Real-time data: choose the right latency and plan for recovery

Define what real time means

We recommend translating the phrase into a measurable delay between an event and its use. An operational alert within seconds and a dashboard refreshed every fifteen minutes do not require the same architecture. Greater freshness can also bring higher costs and operational demands.

Separate component roles

Kafka transports and retains events. Flink processes them, including by event time rather than only arrival time. Its time documentation explains how watermarks track this progress.

At Databricks, the former Delta Live Tables have evolved into Lakeflow pipelines. The official transition page describes the change. Evaluate the selected engine and execution mode rather than inferring latency from a product name.

Test failures and duplicates

Kafka’s processing-guarantee documentation explains that writes to an external system require appropriate coordination. An exactly-once guarantee in one part of a flow does not automatically prevent duplicates in the final application.

We advise testing an outage, recovery, a late event and a schema change. Define idempotency keys and correction rules before production deployment.

Sources and features may change. Check the current version when planning your project.