In my environment, real-time data pipelines often deal with events that arrive late due to network delays, device buffering, or timezone mismatches. If you're using Spark Structured Streaming and ignoring this, you risk silently dropping data or producing incorrect aggregations. I've seen this happen in e-commerce pipelines where order events from mobile apps lag by minutes, causing revenue reports to undercount. The fix isn't complex, but it requires understanding two key concepts: watermark and allowedLateness.
What is a watermark in Spark Structured Streaming?
A watermark defines how long Spark waits for late data before considering it too late to process. When you set a watermark on an event-time column, Spark tracks the maximum event time seen so far and subtracts your delay threshold. Any data with an event timestamp older than the current watermark is dropped. This prevents indefinite state growth and allows Spark to finalize results.
For example, if your events should arrive within 2 minutes of their timestamp, you set a watermark of "2 minutes". Spark will then maintain state only for events within the last 2 minutes of event time, cleaning up older state safely.
How allowedLateness complements watermark
While watermark tells Spark when to drop late data, allowedLateness specifies how much lateness you're willing to tolerate beyond the watermark for updating results. It doesn't extend how long Spark keeps state — that's still governed by the watermark — but it allows late-arriving data (within the allowedLateness window) to trigger updates to already-committed results.
Think of it this way: watermark says "I'll stop tracking state after this point", while allowedLateness says "but if data arrives shortly after, still use it to update outputs". This is crucial for use cases like sessionization or real-time dashboards where you want to correct recent results when stragglers arrive.
Practical example: processing orders with 3-minute latency tolerance
Here’s how I configure a streaming query that handles late-arriving order events:
val orderEvents = spark
.readStream
.format("kafka")
.option("subscribe", "orders")
.option("startingOffsets", "latest")
.load()
.selectExpr("CAST(value AS STRING)")
.select(from_json(col("value"), orderSchema).as("data"))
.select("data.order_id", "data.amount", "data.event_time")
.withWatermark("event_time", "3 minutes")
val hourlySales = orderEvents
.groupBy(
window(col("event_time"), "1 hour"),
col("order_id")
)
.agg(sum("amount").as("total_amount"))
val query = hourlySales
.writeStream
.outputMode("update")
.format("console")
.option("truncate", "false")
.trigger(Trigger.ProcessingTime("1 minute"))
.option("allowedLateness", "3 minutes")
.start()In this setup:
- Watermark at "3 minutes" means Spark drops any order_event with an event_time older than (current max event_time - 3 minutes).
- allowedLateness of "3 minutes" allows events arriving up to 3 minutes after the watermark to still update the window aggregates.
Without allowedLateness, late data would be ignored after the watermark passes. Without watermark, state would grow unboundedly as Spark tries to hold all historical data.
Common pitfalls I’ve encountered
One mistake I see often is setting allowedLateness longer than the watermark. That doesn’t make sense — allowedLateness can’t exceed your watermark because Spark already dropped the state. If you set watermark to "1 minute" and allowedLateness to "5 minutes", Spark will still drop data after 1 minute of lateness; the extra 4 minutes are ignored.
Another issue is confusing processing time with event time. If your timestamps are skewed (e.g., devices with wrong clocks), your watermark logic breaks. Always validate event_time ingestion — I’ve used a simple metric to lag between processing_time and event_time to detect clock drift.
Monitoring and tuning
I monitor the following in my Spark UI and logs:
maxEventTimeandwatermarkmetrics from StreamingQueryListener- Number of dropped rows due to watermark (available in recent Spark versions)
- Lag between current time and latest event_time in partitions
If you see steady increases in dropped rows, your watermark may be too tight. If state size keeps growing, your watermark is too loose or missing. Adjust in 30-second increments based on your observed latency distribution.
When to use this pattern
This approach works best when:
- You have a known upper bound on latency (e.g., "99% of events arrive within 2 minutes")
- You need incremental updates to results (outputMode = "update")
- You can tolerate slight corrections to recent windows
For financial reconciliation where lateness must be zero, you might instead divert late data to a dead-letter queue for batch repair. But for most operational dashboards and KPIs, watermark + allowedLateness gives you the right balance of correctness and efficiency.
Final thoughts
I’ve used this pattern across Kafka, Kinesis, and Delta Lake sources in production. It’s not just about avoiding data loss — it’s about building trust in your real-time systems. When stakeholders see that your numbers adjust gracefully for late data, they stop questioning the pipeline’s reliability.
If you’re starting with Structured Streaming, define your latency SLA first. Then set watermark to match it, and use allowedLateness to give yourself a small buffer for output corrections. Test with a latency injector tool (like Kafka’s kafka-producer-perf-test with custom timestamps) to validate behavior under load.
As I mentioned before in my post about [AI-assisted log analysis](https://furkanikkan.com/urun/ai-destekli-log-analizi-syslog-tan-gecici-iptables-kurali-uretme-101), observability is key — pair this with metrics on watermark lag and update frequency to catch regressions early.
Cover image: Unknown · PDM (Wikimedia Commons / kamu malı) · https://commons.wikimedia.org/wiki/File:Austrian_stock_market_index.jpg