The unglamorous truth about streaming pipelines
Most of the effort in a streaming platform goes not into the processing but into the edges — the connectors, the schema drift, and the question of what happens when a downstream system is having a bad day.
Streaming architecture diagrams have a recognisable shape. A row of sources on the left, a processing engine in the middle, a row of sinks on the right. The middle box gets the attention, the vendor comparisons and the conference talks.
In our experience the middle box is rarely where the project gets difficult.
The edges are the work
A realistic ingest side spans message brokers, log shippers, cloud queues, object storage and device protocols — Kafka, Flume, Kinesis, Pub/Sub, MQTT, JMS, S3, HDFS, and whatever the enterprise service bus happens to be. The output side is just as varied: warehouses, search indexes, key-value stores, files, sockets, and relational databases fronting applications.
That is a dozen or more integration points, each with its own delivery semantics, failure behaviour, authentication model, and definition of "ordered". None of that difficulty lives in the processing engine, and swapping engines does not make it go away.
Practical consequence: when evaluating a streaming platform, weight connector coverage and connector quality far more heavily than processing features. A missing connector is a project. A slightly less elegant windowing API is an afternoon.
Batch and stream are the same pipeline, eventually
Almost every streaming project acquires a batch requirement. Backfill after a bug. Reprocessing when a definition changes. A historical join the real-time path cannot serve.
Teams that treat these as separate systems end up maintaining two implementations of the same business logic, which drift — and the drift is discovered when the batch and streaming numbers disagree in a meeting.
Unifying batch and stream processing behind one definition of the logic is worth real architectural effort, because the alternative is a permanent, low-grade correctness problem nobody has time to fix.
Schema drift is the silent failure
An upstream team adds a field. Nothing breaks. Weeks later someone notices a metric has been quietly wrong since the change.
Streaming makes this worse than batch, because there is no natural checkpoint where a human looks at the output. The pipeline keeps running, which looks identical to the pipeline working.
The defences are unglamorous and they work: schema registries with compatibility rules, explicit handling for unknown fields rather than silent passthrough, and alerting on record shape as well as record volume.
Metrics, and the difference from alerts
Every streaming platform emits metrics — throughput, lag, error counts. Fewer answer the question that matters at 2am: is this pipeline healthy, and if not, what should I do?
Consumer lag on its own is not a health signal. Lag that is rising while input volume is flat is a health signal. Zero errors and zero output is a serious problem that a naive dashboard renders as green.
Design the alerts against failure modes, not against available metrics. The useful question is not "what can we measure" but "what will go wrong, and would we notice".
What to actually decide first
If you are starting a streaming project, the order of decisions we would recommend:
- Enumerate every source and sink, including the ones nobody has mentioned yet. This list, not the engine choice, determines most of the effort.
- Decide how batch reprocessing will work on day one. It will be needed; deciding later means two codebases.
- Choose a schema strategy before the first pipeline ships, because retrofitting one across live producers is painful.
- Write down the failure modes and what should happen in each. This becomes both the alerting spec and the operational runbook.
- Then pick a processing engine.
The engine choice is the most discussed and, in our experience, the least consequential of those five. The projects that struggle rarely do so because they picked the wrong stream processor. They struggle because the edges were underestimated, batch showed up late, and nobody agreed what "healthy" meant.