Start from the decision, not the data
Every pipeline exists to support some downstream decision or action. If that decision is made once a day by a human reviewing a report, streaming ingestion underneath it adds cost and operational complexity with zero corresponding benefit — the decision cadence, not the data arrival rate, sets the requirement.
Streaming earns its complexity when the downstream action is itself automated and needs to react within seconds — fraud blocking, real-time personalisation, operational alerting. Outside that category, it is usually solving a problem nobody actually has.
The hidden cost streaming architecture adds
Streaming systems introduce operational concerns batch pipelines mostly avoid: exactly-once processing semantics, out-of-order event handling, backpressure management, and a permanently running infrastructure footprint rather than a scheduled job.
None of this is prohibitive when the latency requirement genuinely justifies it. It is simply a cost that should be weighed consciously, not assumed away because streaming is the fashionable default.
A simple decision test
Ask: what is the maximum acceptable delay between an event occurring and a decision being made on it, and is that decision automated or human. If the answer is minutes-to-hours and human-reviewed, batch wins on cost and simplicity. If it's seconds and automated, streaming is earning its keep.
Most organisations that run this test honestly find a smaller fraction of their pipelines genuinely need streaming than their current architecture assumes.
Streaming is a latency decision wearing an architecture costume. Ask the latency question first.MNB Data Practice
Key actions
- Identify the downstream decision cadence before choosing architecture
- Reserve streaming for automated, seconds-scale reactions
- Weigh exactly-once and backpressure complexity consciously
- Re-test the latency requirement before defaulting to streaming for new pipelines