Batch vs. streaming, explained with a real pipeline

Most teams reach for streaming too early. Here’s a simple batch job that covers 90% of cases before you ever need Kafka:

# daily batch: load, transform, write
import pandas as pd

def run(date):
    df = pd.read_parquet(f"raw/{date}.parquet")
    df = df[df.amount > 0]
    df.to_parquet(f"clean/{date}.parquet")

No brokers, no consumers, no 3am pages. If your data lands once a day, a job like this is enough.

When streaming earns its keep

Reach for streaming when the value of the data decays in seconds — fraud detection, live dashboards, alerting. If a one-hour delay is fine, batch is almost always cheaper to build and run.

Rule of thumb: start with batch, and let a real latency requirement — not excitement — pull you toward streaming.