Kafka Streams is a stream-processing library that runs inside your application — no separate cluster, no Flink or Spark. It’s powerful and popular. It’s also more complex than a plain Kafka consumer, and the jump from “using Kafka” to “using Kafka Streams” is bigger than people expect.

What Kafka Streams gives you

Over a plain KafkaListener:

  • Windowed aggregations — “count events per user per 5-minute window”
  • Joins — combine events from two topics based on a key
  • State stores — local embedded KV store (RocksDB) for keeping aggregation state
  • Exactly-once — stronger guarantees than plain consumers (with trade-offs)
  • DSL for topology — declarative pipeline definition

For use cases that need these, Streams is genuinely good. For use cases that don’t, a plain consumer is simpler.

A taste

Count order events per customer:

StreamsBuilder builder = new StreamsBuilder();

builder.<String, OrderEvent>stream("orders")
    .groupBy((key, value) -> value.customerId())
    .count(Materialized.as("order-counts-by-customer"))
    .toStream()
    .to("customer-order-counts");

KafkaStreams streams = new KafkaStreams(builder.build(), config);
streams.start();

What happened:

  • Consume from orders
  • Group by customer ID
  • Count occurrences (stateful — uses embedded state store)
  • Publish counts to output topic

A plain consumer version of the same logic would be 4-5× more code, with manual state management.

The operational cost

Kafka Streams apps are not plain services. Key differences:

State stores on disk. Each pod has a local RocksDB. Under /var/lib/streams/.... Backed up to Kafka (changelog topic) for failover.

Restoration takes time. When a pod starts, it restores state from the changelog. Seconds for small state; minutes for large. Deploys are slower.

Rebalancing is painful. Partition reassignment means moving state stores. 2.6+ added warmup replicas to mitigate, but still more expensive than plain consumer rebalancing.

Local storage requirements. State store grows with data. Persistent volumes needed.

Internal topics multiply. Streams creates changelog and repartition topics automatically. What started as 3 topics becomes 15. Retention + monitoring discipline matters more.

When Streams is right

  • Windowed analytics that must be low-latency
  • Joining events from multiple topics to produce enriched events
  • Stateful business logic that must survive restarts
  • Exactly-once required across a multi-step pipeline

When a plain consumer is better

  • Simple transformations — consume, transform, publish
  • Stateless processing — each event is independent
  • Side effects dominate — call a service, send an email, write to DB
  • Small teams that don’t want the operational overhead

Common mistakes

Starting with Kafka Streams for simple pipelines. A @KafkaListener that transforms and re-publishes is cleaner than a Streams topology. Reach for Streams when you need its specific features.

Ignoring state-store sizing. State grows with data. Disk fills. No one notices until the pod dies. Monitor state-store size from day one.

Not testing rebalance behavior. Works great in dev with one instance. Breaks in prod during rolling deploys. Test rolling deploys in staging.

Using Streams for side effects. “When this event arrives, send an email.” Not Streams’s job — use a plain consumer.

Treating it like a database. Interactive queries against the state store are possible but fragile. Prefer publishing to a read-optimized store (another topic, or materialized view).

Exactly-once semantics

Kafka Streams can provide exactly-once processing (input read + state update + output write as one atomic transaction). Enable with:

processing.guarantee=exactly_once_v2

Trade-off: slower throughput (transactional commits). Most teams don’t need strict exactly-once; at-least-once + idempotent downstream is simpler.

Alternatives

  • Plain Kafka consumer — for simple transformations
  • Apache Flink — for complex stream processing at scale; separate cluster needed
  • ksqlDB — SQL-like interface to Kafka Streams; great for analytics-style queries
  • Materialize, RisingWave — newer streaming DBs for stream+SQL work

For most Java services, Kafka Streams inside the service is the right balance when you need stream processing.

Testing

Kafka Streams has a TopologyTestDriver for unit testing topologies without running a Kafka cluster:

TopologyTestDriver driver = new TopologyTestDriver(topology, config);
TestInputTopic<String, OrderEvent> input = driver.createInputTopic("orders", keySerde, valueSerde);
TestOutputTopic<String, Long> output = driver.createOutputTopic("customer-order-counts", keySerde, longSerde);

input.pipeInput("c-1", new OrderEvent(...));
assertThat(output.readKeyValue()).isEqualTo(new KeyValue<>("c-1", 1L));

Fast, deterministic, no infrastructure. Use it liberally.

Closing note

Kafka Streams is the right tool for a specific class of problems — stateful, windowed, or joined stream processing. It’s the wrong tool for “we have Kafka and want to transform messages”. Match the tool to the problem; default to plain consumers until a requirement forces you up the ladder. The teams I’ve seen succeed with Streams committed to its operational model (state store backups, careful deploys, monitoring) from day one. The ones that didn’t ended up migrating away after a few incidents.