A Note on Kafka Streams

The Lightweight Alternative

Spark and Flink are powerful, distributed clusters that require significant infrastructure to manage: cluster provisioning, resource managers (YARN or Kubernetes), monitoring, and operational expertise.

Kafka Streams is a fundamentally different approach. It is not a cluster. It is a regular Java library that you add as a dependency to your microservice application. It runs within your application's JVM process — no separate cluster, no separate deployment, no separate operational team.

// Kafka Streams runs inside your regular Java application
StreamsBuilder builder = new StreamsBuilder();

KStream<String, String> transactions = builder.stream("transactions-topic");

KTable<String, Long> fraudAlerts = transactions
    .filter((cardId, event) -> isSuspicious(event))
    .groupByKey()
    .count();

fraudAlerts.toStream().to("fraud-alerts-topic");

KafkaStreams streams = new KafkaStreams(builder.build(), config);
streams.start();

How it scales: Kafka Streams achieves parallelism by leveraging Kafka's own partitioning. If your Kafka topic has 12 partitions, you can run 12 instances of your application, and each instance will process a subset of partitions. Scaling is as simple as deploying more instances of your service.

Pros:
- Extremely simple to deploy — it's just a library in your existing service.
- No separate cluster to manage, monitor, or pay for.
- Scales naturally with Kafka's partition model.
- Excellent for event-driven microservices that need stream processing.

Cons:
- Less feature-rich than Flink for extremely complex stateful processing (e.g., complex event pattern detection, advanced windowing strategies).
- Scope is limited to data that lives in Kafka — you can't join a Kafka stream with a database table as easily as you can in Flink.
- State is stored locally on each application instance, which can complicate rebalancing when you add or remove instances.

It's an excellent choice for building event-driven microservices that need to do stream processing without the operational burden of a dedicated Flink or Spark cluster.