Blog
Engineering notes from a visual Kafka Streams builder: client-library traps, testing stories, and the reasoning behind design decisions. RSS.
- Calling an external service from Kafka Streams: size the threads, budget every call 31 Oct 2026
A blocking call in a mapValues caps a stream thread at one record per call. Size the threads for it by Little's law. Put a deadline around every call, and keep the calls between two polls under half of max.poll.interval.ms. Measured on one broker, Kafka Streams 4.2.0.
- What max.poll.interval.ms bounds in Kafka Streams, measured 21 Oct 2026
Kafka Streams divides its work into tasks over threads and instances. A thread keeps its share only by polling within max.poll.interval.ms, and a slow remote call strains that. The poll interval bounds a round of calls, not one call. A thread that had been fast is evicted at the defaults. A timeout on every call and a capped max.poll.records keep a thread in its group. Kafka Streams 4.2.0, measured, each run predicted before it ran.
- Debezium change events in Kafka Streams: flatten, route, and survive the delete 22 Sept 2026
A Debezium change event is an envelope: before, after, op, source, and a timestamp or three. The first three things everyone does with it in Kafka Streams are the same — flatten after, route by op, and survive the delete, whose after is null and which is followed by a tombstone. Each is one expression, and each can be proved against a captured change event before anything is deployed.
- A Kafka Streams topology starts in a sandbox, not on your cluster 19 Sept 2026
The expensive part of a Kafka Streams topology is finding out whether the logic is right, and the place to find out is a stack of your own that you are free to break: one command, its own broker, your schemas, a sample of your records. Design it there, prove it against real records, and carry it to the cluster that matters as a file.
- Monitoring Kafka Streams applications: the state a cluster view cannot show 14 Sept 2026
Monitoring Kafka Streams from the cluster shows groups, offsets and lag. What the application is doing — restoring a store, lagging on a standby, losing threads — is in the process, not on the cluster. Why we built a console around the stream, and the State tab that shows what the process knows.
- The compiler was never what you wanted 3 Sept 2026
Reading an Avro topic from Kafka Streams in Java is seven steps, and five of them run again every time you need a field somebody added. What that buys is a compiler — and a compiler answers a narrower question than the one you actually have.
- SpEL in Kafka Streams: from evaluator to editor 31 Aug 2026
Spring's SpEL tokenizer is package-private. What it takes to build an editor on an evaluator: highlighting, completions, and errors a user can act on.
- Kafka Streams topologies you can draw and run 19 Aug 2026
A Kafka Streams diagram shows the shape but not the logic. What changes when you draw the topology with the expressions inside, checked on real records.
- KafkaStreams.state() says REBALANCING. Every thread is dead. 2 Aug 2026
A Kafka Streams client whose processing threads have all died keeps reporting REBALANCING — forever. Why it happens, how to detect it, and what any tool showing raw state() should do instead.