A typical Kafka console reads the cluster: topics, partitions, consumer groups, committed offsets, lag. That is the right picture for most of what a console is for.

A Kafka Streams application is a client of that cluster, and from the cluster’s side it looks like what it is: a consumer group with members, some committed offsets, a few internal topics with a naming pattern.

What the application is doing is not on the cluster. Whether it is restoring a store from its changelog, and how far along it is. Whether a standby is keeping up with its active. Whether the threads Kafka reports as alive are serving the tasks assigned to them. How many times it has rebalanced since it started and how long the last one took. All of that is inside the process, in listeners and gauges the Kafka Streams API exposes to the application itself.

The restore case is the sharpest. The consumer that replays a changelog is not a member of any group, commits no offset, and keeps its position in a local checkpoint file. So while an instance that started with an empty state directory replays millions of records before it can process a single new one, the cluster shows a stable group and, if input keeps arriving, a lag that grows for no reason it can show. The application is doing the one thing that decides when it will be available again, and the cluster has no way to say so.

Anyone who has explained that silence to a colleague, or to a customer on an upgrade call, knows the shape of the problem: the application was fine, or was not, and the picture in front of everyone was of something else.

We built Alginte around the stream because of this gap, on both sides of it.

Visibility into the stream’s shape

The first side was design time. A Kafka Streams topology is a graph, and the DSL, the describe output and every diagram in every article draw it as one, but the drawing never contains what the operators do. So we build the stream as the graph it is: in the modeler you draw the topology, node by node and edge by edge, and the boxes get their bodies.

Each operator carries the expression it applies, and the editor shows a record that is really in the source topic and what that expression makes of it. The derived shape at every node is marked as inferred rather than asserted. You see the stream’s shape, and what it does to your data, before it exists on the cluster. That is visibility into the stream as a thing you are building.

Visibility into the stream’s operation

The second side is run time, available since 0.11.0. Because the console runs the stream, it holds the Kafka Streams client and can read it directly: no agent, no scraping, nothing written to the cluster.

A State tab on every deployed stream shows what the process knows. A restore banner appears only while a store replays its changelog, showing how many of the records it has replayed.

The lifecycle is the state the console derives, not only the one Kafka reports, because the raw client state misleads in two known ways: the console reads REBALANCING with every thread dead as ERROR, and RUNNING with assigned tasks served by nobody as DEGRADED, with the shortfall in red.

Every store partition is listed with the one figure that matters most, read by its role: records left to replay while restoring, distance behind the active on a standby, and the plain changelog lag at steady state. Threads, rebalances with their durations and an event timeline complete it. Beside all of it sit the cluster’s own facts about the same stream, group state, committed offsets, changelog topics, so the two views can be read against each other.

A stream replaying 7.5 million changelog records after its state directory was wiped. The cluster’s view of the same moment is a stable group and no sign of why.

The same page opens for a Kafka Streams application the console does not run, on any cluster it reads with the usual read-only rights. There it shows what the cluster can say, the group, the offsets and lag, the changelog topics and how far each reaches, and marks the rest as unseen: no lifecycle, because a group reads stable while threads die and we would rather show absence than invent one; no restore progress, no store sizes. Every figure the observer cannot see is a dash, never a zero. The docs describe which view you get in each case.

An application this console does not run: the cluster’s facts, and everything it cannot see marked as absent rather than zero.

That is the depth the cluster allows, and it is where the idea points next. An application you run outside Alginte will be able to report its own state to the console, thread state, restore progress and store sizes included, through a small reporter library added to the application, one call before it starts. The console side of that, an endpoint that accepts what a runtime pushes, exists today; the library follows. And the streams the console runs need not run in the console’s own process forever: a headless runtime that hosts them elsewhere and reports back to the same page is the shape we are building toward. Either way the page stays the same. What changes is how much of the stream it is allowed to see.

Together the two sides are one idea. The modeler lets you see the stream you are building; the State tab lets you see the stream you are running. Same object, both ends of its life, with the console built around it. Martin Kleppmann turned the database inside out; this is the console turned inside out, the stream in the middle and the console around it.

What seeing it does not change

Seeing a restore does not make it faster. A large store still takes the minutes it takes to replay; what changes is that those minutes are a number on a page with a bar beside it instead of an unexplained silence. Seeing a dead thread does not bring it back; what changes is that the person who has to decide what to do can tell it from an ordinary rebalance.

See what it is doing

A stateful stream application is a commitment: it holds state, it has to restore it, and it can fail in ways the cluster does not show. Run it where you can see it, so that its state is a page anyone on the team can read, not a question for whoever wrote it. That is what the console is for: a stream you can see while you design it, and a stream you can see while it runs.