Alginte engineering blog

The compiler was never what you wanted

Thu, 03 Sep 2026 00:00:00 GMT

You have an orders topic on a Kafka cluster, its values encoded with Avro against a schema in the Schema Registry. You want the orders worth more than fifty euros on a topic of their own, and you have decided to do it with Kafka Streams — a JVM library, your code, your deployment.

The schema has five fields:

{"type": "record", "name": "Order", "namespace": "com.alginte.demo",
 "fields": [
   {"name": "orderId",    "type": "string"},
   {"name": "customerId", "type": "string"},
   {"name": "item",       "type": "string"},
   {"name": "quantity",   "type": "int"},
   {"name": "priceEur",   "type": "double"}]}

You want one line of logic over them: quantity * priceEur > 50.

Here is everything standing between that line and a topic of big orders.

Seven steps

The route Confluent's own examples take, and many projects with them:

Get the schema out of the registry and into your repository as an .avsc — or, if your team owns the schema in the repository and publishes it to the registry, the other way round. Whichever copy you call the source, there are now two that can disagree.
Add the code generator to your build.
Configure it — source and output directories, and the string type.
Build, producing Order.java under target/generated-sources.
Write the topology against the generated class.
Package the application, with the schema, the class and the serde.
Deploy it somewhere that runs a JVM.

Steps 2 and 3 are this, once — in Maven, though Gradle's equivalent has the same shape:

<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.1</version>
  <executions><execution>
    <phase>generate-sources</phase>
    <goals><goal>schema</goal></goals>
    <configuration>
      <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
      <!-- without this, string fields generate as CharSequence, not String;
           Confluent's own examples set it for the same reason -->
      <stringType>String</stringType>
    </configuration>
  </execution></executions>
</plugin>

And step 5 is the part you actually wanted to write:

builder.stream("orders", Consumed.with(Serdes.Void(), orderSerde))
       .filter((key, order) -> order.getQuantity() * order.getPriceEur() > 50)
       .to("big-orders", Produced.with(Serdes.Void(), orderSerde));

Seven steps, one of them the predicate. Two of them — adding the plugin and configuring it — you do that once.

The filter is deliberately trivial — a real topology joins, aggregates and branches — but the seven steps are identical for twenty operators, because they are charged per project rather than per line of logic. And application.id, bootstrap.servers, the registry URL and your cluster's authentication are missing from the list because they are the price of running against a real cluster and registry, not the price of generating classes.

Those two are not the cost.

Then you want to use a new field

Somebody adds region to the schema. While it sits there unused you are fine — Avro resolves the writer's schema against yours, your generated class does not know the field exists, and nothing needs rebuilding. Schema evolution is doing its job.

Then somebody asks for EU orders only.

The change to your logic is one term: && order.getRegion().equals("EU"). The change to your project is steps 1, 4, 5, 6 and 7 — pull the new schema, regenerate, rewrite, repackage, redeploy.

That is the actual price, and it is charged not per schema change but per schema change you need. Which, over the life of a pipeline, is many of them: fields get added because somebody intends to use them.

What the seven steps buy

They buy the compiler. order.getQuantiy() does not compile. Rename a field, regenerate, and every stale use site turns red before anything runs. The IDE completes field names. Refactoring works.

That is worth having. It is also answering a narrower question than the one you actually have.

The compiler can tell you that getQuantiy() is not a method. It cannot tell you whether quantity * priceEur > 50 is the predicate you meant, whether it matches any record on the topic, or whether the field you are multiplying holds what you think it holds. For that, the seven steps have one answer: deploy it and look.

So the loop you are really in is not edit, compile. It is edit, compile, package, deploy, produce a record, read the output — and it costs the same whether the expression was right or wrong.

A tighter loop

There is another shape for this, and it is the one we build: Alginte, a browser-based topology builder that assembles it at runtime from topics you pick, instead of compiling it into an application you ship.

Point it at orders. The schema comes from the registry at runtime, the five field names arrive as completions, and the predicate is a string — written in SpEL, Spring's expression language, evaluated once per record:

value.get('quantity') * value.get('priceEur') > 50

That expression is evaluated against a real record from the topic while it is being typed — a record off the partition rather than a mock or a fixture, with the answer beside it.

When it is right, it deploys as a Kafka Streams topology: the same library, the same KafkaStreams client, the same rebalances, state stores and changelog topics you would have got from the seven steps. No separate engine is involved. The only thing that changed is how the topology was written.

No .avsc in a repository, no plugin, no target/generated-sources.

The record on the in line is a real one off orders — quantity 3, priceEur 17. The out line is whatever the expression currently returns, and it follows the expression as it is edited: 51, which is the number the > 50 filter is about to judge. Nothing has been deployed.

Mistakes surface in the same place. A method that does not exist on the type the record actually carries is reported while it is being typed, naming the type — the post about building that editor has that one on camera.

That is not a compiler. It is the question the compiler could not answer, asked against real data, answered in seconds. And when somebody adds a sixth field and you want to use it, you type its name.

What you give up

The compiler.

value.get('quantiy') is a valid expression — javac never sees it, so nothing rejects it before it runs. Rename across the project goes too, and the unit tests that constructed Order objects.

What you do not lose is the typo itself. On Avro a misspelt field throws at access — Not a valid schema field: quantiy — so it fails in the editor and fails deployed, identically. That is not the compiler's guarantee, which is made before anything runs. It is the same failure arriving in both places at the same moment: a weaker promise, and a real one.

Some of it returns in a different form — field names completed from the registry, so the typo is never offered; the expression checked against a real record as you type; the same failure surfacing in the editor that would surface in the deployed topology. A different guarantee, weaker in some places, stronger in one.

What does not go is the review. A topology drawn here exports as JSON and imports back, so the thing a reviewer reads and the thing git keeps is a file, not a browser session — the build step is gone, the artifact is not.

You can have the generic half of this without any of the rest. Nothing stops you writing a Java topology that reads GenericRecord and never generates a class: steps 1 to 4 disappear, and so does the compiler, since record.get("quantiy") is a string lookup that javac will not check either. What you are left with is the worst of both — no type safety, and steps 6 and 7 still in front of you.

The generic types are not the point. The loop around them is: the completions, the record on screen, and the deploy that is a click rather than a pipeline.

What that buys is a faster answer to the question you actually had.

And if you would rather write the Java regardless — the stream is one part of a larger application, or your team works that way, or any of the other good reasons — the loop is still worth having first. Draw it here, get the expressions right against real records, then spend the seven steps on logic you already know works. That is a better use of an afternoon than finding out after the deploy.

This came out of building Alginte, a visual Kafka Streams builder — self-hosted, free to run.

SpEL in Kafka Streams: from evaluator to editor

Mon, 31 Aug 2026 00:00:00 GMT

A Kafka Streams application is two things: a graph, and the logic inside it. The DSL writes both in one chain — operators like mapValues, filter and join become the nodes, the chaining becomes the edges, and the lambda inside each operator is where the work happens:

record Enriched(String customer, String product, double total) { }

builder.stream("orders", Consumed.with(Serdes.Void(), orderSerde))
       .mapValues(value -> new Enriched(
               value.getCustomerId(),
               value.getItem(),
               value.getQuantity() * value.getPriceEur()))
       .filter((key, value) -> value.total() > 50)
       .to("orders-enriched", Produced.with(Serdes.Void(), enrichedSerde));

A graph is a picture, so we draw it: operators dragged onto a canvas and wired together.

The same four operators as the Java above, drawn rather than chained — one box each for stream, mapValues, filter and to. The two that need an expression each carry a red badge, and the step above counts them; the source and sink already have their topics. The graph is finished and the contents are not. Watching it built, and what happens after, is the subject of an earlier post.

The body of an operator is not a picture. It is a small computation over one record — and it is the only part of the whole thing doing anything specific to your data. That part, and only that part, is what an expression language is for.

Written as an expression, that body needs no class and no build around it:

{'customer': value.get('customerId'),
 'product': value.get('item'),
 'total': value.get('quantity') * value.get('priceEur')}

The filter's body is one line: value.get('total') > 50.

Writing those bodies in Java is not the hard part. Finding out whether they are right is the hard part: compile, package, point the app at a broker, produce a few records, read what came out. Every step is routine, the whole loop is slow, and it takes exactly as long whether the expression was correct or not.

Evaluated as an expression against a real record while you type, that loop collapses. It widens the set of people who can experiment with stream processing, and it makes learning it cheap — the barrier was never the DSL, it was everything you had to stand up before the DSL would run.

Why SpEL

It is built for this. SpEL has driven routers and transformers in Spring Integration for years: a language built to take expressions at runtime.

It is already there. In a Spring application it arrives on the classpath with the framework.

It can be contained. A Kafka Streams lambda is compiled by whoever owns the deployment — your classpath, your risk. An expression typed into a browser arrives at runtime and runs in a process the author may not own. So containment comes first: a restricted evaluation context, with no type references, no constructors and no bean resolution. Spring provides that deliberately, which is what makes the whole approach possible — though the honest word is containment rather than safety.

There is a tokenizer to reuse. That expression needs a real editor around it, and what that editor shows has to agree with what will actually run. The surest way is to tokenize with the same code the evaluator uses. SpEL has one. Getting hold of it is where this gets interesting.

Where the knowledge comes from

In the Java above, the knowledge came from types — a class on the classpath, there at build time, whether generated from a schema or written by hand.

An expression editor in a browser has none of it. No compile step, no class to consult — the only things that know anything about the data are the Schema Registry and the records sitting on the topic. Everything the editor offers has to come from those two, while someone is typing.

Drawing the graph is the easy half, and on its own it buys very little: a canvas with a bare text box for the logic trades away the help the IDE was giving and offers nothing in its place. Separating the two only pays off if the expression half gets something of its own.

It will not be an IDE — but it has what an IDE cannot: the data. An IDE knows priceEur is a number; it does not know this record's is 17. One expression, evaluated against a real record as you type, is closer to a REPL than to autocompletion.

The editor is Monaco, the component behind VS Code, so it already knows how to highlight tokens, show a completion list and put a marker on a line. What it knows nothing about is SpEL, or your data.

Completions from types, and from content

Completions come from two different places, and the difference is worth keeping visible.

At a source node — reading a topic whose values have a registered schema — the completion list is the schema's own fields with their declared types. That is a contract: something else asserted it, and it holds for every record on the topic.

One operator downstream, after a mapValues, there is no schema any more. The value is whatever the expression returned. The fields offered there are derived by evaluating that expression against a real record and looking at what came out — and the UI marks them (inferred), because that is what they are: an observation of the sample in hand, not a promise. Draw another record and they may differ; if the data is heterogeneous, no single sample will say so.

Inference is not the last word. A node can declare its output in the same vocabulary a source uses — a serde per side, optionally a registered subject — and a node that declares one becomes a fresh start: its completions are a contract again. What you cannot do is type a list of field names, and that is deliberate. Fields belong to schemas, and a shape worth describing is usually a shape worth registering.

The same dropdown at two different nodes, two kinds of knowledge, labelled differently. It would be easy to present both as "fields", and it would be wrong.

An empty topic is not a dead end. With no record to draw, the sample is randomly generated from the registered schema instead, and the preview says which one it is holding. That keeps the editor working before any data exists — while being the weaker evidence of the two, since it demonstrates the shape rather than your data.

One operator after the mapValues, the filter offers customer, product and total — none of which exist in any schema. They are there because the expression on the node behind was evaluated against a real record, and the type hint says so. In the source node the same list would have come from the registry instead. The earlier post has this as a clip.

What an expression receives, and returns

All of that depends on what the deserializer actually hands over — and that is rarely the Java type you would guess from the schema. Avro gives you a GenericRecord, and a field the schema calls a string does not arrive as a String: it arrives as org.apache.avro.util.Utf8, a CharSequence wrapping the raw bytes. Equality still works — value.get('item') == 'Kettle' matches on the orders stream — because SpEL compares CharSequence content. .contains(...) does not, because it is a String method that CharSequence never declared, so it needs a .toString() first. That is the kind of detail that lives in serde Javadoc rather than anywhere you would think to look, and it costs an afternoon the first time.

The same afternoon, spent in under thirty seconds. .contains(...) on an Avro string, the message naming org.apache.avro.util.Utf8 rather than "invalid expression", and .toString() clearing it.

Types are one half of what an editor can offer; the record is the other. Each node's sample preview puts that record in and the evaluated output out, side by side, re-running as the expression changes — so what an expression actually returns is on screen while it is still being written, against the data it will run on.

in is a real record off orders; out is whatever the expression returns, and it follows the expression as that changes — value.get('item') gives "Kettle", then the intro's own quantity * priceEur gives 51, the value the > 50 filter downstream exists to judge. Nothing has been deployed, and the editor never claims either answer is correct. It only makes them impossible to miss.

Avro's type names leak into error messages too, which the next section has to deal with.

Errors a user can act on

SpelCompilerMode.IMMEDIATE exists to make expressions faster by compiling them to bytecode. We use it in the validator, as a type checker. Compiling forces type resolution to happen while someone is still typing, which is exactly when you want to hear that a method does not exist on the type you actually have.

Catching it early, at the right line and column, is half of a diagnostic. The other half is what it says.

SpEL's messages assume a developer who knows the JVM types involved. Ours are read by someone who has never heard of GenericData$Record and should not have to. So the validator rewrites them.

An indexing failure — someone reasonably tries value['item'] — becomes:

indexing with ['...'] is not supported on schema-backed records — use get('fieldName') instead (the completions suggest it). A deployed topology fails the same way on Avro / Protobuf values.

The fix, a pointer at where it is offered, and a promise about the runtime. That last clause matters most: it tells the reader this is not an editor quirk to be worked around, because the deployed topology fails identically.

Everything else gets a blunter rule: any message mentioning an internal sample type has that type replaced with "the sample record". Someone who writes a bad expression should learn what they did, not what class the deserializer happened to return. A few cases are worth naming precisely and get hand-written messages, like "Invalid map access syntax. Use value['key'] without a dot."

The rewriting covers Avro's generic types and Jackson's nodes. Protobuf is not in that set.

The tokenizer you cannot reach

An editor needs three things an evaluator does not provide: syntax highlighting, completions, and errors at the right line and column. All three start from tokens.

Spring has a tokenizer — the one its own parser uses. It is package-private. Against Spring Framework 7.0.8:

class org.springframework.expression.spel.standard.Tokenizer {
  public org.springframework.expression.spel.standard.Tokenizer(java.lang.String);
  public java.util.List<org.springframework.expression.spel.standard.Token> process();
  static {};
}

The constructor and process() are already public. The class is not, so from outside the package there is no way to call them.

The public surface of that package is SpelExpressionParser, SpelExpression and SpelCompiler. Parsing, evaluation, compilation — no tokenization.

SpelExpression does expose a parse tree — getAST() returns SpelNodes with start and end positions, which looks like exactly the supported route we missed. But a parser needs a valid expression, and an editor spends most of its life holding an invalid one. value. is precisely when completions have to fire, and it does not parse; nor does this post's own map literal before its closing brace. Both tokenize.

So we copied Tokenizer, Token and TokenKind from Spring's spel.standard package, under Apache-2.0, into a package of the same name in our own tree — which is what puts those package-private types within reach.

All of it exists to enable one line:

List<Token> tokens = new Tokenizer(inputData).process();

What we add on top is thin, and most of it is a coordinate transform. Spring reports absolute character offsets; editors want line and column. So each Token becomes a SpELToken(start, end, line, charPositionInLine, type, modifiers) — the shape Monaco and LSP want — and goes over a WebSocket to the browser. There they are served through a DocumentSemanticTokensProvider, which is the highlighting: every colour in the expression comes from the same tokenizer the evaluator parses with, so what the editor shows and what runs cannot drift apart.

That is the whole of it. Spring's authors had already written the hard part; nothing was missing but a way to call it. What we carry is not complexity — it is their file.

What it costs

SpEL is not Java. Broadening who can write a transformation costs the type system, the IDE, refactoring, unit tests, and the entire library ecosystem, in exchange for a string in a text box. That trade is defensible for "what does this do to my records" and indefensible for a thousand-line pipeline.

The editor cannot catch a wrong answer. It catches expressions that are invalid — a method that does not exist, a syntax it cannot parse. One that is type-correct and simply always false passes every check. The preview above will faithfully show you the wrong answer; nothing will tell you that is what it is.

Package-private internals carry no compatibility promise. Our copy of the tokenizer can break between Spring versions with no deprecation cycle, and that is our problem rather than Spring's. We took the risk knowingly; it is still a risk.

Protobuf is opaque — its records arrive as a DynamicMessage, which offers no field access by name — and unlike the rest of this, that one is entirely ours to fix. It is on the list.

Is there a better way to do this?

That is our workaround, and we are not claiming it is the right answer. Package-private is a deliberate choice — internals stay private precisely so they can change — and we went around it.

So the question is genuine. If there is a supported path to a SpEL token stream that we missed, we would much rather be told than go on carrying a copy of somebody else's tokenizer. And if there is not, we are probably not the last people who will want one.

Either way the pieces are all there: the tokenizer works, the errors carry positions, the compiler doubles as a type checker. We would just like a supported way to hold one of them.

This came out of building Alginte, a visual Kafka Streams builder — self-hosted, free to run.

Kafka Streams topologies you can draw and run

Wed, 19 Aug 2026 00:00:00 GMT

You have an orders topic. Each record carries a customer, an item, a quantity, and a price. You want the big ones — total over 50 — in a topic of their own, with that total already calculated.

In Kafka Streams, that is a mapValues and a filter: a dozen lines of real logic.

Getting them to run is a different size of job: a build file, a serde configuration, a jar, somewhere to put it, and a redeploy every time you want to check if your expression was right about the data. The work is small. The apparatus around it is not — and none of it tells you anything until the whole thing is up.

That distance is the subject here. Not that Kafka Streams is hard, because it is not — but that the trip from knowing what you want to watching it happen is longer than a dozen lines of logic deserves.

Start with what Kafka Streams already gives you — most of the picture is there, and the missing part is smaller than it looks. Deploy a topology and ask it to describe itself; here is a real one: four operators, reading orders and writing orders-enriched:

Topologies:
   Sub-topology: 0
    Source: Orders-src (topics: [orders])
      --> order-total-enrich
    Processor: order-total-enrich (stores: [])
      --> big-orders-bigOnly
      <-- Orders-src
    Processor: big-orders-bigOnly (stores: [])
      --> Big-Orders-sink
      <-- order-total-enrich
    Sink: Big-Orders-sink (topic: orders-enriched)
      <-- big-orders-bigOnly

That is a graph. Nodes, directed edges, a topological order. The DSL constructs one; describe() prints one, and it is what anyone sketches on a whiteboard when explaining a pipeline to someone else.

Look at what it describes. order-total-enrich is a processor: its name, its position, what feeds it, and what it feeds. What it does lives somewhere else — the lambda inside mapValues, the entire reason that node exists, is in your source rather than in the description. A topology description captures shape, and captures it completely; behaviour was never its job.

What falls between them is the interesting part. A visual representation of a Kafka Streams application — ours included, until recently — tends to be a picture of the shape. Fill the boxes with real bodies and the picture stops being documentation of the program and becomes the program — something you can draw and run.

The data is already there, and so is its schema

When records are produced through a schema registry — as these were — each one carries the id of the schema it was written against. So the shape is described before any topology exists, and a record and its schema are connected before anyone draws anything — you can watch it work.

Note what the value deserializer is not asked for: a subject. The decode succeeding is the connection; the schema tab afterwards only names it.

(Void on the key side because these records genuinely have null keys — anything else renders noise where there is nothing.)

What goes in the boxes

So what should order-total-enrich produce? A customer, a product, and that total. In a Java project, you would write that as a lambda and compile it; here it is an expression, and this is the whole of it:

{'customer': value.get('customerId'),
 'product':  value.get('item'),
 'total':    value.get('quantity') * value.get('priceEur')}

Evaluated once per record, inside the same JVM that is running the topology. The parse happens up front, and the parsed expression is cached, so the per-record cost is evaluation, not parsing.

Reaching for an expression language rather than something more powerful is not a simplification — it follows from who writes it. A Kafka Streams lambda is compiled by whoever owns the deployment, so trust is implicit and never has to be examined. Move authoring into a console and the author is a user: the code arrives at runtime, from outside, and the process it runs in is yours. The obvious alternative — letting people upload compiled code — answers the same need by handing a stranger arbitrary execution inside that process.

We use Spring's SpEL for this. It is embeddable, it has an evaluation context you can restrict deliberately rather than by accident, and in a Spring application it is already on the classpath.

Which makes the sandbox a precondition rather than a feature. Expressions evaluate against a restricted context: no type references, no constructors, no bean resolution. The honest word for that is containment — we can tell you exactly what is blocked, and we cannot prove that nobody will find a way past it.

Drawing it, and watching where the type survives

Now the graph. Four operators dragged out, wired, named, and pointed at orders:

The beat worth pausing on is the source's serde section, about a third of the way in. The value side fills itself in — Avro, subject orders-value — because a registered subject makes the value type a contract. The key side stays empty until you say so, because there is no orders-key subject and nothing to derive it from.

From the source down, each operator either preserves the value or replaces it. filter, peek, repartition, toStream pass it along unchanged; mapValues ends the contract, because after it the value is whatever your expression returned. That is not a property of any tool — it is how the DSL works, and it is the rule anything deriving types has to follow.

The loop

With a contract at the source and a rule for how it propagates, the editor can say something useful while you type — and a real record underneath answers back. Type, look, adjust: that is the loop.

Two different kinds of knowledge appear in that clip, and the difference matters.

In the mapValues node, the completions are the schema's own fields — customerId, item, quantity, priceEur, with their declared types. That is a contract; it came from the registry.

One node downstream, the filter offers total. There is no total in any schema, in any topic, anywhere. It exists because the expression above computed it, and the shape was observed by evaluating that expression against a real record. The UI marks fields like this (inferred) for exactly that reason: it is one record's observation, not a promise. If your data is heterogeneous, one sample will not tell you so.

The record underneath is pulled from the topic, not fabricated. You can draw a different one, roll a fresh one from the schema, or type the case you are actually worried about — the order with the null field, the quantity nobody expected — and watch what your expression does to it. Whatever you put in there, every downstream node reads the same record, so one story flows through the whole chain.

Inference reads one execution path of one sample, so there are nodes where it will be wrong and nodes where it has nothing to offer. Any node can declare its output instead — a serde per side, and a subject if that shape is registered. A node that declares one becomes a fresh start: the walk resumes from it, and everything below reads what you stated rather than what was observed. What you declare beats what was inferred, and both beat a guess.

It is deliberately not a free-form field list. Fields belong to schemas, and a shape worth describing is usually a shape worth registering — so declaring a node's output asks the same question binding a source does, and takes a contract if one exists.

One thing the completions cannot warn you about: the values are runtime objects, not the Java types their names suggest. A field the schema calls a string does not arrive as a String — it arrives as org.apache.avro.util.Utf8, a CharSequence wrapping the raw bytes. Equality still behaves, because SpEL compares CharSequence content, so value.get('item') == 'Kettle' is true when you expect it to be. But .contains(...) is a String method that CharSequence does not declare, so it needs a .toString() first. It is the kind of detail that lives in serde Javadoc rather than anywhere you would think to look.

Why the loop can be trusted

All of which is worth exactly nothing if the answers differ from what the topology does once it is deployed. A preview that disagrees with the runtime teaches you something false and lets you find out at deploy time — worse than showing you nothing at all.

That is not something you can check from the outside, so here is how we check it. Every expression in the test suite runs through all three surfaces. Two you have already seen: the editor's validator, and the preview under the sample. The third is the topology itself — built by the same code that builds a deployed one, running the same serdes, fed a record, and asked what came out of the sink. That leg runs in-process rather than against a broker, which is the only way to do it per expression at test speed, but nothing about the expression's path is simulated.

All three have to agree: same success or failure, same output once serialized, same verdict from a filter. The cases run across Avro, Protobuf, and JSON Schema, and inside those, nested records, unions, arrays, enums, and logical types — and the set only grows: every disagreement we find becomes a case before it becomes a fix. They are usually small and specific, like arithmetic on an Avro field that the editor flagged and the runtime handled perfectly well.

It runs

Configuration, submit, and the deployed topology with live per-node numbers:

Those badges exist because of a naming decision made much earlier. Runtime metrics are tagged with operator names, so joining them back to the picture means knowing what each operator is called — and Kafka's own answer, KSTREAM-MAPVALUES-0000000003, is generated from graph position and shifts the moment you insert an operator upstream. The names in this topology are derived from the nodes instead, which is why the describe() output at the top of this post reads order-total-enrich. That is worth more than legibility: those names also land in JMX, in log lines, and in the internal topic names on your cluster.

Read them carefully, though. The source reports an exact count; everything downstream reports an attributed one, taken from the surrounding subtopology. For a linear chain, the number is correct, but it cannot tell you which records survived a predicate — the filter's effect shows up on the sink topic's offsets, not on the node's badge.

What this is for

There is a stretch of work at the start of any pipeline that is mostly questions: what is actually in these records, does this field mean what its name suggests, what does my transformation do to the awkward ones. Answering those in a project means writing an app to find out. Answering them here takes the time it takes to type an expression, and the answers come from records that are really on the topic.

The loop also explains an architectural choice. Most Kafka consoles are readers: they ask the cluster questions — what topics exist, where are the consumer groups, what do the metrics say — and never execute anything themselves. Evaluating your expression against a real record and showing you what came out is not a question you can ask Kafka. It means building the topology and running it, so a Kafka Streams application deployed from this console runs inside the console's own process.

Having everything in one place pays off after the deploy, too. The sink topic is a topic like any other: the browser that showed you orders in the first clip will show you orders-enriched, so you can read what your expression actually produced — not just how many records got there. That is the round trip closed: the schema you started from, the records you tested against, and the output you caused, all reachable without leaving. Consumer groups, connectors, and ksqlDB sit on the same sidebar for the same reason, and that is the direction the rest of it keeps moving.

That is a genuine trade, not a free win: it puts your expressions in our JVM, which is why the sandbox above came first. If you already have a console you like for browsing a cluster, keep it — this one is shaped the way it is because authoring needed something that could actually run what you wrote.

What stays out of reach

The derivation stops, and where it stops is the honest part.

At a join, we say nothing. A join emits when a pair co-occurs — same key, both sides, inside the window. Showing you its output means either finding a real pair across both inputs, or inventing the side that has not arrived; the second asserts a match that may never happen, and we are not willing to do it. So the walk halts at a join, and the nodes after it are reported unevaluated rather than guessed — until you declare the join's output yourself, which starts a fresh chain from there. Wire only one input and you get a different answer again, because then the chain can reach it but its output is not modelled.

One record at a time is not coverage. You can cycle through real records, generate them from the schema, and write the awkward case by hand — but nothing enumerates the shapes your stream actually contains, and the loop will never tell you about the one you did not think to try. It is a probe, not a proof.

Protobuf values are opaque to all of this — and this one is ours, not Kafka's. They arrive as DynamicMessage, which has no get(String), so field access from an expression is unavailable, not simply awkward. An expression language can be taught to read a DynamicMessage; we have not done it. Meanwhile, the completions decline to offer fields they know an expression cannot reach, which is the right answer and a thin consolation.

Those three are not the same kind of limit. Sampling one record at a time is not something a different tool fixes; it will still be true in whatever you author with instead. Protobuf access is a gap in what we have built, and it can be closed. The join is neither: declining to guess is a decision, and decisions can be wrong.

What a tool can do is derive what is derivable, say nothing where nothing is knowable, and let you run an expression against a record that actually exists before you commit to it.

Try it, and tell us where it fits

This started with an orders topic and a transformation you could describe in one sentence, with a build and a deploy standing between you and knowing whether it was right. The four clips are that distance getting shorter: the schema read out of the records, the expression checked against one of them before it ships, the topology running with its own numbers on it.

What comes out is a Kafka Streams topology that runs as it stands. And if what you wanted was a shape you now understand well enough to go and write in Java, the value has to survive you doing that, and it does: the fields you confirmed and the transformation you settled on do not care where they end up.

Where it stops being useful is the part we cannot answer alone: the pipelines you would want to explore this way, and the ones where none of this would help.

All of this is in the playground: the orders topic with its registered schema, and the topology from these clips deployed and running against it. One command, and you can take it apart instead of taking our word for it.

KafkaStreams.state() says REBALANCING. Every thread is dead.

Sun, 02 Aug 2026 00:00:00 GMT

We found this one the embarrassing way: a seeded demo stream sat in REBALANCING for a day. Not failing — rebalancing. The UI said so. The Streams client said so. The only place the truth existed was a server log nobody was reading.

The trap

Deploy a Kafka Streams topology whose source topic doesn't exist, and here's the exact sequence (Kafka clients 4.x, but the behaviour is old):

The group leader's assignment fails with INCOMPLETE_SOURCE_TOPIC_METADATA; the member receives the error in its assignment.
The StreamThread logs MissingSourceTopicException, transitions PENDING_SHUTDOWN → DEAD, and does not retry. This is deliberate — a missing source topic is not a transient condition Kafka Streams can wait out.
Crucially, the client-level state machine never follows. The StreamsUncaughtExceptionHandler isn't consulted (the thread shut down; it didn't throw), so there's no PENDING_ERROR → ERROR transition. KafkaStreams.state() last saw a rebalance start, and that's where it stays.

The result: a KafkaStreams instance with zero live threads that reports REBALANCING indefinitely. It will never process a record, never error, and never change state again.

Why every Kafka UI has this bug

If your tool renders KafkaStreams.state() — and that's the obvious, documented thing to render — you have this bug. The state enum simply has no value for "all my threads are dead but nobody told the coordinator layer." REBALANCING is the truthful answer to the wrong question.

The signal that does exist is one call away: metadataForLocalThreads() returns per-thread metadata including each thread's state. A client reporting REBALANCING whose thread set is empty — or whose threads are all DEAD — is not rebalancing. It's gone.

The fix, in two halves

Surface it. We derive the displayed state instead of trusting the raw one:

static KafkaStreams.State effectiveState(KafkaStreams.State state,
                                         Collection<ThreadMetadata> threads) {
    if (state == KafkaStreams.State.REBALANCING
            && (threads.isEmpty()
                || threads.stream().allMatch(t -> "DEAD".equals(t.threadState())))) {
        return KafkaStreams.State.ERROR;
    }
    return state;
}

Genuine rebalances have live threads (STARTING, PARTITIONS_ASSIGNED, …) and pass through untouched. In our end-to-end test, deleting a running stream's source topic flips the reported state to ERROR within seconds — where before it showed REBALANCING until someone read the log.

Prevent the common case. The most frequent way to hit this is a typo'd topic name at deploy time. Since a missing source topic is unrecoverable by design, we now validate every source node's topics against the cluster before building the topology, and fail the deploy with the missing names — one listTopics() round-trip. (Fail-open if the listing itself errors: a broker hiccup shouldn't block a deploy that would have succeeded; the state derivation above is the backstop.)

Neither half needs anything from the broker that isn't already public API.

Takeaways

KafkaStreams.state() is the state of the coordinator conversation, not the health of your processing. Dead threads don't move it.
If you're operating Kafka Streams with your own dashboards: alert on thread liveness (metadataForLocalThreads(), or the alive-stream-threads metric), not on state() != RUNNING.
If you're building a tool: derive, don't relay. The raw state is truthful and useless at the same time.

Both fixes shipped in Alginte 0.7.0. The stuck demo that taught us this now recovers in seconds — and deploying against a typo'd topic tells you the topic's name instead of miming a rebalance.