A Kafka Streams application is two things: a graph, and the logic inside it. The DSL writes both in one chain — operators like mapValues, filter and join become the nodes, the chaining becomes the edges, and the lambda inside each operator is where the work happens:

record Enriched(String customer, String product, double total) { }

builder.stream("orders", Consumed.with(Serdes.Void(), orderSerde))
       .mapValues(value -> new Enriched(
               value.getCustomerId(),
               value.getItem(),
               value.getQuantity() * value.getPriceEur()))
       .filter((key, value) -> value.total() > 50)
       .to("orders-enriched", Produced.with(Serdes.Void(), enrichedSerde));

A graph is a picture, so we draw it: operators dragged onto a canvas and wired together.

The same four operators as the Java above, drawn rather than chained — one box each for stream, mapValues, filter and to. The two that need an expression each carry a red badge, and the step above counts them; the source and sink already have their topics. The graph is finished and the contents are not. Watching it built, and what happens after, is the subject of an earlier post.

The body of an operator is not a picture. It is a small computation over one record — and it is the only part of the whole thing doing anything specific to your data. That part, and only that part, is what an expression language is for.

Written as an expression, that body needs no class and no build around it:

{'customer': value.get('customerId'),
 'product': value.get('item'),
 'total': value.get('quantity') * value.get('priceEur')}

The filter’s body is one line: value.get('total') > 50.

Writing those bodies in Java is not the hard part. Finding out whether they are right is the hard part: compile, package, point the app at a broker, produce a few records, read what came out. Every step is routine, the whole loop is slow, and it takes exactly as long whether the expression was correct or not.

Evaluated as an expression against a real record while you type, that loop collapses. It widens the set of people who can experiment with stream processing, and it makes learning it cheap — the barrier was never the DSL, it was everything you had to stand up before the DSL would run.

Why SpEL

It is built for this. SpEL has driven routers and transformers in Spring Integration for years: a language built to take expressions at runtime.

It is already there. In a Spring application it arrives on the classpath with the framework.

It can be contained. A Kafka Streams lambda is compiled by whoever owns the deployment — your classpath, your risk. An expression typed into a browser arrives at runtime and runs in a process the author may not own. So containment comes first: a restricted evaluation context, with no type references, no constructors and no bean resolution. Spring provides that deliberately, which is what makes the whole approach possible — though the honest word is containment rather than safety.

There is a tokenizer to reuse. That expression needs a real editor around it, and what that editor shows has to agree with what will actually run. The surest way is to tokenize with the same code the evaluator uses. SpEL has one. Getting hold of it is where this gets interesting.

Where the knowledge comes from

In the Java above, the knowledge came from types — a class on the classpath, there at build time, whether generated from a schema or written by hand.

An expression editor in a browser has none of it. No compile step, no class to consult — the only things that know anything about the data are the Schema Registry and the records sitting on the topic. Everything the editor offers has to come from those two, while someone is typing.

Drawing the graph is the easy half, and on its own it buys very little: a canvas with a bare text box for the logic trades away the help the IDE was giving and offers nothing in its place. Separating the two only pays off if the expression half gets something of its own.

It will not be an IDE — but it has what an IDE cannot: the data. An IDE knows priceEur is a number; it does not know this record’s is 17. One expression, evaluated against a real record as you type, is closer to a REPL than to autocompletion.

The editor is Monaco, the component behind VS Code, so it already knows how to highlight tokens, show a completion list and put a marker on a line. What it knows nothing about is SpEL, or your data.

Completions from types, and from content

Completions come from two different places, and the difference is worth keeping visible.

At a source node — reading a topic whose values have a registered schema — the completion list is the schema’s own fields with their declared types. That is a contract: something else asserted it, and it holds for every record on the topic.

One operator downstream, after a mapValues, there is no schema any more. The value is whatever the expression returned. The fields offered there are derived by evaluating that expression against a real record and looking at what came out — and the UI marks them (inferred), because that is what they are: an observation of the sample in hand, not a promise. Draw another record and they may differ; if the data is heterogeneous, no single sample will say so.

Inference is not the last word. A node can declare its output in the same vocabulary a source uses — a serde per side, optionally a registered subject — and a node that declares one becomes a fresh start: its completions are a contract again. What you cannot do is type a list of field names, and that is deliberate. Fields belong to schemas, and a shape worth describing is usually a shape worth registering.

The same dropdown at two different nodes, two kinds of knowledge, labelled differently. It would be easy to present both as “fields”, and it would be wrong.

An empty topic is not a dead end. With no record to draw, the sample is randomly generated from the registered schema instead, and the preview says which one it is holding. That keeps the editor working before any data exists — while being the weaker evidence of the two, since it demonstrates the shape rather than your data.

One operator after the mapValues, the filter offers customer, product and total — none of which exist in any schema. They are there because the expression on the node behind was evaluated against a real record, and the type hint says so. In the source node the same list would have come from the registry instead. The earlier post has this as a clip.

What an expression receives, and returns

All of that depends on what the deserializer actually hands over — and that is rarely the Java type you would guess from the schema. Avro gives you a GenericRecord, and a field the schema calls a string does not arrive as a String: it arrives as org.apache.avro.util.Utf8, a CharSequence wrapping the raw bytes. Equality still works — value.get('item') == 'Kettle' matches on the orders stream — because SpEL compares CharSequence content. .contains(...) does not, because it is a String method that CharSequence never declared, so it needs a .toString() first. That is the kind of detail that lives in serde Javadoc rather than anywhere you would think to look, and it costs an afternoon the first time.

The same afternoon, spent in under thirty seconds. .contains(...) on an Avro string, the message naming org.apache.avro.util.Utf8 rather than “invalid expression”, and .toString() clearing it.

Types are one half of what an editor can offer; the record is the other. Each node’s sample preview puts that record in and the evaluated output out, side by side, re-running as the expression changes — so what an expression actually returns is on screen while it is still being written, against the data it will run on.

in is a real record off orders; out is whatever the expression returns, and it follows the expression as that changes — value.get('item') gives "Kettle", then the intro’s own quantity * priceEur gives 51, the value the > 50 filter downstream exists to judge. Nothing has been deployed, and the editor never claims either answer is correct. It only makes them impossible to miss.

Avro’s type names leak into error messages too, which the next section has to deal with.

Errors a user can act on

SpelCompilerMode.IMMEDIATE exists to make expressions faster by compiling them to bytecode. We use it in the validator, as a type checker. Compiling forces type resolution to happen while someone is still typing, which is exactly when you want to hear that a method does not exist on the type you actually have.

Catching it early, at the right line and column, is half of a diagnostic. The other half is what it says.

SpEL’s messages assume a developer who knows the JVM types involved. Ours are read by someone who has never heard of GenericData$Record and should not have to. So the validator rewrites them.

An indexing failure — someone reasonably tries value['item'] — becomes:

indexing with ['...'] is not supported on schema-backed records — use get('fieldName') instead (the completions suggest it). A deployed topology fails the same way on Avro / Protobuf values.

The fix, a pointer at where it is offered, and a promise about the runtime. That last clause matters most: it tells the reader this is not an editor quirk to be worked around, because the deployed topology fails identically.

Everything else gets a blunter rule: any message mentioning an internal sample type has that type replaced with “the sample record”. Someone who writes a bad expression should learn what they did, not what class the deserializer happened to return. A few cases are worth naming precisely and get hand-written messages, like “Invalid map access syntax. Use value['key'] without a dot.”

The rewriting covers Avro’s generic types and Jackson’s nodes. Protobuf is not in that set.

The tokenizer you cannot reach

An editor needs three things an evaluator does not provide: syntax highlighting, completions, and errors at the right line and column. All three start from tokens.

Spring has a tokenizer — the one its own parser uses. It is package-private. Against Spring Framework 7.0.8:

class org.springframework.expression.spel.standard.Tokenizer {
  public org.springframework.expression.spel.standard.Tokenizer(java.lang.String);
  public java.util.List<org.springframework.expression.spel.standard.Token> process();
  static {};
}

The constructor and process() are already public. The class is not, so from outside the package there is no way to call them.

The public surface of that package is SpelExpressionParser, SpelExpression and SpelCompiler. Parsing, evaluation, compilation — no tokenization.

SpelExpression does expose a parse tree — getAST() returns SpelNodes with start and end positions, which looks like exactly the supported route we missed. But a parser needs a valid expression, and an editor spends most of its life holding an invalid one. value. is precisely when completions have to fire, and it does not parse; nor does this post’s own map literal before its closing brace. Both tokenize.

So we copied Tokenizer, Token and TokenKind from Spring’s spel.standard package, under Apache-2.0, into a package of the same name in our own tree — which is what puts those package-private types within reach.

All of it exists to enable one line:

List<Token> tokens = new Tokenizer(inputData).process();

What we add on top is thin, and most of it is a coordinate transform. Spring reports absolute character offsets; editors want line and column. So each Token becomes a SpELToken(start, end, line, charPositionInLine, type, modifiers) — the shape Monaco and LSP want — and goes over a WebSocket to the browser. There they are served through a DocumentSemanticTokensProvider, which is the highlighting: every colour in the expression comes from the same tokenizer the evaluator parses with, so what the editor shows and what runs cannot drift apart.

That is the whole of it. Spring’s authors had already written the hard part; nothing was missing but a way to call it. What we carry is not complexity — it is their file.

What it costs

SpEL is not Java. Broadening who can write a transformation costs the type system, the IDE, refactoring, unit tests, and the entire library ecosystem, in exchange for a string in a text box. That trade is defensible for “what does this do to my records” and indefensible for a thousand-line pipeline.

The editor cannot catch a wrong answer. It catches expressions that are invalid — a method that does not exist, a syntax it cannot parse. One that is type-correct and simply always false passes every check. The preview above will faithfully show you the wrong answer; nothing will tell you that is what it is.

Package-private internals carry no compatibility promise. Our copy of the tokenizer can break between Spring versions with no deprecation cycle, and that is our problem rather than Spring’s. We took the risk knowingly; it is still a risk.

Protobuf is opaque — its records arrive as a DynamicMessage, which offers no field access by name — and unlike the rest of this, that one is entirely ours to fix. It is on the list.

Is there a better way to do this?

That is our workaround, and we are not claiming it is the right answer. Package-private is a deliberate choice — internals stay private precisely so they can change — and we went around it.

So the question is genuine. If there is a supported path to a SpEL token stream that we missed, we would much rather be told than go on carrying a copy of somebody else’s tokenizer. And if there is not, we are probably not the last people who will want one.

Either way the pieces are all there: the tokenizer works, the errors carry positions, the compiler doubles as a type checker. We would just like a supported way to hold one of them.

---

This came out of building Alginte, a visual Kafka Streams builder — self-hosted, free to run.