Prefix caching makes the same prompt produce different logits on a cache hit versus a cache miss. Is that a bug? A bug is a violation of a specification. No specification exists that this behaviou...

Each of the previous three posts assumed the next layer of the stack would save it. Part two showed that KV caches lack an interchange format, assuming the bytes would flow if two vendors merely ag...

Compilers are often described as embarrassingly parallel. They rarely are. A translation unit is independent. The machine has sixteen cores. Yet the frontend turning source into an IR is typically ...

The previous post ended on a pattern: a memory hierarchy works because one component can see the whole path, and disaggregation splits that visibility across two vendors. Scheduling has the same sh...

The last post argued that a KV cache has no interchange format. Grant one anyway. Suppose the two vendors agree on layout, dtype, block size, scale placement and every other axis in that list. The ...

The last post argued that prefill and decode want different computers, and that the industry has started buying them separately. This one is about the handoff, which sounds like the easy part but i...

Three announcements inside a year, all making the same architectural bet. AWS is pairing Trainium with Cerebras: Trainium runs prefill, a Cerebras CS-3 runs decode, and the result ships as a premi...

As of this month, Claude watermarks its text output. Anthropic’s documentation is direct about the mechanism: for text, “it weaves an imperceptible watermark directly into the text itself. You won’...

In September I am presenting a session at CppCon that challenges conventional frontend design: “Escaping the AST: A Data-Oriented, Lock-Free Parallel Compiler Architecture.”1 The core thesis requi...

I have eight devices. I want a matmul whose contracting dimension is split across some of them. So I write the matmul normally, say where the two operands live, and compile: mesh = Mesh(np.array(j...