At Hot Chips 2026 on Tuesday, OpenAI presented Jalapeño, an inference ASIC built with Broadcom.1 One slide from the talk has been circulating since: a request no longer spans two phases but three —...

Prefix caching can cause the same prompt to produce different logits on a cache hit versus a cache miss. In many operational contexts, this is quickly categorized as a defect. However, a bug strict...

Each of the previous three posts assumed the next layer of the stack would save it. Part two showed that KV caches lack an interchange format, assuming the bytes would flow if two vendors merely ag...

Compilers are often described as embarrassingly parallel. They rarely are. A translation unit is independent. The machine has sixteen cores. Yet the frontend turning source into an IR is typically ...

The previous post ended on a pattern: a memory hierarchy works because one component can see the whole path, and disaggregation splits that visibility across two vendors. Scheduling has the same sh...

The last post argued that a KV cache has no interchange format. Grant one anyway. Suppose the two vendors agree on layout, dtype, block size, scale placement and every other axis in that list. The ...

The last post argued that prefill and decode want different computers, and that the industry has started buying them separately. This one is about the handoff, which sounds like the easy part but i...

Three announcements inside a year, all making the same architectural bet. AWS is pairing Trainium with Cerebras: Trainium runs prefill, a Cerebras CS-3 runs decode, and the result ships as a premi...

As of this month, Claude watermarks its text output. Anthropic’s documentation is direct about the mechanism: for text, “it weaves an imperceptible watermark directly into the text itself. You won’...

In September I am presenting a session at CppCon that challenges conventional frontend design: “Escaping the AST: A Data-Oriented, Lock-Free Parallel Compiler Architecture.”1 The core thesis requi...