A KV cache you can fork

Suppose five agent attempts branch from the same twenty thousand token context. Each attempt then sends the shared context again, and a conventional inference server has to process it again. The work is identical every time. This article describes how pretrained-rstr, an MIT licensed inference component from replikativ, turns that state into a value with an identity, so it can be stored, found, restored, and forked instead of recomputed.

In a typical decoder language model, processing the prompt writes one key and one value vector per layer per token. That memory is the KV cache, and every later token is produced by attending over it. Filling it, called prefill, pushes every prompt token through every layer in parallel. Producing each answer token afterwards pushes one token through those layers, conditioned on the prefilled cache. A long prompt therefore dominates the cost of a short answer, and this cache is the most expensive memory in the system.

Existing serving engines have treated that memory as an internal optimization. vLLM and SGLang keep a prefix cache so a repeated prompt within one process is cheaper. A newer layer of systems, led by LMCache, moves KV chunks across GPU, CPU, disk, and remote tiers so they outlive a process and can be pulled by another machine. What none of them hand the caller is the cache as a value: something with a name that can be looked up, restored on a different worker, forked, and reasoned about alongside the rest of an organization’s state. LMCache was the starting point for this work.

The design in one picture

pretrained-rstr separates three questions that a process-local cache never has to answer, and gives each one to the layer that can answer it durably.

What is this state? Datahike holds the answer: each chunk’s content hash, its parent in the token prefix, the model fingerprint it is compatible with, and where a ready replica lives. These are facts, queryable with Datalog, and they never include tensor bytes.

Where are the bytes? Konserve holds the immutable, content addressed chunks, over a local memory-mapped store or an S3-compatible backend. A chunk is written once and never modified.

Who is computing on it right now? Raster, the typed tensor compiler for Clojure underneath, owns the resident page pool on each worker: allocation, sharing between continuations, copy on write, eviction, and the compiled attention graphs that read the pages.

Two movements connect the layers. Publish flows down, from pages to chunks to catalog. Restore flows up, from a catalog lookup to verified chunks to resident pages. The rest of this article follows those movements.

A continuation is a value

The unit being published and restored is a continuation, and it has one exact definition. For a continuation with processed-count = n, the attention state covers positions [0, n), the pending token is evaluated at position n, and the token history includes that pending token. No logits and no transient activations are part of it.

That definition is deliberately small, and smallness is what makes it portable. It holds for CPU execution, contiguous GPU execution, and paged GPU execution alike, so a continuation captured on one path resumes the same causal computation on another. Replaying a transcript instead would rebuild an approximation of that computation from text. The continuation model states the invariants in full.

Identity is causal, not textual

Reuse is only safe if two caches that claim to be the same prefix really are. A chunk’s identity is a Hasch content hash that commits to both its token ids and its parent chunk’s hash. Identical suffix text under two different prefixes therefore produces two different identities. A lookup walks the chain from the root and stops at the first missing or incompatible node, which is exactly the longest reusable prefix.

A second identity covers the executor rather than the tokens. A compatibility fingerprint hashes the weights, the architecture descriptor, the model configuration, the attention-state layout, and a named execution variant into one value. A Q4 packed run and a Q8 packed run of the same checkpoint produce different numbers, so they carry different fingerprints, and state from one cannot be restored into the other.

Neither identity is needed inside a single process, because nothing outside the process ever sees the cache. Both are needed the moment the cache is meant to outlive the process or move between machines.

Chunks for storage, pages for execution

Storage and execution want different granularities.

A 256 token chunk scatters into sixteen 16 token pages on restore. Changing the page size does not change a chunk’s identity, and changing the chunk size does not change the attention result. The store can therefore optimize for object count and sequential transfer while the executor optimizes for allocation and prefix sharing.

Why owning the runtime matters

The operations above are cheap because pretrained-rstr owns the memory they act on. Raster allocates the page pool, and the attention kernels read pages through resident buffer views bound straight into the compiled graph. A lease pins pages while a graph holds a view. A generation counter stops a late transfer from completing into a page that has since been reused. A checkpoint is a retained device-to-host event on pages the pool already governs, and a restore retains the memory-mapped chunk inside the upload event until the device has consumed it. Forking is a page-table edit under the same refcounts.

A cache layer that sits beside an engine cannot do this. LMCache must first detect which of several physical KV layouts vLLM, FlashInfer, or an MLA model handed it, then copy through per-vendor kernels into its own buffers at the engine’s connector points, because the engine’s block manager owns the pages and does not share that authority. That is the right design for a layer that serves vLLM, SGLang, and Dynamo alike, and it is why it cannot offer a fork.

This trade is not free. Owning the runtime means writing the kernels and losing the PyTorch ecosystem and its attention libraries. Raster currently reaches Intel GPUs through Level Zero and Intel, NVIDIA, and AMD GPUs through compatible OpenCL drivers; native CUDA copy streams remain engineering work. pretrained-rstr accepts that in exchange for one memory model that covers kernels, transfers, durability, and branching, on the models it targets.

Forking is copy on write

Pages are immutable until written, so two continuations can share every page of a common prefix. A fork allocates nothing. When one branch appends a token into a partially filled page, that page alone is copied, and from then on the two page lists differ in one entry. The diagram above shows that state: the fork references p0 through p2 and owns its own p3′.

This is the same copy on write argument that cheap isolated branches make for structured data and that a forkable REPL makes for interpreter state, applied to GPU memory. It also has the same shape of limit. Two branches can advance independently from a shared prefix, but there is no semantic merge for divergent attention state. Choosing which branch matters is a decision the application makes, not one the cache prescribes.

The catalog never runs ahead of the bytes

Publish is ordered so that the catalog cannot advertise state that does not exist. A checkpoint captures immutable tensor ranges from the page pool, makes the local chunk durable, waits for the authoritative backend’s receipt when write-behind is configured, and only then transacts the chunk identity into Datahike. A chunk that exists solely in a failed worker-local write is never visible to a lookup.

Restore applies the same discipline in reverse. It verifies each chunk’s content identity and fingerprint before marking the replica ready, and a continuation becomes runnable only after every required page has landed. Datahike is consulted for the lookup and for placement, not per token. Workers keep their hot scheduling state locally, so Datalog queries and network round trips stay off the decode loop.

Route the request to the state

Once state has an identity, a request can be sent to the worker that already holds its prefix rather than moving tensors to the request. The cluster router ranks candidates by exact prefix and predicted time to first token, but its decision is advisory. The selected worker remains the authority for its own device memory: it revalidates the offer against current page state and reserves the projected prompt-plus-generation capacity atomically before accepting. Kabel carries only control messages and token results between them. Tensor bytes stay on the Konserve path.

Clients can access this through an OpenAI-compatible ingress. It accepts a small tested subset of POST /v1/chat/completions, translates HTTP, JSON, chat templates, and tokenization into an ordinary continuation request at the edge, and streams tokens back over server-sent events. From a client, only the base URL changes:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
result = client.chat.completions.create(
    model="gemma-3-270m-it",
    messages=[{"role": "user", "content": "Explain KV caching briefly."}],
    stream=True,
    stream_options={"include_usage": True},
)

The response reports what the routing achieved. usage.prompt_tokens_details.cached_tokens carries the count the worker’s restore returned, not the router’s estimate of what should have been resident. Tool calls, structured outputs, and multimodal content are later slices; the serving boundary lists exactly which fields are accepted today.

How this compares

Two ideas here are shared with the field. Chaining each chunk’s hash through its parent prefix is now standard: vLLM’s automatic prefix caching, SGLang’s RadixAttention, LMCache’s chunked token database, NVIDIA Dynamo’s KV indexer, TensorRT-LLM, and llm-d all do it. Persistence beyond a process is also no longer rare: LMCache’s remote backends, vLLM’s tiered offloading to a filesystem or S3, and llm-d’s use of that tier all survive a restart, while Dynamo deliberately treats KV as transient and recomputable.

The differences are in what is treated as identity and what is handed to the caller.

Keying on a model name is enough inside one deployment where the operator controls what that name means. It is not enough once state is shared across deployments or kept for months, and the vLLM project has an open proposal to add a quantization and dtype digest to shared-store keys for exactly that reason. Hashing the weights themselves, and naming the execution variant, closes that gap at the cost of one pass over the checkpoint at load time.

Recording lineage and placement in a general database rather than a purpose-built index is the other choice. LMCache’s controller offers lookup, pin, move, and compress, and its coordinator tracks fleet-wide placement, which is more operational machinery than pretrained-rstr has. What a Datahike catalog offers instead is that a continuation’s parent chain, fingerprint, and replica placement are ordinary facts that can be joined with whatever else the organization records about the attempt that produced them.

No system in the comparison exposes copy-on-write forking of a resident cache as an operation the caller performs. Engines share prefix blocks internally, and SGLang’s frontend can branch a program over shared prefixes, but the branch is not a durable, named continuation. That gap is what our approach fills. LMCache also does things pretrained-rstr does not attempt: non-prefix reuse through CacheBlend, compression through CacheGen, prefill and decode disaggregation over RDMA, and breadth across datacenter hardware. The two are complementary; the identity and catalog layer here does not assume a particular storage tier underneath it.

What is tested

Current model anchors cover token-exact paged decode, copy-on-write forks, and continuation resume on the supported development hardware. Model-free tests cover content-addressed prefix lookup, catalog and replica state, durable publication, page sharing, admission, eviction, and routed scheduling. Mixed prefill and decode packing, native CUDA copy streams, and production cluster policy remain engineering work.

What it is for

pretrained-rstr supports a growing set of instruction, embedding, and speech models that an organization can run on its own hardware. When inference and storage are configured locally, prompts, continuations, and provenance do not need to leave the operator’s environment. The state behind an answer can be named, stored, and reused with the same discipline as any other durable value in Datahike. The library is experimental and pre-1.0, and feedback is welcome. It is a replikativ component, not yet a shipped Simmis integration.

The branching property is what makes it more than a cache. When an attempt can begin from an exact prefix another attempt already paid for, the marginal cost of a second attempt is the tokens it adds instead of the context it inherits, and abandoning it costs only the pages it dirtied. The same reasoning already governs structured organizational state in this stack. Applied to attention state, it means the cost of exploring several options no longer scales with context size if they share most of their context window.

If you are interested in running pretrained-rstr, contact us and we will help you get started.