Copy-on-write branching: why isolated attempts can stay cheap

IDEAS FOR THE FUTURE SIMMIS. These articles connect implemented research foundations to longer term product direction. They do not describe universal behaviour in the current Simmis product.

Suppose two agents test the same hiring decision from the same company record. If each attempt copies every unchanged record, isolation becomes expensive before either agent does useful work. This article explains the copy on write foundation behind cheap, isolated branches through one question. Should we hire three engineers in Q3 rather than Q4?

When an application forks state managed by Simmis, it need not copy the entire database. The branch shares unchanged structure with its basis, records what diverges, and can be discarded when the attempt ends.

The data structure underneath makes this possible and determines what a fork costs. Understanding that structure also clarifies which costs structural sharing does not remove.

Why full copies become expensive

Many databases optimize their ordinary interface around mutable current state. A naive branching design then creates a full copy for each isolated scenario, with work proportional to the dataset size. Branching a table with one million rows would copy one million rows under that design.

Production databases can also provide snapshots, multiversion concurrency control, or storage level copy on write. A full copy is therefore not the only alternative. The relevant comparison is between materializing every unchanged value and sharing unchanged structure deliberately.

Teams often approximate isolated scenarios by running them in sequence, restoring earlier state, or duplicating spreadsheets and workbooks. Those approaches may work for small analyses, but the manual reconciliation burden grows as scenarios touch more records and depend on one another.

Immutability changes what a fork stores

Persistent data structures create new values while retaining the old ones. Multiple versions can therefore refer to the same unchanged structure.

The hash array mapped trie (HAMT) is a common example. Phil Bagwell introduced the structure in Ideal Hash Trees (2001), and Clojure uses HAMTs for its persistent collections. Leaves hold data, while internal nodes route lookups. The diagram below simplifies that structure.

To update one leaf, the persistent structure creates a new path from the root to that leaf. It copies only the nodes along the path, while the new and old roots both refer to unchanged nodes.

Creating another branch reference can be constant time. The first update on that branch allocates nodes along a path of depth d rather than copying every row. The exact depth and cost depend on the implementation, hash width, branching factor, and changed values.

What this means for parallel controlled work

In a mutable database, isolated scenarios require copies or another isolation mechanism. In a persistent database, separate forks can share most of their unchanged structure. Incremental storage grows with divergent changes and retained paths rather than a full copy of every unchanged value.

For Simmis, this property supports multiple human and agent attempts while accepted managed state remains unchanged. Each branch can be queried independently and shares the structure it has not changed. The system can therefore support more attempts per decision without duplicating the full managed dataset or allowing each attempt to collide with the accepted record.

Forking computation as well as data

Data is only one part of a running attempt. Spindel, the reactive runtime in the stack, applies a related copy on write idea to running computation. An execution context containing signals, cached results, and continuations is a value that the runtime can fork in constant time.

require('[org.replikativ.spindel.core :as s :refer [signal spin track]]
  '[org.replikativ.spindel.incremental.interval :as iv])

def root: s/create-execution-context()
s/with-context(root
  def hires: signal(2)
  def velocity: spin(13 * iv/get-new(track(hires))))

def q3: s/fork-context(root)
s/with-context(q3 swap!(hires clojure.core/+ 3))
;; root still sees 2 hires; q3 sees 5, and both remain live

(require '[org.replikativ.spindel.core :as s :refer [signal spin track]]
         '[org.replikativ.spindel.incremental.interval :as iv])

(def root (s/create-execution-context))
(s/with-context root
  (def hires    (signal 2))
  (def velocity (spin (* 13 (iv/get-new (track hires))))))  ; 13 pts/engineer

(def q3 (s/fork-context root))          ; fork the live model in O(1)
(s/with-context q3 (swap! hires + 3))   ; only this branch hires in Q3
;; root still sees 2 hires; q3 sees 5, and both remain live

A scenario can pair a branched dataset with a branched execution context whose reactive computations remain connected and run again only where a dependency changed. The scope remains limited to registered state managed by Simmis. It does not include every external system or live process. Datahike documents database branching in depth, while Spindel applies the idea to the execution layer.

The git analogy holds

Git provides a familiar analogy. A commit refers to a tree of content addressed objects rather than storing a new full copy of the codebase. A branch is a named reference to a commit, so creating or moving that reference does not copy every file.

Datahike, the database at the bottom of the Simmis stack, applies this idea to data. Every transaction creates a new root pointer into an immutable tree. Named heads can refer to different roots, and a historical query reads the tree rooted at the selected snapshot.

Why Clojure

The Simmis stack is written in Clojure, where standard vectors, maps, and sets are persistent collections by default. Updating one of those values creates a new value that shares unchanged structure with the old value. This default makes the model described above part of ordinary application code.

Other languages also provide immutable or persistent collections. Haskell makes immutable data fundamental, and Scala includes persistent collections in its standard library. Clojure is useful here because persistent collections are its ordinary default and its concurrency primitives build on immutable values. Rich Hickey designed the language around separating identity from value and time from state. Datahike, Stratum, and Spindel apply that distinction to the knowledge graph, columnar analytics, and reactive execution layers.

The distributed boundary introduces another constraint. When state spans multiple machines, network partitions force tradeoffs among consistency and availability. Yggdrasil, our coordination layer, targets causal consistency so that effects follow their recorded causes across nodes. Within one node, immutable snapshots support reproducible reads. Across nodes, results also depend on causal histories and convergence. We treat this as an explicit architectural seam.

The deeper organizational implication

When forking stays cheap, teams do not need to serialize every attempt through one shared workspace. People and agents can branch when a new approach is worth testing, including from another branch.

This changes the cost of controlled exploration. More alternatives can run in parallel, while only an accepted proposal changes the accepted record. Unwanted attempts can end without first undoing changes to shared state.

Persistent data structures can therefore support more attempts per decision while preserving one accepted record. Their practical value still depends on how the surrounding system handles review, merge, retention, and external state.