Draw figures from their cells, paint them with theme tokens, and check figure content

CHANGELOG.md

@@ -13,8 +13,15 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 - A Run whose Turnstile token the server rejects now ends with the server's message instead of re-challenging. The runner used to solve again every time the response carried the challenge marker, so a persistently failing Siteverify (wrong secret, hostname, or action) turned one click into an unbounded loop of solves, Worker POSTs, and Siteverify subrequests. Each Run now earns at most one challenge, and the failure message says to press Run again.
 - Siteverify calls time out after 10 seconds, tokens longer than Cloudflare's documented 2048 characters are rejected without a subrequest, and a `success: true` response with a missing or non-string `hostname` fails closed instead of raising a 500. The gaps surfaced by comparing `_verify_turnstile` with the canonical handler in Cloudflare's Turnstile Spin skill.
 - Production Python packages are locked. Pywrangler 1.17.4 vendors a committed, hash-pinned `pylock.toml` instead of resolving an unpinned `fastapi` at deploy time, and `uv.lock` pins the same versions (FastAPI 0.141.1, Starlette 1.7.0, Pydantic 2.10.6) so the test suite runs against exactly what ships. `tests/test_dependency_locks.py` fails on drift, and CI and `make deploy` fail if a sync would change `pylock.toml`. Starlette 1.7.0 also clears the five advisories against the previously tested 1.0.0.
+- The bytes-and-bytearray figure rendered doubled backslashes (`b'\\x63\\x61\\x66'`) and drew bytes the example never uses (#18). Redrawn from the example's real values, and three new figure contracts close the class of bug: no rendered text may carry a doubled backslash and escape sequences must appear in the example source (13), every arrow is at least 20 units (14), and `site.css` must define every figure token in both schemes (15).
 - Example-page runner wiring (Run interception, Reset, the share button, and keyboard navigation) no longer waits for the CDN-backed highlighter and editor modules: `runner.js` loads `async`, so a slow or unreachable esm.sh cannot stall it — ordered module scripts otherwise execute strictly after every preceding module settles, including their top-level awaits.
 
+### Changed
+
+- Figures now paint with `--fig-ink`, `--fig-ink-soft`, `--fig-accent` and `--fig-soft` tokens instead of literal colours, and `site.css` defines the tokens for both colour schemes. Dark mode recolours every figure with the page; the light "paper chip" behind figures is gone. Figure mono and sans faces follow the page's own font stacks, and mono runs are pinned to the grammar's character advance with `textLength` so divider positions hold on every platform. Paired figures in a banner share a baseline (a two-row subgrid). Five figures lost an accent that marked an arbitrary row (`naming-decisions`, `iteration-loop-selector`, `type-shape-catalog`, `reliability-signal-map`, `container-methods`); nine arrows that were mostly arrowhead were lengthened to the new 20-unit minimum.
+- Ten figures redrawn as mechanisms carrying the lesson's own values: closures is an environment diagram with the returned frame ghosted and one live `__closure__` reference; generators shows the cell's two `next()` calls on `countdown(3)` with the surviving local beside each; recursion shows the example's tree at the deepest frame of `total(tree)` with the values each frame returns; dicts shows the cell's keys hashed to slots with the insert of `"year"`; sentinel iteration shows `read_chunk` yielding `"py"` and `"thon"` and stopping on the empty string; string formatting's format spec is a railroad tracing the cell's `05.1f`, skipped stations ghosted under a bypass rail; bytes-and-bytearray draws the example's bytes as integer slots with the in-place write and moves to the cell that performs it; regular expressions shows the cell's two capture groups landing on `Ada: 10`; values shows `text` bound to a `str` object and the `type()` hop to where `upper()` lives; logging shows the cell's three records crossing the handler's INFO gate. The binding figure is a parametrised factory, so `variables` shows `message → "hi"` and `constants` shows `MAX_RETRIES → 3`.
+- The rubric states that table-shaped figures are redrawn rather than removed, that a figure whose caption names no single element carries no accent, and how cell-0 anchors are scored. The review, principles and queue are in `docs/diagram-upgrade-plan.md`.
+
 ### Added
 
 - `/about` page describing how the site is made — the verified-output pipeline, the runner sandbox, the figure grammar, and the quality gates — with a design-language section that renders the live CSS tokens (color swatches, spacing scale, type specimens, and a real walkthrough cell as a specimen). Linked from the nav on every page and listed in the sitemap.

docs/diagram-upgrade-plan.md

@@ -0,0 +1,179 @@
+# Diagram upgrade plan
+
+Written 2026-09-27 from a review of every attached figure on the live
+site, the gestalt review pages, the grammar, both rubrics and the
+registry. The rubrics remain the scoring instruments. This document
+records what the review found, the principles that follow from it,
+what the first pull request changed, and the queue after that.
+
+## What the review found
+
+The library's strengths are enforced rather than hoped for: the palette,
+stroke weights and typography are locked by contracts, captions assert
+what is true instead of narrating the picture, and the gestalt review
+page paints from the production paint functions. The figures that teach
+best show a mechanism changing state, such as aliasing before and after
+an append, the iterator caret advancing one row at a time, and the MRO
+diamond flattening into a chain.
+
+The weaknesses are measurable.
+
+| finding | measure |
+|---|---:|
+| paint functions using only cell, arrow, dashed line and text | 66 of 124 |
+| `cell` calls / `closed_arrow` calls / `lanes` calls | 236 / 129 / 1 |
+| attachments anchored after cell 0 | 106 of 115 |
+| example figures scoring exactly 9.0 (the rest 9.5, none lower) | 106 of 109 |
+| journey figures scoring exactly 9.0 | 21 of 21 |
+| arrows shorter than 20 units | 9 |
+
+Behind the numbers: figures hard-coded the light palette, so in dark
+mode each one sat on a cream chip; a paired banner had no shared
+baseline; the SVG asked for JetBrains Mono and Source Sans while the
+page loaded no web fonts, so the visitor's operating system chose the
+faces; several selector figures put the accent on whichever row the
+loop reached last; and because no contract read figure text, the
+bytes-and-bytearray figure rendered doubled backslashes for months and
+the regular-expressions figure drew a pattern the example never uses.
+
+## Principles
+
+1. **Show state over time.** One frame is a label. The same frame after
+   each step is a mechanism. Repeating one drawing is cheap in this
+   grammar; `iterator-unroll` and `generator-resume` do it.
+2. **Draw the heap.** Python's picture is frames on the left and objects
+   on the right with arrows crossing between them. The grammar now has
+   an `env` phrase for the frame, and closures uses it.
+3. **Redraw a weak figure; never replace it with a note.** A drawing
+   that would be no worse as a two-column table needs a mechanism
+   drawn in its place. Removing it and recording a rationale leaves the
+   page with less than it had, which the first draft of this plan got
+   wrong for four pages.
+4. **The figure carries the lesson's own names.** Reuse the mechanism
+   by parametrising the paint function, so `variables` shows `message`
+   and `constants` shows `MAX_RETRIES`.
+5. **Accent only what the caption names.** When the caption names no
+   single element, the figure has no accent.
+6. **Check figure content against the example.** Geometry contracts
+   cannot see a wrong label. A content contract can.
+7. **Paint with tokens.** The grammar emits `var(--fig-*)`; the page's
+   stylesheet decides the ink in each colour scheme.
+8. **Typography follows the page.** Mono and sans use the page's own
+   stacks, and mono runs carry `textLength` so divider maths holds on
+   every platform. The italic serif stays for identifiers.
+9. **Rank before scoring.** With 106 of 109 figures at 9.0, the score
+   is a ship gate. Choosing what to redesign next needs a ranking.
+
+## What the first pull request changed
+
+Grammar (`src/marginalia_grammar.py`):
+
+- `INK`, `INK_SOFT`, `EMPHASIS` and `SOFT_FILL` are `var(--fig-*)`
+  references. `FIGURE_TOKENS_LIGHT` and `FIGURE_TOKENS_DARK` hold the
+  values; `figure_token_css()` emits them for the gestalt pages and the
+  social-card shell.
+- `FONT_MONO` and `FONT_SANS` match `public/site.css`; every mono run is
+  pinned to `MONO_ADVANCE` with `textLength`.
+- `ARROW_MIN = 20`; `Canvas` records every arrow's length.
+- `name_box` and `bind` take a width so long names fit.
+- The `env` phrase draws a frame with bindings, optionally ghosted, and
+  returns anchors for heap arrows.
+
+Figures (`src/marginalia.py`):
+
+- `closure-cell` is an environment diagram: the returned frame ghosted,
+  `double` naming a heap function object, one accented `__closure__`
+  reference back into the frame.
+- `generator-resume` replaces `generator-ribbon`: three ribbons for
+  three `next()` calls on the cell's own `countdown(3)`, the surviving
+  local beside each.
+- `format-spec` is a railroad tracing the cell's own `05.1f`: visited
+  stations shaded, skipped stations ghosted under a bypass rail.
+- `bytes-vs-bytearray` draws the example's bytes as integer slots and
+  the in-place write `packet[0] = ord("P")`, anchored to the cell that
+  performs it.
+- `regex-groups` replaces `regex-anchors`: the cell's two capture
+  groups dropped onto the record `Ada: 10`.
+- `value-type-lookup` replaces `value-types`: `text` bound to a `str`
+  object, then the `type()` hop to the class where `upper()` lives.
+- `logging-threshold` replaces `logging-levels`: the cell's three
+  records crossing the handler's INFO gate, `debug` dropped.
+- `variables-bind` and `constants-bind` come from one parametrised
+  factory.
+- Nine arrows lengthened to the minimum; five figures lost an accent
+  that marked an arbitrary row.
+- `literal-forms` and `collections-containers` stay as they were, scored
+  8.5, until a mechanism replaces them.
+
+Page (`public/site.css`, `src/templates/about.html`):
+
+- `--fig-ink`, `--fig-ink-soft`, `--fig-accent` and `--fig-soft` defined
+  for light and dark; the dark-mode paper chip removed.
+- `.cell-banner figure` is a two-row subgrid, so paired figures
+  bottom-align their drawings and start both captions on one line.
+
+Contracts (`tests/test_marginalia_geometry.py`):
+
+- 13, content: no doubled backslash anywhere; an escape sequence in an
+  attached figure must appear in that example's source.
+- 14, arrow length: every arrow at least `ARROW_MIN`.
+- 15, theme tokens: `site.css` defines all four tokens in both schemes
+  with the grammar's values, and every grammar colour is a token.
+
+Rubrics: the redraw rule, the zero-accent rule, the placement note and
+the three contracts are in `docs/example-figure-rubric.md` and
+`docs/journey-visualisation-rubric.md`; the token and typography
+invariants are in `docs/visual-explainer-spec.md`.
+
+## Queue
+
+Ordered by payoff. Each item is one pull request.
+
+1. **State over time** for `context-managers` (enter, body, exit, with
+   the exception path as a second strip), `decorators` (two calls, two
+   wrappers) and `object-lifecycle`.
+2. **Heap drawings** with `env` for `scope-global-nonlocal`, `recursion`
+   (frames with `n` bindings instead of labelled cells) and `unpacking`.
+3. **Mechanisms for the two table figures.** `literals`: the same value
+   spelled three ways converging on one `int` object. `collections-module`:
+   one container's operation, or split the page.
+4. **Parametrise the remaining reused figures**: `iterator-unroll`,
+   `iter-protocol`, `class-triangle`, `operator-dispatch`,
+   `comprehension-equivalence`.
+5. **Move anchors to the cell where the move happens.** For each cell-0
+   attachment, ask whether a later cell holds the mutation, dispatch or
+   resume the figure depicts. Rewrite criterion 10 once the distribution
+   is honest.
+6. **Rank the gestalt.** A pairwise ranking table in the registry, and
+   redesign from the bottom ten. The 8.5 score stays as the gate.
+7. **Widen the content contract** from escape sequences to every mono
+   literal that looks like Python source: it must appear in the
+   example's code or output.
+8. **Buffers and hashes.** `dict-buckets` and `set-buckets` with hash
+   values from the example; `list-append` and `slice-ruler` as header
+   plus buffer.
+
+## What to study
+
+- **Composing Programs** (composingprograms.com) and **Python Tutor**
+  (pythontutor.com) for the environment diagram: frames left, objects
+  right, and how a closure and a recursive call are drawn.
+- **Ned Batchelder, "Facts and myths about Python names and values"**
+  for name-to-value drawings done the same way every time.
+- **Crafting Interpreters** (craftinginterpreters.com) for warm ink on
+  cream, one idea per figure, held consistent across a whole book.
+- **Sam Rose, samwho.dev** ("Hashing", "Memory Allocation") for
+  programmer-facing figures with a locked grammar; the model for the
+  bucket figures.
+- **Lydia Hallie, "JavaScript Visualized"** for the frame designs of
+  scope chain and closures.
+- **Bartosz Ciechanowski** (ciechanow.ski) for figures where exactly
+  one variable changes between frames.
+- **SQLite syntax diagrams** for what a railroad is; `format-spec`
+  follows them.
+- **The Rust Book, chapter 4** and **"Go Slices: usage and internals"**
+  for header-plus-buffer drawings of list, bytearray and slice.
+- **Tufte, Visual Explanations, "the smallest effective difference"**
+  for how little contrast emphasis needs.
+- **Maggie Appleton** and **Julia Evans** for mechanism drawings that stay playful without adding marks.
+- **Distill.pub** for captions that carry the claim.

docs/example-figure-rubric.md

@@ -60,6 +60,9 @@ per-figure scoring.
    figure. The accent goes on the single element the cell prose
    names (the live mutation, the captured cell, the dispatch arrow).
    Three accent marks competing for attention is no emphasis at all.
+   When the caption names no single element, as in a selector where
+   every branch is equal, the figure has zero accents, because an
+   accent on the row a loop happened to reach last marks nothing.
 8. **Restraint (0-1.0)** — no decoration that does not carry
    information. No drop shadows, gradients, ornamental rules,
    non-orthogonal tilts, or marks placed for "balance".
@@ -77,6 +80,14 @@ per-figure scoring.
     cell just made, not stand alone as a generic illustration of the
     example title.
 
+    As of September 2026, 106 of 115 attachments anchor after cell 0,
+    so this criterion is usually scored against the first cell. An
+    opening figure is a legitimate design, and it earns the 0.5 by
+    summarising the first cell's move rather than the title. When a
+    later cell holds the mutation, dispatch or resume the figure
+    depicts, move the anchor there; `bytes-and-bytearray` moved to
+    cell 3 for that reason.
+
 ## Topic gates (cell-shape specific)
 
 - **Binding cells** (assignments, `=`) — show the name-arrow with the
@@ -107,6 +118,34 @@ the figure can merge.
 
 - **One figure per cell, at most.** Two figures on one cell signal
   the cell is doing two things; split the cell instead.
+- **Table figures are redrawn, never removed.** Rows of `name | role`,
+  `type | spellings` or `level | number` are reference material; they
+  score 8.5 at most under criteria 2 and 4 because a table in the prose
+  would do the same work. The remedy is a mechanism in the same slot:
+  `values` became the `text → str` binding with the `type()` hop to
+  where `upper()` lives, and `logging` became three records crossing the
+  handler's level gate. Removing a shipped figure and recording a
+  rationale leaves the page with less than it had; the
+  `no_figure_rationales` registry is for pages whose lesson has no
+  drawable mechanism, and a page that once carried a figure does not
+  qualify.
+- **The figure carries the lesson's own names.** Where one mechanism
+  serves several lessons, parametrise the paint function
+  (`_bind_figure("MAX_RETRIES", "int", "3")`) instead of choosing
+  between placeholder names and a bespoke copy.
+- **Content fidelity, enforced.** No rendered text may contain a
+  doubled backslash, and a mono label containing an escape sequence
+  must appear verbatim in the example's Markdown source. The
+  `bytes-vs-bytearray` figure shipped `b'\\x63\\x61\\x66'` for
+  months because every contract measured geometry and none read the
+  label. *Contract 13 — FigureContentContract.*
+- **Minimum arrow length, enforced.** Every closed arrow is at least
+  `ARROW_MIN` (20) units long; below that it is mostly head. Widen
+  the gap between the boxes, never shrink the head. *Contract 14.*
+- **Both colour schemes, enforced.** Figures paint with the
+  `var(--fig-*)` tokens; `public/site.css` defines all four in the
+  light root and the dark media block with the grammar's values, so
+  no figure needs a light chip in dark mode. *Contract 15.*
 - **figcaption present and declarative.** Captions in the form
   "Two names share one mutable list — appending through one name
   changes the object visible through both." Not "this shows X" or
@@ -231,4 +270,8 @@ example reads as a coherent set rather than independently authored
 diagrams.
 
 The score is a guide, not a substitute for reading the cell beside
-its surrounding prose.
+its surrounding prose. Because 106 of 109 attached figures score 9.0
+and the rest 9.5, the score no longer says which figure to redesign
+next. Use the gestalt page to rank the library pairwise and redesign
+from the bottom, and keep the score as the ship gate
+(`docs/diagram-upgrade-plan.md`).

docs/journey-visualisation-rubric.md

@@ -43,7 +43,10 @@ Score each section figure on a 10-point scale.
 7. **Emphasis scarcity (0-1.0)** — at most one accent mark per
    figure. The accent goes on the single element the section names
    (the live yield, the dispatch arrow, the captured cell). If three
-   things are orange the figure has no emphasis at all.
+   things are orange the figure has no emphasis at all. When the
+   caption names no single element, as in a selector where every branch
+   is equal, the figure has zero accents, because an accent on the row a
+   loop happened to reach last marks nothing.
 8. **Restraint (0-1.0)** — no decoration that does not carry
    information. No drop shadows, gradients, ornamental rules,
    non-orthogonal tilts, or marks placed for "balance".
@@ -95,7 +98,9 @@ Score each section figure on a 10-point scale.
   width the layout gives it (`clamp(280px, 70vw, 640px)`); design at a
   viewBox whose rendered width fits the shared 640px ceiling.
 - **Uses only the four palette constants.** `INK`, `INK_SOFT`,
-  `EMPHASIS`, `SOFT_FILL`. Anything else is grounds for redesign.
+  `EMPHASIS`, `SOFT_FILL`, which resolve to the page's `--fig-*`
+  tokens so a section figure renders in both colour schemes without a
+  chip. Anything else is grounds for redesign.
 
 ## Quality bands