Phase 5 — Validation Evidence and Staleness Contract Check¶
- Date: 2026-08-03
- Status: Evidence recorded for 6 of 7 roadmap scenarios; Phase 5 gate recommended as complete — the remaining scenario (OmniRoute unavailable) is recorded as an opportunistic observation, not a required induced failure (§4, §6)
- Phase: Phase 5 — Validation (Roadmap.md)
- Implementation under validation: commit
5c0973b(Phase 4), branchworktree-phase4-statusline
1. Staleness contract determination¶
Question¶
The [stale] indicator cleared as soon as Claude began processing a new turn
(token count rising), before the assistant finished generating. Does this match the
approved design and implementation contract, or was the marker intended to clear only
after a new assistant message becomes available (i.e., completes)?
Determination — matches the approved design and implementation contract¶
The design keys staleness strictly to the timestamp of the last assistant message in
the transcript, not to message completion. The implementation (isStale) is faithful to
that definition. Claude Code writes a fresh assistant entry to the transcript as soon as
generation begins, so the marker clears at processing start — before completion. This is
the correct, intended behavior. Confirmed by the user on 2026-08-03 after observing
the marker clear at processing start a second time.
Evidence¶
a) Design definition (verified from Statusline/Design/01-display-requirements.md:45-49):
Definition: the resolved route is stale when the wall-clock time minus the last assistant message's timestamp exceeds the threshold. … the marker is a soft signal, not a guarantee of what the next request will use.
The design never states that the marker must persist until an assistant message
completes. The only design text about the marker (01:42) concerns the route value
("the last known value is still shown (with the marker) rather than cleared"), not the
clearing condition.
b) Transcript write pattern (verified locally, 2026-08-03): assistant messages are
written to the transcript progressively during generation. The three content blocks of
one message (id 313c518e) are separate flushed lines with distinct write-time timestamps:
| line | content block | timestamp |
|---|---|---|
| 786 | thinking | 2026-08-03T07:24:04.911Z |
| 787 | text | 2026-08-03T07:24:05.368Z |
| 788 | tool_use | 2026-08-03T07:24:07.688Z |
The model field (message.model = big-pickle) is present on the first (thinking) entry.
A new turn therefore produces a fresh, parseable assistant entry with a fresh timestamp
within milliseconds of generation starting.
c) User observation (2026-08-02/03, reported and confirmed): fresh statusline while
typing; [stale] after overnight idle; [stale] cleared as soon as a new prompt began
processing (token count rising), before the response completed. Confirmed a second time on
2026-08-03.
The only way isStale (now − last-assistant-timestamp > 5 min) can flip from true to
false is a fresh assistant timestamp in the transcript; (b) confirms such an entry exists
at processing start. (a) confirms the design requires nothing more. (c) confirms the
observed behavior.
Corollary observation (not a defect)¶
During an in-flight turn the statusline displays the new in-flight model (known from the first streamed entry) rather than holding the previous route. This is consistent with "the last assistant message" contract and is arguably more correct — the model is determined as soon as the response starts. Noted for completeness; no change requested.
Edge case noted¶
If a turn is aborted after the first entry flushes, the transcript retains a partial
assistant entry with a model and fresh timestamp; the statusline would display that model
(unstale). Benign and consistent with "last parseable assistant message" (02:31); the
route is still a real resolved value.
2. Phase 5 scenario coverage¶
Roadmap.md Phase 5 scenarios vs. evidence gathered:
| Scenario | Status | Evidence |
|---|---|---|
| 1. Direct model selection | Not validated live (optional) | Route logic unit-tested (cases 2, 4); mechanism identical to auto-routing — the statusline does not distinguish how a model was selected |
| 2. Automatic routing | Validated | Live smoke: opencode/big-pickle from real transcript + mapping; matches latest call_logs row (big-pickle\|opencode\|200) |
| 3. Provider/model changes during a session | Validated | Live model-switch (§3b): light session → opencode/mimo-v2.5-free, heavy session → opencode/big-pickle; contract case 8 |
| 4. Missing, malformed, or delayed metadata | Validated | Contract cases 1, 3, 5, 6, 10–13; live failure paths verified (missing transcript → auto/cheap, unparseable stdin → "") |
| 5. OmniRoute unavailable | Opportunistic, not induced (§4) | Degradation is design-inherent (statusline never calls OmniRoute); classifier-outage incident showed statusline independence |
| 6. Slow or failing statusline execution | Validated | Latency test (sub-ms default path), hard timeouts (250 ms transcript / 50 ms mapping) with fallbacks, case 6 failing-read |
| 7. Different terminal widths and display environments | Validated | Manual (§3a): narrowed VS Code terminal — single bounded line, no wrap, no prompt overlap, readable |
3. Manual validation observations (user-reported, 2026-08-03)¶
a) Terminal-width validation¶
The user resized the VS Code terminal significantly narrower than the normal working
width. The statusline remained a single bounded line: it did not wrap, did not overlap
the prompt, and remained readable. Combined with the unit-tested 32-char route truncation
(case 9), this validates the Phase 5 terminal-width scenario (01:56).
b) Model-switch validation (also verified from call_logs)¶
A fresh, lightweight Claude session resolved to opencode/mimo-v2.5-free, while this
long-running heavy session remained opencode/big-pickle.
Verified in call_logs (2026-08-03):
| Time | Request weight (tokens_in) | Resolved |
|---|---|---|
| 07:45:53 | ~31K (fresh light session) | mimo-v2.5-free / opencode — status 200 |
| 07:46:25–46 | ~152K (heavy session) | big-pickle / opencode — status 200 |
This confirms:
- Auto-routing is request-weight dependent — a light request resolves to a lighter
model (mimo-v2.5-free), a heavy context stays on big-pickle.
- The statusline correctly reflects each session's own resolved route — the light
session showed opencode/mimo-v2.5-free while the heavy session kept
opencode/big-pickle. Scenario 3 validated live.
- Both routes remain on the opencode provider; the providers added/enabled by the
user's OmniRoute config change (bazaarlink, cerebras, sambanova, etc.) are passing
connection tests but have not yet won a real auto-route.
- Side note: the 499 rows are OmniRoute's own safety-classifier probes (one probed
deepseek-v4-flash-free and was blocked) — the classifier-gating phenomenon observed
in Phase 1B is also present inside OmniRoute's routing engine.
c) Staleness confirmation¶
The user re-confirmed the [stale] marker clearing as soon as Claude began processing
the next request (before the response completed) — matching the approved design per §1.
4. OmniRoute-unavailable (scenario 5): opportunistic, not a required induced failure¶
Recommendation: record as an opportunistic observation; do not induce the failure.
Rationale:
1. No runtime coupling. The statusline never calls OmniRoute — it reads only the local
transcript (written by Claude Code) and the local mapping file (04-caching-timeout-policy.md).
"OmniRoute unavailable" exercises no statusline code path that is not already
exercised by "transcript stops updating."
2. Identical statusline-visible state. The statusline-visible effect of OmniRoute being
down is that no new assistant messages arrive → the route ages past 5 minutes → [stale]
appears while the last-known route stays visible. That is exactly the behavior already
validated live by the overnight-idle observation (§1c). The statusline cannot — and should
not — distinguish "OmniRoute down" from "user idle."
3. High disruption, marginal information. Inducing the failure requires stopping the
local OmniRoute Docker container, which takes down the user's entire Claude Code workflow
(including this session). The marginal information over the idle observation is minimal.
4. Natural capture is likely. The 2026-08-02 classifier outage already demonstrated the
statusline operating independently of external dependencies. A future natural OmniRoute
outage can be captured opportunistically.
Expected behavior if observed (design contract, not yet live-captured): last-known route
retained, [stale] appended after the threshold, no crash, empty output only on total
statusline failure (unrelated).
5. Known limitations (for the Phase 5 report)¶
- Provider display is best-effort from the empirical mapping (ADR-0006); a model routed through a non-mapped provider shows model-only.
- The mapping is a snapshot (generated 2026-08-02); new models are model-only until the
mapping is refreshed via
Scripts/update-model-provider-mapping.js. - The route is the most recent route (ADR-0005), re-read per event; it is not a per-session history.
- The 5-minute threshold and 32-char truncation are tunable constants (
lib/config.js).
6. Re-evaluation and recommendation¶
Phase 5 gate¶
With the additions in §3, 6 of 7 roadmap scenarios are validated (direct model selection is optional — same code path as auto-routing). The sole remaining scenario (OmniRoute unavailable) is recorded as opportunistic rather than induced (§4).
Recommendation: consider Phase 5 complete. The roadmap's Phase 5 deliverables are satisfied in substance: evidence captured for the important scenarios, known limitations recorded, and the validation report is the remaining formal deliverable.
Next roadmap step¶
Before starting Phase 6 (Documentation and maintenance), close Phase 5 formally:
- Write the Phase 5 validation report (roadmap deliverable) — the validation evidence, the staleness contract determination, and the known limitations; this Findings doc is the source material.
- Promote reviewed conclusions to
Knowledge-Base.md(per the promotion rules): the statusline is read-only w.r.t. OmniRoute; auto-routing is request-weight dependent; assistant entries stream progressively into the transcript;[stale]clearing at processing start is correct. - Update
Roadmap.md"Current state": Phase 4 → Complete (currently still reads "Ready to start"), Phase 5 → Complete, Phase 6 → Ready.
Then Phase 6 — Documentation and maintenance — is the next engineering task: maintainability docs, the mapping-refresh cadence, and operational notes.
Open item for the owner¶
Whether to gate Phase 5 closure on a future opportunistic OmniRoute-down capture (no action needed now) or accept the design-inherent reasoning above and close without it. This decision is the owner's; the recommendation is to close.