Skip to content

Phase 2 — Read-only Verification

  • Date: 2026-08-02
  • Phase: Phase 2 — Architecture, Workstream 2 (read-only verification of open questions)
  • Scope: Latency, resolved-model stability, transcript reliability, freshness, and provider inference. All operations read-only (SQLite via docker run --rm -v omniroute-data:/data:ro; transcript JSONL metadata only). No live traffic, no configuration changes, no credential access.
  • Dataset: call_logs (4147 rows) and the current session transcript (8aa66a3e-5eb3-4536-ad68-3cd534eee262.jsonl, 396 KB). Sampled 2026-08-02 ~07:00 UTC.

Purpose

Answer the Phase 2 open questions that materially affect the architecture, using local evidence. This file records observations; the architecture conclusions drawn from them are in Architecture.md and Decisions.md.

Verified facts

1. The resolved route for auto/cheap is not stable across requests

The combo_name='auto/cheap' rows (482 total) resolve to 20+ distinct (provider, model) pairs, including big-pickle/opencode (192), gemini-3.1-pro-preview/gemini, llama-3.3-70b-versatile/groq, deepseek-ai/deepseek-v4-pro/nvidia, mimo-v2.5-free/opencode, gpt-5.5/blackbox, and others.

  • Status 200 vs 4xx/5xx is mixed: 400, 402, 413, 429, 499, and 502 all appear among auto/cheap resolutions. The auto engine actively probes providers and many candidate requests fail.
  • Implication: a statusline that displays "the resolved route" is displaying the most recent route, not a stable value. The display must tolerate change between invocations.

2. combo_name is the reliable filter for auto-routed traffic, not requested_model

  • requested_model='auto/cheap' appears on only 14 rows — all of them status=499, model=auto/cheap, provider=auto, combo_step_id empty (the blocked, un-resolved requests).
  • combo_name='auto/cheap' appears on 482 rows, including all concrete resolutions above.
  • For direct (non-auto) requests (e.g., requested_model='opencode/big-pickle', 886 rows), combo_name is empty.
  • Implication: any SQLite query that wants "the resolved route for auto traffic" must filter on combo_name, not requested_model.

3. When auto's classifier is unavailable, requests fail rather than resolve

The 14 requested_model='auto/cheap' rows are status=499, provider=auto. This matches the Phase 1B observation that the auto/cheap safety classifier became unavailable and gated read-only Bash operations. The auto routing engine's own safety classification is a dependency of every auto/* request; when it is down, the request does not fall back to a concrete provider.

  • This is a circular dependency: the system under investigation (auto/cheap routing) is required to be healthy to route its own traffic.

4. The transcript carries the resolved model on every assistant message, nested at message.model

  • 78/78 assistant messages carry message.model (100%). Observed values: big-pickle (74), mimo-v2.5-free (4). These match the resolved call_logs.model for the correlated rows.
  • Version drift: Phase 1B read the model field at the top level of the assistant message. In the current transcript the field is nested under message.model. This is the same data, relocated — evidence that the transcript structure can change between Claude Code versions.
  • The transcript does not carry a separate provider field.

5. Latency: transcript read is sub-millisecond; SQLite query is <10 ms; the cost is container startup

Measured on this machine (macOS, Docker Desktop):

Read path Latency
Transcript JSONL tail (tail -n 1) 0.00 s (sub-millisecond)
SQLite query alone (inside a container) 0.00 s (<10 ms)
docker run --rm -v omniroute-data:/data:ro alpine (startup only) 0.21–0.29 s
docker run + apk add sqlite + query (cold install) 0.73–1.13 s
docker run + pre-baked sqlite image (startup + query, inferred) ~0.25 s (borderline)
  • The docker exec omniroute sqlite3 ... path from Phase 1B is not available: the omniroute container does not contain sqlite3, and the plain alpine image does not either (sqlite3 must be installed).
  • Implication for the 300 ms statusline debounce (documented Claude Code constraint): the transcript path is safely within budget; the docker run path is marginal at best (pre-baked image) and over budget with an apk install. The SQLite path is only viable with a pre-baked sqlite3 image or a persistent reader.

6. call_logs.timestamp approximates response completion; the gateway is shared and noisy

  • The transcript's last assistant message (2026-08-02T07:00:42.624Z) correlates to a call_logs row at 2026-08-02T07:00:42.692Z (68 ms later) with combo_name=auto/cheap, model=big-pickle, provider=opencode, status=200, duration=23878 ms. call_logs.timestamp is therefore the response-completion time, not the start time.
  • The same 5-minute window contains connection-test rows (OmniRoute health checks to blackbox, sambanova, openai-*) and direct client requests (opencode/big-pickle with empty combo_name). A raw "last call_logs row" display would frequently show connection-test noise or another client's request.
  • session_tag remains NULL on all rows; session_model_history empty. No session-level filtering is possible from OmniRoute data.

7. Provider inference from the model string is unambiguous in observed data, but not guaranteed

  • Querying successful auto-routed rows (status=200, combo_name non-empty): no model appears under more than one provider. Top mappings: big-pickle→opencode (864), mimo-v2.5-free→opencode (145), auto→openrouter (43), deepseek-ai/deepseek-v4-pro→nvidia (18), deepseek-v4-flash-free→opencode (18), gemini-*→gemini.
  • This is an observation about current data, not a contract. A provider could add a model under another provider at any time. Provider inference from the model string alone is a heuristic, not verified behavior.

Observations

  • The docker run --rm -v omniroute-data:/data:ro read path executed without classifier gating in this session (small sample). Phase 1B saw repeated gating on read-only SQLite commands. This pattern is a good candidate for a statically-allowlisted statusline command, but the gating behavior is non-deterministic and must not be relied upon.
  • The statusline payload (model.display_name) still reflects the requested auto/cheap label (Phase 1B verified fact; configuration unchanged in this session).

Recommendations (feed Phase 3 design)

  1. Use the transcript path as the primary resolved-model source: sub-millisecond, session-accurate, 100% field presence, no Docker dependency. Display message.model from the last assistant message.
  2. Treat provider display as an optional enrichment, not a guaranteed value. Either (a) correlate the transcript timestamp to call_logs for provider (filter combo_name non-empty, requested_model != 'connection-test', status=200) with a fallback, or (b) display model-only, or (c) use an empirical model→provider mapping that is treated as a heuristic and allowed to be wrong.
  3. Define a staleness threshold for the resolved-model display (e.g., show the last known value with a "stale" marker when the transcript has no newer assistant message within N minutes).
  4. If the SQLite path is used at all, it must use a pre-baked sqlite3 image (or a persistent reader) to stay within the latency budget; it must tolerate being cancelled.

Open questions carried forward

  • Does the design display provider at all, given it is not in the transcript and SQLite enrichment is slow and shared-gateway ambiguous?
  • What staleness threshold value is acceptable for an interactive statusline?
  • Can the empirical model→provider mapping be refreshed out-of-band without exceeding the statusline latency budget?