check-ride.ai Voice Agent Accuracy Technical Blog

* This article describes our work as of September 2026. We are continually evolving the architecture and setup, and expect both to change radically over the coming weeks, months and years.

A mock checkride examiner must do three things at once: apply the right aviation facts, assess what the candidate actually demonstrated, and maintain a consistent exam over a long conversation. A correct answer to an isolated question covers only the first part.

We built check-ride.ai around a language model with an explicit ACS scope, a persisted scenario plan, researched evidence, and constrained assessment tools. A flight director controls scenario progression; asynchronous workers prepare research and evaluate additional coverage while the examiner continues speaking. The premise is that supplying the right evidence and enforcing state outside the model should remove specific sources of error.

Our approach is to improve the evidence the examiner receives, constrain how it records decisions, and make those decisions reviewable. We test the evidence source with 300 fixed aviation questions, then audit the assembled examiner using a saved session. Critique returned the most correct answers at the lowest total cost among the tested configurations. In the later session audit, an automated judge supported 158 grading decisions, flagged three as unsupported, and left 13 ambiguous. These are measurements of two parts of the same workflow: acquiring aviation knowledge and applying it to a candidate’s performance.

Deriving the error calculation

Let E₀ mean the initial answer is wrong and E₁ mean the final answer is wrong. A final error comes from one of two mutually exclusive cases: an initial error remains, or an initially correct answer becomes wrong. Adding those cases gives:

P(E₁) = P(E₁ ∩ E₀) + P(E₁ ∩ ¬E₀)

Apply the probability chain rule to each term:

P(E₁) = P(E₀)P(E₁ | E₀) + P(¬E₀)P(E₁ | ¬E₀)

Write p = P(E₀), r = P(E₁ | E₀) for the residual-error rate, and d = P(E₁ | ¬E₀) for the damage rate:

p(final error) = pr + (1 − p)d.

No independence assumption is needed: r and d are conditional rates. An additional stage improves accuracy when it repairs more errors than it creates: p(1 − r) > (1 − p)d. Repeating this calculation across stages requires measuring each stage on the outputs it actually receives.

Search accuracy and cost

Each service received identical questions and choices, with gold labels withheld: Critique /v1/search; Tavily /search with advanced search and advanced answers; Exa /answer, model exa. We scored each service’s own answer deterministically against the pinned Pre-Flight dataset. No downstream answering model or provider fallback was used.

Figure 1. Bars: Wilson 95% accuracy intervals. Cost: recorded usage at public rates, September 8, 2026. Latency includes HTTP attempts, excluding queueing and retry backoff. Rates: Critique, Tavily; Exa reports dollar cost per response.

Critique’s lead over Tavily is six questions: +2.0 percentage points, paired bootstrap 95% interval −2.7 to +6.7, Holm-adjusted McNemar p = 0.497. Both outperform Exa under the same paired tests (adjusted p < 0.001). The cost advantage is observed; the accuracy advantage over Tavily remains unresolved.

Protocol notes. We saved all 900 responses and reused the initial 60 per service. One parser, including explicit option letters and intact choice fields in truncated JSON, was applied to every response; this recovered two correct Tavily answers. Three Exa rate-limit rejections were successfully retried and reported no usage. No final answer remained unparsed. Dataset revision: fdb1e431033db1b9fc7a2e87bdb62478eb0738e1.

Scope. This measures published-key answer-choice accuracy. Pre-Flight emphasizes ground operations and includes disputed keys and missing figures; we have not adjudicated its labels. Benchmark mirrors were excluded in prompts and by native domain filters where supported. Exa Answer has no documented equivalent filter, so retrieval leakage cannot be ruled out.

Critique’s observed error rate was 22%, compared with 24% for Tavily and 45.7% for Exa. That is 51.8% fewer errors than Exa and 8.3% fewer than Tavily; the Tavily difference remains statistically unresolved.

From a search answer to an examiner assessment

Critique researches session-specific ACS topics; the worker compresses each answer into a note, and the ACS tools attach ready notes to the item being examined. The examiner combines that context with the candidate’s actual answer and records an assessment. The sidecar can then credit additional items supported by the exchange.

The studies follow that path. First, we compare providers on a common answer key to select the research service. Then we inspect the examiner’s use of evidence through its recorded grades and justifications. This second test uses candidate exchanges and an assessment rubric, because the output we care about is now a judgment about demonstrated knowledge. The harness below connects the research input to that judgment.

How the synchronous and asynchronous paths connect

The map separates the flight director (scenario/director.py), the sidecar evaluator (voice/cross_credit.py), and the offline session judge. They perform different jobs: scenario progression, additional coverage assessment, and regression evaluation.

Figure 2. Synchronous means the turn depends on the result; blocking database/network work still runs off the shared asyncio loop. Dashed paths dispatch or complete asynchronous work. Research and cross-credit reach the examiner through a later state read. The offline judge does not approve live grades.

Scope and scenario compilation

FAA ACS text is split into tasks, extracted with an LLM, and checked against the exact source-code roster. Session selection applies applicability, task sampling, verbatim deduplication, and required coverage; a deterministic completion pass restores supported minima or rejects the scope. Selection sidecars encode mandatory tasks and item counts separately from the model prompt.

The scenario skeleton is generated alongside session preparation. The selected roster is then compiled into an ordered task-to-beat mapping, with validation, bounded repair, and fallback. This produces the plan the director executes; the live LLM does not regenerate it each turn.

Flight director and assessment contract

The director maps the current task to a beat, accumulates revealed fact IDs, records candidate decisions, and returns the transition instruction. State writes use an expected version; a conflict reloads state and suppresses a duplicate transition. The first bind includes the premise. Subsequent results include only the needed projection.

Assessment tools require a session-valid ACS code. Parent codes return child options; unknown codes return an error and, when unambiguous, a suggestion. They are not silently substituted. Every result carries the current open roster and focus item. Completion is computed from coverage and switches the examiner to debrief mode.

Research jobs and the evidence boundary

Research flags are reusable ACS metadata; note contents are session-specific. Jobs are unique per item, claimed eight at a time with FOR UPDATE SKIP LOCKED, and follow pending → in_progress → ready. Two failed attempts end the job; retries wait 20 seconds and stalled work can be reclaimed after three minutes.

Critique receives the aircraft/airport context; regulatory requests restrict sources to ecfr.gov, faa.gov, and govinfo.gov. Notes are normalized and truncated to 420 characters. Source URLs are stored separately. Only ready note text enters next_focus_item.research_note; a missing note does not block the turn. This compression and the examiner’s use of the note are additional stages that the direct-answer benchmark does not measure.

Sidecar evaluation and cross-credit

A clean satisfactory grade dispatches the sidecar with its justification and timestamp. Evidence is reconstructed from up to 14 preceding conversation rows, coalescing answer fragments; question and answer text are capped at 500 and 900 characters. If that read fails, classification can proceed from the accepted summary alone.

The classifier sees evidence and open target items, deliberately excluding the originating item’s code and description to avoid substituting topic similarity for demonstrated coverage. Default scope is all open items; an optional mode uses equivalence-map siblings. Returned codes and justifications are checked before assessment writes. _attach_scenario=False prevents a background credit from revealing a beat the examiner has not introduced. The next foreground read picks up the changed coverage. Usage and every attempted credit are recorded.

The sidecar evaluates additional coverage after the source grade; it is not a factual veto before that grade commits. Its relevant error is false-positive credit, alongside the repeated questions it prevents.

Streaming, visuals, and replay

Transcript/tool events go through an ordered per-run writer, with timestamps assigned at enqueue and a bounded teardown flush. Audio-frame processors never await database writes. The cascade filters protocol serialization before TTS; realtime paths also scrub persistence/replay, which prevents reuse but cannot retract audio already spoken.

Visual tools are an explicit synchronization point: register the capture request, send the browser command, wait for the persisted frame, inject its pixels, and remove the superseded visual context. Resume instead reconstructs conversation from stored rows, strips pause/stop ceremony, omits tool traffic by default, and refreshes authoritative ACS/scenario state. Provider assignment stays attached to the voice run across reconnects.

Inspectable grades, in the debrief and in our evaluations

The debrief shows each ACS item’s code, description and outcome, plus its recorded grading justification under Examiner Comments. The full conversation transcript is available alongside the assessment. A candidate or instructor can compare what was credited with what was actually said. If a justification credits an explanation the candidate never gave, that discrepancy is available for review at the specific item.

This is the practical meaning of making grading inspectable: the user gets the decision, its stated basis and the exchange used to assess it. The same saved records let our offline evaluator look for inconsistencies systematically. An audit label of unsupported means the evidence fails to justify the grade; the grade and its rationale remain visible for examination.

Study 2: audit the persisted grading decisions

We reused a saved synthetic oral session from the grade_verify_critique configuration: Gemini 3.8 Flash in text mode, a C172 scenario, coaching disabled, and Critique research notes. The archive includes the conversation, tool calls/results and persisted assessment records. Successful Critique lookups and research-note payloads are present in the saved traces. This developmental evaluation exercises the examiner’s text and tool workflow; real-candidate and speech-recognition performance require their own samples.

The offline audit proceeds as follows:

1. Identify the grading records. The audit tracks 174 persisted grade records, each tied to an ACS code, outcome, justification and conversation position. The denominator is these records, not search questions, examiner turns or all ACS coverage entries.
2. Review the candidate evidence. An LLM judge processes transcript windows with relevant tool context, carrying state between windows. For every supplied grade ID it returns supported, unsupported or ambiguous, with a candidate-turn reference, quoted evidence and rationale. A supported label means the automated judge found the recorded assessment justified by the exchange under the rubric.
3. Check allegations and coverage. Code checks that the expected IDs received review. First-pass unsupported allegations receive a separate model review against the local evidence; disagreements become ambiguous. The later audit submitted five allegations to this pass. The final rate is computed from the resulting labels, not from the judge’s overall session score.
4. Keep uncertainty visible. In the later audit, 158 grades were supported, three unsupported and 13 ambiguous. Thus 1.9% of the 161 decided grades were judged unsupported, with 92.5% decisive coverage. Ambiguous grades are excluded from that rate, not counted as correct.

An earlier audit of the same session classified ten of 143 decided grades as unsupported: 7.0%, with 31 ambiguous. We preserve both reports: the difference measures sensitivity to the audit procedure while the examiner’s decisions stay fixed. Our current estimate is therefore 1.9% unsupported among decided grades in the later audit, with judge reliability and broader session coverage still to establish. The next review targets changed and ambiguous findings plus a random sample of supported grades, then applies a consistent rubric across compatible saved runs.

What the two studies establish together

We have evidence for two engineering decisions. Use Critique for research: it achieved 78% answer-choice accuracy at the lowest benchmark cost, with a clear lead over Exa and a small, statistically unresolved lead over Tavily. Evaluate the examiner at the grade level: the saved Critique-enabled session lets us identify supported decisions, unsupported decisions and cases needing review. Its later audit found 98.1% support among decided grades, with 13 additional grades left ambiguous. The first study tests the knowledge service we selected; the second tests the assessments produced by the system using that service.

To measure how much Critique contributes to grading, our next comparison will replay the same candidate exchanges and starting state with research notes off and on, holding the examiner model and rubric fixed. Existing traces supply the test cases. The notes-off grading error rate supplies p; errors retained and introduced with notes on supply r and d. Then p(final grading error) = pr + (1 − p)d quantifies the change on the actual assessment task. Those paired outcomes are the measurements needed to turn the formula into a harness improvement estimate.

We built the harness so that research, exam state and grading can be tested separately and traced through a complete session. We improve it by measuring which changes repair more decisions than they damage, and retain the item-level grades, justifications and transcript so candidates and instructors can inspect the result. As models, prompts and control logic change, these records give us concrete cases to audit and replay.