The Legality Gap: What 1,000+ Graded Orals Reveal About Checkride Readiness

Every session on check-ride.ai is a full mock checkride oral, graded element-by-element against the FAA Airman Certification Standards. A side effect of building it that way is a dataset that, as far as I know, doesn't exist anywhere else: thousands of individual ACS elements, each graded satisfactory / marginal / unsatisfactory with a written justification, across hundreds of complete oral exams.

I spent a couple of days doing a proper exploratory analysis — distributions first, then correlations, then within-pilot designs. This post is what fell out. Some of it flattered my assumptions. A fair amount didn't, and I've kept those parts in.

The one-sentence version: today's applicants are strong exactly where training culture points — judgment, weather, decision-making — and the oral has one consistent blind spot left: legality. Airworthiness, equipment requirements, currency, airspace. It hides because it doesn't surface as "I don't know"; it surfaces as a confident answer that's quietly out of date. The good news, further down: once a mock oral drags one of those into the open, it usually stays fixed.

The dataset (and what I'm not going to tell you)

For this analysis we sampled 1,000+ mock oral sessions from our production database, spanning the last 12 months and pilots in 30+ US states. After excluding sessions that never got past configuration, plus internal accounts, synthetic evaluation runs (we drive automated full-length sessions against production to test examiner quality — they look exactly like real sessions and must be filtered), and test configurations, the working set is several hundred complete orals comprising 10,000+ individually graded ACS elements.

What I'm deliberately not publishing: exact user and session totals, and anything traceable to an individual. Everything below is a percentage or a distribution — the full-resolution breakdown is something we share with partner flight schools, not the open internet. We're small; you understand.

Who's in the data:

Almost all airplane single-engine land, which tracks the training fleet. Geographically it's the sunbelt training corridor you'd expect — Florida, Texas, Georgia, Arizona, and California together account for roughly 44% of pilots who shared a home airport.

Grading, precisely defined

Numbers are only as good as their definitions. For every ACS element the examiner covers, the session records exactly one of:

• Satisfactory — met the ACS standard unprompted: correct, reasonably complete, no misconception the examiner had to fix
• Marginal — partial: a misconception the examiner corrected, an important omission, or substantial prompting to get there
• Unsatisfactory — could not demonstrate the element, even after prompting

Across everything we've graded: 78% satisfactory, 11% marginal, 11% unsatisfactory. For candidates still mid-training, four out of five answers meeting the ACS bar unprompted is genuinely solid — and it sits interestingly close to the ~78–80% national private pilot practical pass rate in AOPA's pass-rate tracking (a different statistic measuring a different thing, but useful context). The interesting part isn't the level. It's the shape — where the remaining fifth concentrates.

Finding 1: the oral is mostly Area I — plan your prep like it

About 63% of every element graded on the platform falls in Area of Operation I — Preflight Preparation. Pilot qualifications, airworthiness, weather, cross-country planning, performance, systems. Not maneuvers. Paperwork, planning, and legality. This matches what DPEs describe publicly (California Aeronautical University's roundup of common checkride themes is representative), but the concentration still surprised me. If your oral prep isn't at least half Area I, you're studying for a different exam than the one that gets given.

One methodological note before going further, because it matters for everything below: exposure and difficulty are different measurements, and this finding is only about exposure. Because Area I is graded ~5× more than everything else, raw miss counts would be dominated by Area I no matter how well pilots knew it. So every difficulty figure in this post is a conditional rate — P(satisfactory | the element was actually assessed) — never a share of total misses.

Finding 2: the curriculum is working — judgment and weather are strengths

Worth saying loudly, because it cuts against a generation of hangar grumbling: the aeronautical-decision-making curriculum has landed. The strongest elements on the platform are the judgment ones — external pressures, passenger distractions, pilot self-assessment, fuel reserves — most sitting at 96–100% satisfactory. And weather, the subject every ground school drills hardest, is one of the strongest knowledge tasks (83.4% in Area I): METARs, TAFs, and winds-aloft answers are overwhelmingly fine. Where training culture concentrates effort, the results show. Which raises the obvious question: what's left?

Finding 3: the one blind spot — legality

Satisfactory rate by task within Area I, private pilot sessions only:

The lower end of that chart is consistent: the National Airspace System, Airworthiness Requirements, and Pilot Qualifications — the dry regulatory material that rarely gets drill time. The specific elements, with their ACS codes (nothing below a minimum sample is reported):

Is this just Area I being tested the most? No — these are conditional rates, and it's worth being precise about what does and doesn't earn "blind spot" status. On pure area-level conditional rates, Navigation actually grades slightly lower than Preflight Preparation overall (72% vs 77% satisfactory) — on roughly 1/25th the exposure, and, as the next section shows, failing a fundamentally different way (blanks, which pilots already know to study). The legality cluster is the blind spot on the conjunction of three things: the lowest element-level conditional rates we measure anywhere (44–66%, against a platform-wide 78%), the highest exposure on the exam (so an elevated miss rate there translates into more real misses than anywhere else), and a miss type that self-study can't catch.

These are all one question, really: "can you, and this airplane, legally make this flight today?" None of this is unique to our examiner — 91.205/91.213 inoperative-equipment reasoning is a well-documented head-scratcher on real private checkrides, and BasicMed is confusing enough that AOPA maintains a dedicated checkride explainer. The same cluster shows up for private, instrument, and commercial candidates alike. It isn't a talent problem; it's an attention problem — this material simply isn't where prep hours go. Which, honestly, is encouraging: of all the things that could be the oral's weak spot, "a well-defined, finite list of regulations" is the most fixable one imaginable.

Finding 4: why it stays hidden — these misses look like answers

Marginal, in our grading, means the pilot had an answer — it was just wrong enough that the examiner had to correct it. That lets us split every miss into "confident-but-off" (marginal) versus "blank" (unsatisfactory):

In Preflight Preparation — the regulatory heart of the oral — more than half of all misses are misconceptions: a currency rule that changed, a BasicMed limit half-remembered, an equipment list from a forum post. In Navigation, by contrast, misses are mostly honest blanks.

This explains the blind spot better than any theory about lazy students. Blanks are self-diagnosing — a pilot who knows they don't know will study. A misconception feels exactly like knowledge from the inside, so no amount of conscientious self-study finds it. Someone — a CFI, a study partner with the ACS open, or an examiner-shaped piece of software — has to probe the answer and push back. That's not a character flaw in pilots; it's just how memory works.

Finding 5: fresh beats experienced — recency is the variable

Commercial applicants — pilots with hundreds of hours — grade below private applicants on the same material:

Not because experienced pilots are worse pilots — because the regulatory knowledge was learned once, years and hundreds of hours ago, and commercial prep culture is maneuver-heavy. Everyone polishes the chandelle; almost nobody re-reads 91.205. The pattern is really the same finding as above wearing a different hat: this material stays sharp only as long as something keeps testing it. (CFI candidates, who have to re-teach the basics, hold the line — 81.3%.)

Finding 6: pilots who think out loud grade better — a lot better

We joined session transcripts to grades and computed each pilot's average words per answer, then correlated it with their satisfactory rate (sessions with enough graded elements to be stable; Spearman ρ = 0.41):

Pilots who answer in structured, complete explanations grade ~17 points above pilots who answer in fragments — and carry half the misconception rate (5.6% vs 12.4% marginal). Correlation isn't causation: better-prepared pilots plausibly both know more and say more. But the practical advice survives the caveat, because it's the same advice DPEs give: answer out loud, in full sentences, showing your chain of reasoning. It plays offense and defense at once — complete answers earn credit, and they give a misconception somewhere to surface before checkride day.

Finding 7: what happens when the same pilot is asked the same thing twice

The lazy marketing claim would be "use us more, score higher." Here's the actual within-pilot analysis instead: we took every case where the same pilot was graded on the same ACS code in two or more separate sessions — 1,100+ paired encounters across dozens of pilots — and compared first encounter to second.

• Elements missed the first time: 62% satisfactory on retest. A miss that gets surfaced, with study pointers generated for exactly that element, more often than not stays fixed — consistent with the retrieval-practice literature, where testing beats re-reading for retention (Roediger & Karpicke, 2006).
• Elements satisfactory the first time: 79% held; 21% wobbled on retest. Knowledge needs maintenance like engines do — "I covered that already" has a shelf life.

Taken together (the difference is statistically real — McNemar's test — but modest either way), the honest summary is: the win isn't a rising score, it's coverage. Finding the misses is most of the value, fixing them mostly works, and the wobble rate is the argument for re-testing to full ACS coverage rather than quizzing your known weak spots until they feel comfortable.

Findings that refused to cooperate (kept in, on principle)

• Geography is boring. Satisfactory rates by census region sit in a tight 77–82% band. A state-by-state league table would be great content and is also statistical malpractice at our sample size — individual heavy users dominate single-state cells — so you get the honest version: the legality gap is everywhere, and where you train explains approximately nothing.
• Cramming looks fine, actually. Among pilots who shared a checkride date, the median first mock oral happens 10 days before the ride — a quarter within 2 days. I was ready to write the sermon about last-minute prep; the data declined to cooperate (late starters actually graded slightly higher — most plausibly because pilots at the end of training are simply more finished, not because cramming works). Directional, small sample, not significant. I'm reporting it anyway because cherry-picking only the findings that sell is exactly what this post is trying not to be.
• Session length doesn't predict quality (ρ ≈ −0.08). Long orals aren't struggling orals; short ones aren't sharp ones.

Methodology, caveats, and why you should only half-believe me

• Selection bias is real. These are pilots motivated enough to seek out mock orals; the general applicant population plausibly skews weaker. Some users arrive precisely because they feel unprepared. We can't fully separate the two.
• The examiner is an AI grading against the ACS, with tool-enforced grading semantics (an element is satisfactory only if met unprompted; every grade carries a written justification). We continuously evaluate examiner quality with a separate LLM-judge harness against known failure modes. It is still not a DPE, and these numbers describe practice performance — I make no claim they predict individual checkride outcomes.
• Minimum sample thresholds apply to every per-task and per-element figure; all difficulty figures are conditional on assessment, never miss-counts; correlations reported are Spearman (rank-based, robust to outliers); the within-pilot analysis pairs identical ACS codes to control for element difficulty. Session-duration figures measure connected time and include thinking pauses.
• Coverage isn't random. Orals proceed through the ACS in order, so later areas are graded only in sessions that ran that far. Cross-area comparisons carry that survivorship caveat (which is one reason the headline claims here rest on element-level rates within Area I — covered near-universally — rather than between-area rankings).

What to do with all this

For pilots: the encouraging read of this whole dataset is that the oral is very passable — the strengths are broad, the blind spot is narrow and well-mapped, and surfaced misses mostly stay fixed. The single highest-yield hour of oral prep the data can identify: have something — a CFI, a study partner with the ACS open, or us — interrogate your legality knowledge specifically, in full spoken answers, before the DPE does. That's it. That's the hour.

For flight schools: everything above is computed from percentages because the full-resolution version — readiness by ACS area, deficiency-ranked tasks, and trends over time across your student body — is what we build for partner schools (aggregate-only and consent-gated; admins never see individual transcripts or per-student grades). If you'd like your students' strong and weak spots mapped this way while there's still time to do something about it, email parth@critique-labs.ai.

I'll rerun this analysis when the dataset is a few multiples larger, and I'll publish the deltas — including the ones that make this post look wrong.

— Parth, founder, check-ride.ai