check-ride.ai vs MockDPE: eval, trial findings, and the full transcript

This post compares public product features and a selected check-ride.ai synthetic evaluation with one short MockDPE trial. The check-ride.ai example was selected as a best-performing run from our archive. That selection favors our product; these are different test conditions, not a controlled head-to-head benchmark or an estimate of either product's typical performance.

Product comparison

Both products advertise AI oral-exam practice, ACS-based assessment, and debriefs. check-ride.ai lists Private, Instrument, Commercial, CFI, CFII, and ATP. MockDPE lists Private, Instrument, and Commercial. Both advertise voice interaction; the MockDPE trial reviewed here used its text-answer mode. check-ride.ai advertises aircraft, airport, weather, and examiner customization; MockDPE advertises airport-based scenarios using live METARs and TAFs. These are documented capabilities, not evidence that one product reliably catches more errors. Sources: check-ride.ai and MockDPE.

How we tested MockDPE

On September 21, 2026, we ran its anonymous instrument trial with the Reece Barrett AI examiner. The synthetic candidate, Alex Morgan, had a profile of 160 total hours and 45 instrument-training hours in a Cessna 172 with Garmin G1000. The displayed route was KLEX to KSDF, with KCVG as the alternate and a planned cruise altitude of 5,000 feet.

The candidate's replies came from gpt-4o-mini using the displayed scenario and a fixed profile. The candidate made mistakes; this tests how the examiner responds to imperfect answers, not a real pilot's competence. We captured 10 confirmed candidate answers and 11 examiner turns before an account gate stopped the trial. We did not obtain a completed oral or debrief.

Problems visible in the MockDPE conversation

1. A destination/alternate weather mix-up went unaddressed. In turn 12, the candidate attributed a chance of light drizzle and 3-mile visibility to destination KSDF. The displayed KSDF TAF has P6SM throughout; the PROB30 ... 3SM -DZ group belongs to alternate KCVG. The examiner asked about a separate ceiling misconception, then moved on without visibly challenging the airport mix-up. That is a meaningful missed follow-up: the answer changes which airport is expected to have reduced visibility.

2. The instrument-currency recovery rule was left oversimplified. In turn 2, the candidate said a pilot who has not met the recent-experience requirements must complete an IPC. The examiner began with “Right” and asked about task wording and proficiency standards, but never clarified when an IPC becomes mandatory. Under §61.57(d)(1), an IPC is required after failing to meet the instrument-experience requirements for more than six calendar months; merely becoming noncurrent does not immediately trigger that requirement. Being noncurrent still restricts acting as PIC under IFR. This needed a timing follow-up or correction. 14 CFR §61.57(c)–(d)

3. The requested forecast trend was not fully resolved. Turn 11 asked about changes throughout the destination TAF. The candidate described deterioration, but the saved forecast later raises the ceiling from 1,500 feet to 2,500, then 3,500, before ending at 2,500, with P6SM visibility throughout. After addressing broken versus overcast, the examiner did not return to that broader trend. This is another gap within the same weather exchange, not an independent repeat failure.

4. A smaller weather-source attribution error also passed without clarification. In turn 20, the candidate called AWC the “FAA's Aviation Weather Center.” AWC is part of NOAA's National Weather Service/NCEP. The examiner replied “Right” and moved to aircraft systems. This is a minor attribution issue, not comparable in consequence to confusing destination and alternate weather. NWS description of AWC

There were also some awkward prompts. For example, turn 3 asked the candidate to distinguish “general holding procedures” from “holding procedures and tasks” without making the operational question clear. My assessment is that a concrete currency-expiration example would have been more useful. That is a judgment about question quality, not a demonstrated false regulatory claim.

MockDPE excerpt score

The validated gpt-5.4 transcript evaluation scored question quality 4/5, responsiveness 3/5, assessment quality 3/5, factual accuracy 4/5, and continuity 4/5. Converting those five applicable axes to the harness's 0–100 scale gives 65/100.

The selected check-ride.ai evaluation

The check-ride.ai baseline is the completed synthetic Instrument Rating evaluation run 6d981f7f-1322-4f87-a503-c7e9d350c4d0, conducted on August 17, 2026 using Gemini 3.7 Flash.

This was a voice oral with a synthetic C172RG/KCWI candidate. It completed its configured oral scope with 38 ACS elements evaluated across 29 candidate answers and 29 examiner turns.

The same LLM judge evaluated both sessions. It scored this check-ride.ai run 98/100 and the MockDPE run 65/100. For the check-ride.ai run, the judge found one isolated examiner factual error involving lost-communications approach timing. It found no duplicate questioning, reasoning leaks, premature completion, ungrounded aircraft claims, tool-text leakage, or turn-taking problems.

The full captured MockDPE conversation follows, including the attempted final answer marked separately. Its wording is preserved, including the synthetic candidate's errors.

Full captured MockDPE transcript and scenario

Departure METAR:

SPECI KLEX 220039Z 02013KT 10SM OVC013 21/18 A3000 RMK AO2 T02060178 $

Destination TAF:

TAF KSDF 212326Z 2200/2306 01010G17KT P6SM BKN015 OVC040 FM220300 01006KT P6SM BKN015 FM220900 02007KT P6SM SCT010 OVC015 FM221500 03007KT P6SM BKN025 OVC035 FM221800 03010KT P6SM BKN035 OVC050 FM230000 03006KT P6SM BKN025

Alternate TAF:

TAF KCVG 212325Z 2200/2306 01009KT P6SM OVC009 PROB30 2200/2206 3SM -DZ OVC006 FM220600 04010KT 5SM BR OVC006 FM221300 04010KT P6SM OVC006 FM221800 03009KT P6SM OVC011

The following 21 turns are the complete confirmed conversation in this capture. No turns have been omitted. This is not a completed oral.

Turn 1 · Examiner

Good morning. I'm Reece Barrett, your designated pilot examiner. We'll be working through an IFR oral exam today based on the scenario you see above. Take a moment to review it, and then we'll get started. Walk me through the currency requirements for instrument flight.

Turn 2 · Candidate

Good morning, Mr. Barrett. To maintain instrument currency, a pilot must have completed at least six instrument approaches, holding procedures, and intercepting and tracking courses within the preceding six months. Additionally, if a pilot has not met these requirements, they must complete an instrument proficiency check with a certified instructor.

Turn 3 · Examiner

Right. To ensure your understanding is complete, walk me through whether the regulation distinguishes between general holding procedures and the specific requirement for holding procedures and tasks, as well as the distinction regarding intercepting and tracking courses?

Turn 4 · Candidate

The regulation does not distinguish between general holding procedures and specific tasks; both are included in the requirement to demonstrate holding proficiency. Similarly, the requirement for intercepting and tracking courses encompasses all related tasks, meaning pilots must be proficient in both areas to meet the currency requirements.

Turn 5 · Examiner

While you have correctly identified the tasks required to maintain your instrument experience, how does the expectation of your performance during a proficiency check differ from the requirements used to maintain your instrument experience during that lookback period?

Turn 6 · Candidate

During a proficiency check, the expectation is for the pilot to demonstrate a higher level of skill and accuracy across all instrument tasks, including approaches, holds, and navigation. This is more rigorous than the currency requirements, which focus on having completed the tasks rather than demonstrating proficiency in them.

Turn 7 · Examiner

To be precise regarding that proficiency check, what specific criteria are you held to, and where can you find the required tasks that define the necessary depth of that performance?

Turn 8 · Candidate

The specific criteria for a proficiency check are outlined in the FAA's Practical Test Standards (PTS) or the Airman Certification Standards (ACS) for instrument ratings. These documents define the required tasks and performance standards that a pilot must demonstrate during the checkride.

Turn 9 · Examiner

Describe the atmospheric conditions required for the formation of radiation fog.

Turn 10 · Candidate

Radiation fog forms when the ground cools rapidly at night, causing the air near the surface to cool to its dew point. This typically occurs under clear skies and calm winds, with high humidity levels, allowing moisture in the air to condense into tiny water droplets, creating fog.

Turn 11 · Examiner

How would you interpret the changes in ceiling and visibility at your destination throughout the duration of the provided terminal aerodrome forecast?

Turn 12 · Candidate

The TAF for KSDF indicates that the ceiling will start at broken clouds at 1,500 feet and gradually lower to overcast at 1,500 feet later in the forecast period. Visibility remains generally good at P6SM, but there is a period with a chance of light drizzle and reduced visibility to 3SM. This suggests that conditions may deteriorate, particularly with the lower ceiling and potential for reduced visibility during that time.

Turn 13 · Examiner

Based on the guidance provided in the Aeronautical Information Manual regarding the definition of a ceiling, does a change in the cloud coverage classification from broken to overcast at the same altitude constitute a change in the height of the ceiling?

Turn 14 · Candidate

No, a change in cloud coverage classification from broken to overcast at the same altitude does not constitute a change in the height of the ceiling. The ceiling remains at the altitude of the cloud layer, which in this case is still 1,500 feet. The classification change indicates a change in cloud coverage, not altitude.

Turn 15 · Examiner

Let's continue. What questions do you have, or shall I move to the next topic?

Turn 16 · Candidate

I don't have any questions at the moment; please feel free to move to the next topic.

Turn 17 · Examiner

Describe the specific sources of weather data you would utilize for your cross-country flight planning, and explain how you would determine which of those sources provides the most reliable information for your departure, en route, and destination segments.

Turn 18 · Candidate

For cross-country flight planning, I would utilize sources such as the Aviation Weather Center, METARs, TAFs, and graphical forecasts. To determine reliability, I would consider the timeliness of the data, the proximity of the reporting stations to my route, and any significant weather advisories or alerts that may affect my flight. Additionally, I would cross-reference multiple sources to ensure consistency in the information provided.

Turn 19 · Examiner

When gathering a pre-flight weather briefing, how do you verify that the information you are accessing is from an officially recognized source?

Turn 20 · Candidate

To verify that the information accessed during a pre-flight weather briefing is from an officially recognized source, I would ensure that I am using resources such as the FAA's Aviation Weather Center, the National Weather Service, or other government-operated platforms. Additionally, I would look for data that is published with official identifiers, such as METAR and TAF reports, which are standardized and widely accepted in aviation.

Turn 21 · Examiner

Right. Speaking of your equipment, walk me through the pitot-static instruments on this aircraft and describe how the system would react if the static port were to become obstructed in flight.

Attempted candidate answer 11 — not confirmed as processed

The following answer appeared as a local echo before the account gate. No examiner response was observed, so it was excluded from the judged conversation above:

In the Cessna 172 with a Garmin G1000, the pitot-static system includes the airspeed indicator, altimeter, and vertical speed indicator. If the static port becomes obstructed in flight, the altimeter would freeze at its current reading, and the vertical speed indicator would show no change. The airspeed indicator would behave erratically, potentially indicating a false airspeed based on the difference between the pitot pressure and the static pressure.

End of capture: “Create a Free Account to Continue.” No further examiner reply or debrief was captured.