Skip to results
salesevals.com/Evaluated Jul 1, 2026

Which models know sales?

26 model configurations coach GPT- and Sonnet-generated synthetic sales calls with hidden ground truth. A judge scores each coaching note from 0–100 on whether it found the real strengths, flaws, and next moves.

Calls
50
Models
26
Evaluations
1300
Benchmark
86.2
50 calls · 1300 evaluationsRank: Sales coaching benchmarkAll available runsBuild-time static dataEvals completed Jul 1, 2026
50 benchmark calls

The 50 calls

Open a call to read its answer key and model scores.

Duolingo Renewal QBR and expansion planning with Amplitude

QBRexcellentSonnet-generated52m · 40 turns
SellerAmplitude
BuyerDuolingo

A renewal QBR between Amplitude (seller) and Duolingo (buyer) in which the Amplitude AE demonstrates exceptional pre-call preparation by opening with specific metric movements pulled from Duolingo's own instance, earns trust by framing the renewal as a formality backed by data, then surgically pivots to two expansion use cases (Amplitude Experiment and Session Replay) grounded in Duolingo's publicly stated DAU obsession and the Duolingo Max onboarding problem. The buyer's VP of Product Growth and a senior PM are engaged and increasingly co-authoring the conversation. The call closes with buyer-originated next steps and a mutual success plan. One minor imperfection: the seller briefly over-explains the MTU pricing mechanics when the buyer had already signaled acceptance, slightly elongating that segment.

Profile
Excellent
Transcript origin
Sonnet-generated
Flaws / Strengths
1 / 4
Duration
52m · 40 turns

What this call should surface

+ strength

Data-led value recap using buyer's own metrics

Research · moderate

+ strength

Experimentation velocity discovery surfaces the expansion problem organically

Discovery · moderate

+ strength

Session Replay framed as a specific 90-day pilot with a defined success metric

Value Alignment · moderate

+ strength

Buyer authors the mutual success plan unprompted by a leading close

Next Steps · subtle

flaw

Seller over-explains MTU pricing mechanics after buyer signals acceptance

Communication Style · subtle

40 speaker turns · 52m timeline

Transcript

The exact speaker-labeled transcript the coach models saw.

Marcus ChenSellerPriya SharmaBuyerPriya NairSellerJordan LeeBuyer
  1. MC

    Marcus Chen

    Seller

    Hey everyone, good to see you — Marcus Chen, Amplitude, I'm the AE on the Duolingo account. Really glad we could get this on the calendar. We've got a lot of good stuff to walk through today. Quick agenda from my side: I want to start with what the data actually shows about the last year — your data, not a generic benchmark deck — then talk through the commercial picture, and then have a real conversation about where the next twelve months could go. Sound good?

  2. PS

    Priya Sharma

    Buyer

    Priya Sharma, VP of Product Growth at Duolingo. I own the DAU strategy and the subscription funnel, so I'm the right person to be in this conversation. Excited to see what you've pulled together — I'll be honest, I'm hoping this is more than a slide deck.

  3. PN

    Priya Nair

    Seller

    Priya Nair, Solutions Consultant on Marcus's team. I'm here for the technical depth — instrumentation, experimentation methodology, that kind of thing. Looking forward to it.

  4. JL

    Jordan Lee

    Buyer

    Jordan Lee, senior PM on the growth team. I live in Amplitude day-to-day, so I'm curious what you found in our data.

  5. MC

    Marcus Chen

    Seller

    Alright, let me pull up the dashboard — give me one second to share my screen.

  6. MC

    Marcus Chen

    Seller

    Okay, you should be seeing my screen now — three charts, all pulled from your Amplitude instance this morning. Let me walk through what jumped out at me. First one: your D7 retention for users who converted to Super Duolingo. Over the last three quarters, that cohort is up 14% quarter-over-quarter. And when I cut it by the users who hit the streak milestone in their first week versus those who didn't — the gap is 22 percentage points. That's not a small number. Second chart is your DAU/MAU ratio trending from Q2 through Q4 last year — it's moved from 0.26 to 0.31, which puts you well above the consumer app median we see across our book. And the third one is where it gets interesting: this is the Duolingo Max trial-to-paid funnel. Strong top of funnel, solid paywall hit rate — and then there's a step where completion drops sharply. I want to come back to that one. But I wanted to start here because this is your data telling a story, and I think it's a good one.

  7. JL

    Jordan Lee

    Buyer

    That third chart — the Max funnel drop-off — where exactly is the break happening? Which step?

  8. MC

    Marcus Chen

    Seller

    It's the step right after the paywall screen — users hit 'Start Free Trial,' we see the confirmation event fire, and then there's a significant drop before the first AI lesson loads. That gap is where we lose them.

  9. JL

    Jordan Lee

    Buyer

    That matches what I've been seeing too — honestly, that step has been a black box for us. We can tell people are dropping, we just can't tell why.

  10. MC

    Marcus Chen

    Seller

    Yeah, that gap is real. So before we go further — how are you currently trying to diagnose it? Like what tools are you reaching for when you want to understand why users drop at that step?

  11. JL

    Jordan Lee

    Buyer

    Honestly? A mix of things. We'll pull the event data in Amplitude, but once we're past what the events can tell us, we're kind of guessing. We've tried looking at load time logs on the engineering side, but that's a separate system. There's no single place where we can just see what a user actually experienced at that step.

  12. MC

    Marcus Chen

    Seller

    Got it. So there's basically a visibility cliff after the paywall event fires. How long has that been the case — is this a recent thing or has that step always been opaque?

  13. JL

    Jordan Lee

    Buyer

    Since we launched Max, basically. The events were never built out past the paywall confirmation — it was kind of a 'we'll come back to this' situation that never got prioritized.

  14. MC

    Marcus Chen

    Seller

    Right, so that gap has basically been baked in since day one. Okay — I want to bring Priya in here, because she actually looked at your event schema before this call and noticed something specific about that step.

  15. PN

    Priya Nair

    Seller

    Yeah, so — I was looking at your event schema in your Amplitude instance ahead of this call. Your Max onboarding flow has really solid coverage at the top: the paywall impression, the trial CTA tap, the confirmation event. But between that confirmation and the first AI lesson load, there are no intermediate events at all. Nothing on the loading state, nothing on the permissions prompt if it fires, nothing on the model initialization step. So from a pure event standpoint, you genuinely cannot tell whether users are dropping because of a slow load, a confusing UI moment, or something else entirely. That's not an instrumentation failure on your team's part — that step is just below the resolution threshold of event tracking. It's exactly the gap Session Replay was built for, because you'd be watching the actual experience rather than inferring it from events that don't exist yet.

  16. JL

    Jordan Lee

    Buyer

    That tracks exactly with what I've been trying to explain to our eng team. So — can Session Replay actually filter replays by experiment variant? Because if I'm running a test on that onboarding step, I need to be able to watch sessions from the control group separately from the treatment.

  17. PN

    Priya Nair

    Seller

    Yes — and that's actually one of the things I'm most excited to show you. You can filter session replays by any property you're already passing as a user property or event property, and experiment variant is just another property. So if you've got a test running on that onboarding step, you pull up the replay list, filter to variant B, and you're watching exactly those sessions. You can even watch them side by side with control sessions if you want to compare the experience directly. And because the replay is stitched to the same event stream you're already tracking in Amplitude, you're not context-switching — the funnel drop-off and the session that caused it are on the same screen.

  18. JL

    Jordan Lee

    Buyer

    Okay, that's — yeah, that's exactly what I'd need. Honestly that answers my question better than I expected.

  19. PN

    Priya Nair

    Seller

    Good. So let me actually bring this back to the broader pilot framing, because I think we have the pieces. Marcus, you want to take that?

  20. MC

    Marcus Chen

    Seller

    Yeah, let me take it. So — Priya Sharma, here's where I'd like to land on the pilot. We've got a real, named problem: the Max onboarding step between paywall confirmation and first AI lesson is a black box. Session Replay closes that gap without any new instrumentation from your team — you'd be watching real sessions from users who hit that step, filtered by experiment variant if Jordan's running a test on it. What I'd propose is a 90-day pilot scoped specifically to that funnel, with a single success metric: ten percent improvement in Max trial-to-paid conversion. That's it. If we hit that, the ROI case writes itself at renewal. If we don't, we know exactly why and we've learned something real about that funnel either way.

  21. PS

    Priya Sharma

    Buyer

    That ten percent number — is that against current baseline, or are you modeling from a specific cohort?

  22. MC

    Marcus Chen

    Seller

    Against current baseline. Your Max trial-to-paid rate over the last sixty days — I pulled it before this call. That's the denominator.

  23. PS

    Priya Sharma

    Buyer

    Good. And honestly that number is achievable — we've seen similar baselines move more than that just from closing instrumentation gaps. So I'm in on the pilot framing.

  24. MC

    Marcus Chen

    Seller

    Jordan, anything you want to add before we move to the commercial side?

  25. JL

    Jordan Lee

    Buyer

    Nothing from me — let's get to the commercial stuff.

  26. MC

    Marcus Chen

    Seller

    Alright. So on the commercial side — your current contract is sitting at forty million MTUs per month, and your actual usage over the last two quarters has been running closer to fifty-two, fifty-three million. So there's an overage we need to true up, and then we size the renewal tier to where you're actually operating. Priya, does that framing make sense before I get into the numbers?

  27. PS

    Priya Sharma

    Buyer

    Yeah, makes sense. We figured it would go up — we've grown a lot this year. What's the new tier look like?

  28. MC

    Marcus Chen

    Seller

    So the new tier — sixty million MTUs per month, which gives you some headroom above where you're running now. On the overage, we'd true that up as a one-time line item at the current per-unit rate. So two separate things: the true-up, and then the new annual rate at the sixty-million tier. The way the tiers work, each band is priced on a per-MTU basis that steps down as you go up — so at sixty million you're actually paying a lower effective rate per user than you were at forty million. The overage calculation runs off the delta between your contracted forty million and your actual monthly peaks, averaged across the two quarters, which works out to roughly — actually, let me just pull up the number directly.

  29. PS

    Priya Sharma

    Buyer

    Right, so what's the actual number?

  30. MC

    Marcus Chen

    Seller

    The true-up is four hundred twelve thousand dollars. New annual rate at sixty million MTUs is in the proposal I'll send right after this.

  31. PS

    Priya Sharma

    Buyer

    Got it. And the annual rate — is that a meaningful jump from where we are today?

  32. MC

    Marcus Chen

    Seller

    It's a step up — I won't pretend otherwise. But given where your MTUs actually landed, it's more of a catch-up than a jump. The proposal will have the exact line items so you can run it through your finance team.

  33. PS

    Priya Sharma

    Buyer

    Okay. Send it over and I'll loop in finance this week. I think we're good on the commercial side — what else did you want to cover before we wrap?

  34. MC

    Marcus Chen

    Seller

    One thing we haven't fully landed on yet — and I want to make sure we leave time for it — is what the next year actually looks like for your team on the product side. Not just the renewal, but whether there's a bigger opportunity here. Can I spend five minutes on that before we wrap?

  35. PS

    Priya Sharma

    Buyer

    Yeah, go for it.

  36. MC

    Marcus Chen

    Seller

    So the question I want to leave us with before we close out — and I'm genuinely curious what your answer is — is: what would need to be true in the next ninety days for next year's renewal to be a no-brainer? Like, not from my side, from yours. What does good look like?

  37. PS

    Priya Sharma

    Buyer

    Honestly? For me it's two things. One — we need to see Max trial-to-paid move. If the Session Replay pilot actually helps us figure out what's happening at that paywall step, and we can point to a number, that's a win I can take to leadership. Two — I need the experimentation story to be cleaner. If Jordan's team is still context-switching between DuoTest and Amplitude six months from now, that's a problem we didn't solve. Jordan, you want to add anything?

  38. JL

    Jordan Lee

    Buyer

    Yeah — both of those. The Max conversion number and getting the experimentation stack consolidated. Those are the two things I'm writing down right now as the actual definition of success. Jordan, what would you add on the technical side before we close out?

  39. MC

    Marcus Chen

    Seller

    Perfect. So let me just lock in what I'm writing down as our two things: Max trial-to-paid moves in the ninety-day pilot, and the experimentation stack gets cleaner — Amplitude and DuoTest living in one view. Jordan, I'll reach out this week to get the technical scoping session on the calendar. Priya, proposal goes to you and finance by Thursday. And let's put a thirty-day check-in on the books so we're not waiting until renewal to see how it's tracking. Sound good?

  40. PS

    Priya Sharma

    Buyer

    Yeah, that works. Thanks both — good call.

Sorted by benchmark score

How each model scored this call

Click a row to read the model's coaching note and the judge's read on it.

186opus 4.8 xhighBestStrong coach output with minor over-coaching
Overall87
Needle recall84
Evidence grounding91
False-positive control80
Prioritization84
Actionability92
Sales instinct90
Technical accuracy91
How this model did

The coach captured the dominant shape of the call: an excellent, highly prepared renewal QBR with a buyer-specific data recap, strong diagnosis of the Max onboarding gap, a well-framed Session Replay pilot, low-friction commercial handling, and a buyer-authored success plan. It also correctly identified the subtle MTU/pricing over-explanation flaw. The main weakness is that the coach somewhat over-penalized the experimentation thread and commercial section: it treated Experiment/DuoTest as a major missed opportunity rather than also recognizing that the buyer did author it as a success criterion and that the seller captured it into next steps. It also introduced a medium-risk critique around not giving the renewal annual rate live, which is transcript-supported but not clearly benchmark-critical and somewhat speculative given the buyer’s low-friction response.

Strongest findings
  • Correctly identified the data-led QBR opening as the central reason the renewal felt value-backed rather than generic.
  • Strongly captured the Max onboarding diagnosis sequence: Marcus asked how Duolingo diagnosed the gap, Jordan called it a black box, and Priya Nair validated the event-schema limitation with technical specificity.
  • Accurately praised the Session Replay pilot framing: named use case, 90-day scope, 10% trial-to-paid success metric, and buyer acceptance.
  • Correctly recognized the buyer-authored close and mutual success plan as a major signal of buy-in.
  • Caught the subtle MTU/commercial flaw: Marcus kept explaining pricing mechanics when the buyer mainly wanted the number and had already accepted the general increase.
Biggest misses
  • The coach did not fully credit the Experiment expansion thread as a positive buyer-authored success criterion; it focused mostly on missing discovery.
  • It over-weighted commercial issues relative to the benchmark. The call’s commercial outcome was low-friction, and the annual-rate deferral did not visibly create buyer resistance.
  • It did not precisely frame the MTU flaw as “continued explanation after buyer acceptance,” though it captured the general over-explanation issue.
  • Some coaching suggestions, especially revenue quantification and renewal-rate range disclosure, are reasonable but go beyond the hidden benchmark and are more speculative than the core findings.
286opus 4.7 highStrong / mostly accurate
Overall86
Needle recall88
Evidence grounding84
False-positive control78
Prioritization81
Actionability93
Sales instinct89
Technical accuracy90
How this model did

The coach output captures the most important realities of the call: this was a very strong renewal QBR, anchored in Duolingo-specific data, with a well-framed Session Replay pilot, strong technical credibility, buyer-authored success criteria, and a minor commercial execution flaw. It is highly actionable and generally well grounded in transcript evidence. The main issues are that it somewhat over-weights the Experiment/DuoTest gap as a high-severity miss relative to an otherwise excellent call, and it includes at least one unsupported invented buyer-language claim around revenue impact.

Strongest findings
  • Correctly identifies the data-led opening as the main reason the QBR earned credibility with Duolingo.
  • Accurately praises the Session Replay motion as diagnosis-led rather than feature-led.
  • Strongly captures the buyer-authored close and the value of Marcus's “what would need to be true” question.
  • Correctly flags the MTU commercial segment as the one real execution blemish in an otherwise strong call.
  • Provides highly actionable coaching drills, especially around commercial choreography and experimentation discovery.
Biggest misses
  • The coach somewhat under-rates the overall call by calling it merely “above-average” and making the experimentation gap a high-severity issue, whereas the hidden profile is closer to excellent with one minor flaw.
  • The Experiment/DuoTest critique is transcript-grounded, but it conflicts with the hidden benchmark's more positive framing of Experiment expansion being opened successfully.
  • The revenue-impact missed opportunity contains an invented buyer quote/style claim, which weakens evidence discipline.
  • The coach adds several extra expansion ideas beyond the benchmark; most are plausible, but some are speculative relative to this specific call.
383fable 5 highStrong coach output with a few prioritization and benchmark-alignment gaps
Overall84
Needle recall80
Evidence grounding91
False-positive control78
Prioritization79
Actionability92
Sales instinct87
Technical accuracy90
How this model did

The coach accurately recognized the main shape of the call: an excellent, data-led renewal QBR with strong Duolingo-specific preparation, excellent SC technical credibility, a well-framed Session Replay pilot, and buyer-authored success criteria at the close. The output is highly transcript-grounded and actionable. The biggest miss is that it did not identify the hidden benchmark’s specific minor flaw: Marcus over-explained MTU tier mechanics after Priya had already signaled acceptance. Instead, the coach reframed the commercial weakness around price anchoring and a live true-up “fumble,” which is supported by the transcript but over-prioritized relative to the benchmark. The coach also treated the experimentation thread mostly as a missed opportunity; this is directionally fair because the seller did not run explicit Experiment discovery, but it under-credits that the buyer did surface experimentation consolidation as a success criterion and that an Experiment expansion path was opened.

Strongest findings
  • Correctly identified the data-led QBR opening as the foundation of trust and renewal momentum.
  • Accurately praised Priya Nair’s schema-level, blame-free technical diagnosis as a major credibility-builder.
  • Nailed the Session Replay pilot framing: specific Duolingo Max funnel, 90-day scope, 10% trial-to-paid target, and baseline readiness.
  • Correctly highlighted the buyer-authored close and the value of Priya Sharma defining renewal success in her own words.
  • Gave highly actionable follow-up coaching, especially around converting the DuoTest/context-switching comment into structured Experiment discovery.
Biggest misses
  • Missed the specific hidden flaw that Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance.
  • Over-prioritized commercial price anchoring relative to the benchmark, which viewed the commercial segment as mostly resolved with only a minor call-economy issue.
  • Under-credited the experimentation expansion path as opened by the buyer-authored success criteria, even though it was fair to note the seller did not proactively discover it earlier.
  • Added a few plausible but not fully evidenced risks, such as pilot commercial terms becoming a finance issue and a dropped buyer question at the close.
483gpt-5.5 mediumStrong, mostly aligned coaching output with a few notable benchmark misses.
Overall84
Needle recall76
Evidence grounding90
False-positive control84
Prioritization80
Actionability90
Sales instinct84
Technical accuracy91
How this model did

The coach correctly recognized the call as a high-quality renewal QBR and captured the biggest transcript-grounded strengths: Duolingo-specific value recap, strong technical diagnosis, Session Replay as a measurable 90-day pilot, and a buyer-owned close. The output is well evidenced and actionable. The main gaps are that it did not identify the subtle MTU-pricing flaw in the benchmark as over-explaining after buyer acceptance, and it treated the experimentation opportunity as under-discovered rather than identifying the benchmarked experimentation-discovery strength. Some commercial critiques are reasonable but slightly over-prioritized given the buyer’s low-friction acceptance.

Strongest findings
  • Correctly identified the data-led QBR opening as a major strength and cited the exact Duolingo metrics used.
  • Strongly captured the Max onboarding black-box pain and the seller’s decision to pursue that live buying signal instead of continuing a generic presentation.
  • Accurately praised Priya Nair’s technical diagnosis of the missing event coverage and her answer about filtering Session Replay by experiment variant.
  • Correctly recognized the Session Replay proposal as a buyer-specific, measurable 90-day pilot with a 10% trial-to-paid conversion target.
  • Identified the buyer-owned close and the two buyer-defined success criteria for the next 90 days.
Biggest misses
  • Did not identify the subtle benchmark flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance and then had to be redirected to the actual number.
  • Did not score the experimentation-discovery thread as the benchmarked strength; instead, it framed experimentation consolidation mainly as an under-discovered missed opportunity.
  • Slightly over-weighted extra commercial and operational critiques relative to the call’s very strong positive outcome and low-friction renewal path.
581gpt-5.5 noneStrong but imperfect coaching evaluation
Overall84
Needle recall74
Evidence grounding88
False-positive control78
Prioritization80
Actionability90
Sales instinct86
Technical accuracy85
How this model did

The coach captured the dominant shape of the call very well: an excellent, customer-specific renewal QBR with strong preparation, credible technical diagnosis, a concrete Session Replay pilot, and buyer-authored success criteria. The output is well grounded in transcript quotes and gives actionable coaching. The main gaps are that it only partially handles the Experiment/experimentation expansion thread and misses the subtle benchmark flaw: Marcus over-explains MTU pricing mechanics after Priya Sharma has already signaled acceptance. Instead, the coach reframes the commercial issue as pricing readiness, which is partly supported but not the most accurate coaching point for this transcript.

Strongest findings
  • Accurately praised the customer-specific opening built on Duolingo’s own Amplitude data, including D7 retention, DAU/MAU, and Max funnel metrics.
  • Correctly identified the Session Replay expansion motion as tied to a specific Duolingo Max onboarding drop-off rather than a generic feature pitch.
  • Highlighted Priya Nair’s strong technical credibility around missing intermediate events and the limits of event-only instrumentation.
  • Captured the value of answering Jordan’s variant-filtering question directly and practically.
  • Correctly recognized the buyer-authored success criteria at the end of the call.
Biggest misses
  • Missed the subtle MTU-pricing flaw: Marcus continues explaining tiers and overage mechanics after Priya Sharma has already accepted the usage-growth framing.
  • Only partially handled the experimentation expansion needle; the coach saw the consolidation signal but did not identify a strong seller-led experimentation discovery motion.
  • Overweighted commercial readiness as the main commercial coaching point when the benchmark issue is more about brevity and reading buyer acceptance signals.
  • Treated a likely transcript speaker-attribution glitch as a call-control problem.
681opus 4.7 xhighStrong, mostly grounded coaching output with some over-criticism
Overall82
Needle recall82
Evidence grounding84
False-positive control73
Prioritization74
Actionability91
Sales instinct86
Technical accuracy88
How this model did

The coach correctly identified the biggest positive patterns in the call: data-led QBR preparation, a tightly scoped Session Replay pilot, strong SC technical credibility, buyer-authored success criteria, and concrete next steps. It also partially caught the subtle commercial flaw, though it framed it more as a math/preparation stumble than the benchmark’s more precise issue: Marcus kept explaining MTU mechanics after Priya had already accepted the framing. The main weakness is prioritization: the coach downgrades an otherwise excellent renewal/expansion call to merely “above-average,” over-weights experimentation as a missed opportunity, and introduces several optional expansion ideas that were not central to the benchmark. Overall, it is a high-quality sales coaching read, but not perfectly aligned to the hidden ground truth’s excellent-call profile.

Strongest findings
  • Correctly recognized the gold-standard QBR opening: Marcus used Duolingo’s own instance data and specific business metrics rather than a generic deck.
  • Accurately praised the SC’s technical credibility and non-blaming diagnosis of the Max onboarding instrumentation gap.
  • Captured the core value-based expansion motion: Session Replay was tied to a named Duolingo funnel, a 90-day pilot, and a 10% trial-to-paid success metric.
  • Correctly identified the buyer-authored mutual success plan and the strength of Marcus’s open-ended closing question.
  • Provided actionable follow-up coaching, especially around quantifying experimentation discovery and preparing commercial numbers.
Biggest misses
  • The coach did not precisely identify the hidden commercial flaw as over-explaining MTU pricing mechanics after the buyer had already signaled acceptance; it reframed the issue as a broader math/preparedness stumble.
  • It under-rated the call overall. The benchmark profile is excellent with one minor flaw, while the coach called it merely above-average and assigned several medium/high risks.
  • It over-penalized the experimentation thread. Deeper discovery was indeed missing, but the buyer did express a clear Experiment-related success criterion and the expansion pipeline was still opened.
  • It introduced several extra product-expansion critiques that are only loosely connected to the transcript and could distract from the highest-leverage coaching points.
  • Some evidence claims were slightly imprecise, especially around DuoTest being named multiple times or being part of pre-call preparation.
781deepseek v4 proStrong but incomplete coaching output
Overall84
Needle recall68
Evidence grounding88
False-positive control78
Prioritization82
Actionability89
Sales instinct87
Technical accuracy91
How this model did

The coach correctly recognized the call as an excellent renewal/expansion QBR and captured the biggest strengths: Duolingo-specific metric recap, strong Max onboarding diagnosis, technically credible Session Replay positioning, a 90-day pilot with a 10% conversion target, and buyer-authored success criteria at the close. The main miss is the subtle commercial flaw: Marcus over-explained MTU pricing mechanics after Priya had already accepted the direction, and the coach instead introduced a different, less-supported commercial risk about not quoting the annual rate live. The coach also only partially captured the Experiment expansion thread; it noticed experimentation consolidation but did not identify the benchmarked consultative discovery pattern around A/B testing volume/tooling.

Strongest findings
  • Correctly identified the data-led QBR opening using Duolingo's own metrics and named product surfaces.
  • Accurately highlighted Priya Nair's technical credibility in diagnosing the missing intermediate events in the Max onboarding flow.
  • Clearly captured the value-based Session Replay expansion motion: specific funnel, 90-day pilot, and 10% trial-to-paid conversion target.
  • Recognized the buyer-authored mutual success plan and the importance of the 30-day check-in before renewal.
  • Mostly grounded its feedback in transcript quotes rather than generic sales platitudes.
Biggest misses
  • Missed the subtle MTU pricing over-explanation after buyer acceptance, which was the primary coaching-worthy flaw in an otherwise excellent call.
  • Only partially captured the Experiment expansion needle; it noticed DuoTest/Amplitude consolidation but did not identify a seller-led discovery sequence around experimentation volume or tooling.
  • Substituted an unsupported commercial risk about not quoting the annual rate live for the actual commercial communication flaw.
  • Did not explicitly call out the commercial segment's call-economy issue: Priya's “Right, so what's the actual number?” suggested Marcus should have been more concise.
881gpt-5.4 highStrong coaching output with good grounding, but it missed the benchmark’s subtle commercial flaw and treated the experimentation expansion thread more as an under-qualified missed opportunity than as a benchmark strength.
Overall82
Needle recall74
Evidence grounding90
False-positive control82
Prioritization76
Actionability88
Sales instinct85
Technical accuracy91
How this model did

The coach correctly recognized the call as high quality and strongly identified the major strengths around Duolingo-specific data preparation, technical diagnosis, Session Replay pilot framing, and buyer-authored next steps. Its evidence is mostly transcript-grounded and its coaching advice is actionable. The main evaluation gap is that it did not catch the specific hidden flaw: Marcus continued explaining MTU pricing mechanics after Priya had already signaled acceptance, prompting her to redirect with “Right, so what’s the actual number?” Instead, the coach focused on a different commercial issue—annual-rate readiness—which is plausible but not the benchmarked flaw and somewhat over-inferred. The coach also framed the Experiment/DuoTest thread as insufficiently discovered, which is grounded in the transcript, but does not fully align with the hidden benchmark’s intended positive needle around experimentation expansion discovery.

Strongest findings
  • Correctly praised the customer-specific, data-led QBR opening with concrete Duolingo metrics.
  • Correctly identified the Max onboarding visibility gap and the strong AE-to-SC technical handoff.
  • Correctly highlighted Session Replay as a buyer-specific, metric-bound 90-day pilot rather than a generic upsell.
  • Correctly recognized the buyer-authored mutual success plan and the strong closing question.
Biggest misses
  • Missed the subtle MTU pricing flaw: Marcus kept explaining mechanics after Priya had already accepted the need to resize and then redirected him to the actual number.
  • Did not align cleanly with the hidden experimentation-discovery strength; it focused on lack of qualification rather than identifying an organic Experiment expansion motion.
  • Over-prioritized commercial readiness and monetization critiques relative to the benchmark’s view of the call as excellent with only one minor commercial-style flaw.
981opus 4.8 maxStrong coaching output with one important benchmark miss
Overall82
Needle recall80
Evidence grounding90
False-positive control76
Prioritization75
Actionability90
Sales instinct84
Technical accuracy88
How this model did

The coach accurately recognized most of the call’s standout strengths: the Duolingo-specific data-led opening, credible technical diagnosis of the Max onboarding gap, the 90-day Session Replay pilot with a 10% trial-to-paid success metric, and the buyer-authored close. The output is well grounded in transcript evidence and highly actionable. Its main gap is that it missed the hidden benchmark’s subtle commercial flaw: Marcus continued explaining MTU tier mechanics after Priya had already accepted the framing, and Priya’s “Right, so what’s the actual number?” was the clue. Instead, the coach emphasized a different commercial issue—the lack of a verbal annual renewal number—which is transcript-based but over-prioritized relative to the ground truth and the buyer’s low-friction acceptance.

Strongest findings
  • Correctly highlighted the best-in-class opening with Duolingo’s own metrics rather than generic benchmark material.
  • Strongly identified the technical credibility created by Priya Nair’s event-schema analysis and explanation of why Session Replay fits the Max onboarding visibility gap.
  • Accurately praised the 90-day Session Replay pilot with a concrete 10% Max trial-to-paid conversion success metric.
  • Correctly recognized the buyer-authored close and mutual success plan as a meaningful buy-in signal.
  • Appropriately noticed that rigorous Amplitude Experiment discovery was not actually performed, despite the buyer later naming experimentation consolidation as important.
Biggest misses
  • Missed the subtle benchmark flaw: Marcus over-explained MTU pricing mechanics after Priya had already accepted the usage/true-up framing.
  • Over-prioritized the lack of a live annual renewal number compared with the transcript’s actual commercial dynamic, which was low-friction acceptance and follow-up proposal routing.
  • Did not coach the exact behavioral lesson from the pricing segment: when the buyer says the increase makes sense, stop justifying the mechanics and move directly to the number or next step.
1080gpt-5.4 xhighGood coaching output with strong coverage of the main positives, but it missed the benchmark’s subtle commercial flaw and only partially handled the experimentation expansion thread.
Overall82
Needle recall72
Evidence grounding88
False-positive control80
Prioritization78
Actionability86
Sales instinct84
Technical accuracy90
How this model did

The coach accurately recognized the call as a strong renewal/expansion QBR, grounded praise in Duolingo-specific metrics, captured the Max onboarding blind spot, credited the 90-day Session Replay pilot, and noted the buyer-authored success criteria at the close. The biggest gap is that the coach did not identify the specific MTU-pricing flaw: Marcus kept explaining tier/overage mechanics after Priya had already accepted the framing. Instead, the coach reframed the commercial issue as lack of pricing crispness and failure to state the annual rate live, which is adjacent but not the hidden benchmark issue. The coach also treated the Experiment/DuoTest thread mainly as underqualified, rather than recognizing the benchmarked expansion motion as a strength; this is partially grounded in the transcript but does not fully match the hidden needle.

Strongest findings
  • Correctly praised the buyer-specific opening metrics from Duolingo’s Amplitude instance.
  • Correctly identified the Max onboarding black box and current-state discovery as the path into Session Replay.
  • Correctly credited Priya Nair’s technical credibility on schema gaps and filtering replays by experiment variant.
  • Correctly highlighted the 90-day Session Replay pilot with a 10% trial-to-paid success metric.
  • Correctly recognized the buyer-authored close around Max conversion and experimentation consolidation.
Biggest misses
  • Missed the exact subtle MTU flaw: Marcus kept explaining pricing mechanics after Priya had already accepted the framing.
  • Only partially matched the experimentation expansion benchmark; the coach saw the DuoTest thread but mostly treated it as underqualified rather than as a strong organically surfaced expansion path.
  • Overweighted commercial crispness as a high-severity risk despite a smooth buyer response and low-friction commercial outcome.
  • Did not explicitly state that the renewal was effectively confirmed and that the MTU overage resolved without meaningful friction, though it did say the renewal likely moved forward.
1180gpt-5.4 noneMostly correct, with one important subtle miss
Overall82
Needle recall74
Evidence grounding89
False-positive control78
Prioritization76
Actionability88
Sales instinct84
Technical accuracy88
How this model did

The coach output correctly recognized the call as a strong, value-led renewal/QBR and captured the biggest strengths: Duolingo-specific data in the opening, a tightly scoped Session Replay pilot with a 10% Max conversion target, strong SC technical credibility, and buyer-authored success criteria at the close. It partially captured the Experiment expansion signal through the DuoTest/Amplitude context-switching discussion, but treated it more as an underdeveloped opportunity than as an organically surfaced expansion thread. The main miss is the hidden flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance. The coach instead diagnosed the commercial issue as insufficient pricing preparedness, which is only weakly supported and misprioritizes the actual coaching point.

Strongest findings
  • Correctly identified the customer-specific, data-led QBR opening as a major strength.
  • Correctly praised the Session Replay expansion as a named Max onboarding use case with a 90-day pilot and a 10% conversion success metric.
  • Correctly highlighted Priya Nair’s technical credibility around event schema gaps and filtering Session Replay by experiment variant.
  • Correctly recognized the buyer-authored close where Priya Sharma defined success around Max conversion movement and experimentation stack consolidation.
Biggest misses
  • Missed the subtle MTU pricing flaw: Marcus kept explaining pricing mechanics after the buyer had already signaled acceptance.
  • Partially missed the benchmark’s positive framing of the Experiment expansion thread, treating it mainly as an underdeveloped opportunity rather than an organically surfaced expansion problem.
  • Over-prioritized commercial pricing preparedness as the main improvement area, which is less aligned with the hidden ground truth than call economy/read-the-room coaching.
  • Did not fully state that the renewal was effectively confirmed and that the expansion pipeline for Session Replay and Experiment had strong buyer buy-in.
1280gpt-5.5 highStrong but not complete. The coach output is well grounded and captures most of the call’s major strengths, especially the customer-data QBR opening, the Session Replay pilot framing, and the buyer-authored close. Its main miss is the subtle benchmark flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance. The coach also only partially captured the Experiment expansion thread and framed it more as an underdeveloped opportunity than as a benchmark strength.
Overall82
Needle recall69
Evidence grounding90
False-positive control84
Prioritization78
Actionability88
Sales instinct84
Technical accuracy86
How this model did

The coach correctly judged the call as high quality and provided extensive transcript-backed coaching. It identified the strongest elements of the call: Duolingo-specific metrics, schema-level technical credibility, a named Max onboarding pain point, a 90-day Session Replay pilot with a 10% conversion target, and a strong closing question that let the buyer define success. However, it missed the specific commercial coaching point in the hidden benchmark: after the buyer accepted the MTU increase framing, Marcus continued into a detailed tier/overage explanation instead of moving crisply to the number. The coach discussed commercial crispness, but for different reasons. It also did not identify the benchmark’s experimentation-discovery needle as a strength; instead it treated experimentation consolidation as a thread the seller should have developed more. Overall, this is a useful and mostly accurate coaching output, but it misses one subtle but important call-economy flaw and partially misaligns on the Experiment expansion assessment.

Strongest findings
  • Correctly recognized the Duolingo-specific value recap as a major QBR strength, with precise evidence around D7 retention, DAU/MAU, and the Max funnel.
  • Correctly identified the Session Replay expansion as value-based rather than feature-based because it was tied to the Max onboarding drop-off and a 10% trial-to-paid conversion target.
  • Correctly praised the SC’s technical credibility, especially the schema-level diagnosis of missing intermediate events and the answer about filtering replays by experiment variant.
  • Correctly highlighted the closing question as a strong move that caused Priya Sharma to articulate the two buyer-defined success criteria.
  • Provided actionable coaching around ROI quantification, stakeholder/process control, pilot dependencies, and Experiment discovery without inventing major unsupported facts.
Biggest misses
  • Missed the subtle MTU-pricing flaw: Marcus continued explaining tier mechanics after Priya had already accepted the usage increase framing, causing her to ask for the actual number.
  • Only partially captured the Experiment expansion benchmark. The coach noticed the DuoTest/Amplitude context-switching issue, but treated it mainly as an underdeveloped opportunity rather than identifying it as a positive expansion thread opened by the call.
  • Slightly under-called the outcome relative to the hidden benchmark. The benchmark views renewal as effectively confirmed and expansion pipeline clearly opened; the coach described the call as strong and likely advanced, but less decisively excellent.
  • The commercial critique focused on missing annual-rate readiness. That is transcript-grounded, but it displaced the more benchmark-relevant coaching point about concise commercial communication after buyer acceptance.
1379gpt-5.5 lowStrong coach output with a few important misses
Overall81
Needle recall70
Evidence grounding84
False-positive control78
Prioritization76
Actionability89
Sales instinct84
Technical accuracy90
How this model did

The coach correctly recognized the call as a high-quality renewal/expansion QBR and captured the biggest strengths: Duolingo-specific value recap, technical diagnosis of the Max onboarding gap, Session Replay framed as a 90-day measurable pilot, and buyer-authored success criteria. The main gap is that it missed the hidden benchmark’s subtle commercial flaw: Marcus kept explaining MTU tier mechanics after Priya had already signaled acceptance. It also only partially captured the Experiment expansion thread, framing it mostly as underdeveloped rather than as a buyer-buy-in expansion opportunity. A few extra critiques were plausible but somewhat over-inferred, especially the claim that not stating the annual rate live created friction and the supposed facilitation issue caused by a likely transcript speaker-label error.

Strongest findings
  • Correctly recognized the call as an excellent, data-led QBR rather than a generic renewal deck.
  • Accurately praised the use of Duolingo’s own D7 retention, DAU/MAU, and Max funnel data to anchor value.
  • Strongly captured the Session Replay expansion motion as a named 90-day pilot with a 10% trial-to-paid success metric.
  • Correctly identified Priya Nair’s schema-level diagnosis and variant-filtering answer as technically credible and buyer-relevant.
  • Correctly called out the buyer-authored success criteria and 30-day check-in as a strong mutual-success-plan close.
Biggest misses
  • Missed the subtle but real MTU-pricing flaw: Marcus kept explaining tier mechanics after Priya had already accepted the usage-growth premise.
  • Only partially captured the experimentation expansion needle; it noticed the DuoTest/Amplitude pain but framed it mainly as underdeveloped rather than as a benchmarked expansion win.
  • Prioritized commercial readiness around stating the annual number live, which is plausible coaching but not the key hidden commercial issue.
  • Over-interpreted a likely transcript speaker-label glitch as a facilitation-control problem.
1479opus 4.7 lowMostly strong coaching output, but slightly over-critical versus the benchmark excellent-call profile and it contradicts the benchmark’s Experiment-strength framing.
Overall82
Needle recall78
Evidence grounding88
False-positive control76
Prioritization72
Actionability86
Sales instinct80
Technical accuracy87
How this model did

The coach accurately captured the biggest transcript-grounded strengths: Marcus opened with Duolingo-specific metrics, Priya Nair added high-credibility schema analysis, Session Replay was framed as a 90-day Max onboarding pilot with a 10% trial-to-paid target, and the close used buyer-authored success criteria with a 30-day check-in. The coach also caught the commercial-delivery issue around meandering MTU/pricing explanation. The main concern is prioritization: the coach makes the Experiment/DuoTest thread the P0 miss and says the seller “completely missed” it, whereas the benchmark treats Experiment expansion as part of the positive outcome. That critique is not baseless—the transcript lacks the quantitative A/B-testing discovery called for in the playbook—but the coach overstates it because Marcus did reflect the experimentation-stack criterion and scheduled technical scoping. A few additional risks, especially the “transcript artifact” delivery glitch and the annual-rate critique, are speculative or over-weighted.

Strongest findings
  • Correctly identified the data-led QBR opening using Duolingo’s own Amplitude metrics, including D7 retention, DAU/MAU, and the Max funnel.
  • Correctly praised Priya Nair’s schema-level analysis as a credibility builder that made the Session Replay use case feel specific and earned.
  • Correctly captured the 90-day Session Replay pilot with a single 10% Max trial-to-paid success metric.
  • Correctly recognized the buyer-authored close and mutual success criteria as a major strength.
  • Correctly flagged the MTU/pricing segment as overly explanatory before delivering the headline number.
Biggest misses
  • The coach is misaligned with the benchmark’s excellent-call profile by making the Experiment thread a high-severity P0 miss rather than treating the call outcome as strongly positive with one minor commercial flaw.
  • The coach overstates the Experiment issue as “completely missed” even though Marcus reflected the DuoTest/Amplitude consolidation goal and assigned technical scoping.
  • The coach expands the pricing critique beyond the hidden flaw by adding speculative concern about not sharing the new annual rate live.
  • The coach treats a likely transcript artifact as a real delivery issue.
  • The overall assessment of “above-average” understates the benchmark’s view that this was an excellent renewal QBR with strong buyer buy-in.
1579gpt-5.4 mediumMostly aligned, with one important subtle miss and some over-weighted commercial critique.
Overall80
Needle recall74
Evidence grounding86
False-positive control76
Prioritization73
Actionability88
Sales instinct83
Technical accuracy90
How this model did

The coach correctly captured the core strengths of the call: Duolingo-specific value recap, strong technical diagnosis, a tightly scoped Session Replay pilot, and buyer-authored success criteria. The output is generally well grounded in transcript evidence and offers actionable coaching. However, it missed the benchmark’s subtle MTU-pricing flaw: Marcus over-explained pricing mechanics after the buyer had already signaled acceptance. Instead, the coach framed the commercial issue as a high-severity lack of pricing transparency, which is only partially supported and overstates the friction. The coach also under-recognized the positive experimentation expansion signal, treating it mostly as a gap rather than as buyer-originated expansion momentum.

Strongest findings
  • Correctly praised the seller for opening with Duolingo’s own metrics rather than generic benchmarks.
  • Accurately identified the Max onboarding black box and the discovery that validated it as a real buyer pain.
  • Recognized Priya Nair’s technical credibility in diagnosing the missing events and tying Session Replay to the instrumentation gap.
  • Correctly highlighted the 90-day Session Replay pilot with a 10% trial-to-paid success metric as excellent value-based expansion framing.
  • Captured the buyer-authored success criteria and concrete follow-up plan near the close.
Biggest misses
  • Missed the subtle MTU-pricing flaw: Marcus continued explaining tier mechanics after Priya had already accepted the usage-based increase.
  • Replaced the benchmark commercial coaching point with a more severe pricing-transparency critique that overstates the actual buyer friction.
  • Did not fully credit the experimentation expansion signal as buyer-originated momentum, even though it did correctly recommend deeper discovery there.
  • Slightly over-prioritized generic executive ROI and stakeholder-mapping improvements relative to the benchmark’s more specific observations.
1679gpt-5.5 xhighStrong coaching output with good grounding, but it missed the benchmark’s subtle commercial flaw and only partially handled the Experiment expansion needle.
Overall82
Needle recall68
Evidence grounding90
False-positive control78
Prioritization76
Actionability90
Sales instinct83
Technical accuracy88
How this model did

The coach correctly recognized the call as a strong renewal QBR, praised the Duolingo-specific data recap, the technical schema diagnosis, the Session Replay pilot framing, and the buyer-authored success question. The output is generally well evidenced and actionable. However, it did not identify the hidden benchmark’s specific minor flaw: Marcus over-explained MTU pricing mechanics after Priya had already accepted the usage-based increase. It also treated the Experiment/DuoTest opportunity mainly as an under-qualified missed opportunity rather than recognizing the benchmarked strength around organically surfacing the experimentation expansion path. Several extra coaching risks are reasonable but somewhat speculative or over-prioritized versus the ground truth.

Strongest findings
  • Correctly identified the Duolingo-specific value recap as a standout strength and quoted the key D7 retention, DAU/MAU, and Max funnel evidence.
  • Correctly praised the SC handoff and schema-level technical diagnosis, including the missing intermediate events between paywall confirmation and first AI lesson load.
  • Correctly recognized that the Session Replay expansion was framed as a measurable pilot rather than a generic feature upsell.
  • Correctly highlighted Marcus’s open-ended renewal-success question as a strong move toward buyer-authored success criteria.
  • Provided actionable follow-up coaching around ROI translation, pilot planning, and deeper Experiment/DuoTest qualification.
Biggest misses
  • Missed the subtle MTU pricing flaw: Marcus kept explaining tier mechanics after Priya had already signaled acceptance, prompting her to redirect to the actual number.
  • Only partially captured the Experiment expansion needle; it noticed the DuoTest/Amplitude signal but did not recognize the benchmarked seller-led discovery pattern around experimentation scaling.
  • Over-weighted additional commercial and implementation risks that are reasonable but not central to the hidden ground truth.
  • Slightly under-credited the strength of the mutual success plan by calling next steps not fully mutualized despite buyer-authored criteria and a scheduled 30-day check-in.
1778opus 4.7 mediumMostly accurate, but materially misprioritized one expansion thread
Overall81
Needle recall80
Evidence grounding88
False-positive control74
Prioritization66
Actionability87
Sales instinct78
Technical accuracy86
How this model did

The coach correctly captured the strongest parts of the call: the Duolingo-specific data recap, the technically credible Session Replay diagnosis, the 90-day pilot with a 10% Max trial-to-paid success metric, the buyer-authored mutual success plan, and the minor commercial over-explanation. The main problem is that the coach elevated “Experiment/DuoTest consolidation was left on the table” as the biggest miss, whereas the benchmark treats the experimentation thread as part of the positive expansion motion and buyer-authored success plan. That creates an under-rating of an otherwise excellent call and overstates a weakness that is only partially supported by the transcript.

Strongest findings
  • Correctly recognized the data-led QBR opening as a gold-standard use of Duolingo’s own Amplitude metrics.
  • Accurately praised Priya Nair’s schema-level technical diagnosis as a trust-building moment.
  • Correctly identified the Session Replay pilot as buyer-specific, time-boxed, and tied to a concrete 10% Max trial-to-paid success metric.
  • Correctly captured the buyer-authored mutual success plan and 30-day check-in as a strong close.
  • Correctly noticed the MTU commercial segment was over-explained and that Priya’s “what’s the actual number?” was a signal to be more concise.
Biggest misses
  • The coach misread or over-penalized the Experiment/DuoTest thread, treating it as a major unaddressed miss rather than a buyer-authored expansion criterion captured for follow-up.
  • The coach’s prioritization is off: the hidden benchmark sees only a minor flaw in an excellent call, while the coach makes a high-severity Experiment miss the central coaching theme.
  • The coach did not fully articulate the call outcome as strongly positive: renewal effectively confirmed, commercial overage resolved without friction, and expansion buy-in created for Session Replay and Experiment-related scoping.
  • The commercial critique about not saying the annual renewal number aloud is plausible but not supported by buyer resistance in the transcript or by the hidden benchmark.
1878gpt-5.4 lowGood but imperfect evaluation: the coach captured the main positive arc and several key strengths, but missed the specific subtle MTU over-explanation flaw and under-credited the Experiment expansion motion.
Overall79
Needle recall74
Evidence grounding86
False-positive control76
Prioritization72
Actionability88
Sales instinct81
Technical accuracy84
How this model did

The coach accurately recognized this as a strong renewal/QBR with excellent preparation, buyer-specific metrics, credible technical diagnosis, a well-scoped Session Replay pilot, and buyer-authored next steps. Its evidence is mostly transcript-grounded and its coaching is generally actionable. However, it diverged from the hidden benchmark in two important ways: it treated the experimentation expansion story primarily as underdeveloped rather than identifying the intended strength around surfacing an Experiment-related expansion path, and it missed the precise commercial flaw—Marcus continuing to explain MTU mechanics after Priya had already accepted the framing. Instead, it over-indexed on pricing readiness/hesitation, which is only partially supported and too severe relative to the benchmark.

Strongest findings
  • Correctly identified the data-led QBR opening as a major strength, with precise evidence from Duolingo’s own metrics.
  • Accurately praised the focused discovery around the Max onboarding black box and the technical handoff to the SC.
  • Clearly captured the 90-day Session Replay pilot with a 10% Max trial-to-paid success metric and buyer buy-in.
  • Correctly recognized the buyer-authored close: Priya defined success, Marcus reflected it back, and the team left with concrete next steps.
Biggest misses
  • Missed the exact subtle MTU flaw: Marcus kept explaining tier mechanics after Priya had already accepted the commercial framing.
  • Under-credited the Experiment expansion motion by treating it mostly as an underdeveloped opportunity rather than a benchmarked expansion strength.
  • Over-prioritized commercial command and pricing readiness, making the commercial segment sound riskier than the transcript and hidden benchmark support.
  • Added several reasonable but non-benchmarked coaching opportunities—finance process, revenue quantification, separate Experiment workshop—that were actionable but less central than the hidden needles.
1977opus 4.8 lowMostly strong, but with a notable miss on the subtle commercial flaw and a few overreaching critiques.
Overall78
Needle recall74
Evidence grounding80
False-positive control68
Prioritization72
Actionability88
Sales instinct83
Technical accuracy86
How this model did

The coach accurately recognized the call as a high-quality renewal QBR and captured the biggest strengths: data-led preparation from Duolingo’s own Amplitude instance, strong Max funnel discovery, precise Session Replay technical/value framing, a 90-day pilot with a measurable success metric, and a buyer-authored close. The main weakness is that it missed the benchmark’s subtle MTU-pricing flaw: Marcus continued explaining tier mechanics after Priya had already accepted the usage increase. Instead, the coach introduced a different commercial critique around not stating the annual renewal rate live, which is transcript-based but over-prioritized relative to the actual issue. The coach also added some unsupported or speculative claims, especially around Priya’s “known style” and revenue-impact expectations.

Strongest findings
  • Correctly recognized the data-led QBR opening as best-in-class preparation using Duolingo’s own metrics.
  • Accurately praised the Max funnel discovery sequence where Jordan articulated the black-box problem before Session Replay was proposed.
  • Strongly identified the SC’s technical credibility around missing intermediate events and why Session Replay fits that instrumentation gap.
  • Correctly highlighted the 90-day pilot with a 10% Max trial-to-paid success metric as excellent value-based expansion scoping.
  • Accurately praised the buyer-authored close and the conversion of Priya’s success criteria into next steps.
Biggest misses
  • Missed the benchmark’s subtle commercial flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance.
  • Substituted a different commercial critique — unstated annual rate — and elevated it more than the transcript supports.
  • Added an unsupported claim about Priya’s “known style” and an unasked revenue-impact question.
  • Treated an apparent transcript attribution artifact as a real facilitation issue.
  • Under-credited the fact that the experimentation consolidation workstream was at least opened and captured in the buyer-authored success plan, even if it was not deeply discovered.
2075glm 5.2Mostly accurate, but over-penalizes an excellent call and misreads/overstates the Experiment and commercial-coaching areas.
Overall80
Needle recall73
Evidence grounding84
False-positive control68
Prioritization65
Actionability88
Sales instinct78
Technical accuracy86
How this model did

The coach correctly identified the strongest parts of the call: the buyer-specific data recap, the Session Replay pilot tied to Max trial-to-paid conversion, and the buyer-authored mutual success plan. It also noticed that the commercial section could have been tighter. However, it materially diverged from the benchmark by treating Experiment as the call’s biggest missed opportunity rather than recognizing the expansion pipeline that was opened through the buyer’s stated consolidation goal and Marcus’s recap/next step. It also missed the precise MTU flaw: the issue was subtle over-explaining after acceptance, not a high-severity commercial fumble or deflection.

Strongest findings
  • Accurately praised the data-led QBR opening using Duolingo’s own metrics and named product surfaces.
  • Accurately identified the Max onboarding black box and Session Replay pilot as the strongest expansion motion.
  • Correctly recognized the buyer-authored close and 30-day check-in as strong mutual success plan behavior.
  • Grounded most major claims in specific transcript quotes rather than generic sales advice.
Biggest misses
  • Misprioritized Experiment as a high-severity miss rather than recognizing the buyer-authored Experiment/consolidation expansion signal emphasized by the benchmark.
  • Missed the exact nature of the MTU flaw: over-explaining after acceptance, not primarily lack of dollar-delta clarity.
  • Overstated commercial risk despite the buyer showing little friction and agreeing to loop in finance.
  • Added some non-benchmark coaching points that are plausible but not strongly supported, such as downside-path language and a script-slip critique.
2175opus 4.8 highGood but not fully aligned with the benchmark
Overall78
Needle recall70
Evidence grounding86
False-positive control64
Prioritization72
Actionability88
Sales instinct80
Technical accuracy84
How this model did

The coach produced a strong, mostly transcript-grounded evaluation of an excellent QBR. It correctly identified the data-led opening, the precise Session Replay pilot framing, the diagnostic discovery around the Max funnel, and the buyer-authored close. However, it missed the benchmark’s subtle MTU pricing flaw: Marcus over-explained the MTU mechanics after Priya had already signaled acceptance. Instead, the coach emphasized a different commercial concern — lack of live annual pricing — and somewhat over-penalized the Experiment/DuoTest thread by treating it mainly as an unclaimed opportunity rather than recognizing that the buyer had clearly made experimentation consolidation part of the mutual success plan.

Strongest findings
  • Accurately identified the QBR’s buyer-specific data-led opening as a major strength.
  • Correctly praised the technical diagnosis of the Max onboarding instrumentation gap and the Session Replay use case.
  • Correctly recognized the 90-day, single-metric Session Replay pilot as a high-quality expansion motion.
  • Correctly highlighted the buyer-authored close and the “what would need to be true” question as excellent closing discipline.
  • Used mostly accurate transcript quotes and provided actionable coaching drills rather than generic advice.
Biggest misses
  • Missed the subtle benchmark flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance.
  • Substituted a different commercial critique — not stating the new annual rate live — and over-weighted it despite no buyer friction in the transcript.
  • Under-credited the Experiment/DuoTest expansion thread as buyer-authored pipeline, even though Priya and Jordan explicitly made it part of the success criteria.
  • Did not clearly state that the renewal was effectively de-risked/confirmed during the call, though it did say the buyer was bought in.
2275opus 4.7 maxStrong but imperfect coaching output. It captured most of the major positive moments, especially the data-led QBR opening, technical credibility, Session Replay pilot framing, and buyer-authored close. However, it missed the benchmark’s subtle commercial flaw around over-explaining MTU mechanics, and it over-prioritized a different commercial critique that the transcript does not support as strongly.
Overall78
Needle recall72
Evidence grounding86
False-positive control66
Prioritization64
Actionability90
Sales instinct79
Technical accuracy88
How this model did

The coach produced a high-quality, well-evidenced assessment of an excellent QBR. It correctly recognized that Marcus anchored the meeting in Duolingo’s own metrics, that Priya Nair’s schema-level specificity earned technical trust, that Session Replay was framed as a 90-day Max funnel pilot with a 10% trial-to-paid success metric, and that the buyer defined the mutual success criteria at the end. The main evaluation issue is prioritization: the coach made the deferred annual renewal rate the top risk, even though Priya did not explicitly ask for the exact annual number and the commercial segment resolved without visible friction. More importantly, the coach missed the hidden benchmark’s subtle flaw: Marcus kept explaining MTU tier mechanics after Priya had already signaled acceptance, prompting her to redirect with “Right, so what’s the actual number?” The coach also treated the experimentation thread mainly as a high-severity miss; that is partially grounded because the seller did not quantify experimentation volume, but it under-credits the buyer-authored success criterion around cleaning up DuoTest/Amplitude fragmentation.

Strongest findings
  • Correctly identified the data-led QBR opening as the model behavior for a renewal with a sophisticated product analytics buyer.
  • Accurately praised Priya Nair’s technical credibility, especially the schema inspection and exact missing events in the Max onboarding flow.
  • Clearly captured the value of framing Session Replay as a scoped 90-day pilot with a single measurable success metric.
  • Correctly recognized the buyer-authored close as a strong signal of alignment and renewal confidence.
  • Provided concrete and actionable coaching drills, especially around commercial number delivery and quantifying experimentation scope.
Biggest misses
  • Missed the subtle benchmark flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance.
  • Over-prioritized the deferred annual renewal number as a high-severity risk despite the buyer accepting the proposal follow-up without friction.
  • Under-credited the experimentation consolidation thread as an opened buyer-authored success criterion, even though it was fair to note that it needed more discovery.
  • Praised the tier-economics explanation as commercial transparency without noting that Priya’s redirect suggested mild impatience.
  • Added a low-value transcript-hygiene critique about name confusion that was not material to sales coaching.
2372opus 4.8 mediumGood but meaningfully off-priority
Overall76
Needle recall68
Evidence grounding82
False-positive control67
Prioritization63
Actionability86
Sales instinct78
Technical accuracy80
How this model did

The coach correctly recognized the strongest parts of the call: Marcus’s buyer-specific data recap, the Max onboarding discovery, the Session Replay pilot with a 90-day/10% conversion goal, and the buyer-authored success close. However, it missed the hidden benchmark’s subtle actual flaw: Marcus over-explained MTU pricing mechanics after Priya had already signaled acceptance. Instead, it over-weighted other critiques, especially commercial vagueness and under-scoped Experiment/DuoTest consolidation. Those critiques have some transcript basis, but they are not the central coaching points in the benchmark and are overstated relative to the buyer’s positive engagement and accepted next steps.

Strongest findings
  • Correctly highlighted the buyer-specific data-led opening with D7 retention, DAU/MAU, and Duolingo Max funnel metrics.
  • Correctly praised the layered discovery around the Max onboarding black box and Jordan’s admission that they could see drop-off but not why.
  • Correctly identified Priya Nair’s schema review and Session Replay positioning as technically credible and buyer-specific.
  • Correctly captured the 90-day Session Replay pilot with a 10% Max trial-to-paid conversion success metric.
  • Correctly recognized the buyer-authored close where Priya Sharma defined the two success criteria in her own words.
Biggest misses
  • Missed the actual subtle benchmark flaw: Marcus continued explaining MTU pricing mechanics after Priya had already signaled acceptance and then redirected him to the number.
  • Over-prioritized commercial vagueness even though the buyer accepted the true-up framing and showed little friction.
  • Over-rotated on Experiment/DuoTest as a major missed opportunity rather than recognizing the broader benchmark view that expansion pipeline for Experiment and Session Replay was opened.
  • Did not cleanly identify the hidden Experimentation discovery strength; instead it mostly treated experimentation as an absence or failure to quantify.
  • Understated how positive the overall call outcome was by assigning relatively low scores to commercial handling and expansion capture.
2472sonnet 4.6Mostly accurate on the major positive themes, but it missed or contradicted two important benchmark needles.
Overall74
Needle recall62
Evidence grounding83
False-positive control67
Prioritization70
Actionability90
Sales instinct76
Technical accuracy86
How this model did

The coach correctly recognized the call as a strong QBR, with excellent data-led preparation, technically specific Session Replay positioning, crisp pilot framing, and a buyer-authored mutual success plan. However, it materially diverged from the benchmark by treating the Experiment expansion thread as the call’s biggest failure rather than a buyer-surfaced expansion opportunity that was incorporated into the success plan, and it completely missed the subtle MTU-pricing flaw. Worse, it explicitly claimed Marcus did not over-explain, which contradicts the benchmark’s main coaching opportunity. The output is still useful and well-evidenced overall, but its prioritization is skewed by over-weighting the experimentation critique and overlooking the intended subtle commercial coaching point.

Strongest findings
  • Accurately identified the data-led opening as best-in-class QBR preparation, with specific Duolingo metrics and charts from the customer’s own Amplitude instance.
  • Correctly praised Priya Nair’s event-schema review as a strong technical diagnosis that earned credibility with Jordan.
  • Correctly recognized the Session Replay pilot as value-based expansion selling because it had a named funnel, 90-day scope, and 10% conversion target.
  • Correctly highlighted the buyer-authored mutual success plan close and Marcus’s open-ended future-state question.
  • Provided highly actionable follow-up coaching, especially around structuring the next scoping session and documenting the Session Replay baseline.
Biggest misses
  • Completely missed and contradicted the subtle MTU over-explanation flaw, which was the benchmark’s main negative coaching point.
  • Over-indexed on the Experiment thread as a failure, whereas the benchmark expected more credit for the buyer-surfaced experimentation expansion opportunity and its inclusion in the success plan.
  • Introduced some evidence inaccuracies, especially claiming Jordan named DuoTest twice.
  • Did not sufficiently distinguish between “Experiment was less scoped than Session Replay” and “Experiment was not meaningfully advanced at all.”
2569sonnet 5Partially accurate, but over-penalizes an excellent call and misses the benchmark’s subtle commercial flaw.
Overall72
Needle recall66
Evidence grounding80
False-positive control58
Prioritization62
Actionability78
Sales instinct70
Technical accuracy84
How this model did

The coach correctly recognized several core strengths: Marcus opened with Duolingo-specific metrics, diagnosed the Max onboarding visibility gap before pitching, used the SC effectively, framed Session Replay as a 90-day pilot with a 10% trial-to-paid success metric, and used a buyer-authored close. However, the coach materially diverged from the benchmark by treating the experimentation expansion thread as a major miss rather than an opened expansion path, and by inventing/overstating commercial-readiness problems while missing the actual subtle flaw: Marcus kept explaining MTU pricing mechanics after Priya had already signaled acceptance. The output is well-evidenced and actionable in many places, but its prioritization is too negative for a strongly positive renewal/expansion QBR.

Strongest findings
  • Correctly identified the data-led value recap using Duolingo’s own metrics as a major strength.
  • Correctly praised the diagnostic sequence around the Max onboarding visibility gap before pitching Session Replay.
  • Correctly recognized Priya Nair’s schema-level technical preparation and answer on filtering replays by experiment variant.
  • Correctly captured the 90-day Session Replay pilot with a 10% Max trial-to-paid conversion success metric.
  • Correctly recognized the open-ended closing question that caused the buyer to define success in her own words.
Biggest misses
  • Missed the actual subtle MTU flaw: Marcus kept explaining pricing mechanics after Priya had already signaled acceptance and was ready for the number.
  • Over-penalized the experimentation thread as a major missed opportunity, whereas the benchmark treats it as an expansion path opened with buyer buy-in.
  • Downgraded the call to a B+ despite the hidden benchmark profile being excellent and the renewal/expansion outcome being strongly positive.
  • Invented or overstated commercial-readiness concerns from a brief live-number moment that did not create buyer friction.
  • Underweighted the strength of the final mutual success plan and 30-day check-in by focusing on possible workstream separation.
2668gemini 3.1 pro previewWorstGood coaching output with strong recognition of the major value-led QBR strengths, but materially weakened by an overreaching commercial critique and by missing the subtle MTU over-explanation flaw.
Overall72
Needle recall66
Evidence grounding78
False-positive control55
Prioritization58
Actionability72
Sales instinct73
Technical accuracy80
How this model did

The coach correctly praised the strongest parts of the call: Marcus opened with Duolingo-specific metrics, Priya Nair diagnosed the Max onboarding instrumentation gap well, Session Replay was framed as a 90-day pilot with a 10% conversion target, and Marcus used a strong buyer-authored close. However, the coach substituted a high-severity “dodging the price question” critique for the actual subtle pricing flaw. The transcript supports that Marcus slightly over-explained MTU mechanics after buyer acceptance; it does not support the claim that he hid an annual price in a trust-breaking way. The coach also somewhat under-credited the overall commercial outcome and Experiment expansion momentum, making the call sound riskier than the hidden benchmark indicates.

Strongest findings
  • Accurately identified the best-in-class data-led QBR opening using Duolingo’s own metrics rather than generic benchmarks.
  • Correctly praised Priya Nair’s technical preparation around the Max onboarding event-schema gap and Session Replay fit.
  • Correctly recognized the 90-day Session Replay pilot with a 10% trial-to-paid conversion target as a strong value-based expansion motion.
  • Correctly highlighted Marcus’s open-ended closing question that caused the buyer to define success criteria in her own words.
Biggest misses
  • Missed the actual subtle pricing flaw: Marcus over-explained MTU tier mechanics after the buyer had already signaled acceptance.
  • Invented or overstated a more severe pricing flaw — “dodging the price question” — that is not supported by buyer reaction or transcript facts.
  • Under-prioritized the overall positive renewal/expansion outcome by making commercial confidence the main coaching plan despite the buyer saying they were good on commercial.
  • Only partially handled the Experiment expansion thread: the coach correctly noted missing early A/B testing discovery, but under-credited the buyer-authored Experiment success criterion at the end.