Which models know sales?
Eighteen model configurations coach the same 25 synthetic sales calls. Each call has hidden ground truth. A judge scores every coaching note from 0–100 on whether it found the real strengths, flaws, and next moves.
- Calls
- 25
- Models
- 18
- Evaluations
- 450
- Mean
- 89.8
The 25 calls
Open a call to read its answer key and how every model did on it.
- CollibraBerkshire HathawayBerkshire Hathaway Data governance discovery across decentralized business units with CollibraEasiestDiscoveryflawed95.4
- StripePavePave Pricing and packaging objection call with StripeCompetitive displacementflawed94.3
- VercelMercuryMercury First discovery for frontend platform consolidation with VercelDiscoveryflawed94.1
- AtlassianDelta Air LinesDelta Air Lines Enterprise discovery for service management modernization with AtlassianDiscoveryflawed94.0
- MongoDBWayfairWayfair Integration deep dive for catalog modernization with MongoDBProduct demoexcellent93.7
- TwilioThe Home DepotThe Home Depot Renewal save call after usage and support concerns with TwilioRenewal saveflawed93.7
- Palo Alto NetworksAppleApple Technical security review for zero trust architecture with Palo Alto NetworksProduct demoexcellent93.2
- AmplitudeDuolingoDuolingo Renewal QBR and expansion planning with AmplitudeQBRexcellent92.4
- OpenAICVS HealthCVS Health AI contact-center transformation discovery with OpenAIDiscoveryexcellent92.0
- GitHubRipplingRippling Product-led expansion discovery for developer workflow with GitHubDiscoveryexcellent91.8
- WorkdayMcKessonMcKesson HR transformation qualification and stakeholder mapping with WorkdayDiscoveryflawed91.1
- AnthropicExxonMobilExxonMobil AI governance and safety review for energy operations with AnthropicProduct demomixed90.9
- CrowdStrikeTargetTarget Security architecture review for endpoint consolidation with CrowdStrikeProduct demoexcellent90.8
- DatadogLinearLinear Technical demo for observability and incident response with DatadogProduct demoexcellent90.4
- ElasticJPMorgan ChaseJPMorgan Chase Technical workshop for search and observability consolidation with ElasticProduct demoexcellent90.4
- NVIDIAWalmartWalmart Executive discovery for AI infrastructure and store operations with NVIDIADiscoveryexcellent89.3
- HashiCorpAmazonAmazon Cloud operating model discussion for internal platform teams with HashiCorpDiscoveryflawed89.1
- ServiceNowFord Motor CompanyFord Motor Company Procurement negotiation for workflow automation with ServiceNowCompetitive displacementmixed88.6
- SnowflakeToastToast Data platform proof-of-concept kickoff with SnowflakeProduct demoflawed87.0
- CloudflareCanvaCanva Competitive displacement discovery for edge security with CloudflareCompetitive displacementflawed85.8
- FigmaThe Walt Disney CompanyThe Walt Disney Company Design collaboration demo with brand and asset workflow discussion with FigmaProduct demomixed85.8
- OktaSweetgreenSweetgreen Executive alignment for identity modernization with OktaQBRmixed85.2
- SalesforceUnitedHealth GroupUnitedHealth Group Healthcare CRM expansion objection handling with SalesforceRenewal savemixed84.9
- SnykRunwayRunway Security review before developer-tool rollout with SnykProduct demomixed82.5
- MicrosoftCostco WholesaleCostco Wholesale Proof-of-concept readout for analytics and productivity workflow with MicrosoftHardestProduct demomixed79.7
Mercury First discovery for frontend platform consolidation with Vercel
This should read as a superficially cordial first discovery call where the Vercel seller keeps the conversation moving but does not earn much strategic depth. The seller is polite and manages to secure a normal follow-up, yet the core coaching truth is that they are underprepared for a fintech frontend-platform consolidation conversation. They rely too heavily on generic BANT qualification, miss or under-explore Mercury’s reliability and compliance cues, and position Vercel mostly as a fast developer workflow tool rather than a platform that could reduce release risk and support governed deployment practices.
- Profile
- Flawed
- Flaws / Strengths
- 4 / 1
- Duration
- 22m · 18 turns
What this call should surface
Over-indexes on basic BANT instead of diagnosing the consolidation problem
Discovery · moderate
Fails to dig into deployment reliability and rollback risk
Technical Knowledge · subtle
Talks past compliance and vendor-risk pressure
Objection Handling · moderate
Underuses Mercury-specific context and fintech implications
Research · subtle
Maintains a professional tone and secures a reasonable follow-up
Next Steps · obvious
Transcript
The exact speaker-labeled transcript the coach models saw.
- NK
Nora Kim
Seller
Hi everyone, thanks for making the time. I’m Nora with Vercel — I work with a handful of fintech and growth-stage engineering teams. I know this is a first conversation, so I was thinking we could keep it pretty lightweight: understand what Mercury’s exploring around frontend platform consolidation, how you’re deploying today, what the timeline or decision process looks like, and then see if there’s a useful next step with Marcus on the technical side. Marcus is here from our solutions team as well. Does that sound okay?
- MV
Marcus Vale
Seller
Hey, everyone — Marcus here. I’m on the solutions side at Vercel, so I can help with workflow or architecture questions as they come up, but Nora can lead us through the discovery.
- EP
Elena Patel
Buyer
Yeah, sounds good. I’m Elena — I lead platform engineering at Mercury. We’re mostly here because our frontend deployment story has gotten a little fragmented across teams, so I’d like to understand where Vercel might fit and what consolidation would actually look like.
- DW
Darius Wong
Buyer
Hi, I’m Darius. I’m on the security and compliance side, mostly here to listen for vendor-risk implications if this moves forward.
- NK
Nora Kim
Seller
Perfect, thanks. Elena, what are the main deployment paths today?
- EP
Elena Patel
Buyer
Sure. There are basically three buckets. Product teams have a couple of customer-facing React and Next.js surfaces that deploy through our internal CI into AWS and CloudFront. Marketing has a separate path that’s a little more agency-friendly. And then we have some internal tools that are, honestly, kind of snowflake-y. The pain is less that any one path is completely broken and more that ownership, previews, and rollback behavior are inconsistent depending on which team you’re talking to.
- NK
Nora Kim
Seller
Got it, that makes sense. Just to size it a bit, how many frontend teams or apps are we talking about, and is there already an active consolidation project with a timeline attached?
- EP
Elena Patel
Buyer
Roughly six or seven teams touch the main web surfaces in some way, but the serious scope is probably three product areas plus marketing. There isn’t a formal funded program yet — it’s more that platform is being asked to come back with options this quarter. And, candidly, after a recent frontend release caused some customer-visible weirdness, there’s more appetite to standardize than there was six months ago.
- NK
Nora Kim
Seller
Yeah, that makes sense — sorry you all had to deal with that. From a Vercel perspective, previews can help teams catch more before merge. Just to understand the project shape, who would own budget or sign-off if platform brings back a recommendation this quarter?
- EP
Elena Patel
Buyer
Probably me to start, with our VP of Eng and finance pulled in if there’s a real proposal. Budget isn’t allocated yet, though. We’d need to justify it against the platform work we’re already doing internally — and part of that justification would be confidence that we’re not just moving the same release-risk problem to a new place.
- NK
Nora Kim
Seller
Totally. And standardizing the workflow is usually a big part of that. Are the main customer-facing apps already on Next.js, or is it more mixed React today?
- EP
Elena Patel
Buyer
It’s mixed. The newer product surfaces are mostly Next.js, older pieces are React with some custom routing, and marketing is its own thing. Framework-wise, we can deal with some variation. The harder part is that a deploy to one surface can have a very different review and rollback path than another, so engineers don’t always know what safety net they’re operating with.
- NK
Nora Kim
Seller
Right, totally. The value we usually see is getting everyone onto the same Git-based flow with preview URLs, so it’s less tribal knowledge per app. On process, besides you and the VP of Eng, does security or procurement need to weigh in before you’d pilot something like Vercel?
- DW
Darius Wong
Buyer
Yeah, I can jump in there. Security would definitely be involved. We’ve gotten a lot stricter on vendor review for anything in the customer-facing delivery path — access model, evidence, audit trail, that kind of thing. It’s not a blocker by default, but it can slow adoption if we don’t have clarity early.
- NK
Nora Kim
Seller
Yeah, absolutely, and we work with a lot of teams where that review is part of the process. At a high level, the reason engineering teams like Vercel is it gives them a really clean Git-based workflow, preview URLs on every PR, and a more consistent path to production instead of each app having its own bespoke setup. I can send over our standard security and platform resources after this, and then maybe the next conversation is more of a workflow mapping with Elena’s platform lead and Marcus on our side.
- DW
Darius Wong
Buyer
That’s fine. If you send the standard packet, include whatever you normally share for vendor review too. We can route it internally before the workflow session.
- NK
Nora Kim
Seller
Perfect, I’ll send a short recap and the standard Vercel resources, including the vendor review packet, later today. Elena, I’ll also include a few times for a workflow-mapping session with whoever owns the current deployment paths on your side, and Marcus can join from ours. Thanks again, both of you — really appreciate the context.
- EP
Elena Patel
Buyer
Thanks, Nora. Email is best — copy Darius and me, and I’ll pull in the right platform lead for the next one.
How each model scored this call
Click a row to read the model's coaching note and the judge's read on it.
196gpt-5.5 noneBestExcellent coaching assessment; strongly aligned to the hidden ground truth.
The coach accurately characterized the call as professional and superficially successful but strategically underdeveloped. It caught the main flaws: Nora relied on basic qualification, failed to dig into Mercury’s reliability/release-risk cues, talked past Darius’s compliance/vendor-review concerns, and positioned Vercel too generically for a fintech buyer. It also appropriately credited the polite opening and concrete follow-up without over-weighting them. The output is well grounded in transcript evidence and adds actionable coaching without materially inventing facts.
- Correctly prioritized the customer-visible release instability as the most important missed discovery moment, not merely a technical aside.
- Accurately identified that Darius’s vendor-review cue required real security/compliance discovery rather than generic reassurance and a standard packet.
- Balanced the assessment well: professional opening and next step were credited, but the call was still judged weakly qualified and strategically shallow.
- Provided highly actionable coaching questions around incident impact, rollback mechanics, security evidence, pilot success criteria, and mutual evaluation planning.
- Used transcript evidence precisely, including the key buyer quotes about rollback inconsistency, release-risk transfer, and access model/evidence/audit trail.
- No major benchmark misses. The only minor gap is that the account-context critique could have more explicitly emphasized Mercury-specific financial workflows in the main assessment, though it was covered in follow-up questions and fintech framing.
- The coach added some adjacent critiques, such as Marcus being underutilized and decision process being partially mapped. These are not hidden-ground-truth needles, but they are transcript-supported and do not distort the evaluation.
296opus 4.7 maxExcellent ground-truth alignment
The coach output correctly reads the call as polite and superficially productive but strategically underdeveloped. It identifies the central hidden flaws: Nora over-relied on BANT-style qualification, failed to dig into Mercury’s customer-visible deployment incident and rollback/release-risk concerns, talked past Darius’s vendor-risk/compliance cue, and positioned Vercel too generically instead of tying value to fintech reliability and governance needs. It also appropriately credits the professional opening and concrete follow-up without overvaluing the booked next step. Evidence is strongly transcript-grounded, with only minor overreach around concepts like release freezes that were not actually stated.
- Correctly identifies the customer-visible deployment incident as the central missed discovery moment and the likely source of executive urgency.
- Strongly captures the compliance/vendor-risk miss by contrasting Darius’s specific security concerns with Nora’s generic developer-workflow response.
- Appropriately balances praise for a professional opening and booked follow-up with the warning that the opportunity remains weakly qualified.
- Provides highly actionable follow-up questions around incident details, rollback mechanics, deploy confidence, security review requirements, and internal build-vs-buy competition.
- Accurately notes that Vercel’s positioning was generic and should have been tied to Mercury’s release-risk, rollback, and fintech governance concerns.
- The coach could have slightly more explicitly called out Mercury-specific customer workflows such as onboarding, dashboard, login, or banking/payment surfaces as examples of where frontend reliability matters.
- A few comments, such as release-freeze cost, go beyond the literal transcript, though they remain directionally relevant to reliability discovery.
395opus 4.7 mediumexcellent
The coach output closely matches the hidden ground truth: it characterizes the call as polite and operationally well-managed but strategically shallow, with the main failures being generic BANT-style discovery, missed reliability/rollback cues, and a generic response to compliance/vendor-risk pressure. It also appropriately limits praise to agenda-setting, buying-process mapping, and concrete next steps. The assessment is highly transcript-grounded and actionable, with only minor overstatements around procurement confirmation and one product-specific rollback recommendation not established in the supplied case materials.
- Correctly identifies the recent customer-visible frontend incident as the deal's likely center of gravity and the biggest missed discovery thread.
- Accurately catches the compliance/vendor-risk moment where Darius offered specific criteria and Nora pivoted to generic developer-experience messaging.
- Appropriately treats the booked follow-up as a limited strength rather than evidence of a well-qualified opportunity.
- Adds useful, transcript-grounded coaching on exploring the internal platform-build alternative and activating Marcus more effectively.
- No material hidden benchmark needle was missed.
- The coach slightly overstated procurement confirmation, which was asked about but not confirmed.
- The coach introduced one specific Vercel rollback capability that was not established in the transcript or supplied research.
495opus 4.7 highExcellent / strongly aligned with ground truth
The coach correctly characterized the call as polite and organized but strategically shallow. It identified the central benchmark issues: Nora over-relied on BANT/process questions, failed to unpack the customer-visible deployment incident and rollback/release-risk concerns, talked past Darius’s compliance/vendor-risk cues, and positioned Vercel generically around developer workflow rather than Mercury-specific fintech reliability and governance outcomes. It also properly credited the professional opening and concrete follow-up without over-weighting them. Minor issues: a few add-on observations go beyond the hidden benchmark, especially the “marketing as low-risk beachhead” claim, which is plausible but not firmly established by the transcript.
- Correctly labeled the call as superficially positive but strategically underdeveloped.
- Identified the customer-visible release issue as the most important missed discovery moment.
- Precisely caught Nora’s pivot from reliability pain to budget/sign-off qualification.
- Precisely caught Nora’s pivot from Darius’s compliance/vendor-risk cue to generic developer-workflow messaging.
- Balanced the critique by crediting the polite agenda, professional tone, and concrete follow-up.
- Provided highly actionable replacement questions and drills for the next call.
- No major benchmark miss. The coach covered all hidden needles with strong evidence.
- The fintech-context critique was accurate but could have been even sharper by naming Mercury-specific customer journeys or surfaces likely to carry the most reliability/compliance risk.
- A few non-benchmark add-ons, especially the marketing pilot/beachhead idea, are plausible but less transcript-grounded than the core findings.
595gpt-5.4 xhighExcellent alignment with the benchmark. The coach correctly characterized the call as polite and operationally tidy but strategically shallow, and it identified the major missed discovery moments around reliability, rollback risk, fintech security/vendor review, and generic BANT-style qualification.
The coach output captures the hidden ground truth very well. It does not over-credit the booked follow-up; instead, it emphasizes that Nora kept the opportunity alive while failing to develop Mercury’s real business case. The strongest parts are the diagnosis of the missed customer-visible release incident, the compliance/vendor-review miss, and the checklist-over-signal discovery pattern. The only mild gap is that the coach could have more explicitly called out lack of Mercury-specific account preparation around banking/customer workflows, though it covered the broader fintech relevance and generic positioning issue.
- Correctly framed the call as superficially positive but strategically underdeveloped, matching the benchmark’s central interpretation.
- Strongly identified the missed customer-visible release incident and explained why it should have become the business-case thread.
- Strongly identified the compliance/vendor-risk miss, including access model, evidence, audit trail, and vendor-review friction.
- Appropriately limited praise for the polite opening and booked follow-up rather than over-scoring the call on hygiene.
- Provided actionable next-call questions and coaching drills that map directly to the missed discovery areas.
- The coach could have been more explicit that Nora showed weak Mercury-specific account preparation by not naming or asking about banking-related customer journeys, such as onboarding, login/dashboard, or money-movement surfaces.
- The coach’s minor claim about procurement as a likely participant is only partially grounded because procurement was not confirmed by the buyer.
- Otherwise, there are no significant misses against the hidden benchmark.
694gpt-5.4 nonestrong_pass
The coach output closely matches the hidden ground truth. It correctly characterizes the call as polite and organized but strategically shallow, with the key missed opportunities centered on generic BANT/checklist discovery, under-explored reliability and rollback risk, and insufficient security/compliance discovery for a fintech buyer. It also appropriately gives limited credit for professional call control and a concrete next step. The only modest gap is that the coach could have more explicitly called out weak Mercury-specific/account-context preparation beyond the broader fintech/compliance framing, but it still substantially captured that issue through comments about generic positioning and lack of tailoring.
- Correctly identifies the call as superficially positive but strategically underdeveloped, rather than over-crediting the booked follow-up.
- Accurately prioritizes the missed reliability/release-risk discovery around customer-visible instability, rollback inconsistency, and confidence in production safety.
- Accurately flags that Nora talked past Darius’s vendor-risk/compliance cue and should have probed access model, evidence, audit trail, controls, and review process.
- Provides practical, transcript-grounded follow-up questions and coaching drills that map directly to the hidden gaps.
- Balances critique with fair praise for Nora’s agenda, tone, stakeholder mapping, and concrete next step.
- The coach could have more explicitly framed one flaw as weak Mercury-specific account preparation, including missed questions about Mercury’s particular customer-facing banking workflows such as onboarding, dashboard, login, or payments.
- It could have been slightly sharper that BANT questions should be background qualification rather than the main discovery spine, although this was mostly covered through the checklist-driven critique.
794gpt-5.4 lowStrong judge-aligned coaching output
The coach output accurately captures the hidden ground truth: the call was cordial and organized, with a reasonable follow-up, but strategically shallow. It correctly identifies Nora’s over-reliance on qualification mechanics, missed reliability/rollback discovery, generic Vercel positioning, and failure to probe fintech-relevant security/vendor-review concerns. The assessment is well grounded in transcript evidence and does not materially invent unsupported issues. Minor over-credit appears in describing the next step as having distinct workflow and security tracks, and there is a small mention of “release freezes” despite no explicit transcript evidence, but these do not undermine the overall quality.
- Correctly identifies the central reliability miss: Elena’s 'customer-visible weirdness' and 'release-risk problem' should have prompted layered discovery on incident impact, rollback, and confidence requirements.
- Accurately flags that Nora talked past Darius’s vendor-risk cue by reverting to generic Git workflow and preview URL messaging instead of probing access model, audit trail, evidence, and approval requirements.
- Balances the assessment well: the coach gives Nora credit for meeting control and next-step hygiene while making clear that the booked follow-up does not compensate for shallow discovery.
- Provides actionable coaching drills and follow-up questions that map directly to the missed discovery moments in the transcript.
- The coach could have more explicitly called out the lack of Mercury-specific business-surface discovery, such as asking about onboarding, login, dashboards, payments, or other high-risk customer-facing financial workflows.
- The coach slightly over-credits the next step as having distinct security and workflow tracks when the transcript shows a vendor packet plus a workflow-mapping session, not a fully formed security workstream.
894gpt-5.4 mediumStrong hit
The coach output closely matches the hidden ground truth. It correctly characterizes the call as professionally managed but strategically underdeveloped, with the seller over-relying on qualification mechanics and generic Vercel workflow messaging while missing the deeper reliability, rollback, and fintech vendor-risk signals. The coach is well grounded in transcript evidence, prioritizes the right coaching themes, and offers concrete follow-up questions and drills. The only meaningful gap is that it only partially develops the Mercury-specific/account-context issue beyond fintech security and generic release-risk language.
- Correctly identifies the customer-visible release incident as the strongest missed urgency signal and explains the discovery questions Nora should have asked.
- Accurately diagnoses the compliance/vendor-risk miss, including the need to ask about evidence, audit trails, access controls, SSO/SAML, data handling, and approval blockers.
- Balances praise for a professional opening and reasonable next step with the more important point that the opportunity remains underqualified.
- Uses strong transcript evidence throughout, including direct buyer quotes about rollback inconsistency, release-risk confidence, and vendor review.
- Provides actionable coaching through prioritized drills, follow-up questions, and specific ways to reframe Vercel’s value around safety nets and deployment confidence.
- The coach only partially addresses the Mercury-specific account-context gap. It mentions fintech and customer-facing delivery path, but does not fully highlight the absence of Mercury-specific workflow hypotheses such as onboarding, login, dashboards, banking flows, payments, or customer trust.
- The coach adds a “solutions consultant was underused” point. This is reasonable and transcript-supported, but it is not one of the hidden benchmark’s central themes and should remain secondary, as the coach mostly treats it.
994gpt-5.5 xhighStrong coach output; closely matches the hidden ground truth with only minor gaps.
The coaching model correctly characterized the call as polite and operationally well-run but strategically shallow. It identified the core flaws: Nora over-relied on qualification/process questions, failed to develop Mercury’s reliability and rollback-risk cues, talked past Darius’s vendor-risk/compliance concerns, and positioned Vercel mostly around generic Git-based workflow and preview URLs. It also appropriately credited the professional opening and reasonable follow-up without overvaluing them. The only modest gap is that the coach did not foreground the broader Mercury-specific fintech/account-context miss as a standalone theme as strongly as the benchmark, though it did address it through reliability, compliance, and customer-facing delivery risk.
- Correctly identified that the customer-visible release issue was the highest-value pain cue and that Nora moved past it into budget/sign-off instead of exploring impact and rollback risk.
- Correctly diagnosed the compliance/vendor-risk miss, especially Darius’s explicit mention of access model, evidence, and audit trail.
- Balanced the assessment well: it credited professional call control and next-step hygiene while still rating the discovery foundation as weak.
- Provided highly actionable follow-up questions and practice drills tied to the actual missed discovery moments.
- The coach could have made the Mercury-specific account-preparation gap more explicit, especially around likely fintech web surfaces such as onboarding, login, dashboard, payments, or customer trust implications.
- The coach added an SC-underutilization theme that is transcript-supported and useful, but it was not one of the core benchmark needles; fortunately it did not distract from the main findings.
1094gpt-5.5 mediumExcellent match to ground truth with only minor omissions/overreach.
The coach output accurately identifies the call as polite and operationally competent but strategically shallow. It strongly catches the main hidden flaws: Nora over-indexed on BANT-style qualification, failed to explore Mercury’s reliability/release-risk cues, talked past Darius’s compliance/vendor-risk concerns, and positioned Vercel too generically around developer workflow. It also appropriately credits the professional tone and reasonable next step without overvaluing the booked follow-up. The main gap is that the coach could have more explicitly framed the seller as underprepared on Mercury-specific fintech/account context, including concrete Mercury web surfaces and trust-sensitive banking workflows. There is also a small unsupported detail about the call being 22 minutes, but it is not material.
- Correctly identifies the central issue: a superficially professional call that remained strategically underdeveloped.
- Strongly catches the missed reliability/release-risk cue after Elena mentions customer-visible frontend instability.
- Strongly catches that Nora talked past Darius’s vendor-risk and compliance requirements with generic developer-workflow positioning.
- Balances praise and criticism well: it credits the recap/resources/workflow-mapping next step but does not let that obscure missed discovery.
- Provides highly actionable coaching questions for incident discovery, rollback-path mapping, vendor-review requirements, pilot criteria, and build-versus-buy comparison.
- The coach could have more explicitly named weak account preparation as its own issue, including Mercury-specific banking workflows and customer trust implications beyond general fintech compliance.
- The output includes a minor unsupported duration reference.
- It slightly expands beyond the hidden benchmark with a low-severity point about Marcus being underutilized, though that point is reasonably grounded and not harmful.
1194gpt-5.5 lowStrong pass
The coach output closely matches the hidden ground truth. It correctly characterizes the call as professional and viable on the surface, while strategically underdeveloped because Nora over-relied on baseline qualification, missed reliability/release-risk cues, talked past security/vendor-review concerns, and positioned Vercel too generically around developer workflow. It also gives appropriate limited credit for the clear agenda, polite tone, and concrete follow-up. The assessment is well grounded in the transcript and highly actionable, with only minor unsupported wording in a few places.
- Correctly identifies that the customer-visible frontend release issue was the most important pain cue and should have triggered deeper discovery.
- Strongly captures the compliance/vendor-risk miss, including the need to clarify access model, evidence, audit trail, SSO/SAML, RBAC, SOC 2, data handling, support, and approval requirements.
- Accurately frames the call as superficially positive but strategically underdeveloped: polite, structured, and with a follow-up, yet weak on business case and risk qualification.
- Gives highly actionable coaching questions for the next call, especially around incident impact, rollback process, workflow mapping, vendor review, and pilot success criteria.
- Appropriately credits the seller’s opening, tone, baseline qualification, and next-step hygiene without letting those strengths mask the deeper discovery gaps.
- The coach could have made the Mercury-specific account-context gap slightly sharper by naming likely high-risk fintech surfaces such as login, onboarding, dashboard, payments, or money-movement workflows.
- A couple of minor phrasing choices overstate transcript wording, especially saying Darius used “control surface.”
1294opus 4.7 lowStrong pass: the coach output closely matches the hidden ground truth.
The coach correctly characterized the call as polite and structurally competent but strategically shallow. It identified the main benchmark flaws: Nora relied on checklist/BANT-style discovery, failed to pursue the customer-visible deployment incident and rollback risk, talked past Darius’s compliance/vendor-review cues, and defaulted to generic developer-experience value. It also appropriately credited the professional tone and clear next step without overvaluing it. Minor deductions only for a few slightly assumptive product or sales inferences beyond the transcript.
- Correctly frames the call as superficially positive but weakly qualified despite a booked follow-up.
- Strongly identifies the missed reliability incident cue and explains why it should have become the business case.
- Accurately calls out Nora’s generic response to Darius’s specific vendor-risk signals around access model, evidence, and audit trail.
- Uses transcript quotes well, especially Elena’s 'customer-visible weirdness,' 'same release-risk problem,' and Darius’s compliance-review language.
- Provides actionable coaching: ask what happened, quantify impact, define deploy confidence, scope security review requirements, and use Marcus at technical trigger moments.
- The coach could have more explicitly called out the lack of Mercury-specific account preparation around banking/customer workflows such as onboarding, login, dashboard, or payments surfaces.
- A few recommendations introduce product or sales assumptions that are plausible but not fully established by the transcript, such as 'instant rollback' and easiest beachhead selection.
- The coach did not deeply separate business impact discovery from technical workflow discovery, though it did identify both categories well.
1394gpt-5.4 highStrong pass: the coach output is highly aligned with the hidden ground truth.
The coach correctly characterizes the call as polished but strategically underdeveloped. It identifies the major hidden flaws: Nora over-relied on qualification mechanics, failed to investigate Mercury’s reliability and rollback-risk cues, handled security/vendor review generically, and positioned Vercel in a feature-led way. It also gives appropriate limited credit for the professional opening and concrete follow-up. The main gap is that the coach only partially isolates the seller’s lack of Mercury-specific fintech/account context as its own issue, though it covers adjacent themes through security, customer-facing risk, and generic positioning.
- Accurately frames the call as operationally competent but strategically shallow, matching the benchmark’s warning not to over-credit the friendly follow-up.
- Strongly identifies the missed reliability/release-risk discovery after Elena’s customer-visible incident and rollback-confidence comments.
- Strongly identifies that Darius’s vendor-risk comments required probing into access model, evidence, audit trails, controls, and approval process rather than a generic packet.
- Provides highly actionable follow-up questions and drills that map to the actual missed moments in the transcript.
- Balances praise and critique well: professional opening and next step are credited, but not allowed to outweigh discovery gaps.
- The coach could have more explicitly named the Mercury-specific account-context gap: Nora did not tailor discovery to fintech banking workflows such as onboarding, dashboard, login, payments, customer trust, or governed release controls.
- The coach covers generic positioning, but could have separated 'feature-led Vercel pitch' from 'lack of Mercury-specific business hypothesis' more cleanly.
- The coach’s extra points about Marcus underuse and pilot selection are useful, but they slightly broaden the critique beyond the hidden benchmark’s core issues.
1494deepseek v4 proExcellent benchmark alignment
The coach accurately recognized the call as superficially professional but strategically shallow. It captured the main hidden issues: Nora over-relied on BANT-style qualification, missed the reliability incident cue, talked past security/vendor-risk requirements, and positioned Vercel generically rather than around Mercury’s fintech-specific release-risk and governance needs. It also correctly gave limited credit for the clear agenda and concrete follow-up. Evidence was well grounded in transcript quotes. The only minor overreach is an extra emphasis on bringing Marcus into the call, which is plausible but not central to the benchmark.
- Correctly framed the call as professional but generic, with weak strategic discovery despite a scheduled follow-up.
- Accurately identified the exact incident cue about “customer-visible weirdness” and the seller’s premature pivot to budget/sign-off.
- Strongly captured the compliance/vendor-risk miss, including the lack of probing on access model, evidence, audit trail, SOC 2, SSO, data handling, and review requirements.
- Balanced critique with fair praise for agenda-setting, role clarity, recap, vendor packet, and workflow-mapping next step.
- Provided actionable replacement questions and coaching drills rather than only diagnosing the failure.
- The coach could have made the Mercury-specific context gap more concrete by naming likely high-risk web surfaces such as onboarding, login, dashboard, money movement, or customer-facing banking workflows.
- The Marcus/co-seller point is somewhat peripheral relative to the benchmark’s core emphasis on reliability, compliance, and fintech-specific discovery.
- Some recommendations slightly imply mapping Vercel controls during the first call; the hidden benchmark mainly required clarifying security requirements and proposing a concrete review path, not necessarily going deep on product/security claims immediately.
1594gpt-5.5 highExcellent coaching evaluation; it captures the hidden ground truth with only minor omissions.
The coach correctly characterizes the call as professional and superficially successful but strategically underdeveloped. It identifies the central issues: Nora relied too much on checklist/BANT qualification, failed to pursue Mercury’s strongest reliability and rollback-risk signals, talked past Darius’s compliance/vendor-risk cues, and positioned Vercel generically around Git workflows and previews rather than Mercury-specific fintech risk. It also appropriately gives limited credit for tone, agenda control, and a reasonable follow-up. The only notable gap is that the coach could have been more explicit about Mercury-specific business surfaces such as onboarding, dashboard, login, payments, or customer-facing financial workflows. A few extra observations, such as Marcus being underused, are not in the hidden needles but are transcript-supported and useful.
- Correctly frames the call as superficially positive but strategically underdeveloped, matching the hidden outcome bias.
- Strongly identifies the missed reliability/rollback discovery after Elena mentions customer-visible instability and release-risk concerns.
- Strongly identifies the missed compliance/vendor-risk discovery after Darius raises access model, evidence, and audit trail requirements.
- Balances criticism with fair credit for Nora’s professional opening, baseline discovery, and concrete follow-up.
- Provides highly actionable coaching drills and follow-up questions that map directly to the missed discovery areas.
- Could have more explicitly named Mercury-specific business workflows such as onboarding, dashboard, login, payments, or customer-facing banking experiences as discovery targets.
- Could have stated even more directly that BANT should be background qualification rather than the spine of the call, though this idea is clearly implied.
- Includes a minor unsupported detail about the call being 22 minutes.
1693sonnet 4.6Excellent coaching assessment with minor grounding issues
The coach model captured the hidden ground truth very strongly: it correctly framed the call as polite but strategically shallow, identified Nora’s BANT/checklist pattern, highlighted the missed reliability and rollback cues, called out the compliance/vendor-risk miss, noted weak fintech-specific tailoring, and still credited the clean next step. The assessment is highly actionable and well-prioritized. Main deductions are for a few unsupported or imprecise claims presented as facts or quotes, such as the call duration, inflated buyer titles, and invented wording like “control surface” or “a release that got more attention than anyone wanted.” These do not materially change the evaluation, but they weaken evidence discipline.
- Correctly diagnosed the call as superficially positive but strategically underdeveloped rather than over-crediting the friendly next step.
- Strongly identified the customer-visible deployment incident as the likely urgency driver and called out the failure to probe impact, recovery, rollback, and reliability criteria.
- Accurately flagged Darius’s vendor-risk comment as a major missed buying-process signal and recommended targeted compliance discovery instead of a generic packet.
- Gave concrete, high-quality follow-up questions and coaching drills that would help Nora recover the next conversation.
- Balanced critique with fair credit for Nora’s opening, agenda control, and confirmed next step.
- The coach did not meaningfully miss any hidden benchmark needle.
- Evidence discipline could be tighter: avoid inventing durations, titles, or transcript-adjacent quotes.
- Some product/fintech recommendations are plausible, but should be framed as discovery areas to validate rather than facts established in the call.
1792gemini 3.1 pro previewStrong match to ground truth
The coach accurately read the call as superficially successful but strategically underdeveloped. It identified the core hidden flaws: Nora relied on BANT-style qualification, failed to unpack the customer-visible deployment incident and rollback/safety-net concerns, talked past Darius’s vendor-risk requirements, and gave generic Vercel/developer-workflow positioning rather than fintech-specific reliability and compliance discovery. The coach also correctly credited the polite structure and reasonable follow-up without over-weighting it. Minor deductions are for slightly broad wording around “ignoring” security despite Nora offering to send a packet, and for a small amount of product-feature prescription that goes beyond the transcript.
- Correctly identified the customer-visible deployment issue as the compelling event Nora should have unpacked.
- Correctly linked the seller’s immediate pivot to budget/sign-off with shallow BANT-driven discovery.
- Correctly saw Darius’s access model/evidence/audit trail comment as a major security/vendor-risk cue, not a side note.
- Appropriately balanced praise for agenda and next steps with the conclusion that the call was strategically weak.
- Provided concrete follow-up questions that would improve discovery around incident impact, rollback, vendor review requirements, and deployment confidence.
- The coach could have been more explicit that Mercury-specific business surfaces — login, onboarding, dashboard, money movement, or other customer-facing financial workflows — should shape discovery.
- It could have more clearly distinguished between a security documentation handoff and a real security-review plan with owners, requirements, and timing.
- It slightly over-prescribed Vercel feature positioning where the better coaching emphasis would be to ask requirements first and only then map capabilities carefully.
1892opus 4.7 xhighWorstStrong pass
The coach output closely matches the hidden ground truth. It correctly frames the call as polite and operationally competent but strategically shallow, with the main failures being BANT-heavy discovery, missed reliability/rollback cues, and weak handling of security/vendor-risk requirements. It also gives appropriate limited credit for the professional close and concrete follow-up. The main gaps are that it only partially develops the Mercury-specific/fintech-account-context issue, and it includes a small amount of unsupported wording, including one invented quote-like phrase.
- Correctly identifies the call as superficially competent but shallow, matching the hidden call-out that a booked follow-up should not be over-credited.
- Very strong diagnosis of the missed incident/reliability moment, including the failure to ask about impact, rollback, response, and success criteria.
- Very strong diagnosis of the compliance/vendor-risk miss, with specific and actionable questions Darius should have been asked.
- Appropriately credits Nora’s professional tone, agenda, and concrete next step without letting those positives dominate the assessment.
- Prioritized coaching plan is practical and well ordered: pain-signal follow-up first, security discovery second, buyer-language value translation third.
- The coach could have more explicitly named the lack of Mercury-specific business preparation, such as not asking about login, onboarding, dashboard, payments, money-movement, or other high-risk customer-facing fintech workflows.
- The coach includes one quote-like phrase that does not appear in the transcript, which slightly weakens evidence discipline.
- The SC-underutilization theme is grounded and useful, but it is not part of the hidden benchmark’s core truth; the coach spends some attention there that could have gone deeper into Mercury-specific fintech context.