Product Manager, Toast IQ Evaluation

Toast creates technology to help restaurants and local businesses succeed in a digital world, helping business owners operate, increase sales, engage customers, and keep employees happy. At the center of this vision is Toast IQ: a single, unified assistant that orchestrates specialized agents across the Toast ecosystem. The Toast IQ Quality team makes sure Toast IQ delivers useful, reliable help to restaurant operators. About the Role As Product Manager, Toast IQ Evaluation, you’ll define what good help looks like for a restaurant operator and build the evidence that tells us whether Toast IQ delivers it. You’ll study real usage, work directly with operators, and turn what you learn into datasets, clear quality standards, and priorities for improvement. Your work will help teams across Toast understand where the assistant succeeds, where it falls short, and identify new opportunities for improvement. This is a hands-on product role. Much of your week will be spent reviewing sessions, labeling examples, investigating failures, and building better ways to do that work. Restaurant judgment, attention to detail, and the ability to turn evidence into action are central to the job. You’ll be the team’s first restaurant domain expert, working alongside engineers from one prioritized backlog and reporting through Product. A Day in the life (Responsibilities): • Understand the operator’s experience. Review real Toast IQ sessions every week. Describe what went wrong in plain operator language. Work directly with operators, including in live sessions, to check that our evaluations reflect what matters to them. • Find and prioritize recurring failures. Group observations into failure modes, measure how often they occur, and assess their impact on operators. Use that evidence to identify the problems most worth solving. • Build trustworthy evaluation datasets. Create and maintain labeled examples covering common workflows, edge cases, and known failures. • Verify that improvements hold up. Validate fixes in real sessions and turn important production failures into regression cases that catch the same problems in future releases. • Help teams across Toast evaluate their work. Lead reviewer calibration sessions and help product teams create and maintain evaluation cases for their capabilities. • Turn findings into product decisions. Surface tools that aren’t delivering value, intents Toast IQ can’t handle, and workflows that need a product team’s attention. Make the operator’s problem and its impact clear enough to act on. • Build tools that make the work better. Use AI coding tools to create faster review interfaces, automate recurring analysis, monitor common failures, and prototype fixes. Look for opportunities to improve every step from session review through validation. What you'll need to thrive (Requirements) • Restaurant knowledge: You’ve worked in a restaurant or closely with operators. You understand how decisions about menus, labor, sales, and inventory play out during a busy Friday night. This matters more than your years of product experience. • Care for detailed work: You’re comfortable reading hundreds of transcripts, labeling examples by hand, and keeping datasets accurate and current. • Comfort with structured data: You can investigate data using a database, queries, or an AI coding tool and check that the results are trustworthy. • Precise thinking and writing: You can turn an ambiguous observation into a clear failure definition or scoring rule that someone else can apply consistently. • Clear communication: You can explain what happened, why it matters to operators, and what should change in language that engineers, product teams, and leadership can use. • Curiosity about AI systems: You want to understand ho