About the role

What you’ll do

• Own Agent Intelligence evaluation, observability, and feedback systems
• Set agent quality standards for production workflows
• Build platforms to evaluate performance and trace failures
• Monitor agent behavior and drive continuous improvement
• Collaborate across engineering, data science, and business teams
• Explain trade-offs clearly to senior leaders

What you’ll need

• 7+ years across product management and engineering, including 3+ years as a product manager on technical platform, infrastructure, data, or ML products
• Systems engineering background building or operating distributed systems, data platforms, infrastructure, or ML systems
• Experience building and running evaluation, quality, or observability systems for production ML or LLM products at scale
• Direct experience with multi-component systems such as agents, tool use, retrieval pipelines, or orchestrated services
• Hands-on data skills in SQL and Python with strong evaluation judgment using code-based checks, LLM judges, and human review
• Experience with LLM evaluation and observability tools, OpenTelemetry, and agent protocols such as MCP and A2A

Original job description