American Express is seeking a Senior Technical Product Manager to own shared evaluation and observability tools for agent-enabled customer experiences. The role will define the product roadmap, drive adoption, and help teams measure agent performance, diagnose failures, and improve workflows.
What you’ll do
Own the product roadmap for agent evaluation and observability tools
Define requirements for shared workflows that measure agent performance and diagnose failures
Drive adoption and iteration of quality tooling across teams
Enable teams to improve customer experiences through evidence and measurement
Translate ambiguous quality problems into usable tools and repeatable workflows
Balance customer outcomes, engineering constraints, and privacy or risk requirements
What you’ll need
6-8+ years in product management, product development, software engineering, technical platforms, analytics, quality tooling, or AI products
Experience owning evaluation, testing, observability, analytics, or related technical products through launch, adoption, and iteration
Practical understanding of AI or agent-enabled systems, including nondeterministic behavior and tool/API interactions
Experience with LLM or agent evaluation, scenario design, human review, model-assisted scoring, or production failure analysis
Familiarity with traces, logs, APIs, MCP, function calling, telemetry standards, evaluation frameworks, or technical troubleshooting
Cross-functional leadership and clear communication, balancing customer outcomes, engineering constraints, and privacy or risk requirements
Original job description
American Express is hiring a Senior Technical Product Manager to lead shared evaluation and observability tools for agent-enabled customer experiences. The role owns the product roadmap, requirements, adoption, and improvement of workflows that help teams measure agent performance, diagnose failures, and improve experiences.6-8+ years of experience in…