Senior AI Engineer - Agent Evaluation Specialist at Relevance Lab · Los Angeles metropolitan area · Joined
RL
Senior AI Engineer - Agent Evaluation Specialist
Relevance Lab · Los Angeles metropolitan area · Just now · 0 applicants
NewHybridFull-timeSenior
Pay
Pay not listed
Workplace
Hybrid
Job type
Full-time
Level
Senior
Source: jobright · Posted by joined-scrapper
About the role
This role focuses on evaluating LLM-powered agents rather than primarily building them, including implementing agent evaluation frameworks, creating and maintaining benchmark/golden datasets, and using Langfuse to trace, score, monitor, and troubleshoot agent performance.
What you’ll do
Implement agent evaluation frameworks
Develop and maintain golden datasets with Business Analysts and stakeholders
Use Langfuse to trace agent execution
Capture evaluation scores
Monitor agent performance
Troubleshoot agent failures
What you’ll need
Hands-on experience designing, implementing, and running LLM-based evaluation frameworks
Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets
Practical experience instrumenting AI agents using Langfuse
Strong understanding of end-to-end agent evaluation
Demonstrated implementation experience with agent evaluation frameworks, not just theoretical knowledge
Original job description
Relevance Lab is seeking an AI Engineer focused on evaluating LLM-powered agents rather than primarily building them. The role centers on implementing agent evaluation frameworks, developing benchmark datasets, and using Langfuse to trace, score, monitor, and troubleshoot agent performance.Hands-on experience designing, implementing, and running LLM-based…