Hiring - AI Engineer at TMS, LLC · Los Angeles, CA · Joined
TL
Hiring - AI Engineer
TMS, LLC · Los Angeles, CA · Just now · 0 applicants
NewHybridContractSenior
Pay
Pay not listed
Workplace
Hybrid
Job type
Contract
Level
Senior
Source: jobright · Posted by joined-scrapper
About the role
TMS, LLC is hiring an AI Engineer to specialize in evaluating LLM-powered agents, focusing on building evaluation frameworks and golden datasets rather than developing agents. The role involves using Langfuse to trace and monitor agent execution, and collaborating with business analysts to maintain benchmarks.
What you’ll do
Evaluate LLM-powered agents for response quality, accuracy, relevance, and reliability
Design, implement, and run LLM-based evaluation frameworks
Collaborate with Business Analysts and stakeholders to create, validate, and maintain golden datasets
Instrument AI agents using Langfuse for tracing and monitoring
Capture evaluation scores, monitor performance, and troubleshoot failures
Test agent behavior, tool usage, reasoning workflows, and response quality
What you’ll need
14+ years of experience
Hands-on experience designing, implementing, and running LLM-based evaluation frameworks
Experience collaborating with Business Analysts and stakeholders on golden datasets
Practical experience instrumenting AI agents with Langfuse
Strong understanding of end-to-end agent evaluation
Actual implementation experience with agent evaluation frameworks
Original job description
TMS, LLC is hiring an AI Engineer – Agent Evaluation Specialist to evaluate LLM-powered agents rather than focus primarily on building them. The role centers on agent evaluation frameworks, golden datasets, and monitoring agent execution and performance with Langfuse.Hands-on experience designing, implementing, and running LLM-based evaluation frameworks…