Post a Job

Millions Of Active Candidates

For Employers

VIEW SIMILAR JOBS →

Loading, Please wait..!!

Ai Evaluations Engineer

Posted on: Oct 13, 2026

VeeRteq Solutions Inc
New York, New York
Onsite
Full-time

Save this job

Apply Here APPLY LATER

Curious about compensation?

Explore the historical salary trends, average pay, and estimated compensation for Ai Evaluations Engineer roles in New York.

View Salary Guide →

Over 12 millions jobs on Snaprecruit.
Find your match !

VIEW SIMILAR JOBS →

Job Title :

Ai Evaluations Engineer

Job Type :

Full-Time

Job Location :

New York New York United States

Remote :

Job Description :

Role: AI Evaluations Engineer (Senior)

Location: US (remote)

Duration: 12 Months +

Key Responsibilities

Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)
Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows
Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety
Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices

Must Have

5-7+ years building production-grade ML or GenAI systems; advanced Python
Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring
LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity
Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety
Agentic framework experience (Strands, LangGraph, CrewAI or similar)
AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have

Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)
Agent-as-a-Judge framework; HITL evaluation pipelines
Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction
RAG quality evaluation (faithfulness, context precision, recall)
CI/CD integration for eval regression; observability tooling (tracing, structured logging)
Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design
Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)
Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

Position Details

Posted:

Oct 13, 2026

Reference Number:

4601-15867

Employment:

Full-time

City:

New York

Job Origin:

CIEPAL_ORGANIC_FEED

Share this job:

Ai Evaluations Engineer Apply

Click on the below icons to share this job to Linkedin, Twitter!

Role: AI Evaluations Engineer (Senior)

Location: US (remote)

Duration: 12 Months +

Key Responsibilities

Design and operate end-to-end evaluation frameworks for agentic AI systems (offline regression, online monitoring, A/B testing)
Define quality rubrics, scoring architectures, and evaluation standards across multi-agent workflows
Build LLM-as-judge and Agent-as-judge pipelines assessing trajectory, tool-call accuracy, groundedness, and safety
Own and deliver evaluation infrastructure end-to-end; mentor teams and uplevel customers on evals best practices

Must Have

5-7+ years building production-grade ML or GenAI systems; advanced Python
Hands-on experience with agent evaluation: offline/on-demand regression testing and online/continuous live-traffic scoring
LLM-as-judge proficiency: rubric design, scoring at session, response, and tool-call granularity
Core eval metrics: task completion, tool-call accuracy, trajectory quality, groundedness, safety
Agentic framework experience (Strands, LangGraph, CrewAI or similar)
AWS Bedrock (Models, Guardrails) + CloudWatch; Lambda, S3, IAM fundamentals

Should Have

Bedrock AgentCore Evaluations (on-demand SDK runner, online monitoring, A/B testing)
Agent-as-a-Judge framework; HITL evaluation pipelines
Benchmarking frameworks (tau-bench, SWE-bench, GAIA) and custom dataset construction
RAG quality evaluation (faithfulness, context precision, recall)
CI/CD integration for eval regression; observability tooling (tracing, structured logging)
Legacy technology familiarity (Java/Spring, Sybase, DB2, or COBOL) as source context for agentic modernization or integration pattern design
Financial services or regulated-industry experience (compliance-aware delivery: trade surveillance, data controls, audit requirements)
Large-scale portfolio delivery experience - reusable patterns or components applied across 50+ applications, including wave planning and throughput measurement

Please wait..!!

Millions Of Active Candidates

Ai Evaluations Engineer