Loading, Please wait..!!
AI Evaluation Specialist
-
Posted on: Nov 04, 2026
-
United, West Virginia
-
Remote
-
Full-time
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for AI Evaluation Specialist roles in West Virginia.
View Salary Guide →Find your match !
Job Title :
AI Evaluation Specialist
Job Type :
Full-Time
Job Location :
United West Virginia United States
Remote :
Yes
Job Description :
Job Description
Position Title: Senior Computer Programmer – AI Evaluation Specialist
\nLocation: Remote (Continental U.S.)
\nCompany: JR Software Solutions Inc. (JRSS)
\nEmployment Type: Full-time
\nAbout JR Software Solutions Inc.
\nJRSS is a certified Women-Owned Small Business (WOSB), Small Disadvantaged Business (SDB), and HUBZone business headquartered in Tampa, FL. We specialize in Cloud Services, AI and Machine Learning, Big Data and BI Analytics, Enterprise Resource Planning, IT Project Management, and Staff Augmentation for Federal Civilian agencies (e.g., CDC) and Fortune 500 clients. Our AI/ML teams help public health agencies adopt generative AI responsibly, with the rigor, transparency, and safeguards that federal missions require.
\nPosition Overview
\nThe JRSS AI/ML Team is seeking an AI Evaluation Specialist to determine the effectiveness, safety, usability, and readiness of enterprise AI tools for adoption in a federal public health environment. You will partner with product managers, data scientists, AI engineers, and security and privacy teams to run structured assessments of generative AI products and platforms, including OpenAI tools and life sciences models such as Rosalind. Your findings will guide which AI solutions are adopted, comparing them across quality, accuracy, reliability, safety, privacy, and security.
\nKey Responsibilities
\n- \n
- Plan and execute hands-on evaluations of AI tools, including OpenAI products and life sciences models such as Rosalind, using representative business scenarios, workflows, prompts, datasets, and user personas. \n
- Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness. \n
- Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications. \n
- Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders. \n
- Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost. \n
- Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities. \n
- Translate evaluation findings into clear, actionable recommendations for technical and business stakeholders. \n
- Stay current with advancements in generative AI, model evaluation, AI safety, and enterprise AI products, and incorporate relevant methods into the evaluation program. \n
- Contribute to knowledge sharing and a strong, collaborative team environment. \n
Required Qualifications
\n- \n
- Bachelor's degree. \n
- 5+ years of experience in model evaluation or model quality assurance, with hands-on experience evaluating AI-enabled products or services. \n
- Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria. \n
- Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes. \n
- Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost. \n
- Familiarity with common generative AI architectures and capabilities, including LLMs, RAG, AI agents, APIs, enterprise search, and tool or connector integrations. \n
- Must be a U.S. Citizen or Lawful Permanent Resident (Green Card holder). \n
- Ability to obtain and maintain a Public Trust clearance. \n
Preferred Qualifications
\n- \n
- Master's degree. \n
- Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind, or comparable generative AI solutions. \n
- Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing. \n
- Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows. \n
- Experience building dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results. \n
- Proficiency with Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics and summarize findings. \n
- Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI. \n
What We Offer
\n- \n
- 401(k) plan with a 4% company match. \n
- Two weeks of paid time off. \n
- Comprehensive medical, dental, and vision insurance. \n
- Ongoing investment in professional growth and development. \n
Equal Opportunity Statement
\nJR Software Solutions Inc. is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.
Similar Jobs You May Be Interested In
Medical Coder / CDI Specialist - AI Evaluation
Ai Safety Evaluation Specialist
AI Evaluation Specialist
Video Annotation Specialist, Part-Time AI Video Evaluation - Remote (United States)
Position Details
Posted:
Nov 04, 2026
Reference Number:
1d5f2d707ac633d1
Employment:
Full-time
City:
United
Job Origin:
ziprecruiter
Share this job:
Please wait..!!

