Skip to main content
Loading
loadingbar
Loading, Please wait..!!

AI Test Engineer

  • AI Test Engineer job type Posted on: Oct 28, 2026
  • AI Test Engineer experience level Mindlance
  • AI Test Engineer employment type Ann Arbor, Michigan
  • AI Test Engineer employment type Onsite
  • AI Test Engineer salary Full-time

Point Apply Here APPLY LATER

Curious about compensation?

Explore the historical salary trends, average pay, and estimated compensation for AI Test Engineer roles in Michigan.

View Salary Guide →

Job Title :

AI Test Engineer

Job Type :

Full-Time

Job Location :

Ann Arbor Michigan United States

Remote :

No

Jobcon Logo Job Description :

Job Description

Job Description
6 months 100% onsite. After 6 months 4 days on site and 5th day remote. Candidate will pick their remote day.

Mid-Level AI Software Test Engineer
Position Summary
We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.
The ideal candidate has a strong software testing background, experience building automated test frameworks, and an interest in AI technologies such as LLMs, agents, RAG systems, MCP integrations, and machine learning models. AI engineering organizations increasingly emphasize AI evaluation frameworks, automation, reliability, governance, and production quality, making testing a critical function in AI delivery.
Key Responsibilities
AI Quality Engineering
• Develop comprehensive testing strategies for AI applications, platforms, and services.
• Validate AI model outputs for accuracy, consistency, reliability, and safety.
• Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.
• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.
• Validate AI guardrails, business rules, permissions, and governance controls.
• Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations.
Automation
• Build and maintain automated test frameworks for AI applications.
• Develop automated evaluation pipelines for AI responses and workflows.
• Integrate AI testing into CI/CD pipelines.
• Implement automated quality scoring and regression detection.
• Create reusable test data, mocks, simulators, and validation frameworks.
Platform & Integration Testing
• Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities.
• Validate integrations with external systems, data sources, and enterprise services.
• Verify performance, reliability, scalability, and resiliency of AI workloads.
• Execute load and stress testing for AI services.
Collaboration
• Partner with software engineers, AI engineers, product owners, architects, and security teams.
• Participate in design reviews and provide quality feedback during development.
• Contribute to test strategy, quality standards, and best practices.
• Support production readiness reviews and defect triage activities.
Required Qualifications
• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or related field.
• 3–6 years of software testing, QA automation, or quality engineering experience.
• Experience developing automated test solutions using:
o Python
o Java
o JavaScript/TypeScript
o C#
• Experience with API testing and automation tools.
• Strong understanding of:
o Test automation
o Software development lifecycle (SDLC)
o Agile methodologies
o CI/CD pipelines
• Experience testing distributed systems, web applications, and APIs.
Preferred Qualifications
• Experience testing:
o Generative AI applications
o LLM-based systems
o AI agents
o RAG applications
o MCP-based integrations
• Familiarity with:
o OpenAI
o Claude
o Gemini
o Azure OpenAI
• Experience building evaluation and benchmarking frameworks for AI solutions.
• Experience testing cloud-native applications on Azure, AWS, or GCP.
• Knowledge of responsible AI, AI governance, and AI risk management practices. AI initiatives are typically expected to align with enterprise governance, risk, privacy, and compliance requirements.
Technical Skills
AI Testing
• Prompt Testing
• Response Evaluation
• Hallucination Detection
• Agent Workflow Validation
• RAG Validation
• AI Safety Testing
• AI Regression Testing
• AI Benchmarking
Quality Engineering
• Test Automation
• API Testing
• Integration Testing
• Performance Testing
• Load Testing
• Security Testing
• Defect Analysis
• Root Cause Investigation
Tools & Technologies
• Selenium
• Playwright
• Postman
• JUnit / PyTest
• GitHub Actions
• Jenkins
• Docker
• Kubernetes
• Cloud Platforms (Azure, AWS, GCP)
Success Measures
• Establishes reliable automated AI testing coverage.
• Detects AI quality issues before production deployment.
• Reduces regression defects across AI releases.
• Improves confidence in AI model and agent behavior.
• Enables safe, scalable delivery of AI-powered features.
• Ensures AI solutions meet quality, security, and governance requirements.
Example Projects
• Testing AI chat assistants and copilots
• Validating MCP tools and AI agent workflows
• Evaluating RAG search quality and grounding accuracy
• Automating AI response evaluation frameworks
• Testing AI-powered trading assistants and workflow automation
• Performance testing AI services and orchestration platforms
Typical Level
Experience: 3–6 years software quality engineering, with 1–3 years exposure to AI/ML or Generative AI technologies.
Equivalent Titles:
• AI Software Test Engineer
• AI Quality Engineer
• Generative AI Test Engineer
• AI Automation Engineer
• AI Validation Engineer
• Software Engineer in Test (AI) (SDET-AI)
This role is suitable for an engineer who can independently own AI testing and automation efforts while partnering closely with AI developers, architects, and product teams to ensure production-quality AI solutions.

“Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of – Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans.”

Jobcon Logo Position Details

Posted:

Oct 28, 2026

Reference Number:

a9f37c67

Employment:

Full-time

City:

Ann Arbor

Job Origin:

ziprecruiter

Share this job:

  • linkedin

Jobcon Logo
A job sourcing event
In Dallas Fort Worth
Aug 19, 2017 9am-6pm
All job seekers welcome!

AI Test Engineer    Apply

Click on the below icons to share this job to Linkedin, Twitter!

Job Description

Job Description
6 months 100% onsite. After 6 months 4 days on site and 5th day remote. Candidate will pick their remote day.

Mid-Level AI Software Test Engineer
Position Summary
We are seeking a Mid-Level AI Software Test Engineer to design, automate, and execute testing strategies for AI-powered applications, machine learning systems, AI agents, copilots, and generative AI solutions. This role combines traditional software quality engineering with emerging AI validation techniques to ensure AI systems are reliable, safe, accurate, performant, and production-ready.
The ideal candidate has a strong software testing background, experience building automated test frameworks, and an interest in AI technologies such as LLMs, agents, RAG systems, MCP integrations, and machine learning models. AI engineering organizations increasingly emphasize AI evaluation frameworks, automation, reliability, governance, and production quality, making testing a critical function in AI delivery.
Key Responsibilities
AI Quality Engineering
• Develop comprehensive testing strategies for AI applications, platforms, and services.
• Validate AI model outputs for accuracy, consistency, reliability, and safety.
• Design and execute functional, integration, end-to-end, regression, and performance tests for AI solutions.
• Create test cases for prompt-driven, agentic, and retrieval-based AI workflows.
• Validate AI guardrails, business rules, permissions, and governance controls.
• Perform adversarial, negative, and edge-case testing to identify model failures and hallucinations.
Automation
• Build and maintain automated test frameworks for AI applications.
• Develop automated evaluation pipelines for AI responses and workflows.
• Integrate AI testing into CI/CD pipelines.
• Implement automated quality scoring and regression detection.
• Create reusable test data, mocks, simulators, and validation frameworks.
Platform & Integration Testing
• Test AI agents, workflows, APIs, MCP integrations, and tool-calling capabilities.
• Validate integrations with external systems, data sources, and enterprise services.
• Verify performance, reliability, scalability, and resiliency of AI workloads.
• Execute load and stress testing for AI services.
Collaboration
• Partner with software engineers, AI engineers, product owners, architects, and security teams.
• Participate in design reviews and provide quality feedback during development.
• Contribute to test strategy, quality standards, and best practices.
• Support production readiness reviews and defect triage activities.
Required Qualifications
• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or related field.
• 3–6 years of software testing, QA automation, or quality engineering experience.
• Experience developing automated test solutions using:
o Python
o Java
o JavaScript/TypeScript
o C#
• Experience with API testing and automation tools.
• Strong understanding of:
o Test automation
o Software development lifecycle (SDLC)
o Agile methodologies
o CI/CD pipelines
• Experience testing distributed systems, web applications, and APIs.
Preferred Qualifications
• Experience testing:
o Generative AI applications
o LLM-based systems
o AI agents
o RAG applications
o MCP-based integrations
• Familiarity with:
o OpenAI
o Claude
o Gemini
o Azure OpenAI
• Experience building evaluation and benchmarking frameworks for AI solutions.
• Experience testing cloud-native applications on Azure, AWS, or GCP.
• Knowledge of responsible AI, AI governance, and AI risk management practices. AI initiatives are typically expected to align with enterprise governance, risk, privacy, and compliance requirements.
Technical Skills
AI Testing
• Prompt Testing
• Response Evaluation
• Hallucination Detection
• Agent Workflow Validation
• RAG Validation
• AI Safety Testing
• AI Regression Testing
• AI Benchmarking
Quality Engineering
• Test Automation
• API Testing
• Integration Testing
• Performance Testing
• Load Testing
• Security Testing
• Defect Analysis
• Root Cause Investigation
Tools & Technologies
• Selenium
• Playwright
• Postman
• JUnit / PyTest
• GitHub Actions
• Jenkins
• Docker
• Kubernetes
• Cloud Platforms (Azure, AWS, GCP)
Success Measures
• Establishes reliable automated AI testing coverage.
• Detects AI quality issues before production deployment.
• Reduces regression defects across AI releases.
• Improves confidence in AI model and agent behavior.
• Enables safe, scalable delivery of AI-powered features.
• Ensures AI solutions meet quality, security, and governance requirements.
Example Projects
• Testing AI chat assistants and copilots
• Validating MCP tools and AI agent workflows
• Evaluating RAG search quality and grounding accuracy
• Automating AI response evaluation frameworks
• Testing AI-powered trading assistants and workflow automation
• Performance testing AI services and orchestration platforms
Typical Level
Experience: 3–6 years software quality engineering, with 1–3 years exposure to AI/ML or Generative AI technologies.
Equivalent Titles:
• AI Software Test Engineer
• AI Quality Engineer
• Generative AI Test Engineer
• AI Automation Engineer
• AI Validation Engineer
• Software Engineer in Test (AI) (SDET-AI)
This role is suitable for an engineer who can independently own AI testing and automation efforts while partnering closely with AI developers, architects, and product teams to ensure production-quality AI solutions.

“Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of – Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans.”

Loading
Please wait..!!

Choose which optional categories you allow. Optional switches start turned off until you enable them. Withdrawing consent takes effect immediately.