Loading, Please wait..!!
AI QA Engineer
-
Posted on: Oct 28, 2026
-
New York, New York
-
Onsite
-
Full-time
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for AI QA Engineer roles in New York.
View Salary Guide →Find your match !
Job Title :
AI QA Engineer
Job Type :
Full-Time
Job Location :
New York New York United States
Remote :
No
Job Description :
Job Description
AI QA Engineer
LocationHybrid / Remote
Employment TypeFull-time
Job SummaryWe are seeking an AI QA Engineer to ensure the quality, reliability, safety, and performance of AI-powered applications and machine learning solutions. The ideal candidate will design and execute testing strategies for AI systems, including large language model (LLM) applications, generative AI features, machine learning models, and AI-driven workflows. This role requires expertise in traditional software quality assurance combined with an understanding of AI model behavior, prompt engineering, and automated testing.
Key Responsibilities- Design and implement test strategies for AI and machine learning applications.
- Develop and maintain automated test frameworks for AI-powered systems.
- Validate LLM responses for accuracy, consistency, relevance, and safety.
- Create and manage test datasets, prompts, and evaluation benchmarks.
- Test AI features for functionality, usability, performance, security, and reliability.
- Identify, document, and track defects using bug management tools.
- Perform regression, integration, API, end-to-end, and performance testing.
- Evaluate AI systems for hallucinations, bias, toxicity, and prompt injection vulnerabilities.
- Collaborate with AI engineers, data scientists, product managers, and developers to resolve issues.
- Analyze model outputs and recommend improvements to prompts or workflows.
- Monitor AI application performance after deployment.
- Ensure compliance with organizational quality standards and AI governance policies.
- Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
- 3 5 years of experience in software quality assurance or test automation.
- Experience testing web, mobile, or API-based applications.
- Familiarity with AI, machine learning, or generative AI applications.
- Strong understanding of software testing methodologies and QA best practices.
- Experience with Agile or Scrum development environments.
- Experience testing LLM-based or generative AI applications.
- Knowledge of AI evaluation frameworks and benchmarking techniques.
- Familiarity with prompt engineering and AI model behavior.
- AI or cloud certifications (AWS, Microsoft Azure, Google Cloud, etc.).
- Experience with MLOps or AI deployment pipelines.
- Manual and automated testing
- Test case design and execution
- API testing (Postman, REST Assured)
- Automation tools (Selenium, Playwright, Cypress)
- Programming (Python, Java, JavaScript, or C#)
- SQL and database validation
- Git and CI/CD pipelines
- Test management tools (Jira, TestRail, Zephyr)
- Performance testing (JMeter, k6)
- AI model evaluation techniques
- Prompt engineering
- LLM testing and validation
- AI safety testing (hallucinations, bias, toxicity, prompt injection)
- Basic machine learning concepts
- JSON and REST APIs
- Analytical thinking
- Strong attention to detail
- Problem-solving
- Communication and documentation
- Collaboration with cross-functional teams
- Critical thinking
- Continuous learning
- Time management
- AI-powered enterprise applications
- Generative AI products
- Machine learning platforms
- SaaS applications
- Conversational AI or chatbot testing
- Cloud-native applications
- Defect detection rate
- Test automation coverage
- AI response quality and accuracy
- Reduction in production defects
- Test execution efficiency
- AI safety and compliance metrics
- Release quality
- Customer satisfaction
- Mean time to detect (MTTD) and resolve (MTTR) issues
- LangChain or similar AI orchestration frameworks
- AI evaluation frameworks (e.g., DeepEval, Ragas, LangSmith)
- Vector databases
- Docker and Kubernetes
- MLOps tools
- Cloud platforms (AWS, Azure, Google Cloud)
- Data annotation and synthetic data generation
- Security testing for AI systems
- Accessibility testing
- Experience with Retrieval-Augmented Generation (RAG) systems
Similar Jobs You May Be Interested In
Senior QA Automation Engineer - Playwright / TypeScript / AI
AI Testing- QA Engineer
QA Engineer – AI/GenAI
QA Engineer - Browser Test Specialist - AI Trainer
Software Engineer AI/ML QA Engineer
AI QA and Test Engineer
Qa Automation Engineer Playwright Ai
QA Automation Engineer – AI
QA Engineer: AI-Assisted Testing
Position Details
Posted:
Oct 28, 2026
Reference Number:
77247bde
Employment:
Full-time
City:
New York
Job Origin:
ziprecruiter
Share this job:
Please wait..!!

