Loading, Please wait..!!
SDET AI Lead/Senior/Junior
-
Posted on: Nov 18, 2026
-
New York, New York
-
Onsite
-
Full-time
Curious about compensation?
Explore the historical salary trends, average pay, and estimated compensation for SDET AI Lead/Senior/Junior roles in New York.
View Salary Guide →Find your match !
Job Title :
SDET AI Lead/Senior/Junior
Job Type :
Full-Time
Job Location :
New York New York United States
Remote :
No
Job Description :
Job Role: SDET AI Lead/Senior/Junior
Job Location: NY/Alpharetta, GA- Day 1 onsite (3 days work from Office, 2 days' work from home)
Job Type: Contract
Lead role- New York
Senior Role- Alpharetta, GA
Junior Role- Alpharetta, GA
About the Role
We are seeking a Lead Software Development Engineer in Test (SDET) to lead our quality engineering strategy and build scalable, reliable, and intelligent test automation solutions. This is a hands-on leadership role combining software engineering, test architecture, mentoring, CI/CD, and artificial intelligence.
The ideal candidate has strong expertise in Python, Playwright, pytest, API testing, automation frameworks, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and AI evaluation. You will collaborate with software engineers, product managers, DevOps teams, data scientists, and AI engineers to improve quality throughout the product lifecycle.
- Lead the design, development, and maintenance of automation frameworks using Python, Playwright, and pytest.
- Define test strategies covering UI, API, integration, end-to-end, regression, smoke, contract, exploratory, and non-functional testing.
- Create scalable, maintainable, and reusable test utilities, fixtures, mocks, test data, and service virtualization.
- Integrate automated testing into CI/CD pipelines using GitHub Actions, GitLab CI, Jenkins, or Azure DevOps.
- Test web applications, REST and GraphQL APIs, microservices, distributed systems, databases, and event-driven architectures.
- Review requirements, user stories, prompts, retrieval workflows, and technical designs to identify risks and define test coverage.
- Investigate test failures, reduce flaky tests, identify root causes, and collaborate with engineering teams to resolve defects.
- Establish coding standards, quality gates, test reporting, dashboards, metrics, and automation best practices.
- Mentor SDETs, QA engineers, and developers in software quality, automation, and AI testing.
- Evaluate and introduce AI-assisted testing, including test generation, intelligent test selection, failure analysis, defect prediction, and test maintenance.
- Design testing strategies for LLM applications, including prompt validation, response quality, hallucination detection, toxicity, bias, safety, latency, cost, consistency, and reliability.
- Test RAG systems, including document ingestion, OCR, preprocessing, chunking, metadata, embeddings, vector databases, retrieval, context assembly, grounded responses, citations, permissions, stale data, and retrieval failures.
- Build automated evaluation frameworks using benchmark datasets, expected responses, deterministic assertions, human feedback, and model-based evaluation.
- Promote the secure, responsible, explainable, and maintainable use of AI in testing.
- Lead-level experience in software testing, quality engineering, or SDET roles, with the ability to own strategy and influence engineering practices.
- Strong hands-on automation development skills using Python, Playwright, and pytest.
- Experience designing, implementing, and maintaining automation frameworks at scale.
- Strong knowledge of UI, API, integration, end-to-end, regression, contract, functional, and non-functional testing.
- Experience testing web applications, APIs, microservices, and distributed systems.
- Practical experience testing LLM-powered applications and RAG architectures.
- Understanding of embeddings, semantic search, vector databases, context retrieval, prompt engineering, and response generation.
- Experience creating AI test datasets, evaluation suites, prompts, expected outputs, edge cases, and quality criteria.
- Knowledge of LLM and RAG metrics, including correctness, relevance, faithfulness, groundedness, context precision, context recall, completeness, safety, and toxicity.
- Experience with Git, CI/CD, logging, debugging, test reporting, and Agile development.
- Strong leadership, mentoring, communication, analytical, and problem-solving skills.
- Experience with Playwright tracing, screenshots, video capture, parallel execution, fixtures, and test sharding.
- Experience with requests, Postman, Allure, JUnit, and comparable testing tools.
- Familiarity with Ragas, DeepEval, promptfoo, LangSmith, or similar LLM evaluation frameworks.
- Experience with OpenAI, Azure OpenAI, Anthropic, Google Vertex AI, or Amazon Bedrock.
- Experience with Pinecone, Weaviate, Milvus, Qdrant, Chroma, pgvector, or other vector databases.
- Knowledge of Docker, Kubernetes, AWS, Azure, GCP, SQL, NoSQL, observability, performance, accessibility, and security testing.
- Experience leading a quality engineering or SDET team.
- Experience testing conversational agents, AI copilots, autonomous agents, multi-turn workflows, and tool-calling systems.
- Familiarity with prompt injection, jailbreaks, data leakage, unauthorized retrieval, insecure output handling, and adversarial testing.
- Experience testing machine learning models, production RAG platforms, or regulated applications.
- Knowledge of responsible AI, privacy, security, governance, and compliance.
- Contributions to open-source testing or AI evaluation projects.
Position Details
Posted:
Nov 18, 2026
Reference Number:
34403-5039
Employment:
Full-time
City:
New York
Job Origin:
CIEPAL_ORGANIC_FEED
Share this job:
Please wait..!!

