Skip to main content
Loading
loadingbar
Loading, Please wait..!!

Quality Assurance Tester

  • Quality Assurance Tester job type Posted on: Nov 02, 2026
  • Quality Assurance Tester experience level Matlen Silver
  • Quality Assurance Tester employment type New York, New York
  • Quality Assurance Tester employment type Onsite
  • Quality Assurance Tester salary Full-time

Point Apply Here APPLY LATER

Curious about compensation?

Explore the historical salary trends, average pay, and estimated compensation for Quality Assurance Tester roles in New York.

View Salary Guide →

Job Title :

Quality Assurance Tester

Job Type :

Full-Time

Job Location :

New York New York United States

Remote :

No

Jobcon Logo Job Description :

Global Markets GenAI QA TesterLocation: New York, NY (2BP – onsite interview required)Duration: 18 monthsWork Arrangement: W2 contract onlyExperience: 5–10 years**W2 ONLY NO SPONSORSHIP ALLOWED**Job DescriptionBank of America is seeking a QA Tester with experience in Generative AI, test automation, and Python-based applications to support its Global Markets GenAI platform. This role will focus on validating LLM outputs, evaluating model quality, and testing the underlying AI infrastructure to ensure reliability, accuracy, and performance.The platform, Orchestra, enables secure access to external large language models (LLMs) while enforcing user authorization and controlled integrations. The ideal candidate has hands-on experience applying AI to testing, building automation solutions, and evaluating LLM responses.Key ResponsibilitiesDevelop and execute test strategies, test cases, and automated test suites for AI-powered applications and infrastructure.Evaluate LLM outputs and establish methods to measure response quality, accuracy, and reliability.Apply LLM-as-a-Judge techniques to assess model responses and identify quality issues.Develop or leverage AI agents to automate testing workflows and improve testing efficiency.Automate Python application testing using PyTest and Playwright.Use tools such as LangChain and LangSmith to inspect, trace, and evaluate model outputs.Perform database testing and data validation across MongoDB and other data platforms.Manage defects, report testing progress, and escalate risks and issues.Collaborate with developers, data scientists, product owners, and platform teams to support releases and continuous improvement.Ensure testing practices comply with enterprise security, data privacy, and governance requirements.Required Qualifications5–10 years of experience in QA, software testing, test automation, or a related technical role.Hands-on experience testing Python applications using PyTest.Experience with Playwright for test automation.Experience with MongoDB or other NoSQL databases.Practical experience using LLMs in testing, including LLM-as-a-Judge techniques or AI-agent-based test automation.Understanding of how to evaluate LLM responses and measure model quality.Familiarity with AI testing tools and frameworks such as LangChain and/or LangSmith.Strong analytical, troubleshooting, communication, and problem-solving skills.Preferred QualificationsExperience testing AI/ML applications or Generative AI platforms.Exposure to SQL, NoSQL databases, and data validation.Familiarity with Azure, Google Cloud Platform (GCP), or AWS.Experience working in Agile environments using Jira and DevOps practices.Financial services or banking industry experience.Experience collaborating with cross-functional and onshore/offshore teams.

Jobcon Logo Position Details

Posted:

Nov 02, 2026

Reference Number:

10440_4483576482

Employment:

Full-time

City:

New York

Job Origin:

APPCAST_CPC

Share this job:

  • linkedin

Jobcon Logo
A job sourcing event
In Dallas Fort Worth
Aug 19, 2017 9am-6pm
All job seekers welcome!

Quality Assurance Tester    Apply

Click on the below icons to share this job to Linkedin, Twitter!

Global Markets GenAI QA TesterLocation: New York, NY (2BP – onsite interview required)Duration: 18 monthsWork Arrangement: W2 contract onlyExperience: 5–10 years**W2 ONLY NO SPONSORSHIP ALLOWED**Job DescriptionBank of America is seeking a QA Tester with experience in Generative AI, test automation, and Python-based applications to support its Global Markets GenAI platform. This role will focus on validating LLM outputs, evaluating model quality, and testing the underlying AI infrastructure to ensure reliability, accuracy, and performance.The platform, Orchestra, enables secure access to external large language models (LLMs) while enforcing user authorization and controlled integrations. The ideal candidate has hands-on experience applying AI to testing, building automation solutions, and evaluating LLM responses.Key ResponsibilitiesDevelop and execute test strategies, test cases, and automated test suites for AI-powered applications and infrastructure.Evaluate LLM outputs and establish methods to measure response quality, accuracy, and reliability.Apply LLM-as-a-Judge techniques to assess model responses and identify quality issues.Develop or leverage AI agents to automate testing workflows and improve testing efficiency.Automate Python application testing using PyTest and Playwright.Use tools such as LangChain and LangSmith to inspect, trace, and evaluate model outputs.Perform database testing and data validation across MongoDB and other data platforms.Manage defects, report testing progress, and escalate risks and issues.Collaborate with developers, data scientists, product owners, and platform teams to support releases and continuous improvement.Ensure testing practices comply with enterprise security, data privacy, and governance requirements.Required Qualifications5–10 years of experience in QA, software testing, test automation, or a related technical role.Hands-on experience testing Python applications using PyTest.Experience with Playwright for test automation.Experience with MongoDB or other NoSQL databases.Practical experience using LLMs in testing, including LLM-as-a-Judge techniques or AI-agent-based test automation.Understanding of how to evaluate LLM responses and measure model quality.Familiarity with AI testing tools and frameworks such as LangChain and/or LangSmith.Strong analytical, troubleshooting, communication, and problem-solving skills.Preferred QualificationsExperience testing AI/ML applications or Generative AI platforms.Exposure to SQL, NoSQL databases, and data validation.Familiarity with Azure, Google Cloud Platform (GCP), or AWS.Experience working in Agile environments using Jira and DevOps practices.Financial services or banking industry experience.Experience collaborating with cross-functional and onshore/offshore teams.

Loading
Please wait..!!