Senior Quality Engineer (AI)

Qualifications we are looking for:

• 7+ years of experience in Quality Engineering, Software Development, or a related technical discipline, including recent hands-on experience with AI/ML or LLM-based systems.
• Experience designing validation or evaluation frameworks for non-deterministic or probabilistic systems such as Large Language Models (LLMs) or Machine Learning applications.
• Strong understanding of Retrieval-Augmented Generation (RAG), agentic workflows, and LLM evaluation concepts, including hallucination detection, groundedness, relevance, and consistency.
• Hands-on experience with Large Language Models (LLMs) and prompt engineering, including prompt design, optimization, and regression validation.
• Practical experience using Claude Code, Claude SDK, or similar AI-assisted development platforms and agentic coding tools.
• Proficiency with Node.js for building automation tools, validation pipelines, and engineering utilities.
• Experience working with MongoDB to store, manage, and analyze evaluation results, validation data, or AI output logs.
• Experience integrating automated validation processes into CI/CD pipelines.
• Experience designing semantic validation approaches that evaluate meaning, relevance, and contextual accuracy beyond traditional software testing.
• Strong understanding of AI quality metrics, confidence scoring, output consistency, and production monitoring.
• Solid knowledge of data validation principles, including schema validation, business rule compliance, and data consistency.
• Excellent cross-functional collaboration skills, with experience partnering across Quality Engineering, AI/ML Engineering, Software Engineering, and Data Governance teams.

Nice to have:

• Experience with prompt regression testing tools or frameworks.
• Familiarity with AI governance, fairness, or bias-detection practices.
• Experience with tools such as Playwright, RestSharp, or similar UI/API test automation.
• Prior experience standing up a validation or evaluation function from scratch.
• Exposure to confidence scoring or guardrail systems for production AI.
• Experience designing or orchestrating multi-agent systems in production environments

Job duties:

• Define and lead the AI validation strategy for LLMs, RAG systems, and AI agents.
• Build automated validation pipelines integrated into CI/CD.
• Establish evaluation frameworks covering correctness, relevance, groundedness, consistency, and hallucination rate.
• Design and maintain prompt regression testing to catch silent quality degradation after model or prompt changes.
• Introduce semantic validation techniques that go beyond traditional QA (evaluating meaning, not just structure).
• Lead production monitoring of AI output quality and define acceptable output ranges for non-deterministic systems.
• Partner with engineering teams to validate AI-generated code, test cases, and AI-assisted decision workflows.
• Introduce AI guardrails and confidence scoring to flag unsafe or low-confidence outputs.
• Build validation checkpoints across multi-step agent pipelines.
• Define and track quality metrics/KPIs (defect reduction, output consistency, prompt regression stability, adoption confidence).