AI Quality Analyst

Job Title: AI Quality Analyst

Location: Pune

Experience: 410 Years

Role Overview

Ensure the quality, reliability, and robustness of AI/ML systems by designing and executing comprehensive testing strategies for non-deterministic models, including LLM-based applications.

• Proven experience in testing AI/ML-powered applications, with a strong understanding of how to validate non-deterministic systems.
• Hands-on experience with LLM testing tools such as RAGAs, DeepChecks, OpenAI Evals, or DeepEval is highly desirable.
• Expertise in creating test plans, designing test cases, and executing functional, performance, and regression tests for AI models, focusing on bias, accuracy, and robustness.
• Strong proficiency in Python and frameworks like PyTest for building automated test suites.
• Good experience testing RESTful APIs and backend services to ensure seamless data flow and integration.
• Familiarity with CI/CD pipelines (Jenkins, Kubernetes) and cloud platforms (GCP, AWS) is a major plus.
• A methodical approach to identifying, documenting, and tracking bugs in complex AI systems.

Key Responsibilities

• Design and execute tests for AI/ML and LLM-powered applications
• Validate non-deterministic systems for accuracy, bias, and robustness
• Create and maintain test plans, test cases, and automation suites
• Use tools like RAGAs, DeepChecks, OpenAI Evals, DeepEval
• Perform functional, regression, and performance testing
• Test REST APIs and backend integrations
• Integrate testing into CI/CD pipelines (Jenkins, Kubernetes)
• Identify, document, and track defects in AI systems

Required Qualification

• Proven experience in testing AI/ML applications
• Strong understanding of LLM testing and validation techniques
• Proficiency in Python and PyTest for automation
• Experience with API and backend testing
• Ability to evaluate bias, accuracy, and model performance
• Strong analytical and problem-solving skills

Preferred Qualifications

• Hands-on experience with LLM evaluation tools (RAGAs, DeepEval, etc.)
• Familiarity with GCP or AWS cloud platforms
• Experience with CI/CD tools (Jenkins, Kubernetes)
• Exposure to AI quality frameworks and metrics