AI LLM QA Engineer-Python

Job Summary

We are looking for a highly motivated AI/LLM QA Engineer with strong expertise in Python-based test automation and LLM evaluation techniques. The ideal candidate will have hands-on experience in testing AI/ML or LLM-based systems, ensuring quality, reliability, and performance of intelligent applications. You will play a key role in building AI Quality Engineering frameworks, validating LLM outputs, and integrating testing pipelines within modern cloud and orchestration ecosystems.

Key Responsibilities

• Design, develop, and execute automated test frameworks for AI/LLM systems using Python.
• Perform LLM evaluation and validation using: Semantic similarity, Embeddings-based scoring, Regression testing strategies for model outputs
• Build and maintain test suites using Pytest, TestNG, JUnit, Cucumber, or Behave.
• Conduct API testing and integration testing using libraries like boto3, requests, etc.
• Generate rich test reports using tools like pytest-html and enable parallel execution via pytest-xdist.
• Collaborate with data scientists and ML engineers to define evaluation benchmarks and test datasets.
• Implement end-to-end validation of AI workflows, including orchestration frameworks.
• Integrate tests into CI/CD pipelines (GitLab, Jenkins).
• Monitor system performance and quality metrics using tools like Grafana.
• Manage defect tracking, test cases, and reporting using Jira and X-Ray. Mandatory Skills Programming & Automation
• Strong hands-on experience with Python
• Experience with: o boto3, requests o pytest, pytest-html, pytest-xdist
• Knowledge of testing frameworks: o Pytest / TestNG / JUnit / Cucumber / Behave AI/LLM Testing Expertise
• Experience in Software QA/Testing for AI/ML or LLM-based systems
• Strong knowledge of: o LLM evaluation techniques (semantic similarity, embeddings) o Regression testing for AI outputs o Prompt/output validation Orchestration Frameworks
• Experience with at least one:
• LangChain / LangGraph
• LlamaIndex
• DSPy
• OpenAI Assistants / Actions
• Similar AI orchestration tools DevOps & QA Tools
• CI/CD Tools: GitLab, Jenkins • Monitoring: Grafana
• Test & Issue Tracking: Jira, X-Ray
• Exposure to AI Quality Engineering practices