Job Description
About the Role:
We are seeking a highly skilled AI/LLM QA Engineer with expertise in Agentic and Multi-Agent system testing to join our team. The ideal candidate will have a strong background in software QA/testing combined with hands-on experience in AI/ML or LLM-based applications. You will be responsible for validating the performance, accuracy, and robustness of next-generation AI systems, including orchestration-driven workflows and autonomous agents.
Key Responsibilities:
- Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
- Perform LLM evaluation using metrics such as:
1. Exact match / Soft match
2. BLEU / ROUGE
3. BERTScore
4. Semantic similarity using embeddings
- Develop and maintain prompt testing frameworks and validate output consistency across use cases.
- Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
- Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
- Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
- Identify defects, performance issues, and edge cases in AI systems.
- Integrate testing processes into CI/CD pipelines.
Mandatory Skills Core AI/LLM Testing:
- Hands-on experience in Agentic & Multi-Agent Testing
- Expertise in LLM evaluation frameworks and methodologies
- Strong understanding of:
1. Prompt engineering validation
2. Guardrails and safety mechanisms
3. Output evaluation using embeddings and NLP metrics
Software QA Experience:
- 3+ years in Software QA/Testing
- Minimum 2+ years of experience in AI/ML or LLM-based systems
Programming:
- Proficiency in Python OR TypeScript/JavaScript
Orchestration Frameworks - Experience with at least one:
- LangChain / LangGraph
- LlamaIndex
- DSPy
- OpenAI Assistants / Actions
- Or similar agentic orchestration frameworks
Tools & Platforms:
- CI/CD: GitLab, Jenkins
- Monitoring: Grafana
- Test Management & Tracking: Jira, X-Ray
- Exposure to AI Quality Engineering practices
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.