Gen AI Testing - Sr. Engineer

* 5 8 years in QA automation, with 1 3 years in GenAI / API-based testing.
Key Responsibilities:
* Develop and maintain automated evaluation pipelines.
* Implement evaluation scripts using Python frameworks (e.g., DeepEval, custom frameworks)
* Integrate LLM/Chatbot APIs and agent workflows into evaluation pipelines
* Execute dataset-driven evaluations and capture and process responses.
* Support manual test scenario execution and validation
* Assist in dataset creation and enrichment
* Generate evaluation reports, and logs
* Debug and troubleshoot execution issues.
* Enable CI/CD integration for continuous evaluation.

Key Skills :
Core GenAI Evaluation Skills:
* Experience with evaluation frameworks (e.g., DeepEval or Arize)
* Understanding of LLM-as-a-Judge (G-Eval) methodology
* Strong prompt engineering and evaluation design skills
* Experience in manual evaluation of LLM outputs

Test Strategy
JD for GenAI Evaluation QA

Technical Skills:
* Strong programming in Python
* Experience in API testing and integration
* Proficiency in JSON handling, parsing, and data processing
* Automation framework development/integration.
* Knowledge of logging, reporting, and debugging tools

Agent Manual Testing & Dataset Skills:
* Experience in:
o Test scenario creation for GenAI use cases
o Manual validation of LLM responses (qualitative assessment)
o Dataset creation and curation
o Writing expected outputs or golden answers.
* Ability to design edge cases, negative scenarios and adversarial inputs (prompt injection, jailbreaks)

Domain & QA Skills:
* Strong foundation in software testing principles:
o Functional, integration, regression testing
* Experience in test design, defect tracking, and reporting.
* Strong analytical and problem-solving skills.
* Conversational AI testing experience.
* Understanding of AI agent behavior, workflows, and edge cases.