Experience
3 - 7 yrs
Job Location
Pune, India
Vacancy
1
Designation
Python Automation Engineer
Job Type
Not specified
Job Description
Senior Quality Assurance Engineer - AI GenAI Applications We are seeking a Senior Quality Assurance Engineer to drive quality assurance for AI-powered chat applications, RAG-based knowledge retrieval systems, Agentic AI workflows, and intelligent support automation solutions. The ideal candidate will ensure the quality, reliability, accuracy, and performance of AI-driven products through a combination of manual and automated testing practices. Key Responsibilities
- Define and execute QA strategies for AI Assistant and Generative AI applications.
- Design and perform functional, integration, regression, API, and end-to-end testing.
- Validate chatbot responses for accuracy, relevance, context retention, and overall user experience.
- Test RAG pipelines, knowledge retrieval, citation accuracy, and response grounding.
- Validate Agentic AI workflows, tool integrations, task orchestration, and multi-step reasoning processes.
- Perform API testing and automation using Postman and REST services.
- Analyze application logs, traces, and production issues using Kibana, Elasticsearch, and observability tools.
- Monitor AI behavior, prompts, token usage, and response quality using Langfuse or similar LLM observability platforms.
- Develop and maintain AI test automation frameworks.
- Conduct performance, load, and reliability testing for AI applications.
- Identify defects, perform root cause analysis, and collaborate closely with developers for timely resolution.
- Ensure compliance with quality standards, release readiness criteria, and Definition of Done (DoD).
- Actively participate in Agile/Scrum ceremonies and quality planning activities.
Required Skills Experience
- 6+ years of experience in Software Quality Assurance and Test Automation.
- Hands-on experience testing AI Chatbots, Conversational AI, Generative AI, or RAG-based applications.
- Strong expertise in API testing, Postman, and test automation frameworks.
- Strong proficiency in Python and PyTest for test automation, API validation, data processing, and AI/LLM testing workflows.
- Experience with Langfuse, Kibana, Elasticsearch, and AI observability/monitoring tools.
- Good understanding of LLMs, Prompt Engineering, RAG architectures, Vector Databases, and Agentic AI concepts.
- Experience with Selenium, Playwright, Cucumber, or similar automation tools.
- Strong analytical, debugging, problem-solving, and communication skills.
- Experience working in Agile/Scrum-based enterprise product development environments.
Good to Have
- Knowledge of Fitness testing framework.
Education Bachelors or Masters degree in Computer Science, Engineering, Information Technology, or a related field, with proven experience in enterprise-scale QA delivery.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.