Job Description
Role Summary
The GenAI Specialist Rating evaluates AI-generated responses against defined quality rubrics
to determine whether responses meet gold-standard expectations. Specialists assess factual
accuracy, grounding, completeness, helpfulness, safety, and user satisfaction while providing
clear rationale for every decision.
Their work directly contributes to high-quality evaluation datasets used for model training,
benchmarking, and calibration.
Key Responsibilities
- Evaluate AI-generated responses using standardized evaluation rubrics
- Assess factual accuracy of geographic and local information
- Verify that responses are properly grounded in available map and place data
- Evaluate:
Factuality
Grounding
Helpfulness
Completeness
Relevance
Clarity
Safety
Instruction following
- Identify hallucinations and unsupported claims
- Distinguish between critical and minor defects
- Provide detailed justification for ratings
- Escalate ambiguous or policy-sensitive cases
- Maintain consistency across evaluation tasks
- Participate in calibration exercises
- Contribute feedback to improve evaluation guidelines
Core Skills
Generative AI Evaluation
- Understanding of LLM behavior
- Prompt-response evaluation
- Hallucination detection
- Response quality assessment
Analytical Thinking
- Critical reasoning
- Evidence-based decision making
- Attention to detail
- Pattern recognition
Maps Knowledge
- Geographic reasoning
- Navigation concepts
- POIs (Points of Interest)
- Local search behavior
- Route interpretation
Language Skills
- Excellent written English
- Reading comprehension
- Ability to interpret nuanced prompts
Secondary Skills
- Search verification techniques
- Knowledge of local businesses
- Travel and navigation terminology
- Familiarity with Maps products
- Basic data annotation experience
- Spreadsheet proficiency
Ancillary Skills [Good to know]
- SQL (basic)
- Annotation platforms
- Jira
- Documentation tools
- AI evaluation platforms
Qualifications
- Work Experience: 03 years in data annotation, content evaluation, technical writing,
or a related field. Experience with AI/LLM evaluation or editing is preferred.
- Educational Qualifications: Bachelor’s degree in Linguistics, English, Computer
Science, Humanities, or a related field, or equivalent practical experience.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.