Job Description
Role & responsibilities
We are looking for a candidate with experience in Data Engineering Automation to help transform DAs Azure Databricks data platform into a highly autonomous, self-improving data engineering ecosystem.
The ideal candidate will possess skills in modern Data Engineering together with experience in creating, designing, orchestrating, managing, and continuously improving AI Agents capable of automating Data Engineering workflows.
This role goes significantly beyond only ETL. The engineer will develop intelligent AI Agent with the available ecosystem, capable of generating, validating, testing, optimizing, monitoring and improving Data Engineering pipelines with minimal human intervention.
The goal is to progressively increase automation while improving reliability and engineering productivity
Primary Responsibilities - Agentic Data Engineering:
Design AI Agents capable of automating:
- ETL, ELT
- Data Pipeline creation, SQL generation, PySpark generation
- Delta Live Tables creation
AI Agent Orchestration
Develop or use orchestration frameworks enabling multiple specialized agents to collaborate, including:
- Data Discovery Agent, Code Generation Agent, Testing Agent, Validation Agent, Optimization Agent
Leverage orchestration frameworks such as LangGraph, Azure AI Agent Service, MCP-compatible orchestration frameworks
Programming Python, SQL REST APIs + JSON, YAM, Git
Desired Experience
Candidates should have demonstrated experience in: Workflow automation, Data engineering, Agent orchestration, Evaluation harnesses Enterprise-scale automation, Multi-agent systems, LLM applications
Soft Skills Systems thinking, Strong analytical problem-solving, Strong communication with technical and business stakeholders, Continuous learning mindset, Ownership and accountability and Passion for automation and innovation
Preferred candidate profile