Job Description Engineering Manager (Databricks / Data Engineering)

Job Title

Engineering Manager Databricks / Data Engineering

Job Summary

We are seeking an experienced Engineering Manager to lead a team of Data Engineers in designing, developing, and delivering scalable data platforms on Databricks. The role involves driving technical excellence, people leadership, delivery management, stakeholder engagement, and continuous improvement while ensuring high-quality, secure, and reliable data solutions.

Key Responsibilities

Technical Leadership

Lead the design and implementation of scalable data platforms using Databricks.
Define technical architecture, coding standards, and engineering best practices.
Drive adoption of modern data engineering patterns including Medallion Architecture.
Review solution designs, code, and performance optimization strategies.
Ensure data security, governance, and compliance standards are followed.

Delivery Management

Own end-to-end delivery of multiple data engineering initiatives.
Plan sprint execution and manage project timelines.
Identify and mitigate delivery risks and dependencies.
Ensure production readiness and successful release management.
Drive continuous improvement in engineering processes.

People Management

Lead and mentor a team of Data Engineers and Technical Leads.
Conduct performance reviews and career development discussions.
Support hiring, onboarding, and capability development.
Foster a culture of collaboration, innovation, and accountability.

Stakeholder Management

Collaborate with Product Owners, Architects, Business stakeholders, and Platform teams.
Communicate delivery progress, risks, and technical decisions.
Manage stakeholder expectations and prioritize business requirements.

Operational Excellence

Ensure platform reliability, availability, and performance.
Drive root cause analysis for production incidents.
Improve monitoring, alerting, and observability.
Optimize cloud infrastructure and operational costs.

Required Skills

Databricks

Databricks Workspace
Delta Lake
Delta Live Tables (DLT)
Unity Catalog
Databricks Workflows
Auto Loader
Structured Streaming
MLflow (preferred)
Photon Engine
Databricks Asset Bundles (preferred)
AI Skills Nice to Have Genie & Cursor AI

Cloud Platforms

Microsoft Azure
AWS or Google Cloud Platform
Cloud Storage (ADLS, S3, GCS)

Data Engineering

Apache Spark (PySpark and Spark SQL)
Python
SQL
Data Warehousing
ETL/ELT Design
Medallion Architecture
Batch and Streaming Data Pipelines
Data Modeling (Star, Snowflake, Data Vault)

DevOps & CI/CD

Git
Azure DevOps / GitHub
CI/CD Pipelines
Terraform (preferred)
Infrastructure as Code

Database Technologies

SQL Server
Oracle
Snowflake
PostgreSQL
NoSQL databases (preferred)

Leadership Skills

Engineering leadership
Team management
Agile delivery
Stakeholder management
Risk management
Conflict resolution
Coaching and mentoring
Resource planning
Budget and capacity planning

Preferred Experience

10 15+ years of experience in Data Engineering.
3 5+ years of experience leading engineering teams.
Hands-on experience with Databricks on Azure, AWS, or GCP.
Experience building enterprise-scale data platforms.
Strong understanding of cloud-native architectures and data governance.
Experience with performance tuning, cost optimization, and production support.

Nice to Have

Databricks Certified Data Engineer Professional
Azure Data Engineer Associate
Azure Solutions Architect
Experience with Apache Kafka or Event Hubs
Knowledge of AI/ML pipelines and Generative AI
Experience with Data Mesh or Data Fabric architectures