Data Engineering Manager

Role & responsibilities

• Lead and manage the Data Engineering team responsible for building scalable, reliable, and high-performance data platforms and pipelines.
• Define and execute the data engineering strategy, technical roadmap, architecture, and delivery plans aligned with business objectives.
• Design and oversee scalable data pipelines, data warehouses, data lakes, lakehouse platforms, and data integration solutions.
• Drive development of batch and real-time data processing solutions using technologies such as Spark, Kafka, Airflow, Databricks, Snowflake, and cloud-native data services.
• Establish best practices for ETL/ELT, data modeling, data quality, data governance, data security, metadata management, and data observability.
• Partner with Data Science, Analytics, Product, Engineering, Business Intelligence, and Platform teams to deliver high-quality data solutions.
• Own the architecture and technical direction for cloud-based data platforms across AWS, Azure, or GCP.
• Drive modernization initiatives including cloud migration, lakehouse adoption, pipeline automation, and legacy data platform transformation.
• Ensure data platforms meet requirements for scalability, reliability, performance, security, availability, and cost efficiency.
• Establish standards for data architecture, coding practices, testing, CI/CD, deployment, monitoring, and operational excellence.
• Identify and resolve data pipeline failures, performance bottlenecks, data quality issues, and architectural challenges.
• Drive data quality, lineage, governance, privacy, and compliance initiatives in collaboration with security and governance teams.
• Optimize data infrastructure and workloads for performance and cloud cost efficiency.
• Evaluate emerging data technologies, tools, and frameworks and recommend solutions based on business and technical requirements.
• Manage engineering capacity, resource allocation, project priorities, technical dependencies, and delivery commitments.
• Hire, mentor, coach, and develop Data Engineers and Technical Leads to build a high-performing engineering organization.
• Conduct performance reviews, career planning, and technical mentoring for team members.
• Communicate technical strategy, project progress, risks, data platform metrics, and business impact to senior leadership.

Preferred candidate profile

• 5 to 10 years of experience in Data Engineering, Big Data, Data Platform Engineering, or related technology roles, with demonstrated leadership experience.
• Proven experience managing or leading Data Engineering teams and delivering large-scale data platforms.
• Strong expertise in Data Engineering, Data Architecture, Data Warehousing, Data Lakes, Data Modeling, ETL/ELT, and Data Pipelines.
• Strong hands-on experience with SQL and Python; experience with Scala or Java is an added advantage.
• Strong experience with Apache Spark, PySpark, Kafka, Airflow, Databricks, Snowflake, Hadoop, or equivalent data technologies.
• Experience designing and implementing batch processing, real-time/streaming data pipelines, distributed data processing, and data integration solutions.
• Strong knowledge of cloud data platforms such as AWS, Azure, or GCP.
• Experience with technologies such as AWS Glue, Redshift, EMR, Azure Data Factory, Azure Synapse, Google BigQuery, Dataflow, dbt, or equivalent platforms.
• Strong understanding of Data Warehouse, Data Lake, Data Lakehouse, Delta Lake, dimensional modeling, data governance, data quality, and metadata management.
• Experience implementing CI/CD, DevOps, Infrastructure as Code, data testing, monitoring, and DataOps practices.
• Strong understanding of data security, access control, privacy, compliance, data lineage, and governance.
• Proven experience optimizing data pipelines, distributed workloads, storage, query performance, and cloud infrastructure costs.
• Strong technical leadership skills with experience in architecture decisions, technical design reviews, code reviews, mentoring, and engineering best practices.
• Proven ability to hire, mentor, coach, and develop Data Engineers and Technical Leads.
• Strong stakeholder management, communication, problem-solving, decision-making, and cross-functional collaboration skills.
• Ability to work effectively with Data Scientists, BI/Analytics, Product, Engineering, Cloud, Security, and Business teams.
• Bachelor's degree in Computer Science, Information Technology, Data Engineering, Mathematics, Statistics, or a related technical discipline.
• B.E./B.Tech/MCA/M.Tech/M.Sc. or equivalent qualification preferred.
• Certifications such as AWS Data Engineer/Analytics, Azure Data Engineer, Google Professional Data Engineer, Databricks, or Snowflake are an added advantage.