Data Engineer_Azure

Responsibilities

• Execute Legacy-to-Cloud Migrations: Lead the end-to-end migration of complex datasets from on-premise legacy systems (including Mainframe and Informatica-based workflows) to the Azure.
• Develop Databricks Pipelines: Build and optimize high-throughput ETL/ELT pipelines using PySpark and Delta Live Tables (DLT) to ensure data consistency and reliability.
• Integrate Hybrid Workflows: Map and translate legacy logic (Informatica mappings and Mainframe COBOL/Copybooks) into modern, code-based transformations within Azure Data Factory (ADF) and Databricks.
• Performance Tuning & MLOps: Monitor and tune Databricks clusters for cost-efficiency and performance; implement CI/CD pipelines via Azure DevOps to automate data deployments.

Required Technology stack

• Azure (Data Factory, ADLS Gen2, Synapse, Azure DevOps), Azure Databricks, Apache Spark (PySpark/Scala), Delta Lake, Advanced Python, SQL, and Shell Scripting, ADF Triggers, Databricks Workflows, or Airflow

Good to have

• Informatica PowerCenter (mappings/workflows), Mainframe (DB2, VSAM, JCL).

Experience & Qualifications

• 8+ years of experience in Data Engineering.
• Bachelors or Masters Degree in Computer Science, Information Systems, or a related field.
• Proven track record of a full-cycle migrations from on-premise environments to Azure.
• Expertise in Databricks SQL and Spark core, specifically focusing on performance optimization of large-scale joins and aggregations.
• Azure Certification: AZ-204 (Developer) or AZ-305 (Solutions Architect) is a significant advantage