Lead I - Data Engineering

We are seeking an experienced Azure Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to design, develop, and maintain scalable data pipelines and analytics solutions. The ideal candidate will work closely with business stakeholders, data scientists, and engineering teams to deliver high-quality data integration and transformation solutions on the Azure cloud platform.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory.
• Build scalable data processing solutions using Azure Databricks and Apache Spark.
• Ingest data from multiple sources including databases, APIs, flat files, and cloud storage.
• Develop and maintain data transformation workflows using PySpark, Spark SQL, and SQL.
• Implement data quality checks, monitoring, and error-handling mechanisms.
• Integrate data solutions with Azure services such as:
  • Azure Data Lake Storage
  • Azure Synapse Analytics
  • Azure SQL Database
• Optimize Spark jobs for performance and cost efficiency.
• Collaborate with data architects and business teams to understand requirements and deliver solutions.
• Implement CI/CD pipelines and deployment automation.
• Ensure security, governance, and compliance standards are followed.

Required Skills

• Strong experience with:
  • Azure Databricks
  • Azure Data Factory
  • Apache Spark
  • PySpark
  • SQL
• Experience with Azure cloud services.
• Knowledge of ETL/ELT concepts and data warehousing.
• Experience working with large-scale structured and unstructured datasets.
• Understanding of Delta Lake architecture.
• Experience with Git-based version control.
• Familiarity with REST APIs and data integration patterns.