Hiring For Azure Data Engineer (Databricks & ADF)

Job Description

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Databricks.• Develop data transformation and processing logic using PySpark, Python, and Spark SQL.• Build and maintain Databricks notebooks, workflows, and Spark-based data processing jobs.• Develop and manage Azure Data Factory pipelines, including activities, parameters, triggers, and dependencies.• Perform data cleansing, transformation, validation, and integration from multiple data sources.• Optimize PySpark and Spark SQL jobs for performance, scalability, and efficient resource utilization.• Troubleshoot data pipeline failures and production issues and implement appropriate fixes.• Work with large datasets and ensure data quality, accuracy, and reliability.• Use Azure DevOps for source-code management, branching, pull requests, and CI/CD processes.• Follow coding standards, best practices, and development processes.• Collaborate with business analysts, data engineers, developers, and other stakeholders to understand requirements and deliver data solutions.• Prepare technical documentation for pipelines, transformations, and data processes.• Support deployment and production releases across different environments.

Qualifications

• Bachelor's or Master's degree in Engineering or Technology• 3-5 years of experience