Azure Databricks Engineer

Azure Databricks Engineer

Location: Bangalore

Experience:6-9 Y

Job Summary:

We are seeking a highly skilled and experienced Azure Databricks Engineer tojoin our data engineering team. The ideal candidate will have deep expertise in building scalable data pipelines, optimizing big data workloads, and leveraging Azure cloud services to deliver high-performance analytics solutions. This role requires strong hands-on experience with Apache Spark, Databricks, and Azure Data Services, along with a solid understanding of data architecture and engineering best practices.

Key Responsibilities:

Design, develop, and maintain scalable data pipelines using Azure Databricks,
strong in Python, Pyspark and Apache Spark

Collaborate with data scientists, analysts, and business stakeholders to
understand data requirements and deliver robust solutions.

Optimize Spark jobs for performance and cost-efficiency in cloud environments.

Integrate data from various sources including Azure Data Lake, Azure SQL, Blob
Storage, and external APIs.

Implement data quality checks, monitoring, and alerting mechanisms.

Work with Delta Lake for ACID-compliant data storage and efficient data
versioning.

Automate workflows using Azure Data Factory, Airflow, or similar orchestration
tools.

Ensure security, governance, and compliance across data platforms.

Mentor junior engineers and contribute to best practices and standards.

Required Skills & Qualifications:

4- 5 years of experience in data engineering

Strong proficiency in Apache Spark, PySpark, and SQL.

Hands-on experience with Azure Databricks, Azure Data Lake, Azure Data Factory,
and Azure Synapse Analytics.

Solid understanding of Delta Lake, Parquet, and other big data formats.

Familiarity with CI/CD pipelines and DevOps practices in data engineering.

Excellent problem-solving and communication skills.