Lead Data Engineer, Databricks

Senior Data Engineer (Databricks + CosmosDB)

Location: Bangalore (Hybrid - 3 days from client office)

Experience: 8-10 Years

Job Summary

We are seeking a highly skilled Senior Data Engineer with deep expertise in Azure Databricks and Cosmos DB. The ideal candidate will be responsible for designing, building, and optimizing scalable, cloud-native data platforms and data pipelines. This role requires a hands-on approach to developing high-performance data solutions that support analytics, AI/ML initiatives, real-time processing, and enterprise reporting requirements while ensuring data quality, security, and operational excellence.

Key Responsibilities

1. Data Engineering & Pipeline Development

   • Design, develop, and maintain scalable batch and real-time data pipelines using Databricks and Apache Spark.
   • Build robust ETL/ELT processes to ingest, transform, and curate large volumes of structured and unstructured data.
   • Develop reusable data engineering frameworks and accelerators.
   • Optimize data processing jobs for performance, scalability, and cost efficiency.

2. Databricks Platform Development

   • Build and maintain data solutions using Azure Databricks.
   • Develop notebooks and workflows using Apache Spark RDD, Apache Spark SQL, and Python.
   • Implement Delta Lake architectures for reliable and efficient data management.
   • Configure and optimize Databricks clusters and workloads.
   • Support CI/CD implementation for Databricks deployments.

3. Cosmos DB Development & Optimization

   • Design and implement scalable solutions using Azure Cosmos DB (NoSQL).
   • Model data effectively for NoSQL workloads.
   • Optimize partition strategies, indexing policies, and query performance.
   • Manage data replication, consistency models, and throughput provisioning.
   • Integrate Cosmos DB with analytics and downstream systems.

4. Azure Data Platform Integration

   • Develop and integrate end-to-end solutions using Azure services such as Azure Data Factory, Azure Functions, and Azure Logic Apps.

5. Real-Time & Streaming Data Processing

   • Build streaming data pipelines using Spark Structured Streaming.
   • Process event-driven workloads from Azure Event Hubs, Kafka Stream, or IoT sources.
   • Implement near real-time analytics solutions integrated with Cosmos DB and Databricks.

6. DevOps & CI/CD

   • Implement Devsecops practices for Data Engineering solutions.
   • Automate deployment pipelines using Azure DevOps or GitHub Actions.
   • Manage source control and release processes.

Mandatory Skills

• Azure Databricks
• Apache Spark RDD
• Apache Spark SQL
• Delta Lake
• Azure Cosmos DB (NoSQL)

Preferred Skills

• Delta Live Tables (DLT)
• Azure Databricks

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 8-10 years of experience in data engineering or related roles.
• Strong understanding of cloud engineering principles and modern data architectures.
• Excellent problem-solving skills and ability to work in a fast-paced environment.