Role Overview

Service Line: Data Analytics Unit

Responsibilities

Azure Databricks Platform Administration: Administer and manage Azure Databricks workspaces, clusters, and jobs
Configure and maintain Databricks environments (Dev, QA, Prod)
Monitor cluster performance, job execution, and system health
Manage notebooks, libraries, and dependencies
Perform upgrades, patches, and platform maintenance
Cluster Management Optimization: Design and manage cluster configurations (auto-scaling, instance types, pools)
Optimize job performance, resource utilization, and execution times
Troubleshoot cluster failures, memory issues, and job bottlenecks
Implement best practices for Spark performance tuning
Security Governance: Implement and enforce data security and access controls (RBAC, ACLs, IAM)
Manage workspace security, secrets (Key Vault integration), and credential passthrough
Ensure compliance with enterprise security and governance policies
Implement data governance, auditing, and lineage tracking
Azure Cloud Integration: Integrate Databricks with Azure services such as: Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, Azure Data Factory (ADF), Azure Synapse Analytics
Manage connectivity and networking (VNet, Private Endpoints, NSGs)

Educational Requirements

Technical and Professional Requirements

Core Skills1015 years of IT experience with strong Azure Databricks administration
Hands-on expertise in Apache Spark and Databricks platform management
Strong experience with Azure cloud services (ADLS, ADF, Key Vault, Networking)
Proven experience in production support and platform operations
Deep knowledge of cluster configuration, job scheduling, and performance tuning
Strong understanding of Spark architecture and optimization techniques
Experience with Terraform / ARM / Bicep (IaC tools)
Knowledge of security models, IAM, RBAC, and data governance
Familiarity with CI/CD tools and DevOps practices

Preferred Skills

Azure Databricks Platform Admin

Job Description