Big Data Platform Engineer

Experience Requirements

8+ years of overall IT experience 5+ years of experience in Cloud Infrastructure Engineering 4+ years of experience working with Kubernetes platforms 3+ years of experience supporting enterprise-scale Data Platforms 3+ years of experience in DevOps/SRE practices 2+ years of experience working with Azure Data Services

Mandatory Skills

Candidates must possess strong hands-on experience in the following areas: Apache NiFi Azure Event Hub Azure Functions Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered.

1. Azure Kubernetes Service AKS

Strong Expertise

AKS cluster deployment and administration Cluster upgrades and patch management Node pool management Ingress Controllers, including NGINX and Application Gateway RBAC implementation AKS security hardening Autoscaling using HPA, VPA, and Cluster Autoscaler Persistent storage management Troubleshooting cluster issues Network policies

Experience

Production support of AKS environments Multi-node and multi-environment AKS setups High availability and disaster recovery implementation

2. Kubernetes and Azure Networking

Kubernetes Networking Kubernetes networking concepts Services, Ingress, and Load Balancers DNS resolution Pod-to-Pod communication Pod-to-Service communication Network Policies Service Mesh concepts, preferably Istio CNI Architecture

Azure Networking

Virtual Networks Subnets Private Endpoints Peering Network Security Groups User Defined Routes Azure Firewall VPN connectivity

3. Docker and Containerization

Docker image creation and optimization Container lifecycle management Registry management Security scanning Multi-stage builds Container troubleshooting Azure Container Registry Image versioning and governance

4. Azure DevOps and CI/CD

CI/CD Expertise Azure DevOps Pipelines YAML pipelines Build and Release pipelines Git repositories Branching strategies

Infrastructure Deployment

Infrastructure as Code ARM Templates Bicep Terraform

Automation

Deployment automation Environment promotion Release management

5. Monitoring and Observability

Azure Monitor

Monitoring strategy implementation Alert configuration Dashboard creation

Log Analytics

KQL query development Performance analysis Troubleshooting

Application Monitoring

Azure Application Insights Distributed tracing Metrics collection

AKS Monitoring

Cluster health monitoring Pod performance monitoring Resource optimization

6. Azure Event Hub Candidates must have strong hands-on experience in: Event Hub architecture Namespace management Consumer groups Scaling and throughput units Monitoring and troubleshooting Security and access control 7. Azure Functions Candidates must have hands-on experience in: Deployment and operations support Function App management Scaling configuration Managed Identity integration Monitoring and troubleshooting CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in: NiFi cluster setup and maintenance Flow deployment Security configuration Performance tuning Monitoring Troubleshooting Preferred NiFi Registry Cluster scaling High availability implementation 9. Infrastructure as Code Must-Have Skills Terraform Bicep ARM Templates

Required Capabilities

Platform provisioning Environment consistency Version-controlled infrastructure Good-to-Have Skills Platform Engineering GitOps using ArgoCD or Flux Platform automation Self-service platform concepts Internal Developer Platform Observability Grafana Prometheus OpenTelemetry Security Microsoft Defender for Cloud

Kubernetes security best practices Container security Secrets management Vulnerability management Data Platform Azure Data Factory Synapse Analytics Cloud Multi-region architecture Disaster recovery planning Landing Zone implementation Scripting PowerShell Bash Python

Key Responsibilities

Manage and support AKS-based Big Data platform infrastructure. Administer Apache NiFi, Azure Event Hub, and Azure Functions. Implement and maintain CI/CD pipelines using Azure DevOps. Monitor platform health using Azure Monitor and Log Analytics. Troubleshoot platform, networking, and infrastructure issues. Implement infrastructure automation through Terraform and Bicep. Ensure platform security, governance, and compliance. Collaborate with Data Engineering teams for platform enablement. Perform upgrades, patching, capacity planning, and optimization. Support production incidents and platform reliability initiatives.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.