Senior Data Engineer

Job Purpose


Job Purpose We are seeking a highly skilled Python Cloud Platform Engineer to support, modernize, and migrate Python-based applications currently running on Linux servers with load balancers to containerized environments on Kubernetes, specifically Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS). Log extraction and RCA debug using Azure Data Explorer.
Key Roles

Support and maintain Python applications running on Linux environments

Manage load-balanced application architectures to ensure high availability and performance

Analyse existing Linux/VM-based workloads and plan migration to Kubernetes

Containerize Python applications using Docker

Deploy, manage, and scale applications on Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS)

Design and configure Kubernetes resources (Deployments, Services, Ingress, ConfigMaps, Secrets)

Perform application and infrastructure migration from on-prem / VM setups to AKS and EKS

Implement CI/CD pipelines for automated build and Kubernetes deployments

Monitor and troubleshoot application, container, and cluster-level issues

Optimize performance, scalability, and reliability in cloud-native environments

Implement security best practices for Kubernetes and cloud platforms

Duties and Responsibilities


Key Responsibilities

Translate business requirements into technical solutions in collaboration with the PMO team.

Own end-to-end delivery of data projects, ensuring on-time execution and adherence to quality standards.

Design technical architecture and guide development efforts for enhancements and new projects.

Develop and maintain robust ETL pipelines and data integration modules across systems.

Ensure high data quality, platform stability, and resolution of critical process issues.

Monitor and resolve performance bottlenecks in data workflows and programs.

Establish best practices, standard operating procedures, and drive their implementation across teams.

Act as a liaison with business users and product managers to support daily data needs and strategic initiatives.

Coordinate with internal and external development teams to troubleshoot and resolve issues efficiently.

Manage workload through effective planning, prioritization, and progress tracking

Key Decisions / Dimensions


Key Decisions / Dimensions::

Making critical decisions during production issues, including root cause analysis, quick fixes, and long-term resolutions.

Prioritizing and escalating support tasks effectively to minimize downtime and business impact.

Driving decisions around technical design, architecture, and optimization to ensure performance, scalability, and maintainability of solutions.

Major Challenges


Major Challenges

Balancing development timelines with on-time delivery in a dynamic and evolving technical environment.

Coordinating with multiple internal and external stakeholders to align priorities, resolve dependencies, and ensure smooth execution.

Ensuring code quality, data integrity, and performance while scaling data solutions across diverse systems and platforms.

Required Qualifications and Experience


Required Skills Experience

Strong experience with Python development (FastAPI, Pyspark) - Must Have

Strong Linux system administration skills. - Must Have

Hands-on experience with Docker and Kubernetes.

Proven experience with AKS and EKS.

Experience with load balancers and traffic routing.

Understanding of microservices architecture.

Experience with Git, CI/CD pipelines, and automation tools.

Knowledge of cloud networking concepts.

Azure Databricks - PySpark, SQL

Azure Data Explorer as Time Series Database with Kusto Query Language (KQL) As Programming Language - Good to Have

Soft Skills:

Strong problem-solving and analytical thinking

Good communication and stakeholder management

Ability to work independently and as part of a team in a fast-paced environment

Strong attention to detail and data integrity