Senior Data Engineer

Senior Data Engineer

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Python & PySpark
• Develop and optimize ETL/ELT workflows for structured and semi-structured data
• Work with Azure data services to ingest, process, and store large datasets
• Write efficient SQL queries for data analysis, transformation, and reporting
• Collaborate with data analysts, data scientists, and business teams to deliver reliable datasets
• Implement data quality checks, monitoring, and performance tuning
• Handle batch and (if applicable) streaming data workloads
• Follow best practices for data security, governance, and documentation

Mandatory Skills

• Python PySpark
• Azure

Required Skills

Programming & Data Processing

• Strong hands-on experience with Python
• Solid expertise in PySpark / Apache Spark
• Experience with data transformations, joins, aggregations, window functions

Databases & SQL

• Advanced SQL (CTEs, indexing, query optimization)
• Experience with relational and analytical databases

Cloud & Azure

• Hands-on experience with Microsoft Azure, including:
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage (ADLS)
  • Azure Synapse Analytics
  • Azure Databricks
• Understanding of cloud-based data architecture

Data Engineering Concepts

• ETL vs ELT
• Data modeling (Star / Snowflake schemas)
• Partitioning, caching, performance tuning
• Handling large-scale datasets

Good to Have

• Experience with streaming tools (Kafka, Event Hubs)
• Knowledge of CI/CD pipelines for data workflows
• Exposure to Airflow / Azure Data Factory scheduling
• Understanding of DevOps / Git / version control
• Basic knowledge of data governance & compliance

Education

• Bachelor s degree in Computer Science, Engineering, or related field(Equivalent practical experience is welcome)

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.