Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
Implement data validation and quality checks to ensure accuracy and consistency.
Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency .
Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
Provide technical guidance and support to team members and stakeholders as needed.

Desirable Competencies:

8+ years of work experience.
Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
Experience in Databricks and Delta tables.
Data engineering experience with Spark/Scala/Python.
Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
Experience in Azure DevOps and CI/CD pipelines.
Working knowledge of Python.
Comfortable participating in 2-week sprint development cycles.

Sr. Data Engineer

Job Description