Data Engineer

Primary Responsibilities:

• Data Pipeline Development: Design, develop, and maintain efficient and scalable data pipelines using Python and Azure technologies
• Data Integration: Integrate data from various sources, ensuring data quality and consistency
• Azure Cloud Services: Utilize Azure cloud services for data storage, processing, and analytics, including Azure Data Factory, Azure Databricks, and Azure SQL Database
• CI/CD Implementation: Implement and manage CI/CD pipelines to automate the deployment of data solutions
• Data Modeling: Develop and maintain data models to support business intelligence and analytics needs
• Performance Optimization: Optimize data processing and storage solutions for performance and cost-efficiency
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs
• Documentation: Create and maintain comprehensive documentation for data pipelines, processes, and solutions
• Troubleshooting: Identify and resolve data-related issues and bottlenecks

Qualifications - External

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent in related field
• 7+ years of experience in data engineering, with a solid focus on Python, Azure, CI/CD, and deployment
• Proficiency in Python for data processing and automation
• Experience with Azure cloud services (Azure Data Factory, Azure Databricks, Azure SQL Database, etc.)
• Experience with containerization and orchestration tools (e.g., Docker, Kubernetes)
• Knowledge of SQL and NoSQL databases
• Familiarity with data warehousing concepts and technologies
• Solid understanding of CI/CD principles and tools (e.g., Jenkins, GitLab CI/CD)
• Proven excellent problem-solving and analytical skills
• Proven solid communication and collaboration abilities
• Proven ability to work independently and as part of a team

Proven detail-oriented with a focus on quality and accuracy