Job Description
Main Roles and Responsibilities :
- Design, develop, and implement ETL pipelines for the continuous deployment and integration.
- Collaborate with other data engineers and data scientists to understand data requirements and optimize data solution processes.
- Assure that data is cleansed, mapped, transformed, and otherwise optimised for storage and use according to business and technical requirements
- The ability to automate tasks and deploy production standard code (with unit testing, continuous integration, versioning etc.)
- Load transformed data into storage and reporting structures in destinations including data warehouse, high speed indexes, real-time reporting systems and analytics applications
- Build data pipelines to collectively bring data together
- Other responsibilities include extracting data, troubleshooting and maintaining the data warehouse
- Implement best practices for pipeline building and governance.
- Troubleshoot and resolve issues related to pipeline deployment and performance.
- Ensure compliance with security and data privacy standards
- Solution design using Microsoft Azure services and other tools
Required skills and experience :
- 2-3 years of experience in Azure Data Analytics, Synapse, DevOps.
- Proficiency in programming languages such as Python, SQL, Py-Spark, Spark-SQL.
- Experience with Azure: ADLS, Synapse, Stream Analytics, SQL DW, Analysis Services, Serverless Architecture, ARM Templates
- Experience with cloud platforms like Azure, AWS or and their respective services.
- Knowledge of CI/CD pipelines, automation tools and version control systems like Git.
- Strong problem-solving skills and ability to troubleshoot complex issues.
- Experience with monitoring tools.
- Ability to work collaboratively in cross-functional teams.
Nice to have:
- Should have Data Engineer certification from any discipline - Azure, AWS.
- Azure Synapse Studio, Data bricks, Fabric
- Familiarity with agile software development lifecycle (SCRUM, Kanban, etc.)
Location - Hyderabad,Ahmedabad,pune,chennai,kolkata.