Job Description
Your future duties and responsibilities
- Design scalable batch and streaming architectures
- Develop advanced Airflow DAGs with dynamic task mapping
- Build robust REST API ingestion pipelines
- Implement Kafka producers/consumers
- Write and optimize SQL/PLSQL queries and data models for transformations
- Ensure data quality, monitoring, and reliability
- Error handling, fault tolerance. Data triaging
- Ensure idempotent and incremental data loads
- Api rate limits and retries at scale
Required qualifications to be successful in this role
- 7-10+ years of experience in Data Engineering
- Strong Python programming
- Expertise in Apache Airflow, DAG Coding, Kafka
- Experience with REST API integrations
- Advanced SQL and PL/SQL knowledge
- Experience handling large-scale data pipelines
- Cloud Platforms (AWS/GCP/Azure)
Good to Have
- Oracle database knowledge
- Exposure to CI/CD pipelines and containerization (Jenkins/ Git/ Docker/Kubernetes)
- Exposure to data governance and compliance frameworks
Please share your resumes to silvia.k@cgi.com