Job Description
We are looking for a skilled Python + PySpark Developer with strong expertise in data engineering, ETL development, and cloud-based data processing. The ideal candidate should have hands-on experience building scalable data pipelines using Python, PySpark, and AWS services while ensuring high performance, data quality, and reliability.
Key Responsibilities :
- Design, develop, deploy, and maintain scalable ETL and data processing pipelines using Python and PySpark.
- Build and optimize batch and distributed data processing workflows for large-scale structured and unstructured datasets.
- Develop complex SQL queries, stored procedures, and data transformation logic to support analytics and business reporting.
- Design and implement data migration and integration solutions across multiple platforms using AWS services such as Glue, Lambda, and S3.
- Perform data cleansing, validation, transformation, and quality assurance to ensure data integrity.
- Optimize PySpark applications for performance, scalability, and efficient resource utilization.
- Collaborate with data engineers, data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
- Develop reusable frameworks, utilities, and automation scripts to improve data engineering processes.
- Write unit tests and ensure adherence to coding standards, version control, and CI/CD best practices.
- Troubleshoot production issues, perform root cause analysis, and implement preventive measures.
- Maintain technical documentation and support deployment and release activities.
Required Skills & Experience :
- 5- 7 years of experience in Python development and Data Engineering.
- Strong proficiency in Python programming.
- Hands-on experience with PySpark for distributed data processing.
- Strong knowledge of Pandas and NumPy for data manipulation and analysis.
- Expertise in SQL with experience writing complex queries and optimizing database performance.
- Experience building ETL/ELT pipelines on AWS using services such as AWS Glue, Lambda, S3, and IAM.
- Good understanding of distributed computing concepts and Spark architecture.
- Experience with relational databases and data warehousing concepts.
- Familiarity with Git, CI/CD pipelines, and Agile development methodologies.
- Strong analytical, debugging, and problem-solving skills.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.