Experience
4 - 9 yrs
Job Location
Bengaluru, India
Vacancy
1
Designation
Pyspark Developer
Job Type
ONSITE
Job Description
Role Descriptions: AWS GLUE & PySpark Developer
Role Overview
We are looking for an experienced Senior AWS Glue & PySpark Developer to lead the ETL development and support function. This role requires strong technical leadership, design expertise, and the ability to mentor team members while ensuring scalable and reliable data integration solutions.
Experience: 4+ Years
Location: Bangalore
Required Skills & Qualifications
4+ years of experience in data engineering, with at least 3 years in AWS-based data solutions
Strong expertise in Python and PySpark for large-scale data processing
Hands-on experience with AWS Glue (jobs, crawlers, Data Catalog, bookmarks), Lambda, S3, IAM, and VPC networking
Advanced knowledge of PostgreSQL, including schema design, query optimization, indexing, and migration from SQL Server
Proven experience in building production-grade ETL/ELT pipelines with robust error handling, retry mechanisms, and data quality checks
Strong understanding of data warehousing principles, dimensional modeling, and reconciliation techniques
Experience with Git, branching strategies, and CI/CD pipelines
Excellent problem-solving, leadership, and communication skills
Key Responsibilities
Design and architect end-to-end data pipelines using AWS Glue, PySpark, Lambda, and PostgreSQL
Lead migration initiatives by translating legacy SSIS workflows (including configurations, connections, file paths, and schedules) into AWS-native solutions
Develop scalable and reusable PySpark frameworks for data transformation, validation, reconciliation, and error handling
Implement workflow orchestration using AWS Step Functions or similar tools for complex pipeline coordination
Drive database migration efforts from SQL Server to Aurora PostgreSQL, including schema design, data migration, and performance optimization
Establish monitoring, logging, and alerting mechanisms using CloudWatch, SNS, and custom dashboards
Mentor junior developers, conduct code reviews, and enforce coding standards and best practices
Collaborate with stakeholders to define requirements, plan sprints, estimate effort, and ensure timely project delivery
Preferred Qualifications
Experience migrating SSIS packages to AWS Glue or similar cloud ETL platforms
Domain expertise in financial services, insurance, or superannuation
Knowledge of AWS Lake Formation, Athena, or Redshift
Experience with Apache Airflow (MWAA) for orchestration
Familiarity with data governance and regulatory compliance requirements
Relevant AWS certifications (e.g., AWS Certified Data Analytics, Solutions Architect)
Skills: Digital : AWS Mobile Hub & AWS Mobile Services
Experience Required: 4+ years Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Role Overview
We are looking for an experienced Senior AWS Glue & PySpark Developer to lead the ETL development and support function. This role requires strong technical leadership, design expertise, and the ability to mentor team members while ensuring scalable and reliable data integration solutions.
Experience: 4+ Years
Location: Bangalore
Required Skills & Qualifications
4+ years of experience in data engineering, with at least 3 years in AWS-based data solutions
Strong expertise in Python and PySpark for large-scale data processing
Hands-on experience with AWS Glue (jobs, crawlers, Data Catalog, bookmarks), Lambda, S3, IAM, and VPC networking
Advanced knowledge of PostgreSQL, including schema design, query optimization, indexing, and migration from SQL Server
Proven experience in building production-grade ETL/ELT pipelines with robust error handling, retry mechanisms, and data quality checks
Strong understanding of data warehousing principles, dimensional modeling, and reconciliation techniques
Experience with Git, branching strategies, and CI/CD pipelines
Excellent problem-solving, leadership, and communication skills
Key Responsibilities
Design and architect end-to-end data pipelines using AWS Glue, PySpark, Lambda, and PostgreSQL
Lead migration initiatives by translating legacy SSIS workflows (including configurations, connections, file paths, and schedules) into AWS-native solutions
Develop scalable and reusable PySpark frameworks for data transformation, validation, reconciliation, and error handling
Implement workflow orchestration using AWS Step Functions or similar tools for complex pipeline coordination
Drive database migration efforts from SQL Server to Aurora PostgreSQL, including schema design, data migration, and performance optimization
Establish monitoring, logging, and alerting mechanisms using CloudWatch, SNS, and custom dashboards
Mentor junior developers, conduct code reviews, and enforce coding standards and best practices
Collaborate with stakeholders to define requirements, plan sprints, estimate effort, and ensure timely project delivery
Preferred Qualifications
Experience migrating SSIS packages to AWS Glue or similar cloud ETL platforms
Domain expertise in financial services, insurance, or superannuation
Knowledge of AWS Lake Formation, Athena, or Redshift
Experience with Apache Airflow (MWAA) for orchestration
Familiarity with data governance and regulatory compliance requirements
Relevant AWS certifications (e.g., AWS Certified Data Analytics, Solutions Architect)
Skills: Digital : AWS Mobile Hub & AWS Mobile Services
Experience Required: 4+ years Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.