PySpark Developer (5-10 yrs)

Role Overview :

We are looking for a skilled PySpark Developer to design, develop, and optimize large-scale data processing solutions. The ideal candidate will have strong experience in PySpark, distributed data processing, and ETL development, with the ability to build scalable and high-performance data pipelines that support analytics, reporting, and business intelligence initiatives.

Key Responsibilities :

- Design, develop, and maintain scalable data pipelines using PySpark and Apache Spark.

- Build and optimize ETL/ELT processes for processing large volumes of structured and unstructured data.

- Develop data ingestion, transformation, validation, and enrichment workflows across multiple data sources.

- Work with Hadoop ecosystem components and distributed computing frameworks to process large-scale datasets efficiently.

- Optimize Spark jobs for performance, scalability, reliability, and cost efficiency.

- Implement data quality checks, monitoring, and error-handling mechanisms within data pipelines.

- Collaborate with data engineers, data architects, analysts, and business stakeholders to understand requirements and deliver data solutions.

- Perform data analysis, troubleshooting, and root cause investigations for production issues.

- Ensure adherence to coding standards, data governance policies, and best practices.

- Support deployment, testing, and maintenance of data engineering solutions across development and production environments.

- Contribute to automation, performance tuning, and continuous improvement initiatives.

Required Skills & Experience :

- 5-10 years of experience in Data Engineering, Big Data Development, or related roles.

- Strong hands-on experience with PySpark and Apache Spark.

- Experience working with Hadoop ecosystem technologies.

- Strong knowledge of distributed systems and large-scale data processing frameworks.

- Experience designing and developing ETL/ELT pipelines.

- Proficiency in Python and SQL.

- Strong understanding of data modeling, data transformation, and data integration concepts.

- Experience working with relational and/or NoSQL databases.

- Familiarity with data warehousing and analytics platforms.

- Strong problem-solving, debugging, and performance optimization skills.

Preferred Skills :

- Experience with cloud platforms such as AWS, Azure, or GCP.

- Exposure to Databricks, EMR, or other Spark-based managed platforms.

- Knowledge of workflow orchestration tools such as Airflow, Oozie, or Azure Data Factory.

- Experience with Kafka or other streaming technologies.

- Familiarity with CI/CD practices and DevOps concepts for data engineering.

- Experience working in Agile development environments.