Pyspark Developer

PySpark Developer

Experience: 7+ Years
Location: Pune & Chennai
Joining: Immediate / 015 days preferred

Job Summary:

We are seeking a skilled PySpark Developer with 7+ years of experience in big data technologies. The ideal candidate should have strong expertise in PySpark, data processing, and building scalable data pipelines. You will be responsible for designing and optimizing large-scale data workflows in a distributed environment.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using PySpark
• Process and analyze large datasets in distributed environments
• Work with Apache Spark, Hadoop ecosystem, and big data tools
• Optimize data processing performance and ensure efficiency
• Develop ETL/ELT workflows for data ingestion and transformation
• Collaborate with data engineers, analysts, and stakeholders
• Ensure data quality, consistency, and governance
• Troubleshoot and resolve production issues in data pipelines
• Document technical designs, processes, and workflows

Required Skills:

• Strong hands-on experience in PySpark and Apache Spark
• Expertise in Python programming
• Experience with Hadoop ecosystem (HDFS, Hive, etc.)
• Strong SQL skills and data manipulation capabilities
• Experience in building ETL pipelines
• Knowledge of data warehousing concepts
• Familiarity with distributed computing and optimization techniques

Preferred Skills:

• Experience with Cloud platforms (AWS / Azure / GCP)
• Knowledge of Databricks or similar platforms
• Experience with Airflow, Kafka, or streaming technologies
• Exposure to Delta Lake or Lakehouse architecture
• Understanding of CI/CD pipelines and DevOps practices

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, IT, or related field
• Relevant certifications in Big Data or Cloud technologies are a plus