Pyspark Developer

Role & responsibilities

Job Summary

We are seeking a skilled and motivated PySpark Engineer to design, develop, and optimize large-scale data processing solutions using PySpark and distributed computing technologies. The ideal candidate will have strong experience in building scalable ETL pipelines, working with big data platforms, and collaborating with cross-functional teams to deliver high-quality data solutions.

Roles & Responsibilities

• Design, develop, and maintain scalable data processing applications using PySpark.
• Build and optimize ETL/ELT pipelines for structured and unstructured data.
• Process large datasets efficiently using Apache Spark.
• Integrate data from relational databases, cloud storage, APIs, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Engineers, Data Scientists, Business Analysts, and Architects to deliver business solutions.
• Develop reusable frameworks and follow coding standards for data engineering projects.
• Perform data validation, data quality checks, and troubleshoot production issues.
• Monitor production jobs and resolve performance bottlenecks.
• Participate in code reviews and maintain technical documentation.
• Work closely with cross-functional teams throughout the software development lifecycle.

Preferred candidate profile

Required Skills

• 4-7 years of experience in Data Engineering.
• Minimum 3 years of hands-on experience in PySpark development.
• Strong Python programming skills.
• Hands-on experience with Apache Spark, Spark SQL, Spark DataFrames and RDDs.
• Strong SQL and ETL development experience.
• Experience with distributed computing concepts.
• Experience with AWS and/or Azure cloud platforms.
• Familiarity with Git version control.
• Experience with workflow orchestration tools like Airflow or Azure Data Factory.
• Good understanding of Data Warehousing concepts and Dimensional Modeling.
• Excellent analytical, debugging and problem-solving skills.

Preferred Skills

• Databricks
• Delta Lake
• Lakehouse Architecture
• Docker
• Kubernetes
• CI/CD Pipelines
• DevOps practices for Data Engineering

Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Location

• Hyderabad / Pune / Coimbatore

Notice Period

• Immediate to 30 days preferred.