PySpark Developer

Educational Requirements

• Bachelor of Engineering
• BTech
• BSc
• BCA
• MTech
• MSc
• MCA

Service Line Responsibilities

• Develop and maintain data pipelines using PySpark
• Process and analyze large-scale datasets in distributed environments
• Design and implement ETL/ELT workflows
• Optimize Spark jobs for performance and scalability
• Work with data stored in HDFS, Hive, or cloud storage (S3, ADLS)
• Collaborate with data engineers, analysts, and business teams
• Ensure data quality, integrity, and governance
• Debug and troubleshoot data processing issues
• Automate workflows using scheduling tools (Airflow, Oozie, etc.)
• Write clean, scalable, and efficient code

Technical and Professional Requirements

• Strong proficiency in Python and PySpark
• Good experience with Apache Spark (RDDs, DataFrames, Spark SQL)
• Knowledge of Hadoop ecosystem (HDFS, Hive)
• Experience in ETL pipeline development
• Familiarity with SQL and database concepts
• Experience with data formats (Parquet, ORC, JSON, CSV)
• Basic understanding of distributed computing concepts
• Exposure to version control tools (Git)

Technical and Professional Requirements Preferred Skills