Python-Pyspark Developer

Educational RequirementsBachelor of Engineering,BTech,BCA,BSc,MTech,MSc,MCAService Line Data Analytics UnitResponsibilities

• We are looking for an experienced Python PySpark Developer to design, develop, and optimize large-scale data processing systems. The ideal candidate will work on big data platforms, build scalable ETL pipelines, and process high-volume datasets using Spark and Python.
• Data Engineering Development
• Develop and maintain data pipelines using Python and PySpark
• Process and transform large datasets in distributed environments
• Build scalable ETL/ELT workflows
• Big Data Processing
• Work with Apache Spark (PySpark) for batch and real-time processing
• Optimize Spark jobs for performance and efficiency
• Handle structured and unstructured data
• Data Integration
• Ingest data from multiple sources: Databases (SQL/NoSQL) APIs Files (CSV, JSON, Parquet)
• Integrate with data platforms like:Hadoop (HDFS) Cloud (AWS, Azure, GCP)
• Performance Optimization
• Tune Spark jobs (partitioning, caching, parallelism)
• Optimize SQL queries and transformations
• Improve data processing efficiency and cost
• Collaboration
• Support
• Work with data engineers, data scientists, and analysts
• Translate business requirements into technical solutions
• Participate in code reviews and agile development practices
• Monitoring Troubleshooting
• Debug and resolve issues in data pipelines
• Monitor job execution and data quality
• Ensure reliability and availability of data workflows

Technical and Professional Requirements: Primary skills:Python, PysparkPreferred Skills: Technology->OpenSystem->Python - OpenSystem->Python Technology->Big Data - Data Processing->PySpark