Pyspark & Azure (5 locations)

JOB LOCATION: Bengaluru, Chennai, Pune, Kolkata, Gurugram

Key Requirements:

• Strong proficiency in Python and PySpark
• Experience working on Spark SQL, RDD, and DataFrame APIs
• Good understanding of Hadoop ecosystem (Hive, HDFS, YARN)
• Knowledge of data formats like Parquet, Avro, JSON, etc
• Experience with SQL and writing efficient queries
• Familiarity with job orchestration tools (Airflow, Oozie, or similar)
• Version control systems like Git
• Exposure to Azure is mandatory

Key Responsibilities:

• Design, develop, and maintain robust ETL/ELT pipelines using PySpark and other big data technologies. Optimize Spark jobs for performance and scalability
• Implement data transformations, aggregations, and joins over large datasets. Perform batch and real-time data processing tasks
• Collaborate with data scientists, analysts, and other engineers to understand requirements and deliver quality solutions. Integrate PySpark solutions with data warehouses (like Hive, Redshift, Snowflake) and other data stores
• Write clean, maintainable, and well-documented code. Follow version control and CI/CD practices using tools like Git, Jenkins, or Azure DevOps
• Troubleshoot data quality and performance issues. Monitor pipeline health and ensure SLAs are met
• Work with tools and platforms like Hadoop, Hive, HDFS, AWS EMR, Databricks, or Azure Synapse as required.