Job Description
We are looking for a highly skilled PySpark Developer with strong expertise in PySpark, Scala, and Big Data technologies to join our Data Engineering team.
The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions while supporting enterprise-wide data transformation initiatives.
The role requires hands-on experience in distributed computing, data engineering, ETL development, and analytics platforms to deliver scalable and high-performance data solutions.
Key Responsibilities :
Data Engineering & Development :
- Design, develop, and maintain scalable data processing applications using PySpark and Scala.
- Build robust ETL/ELT pipelines to process structured, semi-structured, and unstructured data.
- Develop reusable data ingestion, transformation, and validation frameworks.
- Optimize data workflows for performance, scalability, and reliability.
Big Data Processing :
- Work with large-scale datasets using Apache Spark and distributed computing frameworks.
- Develop Spark jobs for batch and real-time data processing.
- Implement data partitioning, caching, and optimization strategies to improve performance.
- Monitor and troubleshoot Spark applications in production environments.
Requirement Analysis & Solution Design :
- Collaborate with business stakeholders to understand data and reporting requirements.
- Translate business needs into technical specifications and functional designs.
- Participate in solution architecture discussions and provide technical recommendations.
- Prepare effort estimates, impact assessments, and implementation plans.
Research & Proof of Concepts (POCs) :
- Explore emerging technologies, frameworks, and tools relevant to data engineering.
- Conduct feasibility studies, literature reviews, and technology evaluations.
- Build Proof of Concepts (POCs) for innovative data solutions and modernization initiatives.
- Recommend best-fit solutions based on business and technical requirements.
Data Integration & ETL :
- Develop and maintain data integration solutions across multiple systems.
- Ensure data quality, consistency, and governance standards are met.
- Support data migration and transformation initiatives.
- Work with cloud and on-premises data platforms.
Production Support & Troubleshooting :
- Diagnose and resolve data processing failures and performance issues.
- Perform root cause analysis for production incidents.
- Ensure timely resolution of critical issues while maintaining service-level commitments.
- Support deployment and release management activities.
Agile Delivery :
- Participate in Agile ceremonies including sprint planning, stand-ups, reviews, and retrospectives.
- Collaborate with architects, business analysts, data scientists, QA teams, and stakeholders.
- Contribute to continuous improvement initiatives and knowledge-sharing activities.
Organizational Contributions:
- Support organizational initiatives, innovation programs, and capability-building efforts.
- Mentor junior team members and promote engineering best practices.
- Contribute to reusable frameworks, accelerators, and internal knowledge repositories.
Required Skills & Qualifications:
Technical Skills:
- Strong hands-on experience in PySpark and Scala.
- Expertise in Apache Spark architecture and distributed computing concepts.
- Experience with Big Data ecosystems such as Hadoop, Hive, HDFS, and Spark SQL.
- Strong knowledge of ETL development and data transformation processes.
- Proficiency in SQL and database technologies.
- Experience working with large-scale data processing environments.
- Knowledge of performance tuning and optimization techniques for Spark applications.
- Familiarity with version control tools such as Git.
Preferred Skills:
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP).
- Exposure to Databricks, Snowflake, Delta Lake, or Lakehouse architectures.
- Knowledge of Kafka, Airflow, or other data orchestration tools.
- Experience in data warehousing and data modeling concepts.
- Familiarity with CI/CD and DevOps practices.
- Exposure to machine learning pipelines and analytics platforms.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.