Python, Spark Scala Developer

Educational Requirements Bachelor of Engineering

Service Line Data Analytics Unit

Responsibilities Data Engineering DevelopmentDesign, develop, and maintain scalable batch/stream data pipelines using Spark with Scala and Python.Implement efficient transformations, joins, aggregations, and data quality checks for large datasets.Build reusable frameworks/utilities to standardize pipeline patterns and reduce delivery time.Performance ReliabilityTune Spark jobs (partitioning, caching, shuffles, memory/executor settings) to improve performance and cost efficiency.Troubleshoot production issues, perform root-cause analysis, and implement preventive fixes.Ensure reliability through robust logging, monitoring hooks, and failure-handling strategies.Collaboration DeliveryWork with stakeholders to refine requirements and deliver well-documented, production-ready solutions.Conduct code reviews, enforce best practices, and mentor team members on Spark/Scala/Python patterns.Contribute to CI/CD-friendly development practices including testing, version control, and release readiness.

Additional Responsibilities: Bachelors degree in Computer Science, Engineering, or a related field (or equivalent practical experience).59 years of experience in data engineering or backend development with strong hands-on delivery ownership.Strong proficiency in Python and Apache Spark for large-scale data processing.Solid experience with Scala for Spark-based development and production-grade implementations.Working knowledge of Hive and SQL-based data querying/processing concepts.

Technical and Professional Requirements: Technology->Analytics - Packages->Python - Big Data,Technology->Big Data - Data Processing->Spark,Technology->Big Data - Data Processing->Scala

Preferred Skills: Technology->Analytics - Packages->Python - Big Data, Technology->Big Data - Data Processing->Spark