Educational Requirements

Bachelor of Engineering

Service Line

Data Analytics Unit

Responsibilities

Data Engineering DevelopmentDevelop and maintain batch/stream data processing jobs using Spark with Scala and/or Python.Build reusable components and utilities to standardize ingestion, transformation, and validation patterns.
Implement efficient transformations, joins, aggregations, and partitioning strategies for large-scale datasets.
Performance, Quality ReliabilityTune Spark jobs for performance (e.g., caching, shuffles, partition sizing) and troubleshoot failures or bottlenecks.
Write unit/integration tests and ensure code quality through reviews, refactoring, and documentation.
Monitor job runs, analyze logs/metrics, and drive continuous improvements in stability and runtime.
Data SQLUse SQL to validate data, reconcile outputs, and support downstream reporting/analytics needs.
Collaborate with stakeholders to clarify requirements and translate them into robust data processing logic.

Additional Responsibilities:

Bachelors degree in Computer Science, Engineering, or a related field.
25 years of hands-on experience building solutions with Python.
25 years of experience working with Apache Spark for large-scale data processing.
25 years of experience developing with Scala (or strong ability to work across Scala and Python codebases).
Working knowledge of SQL for querying, validation, and troubleshooting datasets.
Strong problem-solving skills and ability to collaborate effectively in a team environment.

Preferred Skills:

Python, Spark Scala Developer

Job Description