Experience
8 - 10 yrs
Salary (CTC)
₹23.9L - ₹27.2L
Job Location
Hyderabad, India
Vacancy
1
Designation
Senior Big Data Engineer
Job Type
Not specified
Job Description
Mandatory
- Scala and Python
- Apache Spark (batch streaming) - must!
- Deep knowledge of HDFS internals and migration strategies.
- Experience with Apache Iceberg (or similar table formats like Delta Lake / Apache Hudi) for schema evolution, ACID transactions, and time travel.
- Running Spark and/or Flink jobs on Kubernetes (e.g., Spark-on-K8s operator, Flink-on-K8s).
- Experience with distributed blob storages like Ceph or AWS S3 and similar
- Building ingestion, transformation, and enrichment pipelines for large-scale datasets.
- Infrastructure-as-Code (Terraform, Helm) for provisioning data infrastructure.
- Ability to work independently while guiding juniors.
Essential functions
Responsibilities
- Design and implement data pipelines for migration from HDFS/Hive to cloud object storage (e.g., S3, Ceph).
- Optimize Spark (and optionally Flink) jobs for performance and scalability in a Kubernetes environment.
- Ensure data consistency, schema evolution, and governance with Apache Iceberg or equivalent table formats.
- Support migration strategy definition by providing technical input and identifying risks.
- Mentor junior developers and review their code / design decisions.
- Collaborate with platform engineers, cloud architects, and product stakeholders to align technical implementation with project goals.
- Troubleshoot complex distributed system issues in data pipelines or storage integration.
Qualifications
Mandatory
- Scala and Python
- Apache Spark (batch streaming) - must!
- Deep knowledge of HDFS internals and migration strategies.
- Experience with Apache Iceberg (or similar table formats like Delta Lake / Apache Hudi) for schema evolution, ACID transactions, and time travel.
- Running Spark and/or Flink jobs on Kubernetes (e.g., Spark-on-K8s operator, Flink-on-K8s).
- Experience with distributed blob storages like Ceph or AWS S3 and similar
- Building ingestion, transformation, and enrichment pipelines for large-scale datasets.
- Infrastructure-as-Code (Terraform, Helm) for provisioning data infrastructure.
- Ability to work independently while guiding juniors.
Would be a plus
- Experience with Apache Flink
- Prior experience in migration projects or large-scale data platform modernization.
- Apple experience preferred (to enable him/her to get up to speed on our tooling set quickly and more independently)
We offer
- Opportunity to work on bleeding-edge projects
- Work with a highly motivated and dedicated team
- Competitive salary
- Flexible schedule
- Benefits package - medical insurance, sports
- Corporate social events
- Professional development opportunities
- Well-equipped office