Big Data Engineer

Location: Pune, India
Experience: 6 to 8 Years

UST is a leading digital transformation solutions company that partners with global enterprises to drive innovation and business value. We are looking for an experienced Big Data Engineer to join our growing Data & Analytics team and play a key role in building scalable data platforms and pipelines that power analytics and business insights.

We are seeking a highly motivated Big Data Engineer with 6 to 8 years of experience in designing, building, and maintaining large-scale data processing solutions. The ideal candidate should possess strong expertise in distributed data processing frameworks, cloud technologies, and modern data engineering practices. This role requires an individual contributor who can take ownership of data pipelines, optimize performance, and collaborate with cross-functional teams to deliver reliable and scalable data solutions.

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build robust solutions for ingesting, transforming, and processing large-scale structured datasets.
• Ensure data quality, reliability, scalability, and availability across the data platform.
• Collaborate with Product Managers, Architects, and Engineering teams to define and implement technical solutions.
• Participate in code reviews, testing, CI/CD, and deployment activities to maintain engineering excellence.
• Own data platform components and drive end-to-end delivery and support.
• Troubleshoot and resolve performance bottlenecks within data pipelines and distributed systems.
• Continuously improve data engineering practices, development workflows, and platform capabilities.
• Leverage modern AI-assisted and agentic development tools to improve productivity, testing, and code quality.
• Mentor junior team members and contribute as a senior individual contributor across multiple product initiatives.

• 6 to 8 years of hands-on experience in Big Data or Data Engineering.
• Strong understanding of distributed storage and processing architectures including Data Lake and Lakehouse concepts.
• Proficiency in Java or Scala (preferred); strong Python experience is also acceptable.
• Extensive experience with Apache Spark for distributed data processing.
• Hands-on experience with workflow orchestration tools such as Apache Airflow or equivalent.
• Experience working with Databricks for large-scale data engineering workloads.
• Strong experience with AWS services, including:
  • Amazon S3
  • AWS Glue
  • Amazon EMR
• Ability to write clean, optimized, maintainable, and scalable code.
• Experience utilizing AI-powered coding or agentic development tools within the software development lifecycle.
• Strong understanding of:
  • Data Structures
  • Algorithms
  • Object-Oriented Programming (OOP)

• Apache Spark
• Databricks
• Data Lake / Lakehouse Architecture
• Batch & Streaming Data Processing

• Java
• Scala
• Python

• AWS
  • S3
  • Glue
  • EMR

• Apache Airflow
• Dagster (Good to Have)

• Parquet
• Avro
• ORC
• JSON

• Git
• CI/CD Tools
• Claude Code or equivalent AI-assisted coding tools

• Docker
• Terraform (Good to Have)

• Exposure to modern data platform architectures.
• Experience implementing data quality and monitoring frameworks.
• Knowledge of cloud-native data engineering best practices.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work in a fast-paced, collaborative environment.