Job Description
Job title : Lead Spark+Scala+Kafka engineer
Location : Bangalore
Years of experience : 8 to 15 Years
Joining time : Immediate to 30 Days
Key Responsibilities :
- Develop batch and real-time pipelines using Spark, Scala, and Kafka / Pub/Sub.
- Ingest, transform, and process large volumes of data.
- Work with GCP services such as: BigQuery, Cloud Storage, Cloud compressor, Dataflow/Dataproc
- Ensure data quality, performance, and reliability.
- Understand and manage downstream impacts of data loads (reporting, compliance, data quality).
- Follow best practices for scalable and maintainable data solutions.
- Collaborate with teams to deliver business-ready data platforms.
Required Skills & Experience :
- 8 to 15 years of experience in IT or professional services experience in IT delivery or large-scale IT analytics projects
- Mandatory programming experience in Scala
- Solid experience with Apache Spark.
- Experience with Kafka or Google Pub/Sub for streaming data.
- Experience with ETL and data integration tools (Dataflow, Spark, Dataprep, Python).
- Expertise in at least two of these technologies: Relational Databases, Analytical Databases, NoSQL databases.
- Implement data pipelines to automate the ingestion, transformation, and augmentation of data sources, and provide best practices for pipeline operations.
- Capability to work in a rapidly changing business environment and to enable simplified user access to massive data by building scalable data solutions
- Advanced SQL writing and experience in data mining (SQL, ETL, data warehouse, etc.) and using databases in a business environment with complex datasets
- Ability to work independently and guide teams in a fast- paced environment.
- Certified in Google Professional Data Engineer/ Solution Architect is a major Advantage
Good to Have :
- Strong hands-on experience as a Data Engineer.
- Experience with Python or Java
- Exposure to Machine Learning on GCP
- Google Professional Data Engineer or Solution Architect certification
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.