Gcp Data Engineer

Senior Data Engineer - GCP, Big Query, PySpark & Kafka
Location
Hyderabad (Hybrid)
Experience
5 to 13 Years
Role Overview
We are looking for a highly skilled Data Engineer with strong expertise in building scalable, cloud-native data platforms on Google Cloud Platform (GCP). The ideal candidate should have extensive hands-on experience with Python, PySpark, BigQuery, Kafka, DBT, Terraform, Kubernetes, SQL, CI/CD, and ETL pipeline development. The candidate will be responsible for designing, developing, optimizing, and supporting large-scale data processing solutions that drive business insights and analytics.
Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing frameworks on GCP.
• Build and optimize batch and real-time ETL/ELT pipelines using Python, PySpark, Kafka, and BigQuery.
• Develop high-performance data solutions for large-scale structured and unstructured datasets.
• Create and maintain data models using DBT and ensure adherence to data governance standards.
• Design streaming solutions using Kafka and event-driven architectures.
• Implement infrastructure as code (IaC) using Terraform for automated cloud provisioning.
• Deploy and manage containerized applications using Kubernetes.
• Build automated CI/CD pipelines for data platform deployments and releases.
• Optimize BigQuery performance, partitioning, clustering, and query tuning.
• Collaborate with data scientists, analysts, architects, and business stakeholders to build data products.
• Troubleshoot production issues and drive continuous improvements in platform reliability and performance.
• Ensure data quality, scalability, security, and operational excellence across the data ecosystem.

Must Have Skills
Data Engineering

• Python
• PySpark / Apache Spark
• Advanced SQL
• ETL / ELT Pipeline Development
• Data Warehousing Concepts
• Data Modeling

Google Cloud Platform (GCP)

• BigQuery
• Dataflow
• Pub/Sub
• Cloud Composer
• Cloud Storage (GCS)
• Cloud Functions (Good to Have)

Streaming & Messaging

• Apache Kafka
• Event-Driven Architecture
• Real-Time Data Processing

DevOps & Cloud Engineering

• Terraform
• Kubernetes
• Docker
• CI/CD Pipelines
• Git/GitHub/GitLab

Data Transformation

• DBT (Data Build Tool)

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field.
• 5-13 years of experience in Data Engineering and Cloud-based Data Platforms.
• Strong hands-on expertise in GCP ecosystem and BigQuery.
• Experience building enterprise-grade data platforms and large-scale distributed data processing systems.
• Strong understanding of cloud-native architectures and modern data engineering practices.
• Excellent problem-solving and analytical skills.
• Strong communication and stakeholder management capabilities.

Preferred Skills

• Experience with Data Lake and Lakehouse architectures.
• Knowledge of Airflow/Cloud Composer.
• Exposure to BI/Analytics platforms such as Tableau, Looker, or Power BI.
• Experience in Financial Services, Banking, Insurance, or Enterprise Data Platforms.
• Exposure to Data Governance and Data Quality tools.

What We Are Looking For

• Strong hands-on coding experience in Python and PySpark.
• Deep expertise in BigQuery performance optimization.
• Proven experience with Kafka-based streaming solutions.
• Strong Terraform and Kubernetes implementation experience.
• Experience building robust CI/CD pipelines for data platforms.
• Ability to design end-to-end cloud-native data solutions on GCP.