Gcp Data Engineer

Role & responsibilities

Job Overview

We are seeking a skilled GCP Data Engineer to design, build, and optimize our data pipelines and architecture on Google Cloud Platform. In this role, you will work with cross-functional teams to build scalable, real-time, and batch data processing systems using modern GCP tools. You will be responsible for transforming raw data into actionable insights for downstream analytics and machine learning applications.

Key Responsibilities

• Pipeline Design & Development: Build, deploy, and maintain scalable batch and real-time ETL/ELT pipelines using GCP Dataflow (Apache Beam) and Apache Airflow (Cloud Composer).
• Data Warehousing: Design, optimize, and manage large-scale data warehouses and data marts in Google BigQuery, ensuring high performance and cost efficiency through effective partitioning, clustering, and SQL optimization.
• Stream & Event Processing: Ingest and process high-throughput real-time streaming data using Google Cloud Pub/Sub.
• Big Data Processing: Configure, manage, and execute distributed data processing workloads using Google Cloud Dataproc (Apache Spark/Hadoop).
• Data Preparation & Quality: Clean, transform, and wrangle complex structured and unstructured datasets using Google Cloud Dataprep and custom Python scripts.
• Code & Query Development: Write efficient, maintainable Python code and complex SQL queries for data ingestion, transformation, and validation.
• Optimization & Monitoring: Monitor pipeline performance, troubleshoot failure points, and optimize execution time and cloud infrastructure costs.
• Collaboration: Partner with Data Analysts, Data Scientists, and Software Engineers to understand data requirements and deliver business-ready datasets.

Required Qualifications & Skills

• Hands-on GCP Experience: Solid experience building end-to-end data pipelines using Google Cloud Platform native tools.
• Core Languages: Advanced proficiency in Python and standard ANSI/Google Standard SQL.
• Data Warehousing: Expertise in BigQuery architecture, query optimization, view creation, and cost management.
• Orchestration & ETL: Strong experience orchestrating workflows using Apache Airflow and building transformations with Dataflow or Dataprep.
• Big Data & Streaming: Proven experience handling streaming architectures with Pub/Sub and big data cluster processing via Dataproc.
• Data Modeling: Solid understanding of relational database design, data modeling concepts (Star Schema, Snowflake Schema), and data lake architectures.

Preferred candidate profile

• Google Cloud Certified Professional Data Engineer certification.
• Knowledge of DevOps principles, CI/CD pipelines, and Infrastructure as Code (Terraform).
• Experience with Git, Docker, and Linux environments.