Lead Snowflake Data Engineer

Job title: Lead Snowflake Data Engineer

Primary Skill: Snowflake (Snow SQL, Snow PLSQL and Snowpark, SNowflake DBT)

Location: Coimbatore

Key Skills:

Main Technology Focus

• Snowflake with strong hands-on experience inDBT
• AWS, Python, SQL, PySpark
• Data engineering set-up with strong programming skills

Requirements:

• Looking for engineer for information warehouse
• Warehouse is based on AWS/Azure, DBT, Snowflake.
• Strong programming experience with Python.
• Experience with workflow management tools like Argo/Oozie/Airflow.
• Experience in Snowflake modelling - roles, schema, databases
• Experience in data Modeling (Data Vault).
• Experience in design and development of data transformation pipelines using the DBT framework.
• Experience with Cloud providers, preferably AWS
• The warehouse must be scalable, secure and well performant. The solution should be reusable and support future use cases.
• This will be used as storage layer for the platform
• Essential part of the warehouse is integration with monitoring and observability capabilities.

Responsibilities:

• Snowflake Data Modeling: Design and implement scalable Snowflake data models, optimized for data ingestion and analytics requirements.
• ETL Pipeline Development: Build and maintain robust ETL pipelines to integrate data from multiple sources into Snowflake, ensuring data integrity and consistency.
• Performance Optimization: Optimize Snowflake usage and storage, tuning query performance and managing data partitions to ensure quick, reliable access to data.
• Data Security & Governance: Implement best practices in data security, role-based access control, and data masking within Snowflake to maintain compliance and data governance standards.
• Automation & Workflow Management: Utilize tools such as dbt and Apache Airflow to schedule data processing and automate pipeline monitoring.
• Collaboration & Troubleshooting: Partner with data scientists, business analysts, and other stakeholders to address complex data challenges and troubleshoot Snowflake-related issues effectively.
• Documentation & Reporting: Develop comprehensive documentation for data structures, ETL workflows, and system processes to ensure transparency and knowledge sharing within the team.