Experience
5 - 8 yrs
Job Location
Noida, India
Vacancy
1
Designation
Lead Data Engineer
Job Type
ONSITE
Job Description
Snowflake Migration Leadership: Led all phases of migration including but not limited to planning to optimization, ensuring a smooth transition with data integrity and minimal service disruption
Data Pipeline Development & Orchestration:
Lead the design and implementation of scalable ETL/data pipelines using Python and Luigi for data processing
Ensure efficient data processing for high-volume clickstream, demographics, and business data
Data Pipeline Development & Orchestration:
Lead the design and implementation of scalable ETL/data pipelines using Python and Luigi for data processing
Ensure efficient data processing for high-volume clickstream, demographics, and business data
Cloud Infrastructure Management:
Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security
Support data storage and retrieval workflows using S3 and SQL-based storage solutions
Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security
Support data storage and retrieval workflows using S3 and SQL-based storage solutions
Legacy System Maintenance & Upgrade Initiatives:
Oversee legacy framework maintenance, identify improvement areas, and propose cloud migration or modernization plans
Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints
Oversee legacy framework maintenance, identify improvement areas, and propose cloud migration or modernization plans
Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints
Monitoring & Troubleshooting:
Develop monitoring solutions to track system health, performance, and pipeline accuracy
Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues
Develop monitoring solutions to track system health, performance, and pipeline accuracy
Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues
Documentation & Knowledge Sharing:
Document workflows, troubleshooting procedures, and code for system transparency and continuity
Provide mentoring and training to team members on best practices and technical skills
Document workflows, troubleshooting procedures, and code for system transparency and continuity
Provide mentoring and training to team members on best practices and technical skills
Minimum Qualifications:
Experience: 5-8 years of experience in data engineering, DevOps, or a related technical field
Snowflake Expertise: 3+ years of hands-on experience on Snowflake
Candidates must have hands-on experience with Snowflakes core functionalities, including designing database schemas, efficiently loading data (using features like Snowpipe or the COPY into command), optimizing query performance, and managing user roles and access within Snowflake
Python: Strong Python skills for scripting migration processes, custom ETL, and API interactions with Snowflake
Programming & Scripting: Strong programming skills in Python and Linux Bash for automation and data workflows
Framework Proficiency: Hands-on experience with Luigi for orchestrating complex data workflows
Data Processing & Storage: Expertise in Hadoop ecosystem tools and managing SQL databases for data storage and query optimization
AWS Cloud Services: In-depth knowledge of AWS EC2, S3, RDS, and EMR to deploy and manage data solutions
Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines
Communication & Leadership: Proven ability to lead projects, communicate with stakeholders, and guide junior team members
Additional Experience Desired:
Snowflake Expertise: 3+ years of hands-on experience on Snowflake
Candidates must have hands-on experience with Snowflakes core functionalities, including designing database schemas, efficiently loading data (using features like Snowpipe or the COPY into command), optimizing query performance, and managing user roles and access within Snowflake
Python: Strong Python skills for scripting migration processes, custom ETL, and API interactions with Snowflake
Programming & Scripting: Strong programming skills in Python and Linux Bash for automation and data workflows
Framework Proficiency: Hands-on experience with Luigi for orchestrating complex data workflows
Data Processing & Storage: Expertise in Hadoop ecosystem tools and managing SQL databases for data storage and query optimization
AWS Cloud Services: In-depth knowledge of AWS EC2, S3, RDS, and EMR to deploy and manage data solutions
Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines
Communication & Leadership: Proven ability to lead projects, communicate with stakeholders, and guide junior team members
Additional Experience Desired:
Migration Tools & Strategies: Experience with specific data integration and ETL tools like Fivetran or Matillion, which are often used for efficient data loading and transformation into Snowflake, would be a significant advantage
Knowledge of custom scripting for migration is also valuable
Data Governance in Snowflake: Understanding how to apply data governance, security protocols, and compliance frameworks within Snowflake is highly desirable
This includes knowledge of features like dynamic data masking, row-access policies, and external tokenization
CI/CD for Data Pipelines: Experience with CI/CD practices specifically for data pipelines, including automated testing and deployment of Snowflake objects (eg, using tools like dbt), would be a plus
Security Practices: Understanding of data security practices, data governance, and compliance for secure data processing
Automation & CI/CD: Familiarity with CI/CD tools to support automation of deployment and testing
Big Data Technologies: Knowledge of big data processing tools like Spark, Hive, or related AWS services
Advanced Analytics: Background in analytics or data science to contribute to more data-driven decision-making
Cross-Functional Collaboration: Experience collaborating with non-technical teams on business goals and technical solutions
Knowledge of custom scripting for migration is also valuable
Data Governance in Snowflake: Understanding how to apply data governance, security protocols, and compliance frameworks within Snowflake is highly desirable
This includes knowledge of features like dynamic data masking, row-access policies, and external tokenization
CI/CD for Data Pipelines: Experience with CI/CD practices specifically for data pipelines, including automated testing and deployment of Snowflake objects (eg, using tools like dbt), would be a plus
Security Practices: Understanding of data security practices, data governance, and compliance for secure data processing
Automation & CI/CD: Familiarity with CI/CD tools to support automation of deployment and testing
Big Data Technologies: Knowledge of big data processing tools like Spark, Hive, or related AWS services
Advanced Analytics: Background in analytics or data science to contribute to more data-driven decision-making
Cross-Functional Collaboration: Experience collaborating with non-technical teams on business goals and technical solutions
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.