Lead Data Engineer

Key Responsibilities - Lead the development of scalable ETL/ELT pipelines and data models to process high-volume data. - Design and optimize data processing applications across platforms like Databricks, Snowflake, AWS, and Azure. - Build and manage data pipelines using Apache Airflow for orchestration and Airbyte for data integration. - Architect solutions for Change Data Capture, batch processing, and real-time/streaming data processing. - Develop and manage REST APIs for data ingestion and distribution to applications and users. - Leverage AI tools to enhance the development and deployment of data pipelines. - Oversee CI/CD pipelines and DevOps deployments using Docker, Kubernetes, and monitoring tools. - Mentor team members in best practices for data governance, quality, and security. Overview - As a Lead Data Engineer at our consulting firm, you will play a pivotal role in shaping the data architecture and engineering landscape. - This position requires an expert with multi-platform proficiency, particularly in Databricks, Snowflake, AWS, and Azure, to lead scalable data solutions. - You will work closely with cross-functional teams to integrate MLOps/DevOps practices and guide a team in delivering production-ready solutions. - The role demands innovation in solving complex data challenges using a diverse toolkit and requires strong leadership skills to mentor and develop team members. - Your expertise will help build robust, high-volume data processing pipelines and drive the adoption of AI-augmented deliverables. - The position is based in Bangalore and Gurgaon, catering to high-profile clients in the consulting sector. - You will be responsible for ensuring data governance, quality, and security while staying ahead of industry trends in MLOps and GenAI. - This role offers an opportunity to collaborate with data scientists and analysts to ensure data readiness for machine learning projects and contribute to the evolution of CI/CD pipelines. Requirements - 5+ years of experience in Data Engineering with expertise in data lakes and warehouses. - Proficiency in Change Data Capture (CDC), batch, and real-time data processing. - Experience with multiple cloud data platforms such as Databricks, Snowflake, AWS, or Azure. - Advanced SQL skills for large-scale data processing and analytics. - Strong programming skills in Python, with experience in asynchronous programming and threading. - Experience in creating and managing REST APIs using frameworks like FastAPI or Flask. - Hands-on experience with Apache Airflow for workflow orchestration and Airbyte for data integration.