Weekend Drive-Virtual-Saturday

Strong expertise in SQL, PySpark, and Python to build scalable batch and distributed data pipelines.

• Ability to write and tune complex SQL for analytics on distributed query engines.
• Experience designing high-performance data processing using Spark SQL and DataFrames.
• Proficiency in developing Python-based automation scripts and reusable data engineering components.
• Hands-on experience with distributed SQL engines such as Starburst or similar technologies.
• Practical knowledge of lakehouse architecture using Delta Lake, Iceberg, or equivalent platforms.
• Experience working with cloud object storage such as S3, ADLS, or similar environments.
• Understanding of ETL/ELT processes and managed data services for ingestion and transformation.
• Solid knowledge of relational and non-relational databases as source systems.
• Ability to implement data quality checks, including validation, monitoring, and reconciliation.
• Experience in analytical data modeling, including dimensional and star schema design.
• Familiarity with Airflow, Autosys, or similar orchestration tools.
• Exposure to CI/CD, DevOps practices, and Git for collaborative pipeline development.
• Strong skills in debugging data issues, validating transformations, and optimizing performance.
• Proven problem-solving ability for complex data engineering challenges.
• Ability to leverage AI coding tools like GitHub Copilot, Devin, or Claude to improve productivity and code quality.