Databricks Platform Engineer

Staff Databricks Platform Engineer
Scientific Games
AI, Data & Infrastructure

Job Summary

Scientific Games is seeking an experienced Staff Databricks Platform Engineer to design, build, and optimize enterprise-scale data engineering solutions on the Databricks Lakehouse Platform running on AWS. This role will develop scalable data pipelines, reusable engineering frameworks, and production-ready Data Products that support business-critical analytics and operational data services across the enterprise. Working closely with the Principal Enterprise Data Architect and Principal Analytics Engineer, the Staff Databricks Platform Engineer transforms enterprise architecture into scalable, secure, and maintainable engineering solutions while promoting software engineering excellence and modern data engineering practices.

This is a highly hands-on Staff-level technical leadership role. The successful candidate is expected to spend most of their time designing, developing, reviewing, optimizing, and mentoring while remaining deeply involved in implementing the organization's most complex Databricks engineering solutions.

Success in this role requires exceptional technical depth, strong software engineering skills, and the ability to influence engineering standards through technical expertise and collaboration.

Scope

Owns the engineering implementation of the enterprise Data Platform built on the Databricks Lakehouse Platform running on AWS.

Success is measured by:

• Scalable and reusable Data Products
  • Engineering quality and software craftsmanship
  • Platform scalability, performance, and reliability
  • Reusable engineering frameworks
  • Developer productivity
  • Reliable delivery of production solutions

Job Duties / Key Accountabilities

Databricks Platform Engineering

• Design, develop, and optimize scalable enterprise data pipelines using Databricks, Spark, Delta Lake, SQL, and Python.
  • Build and maintain Medallion Architecture (Bronze, Silver, Gold).
  • Develop reusable ingestion frameworks supporting batch, streaming, APIs, and Change Data Capture (CDC).
  • Engineer scalable Data Products supporting analytics, self-service, and AI.
  • Optimize Spark workloads for performance, scalability, reliability, and cost.

Software Engineering

• Develop clean, modular, maintainable, and testable software.
  • Lead code reviews and promote software engineering best practices.
  • Implement automated testing, CI/CD, and engineering quality standards.
  • Build reusable engineering frameworks and shared libraries.

Data Engineering

• Build production-grade ETL/ELT pipelines.
  • Implement Structured Streaming, Auto Loader, and event-driven engineering patterns.
  • Develop automated validation, reconciliation, and data quality capabilities.
  • Troubleshoot complex production engineering issues.

Performance & Platform Optimization

• Optimize Spark execution plans, Delta Lake performance, partitioning, and storage.
  • Improve observability through logging, monitoring, diagnostics, and metrics.
  • Continuously improve platform performance and cost efficiency.

Technical Leadership

• Serve as the Staff-level technical expert for Databricks engineering.
  • Lead the design and implementation of the organization's most complex Databricks engineering solutions.
  • Mentor engineers and provide technical leadership through design reviews and architectural collaboration.
  • Partner with the Principal Enterprise Data Architect and Principal Data Engineer.
  • Participate in technical planning, estimation, backlog refinement, and sprint execution.

Qualifications / Skills / Knowledge

Required

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related discipline.
  • 8+ years designing and building enterprise data engineering solutions.
  • 5+ years hands-on Databricks Lakehouse Platform experience.
  • Expert Python, SQL, Spark, and PySpark.
  • Deep expertise with Delta Lake, Unity Catalog, and Medallion Architecture.
  • Experience building scalable Data Products and reusable engineering frameworks.
  • Strong AWS experience.
  • Strong software engineering practices including Git, testing, CI/CD, design patterns, and code reviews.
  • Strong Spark performance tuning experience.
  • Demonstrated technical leadership and mentoring experience.

Desired

• Databricks Certified Data Engineer Professional.
  • Experience with Delta Live Tables, Databricks Workflows, Databricks Asset Bundles, and Terraform.
  • Experience with Infrastructure as Code and engineering observability.
  • Experience in Gaming, Lottery, Financial Services, or other regulated industries.

Authority / Decision Making

Authority To

• Define engineering implementation patterns aligned with approved architecture.
  • Establish coding standards and reusable engineering frameworks.
  • Review engineering designs and code quality.
  • Drive engineering best practices across Databricks development.

Requires Approval For

• Architectural changes outside approved enterprise standards.
  • Platform investments beyond approved technology strategy.
  • Technology adoption outside approved engineering standards.

Key Contacts

Internal

• Head of AI, Data & Infrastructure
  • Principal Enterprise Data Architect
  • Principal Data Engineer
  • Platform Engineering
  • Product Leadership
  • Data Governance
  • Data Science & ML

External

• Databricks
  • AWS
  • Strategic Technology Partners
  • Engineering Communities

Language Skills

Required: English
Desired: Additional languages considered an asset

Job Conditions.
• Regular collaboration with engineering teams across India and North America.
• Flexible work schedule with recurring overlap during North American business hours.
• Occasional international travel ( up to 10%).
• Participation in engineering design sessions, architecture reviews, sprint planning, and technical workshops.