Experience
10 - 15 yrs
Job Location
Mangaluru, India
Vacancy
1
Designation
Senior Data Engineer
Job Type
ONSITE
Job Description
Role Summary
We are seeking a highly experienced Senior Data Engineer with strong expertise in Teradata Administration, Databricks, and AI/ML to support a Teradata Utilization Analysis engagement. The ideal candidate will analyze Teradata platform utilization using DBQL logs, metadata, and workload statistics to identify cost optimization opportunities and deliver scalable, log-driven analytical solutions using the Databricks platform. Candidates with Databricks Machine Learning or Databricks Generative AI Associate certification are highly preferred.
Key Responsibilities Teradata Utilization Analysis - Ingest, validate, and analyze 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and execution steps.
- Analyze Teradata system metadata and workload statistics to identify unused datasets, inactive partitions, and read-only data.
- Capture CPU, IO, and workload utilization metrics using Teradata ResUsage.
- Develop recommendations for dataset archival, storage optimization, and platform cost reduction.
- Classify datasets into hot, warm, and cold storage tiers based on usage patterns.
- Design and develop scalable data pipelines using Databricks and Apache Spark.
- Build reusable notebooks and workflows for log-driven analytics.
- Develop Delta Lake-based data pipelines for reliable and performant processing.
- Utilize Databricks SQL and Power BI to build dashboards, heatmaps, and analytical reports.
- Implement data governance using Unity Catalog.
- Build ML models to detect workload anomalies and predict cold-data candidates.
- Apply clustering and classification algorithms for dataset categorization.
- Develop feature engineering pipelines using time-series log data.
- Integrate LLMs for SQL log interpretation and automated recommendation generation.
- Track experiments using MLflow for reproducibility.
- Integrate metadata from Autosys, DataStage, MagicWand, and other enterprise systems.
- Assess ETL pipelines and recommend decommissioning of unused workflows.
- Support enterprise data modernization and cloud migration initiatives.
- Prepare Observation Reports, recommendation documents, workshop notes, and executive presentations.
- Present technical findings and cost optimization recommendations to customer stakeholders.
- Collaborate with architects, DBAs, data engineers, and business teams throughout the engagement.
- Bachelors or Masters degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
- 10+ years of relevant experience in Data Engineering, Teradata Administration, and Analytics.
- Strong experience in enterprise-scale data platforms and cloud-native data engineering.
- Experience delivering advisory or consulting engagements is preferred.
- Teradata Administration
- Teradata DBQL
- Teradata System Views
- Space Metadata Analysis
- Teradata SQL
- Performance Tuning
- ResUsage
- Workload Statistics
- BTEQ
- FastExport
- Teradata Parallel Transporter (TPT)
- Data Classification
- ETL Assessment
- Autosys
- DataStage
- Databricks Workspaces
- Databricks Jobs
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Databricks Notebooks
- Databricks Workflows
- Unity Catalog
- Databricks SQL
- Python
- Pandas
- NumPy
- SQL
- Plotly
- Matplotlib
- Power BI
- Machine Learning
- Scikit-learn
- MLflow
- Generative AI
- Large Language Models (LLMs)
- Feature Engineering
- Time-Series Analytics
- Clustering Algorithms
- Classification Algorithms
- Natural Language Processing (NLP)
- Experience with DataStage orchestration log parsing.
- Experience in cloud migration readiness assessments.
- Data platform cost optimization expertise.
- Healthcare payer domain experience with HIPAA awareness.
- Strong technical documentation and advisory reporting skills.
- Experience creating executive-level presentations and customer-facing deliverables.
- Databricks Machine Learning Associate (Highly Preferred)
- Databricks Generative AI Associate (Highly Preferred)
- Databricks Data Engineer Professional
- Azure Databricks Certification
- AWS Certified Data Engineer
- Microsoft Azure Data Engineer Associate
- Teradata Certification
- Data Engineering
- Teradata Administration
- Databricks Development
- Machine Learning
- Generative AI
- Data Platform Optimization
- Performance Tuning
- Data Analytics
- Solution Architecture
- Problem Solving
- Stakeholder Management
- Technical Consulting
- Executive Communication
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.