Big Data Engineer

Role & responsibilities

Must Have

- 6+ years data engineering experience

- Strong programming skills in Python

- Hands on experience with Big Data tech (e.g., Hadoop) and Spark

- Comfort working in Agile, highly collaborative teams

- Strong written and verbal communication skills

Good to have

- Experience with cloud platforms (AWS EMR, S3 Lamda, or Azure), Terraform (provisioning and Infra), Scala, Java, Databricks

- Performance tuning and code optimization experience

Must Have

8+yrs

- PySpark: Experience with Apache Spark using the PySpark for distributed data processing, including DataFrame operations, window functions, and Spark SQL.

- Python Programming: Strong knowledge of Python, including advanced features and best practices.

- AWS Services: Familiarity with AWS S3 (for data storage and access), EMR, Lambda, SQS, SNS, IAM etc and Boto3 (AWS SDK for Python).

- Data Engineering: Understanding of ETL processes, schema design, and data validation.

- SQL: Proficiency in writing and optimizing SQL queries, especially within Spark SQL

- Strong analytical and Problem solving skills

Preferred / Ideal to have

- Experience with other AWS services - SQS, SNS, IAM etc

- Infrastructure as Code (IaC) with tools like Terraform

- Experience with Git and branching strategies and Code quality tools (SonarQube)

- RESTful API development and documentation (e.g., Swagger/OpenAPI/Fast API)

- Data pipeline orchestration (Airflow, Step Functions)

- Scripting for automation (Bash, Python, PowerShell)

Good to have –

- Any knowledge on Databricks is added advantage

- Data Quality and Validation Frameworks

WFO all 5days

Bangalore/Hyderabad

F2F interview

6+yrs

Immediate to 15days only

Mok@teksystems.com