Sr SDE-Spark

Unify Technologies is looking for Sr SDE-Spark to join our dynamic team and embark on a rewarding career journey

• We are seeking a highly skilled and experienced Senior Software Development Engineer with expertise in Apache Spark to join our dynamic engineering team
• The ideal candidate will have a strong background in big data processing, distributed computing, and software development
• As a Senior SDE specializing in Apache Spark, you will play a key role in designing, implementing, and optimizing data processing workflows to support our organization's data-driven initiatives

Responsibilities:

• Spark Application Development:Design, develop, and maintain Apache Spark applications for large-scale data processing
• Implement efficient data transformations, aggregations, and analytics using Spark RDDs and DataFrames
• Distributed Computing:Work on distributed computing frameworks and architectures to ensure scalability and performance of Spark applications
• Optimize Spark jobs for resource utilization and execution speed
• Data Processing Workflows:Design end-to-end data processing workflows, integrating Spark with other components of the data pipeline
• Collaborate with data engineers and data scientists to implement data processing requirements
• Optimization and Performance Tuning:Identify and address performance bottlenecks in Spark applications
• Implement optimizations to enhance the efficiency of data processing jobs
• Data Integration:Integrate Spark applications with various data sources and sinks, including databases, data lakes, and external APIs
• Ensure data quality and reliability in integration processes
• Monitoring and Debugging:Implement monitoring solutions to track the performance and health of Spark applications
• Debug and troubleshoot issues to ensure the reliability of data processing workflows
• Code Reviews and Collaboration:Conduct code reviews and provide constructive feedback to peers
• Collaborate with cross-functional teams, including data engineers, data scientists, and software developers
• Documentation:Maintain accurate and up-to-date documentation for Spark applications, workflows, and development processes
• Provide documentation for troubleshooting and knowledge transfer
• Requirements:Bachelor's degree in Computer Science, Information Technology, or a related field
• Master's degree is a plus
• Proven experience as a Software Development Engineer with a focus on Apache Spark
• In-depth knowledge of Spark architecture, internals, and best practices
• Proficiency in programming languages such as Scala or Python for Spark development
• Strong understanding of big data processing concepts and distributed computing
• Experience with Spark SQL, Spark Streaming, and related Spark libraries
• Familiarity with data pipeline orchestration tools (eg, Apache Airflow)
• Knowledge of data integration and ETL processes
• Strong problem-solving and analytical skills
• Excellent communication and collaboration skills
• Ability to work independently and contribute to a collaborative team environment