6 to 8 years of experience in Web Scraping or Crawling through Scrapy, Selenium or other frameworks and related libraries (like beautifulsoup, puppeteer).
Should be expert on latest version of Python.
Should have very good experience on fetching data from multiple online sources, cleanse it and build APIs on top of it.
Good understanding of data structure and algorithms, as well as how they affect system performance in real-world applications
Sound knowledge in bypassing Bot Detection techniques.
Web RestFul APIs / Microservices Development Experience.
Think deeply about developing large scale scraping tools including data integrity, health and monitoring systems.
Develop a deep understanding of our vast data sources on the web and know exactly how, when, and which data to scrape, parse and store.
Work with SQL and NoSQL databases to store raw data.
Develop frameworks for automating and maintaining constant flow of data from multiple sources.
Good knowledge of distributed technologies, real-time systems of high throughput, low latency, and highly scalable systems.
Work independently with little supervision to research and test innovative solutions.
Should have a strong passion for coding.
Must take quality, security and performance seriously.
Ability to pair with other engineers and cross-team as needed.
Excellent communication skills, including the ability to present effectively to both business and technical audiences at all levels of the organization.

Tech lead Web Crawling

Job Description