Experience
9 - 14 yrs
Salary (CTC)
₹2,500,000 - ₹4,000,000
Job Location
Not specified
Vacancy
4
Designation
Python Developer Lead
Job Type
ONSITE
Job Description
About the Role
We are seeking an experienced Python Lead with deep expertise in Scrapy and large-scale web data extraction solutions. The ideal candidate will lead the design, development, and deployment of high-performance web scraping frameworks, mentor engineering teams, and drive data collection initiatives that support business intelligence and AI/ML applications.
Key Responsibilities
- Lead the development and maintenance of scalable web scraping solutions using Python and Scrapy.
- Design and implement distributed crawling architectures for high-volume data extraction.
- Build robust data pipelines for collecting, processing, and storing structured and unstructured web data.
- Optimize crawlers for performance, reliability, and scalability.
- Handle anti-bot mechanisms, rate limiting, CAPTCHAs, proxies, and dynamic website scraping.
- Develop APIs and backend services to support data collection workflows.
- Collaborate with Data Engineering, AI/ML, and Product teams to understand data requirements.
- Ensure adherence to coding standards, best practices, and security guidelines.
- Conduct code reviews, mentor junior developers, and provide technical leadership.
- Troubleshoot and resolve production issues related to web scraping infrastructure.
Required Skills
- 8+ years of experience in Python development.
- 4+ years of hands-on experience with Scrapy Framework.
- Strong knowledge of web scraping, crawling, and data extraction techniques.
- Expertise in Python libraries such as:
- Scrapy
- Requests
- BeautifulSoup
- Selenium / Playwright
- Pandas
- Strong understanding of:
- HTML, CSS, JavaScript
- REST APIs
- XPath and CSS Selectors
- JSON/XML data processing
- Experience with databases:
- PostgreSQL
- MySQL
- MongoDB
- Redis
- Experience working with cloud platforms (AWS/Azure/GCP).
- Proficiency in Git, CI/CD pipelines, and Docker.
Preferred Qualifications
- Experience with distributed scraping frameworks and large-scale data collection systems.
- Exposure to AI/ML data preparation pipelines.
- Knowledge of message queues such as RabbitMQ or Kafka.
Nice to Have
- Experience with LLMs, Generative AI, or data collection for AI model training.
- Exposure to Computer Vision or NLP data acquisition projects.
- Contributions to open-source scraping or data engineering projects.
No Referrers Available
There are currently no referrers available for this job. You can still apply, will let you know once there is any referrer available.