Design, develop, and maintain robust web scraping pipelines using Python frameworks such as Scrapy, Selenium, Playwright, and Beautiful Soup. Implement headless browser automation and proxy rotation to extract structured and unstructured data at scale.
Data Processing & Analysis
Build ETL/ELT workflows with pandas, NumPy, and Dask for efficient data cleansing, transformation, and normalization. Integrate API-based data sources (REST, GraphQL) and schedule batch or real-time ingestion via Apache Airflow or Prefect.
Insight Generation & Visualization
Leverage Python libraries (Matplotlib, Seaborn, Plotly) and BI tools (Tableau, Power BI) to convert raw data into actionable dashboards. Produce executive-ready reports highlighting key trends, anomalies, and opportunities.
Data Intelligence & Machine Learning
Collaborate with data science teams to deploy simple ML models (scikit-learn, XGBoost) for classification, clustering, and predictive analytics. Implement NLP pipelines (spaCy, NLTK) to extract semantics from text-based web content.
AI-Driven Automation
Explore AI-powered scraping and data enrichment services (Diffbot, Import.io with AI modules) and leverage OpenAI APIs or LangChain for intelligent data parsing, summarization, and anomaly detection.
Performance & Reliability
Ensure scraper resiliency by handling anti-bot measures, CAPTCHAs, and dynamic content. Optimize code for concurrency (Asyncio, multiprocessing) and implement unit tests (pytest) and CI/CD pipelines (GitHub Actions, Jenkins).
Proof of Impact
Demonstrate successful delivery of high-volume scraping projects processing 5M+ records monthly with 99% data accuracy and generation of insights that led to a 25% improvement in decision-making speed.

Required Qualifications

Bachelor s degree in Computer Science, Data Science, or related field with fluency in English communication
Proficiency in Python, web frameworks (Flask, FastAPI), and SQL/noSQL databases (PostgreSQL, MongoDB)
Familiarity with cloud environments (AWS Lambda, EC2, S3; Azure Functions) and containerization (Docker, Kubernetes)
Strong problem-solving skills, attention to detail, and ability to translate complex data challenges into scalable solutions

Preferred: Experience with graph databases (Neo4j), vector databases (Pinecone), and real-time streaming (Kafka, AWS Kinesis).

Python Developer

Job Description

No Referrers Available