Data Engineer (Web ScrapingCrawling)

Job Title: Web Crawling Engineer

Experience:3-5 Years

Location: Hyderabad

We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks anti-blocking techniques and proxy management along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

• Design develop and maintain scalable and efficient web crawlers for data extraction.
• Work with Scrapy Requests and other Python libraries to implement crawling solutions.
• Handle anti-scraping measures including IP blocks rate limits and captchas.
• Implement and manage proxy rotation and session management strategies.
• Ensure high-quality and structured data extraction cleaning and storage.
• Monitor crawler performance troubleshoot issues and optimize for reliability and efficiency.
• Collaborate with the data engineering team to integrate crawled data into pipelines.
• Stay updated with the latest tools techniques and best practices in web crawling.

Required Skills & Qualifications

• Strong programming skills in Python.
• Proven experience with Scrapy Requests BeautifulSoup Selenium (if required).
• Hands-on experience solving blocking issues handling
• Good understanding of proxy usage rotation and fingerprinting avoidance techniques.
• Knowledge of HTTP cookies headers sessions and request/response cycles.
• Experience with databases (SQL/NoSQL) for storing crawled data.
• Strong debugging problem-solving and analytical skills.

Good to Have

• Experience with cloud environments (AWS GCP Azure).
• Knowledge of data pipelines and ETL processes.

Education & Experience

• Bachelors degree in Computer Science Information Technology or equivalent practical experience.
• 3 years of experience in web crawling or related fields

OSI Digital was founded in 1993 in California and now has offices in the US Canada India Philippines Dubai and Australia. Our client base ranges from start-ups to Fortune 500 corporations including Barnes & Noble College Hyundai Toyota Gilead Allergan Yamaha Planview Globe and Sketchers. OSI Digital has developed strong partner alliances with the worlds leading technology providers such as Salesforce Boomi Tableau Oracle Microsoft and Amazon Web Services.