Job Description

About Us : We are a dynamic and innovative team specializing in building scalable and efficient web scraping solutions for complex e-commerce platforms and diverse web environments. Our team is proficient in a wide range of technologies and advanced web scraping techniques, ensuring high-quality data extraction and delivery Job Summary: We are seeking a talented Web Scraping Engineer / Data Extraction Specialist to join our growing team. The ideal candidate will have a strong background in web scraping, data extraction, and backend technologies. You will be responsible for designing, developing, and maintaining robust web scraping solutions, handling dynamic content, and overcoming anti-crawling measures. Responsibilities: Develop and maintain scalable web scraping scripts using Python, JavaScript, and related frameworks (e.g., Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js). Implement advanced web scraping techniques, including API interception, sitemap parsing, and handling dynamic content. Design and build data pipelines for efficient data extraction, processing, and storage. Manage and optimize data extraction workflows, ensuring high speed and accuracy. Implement anti-crawling solutions, including IP rotation, proxy management, and CAPTCHA bypassing. Collaborate with cross-functional teams to gather requirements and deliver complex data solutions. Utilize backend technologies such as Flask, FastAPI, Django, Node.js, Spring Boot, and relational databases (PostgreSQL, MySQL) for data storage and API development. Work with cloud platforms like Azure and leverage services such as AzureML and ADLS GEN2. Employ data processing techniques using libraries like NumPy and Pandas. Use tools like Postman, MITM, and DevTools for API testing and network traffic analysis. Apply machine learning and NLP techniques for data analysis and processing (e.g., sentiment analysis, content classification). Set up and manage server-side scraping environments. Monitor and troubleshoot scraping scripts to ensure optimal performance. Required Skills: Strong proficiency in Python and/or JavaScript. Experience with web scraping libraries and frameworks (Scrapy, Selenium, Puppeteer, Beautiful Soup, Cheerio.js). Knowledge of backend technologies (Flask, FastAPI, Django, Node.js, Spring Boot). Experience with relational databases (PostgreSQL, MySQL). Understanding of HTTP/HTTPS protocols and API communication. Familiarity with cloud platforms (Azure). Ability to handle dynamic content and JavaScript-heavy websites. Experience with anti-crawling techniques (IP rotation, proxies, CAPTCHA bypassing). Data processing and analysis skills (NumPy, Pandas). Experience with API testing tools (Postman). Knowledge of machine learning and NLP concepts is a plus. Strong problem-solving and debugging skills. Excellent communication and collaboration skills. Experience: 3+ years of professional experience in web scraping or related fields. Education: Bachelor's degree in Computer Science, Engineering, or a related field