Web scraping at scale presents significant challenges, especially when dealing with millions of hyperlinks per request and handling 10,000+ requests per minute. The system must efficiently crawl web pages, extract hyperlinks recursively, and store them while ensuring