In this session, Rafael Levi from Bright Data discusses how to efficiently collect large-scale data using Large Language Models (LLMs) without incurring excessive token costs. He highlights the challenges of scraping websites protected by CAPTCHA and bot detection systems, such as Walmart and other aggressive marketplaces. Instead of directly parsing every page with an LLM, which is token-intensive and costly, he advocates building automated scraping pipelines using Bright Data’s tools. These pipelines leverage the company’s Managed Collection Platform (MCP) to extract HTML, identify selectors, and create scrapers that can run, maintain, and self-heal without constant manual intervention.
Levi demonstrates how Bright Data’s MCP integrates with LLMs to build scrapers quickly and efficiently. Using Cloud Code, he shows how an agent can be instructed to scrape product listings from websites like Walmart and Very.com by specifying search keywords and page limits. The MCP handles the complexities of anti-bot measures, including CAPTCHA solving and browser automation, enabling the scraper to access data that would otherwise be blocked. This approach not only saves significant tokens—up to 62% in some cases—but also reduces the time and effort traditionally required to build and maintain scrapers, which often took days or weeks.
The MCP offers a suite of tools including over 500 pre-built APIs for popular domains, remote browser infrastructure, and advanced anti-bot bypass technologies. It can fetch full HTML or simplified markdown versions of pages to optimize token usage further. Levi emphasizes that the MCP is particularly valuable for domains protected by heavy anti-bot systems like Akamai, Data Dome, and Cloudflare. The platform also supports scheduled scraping tasks and real-time monitoring, allowing users to set up listeners for personal or enterprise use cases, such as tracking apartment listings or booking restaurant reservations automatically.
Levi clarifies that Bright Data only deals with public data and respects website terms of service, avoiding scraping behind login walls or private data. He shares that the company has successfully defended its practices in court, reinforcing that public data remains accessible regardless of collection methods. Additionally, the platform supports interactive actions on websites, such as filling out forms and clicking buttons, mimicking human behavior with pre-recorded mouse movements and typing patterns to avoid detection. However, logging into accounts remains outside the scope due to privacy and legal considerations.
In conclusion, Rafael Levi presents Bright Data’s MCP as a powerful solution for building scalable, self-maintaining scraping pipelines that integrate seamlessly with LLMs. This approach drastically reduces token consumption and operational headaches associated with traditional scraping. He encourages attendees to leverage these tools for both large-scale data collection and personal automation tasks, emphasizing the importance of keeping public data accessible. Levi invites further discussion and offers support for anyone facing challenges in data access or scraping, underscoring the platform’s flexibility and robustness in navigating today’s complex web environments.
