The video showcases building a robust brand monitoring app that scrapes user-facing AI platforms like ChatGPT and Google using Bright Data’s web scrapers, emphasizing real-world application over simple API calls and integrating asynchronous job management with Ingest for scalability and reliability. It also highlights best practices in context engineering, troubleshooting, and suggests future enhancements like scheduled scans and alerts to provide businesses with comprehensive, timely brand insights.
The video demonstrates how to build a valuable brand monitoring application using coding agents like Claude Code or Cursor, focusing on real business needs rather than toy projects. The app tracks mentions and sentiments of specified brands across multiple AI providers such as ChatGPT, Perplexity, Gemini, Grok, Co-Pilot, and Google search results. This is crucial because AI platforms are increasingly replacing traditional search engines, and brands need to know if they are being recommended or mentioned by these AI tools to maintain visibility and customer traffic.
A key insight shared is that directly calling API endpoints of these AI providers is insufficient for this application. Instead, the app scrapes the actual user-facing web applications to capture the authentic user experience, which often differs from API responses. This approach also accounts for geolocation-based variations in responses. The video introduces Bright Data’s AI-powered web scrapers as the solution for scraping these platforms, explaining how their synchronous and asynchronous scraping APIs work, including polling for results and handling snapshot IDs for longer scrapes.
The presenter walks through setting up the project using a Next.js boilerplate with React, SQLite, and Drizzle ORM, and integrating Bright Data’s scraping services by configuring API keys and dataset IDs. They emphasize the importance of context engineering when working with coding agents, providing detailed documentation and examples to guide the agent in implementing the scraping logic robustly. The video also covers troubleshooting provider-specific input requirements and response handling to ensure all scrapers work correctly.
To enhance robustness and scalability, the video introduces Ingest, a background job queue system that allows scraping tasks to run asynchronously and in parallel. This setup supports retrying failed jobs, concurrency limits to avoid rate limiting, and persistence so that jobs continue even if the server or browser restarts. The integration with Ingest significantly improves the user experience by enabling scans to run reliably in the background, with progress and results visible in both the app and the Ingest dashboard.
Finally, the video suggests further enhancements such as scheduling regular scans to monitor changes in brand mentions over time and setting up alerts for significant events. This would provide businesses with historical data and timely notifications, making the tool even more valuable. The presenter encourages viewers to experiment with these features and shares additional resources for building beautiful UIs and mastering coding agents, inviting feedback and engagement from the audience.
