The creator uses Carpathy’s auto research project combined with Claude and OpenAI’s Codex to set up an automated AI hacker that tests the security of their Vibecoded site, focusing on protecting paywalled MD files behind a token-based download system. After multiple iterative attack attempts and AI-driven analysis, the system found no critical vulnerabilities, demonstrating strong site defenses and highlighting the effectiveness of AI-powered red teaming for proactive web security.
In this video, the creator explores using Carpathy’s auto research project combined with Claude, an AI model, to set up an automated red teaming hacker aimed at testing the security of their Vibecoded website. The main goal is to determine if the site’s MD files, which are protected behind a paywall, can be breached or stolen. To achieve this, the setup involves an evaluation or scoring function that rates each hacking attempt from zero to 100, allowing the AI hacker agent to learn from past attempts and improve over time by keeping successful strategies and discarding ineffective ones.
The creator demonstrates the technical setup using Cloud Code, showing how the AI is instructed via program files and attack scripts to run timed attack attempts on the website. Each attempt is evaluated, logged, and used to refine future attacks in a loop similar to Carpathy’s auto research methodology. Initially, the attack attempts were set to two minutes but were later extended to five minutes to allow more thorough testing. After several runs, the AI had not found any critical vulnerabilities, indicating the site’s defenses were holding up well.
To further enhance the testing, the creator integrates OpenAI’s Codex to analyze the results and suggest new experiments or attack vectors. Codex provides prioritized recommendations, which are then incorporated back into the Cloud Code setup for additional testing rounds. This iterative process helps ensure a comprehensive security evaluation by leveraging AI-generated insights to explore potential weaknesses that might have been missed in earlier attempts.
The video also covers specific tests related to the site’s token-based download system, confirming that the token mechanism works as intended, including download limits and expiration times. The AI was able to download files only when provided with a valid token, and attempts to bypass this protection failed. The creator notes that while the token URL could potentially be shared post-purchase, this is an acceptable risk given the limited download window and count, and does not constitute a significant vulnerability.
In conclusion, after 16 automated experiments, the AI hacker was unable to access the protected MD files without the proper token, demonstrating strong site security. The creator expresses satisfaction with the results and encourages others with web-coded sites to use similar AI-driven red teaming tools as a proactive security measure. They emphasize the ethical use of such tools for defensive purposes only and highlight the educational value and practical benefits of integrating AI into security testing workflows.