Scraping 250k+ URLs using Claude Code (via Telegram)
This Saturday, I was about to leave for a movie with my friends and suddenly thought of experimenting with newly launched Claude Code channels. I set up Telegram to work with a Claude Code session, kept my laptop on, and then left for the movie, but when I was in the cab I started chatting about the scraping project and asked Claude to give me suggestions and ideas about how this will work.
And by the time I reached, Claude was already setting up the project and ready to start the scraping process. I give the final confirmation and get busy watching the movie. When I checked my phone during the interval, it had sent me a bunch of messages and the process was still running. It had discovered 260k URLs to scrape and had already completed scraping 36k URLs as you see below.
Whenever I messaged "Progress?" via Telegram, it quickly sent me a summary like above. The scraping process ran for ~8 hours, and it was still running when I returned home. And after a few hours, when the process was complete, it sent me this message confirming the completion.
Around ~10k URLs failed which is acceptable for a process this huge, but it had also failed to capture a few data points so I asked it to grab those as well and then the process again ran for ~3 hours. Finally, I had everything I needed. I, then, asked it to update the scraping script so we have the final polished data when we run the process the next time.
All I would say is, thank you, Claude.
Webmentions