Web Scraping Roadmap
The roadmap started as a guide for a friend learning to collect web data and grew into a public resource. It covers requests and parsing, dynamic pages, data cleaning and storage, and good practice around rate limits and site terms, with every stage shown both in hand-written code and with AI coding assistants.
Authors
PhD Student in Geospatial AI
I am a PhD student at the Chair of Cartography and Visual Analytics
at the Technical University of Munich, supervised by Prof. Liqiu Meng and
mentored by Prof. Yu Feng. I came to AI from maps. I studied geodesy,
cartography and GIS at Wuhan University, where I first used language
models to make sense of geospatial code, and I now spend most of my time
on the AI side of GeoAI. Besides research, I supervise Master’s students
and maintain open learning resources on LLM agents and web scraping.