I've recently been working on some enhancements to Webscraper so that it can handle some of the odd problems encountered while scraping the US Yellowpages.com. It's doing that pretty well now, with some setting up. (though scraping is a constantly-shifting area.)
Users' needs are usually pretty specific. I wondered whether I could make a much simplified version of WebScraper, pre-configured to scan that particular site.
It turns out that they provide an API, which changes the game. This app can be much more efficient. The trade-off is that there is a 'fair use' limit per API key, but this is very generous and the limit I've built into my app is pretty hard to hit.
This is the interface.
In return for such a simple interface, all configuration is fixed. It's locked to yellowpages.com and the output file is pre-configured too.
For more flexibility (or to scrape a different site) please look at WebScraper.
Post a Comment