I was surprised the other day to see this:
I have to confess that I'd been working on robots.txt functionality in Scrutiny and had accidentally left the robots.txt file in place.
That's a cautionary tale, but not the one I wanted to tell. My file was very similar to this (with a couple of urls in the Google disallow list) - the screenshot is from www.robotstxt.org
I've ruled out a few possibilities but I see from Google's own developer help that the name of the Google robot is actually googlebot not Google. I'm assuming that this is the reason for my problem. If you can confirm this or know better, please leave a message in the comments below.
Saturday 16 February 2013
Monday 4 February 2013
I've been thinking about adding graphing functionality to Organise's report manager for a long time.