Server returning 400 for url with no referer

I've had an interesting support problem this morning and thought that it might be useful to log the answer here.

The problem was Scrutiny not being able to get past the starting url - reporting '400 bad request'. But the same url would return the expected page in a browser.

It seems that this particular server doesn't like not being sent a 'referer' field. Scrutiny does send a referer for all other pages it crawls, filled in with the url of the page that the link appears on. But by definition there is no referer for the starting url and at present it doesn't send one.

Going to Advanced settings and entering 'referer' as the name of the first custom header field, with any valid url (including 'http://') as the value then the crawl worked.

Sending an empty string or a space for the value doesn't seem to work, so I'm not sure what the browsers do (If anyone knows the answer to this I'd be grateful)

Testing page weight where gzip is being used

Webmasters using Scrutiny and Reactivity can now see uncompressed and compressed size of files and  therefore easily see the benefit of their servers' gzip service

Following a support request, I've been looking at the way that these apps show file size and page weight where gzip compression is being used.

The idea is simple and effective - the server sends the file compressed and it's re-inflated at the client side. The transfer time is thus reduced.

Scrutiny and Reactivity were giving file size as the uncompressed size.

It was easy to enhance these apps to take this into consideration. They now show both the compressed and uncompressed size of files, and both totals.

The enhancement is now available in Scrutiny 3.1 and ReActivity 1.1

Making best use of Scrutiny's SEO and keyword analysis

I've always tried to write for the human reading the page.

Happily this approach seems to be the Panda- and Penguin-proof one. And of course it makes good long-term sense. It's Google's job to give the searcher the best page, rather than a crap one that's used some clever tricks to get a good rank. Google doesn't want to promote that page and the user doesn't want to see it. Going forward they'll get better at their job and good content will be king.

With this in mind, is a keyword strategy the right way to go?

Brian Clark says yes.

In The Business Case for Agile Content Marketing, he says that Google is getting smarter but still needs help. And that it’s still important to gently tweak your content so that Google knows exactly who are the right people to deliver it to. If you don't use the words "green widgets" in the right locations and frequency, then the search engine won't know that's what your page is about.

I'm not going to try to fine-tune Scrutiny to analyse your content in line with Google's latest update, because they don't tell us anyway, it's constantly changing and different search engines will weight things differently. Instead it will help you get the basics in place, get you thinking about the right keywords and synonyms and show you how well (or not) you're using those words.

After crawling your site, Scrutiny's SEO window will show you a list of your pages. See those with missing title, description or headings can be seen by choosing the appropriate option from the 'Filter' button.

Simply type a keyword or phrase into the search box and the list will be filtered to show only pages containing that phrase. You'll also see a count in various columns to show you the occurrences of the phrase in the url, title, description, headings.

It will also count occurrences in the content, but this is a feature that you have to turn on in Preferences. (switched off by default, only because it slows the crawl and uses disc space).

Scrutiny is free to use unrestricted for 7 scans, and then only 55 GBP for a lifetime licence. More information and download at Scrutiny's home page.

Tutorial - making a custom Bin-it theme

When OSX was first released, Apple moved the trash from the desktop to the dock, which upset a lot of people who found the earlier desktop trash more convenient*.

I first made Bin-it in 2006 to add a desktop trash to OSX, and shortly afterwards collaborated with Chris Knight to add a very quick and easy progressive indication of the trash level with a changing icon.

Built-in themes include the standard OSX trash (with added levels of trash) and for the retro look, pixellated OS7 and OS9 cans.

Here's how to add your own:


1. Prepare between two to six images, 128 pixels x 128 pixels with transparent background. Save in a format that preserves the transparency such as .png or .tiff

2. Go to Preferences > Themes and click the [+] button

 3. Drag-and-drop your images into the image wells. You can also add sound files if you like, again by drag-and-drop

 4. Give your new theme a name and OK

 5. If you have only two images (as I have here) you can just fill in the first and last wells. Although if you drop the empty / full images into wells 2 -> 5 you can control the point at which the bin appears full. OK that sheet and use the Threshold slider to further control the point at which the bin appears full.

More information and the latest version of Bin-it is available at http://peacockmedia.co.uk/bin-it, free for existing licence holders, or £4.95 if not.

* The big inconvenience of the desktop trash was that it was often covered with a window. Note that Bin-it allows you to choose its level: desktop, floating or 'keep on top'.

AuthorRank and Blogger blogs

Have you seen posts about how important a 'rel = author' link could be for your Google page rank?

It does two things:

• Displays your Google+ profile image by your pages in search results, which increases visibility and trust in your page.
• It's also said that pages with a linked author will rank better than pages without. 

Google give two ways to achieve this. One is to a link your page to your Google profile like this:

<a href="https://plus.google.com/109412257237874861202?rel=author">Google</a>

(obviously using the id number of your profile)

Note that you also need to update your G+ profile to include websites that you author - edit your profile and add your websites to "Contributor to".

Implementing this in my regular websites was a simple copy and paste.

Then I turned to my Blogger blogs. I found that Google already adds the important 'rel=author' code if your name is shown at the bottom of your posts. But it was linking to my Blogger profile (out of date and has a distant photo). The answer was simple.

If you go to the Blogger home page and click the little cog near the top-right, there's an option called 'Google+'. Clicking this allowed me to very easily switch from my Blogger profile to my G+ profile. This simple action meant that all of my posts on all of my blogs now contain the rel=author link to my G+ profile.

Google futher helps you by asking you whether you want to add your blogs to the 'contributor to' list on your profile (if you've not already added all of them in the step above).

Scrutiny on Tiger

This is Scrutiny running and looking good on 10.4 (Tiger).

It's looking particularly good to me tonight because I've just spent the best part of the last 24+ hours trying to track down a very elusive bug, only apparent in 10.4 and without the benefit of developer tools on this machine due to lack of free space.

I was vexed to find that I'd introduced said beastie at some point (into both Integrity and Scrutiny) without spotting it until it was reported yesterday.

Will shortly have fixed versions of both apps uploaded followed by a good and much longed-for night's sleep.

I've noticed a couple of other minor cosmetic problems, which I will enjoy working on. It's very good to know that people are still using the older-but-still-lovely systems!

The beautiful snow leopard

[post originally written after the release of Lion and copied from a different blog system]

Isn't she gorgeous? I'm now appreciating that the operating system she represents is beautiful too.

I've found plenty of things about Lion [edit: and now Mountain Lion] that I really don't like.

I found a secondhand machine for day-to-day use and now that I'm back on Snow Leopard I love it.

Something else has made me think about users of older systems too. A user of my new product Scrutiny asked about ppc support. The answer is that I have been forced to move to XCode 4 in order to find and fix Lion problems. However, XCode 4 doesn't allow me to build a version which runs on ppc machines. Therefore I released the beta as intel / 10.5 upwards.

It seems that maybe I can use XCode 4 to build for ppc and all of these things have made me think that it would be a good thing to support older macs and versions of OSX.

Rather than try the XCode 4 conversion (I've found messing with build settings to be hair-tearingly frustrating) I may instead rebuild the products using XCode 3 and work in that.

Either way, I'm aiming to support Integrity and Scrutiny on ppc and intel, 10.4 and above if possible, or 10.5 and above if not.

[update: I'm now able to build my apps as a single version to run on 10.4 updwards and code-sign them for 10.8's Gatekeeper too.]

Tracking down zombies in objective-C applications

Aren't the worst bugs the ones that cause crashes infrequently seemingly randomly?

Even when you think you've found the problem and run the app successfully 100 times, you still can't be 100% sure you've got it! It's like trying to prove a negative.

Tracking down such a problem can be like the proverbial needle in a haystack. You can get clues from the crash report or debugger console, commenting out lines can work if the problem is fairly reproduceable but a good analysis tool can save many hours of poring over code and testing and re-testing.

In my experience a problem like this is most likely caused by object alloc / release. In such cases XCode will report 'EXC_BAD_ACCESS' which means (most likely but not always) that you're trying to access an object which has been released. 

Instruments with Zombies (XCode>Run with performance tool>Zombies) is a little baffling at first but is well worth it. If the bad access happens, it'll show you the life cycle of the object in question, where and when it was retained and released.