Important new feature for those attempting to crawl a website with authentication

Scanning a website as an authenticated user is a common reason for people turning to Scrutiny.

The process necessarily involves some trial and error to get things set up properly, because different websites use different methods of authorisation and sometimes have unusual security features.

Scrutiny now has an important new feature. Some login forms use a 'security token'. I'm not going to go into details (I wouldn't want to deprive my competitors of the exasperations that I've just been through!)

There's a simple checkbox to switch this feature on (available since Scrutiny v6.4), and this may enable Scrutiny to crawl websites that have been uncooperative so far. (This may well apply to websites that have been built using Expression Engine).

All the information I have about setting Scrutiny up to scan your site (or member-only pages etc) which requires authentication is here.

Version  6.4 is in beta as I write this, if you're interested in trying it, please just ask.

Small print: Note that some care and precautions (and a good backup) are required because scanning a website as an authenticated user can affect your website. Yes, really! Use the credentials of a user with read access, not author, editor or administrator.

Link checking Wordpress websites using Scrutiny

This article is for you if you want to scan your Wordpress site for broken links, SEO issues, generate a sitemap, or perform spelling / grammar checks. It also applies to other CMS's which generate 'SEO-friendly' urls.

If you want to start your crawl at the root, www.my-wordpress-site.com then just go ahead, it should be fine and you can limit your crawl if you need to by including or excluding partial urls (blacklisting or whitelisting).

But if you wish to start deep within a site, Scrutiny will limit its crawl to pages within and below that directory. eg www.my-wordpress-site.com/publications/all-publications/

You may know that all-publications is a page. But to Scrutiny it looks like a directory (it's not the trailing slash, it's the lack of a file extension) and it will dutifully check all links on that page but only follow (crawl) links which are in the same 'directory' or below. Therefore your pages won't be fully crawled as you expect.

Since v5.5, Scrutiny has a new option on the Settings screen which allows you to tell Scrutiny that urls are in this form.

It means 'the last part of this url is a filename, not a directory' and so in our example, a crawl starting at www.my-wordpress-site.com/publications/all-publications/  would be limited to the /publications/  directory which is probably what you would expect.

How to test a website which requires authentication

[last updated 23 Mar 2021]
This process will involve a little experimentations because of the different ways that authentication can work. It may be as simple as checking a box.

It also involves a risk of changing or deleting your pages - yes really. With some content management systems, the buttons to perform these actions can look like links to Scrutiny and it will dutifully try to click them.  Here are some precautions:

• If possible, use an account which has access to view the site but no higher
• If you know the url(s) of links which could perform changes, use Scrutiny's blacklist feature to make sure that they're 'not checked'
• Make sure that your site is backed up and that you're prepared to restore if necessary

So, we're ready to go. Work through these steps until Scrutiny successfully crawls the site as an authenticated user. 

1. Go to Advanced Settings and Check 'Attempt to authenticate'. You will see a warning - read it, heed it and OK it.


2. Step 2 used to be:

Log into your site using Safari, using an account which has read access but no higher.  If your site has an option for 'keep me logged in' then check that. (step 9 should be up here - do that now) Then try to crawl the site.

However, that method won't work if you're on Yosemite or newer because of tighter security in MacOS making cookies browser-specific and no longer systemwide. If you are on 10.9 or earlier, do that ^.

If you're on 10.10 or higher, here's the new step 2. Check the 'attempt to authenticate' button.

It's a simple workaround - if your website tracks you using a cookie (rather than session id) then you can use a simple browser window here to log in. If you check the 'handle cookies' button, then when you start Scrutiny's crawl, it should retain and use the cookie you just collected. 

(Version 10.3.1 updated the webview used in the Log in window. This helped the functionality to work with some sites and broke the functionality for others. Version 10.3.3 offers a choice - try the legacy version first, if that doesn't work, try the other.)

3. This is worth a try, I have seen it work. It may or may not work when you try it in your browser - currently it works in some browsers but not others, but may still work if you try it in Scrutiny.

Add the username and password to your starting url, in the form:

http://user:password@example.com

4. If that doesn't work, enter the username and password into the top two boxes in the Advanced Settings window and try again.

5. If your site uses a web form to send the authentication details ( eg Wordpress) then find out the names of the username and password fields. Here's a snippet of the source for the site above, and you'll see that the names of the fields in this case (a Wordpress site) are 'log' and 'pwd'. Enter these in the second pair of boxes and try the crawl again.

<input type="text" name="log" id="user_login" class="input" value="" size="20" />
<input type="password" name="pwd" id="user_pass" class="input" value="" size="20" />

6. You may need to experiment with your starting url too. If you're using the web form fields described in step 4 then Scrutiny will send these by POST request but it'll only send them to your starting url. (The site should use a cookie or some kind of session id after that.) Again, if you check the source code of your login form, find the form action and use that url as your starting url. 

7. If authentication still not working, check your html login form for hidden fields (or visible ones) which may be necessary for the login to work. [Note step 7 - this step may now be superseded by a new checkbox.] Since v5.5 you can enter any field names and values to be included in the POST request:

8. If your login form uses a security token - check the box and try again. (This may take care of step 7 so it may not then be necessary to add the names / values of  hidden fields). This feature is available in Scrutiny v6.4 onwards.

9. If no joy, this may work: Using a custom header field in the Advanced settings panel, set Field to 'Authorization' and Value to 'Basic [base64 encoded credentials]'. Credentials should  be in the form username:password and an encoded version can be obtained here

10. Once logged in, there may be a 'logout' link on your pages. Obviously you don't want Scrutiny to log itself out on the first pageful of links, so you may have to blacklist such links (see screenshot above).

11. If you've tried all of this and are still unable to log in, please contact Scrutiny support. Be ready to let us have the details for a test user account with read access but no higher.