Productivity improvements in Integrity and Scrutiny

Over the years the web crawling engine that's shared by Integrity and Scrutiny has become faster, more efficient, more accurate and free of problems. It does what it does really well.

But the interface hasn't kept up; it displays its results and then you're on your own. Support calls have been about what people want to do next:

"Where is this link that it's reporting?"

"I only want to copy that URL but there doesn't seem to be an easy way to do it"

I hope that the new version 4.2 will help with such tasks. Useful functions such as 'Copy URL', 'Visit', 'Highlight' and the exciting new 'Locate' function can be found via context menus, buttons, keyboard shortcuts and menus.

First of all, the 'by link' view is an expandable view, meaning that the list of pages that the link appears on can more intuitively be seen from that view without having to open the link inspector:

The link inspector is improved. You can still double-click in its table to either visit or highlight (according to your preferences) but you can now also pop up a context menu with a number of options, or use the new buttons to visit, highlight or locate:

The Locate function is a big help in those situations where you're not quite sure how the crawler has found a certain page - maybe it's an old page you thought you'd orphaned. Previously it was possible to trace the path but it was time-consuming. Now you can call up a list of the clicks required from your starting point to the link in question:

These useful functions are available from context menus - right-click or command-click the link or page to pop up a menu:

The new versions of Integrity and Scrutiny containing these features are available for download:

http://peacockmedia.co.uk/scrutiny/

http://peacockmedia.co.uk/integrity/

And on the Mac App Store shortly.

Periodic Table Of SEO Success Factors

I've just seen this wonderful graphic showing the factors which influence your search engine ranking with an indication of how each is weighted (and which ones work against you).

For the large version complete with lots of explanation.

Scrutiny will be able to help you with many of these things.

Find missing meta tags

[Updated 24 Jun in line with Scrutiny and Integrity Pro v9]

Meta keywords may not be as important now as they used to be, but your meta title is one of the most important SEO factors and your meta description will appear on search result pages and net you click-throughs.

Here's how to check your site to see whether these things are in place using Scrutiny and Integrity.

1. First you need to scan your site. Scrutiny: At the Sites screen, press 'New' and type your starting url. Press 'Next' to see the default settings, press 'Next' again to accept those settings. Press 'Go' beside 'SEO'.

Integrity: Type your starting url into the address bar and press Go. Integrity Plus and Pro, First press the [+] button below the left-hand pane to add a new site.

2. When the crawl has finished, the SEO screen will open

3. Use the Filter box to select 'missing title' or 'missing description'

4. Scrutiny and Integrity Pro have a Meta Data tab under the SEO tab. There are many columns that you can switch onto show various meta data. So if you're interested in the Twitter tags, then you can switch on those columns and see which pages contain those tags or are missing them.

How to test a website which requires authentication

[last updated 23 Mar 2021]
This process will involve a little experimentations because of the different ways that authentication can work. It may be as simple as checking a box.

It also involves a risk of changing or deleting your pages - yes really. With some content management systems, the buttons to perform these actions can look like links to Scrutiny and it will dutifully try to click them.  Here are some precautions:

• If possible, use an account which has access to view the site but no higher
• If you know the url(s) of links which could perform changes, use Scrutiny's blacklist feature to make sure that they're 'not checked'
• Make sure that your site is backed up and that you're prepared to restore if necessary

So, we're ready to go. Work through these steps until Scrutiny successfully crawls the site as an authenticated user. 

1. Go to Advanced Settings and Check 'Attempt to authenticate'. You will see a warning - read it, heed it and OK it.


2. Step 2 used to be:

Log into your site using Safari, using an account which has read access but no higher.  If your site has an option for 'keep me logged in' then check that. (step 9 should be up here - do that now) Then try to crawl the site.

However, that method won't work if you're on Yosemite or newer because of tighter security in MacOS making cookies browser-specific and no longer systemwide. If you are on 10.9 or earlier, do that ^.

If you're on 10.10 or higher, here's the new step 2. Check the 'attempt to authenticate' button.

It's a simple workaround - if your website tracks you using a cookie (rather than session id) then you can use a simple browser window here to log in. If you check the 'handle cookies' button, then when you start Scrutiny's crawl, it should retain and use the cookie you just collected. 

(Version 10.3.1 updated the webview used in the Log in window. This helped the functionality to work with some sites and broke the functionality for others. Version 10.3.3 offers a choice - try the legacy version first, if that doesn't work, try the other.)

3. This is worth a try, I have seen it work. It may or may not work when you try it in your browser - currently it works in some browsers but not others, but may still work if you try it in Scrutiny.

Add the username and password to your starting url, in the form:

http://user:password@example.com

4. If that doesn't work, enter the username and password into the top two boxes in the Advanced Settings window and try again.

5. If your site uses a web form to send the authentication details ( eg Wordpress) then find out the names of the username and password fields. Here's a snippet of the source for the site above, and you'll see that the names of the fields in this case (a Wordpress site) are 'log' and 'pwd'. Enter these in the second pair of boxes and try the crawl again.

<input type="text" name="log" id="user_login" class="input" value="" size="20" />
<input type="password" name="pwd" id="user_pass" class="input" value="" size="20" />

6. You may need to experiment with your starting url too. If you're using the web form fields described in step 4 then Scrutiny will send these by POST request but it'll only send them to your starting url. (The site should use a cookie or some kind of session id after that.) Again, if you check the source code of your login form, find the form action and use that url as your starting url. 

7. If authentication still not working, check your html login form for hidden fields (or visible ones) which may be necessary for the login to work. [Note step 7 - this step may now be superseded by a new checkbox.] Since v5.5 you can enter any field names and values to be included in the POST request:

8. If your login form uses a security token - check the box and try again. (This may take care of step 7 so it may not then be necessary to add the names / values of  hidden fields). This feature is available in Scrutiny v6.4 onwards.

9. If no joy, this may work: Using a custom header field in the Advanced settings panel, set Field to 'Authorization' and Value to 'Basic [base64 encoded credentials]'. Credentials should  be in the form username:password and an encoded version can be obtained here

10. Once logged in, there may be a 'logout' link on your pages. Obviously you don't want Scrutiny to log itself out on the first pageful of links, so you may have to blacklist such links (see screenshot above).

11. If you've tried all of this and are still unable to log in, please contact Scrutiny support. Be ready to let us have the details for a test user account with read access but no higher.

Test the load speed of a page and all of its elements

This tutorial uses Scrutiny or Reactivity. The page test is a component of Scrutiny, but is also built as a standalone app called Reactivity.

You can get the same results from this online demo [link removed as out of date], which runs the same app but it doesn't have the user interface and the results are presented on a web page.

1. If you're using Reactivity you'll be presented with this window. If you have Scrutiny, call this window with Window > Page analysis,  command-2 or switch to the SEO tab and press 'Full Page Analysis'.

2. Type the address of the page that you'd like to test and press Go. To test more than one page at a time, simply open another window - In Scrutiny repeat step 1. In Reactivity use File > New or command-N

3. The app will find all elements that make up your page - the html file, the images, javascript files and css files. It will load each element separately without multi-threading so that you get the most accurate timings.

4. Remember that the response time and load time will reflect the server's response time and speed, but will also be affected by the speed of your own internet connection.

5. You will see a total weight for the whole page. (The one in the example is a bit hefty). You will be able to spot any elements on your page which are particularly big, and you'll also be able to spot any elements which are slowing your page down -  third party code or plugins often cause such problems and affect your page's response - maybe even your Google page rank.

6. If your site uses compression, then you'll see the compressed as well as the uncompressed size for each element as well as totals, and see how effective that is being.

How to generate an XML sitemap using Scrutiny

[Updated 6 March 2017 in line with Scrutiny v7]

1. First you need to scan your site. If you don't already have a configuration for your site, at the Sites screen, press the + button and type your starting url. 

2. Press Scan now. When that's finished, choose Sitemap from the list of tasks. Check that this list contains your pages as you expect. See the manual for more information about how to exclude pages or why some might not be appearing

3. 'Change frequency' indicates to Google how often you feel that you update your pages. This doesn't oblige them to crawl your site any more or less regularly. If you generally update your pages more regularly or less regularly than weekly, you might like to change the default setting and press 'Update all'.

4. The 'Priority' you give your pages is a measure of how important you think each of your pages. Priority must be a value beetween 0.0 and 1.0. I suggest that you mark your home page as 1.0, one important page as 0.9, slightly less important pages as 0.5 and then decreasing from there. If you choose the 'Automatic' setting, Scrutiny will mark your starting url as 1.0 and then calculate the others based on the number of clicks from the home page, and use a logarithmic scale. ie one click from first page = 0.5, two = 0.3, three = 0.2 with all other pages = 0.2

4. For more specific control, you can set up some 'rules' to specify priority and update frequency for certain pages or sections of your site. You don't need to enter a full url - like the black and white lists on the front settings screen, you only need to enter a partial url. This way it is possible to specify a particular url or a section of the site, eg "/engineering/"

5. When you're ready, go to Export > XML  and choose where you save the sitemap file.

6. See the search engine's own help for details of how to submit your sitemap.

ScreenSleeves screensaver - Installing and setting preferences

Important note: The article below applies to the old screensaver version of ScreenSleeves. This has been superseded by a standalone app, but the appearance and the options are pretty much the same as described below. The settings for the new app are found in the app's Preferences.

It's a shame that the Lounge screensaver seems to have disappeared. It displays only the currently playing album cover with some details in a very neat, easy-on-the-eye-way (a la Front Row). Being a screensaver it just kicks in if you leave your computer or mouse to your hot corner.

So I set about writing a similar app. But with Spotify support too, which I would have liked in Lounge - I use both iTunes and Spotify equally.

Screensleeves detects which is playing and displays cover art and details for the currently-playing track:

As with most screensavers there are some options, and it's not always obvious how to find these options. So here's a quick tutorial.

[4 Aug 2014 - this page updated to reflect the new options. If you don't have all of the options below, please download and install the current version]

Installing

Simply double-click the package icon from within the downloaded dmg. System Preferences will open and confirm that you'd like to install the saver. It'll do so and show you a preview. Note that with earlier systems you may need to click the saver's name in the left-hand list if you want to switch to it.

Accessing options

After you've installed, you can get back to the options via System Preferences > Desktop & Screen Saver > Screen Saver   and press 'Options'

What those options mean

1. Theme: Choose from Classic, Text, Mosaic, Full-Screen or Random. Random chooses one of the graphic themes (not Text) each time the screensaver kicks in. Preview window in System Prefs will not be the same theme as the one chosen when the saver kicks in for real.

2. Artwork size: The artwork and text size will adjust itself to be suitable for your screen size, but you can make the default size larger or smaller.

3. 3D effect: The cover can be displayed flat or at an angle. This slider adjusts the angle of the effect. Works with Classic and Full-Screen themes

4. Font size: Adjusts the size of the text. With some themes the font size may also be sized relative to the artwork size.

5. Rating / Popularity: If iTunes is playing, your star rating will be displayed and you can press keys 0-5 to change the rating. If Spotify is playing, the Spotify popularity will be shown. You are unable to change this except by playing the tune which is a vote for it.

6. A Progress bar can be displayed showing the position within the current track.

7. Show Total track time / Time remaining: If progress bar is showing it can display a counter showing either of these things.

8. Keep centred and static: If you are sure your screen isn't burnable, you can fix the position of the artwork and text. It will only fade when the track changes.

9. If no artwork available: These buttons choose what Screensleeves does. It can display a default image, or no image. Since v3.4.3 it's been able to very sucessfully fetch album art from the web. The last box about Snowtape is there because Snowtape currently provides artwork which is low-quality and often not the correct cover. With the box checked, SS will try to override the artwork Snowtape provides with a cover fetched from the web.

10. Always find from the web with Snowtape / Rdio: Depending on the type of music you're listening to, Snowtape may not provide the best album cover for the track playing. In both cases, these players provide lower-res images than is desirable when Screensleeves is displaying larger artwork sizes. This option forces Screensleeves to source a better quality and perhaps more appropriate image from the web.

11. ScreenSleeves can default to any of the other screensavers if necessary. Just choose your favourite from the list.  Note that the alternative won't show in the preview window. You'll need to press 'Test' or Preview' or use your hot corner

12. This box allows you to choose whether the alternative saver cuts in if there's no music playing, or if no music player is running. Checking this box means that if there's a player available, but paused, ScreenSleeves will display a 'Paused' message and allow you to start the player using the spacebar.

13. If both players are running but paused, these buttons allow you to choose which one starts playing when you press the spacebar.

14. Setting a modifier helps to prevent the rating of a track being changed accidentally. 'none' means that you can just press keys 0-5 to set the rating. 'control', 'alt', or 'cmd' mean that you have to press that key along with 0-5 to change the rating.

15. If you're going to sleep to music, or for some other reason want the screen to black out when the music stops or after a delay, here are the options.

Triggering the screen saver

As per any screensaver, it'll kick in after a preset time, or you can trigger it manually by setting up a Hot Corner. While your System Preferences window is showing, you can use the 'Test' or 'Preview' button for a full-screen preview.




Download

In case you missed it at the top of this post, the ScreenSleeves download is here