Scrutiny and WIX sites

Wix make it easy to create a website, but their reliance on javascript and plugins (including Flash apparently) makes it a challenge for webcrawlers, including Googlebot, to index the content.

It seems that Google have had to go out of their way to index these sites. There used to be a hack to get at plain html content, which I used to give as a workaround to users wanting to scan Wix sites, but that no longer appears to work.

As someone with a background in website accessibility, this all seems appalling to me*.

But they're gaining ground, I seem to be seeing more support questions about scanning WIX sites. (and their *!^%$ advert that I have to skip before almost every Youtube video I watch. When is Yotube Red going to be available in the UK??)

Unless anyone knows otherwise (please, please tell me if you know), WIX relies on a/ providing its users with an xml sitemap for submission to Google, and b/ Google being able to render js before indexing a page (again, as an old-fashioned gal, something I heartily disapprove of* )

Scrutiny has long been able to render javascript before parsing page content. There have always been a small number of sites which have menus and links that are written by js when the page loads. I've recently done some work to ensure proper scanning of such a site.

So... here's my best advice for scanning a Wix site using Scrutiny (Integrity and Integrity Plus don't have the necessary js option).

1. Make sure you have Scrutiny 7.4.1 or higher, which contains some necessary improvements to the js functionality. The most up-to-date version of scrutiny is always here.

2. Switch on 'Render Javascript' which is in the Advanced settings for your site.

3. Turn down your number of threads. You can see I'm using 3 here (first notch). Using the js option is very heavy in terms of client-side processing. You don't want pages timing out without having loaded because Scrutiny is working hard at rendering a large number of pages. If your website scans fully then maybe edge this up a bit but best to err on the side of caution.

4. Go and make a cup of tea. This is going to take some time.
5. After the scan, switch on the Target Size column of the 'by link' table and check whether any of the Internal links have a target size of zero (switch the Filter button to Internal to see this more easily). If an internal page has zero kb as the target size, then it may have timed out before the page had finished loading / rendering, which means that you may have an incomplete scan.

If that looks well then you're good to go with your data, which should be complete.



* It's not the javascript I disapprove of, but the fact that the page is 'invisible' without the js being rendered. Accessibility guidelines state that we must "Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page." (Web content accessibility guidelines, checkpoint 6.3 -  a priority 1)
SaveSave

Sharpresize v2 now fully released

SharpResize v2 is now out of beta. It's a lightweight application (ie quick to start up) for making a better thumbnail / resized for web  image. The great result is due to the combination of the resizing algorithm it uses along with an additional sharpening slider.

Despite being a lightweight app it now has a good selection of adjustment sliders and built-in filters. (sepia, black and white, posterize etc).

We've decided to keep the price down. In fact as it's now priced in dollars (1.95 USD) it's a little less in real terms than the previous GBP price.

More information and the download are here

The guide is here.

Alongside our recent post about supporting International Domain names within our web crawling tools, we're very proud and excited to announce that work has started on translating some apps and some web pages into other languages. Initially French, and initially Integrity, Integrity Plus and Scrutiny.

It's impossible to do all of the work at once, it will take a little while but the web pages for Integrity, Integrity Plus and Scrutiny now allow you to choose (top-right) English or French versions of the pages.

Localized context help in French is shortly to go into those apps, followed by the rest of the text within the apps.

When is a link internal.....

Here's an interesting situation.

The website being scanned was http://peacockmedia.software.  As you can see, the link found during the crawl is a link to a page on peacockmedia.co.uk . That looks like an external link, and so Scrutiny has marked it as an external link, therefore checked its status but not followed it.

However, it redirects to a url on peacockmedia.software - so the ultimate destination is an internal one.

There are a bunch of pages which will only be discovered if this link is considered internal.

(the pages are old manual pages - I want them to be orphaned, but that's beside the point...)

The question is - should Integrity and Scrutiny consider that link *internal* (and therefore follow it and discover more internal pages) or *external* (so it's checked but then is a dead end). Remember that the original link is to an external domain.

Heads-up : Internationalised Domain Names (IDNs) supported in our web crawlers

We're a UK-based concern, our apps have been almost always available in a single language - UK English (or just English as we call it here in England!)  The vast majority of our users as I write this are from English-speaking countries.

Our alphabet entirely consists of characters available in ascii, and so there has been little call for Integrity, Integrity Plus and Scrutiny (and other tools based on the same engine) to support domain names - ie domain names which contain characters not found in the ascii character set.

But now we've started work on localisation of our apps and web pages, and have received the odd question concerning IDNs.

Let's not confuse this with unusual characters in the path and filename of the url. Our apps have long supported these. You may still see the non-ascii characters displayed, but behind the scenes, those characters are encoded before the http request is put together, usually using a percent-encoding system.

The method is similar with the domain name, but using a a more complex and clever system of character encoding. Browsers (and our web crawlers) often still display the user-friendly unicode version.

You can enter your starting url in the unicode form or the 'punycode' form and it'll be handled correctly. The same goes for unicode or punycode links found on your pages.

Personally, I'm not keen, this does allow for spoofing of legitimate domains using similar characters. There are rules excluding many characters for these reasons.

After lots of extra homework for us, Scrutiny is now handling IDNs, and is in testing.

[update 26 May 2017] Integrity and Integrity Plus also have this enhancement and are also in testing.

If this is useful for you, and you'd like to try the new version (remembering that there may still be the odd bug to iron out)  then you're very welcome to download and use it (with the condition that you let us know about any issues you spot.)

[edit - links taken out, out of date]

List of all of a site's images, with file sizes

A recent enhancement to Scrutiny and Integrity make it easy to see a list of all images on a site, with file size.

It was already possible to check all images (not just the ones that are linked, ie a href = "image.jpg" but the actual images on the page, img src = "image.jpg"  srcset = "image@2x.jpg 2x" )

The file size was also held within Scrutiny and Integrity, but wasn't displayed in the links views.

Now it is. It's a sortable column and will be included in the csv or html export.

Before the crawl, make sure that you switch on checking images:

or

You may need to switch on that column if it's not already showing - it's called 'Target size'.

Once it is showing, as with other columns in these tables, you can drag and drop them into a different order, and resize their width.

To see just the images - choose Images from the filter button over on the right (Scrutiny and Integrity Plus)

If you're checking other linked files (js or css) then their sizes may be displayed, but will probably have a ? beside them to indicate that the file size shown has not been downloaded and the uncompressed size verified (the size shown is provided in the server header fields).

This last point applies to Integrity and Integrity Plus, and will appear in Scrutiny shortly.

Note that all of this is just a measure of the sizes of all files found during a crawl. For a comprehensive load speed test on a given page, Scrutiny has such a tool - access it with cmd-2 or Tools > Page Analysis

Post-apocalyptic life skills : handwriting

Self-service is great, if all goes as planned then it's efficient and easy for both customer and seller. (Though I still resist the self-service tills at my local convenience store - I think I care too much about the 4 or 5 jobs that have clearly gone redundant as a result. Plus - the store is reaping the benefit of the customer doing those ladies' jobs, without paying the customer in any way for doing that job.)

Anyway. When it doesn't go as planned is when the fun and games start.

Today I found myself needing to get back to a customer urgently, but the email address bounced as 'user not known'. Leaving me with a postal address.

The old business-letter-writing skills came back pretty easily; my address on the right, their address below and to the left, sincerely if you know their name, faithfully if you don't etc.

(I'm an old-fashioned kinda girl. Printing off the attempted email and putting that in an envelope wasn't going to happen.)

I was wrong to think that it would be quicker to write the address on the envelope than to work out how to print it on a label or put the envelope through the printer.

I draw a lot, but I don't write very much at all now. My handwriting was never great, it seems to be appalling now. After several practice goes, I got a result that wasn't as hideous as the first ones.

.... then turned it over and found that the address was upside-down! How crap does that look?

I'm not going to be among the survivors after the aliens come and hit us with a big EMP device.

Don't we love rules?

I've just Googled 'html5 page structure' to check something. The post that appeared first consists of a very simple code snippet followed by around a hundred comments, some asking questions but mostly asserting an opinion. (none spammy or trolly).

Without wanting to get very deeply into the actual topic itself (and maybe risking the odd comment myself....) a large number of those comments are about whether a section tag is allowed within an article tag and vice versa. It seems that anything goes as far as these two are concerned. But my, don't we love rules?

It reminds me of those areas of cities where the pavement and road have been paved over into one homogenous area. I hate them; surely it's better if there are rules that everyone knows?

It occurs to me that the vast number of relevant comments could be responsible for that post being ranked #1 for such a popular search term. I'm sure there's a lesson there somewhere...

Hidden Gems in Scrutiny 7: Locate a broken link

This tip applies to Integrity, Integrity Plus and Scrutiny.

So the app is reporting a broken link (or maybe it's a redirect you're interested in, or just a good link). You can easily see, copy or visit the target of the link, or the page it appears on. But how did the crawl find this particular page?

The Locate function will tell you. First open the link inspector by double-clicking on the link in one of the Links views Then highlight the 'appears on' page you're interested in, and click 'Locate'.

It won't show you every possible route to that link, but it will show you the shortest.

note that there's a context menu there too with these options.

You may have noticed that the link inspector and the context menus have a 'Highlight' option too. If you're having trouble seeing the link on the page, the Highlight option will do its best to open the page and apply yellow highlighter.

Scrutiny plus 11 other apps for $19.99

[Edit 14/6/17] this bundle has now finished.

 Bundlehunt have just gone live with an offer that includes Scrutiny.

Choose 12 from more than 60 premium apps and pay $19.99

The Scrutiny licence is the full per-user licence for the current version, 7.3.1, usually $115.

Improvements since 7.3 include

• Improved insecure content reporting - the table now shows all issues - secure pages which contain links to insecure ones, and pages with mixed content.
• Manage the autosaved data and selectively trash what you don't need.
• see all images with their file size
• Other small fixes and enhancements

SharpResize Help, updated for v2.0

This article uses SharpResize and is updated for version 2, which creates better thumbnails. Version 2 is available shortly as beta.

In this article:

• Opening an Image
• Quality and Size
• Adjustments
• Additional Output Sizes
• Batch Image Processing
• Saving

Opening an Image

There are several ways to get your image into SharpResize.

• New from clipboard using File > New From Clipboard or cmd-N
• Drag the image onto application's dock icon
• Drag the image onto application icon or alias
• Drag the image into SharpResize's image well
• File > Open or cmd-O

Tip: You can grab an area of the screen using cmd-ctrl-shift-4. Drag a rectangle and see the width and height of your selection. It's not an easy shortcut to remember but worthwhile.

In addition there are toolbar buttons for New from Clipboard, Open and Save.

Retina Screens

A note about retina screens. This is a confusing subject, pixels can refer to the number of dots within an image and the actual physical pixels on the display, or it can be used as a a measurement similar to points.

Because SharpResize is designed primarily for resizing images for the web, it will display your image at '1x' - ie the size that it would appear if you specified your size in pixels in a CSS. If you're using a retina screen, then this would obviously look fuzzy and you wouldn't be able to make proper judgement when making adjustments to sharpness etc.

So SharpResize has a setting in Preferences:

As it says, the 2x version of your image will be displayed while you're working. This setting should be auto-detected when you first run SharpResize.

If you want to save the actual version of the image that you're seeing while you're working, make sure that you have 2x selected in the Additional sizes.

Quality and size

When you open a new image, the width and height won't immediately set themselves to the size of your image because Sharpresize is about creating thumbnails, and the width and height fields may be set to the size that you want your thumbnail to end up.

If you do want to set the output size to the actual size of the image, press 'Actual' or cmd-=

If you want the resized image to have the same aspect ratio as the original, make sure the padlock  is in the 'locked' position.

If the padlock is in the 'unlock' position, then you can set the height and width to whatever you like, and you'll see additional controls relating to whether you want to crop your original image to size or add white, grey or black etc.

Resizing will be performed with Lanczos sharpening which is noticeably better than the more usual bicubic. You will see the output image displayed in the image well.

The Sharpening slider applies a further filter to further improve a resized image. Use judiciously - too much will result in an artificial look. The best setting here will depend on the image. Most pictures seem to look best with the slider about a third of the way. If the image includes text, any more than a small amount of sharpening can spoil the anti-aliasing and make the text look jagged.

Under the Output File tab, the Quality slider sets the jpeg compression of the output. (If you have the jpg output file format selected.) For maximum quality you'll also get the maximum file size but the file size can be reduced significantly (maybe halved) by backing off that slider a tiny bit below maximum.

Adjustments

Version 2 has more controls for adjusting the output image, and also a few filters, such as Sepia and Black and White.

Like a little extra vibrance with your sharpening? No problem.

Additional Output Sizes

The height and width fields are important, they'll set the dimensions of the primary output image for all input images.  You can now create and select additional sizes (for example, a banner-format image, or a square image, or a @2x image).  These can be relative to the original image size, or your primary output image, or have absolute dimensions.

Batch Image Processing

Version 2 allows you to drag and drop or open multiple images. They'll all be resized using the height and width that you've set and all using the same sharpness and adjustments that you've set. if you've created and selected additional output sizes, each input image will be saved with all output sizes.

Saving

You can save the image using File > Save, cmd-S or the toolbar Save button.

If you have opened multiple images, you will be asked to choose a folder for the output images (at least one output image for each that you've opened). Files will be saved with their original names. If you've chosen additional sizes for each image, the filename modifier will be applied.

There's a preference to set whether SR saves all of these images directly in the folder that you've chosen, or whether it creates a new folder with a unique name and saves the images within that. The latter method is set by default, and is less likely to cause filename conflicts.

If you have opened a single image (regardless of how many output images you want) then you'll be asked to choose a filename, and your output images will be saved with that filename, with modifier where appropriate.

This *one weird trick* will cure your website of all ills

Do people really fall for those headlines? Surely we're all aware by now that anything that sounds too good to be true always is. And that if you ever do take the plunge into that clickbait labyrinth and ever do find the promised information, it's sure to disappoint.

A spam email about Mother's Day, which turned out to be selling cleaning products, nearly made funniest spam email this week (also the one from Dick Richardson offering to send a bag of gummy dicks to the person of your choice for $15) but all of that was pipped at the post by this one:

When I say funny, I do find it sad that it's reinforcing and preying on fear and anxiety. The funny part is the '4Idiots' part.

I can almost understand how the 'for Dummies' series works, people are often self-deprecatory when it comes to technical things.

But what man with anxiety about his height is going to click a link that says 'for idiots'?

I'm still wondering whether it's a parody. The author appears to be saying 'you'd have to be an idiot to believe this!'

(For the record, the subject line mentions HGH. This can significantly increase height but only if you haven't stopped growing, this may work into someone's twenties but there's no mention of that here.)

Anyway. You clicked a 'one weird trick' link and arrived here. Maybe it's not an empty promise this time; we're responsible for Scrutiny, which can quickly scan your website and report broken links, SEO issues, spelling and much more.

Scrutiny is the premium product. For no-frills link-checking, there's the free Integrity, and for more features, it's paid sister Integrity Plus. Too good to be true?  The paid apps offer free trials, so you've nothing to lose by finding out.



SaveSave

Hidden Gems in Scrutiny 7: Run multiple website scans (and other things) at the same time

This may be Scrutiny's coolest trick (other than its main job of zipping through a website like a dose of salts and presenting loads of data).

It may not really be a hidden gem, because it's mentioned in lots of places and one of the main new features of version 7, but if you've just downloaded Scrutiny and run it, you may have missed it.

As the title of this post suggests, Scrutiny 7 allows you to open multiple windows and scan a site or do anything else you like within them, examine some data or configure the settings for another site.

Unsurprisingly, File>New (cmd-N) will open a new window for you to use as you see fit.

Google: "No more deceptive download buttons" (Feb 2006). Still waiting.

I've just come across this article from Feb 2006 on the Google Security Blog called No More Deceptive Download Buttons. It was mentioned in a recent article on the Google Webmaster Central Blog.

It's a problem that I've been very frustrated about in recent years. This is Scrutiny's page on MacUpdate. This is an incredibly important page for me; Scrutiny is my flagship product and MU are probably the leading mac app site outside of Apple's Mac App Store (which I have real problems with and don't support).

At the size you're seeing this screenshot, you'll find it hard to pick out the real 'download' link for Scrutiny, but you won't have any trouble seeing three massive download buttons which are unrelated.

My example here is from MacUpdate because of their importance to me, but the problem is everywhere, I'm sure you're more than familiar with it and are already skilled in hunting for the real link among the ads, or are blocking ads.

To be fair, the number of deceptive download buttons on that page does vary (There are more ads as you scroll down) and you get a different selection of ads every time you view the page (Google ads, via Doubleclick) but it's unusual for me not to see at least two big green download buttons high up that page.

It would be nice to see an end to this, but in these days of people expecting the product to be free, advertising comes into play, and advertisers will do what they can to attract that click.

Hidden gems: Scrutiny 7's Autosave feature

Some of Scrutiny's coolest tricks may not be terribly well documented, so I thought I'd start a series of articles here.

Previous versions of Scrutiny had an 'Autosave' checkbox. With the feature turned on, any data that was in memory would still be there after quitting (deliberately or unintentionally) and re-starting the app.

(Note that this option is in Scrutiny's Preferences, not the site settings.)

In Version 7, the same option is there but the app goes much further. With the feature switched on, Scrutiny will save the data from the most recent crawl of each site. This means all data; link check results, sitemap, SEO data, spell check, everything.

If crawl data exists for a site, then you'll see the database icon with the site's configuration. This takes you straight to the Results selection screen.

Since version 7.3 this feature is on by default for new users. So if you are scanning very large sites and short of disc space, you may want to switch the feature off (note that you can still save and load data manually using File > Save data and File > Open). But you may not want to switch Autosave off, because this window (below) shows the data that Scrutiny is keeping (with dates and sizes) and you can use this 'data manager' to selectively trash the data that you don't need. The table can be sorted by size, scan date or website name.

This is found under Tools > Manage Autosaved Data (cmd-5)

Updated video introduction to Scrutiny for Mac

The video introduction for Scrutiny had become a little out-of-date, it showed version 6. So here's an updated video showing the current version, 7.3.

It'll be of particular interest to users of Integrity or Integrity Plus, and anyone who develops or helps customers with SEO or other website issues.

Application home page http://peacockmedia.software/mac/scrutiny