Scrutiny and WIX sites

Wix make it easy to create a website, but their reliance on javascript and plugins (including Flash apparently) makes it a challenge for webcrawlers, including Googlebot, to index the content.

It seems that Google have had to go out of their way to index these sites. There used to be a hack to get at plain html content, which I used to give as a workaround to users wanting to scan Wix sites, but that no longer appears to work.

As someone with a background in website accessibility, this all seems appalling to me*.

But they're gaining ground, I seem to be seeing more support questions about scanning WIX sites. (and their *!^%$ advert that I have to skip before almost every Youtube video I watch. When is Yotube Red going to be available in the UK??)

Unless anyone knows otherwise (please, please tell me if you know), WIX relies on a/ providing its users with an xml sitemap for submission to Google, and b/ Google being able to render js before indexing a page (again, as an old-fashioned gal, something I heartily disapprove of* )

Scrutiny has long been able to render javascript before parsing page content. There have always been a small number of sites which have menus and links that are written by js when the page loads. I've recently done some work to ensure proper scanning of such a site.

So... here's my best advice for scanning a Wix site using Scrutiny (Integrity and Integrity Plus don't have the necessary js option).

1. Make sure you have Scrutiny 7.4.1 or higher, which contains some necessary improvements to the js functionality. The most up-to-date version of scrutiny is always here.

2. Switch on 'Render Javascript' which is in the Advanced settings for your site.

3. Turn down your number of threads. You can see I'm using 3 here (first notch). Using the js option is very heavy in terms of client-side processing. You don't want pages timing out without having loaded because Scrutiny is working hard at rendering a large number of pages. If your website scans fully then maybe edge this up a bit but best to err on the side of caution.

4. Go and make a cup of tea. This is going to take some time.
5. After the scan, switch on the Target Size column of the 'by link' table and check whether any of the Internal links have a target size of zero (switch the Filter button to Internal to see this more easily). If an internal page has zero kb as the target size, then it may have timed out before the page had finished loading / rendering, which means that you may have an incomplete scan.

If that looks well then you're good to go with your data, which should be complete.



* It's not the javascript I disapprove of, but the fact that the page is 'invisible' without the js being rendered. Accessibility guidelines state that we must "Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page." (Web content accessibility guidelines, checkpoint 6.3 -  a priority 1)
SaveSave

Sharpresize v2 now fully released

SharpResize v2 is now out of beta. It's a lightweight application (ie quick to start up) for making a better thumbnail / resized for web  image. The great result is due to the combination of the resizing algorithm it uses along with an additional sharpening slider.

Despite being a lightweight app it now has a good selection of adjustment sliders and built-in filters. (sepia, black and white, posterize etc).

We've decided to keep the price down. In fact as it's now priced in dollars (1.95 USD) it's a little less in real terms than the previous GBP price.

More information and the download are here

The guide is here.

Alongside our recent post about supporting International Domain names within our web crawling tools, we're very proud and excited to announce that work has started on translating some apps and some web pages into other languages. Initially French, and initially Integrity, Integrity Plus and Scrutiny.

It's impossible to do all of the work at once, it will take a little while but the web pages for Integrity, Integrity Plus and Scrutiny now allow you to choose (top-right) English or French versions of the pages.

Localized context help in French is shortly to go into those apps, followed by the rest of the text within the apps.

When is a link internal.....

Here's an interesting situation.

The website being scanned was http://peacockmedia.software.  As you can see, the link found during the crawl is a link to a page on peacockmedia.co.uk . That looks like an external link, and so Scrutiny has marked it as an external link, therefore checked its status but not followed it.

However, it redirects to a url on peacockmedia.software - so the ultimate destination is an internal one.

There are a bunch of pages which will only be discovered if this link is considered internal.

(the pages are old manual pages - I want them to be orphaned, but that's beside the point...)

The question is - should Integrity and Scrutiny consider that link *internal* (and therefore follow it and discover more internal pages) or *external* (so it's checked but then is a dead end). Remember that the original link is to an external domain.

Heads-up : Internationalised Domain Names (IDNs) supported in our web crawlers

We're a UK-based concern, our apps have been almost always available in a single language - UK English (or just English as we call it here in England!)  The vast majority of our users as I write this are from English-speaking countries.

Our alphabet entirely consists of characters available in ascii, and so there has been little call for Integrity, Integrity Plus and Scrutiny (and other tools based on the same engine) to support domain names - ie domain names which contain characters not found in the ascii character set.

But now we've started work on localisation of our apps and web pages, and have received the odd question concerning IDNs.

Let's not confuse this with unusual characters in the path and filename of the url. Our apps have long supported these. You may still see the non-ascii characters displayed, but behind the scenes, those characters are encoded before the http request is put together, usually using a percent-encoding system.

The method is similar with the domain name, but using a a more complex and clever system of character encoding. Browsers (and our web crawlers) often still display the user-friendly unicode version.

You can enter your starting url in the unicode form or the 'punycode' form and it'll be handled correctly. The same goes for unicode or punycode links found on your pages.

Personally, I'm not keen, this does allow for spoofing of legitimate domains using similar characters. There are rules excluding many characters for these reasons.

After lots of extra homework for us, Scrutiny is now handling IDNs, and is in testing.

[update 26 May 2017] Integrity and Integrity Plus also have this enhancement and are also in testing.

If this is useful for you, and you'd like to try the new version (remembering that there may still be the odd bug to iron out)  then you're very welcome to download and use it (with the condition that you let us know about any issues you spot.)

[edit - links taken out, out of date]

List of all of a site's images, with file sizes

A recent enhancement to Scrutiny and Integrity make it easy to see a list of all images on a site, with file size.

It was already possible to check all images (not just the ones that are linked, ie a href = "image.jpg" but the actual images on the page, img src = "image.jpg"  srcset = "image@2x.jpg 2x" )

The file size was also held within Scrutiny and Integrity, but wasn't displayed in the links views.

Now it is. It's a sortable column and will be included in the csv or html export.

Before the crawl, make sure that you switch on checking images:

or

You may need to switch on that column if it's not already showing - it's called 'Target size'.

Once it is showing, as with other columns in these tables, you can drag and drop them into a different order, and resize their width.

To see just the images - choose Images from the filter button over on the right (Scrutiny and Integrity Plus)

If you're checking other linked files (js or css) then their sizes may be displayed, but will probably have a ? beside them to indicate that the file size shown has not been downloaded and the uncompressed size verified (the size shown is provided in the server header fields).

This last point applies to Integrity and Integrity Plus, and will appear in Scrutiny shortly.

Note that all of this is just a measure of the sizes of all files found during a crawl. For a comprehensive load speed test on a given page, Scrutiny has such a tool - access it with cmd-2 or Tools > Page Analysis

Post-apocalyptic life skills : handwriting

Self-service is great, if all goes as planned then it's efficient and easy for both customer and seller. (Though I still resist the self-service tills at my local convenience store - I think I care too much about the 4 or 5 jobs that have clearly gone redundant as a result. Plus - the store is reaping the benefit of the customer doing those ladies' jobs, without paying the customer in any way for doing that job.)

Anyway. When it doesn't go as planned is when the fun and games start.

Today I found myself needing to get back to a customer urgently, but the email address bounced as 'user not known'. Leaving me with a postal address.

The old business-letter-writing skills came back pretty easily; my address on the right, their address below and to the left, sincerely if you know their name, faithfully if you don't etc.

(I'm an old-fashioned kinda girl. Printing off the attempted email and putting that in an envelope wasn't going to happen.)

I was wrong to think that it would be quicker to write the address on the envelope than to work out how to print it on a label or put the envelope through the printer.

I draw a lot, but I don't write very much at all now. My handwriting was never great, it seems to be appalling now. After several practice goes, I got a result that wasn't as hideous as the first ones.

.... then turned it over and found that the address was upside-down! How crap does that look?

I'm not going to be among the survivors after the aliens come and hit us with a big EMP device.