Development environment for hand-coding websites - update

Moving from an app built for myself, to a product that I expect other people to use has been a much longer process than I would have imagined. The very long list of small fixes and enhancements makes me realise what I'm prepared to live with and work around.

The first public release happened a little while ago and yesterday it received an update with lots of rough edges smoothed off.

If it turns out that I'm the only person who wants to hand-code html/css/js then that's fine, my own tool is much nicer to use than it has been for most of its life. 

The current version is entirely free. Download is here. No card details, not even an email address. The only thing I do ask for is feedback.

The browser padlock and why it might not appear

It's important to have an SSL certificate these days if your site is to have any credibility.

Even if you do have a valid certificate in place, you may still find that a browser refuses to display the padlock. Different browsers have their own criteria and display the information in different ways, but we've generally moved from 'a padlock when the site is secure' to a clear 'site insecure' warning.

The image above illustrates this. The site does have a valid certificate in place.  My two favourite browsers do both have developer tools which allow you to drill down and find the reason(s) for the warnings.

That's good for a single page that you know has a problem. But if you're a Scrutiny user, you want to be notified of any such problems on any page of your site.

Scrutiny has long had features to help you with migration to https://. It alerts you to old links to your http:// pages and pages which have mixed content. (images or linked files which are http://)

As mentioned above, browsers vary in their criteria for displaying the padlock. As from v9.8.0, Scrutiny makes additional checks / warnings:

The insecure content alert/report will now include:

• insecure urls found in certain meta tags, such as open graph or Twitter cards.
• insecure images, whether hosted internally or externally
• insecure form action urls, even if the 'check form action' is switched off.

Testing website accessibility (WCAG / ADA compliance) using Scrutiny

No software can test your website and declare it ADA, or more specifically WCAG, compliant because some of the checks need to be made by a human or are subjective.

For example, is a heading, title or link text meaningful? Only a human can judge. But software can tell you whether headings and title of a reasonable length are present and thus report pages of possible concern.

Having said that, there are certain very important things that automated testing does do very well, such as checking for images without alt text.

With that in mind, here is a list of the ways that Scrutiny can help. The checkpoint numbers relate to the WGAC 2.0 requirements.



Alt text (1.1.1):  "non-text content that is presented to the user has a text alternative" (level A)

• Scrutiny can report images without alt text

Adaptable website structure (1.3.1, 1.3.2): Properly marked up and well-organised headings  (level A) and
Section headings (2.4.10) "Section headings are used to organize the content." (level AAA)

• Scrutiny can report pages with more than one h1 tag. For a specific page, it can show you the outline (ie just the headings, indented)

• Scrutiny's Robotize feature can display a 'text-only' view of a web page and let you browse the site, with headings and links listed separately. This is a good way to test this checkpoint.

Keyboard accessible (2.1): "Make all functionality available from a keyboard."

• if Scrutiny crawls a website fully (particularly with the 'run js' option switched off) then the navigation links are correctly-formed hyperlinks and it should be possible to tab through them using a keyboard and therefore navigate the site. (NB Scrutiny does not currently test / report form fields or buttons)

Page titles (2.4.2): "Web pages have titles that describe topic or purpose." "The title of each Web page should: Identify the subject of the Web page, Make sense when read out of context, Be short"  (level A)

• Scrutiny can report pages which have a non-unique title, and pages which have a title which is too short / too long.

Link text (2.4.4): "The purpose of each link can be determined from the link text alone or from the link text together with its programmatically determined link context" (level A)

• Scrutiny can report empty links as bad links (this option defaults to off, needs to be switched on)
• Scrutiny makes it easier to check manually for meaningful link text.  Cast your eye down the 'link text' column of the links / flat view (or sort the table by this column) and look for link text that doesn't explain the purpose of the link.

Descriptive headings and labels (2.4.6): "Headings and labels describe topic or purpose."  "Determine if the Web page contains headings. Check that each heading identifies its section of the content." (level AA)

• Scrutiny can report pages with no h1
• Scrutiny can include headings (h1's h2's etc in separate columns) in the SEO report to make it easier to scan them by eye and pick out non-descriptive ones

Parsing (4.1.1): Make sure HTML code is clean and free of errors, particularly missing bracket closes. Also, make sure all HTML elements are properly nested.

• Scrutiny provides a way to validate the html code for a specific page using the w3c validator. Your website is likely to be template-based (ie the code for the design of the page is likely to be identical throughout the site) then validating the home page (and certain other pages if the design varies for differnet types of page) is a good indication of validation throughout the site.
• Since version 10, Scrutiny runs more html validation tests of its own over every page while scanning. 

Website archiving utility, version 2

Watchman for Mac is a utility that can scan a single page or a whole site on schedule, it'll archive the page(s) and alert the user to any changes in the code or visible text.

As it builds its archive it's possible to browse the historical versions of the pages, as they stood on particular dates. It displays a 'living' version of the historical pages, with javascript and stylesheets also archived.

We've just made a version 2 beta available. It features a 'non-UI' or headless mode, which means that it can remain in the background and not interrupt the user when a scheduled scan starts. Its windows can still be accessed from a status bar menu.

https://peacockmedia.software/mac/watchman/

Version 1 is still available. It's free and will remain free. The new beta is also available and free at present, but version 2 will eventually require a licence.

New app - 'Time machine for your website'

[Edit 5 Dec - video added - scroll to bottom to see it]
[Edit 18 Dec - beta made public, available from website]
[Edit 10 Mar 2019 - version 2 made public]
[Important note 17 Mar 2019: Name of app has had to be changed slightly from Watchman to Website Watchman because of a conflict with a Linux open source title]

We kick off quite a few experimental projects. In most cases they never really live up to the original vision or no-one's interested.

This is different. It's already working so beautifully and is proving indispensable here, I'm convinced that it will be even more important than Integrity and Scrutiny.

So what is it?

It monitors a whole website (or part of a website, or a single page) and reports changes.

You may want to monitor a page of your own, or a competitor's or a supplier's, and be alerted to changes. You may want to simply use it as a 'time machine' for your own website and have a record of all changes. There are probably use-cases that we haven't thought of.

You can easily schedule an hourly, daily, weekly or monthly scan so that you don't have to remember to do it, and the app doesn't even need to be running, it'll start up at the scheduled time.

Other services like this exist. But this is a desktop app that you own and are in control of. It goes very deep. It can scan your entire site, with loads of scanning options just like Integrity and Scrutiny, plus blacklisting and whitelisting of partial urls. It doesn't just take a screenshot, it keeps its own record of every change to every resource used by every page. It can display a page at any point in time - not just a screenshot but a 'living' version of the historic page using the javascript & css as it was at the time.

It allows you to switch between different versions of the page and spot changes. It'll run a comparison and highlight the changes in the code or the visible text or resources.

It stores the website in a web archive, you can export any version of any page at any point in time as a screenshot image or a collection of all of the files (html, js, css, images etc) involved in that version of that page.

The plan was to release this in beta in the New Year. But it's already at the stage where all of the fundamental functionality is in place and we're using it for real.

If you're at all interested in trying an early version and reporting back to us, you can now download the beta from the website. [Edit: version 1 is now the stable release and is free, version 2 is also free and is in beta]

The working title has been Watchtower, but it won't be possible to use that name because of a clash with the 'Watchtower library' and related apps. It'll likely be some variation on that name.

HTMLtoMD and Webscraper - a comparison converting & archiving a site as MarkDown

I noticed a few people visiting the website as a result of searching for "webpage to markdown" and similar.

Our app HTMLtoMD was designed to do exactly that, and is free. But it was experimental and hasn't received very much development in a long time.

It still does its job, but our app Webscraper can also do that job, with many more options, and is a much more active product, it's selling and is under regular development now.

I thought it was worth writing a quick article to compare the output and features of the two apps.

First HTMLtoMD. It's designed for one job and so is simple to use. Just type or paste the address of your page or homepage into the address bar and press Go. If you choose the single page option, you'll immediately see the page as markdown, and can copy that markdown to paste somewhere else.

If you don't choose the single page option, the app will scan your site, and then offer to save the pages as markdown.

Here's the big difference with WebScraper. HTMLtoMD will save each page separately as a separate markdown file:

So here's how to do the same thing with Webscraper. It has many more options (which I won't go into here.) The advanced ones tend not to be visible until you need them. So here's WebScraper when it opens.

For this task, simply leave everything at defaults, enter your website name and make sure you choose "collect content as markdown" from the 'simple setup' as shown above. Then press Go.

For this first example I left the output file format set to csv.  When the scan has run, the Results table shows how the csv file will look.  In this example we have three field names;  url, title, content - note that the 'complex setup' allows you to choose more fields, for example you might like to collect the meta description too.

You may have noticed in one of the screenshots above that the options for output include csv, json and txt. Unlike HTMLtoMD, Webscraper always saves everything as a single file. The csv output is shown above. The text file is also worth looking at here. Although it is a single file, it is structured.   I've highlighted the field headings in the screenshot below. As I mentioned above, you can set things up so that more information (eg meta description) is captured.

[Update]

It occurred to me after writing the article above, that collecting images is a closely-related task. WebScraper can download and save all images while performing its scan. Here are the options, you'll need to switch to the 'complex setup' to access these options:

If you're unsure about any of this, or if you can't immediately see how to do what you need to do, please do get in touch. It's really useful to know what jobs people are using our software for (confidential of course). Even if it's not possible, we may have suggestions. We have other apps that can do similar but different things, and have even written custom apps for specific jobs.

Introductory offer on full release of 404bypass

Problem: you've moved your website. For various reasons the urls of existing pages may have changed.

Solution: a .htaccess file at the root of your old site which redirects old urls to new ones.

But there are a large number of such pages. Matching them up and compiling the redirect file is going to take time....

404bypass scans your old and new sites, 'smart matches' the pages and offers you a table of results. You can make any corrections manually before generating a .htaccess file, or a csv (or any other type of file you like, using a template configuration system). There are more details and screenshots here.

The beta period has come to an end, 404bypass is on full release now. For a limited period, it's available at an introductory price of $5 ($10 when it reverts to full price).

Download and try it here (30 day trial).

Crawling a website that requires authentication

This is a big subject and gets bigger and more complicated as website become increasingly clever at preventing non-human visitors from being able to log in.

My post How to use Scrutiny to test a website which requires authentication has been updated a number of times in its history and I've just updated it again to include a relatively recent Scrutiny feature. It's a simple trick involving a browser window within Scrutiny which allows you to log into your site. If there's a tracking cookie, that's then retained for Scrutiny's scan.

It used to be possible to simply log in using Safari - Safari's cookies seem to have been systemwide, but after Yosemite, a browser's cookies seem to be specific to that browser.

The reason for this all being on my mind today is that I've just worked the same technique into WebScraper. I wanted to compile a list of some website addresses from pages within a social networking site which is only visible to authenticated users.

Webscraper doesn't have the full authentication features of Scrutiny but I think this method will work with the majority of websites which require authentication.

(This feature, and others, are in Webscraper 1.3 which will be available very shortly)
SaveSave

Webscraper from PeacockMedia - usage

[updated 23 Apr 2018 for version 4]

[reviewed 29 Aug 2021]

I've had one or two questions about using WebScraper. There's a short demo video here  but if, like me, you prefer to cast your eye over some text and images rather than sit through a video, then here you go:

1. Type your website address (or starting url for the scan). Like Integrity / Scrutiny (Webscraper uses the same engine) the crawl will be limited to any 'directory' implied in the url.

2. Configure your output. If it's a single piece of information you want to extract from each page, you can use the Simple Setup. If you want to set up a number of columns, use the Complex Setup. Toggle between these two options below the address bar.

You must configure your output file before scanning, and then the app crawls your site, collecting the data as it goes. This is more efficient than the way that the first version of Webscraper worked but it does mean that if you want to change the configuration of your output file, you'll need to re-scan.

If you choose 'Complex setup' you'll need to configure your output file here. When you add a  column you can choose  basic metadata (title, description etc), a class or id, a regular expression (regex) or content (as plain text, html, markdown or an outline).

3. Test or run. You'll be able to either begin the scan, or run a short test. The 'Run test' button will perform a very short scan of a few pages and present your output as a quickview. If all looks well, you can press Go, or if you need to make changes, you can head back to the Output file configuration.

4. When the scan is complete, the Results tab will open. You can export this using the export button above the table. It uses the options you set in the 'Output file format' tab. 

Note that the Save Project option from the File Menu will only save your setup, not the scan data.

A common scenario is that the data you want isn't defined by a unique class or id. In these cases a regular expression can be used, there's a detailed tutorial here.

New Project - WebScraper for OSX

I love starting new things. This project uses the Integrity V6 Engine for the crawling which means that I could get right on and build the output functionality.

I noticed that this is something people have been trying to use Scrutiny's search functionality to achieve. Scrutiny will report which pages contain (or don't contain) your term in the text or the entire code. And you can export results to csv and choose columns.

But Scrutiny (currently) can't extract data from particular css classes or ids.

This is where WebScraper comes in. It quickly scan a website, and can output the data (currently) as csv or json. (Anyone want xml?) The output can include various meta data (more choices to be added), the entire content of each page (as text, html or markdown) and can extract parts of the pages (currently a named class or id of divs or spans).

Webscraper is new and in beta. Please use it for free and please get in touch with any requests, bug reports or observations.

http://peacockmedia.software/mac/webscraper/

There's a short demo video here