A sneaky peek at what's coming in early 2021

Scrutiny and Integrity have received constant attention and frequent updates (often weekly). 

But they feel like mature applications. They sell well, are well-used, we use them to provide services and yet reported problems are very few.

Requested features / functionality have been few too and we haven't had anything major on the radar for these apps. 

However, if you've followed recent developments you'll know that the 'warnings' functionality has been progressing. 

Traditionally, links are coloured orange if there's a redirection. The redirection itself is certainly not an error and users may or may not be concerned about them. (So warnings about redirections can be switched off.)

For some time, other things have been noted and reported as 'warnings' (and listed separately from the redirections in the link inspector) and people asked for a way to view / export these warnings. Scrutiny has been gaining new ways to view warnings (eg columns in tables, a tab in the link inspector). More recently warnings (again, distinct from redirections) have been presented in their own table.

This in itself is major new functionality. It has happened incrementally in later minor releases of version 9.  Version 10 marks what has so far been more of an evolution.

It can go much further. Way back, Scrutiny had the ability to pass every page that it scanned to the w3 validator. Although it was a little clunky internally, it was a very popular feature, and therefore frustrating when w3.org released the 'nu' validator which didn't have the API that allowed us to use it in the same way as before.

Because of the way that our crawling / parsing  engine works, it's easy to spot and report a few more html validation problems. We're working hard on this now. When the v10 beta is released after the new year, it will be able to report a number of important validation problems. It won't be a full html validator just yet, but the list of problems that it can report will grow through 2021.

The html validation functionality applies to Scrutiny and Integrity Pro only.

Scrutiny version 10 will be a free upgrade from the current version, although the word is that there may be a small price increase for 2021. (hint: if you're thinking about buying, now is a good time.) 

[update 8 Jan 2021] The first public beta of v10 is available for download. Important: It will be a free upgrade. It will ask for a key. If you already have a key it will accept your existing one. Just get in touch if you get a 'too many activations' message, the key just needs resetting.

[update 29 Aug 2021] Version 10 has been through some minor point updates and is stable. The warnings functionality continues to gain new warnings and be improved. 

~ Shiela

Changes to nofollow links : sponsored and ugc attributes : how to check your links

Google announced changes last year to the way they'd like publishers to mark nofollow links.

The rel attribute can also contain 'sponsored' or 'ugc' to indicate paid links and user-generated content. A while ago, nofollow links were not used for indexing or ranking purposes. But this is changing. Google will no longer treat them as a strict instruction to not follow or index.

This article on moz.com lists the changes and how these affect you.

As from version 9.5.6 of Integrity (including Integrity Plus and Pro) and version 9.5.7 of Scrutiny, These apps allow you to see and sort your links according to these attributes.

There was already a column in the links views for 'rel' which displayed the content of the rel attribute, and a column for 'nofollow' which displayed 'yes' or 'no' as appropriate. Now there are new columns for 'sponsored' and 'ugc' (also displaying yes/no for easy sorting). Many of the views have a 'column' selector . If visible, these columns will be sortable and they'll be included in csv exports.

New feature for Website Watchman and spin-off website archiving app

I wouldn't be without Website Watchman scanning the PeacockMedia website on schedule every day. As with Time Machine, I can dial up any day in the past year and browse the site as it appeared on that day. And export all files associated with any page if I need to.

It also optionally alerts you to changes to your own or anyone else's web page(s).

It took a lot of development time last year. I have (literally) a couple of reports of websites that it has trouble with but on the whole it works well and I think the tricks it does are useful.

Since it went on sale early this year, it has sold. But not in the numbers I'd hoped. Maybe it's just one of those apps that people don't know that they need until they use it.

Here's the interesting part. Of the support requests I've had, more have been on one question than any other. Frustrated emails asking how to make it export the entire site that they've just scanned.   What those people are after is an app which does a 'one-shot' scan of a website and then saves all the files locally. It's a reasonable question because WW's web page talks about archiving a website.

My stock answer is that WW is designed to do two specific things, neither of which is running a single scan and exporting the entire thing.  Like Time Machine, It does hold an archive, which it builds over time, and that's designed to be browsed within WW and any files that you need from a given date are recoverable.

There is of course quite a well-known and long-standing app which does suck and save an entire site. I've had conversations with my users about why they're not using that.

So knowing about those requirements, and owning software which already nearly does that task, the time seems right to go ahead.

I've mused for a while about whether to make an enhancement to Website Watchman so that it can export the files for an entire site (remember that it is designed to make multiple scans of the same website and build an archive with a time dimension, so this option must involve selection of a date if more than one scan has been made) or whether what people are after is a very simple 'one-trick' app which allows you to type a starting URL, press Go, choose a save location and job done.

I've decided to do both. The new simple app will obviously be much cheaper and do one thing. Website Watchman will have the new full-site export added to its feature list. I'm using WebArch for the new app as a working title which may stick.

So where are these new things?

It's been more than a week since writing the news above. Even though we already had toes in this water with the archive functionality (such as it is) in Integrity Plus and Pro and Scrutiny, it turns out that it's no easy thing to make a local copy of a website browsable. In some cases it's easy and the site just works. But each site we tested was like opening a new Pandora's box, with new things that we hadn't taken into account. It's been an intense time but I'm really happy with the stage it's at now. I'd like to hear from users in cases where the local copy isn't browsable or problems are seen.

The new functionality is in Website Watchman from version 2.5.0. There's a button above the archive browser, and a menu item with keyboard shortcut under the File menu.

The new simple app is WebArch and is here for free download.

Press Release - Integrity Pro v9 released

Integrity Pro version 9 is now fully released. It is a free update for existing licence holders.

The major new features are as follows:

• Improved Link and Page inspectors. New tabs on the link inspector show all of a url's redirects and any warnings that were logged during the scan.

• Warnings. A variety of things may now be logged during the scan. For example, a redirect chain or certain problems discovered with the html. If there are any such issues, they'll be highlighted in orange in the links views, and the details will be listed on the new Warnings tab of the Link Inspector.
• Rechecking. This is an important part of your workflow. Check, fix, re-check. You may have 'fixed' a link by removing it from a page, or by editing the target url. In these cases, simply re-checking the url that Integrity reported as bad will not help. It's necessary to re-check the page that the link appeared on. Now you can ask Integrity to recheck a url, or the page that the url appeared on. And in either case, you can select multiple items before choosing the re-check command.
• Internal changes. There are some important changes to the internal flow which will eliminate certain false positives.

More general information about Integrity Pro is here:
https://peacockmedia.software/mac/integrity-pro/

Press Release - Scrutiny 9 Launched

Scrutiny version 9 is now fully released. It is a free update for v7 or v8 licence holders. There is a small upgrade fee for holders of a v5 / v6 licence. Details here: https://peacockmedia.software/mac/scrutiny/upgrade.html

The major new features are as follows:

• Improved Link and Page inspectors. New tabs on the link inspector show all of a url's redirects and any warnings that were logged during the scan.

• Warnings. A variety of things may now be logged during the scan. For example, a redirect chain or certain problems discovered with the html. If there are any such issues, they'll be highlighted in orange in the links views, and the details will be listed on the new Warnings tab of the Link Inspector.
• Rechecking. This is an important part of your workflow. Check, fix, re-check. You may have 'fixed' a link by removing it from a page, or by editing the target url. In these cases, simply re-checking the url that Scrutiny reported as bad will not help. It's necessary to re-check the page that the link appeared on. Now you can ask Scrutiny to recheck a url, or the page that the url appeared on. And in either case, you can select multiple items before choosing the re-check command.
• Internal changes. There are some important changes to the internal flow which will eliminate certain false positives.
• Reporting. The summary of the 'full report' is customisable (in case you're checking a customer's site and want to add your own branding to the report). You now have more choice over which tables you include in csv format with that summary.

A full and detailed run-down of version 9's new features is here:
https://blog.peacockmedia.software/2019/05/scrutiny-version-9-preview-and-run-down.html

More general information about Scrutiny is here:
https://peacockmedia.software/mac/scrutiny/

Press release - Slight change of name and full launch for Website Watchman

Press release
Mon 25 Mar 2019


begins---------------

Watchman from PeacockMedia is now fully released and it's now called Website Watchman. The original name clashed with another project, but Website Watchman better sums up the app.

It'll be useful to anyone with a mac and a website. It does a very impressive trick. As it scans a website and checks all of the files that make up a page; html, js, css, images linked documents. It archives any changes. So the webarchive that it builds is four-dimensional. You can browse the pages, and then browse the versions of that page back through time. You view a 'living' version of the page, not just a snapshot.

As the name suggests, you can watch a single page, part of a website or a whole website, or even part of a page, using various filters including a regex, and be alerted to changes.

Thank you for your attention and any exposure that you can give Website Watchman.

https://peacockmedia.software/mac/watchman

ends---------------

Website archiving - Watchman's commercial release

[NB since version 2.1.0, we have had to make a slight change to the name, its full title is now Website Watchman.]

It has been a (deliberately) long road but Watchman for Mac now has its first commercial release.

This product does such a cool job that I've long believed that it could be as important to us as Integrity and Scrutiny. So I've been afraid to rush things. Version zero was officially beta, and a useful time for discovering shortcomings and improving the functionality. Version one was free. Downloads were healthy and feedback slim, which I take as a good sign. Finally it's now released with a trial period and reasonable introductory price tag. Users of version one are welcome to continue to use it, but it obviously won't get updates.

So what does it do? In a few words. "Monitor and archive a website".

There are apps that monitor a url and alert you to changes. There are apps that scan an entire website and archive it.

Watchman can scan a single page, part of a website or a whole website. It can do this on schedule - hourly, daily, weekly, monthly. It can alert you to changes. It builds a web archive which you can view (using Watchman itself or the free 'WebArchive Viewer' which is included in the dmg). You can browse the urls that it has scanned, and for each, view how that page looked on a particular day.

We're not talking about screenshots but a 'living' copy of the page. Watchman looks for and archives changes in every file, html, css, js and other linked files such as pdfs.  You can obviously export that page as a screenshot or a collection of the files making up that page, as they stood on that date.

A 'must have' for every website owner?

Try Watchman for free / buy at the introductory price.

New website monitor / archive utility for Mac arrives at full stable release and is still free

Watchman is an easy-to-use website monitoring / archiving utility.

You can use it to watch a single page, a part of a website or a whole website. It can run on schedule (hourly, daily, weekly, monthly) and alert you to the changes you're interested in, which could be visible text, source, resources appearing on the page, its status, or you can simply leave it to archive all changes to all files. You can set up multiple website configurations, each with their own schedule. It uses the system's launchd, meaning that Watchman doesn't have to be left running, it'll just start as needed.

Watchman uses the same fast, efficient crawling engine as Scrutiny and Integrity, which has been developed over 12 years and offers a huge amount of configuration and tuning. This is coupled with a new web archive format.

Its web archive format can store changes like a Time Machine backup. You can view any page as it appeared on a certain date. When you do so, you're viewing a 'living' version of the page, with its css and javascript running as it was at the time, not a simple screenshot. You can of course export a version of a page as an image, or as a collection of all the files under their original filenames, as they were on that date. You can switch between versions of a page to compare them.

It's a desktop app running on your own Mac, so you own your own data.

It's early days, there are many more features in the pipeline, but for now it's stable and doing invaluable work. And version 1.x is free to download and use. (The next major version may not be free, or may be 'freemium' but the current version will continue to work and remain free.)

I've been flagging the release of Watchman for a while, it's been a long time since I've been so excited about a new project and I believe it'll become a more important title for us than Scrutiny.

https://peacockmedia.software/mac/watchman

New app - 'Time machine for your website'

[Edit 5 Dec - video added - scroll to bottom to see it]
[Edit 18 Dec - beta made public, available from website]
[Edit 10 Mar 2019 - version 2 made public]
[Important note 17 Mar 2019: Name of app has had to be changed slightly from Watchman to Website Watchman because of a conflict with a Linux open source title]

We kick off quite a few experimental projects. In most cases they never really live up to the original vision or no-one's interested.

This is different. It's already working so beautifully and is proving indispensable here, I'm convinced that it will be even more important than Integrity and Scrutiny.

So what is it?

It monitors a whole website (or part of a website, or a single page) and reports changes.

You may want to monitor a page of your own, or a competitor's or a supplier's, and be alerted to changes. You may want to simply use it as a 'time machine' for your own website and have a record of all changes. There are probably use-cases that we haven't thought of.

You can easily schedule an hourly, daily, weekly or monthly scan so that you don't have to remember to do it, and the app doesn't even need to be running, it'll start up at the scheduled time.

Other services like this exist. But this is a desktop app that you own and are in control of. It goes very deep. It can scan your entire site, with loads of scanning options just like Integrity and Scrutiny, plus blacklisting and whitelisting of partial urls. It doesn't just take a screenshot, it keeps its own record of every change to every resource used by every page. It can display a page at any point in time - not just a screenshot but a 'living' version of the historic page using the javascript & css as it was at the time.

It allows you to switch between different versions of the page and spot changes. It'll run a comparison and highlight the changes in the code or the visible text or resources.

It stores the website in a web archive, you can export any version of any page at any point in time as a screenshot image or a collection of all of the files (html, js, css, images etc) involved in that version of that page.

The plan was to release this in beta in the New Year. But it's already at the stage where all of the fundamental functionality is in place and we're using it for real.

If you're at all interested in trying an early version and reporting back to us, you can now download the beta from the website. [Edit: version 1 is now the stable release and is free, version 2 is also free and is in beta]

The working title has been Watchtower, but it won't be possible to use that name because of a clash with the 'Watchtower library' and related apps. It'll likely be some variation on that name.

New free flashcard / Visualisation & Association method for MacOS

Vocabagility is more than a flashcard system, it's a method. Cards are selected and shuffled, one side is shown. Give an answer, did you get it right? Move on. As quick and easy as using a pack of real cards in your pocket.

The system also encourages you to invent an amusing mental image linking the question and answer (Visualization and Association)

Cards that you're not certain about have a greater probability of being shown.

This is an effective system for learning vocabulary / phrases for any language but could be used for learning other things too.

Download Vocabagility for Mac for free here.

Dark mode and Integrity / Integrity Plus / Integrity Pro / Scrutiny

I have to admit that I really love dark mode. It's very easy on the eye and it jarrs a little when you have to look at a web page with a white background.

(Does anyone know whether it's possible for a website to detect a mac's dark mode setting and display a dark version using an appropriate css? Let me know.)

I did naively expect that the OS would simply draw all windows and controls in the dark colours. But it's up to each developer to build their apps under the new SDK. And carefully check for hard-coded colours and unsuitable images within the app.

We're just about there with Integrity / Integrity Plus / Integrity Pro / Scrutiny  and it's been a pleasure to do.  The 'dark mode enabled' version of all of this family of apps will be 8.1.5 (a minor point release, as there are very few functional changes).

[update 26 July 18 and again 8 Aug 18] Scrutiny 8.1.8 is available and looks great under dark mode. Obviously will only look dark on 10.14 with dark mode selected.

[update 30 July 18 and again 8 Aug 18] Integrity, Integrity Plus and Integrity Pro are also available with dark mode enabled. As above, 10.14 is required to see them in dark mode

Migration to secure (https://) site

This is an older article. The information in it is still correct but there's a newer article here with new screenshots and revised content.

This has been a big week but peacockmedia.software is now https://

This is a well-overdue move. Google have been offering small carrots for a long time, but at the end of this month, they'll be adding a stick as well. They're switching from informing users when a connection is secure, to warning users if a connection is insecure. Google Chrome is making this move but other browsers are expected to follow suit.

Well-informed web users will know whether they really need a connection to be secure or not, but I suspect that even for those users, when this change really takes hold, the red unlocked padlock will start to become an indicator of an amateur or untrustworthy site.

Once the certificate is installed (which I won't go into) then you must weed out links to your http:// pages and pages that have 'mixed' or 'insecure' content, ie references to images, css, js and other files which are http://.

Scrutiny makes it easy to find these.

1. Find links to http pages and pages with insecure content.

First you have to make sure that you're giving your https:// address as your starting url, and make sure that these two boxes are ticked in your settings,

and these boxes ticked in your Preferences,

After running a scan, Scrutiny will offer to show you these issues,

You'll have to fix-and-rescan until there's nothing reported. (When you make certain fixes, that will reveal new pages to Scrutiny for testing).

2. Fix broken links and images

Once those are fixed, there may be some broken links and broken images to fix too (I was copying stuff onto a new server and chose to only copy what was needed. There are inevitably things that you miss...) Scrutiny will report these and make them easy to find.

3. Submit to Google.

Scrutiny can also generate the xml sitemap for you, listing your new pages (and images and pdf files too if you want).

Apparently Google treats the https:// version of your site as a separate 'property' in its Search Console (was Google Webmaster Tools). So you'll have to add the https:// site as a new property and upload the new sitemap.

[update 15 Jul] I uploaded my sitemap on Jul 13, it was processed on Jul 14.

4. Redirect

As part of the migration process, Google recommends that you then "Redirect your users and search engines to the HTTPS page or resource with server-side 301 HTTP redirects"  (full article here)

Announcing Vinyl Shine for Mac

This is probably the most satisfying new release ever from NPD at PeacockMedia Towers. It represents a new direction of exploration for us.

Work on this was prompted by me failing to find a plugin component that could be used by my favourite sound recorder as a filter to remove pops and crackles, or a plugin for a player to filter the pops in real time during playback.

The work has involved a steep learning curve with Apple's AVAudioEngine. The documentation is far from thorough and there's very little sample code around. That has not been fun but learning more about sound processing (DSP) and getting hands-on with some C++ really has been enjoyable.

We now have a standalone app that will open a sound file, apply pop filtering (pretty quickly - currently <6s for a 4 minute track, but I hope to significantly improve this).

The pop filter is applied, along with EQ and normalisation. For some reason, LPs seem to vary quite widely in sound (I think this is a different thing from playback equalization curves for early LPs and 78s.) Either way we intend that the graphic EQ in Vinyl Shine be flexible enough to handle all of this.

Options are minimal. A commercial app I currently use for pop filtering works ok but the result isn't perfect and it offers a shedload of options. What do those options mean? What am I supposed to change? Surely a pop is a pop, identify and remove it. Fewer options is the Mac way.

Vinyl Shine allows you to listen to the result and toggle between original and cleaned audio. And save the result as a new file when you're happy. (The final offline render is very quick, ~3s for a 4m song).

For the time being it currently exists as a standalone app and is working pretty well, making crackly recording much more listenable. It'll shortly be available as a free beta.

pops, clicks and crackles are identified, highlighted visually and repaired 

Option to download images in WebScraper

WebScraper for Mac now has the option to download and save images in a folder, as well as, or instead of, extracting data from the pages.

WebScraper crawls a whole website and it can extract information based on a class or id, or dump the content as html, plain text or markdown.

By just checking that box and choosing a download location, it can also dump all images to a local folder. That includes those found linked (the target of a regular hyperlink), those in a <img src , and those found in an srcset.

If you only want particular images, you can select only those which match a partial url or regex.

This functionality is in version 4.1.0 which is available now.

Webscraper version 4 released

Webscraper from PeacockMedia makes it quick and easy to extract data from a website.

It has some 'simple setup' options if you simply want to archive content or collect certain meta data. For more complex tasks you can add columns to your output file which extract data based on certain class or id, or even by regular expression (regex). (A tutorial is here).

version 4 is now released which has been updated to use the Integrity v8 engine. That makes some efficiency improvements, and allows better control if you need to limit the requests that WS makes to your server:

The app has a very modest price tag of $10 and you can use it for free to test whether it works for you.

Scrutiny version 8 released

Some serious 'under the hood' improvements have been made in Scrutiny version 8. Benefits include more efficient scanning of your website and more data collected.

we've invested a lot of time in something that isn't a killer new feature, but will make things run more smoothly, efficiently and reliably, and give you more information.

There have been a number of requests to add more information about the properties of a link , eg hreflang and more of the 'rel' attributes. HTML5 allows loads of allowable values in the 'rel' and if the one you want doesn't have its own yes/no column in the new apps then you can view the entire rel attribute. You can switch these columns on in any of the link tables as well as the one in the link inspector.

The information displayed in Scrutiny's SEO table will include more data about the pages, notably og: and twitter card meta data which is important to a lot of users.

Redirect information wasn't stored permanently, only the start and end of a redirect chain (the majority are a single step. But if there is more than one redirect, you'll want to know all the in-between details)

It's becoming increasingly necessary to apply some limits on the rate of the requests that Integrity / Scrutiny make.

Your server may keep responding no matter how hard you hit it. That's great; turn up those threads and watch Integrity / Scrutiny go through your site like a dose of salts.

If you do need to limit the scan, you'll no longer have to turn down the threads and use trial and error with the delay field. You'll be able to simply enter a maximum number of requests per minute, while still using multiple threads for efficiency.

Bag a bargain! Scrutiny included in Creatable bundle

The Creatable Pick A Bundle 2018 contains 28 varied utilities, including our own Scrutiny website scrutineer. You can choose ten for $39 or all of them for $99.

If you're a user of Philips Hue or LIFX bulbs, our own Mac apps for these lighting systems are also included.

So that's Scrutiny plus nine other useful apps for less than half the regular price of Scrutiny.

Check out the bundle now
SaveSave

Big news - Integrity Pro

Recently I wrote about the new releases of Integrity and Integrity Plus and how, despite being +2 major version numbers, they're unlikely to get many people this excited.

Something that I hope will light more fires is Integrity Pro! It's not intended as a replacement for Scrutiny, but it will bring the SEO and spellcheck functionality into Integrity and be priced midway between Integrity Plus and Scrutiny.

Thus making it a natural and affordable step for Integrity Plus users. There will be a fair upgrade path for Integrity Plus users, and free for any licensed Scrutiny user who wants the more classic interface and doesn't need the full weight of Scrutiny.

Integrity and Integrity Plus v8 are available now.

Update: Integrity Pro is available in full now.

SaveSave

Black Friday / Cyber Monday offers on Scrutiny for Mac and Integrity Plus

50% off Scrutiny

For Black Friday / Cyber Monday

If you're a user of Integrity or Integrity Plus, or have trialled Scrutiny, we hope that this discount will help you to make the decision to add Scrutiny for Mac to your armoury. 

Simply buy through the app or Scrutiny's home page, the discount will be applied. Exp 28-11-2017


The offer will run for the next week or so, please feel free to share this offer, or use it to buy more licences if you have multiple users.

50% off Integrity Plus

For Black Friday / Cyber Monday

If you're a user of the free Integrity, you may like to learn about the extra power that Integrity Plus gives you. Search / filter / export your link results, Generate an XML sitemap and more. 

This offer is being run by MacUpdate, visit Integrity Plus's page at MacUpdate to take advantage. Exp 28-11-2017

The offer will run for the next week or so, please feel free to share this offer or use it to buy more licences if you have multiple users.

Reminder - Scrutiny in current Bundlehunt app bundle

[Edit 14/6/17] this bundle has now finished.

A reminder that Scrutiny is included in the current bundle at Bundlehunt for a few more days only. For a fraction of the price of Scrutiny, you can obtain it along with 11 other Mac apps.