Monday, 18 March 2019

Composing tune for C64 game - part 3 - generating assembly straight from sheet music

This post is going to combine musical notation, music XML, cocoa programming (objective-C for MacOS) and retro game programming (specifically the music) in assembly for C64.

I'm guessing that might make this an extremely niche post, so if you're reading because you're interested in all or some of those things, or if there's some other reason why you'd like to convert sheet music to another format, do let me know if the comments so that I know I'm not entirely alone!

In the last couple of posts in this series I wrote about how I was tinkering around with a little tune for a game I'm writing for C64. Writing out the notation is something I'm very comfortable with.

I eventually arrived at a great and efficient way to store and play the music within my game. This involved using a kind of 'bytecode', combining instructions and data. That looks like this:
That looks pretty laborious to write out by hand, particularly looking up the high and low oscillator settings for each note. And it is.

Anyway, that's done for this game, I have something I'm pretty happy with, other than the voice settings. You can hear that in the last part of this series if you're interested.

But with future composing efforts in mind, it occurred to me that I could seriously turbocharge this process, and pretty much generate the assembly language straight from the sheet music.

First step: parse the music XML (which any music notation software should be able to export to). At first, to save time, I downloaded an open-source class for turning XML into NSDictionary. That appeared to work but after some work I realised that it wasn't handling self-closing tags in the XML, which my musicXML was full of. So that wasn't a time-saver and to be fair the author of the xml -> NSDictionary class has made it clear that it's now out of date and unsupported.

So back to square one and writing code using NSXMLParser. That involves writing a lot of code, but it's useful in some ways - it serialises the data, you can take what you want from it and ignore what you don't want. It was probably no more code than I had to write to extract the data after the XML had been converted to an NSDictionary.

Here's where we are, in one afternoon (including the abandoned effort and restarting) I can open any musicXML file and see this:
The next step is converting those note values to the actual SID chip values, and then we're a short step away from getting this little programme to generate the assembly that I can simply drop into a game.

Saturday, 16 March 2019

SID tune for C64 homebrew game - part 2

The development of the music for this game has gone so well - way better than the development of the gameplay itself, which I'm having serious doubts about!

Writing a silly tune was the easiest part. I've now improved that - it has a 'chorus' which is deliberately reminiscent of 'English Country Garden'. The main part of the tune is meant to accompany sheep trotting / blundering around a field.

It even has a fourth part now, although I can only use three because the C64's sound chip - SID - has three tone voices.
And here's how it sounds using the Vice C64 emulator:

Disclaimer: Tonight I've been messing around with voices and ASDR envelopes, and haven't yet found settings that I really like. The bass in particular is raspy here, I want a smooth sound, but when I use the triangle waveform it sounds very quiet even with the highest sustain volume.

In the first part of this little diary, I mentioned that I started to use Derek Morris' sound library. It allowed me to 'code' the tune with his user-friendly constants.
It worked with three voices at once but I had trouble with the synchronisation. At first I devised a method of breaking the music down into bars, and having the main program loop kick off each bar of the three voices at the same time. That's why there's a cmdEnd at the end of each bar in the code above.

That worked. To a point. The music was in sync at the start of each bar. Within each bar, things got a little rough, and I started to fiddle with the delays on certain notes to try and straighten that out. That worked. To a point. That was all a bit of a fudge, so I carefully read DM's code to make sure that it was doing what I assumed it was doing. It really was, and it's beautifully-written (I don't mean to sound patronising. I mean that I've learned so much by reading his code and his book.)

But after manually following the loops in his code around and around as if I were the processor, noting down variable values on paper, I found that the delay loops were delaying for one more cycle than I thought they were. It's to do with counting down to zero. You get one more delay than the number you specify after the CmdDelay. I wouldn't call this a bug, Mr Morris designed this for playing a sound effect on a single voice so this point is insignificant in that context.

It took a single 'dec' instruction in the right place, and the soundlib is playing my music perfectly. (although faster than before, so I've had to re-write the delay values in my sound command files so that there are 48 in each bar rather than 32). I no longer need my 'bar-by-bar' system to keep everything in sync, because as long as the delay count is the same in each bar, everything will stay in sync. That saves one byte per bar per voice, plus my sync routine and the execution time of that each programme loop.

Manually translating the notated music to the sound library instructions is very laborious. Particularly looking up the high and low note values for each note in Appendix E of the Prog Ref Guide (I bought a copy of the C64 one in the end). I'm thinking about how I might write a tool(s) to help with this, because I've really enjoyed the composition part and want to do a lot more. Possibly even a mac app to take the sheet music (as musicXML) and build those sound commands. 

Wednesday, 13 March 2019

Raspberry Pi Zero W - baby steps

I don't know why I've waited so long to do this. I love messing around with home automation, and here's a fully-functional computer with wireless and 20-odd GPIO (input-ouput) pins.

I've also been meaning to begin using Linux (technically I already do, I use MacOS's Terminal a lot, and have set up and run a couple of Ubuntu servers with python and mysql). The Pi has a desktop, USBs for keyboard and mouse, HDMI output. This one has 1Ghz processor 256Mb of ram and for a HD, whatever free space is on the card that you put in. All of this on a board which is half the size of a credit card and costs less than a tenner (or less than a fiver for the non-wireless version).

When we finally invent time travel, or at least find a way to communicate across time (as per Gibson's The Peripheral which I heartily recommend)  my teenage self will be astounded by this information. I remember the day when I first heard the word 'megabyte'. It wasn't far off the day that I felt very powerful after plugging an 8k expansion into my computer.

Anyhow, back to the plot. What I've learned so far is that that the 'less than a tenner' Raspberry Pi Zero W is 'bare bones'. I've bought a few bits and pieces that have cost much more than the computer(!) including a header for the GPI pins, a breadboard & components kit, pre-loaded micro SD (effectively the HD and OS), mini HDMI to proper-size HDMI adaptor.

Monday, 11 March 2019

Website archiving - Watchman's commercial release

[NB since version 2.1.0, we have had to make a slight change to the name, its full title is now Website Watchman.]

It has been a (deliberately) long road but Watchman for Mac now has its first commercial release.
This product does such a cool job that I've long believed that it could be as important to us as Integrity and Scrutiny. So I've been afraid to rush things. Version zero was officially beta, and a useful time for discovering shortcomings and improving the functionality. Version one was free. Downloads were healthy and feedback slim, which I take as a good sign. Finally it's now released with a trial period and reasonable introductory price tag. Users of version one are welcome to continue to use it, but it obviously won't get updates.

So what does it do? In a few words. "Monitor and archive a website".

There are apps that monitor a url and alert you to changes. There are apps that scan an entire website and archive it.

Watchman can scan a single page, part of a website or a whole website. It can do this on schedule - hourly, daily, weekly, monthly. It can alert you to changes. It builds a web archive which you can view (using Watchman itself or the free 'WebArchive Viewer' which is included in the dmg). You can browse the urls that it has scanned, and for each, view how that page looked on a particular day.

We're not talking about screenshots but a 'living' copy of the page. Watchman looks for and archives changes in every file, html, css, js and other linked files such as pdfs.  You can obviously export that page as a screenshot or a collection of the files making up that page, as they stood on that date.

A 'must have' for every website owner?

Try Watchman for free / buy at the introductory price.

Friday, 8 March 2019

SID tune for C64 homebrew game - part 1

It's funny how interests from your teens remain with you, how easy it is to pick them up again and how comfortable they feel.

I'll probably write more about the revival of this hobby and my collection of 8-bit computers. Let's just say for now that I've begun a C64 puzzle game based on my character Yvonne the Sheep and her friends - and enemies.

Writing for 6502 (or 6510 in this case, which makes no difference as far as the assembly language is concerned) is like putting on an old familiar comfortable jumper.

I wasn't particularly keen on doing music, I did think about just forgetting it and just going with sound effects.

But a silly tune came into my head that seemed suitable for sheep pootling around a field. I'm not a composer of any great experience but I am at home with sheet music. Notating it (in 3 voices and possibly a fourth later) was the easiest way for me to get the tune out of my head and onto 'paper'.

Before I knew it I was really hooked on this aspect of the game.
Before I started to write my code I had bought Derek Morris' book, Retro Game Dev: C64 Edition. (Most of my teen experience was ZX81, Vic20, Speccy and then Amiga, I seemed to leapfrog the C64).

It contains loads of tips and tricks, plus there's downloadable code, discussed in the book, for two simple games, a space shooter and a platformer. This includes his sound library, which I thought I might try to use because that means that I can get straight on with writing the music. It uses an execute buffer; you simply pass a pointer to a bunch of commands and data to start playing the effect / music. Writing the commands is almost like 'coding the music', which is an approach I like.

At this point, someone may be reading this and thinking "why isn't she using _____ because that would be a much easier way to enter the data and for the game to play the music".

If that's you, please tell me. I'd really like to hear. I'm coming to this fresh after a long break, and when making games in my teens I didn't add music. I've not been an active part of a retro homebrew game community (yet) and don't know the tricks or the tools that people are using (if any).

Back to the story, using DM's sound library worked well with three simultaneous voices*. But there's a snag. The voices aren't staying in sync. Maybe I've misunderstood the way that the delays work (for the length of the notes and the gaps between them). I've tried to make sure that the delay counts total the same in each bar of each voice, but maybe other commands count as cycles too. Or maybe it's the attack /decay that's throwing it out (the more notes in a bar, the more it lags). I'll have to read the code a bit more thoroughly and see what it's doing. Or maybe it's a good idea anyway to modify things so that it syncs the voices. Perhaps by writing each bar of each voice as a separate buffer so that the start of each bar for each voice can be triggered at the same time.

* I'm not a windows user, and therefore not using the IDE that Derek suggests. The tools I've gathered and like include TMPx for assembling, and so I've had to edit the syntax of Derek's code.

Monday, 4 March 2019

Website archiving utility, version 2

Watchman for Mac is a utility that can scan a single page or a whole site on schedule, it'll archive the page(s) and alert the user to any changes in the code or visible text.

As it builds its archive it's possible to browse the historical versions of the pages, as they stood on particular dates. It displays a 'living' version of the historical pages, with javascript and stylesheets also archived.

We've just made a version 2 beta available. It features a 'non-UI' or headless mode, which means that it can remain in the background and not interrupt the user when a scheduled scan starts. Its windows can still be accessed from a status bar menu.

Version 1 is still available. It's free and will remain free. The new beta is also available and free at present, but version 2 will eventually require a licence.

Wednesday, 27 February 2019

Vic20 Programmer's Reference Guide

Having become more and more interested in the computers that started it all for me, I've been poking (geddit?) around in my attic among my collection of 8-bit computers, running emulators and buying hardware (leads and adaptors) and starting to use some of these machines again.

I have quite a collection - there was a time when you could easily find them at car boot sales for a fiver,  and I think that one of the Vic20s I have is the one I was given as a teenager.

I've always thought that I'd return to these computers, maybe in retirement, but it's happened a bit sooner.
There's something very exciting about these early home 8-bit machines. You're presented with a flashing cursor and in order to do anything, you need to type a command or two in basic (LOAD and RUN if you wanted to play a game). I believe this fact is why so many of my generation became so interested in software development, and it's something that was lost when a/ many turned to consoles because they only wanted to play games and b/ computers gained more graphic interfaces, separating the user from the workings.

Back to the point. I've begun writing a game for C64. I couldn't help myself. After much searching of the attic, I couldn't find a copy of the C64 Programmer's Reference Guide. I easily found it as a scanned pdf online, which is fine, but it's harder to use that than simply flicking through the pages of a book. (I will buy a copy, they come up on eBay. No doubt I will then find a copy I already owned.)

One of the things I really wanted was a reference guide for the instruction set. Although the C64 has a 6510 processor and the Vic20 a 6502, the two processors are identical but for a small difference. And I did find a Vic20 Prog Ref Guide, and the instruction set reference is identical.
It'll be handy when I make the Vic20 version of my new game.

It's very well-thumbed - I was as obsessed with programming as I am now. This is clearly my own original copy. It has some lined A4 with notes in my handwriting. It feels weird handling the book, a little bit like I've gone back 38 years in time, grabbed the book and brought it back with me.
(This page has my own instructions for using the monitor programme that I wrote in machine code, as a tool for writing more machine code. I used to write my programmes on paper and assemble them by hand before typing in the hex. I'm sure the monitor is on a tape somewhere in the attic.)

End note:
As part of my foray into this new-old world, I've discovered that there are some amazing people producing hardware and games (on cartridge/disc/tape) for these old computers. To support and encourage that, I've begun this project.

Friday, 18 January 2019

scraping Yelp for phone numbers of all plumbers in California (or whatever in wherever)

I've written similar tutorials to this one before, but I've made the screenshots and written the tutorial this morning to help someone and wanted to preserve the information here.

We're using the Webscraper app. The procedure below will work while you're in demo mode, but the number of results will be limited.

We enter our starting url. Perform your search on Yelp and then click through to the next page of results. Note how the url changes as you click through. In this case it goes:

(I added in &start=0 on the first one, that part isn't there when you first go to the results, but this avoids some duplication due to the default page and &start=0 being the same page).

So our starting url should be:

In order to crawl through the results pages, we limit our crawl to urls that match the pattern we've observed above. We can do that by asking the crawl to ignore any urls that don't contain

In this case we're going to additionally click the links in the results, so that we can scrape the information we want from the businesses' pages. This is done on the 'Output filter' tab. Check 'Filter output' and enter these rules:
URL contains /biz/
and URL contains ?osq=Plumbers
(The phone numbers and business names are right there on the results pages, we could grab them from there, but for this exercise we're clicking through to the business page to grab the info from there. It has advantages.)

Finally we need to set up the columns in our output file and specify what information we want to grab. On the business page, the name of the business is in the h1 tags, so we can simply select that. The phone number is helpfully in a div called 'biz-phone' so that's easy to set up too.

Then we run by pressing the Go button. In an unlicensed copy of the WebScraper app, you should see 10 results. Once licensed, the app should crawl through the pagination and collect all (in this case) 200+ results.


I was able to get all of the results (matching those available while using the browser) for this particular category. For some others I noticed that Yelp didn't seem to want to serve more than 25 pages of results, even when the page said that there were more pages. Skipping straight to the 25th page and then clicking 'Next' resulted in a page with hints about searching.

This isn't the same as becoming blacklisted, which will happen when you have made too many requests in a given time. This is obvious because you then can't access Yelp in your browser without changing your IP address. One measure to avoid this problem is to use ProxyCrawl which is a service that you can use by getting yourself an account (free initially), switch on 'Use ProxyCrawl' in WebScraper and enter your token in Preferences.

Wednesday, 9 January 2019

New website monitor / archive utility for Mac arrives at full stable release and is still free

Watchman is an easy-to-use website monitoring / archiving utility.

You can use it to watch a single page, a part of a website or a whole website. It can run on schedule (hourly, daily, weekly, monthly) and alert you to the changes you're interested in, which could be visible text, source, resources appearing on the page, its status, or you can simply leave it to archive all changes to all files. You can set up multiple website configurations, each with their own schedule. It uses the system's launchd, meaning that Watchman doesn't have to be left running, it'll just start as needed.

Watchman uses the same fast, efficient crawling engine as Scrutiny and Integrity, which has been developed over 12 years and offers a huge amount of configuration and tuning. This is coupled with a new web archive format.

Its web archive format can store changes like a Time Machine backup. You can view any page as it appeared on a certain date. When you do so, you're viewing a 'living' version of the page, with its css and javascript running as it was at the time, not a simple screenshot. You can of course export a version of a page as an image, or as a collection of all the files under their original filenames, as they were on that date. You can switch between versions of a page to compare them.

It's a desktop app running on your own Mac, so you own your own data.

It's early days, there are many more features in the pipeline, but for now it's stable and doing invaluable work. And version 1.x is free to download and use. (The next major version may not be free, or may be 'freemium' but the current version will continue to work and remain free.)

I've been flagging the release of Watchman for a while, it's been a long time since I've been so excited about a new project and I believe it'll become a more important title for us than Scrutiny.