modelrail.otenko Trains, computers, vintage, retro, model railways… the lot…

Commodore 64: Fixing RS-232 Serial Limitations

This is going to be a bit of a rant, so I apologise in advance. I have just spent a good 75 hours getting the C64 to talk to an Arduino via RS-232 and each step of the way has been painful. Initially, communications were sorted and data made it across to the Arduino. All wasn't as it should be as I realised that the character mappings (PETSCII vs ASCII) meant that the data had to be translated. Once that was sorted, it was a matter of sending data back. Again, character mappings were required. Past this, I then wanted to send 64-bits of raw data. Not characters, but 8-bits-to-a-byte raw data as I wanted up to 64 sensors and therefore only 8 bytes to transmit this. Turns out that the C64 is hard-coded to transmute the serial value zero to something else...

Diagnosing the issue

I'd already built up a large-ish application on the C64 for controlling the trains. There was also a large amount of code on the Arduino side. Due to this, any support libraries or includes or other interrupt-hacking routines could've been interjecting and mangling data. After a lot of to-ing and fro-ing, compiling on windows and switching to SD card... I got jack of the speed of which I was able to debug on real hardware.

Fortunately, VICE came to my rescue. Not only does it have a debugger, but it also emulates the RS-232 User Port Serial and shit... it even reproduced the error! Let me show you how to set that up...

Configuring VICE's User Port Serial

Open VICE and choose Settings -> Catridge/IO Settings -> RS232 userport settings.... Enable the RS232 userport emulation, leave it as device 1 and set the baud rate to 2400. Hit OK.

Back to Settings -> RS232 settings.... We want to edit RS232 device 1. Fill in the text box with the relevant IP and port of the machine you wish to communicate with. If you're going to use my TCP server below, then enter 127.0.0.1:25232.

Close VICE. Next time you open it, VICE will attempt to connect to a TCP Server listening on your localhost IP on port 25232. You can configure this to whatever you want, but we are going to use the default. VICE will then treat the connection as RS-232 and provide any data received to the internal C64 user port. It will also send out any data sent to the user port via this channel.

Now that we're done with the settings, we need to give it something to connect to.

TCP Server in C#

Download the code here. I've rigged up a very simple C# TCP server for VICE to connect to. The TCP server must always be loaded first, otherwise VICE will continue silently and never send out any data. The code is also overly-primitive and, upon closing VICE, the TCP server will need to be restarted prior to starting another instance of VICE.

Once running, you can use the 1-8 keys to set the bits of the byte to be sent. Alternatively, pressing any keys of the alphabet will set the byte to that letter in ASCII.

Finally, hitting Enter will transmit the byte.

C64 RS-232 Serial Test Program

Grab the Serial Test Application here that I wrote to test the RS-232 serial port on the C64 using Johan's driver. There's a batch file in there that you can use to run the program. Just make sure it knows where VICE is.

Make sure the TCP Server is loaded first, then run the batch program. It's set to auto-start the binary. If this doesn't work, then you can open VICE and via Settings -> Peripheral Settings... configure the directory where the binary is. You can then LOAD it as per usual.

Once loaded, you'll be in the app. It's pretty straightforward and will just print out the data that has been received.

Make sure that the TCP Server has shown that the connection has been established. You can now press the keys as per the instructions above on the TCP Server and send data. At the bottom is a timer and a number of bytes received. As each byte comes in, it'll be displayed up the top. It'll also be numerically represented down below, next to in:, above the clock.

At this point, type random letters from the alphabet and send them to the C64. You'll note that lowercase get translated to uppercase thanks to the ASCII to PETSCII translation. Tinker with the 1-8 keys to set the relevant bits in the byte to send and then hit enter. Watch that 1 = 1, 2 = 2, etc... until you try to send a raw zero... bugger.

Debugging the problem

Ok, we've now worked out that, even in the land of emulation, sending a zero to the RS-232 serial port on the C64 produces an 0x0d. This rules out the 'client'... both the Arduino and my TCP Server send raw zeroes and both VICE and a real C64 render the wrong character once received. From here, it could be the custom driver, the cc65 support libraries or something much more evil.

Debugging on real hardware is something I'm scared of... I'm so used to multiple windows and multi-tasking that I'd like to do it in a more comfortable environment. Thankfully VICE has a monitor that can help us. Open this up via the File menu and prepare to delve into the land of assembler! Note that, when open, the C64 emulation is paused.

Once the emulator and application are loaded, choose View -> Disassembly Window. You'll get a little window showing the entire memory of the emulated C64, disassembled. This means that VICE, with it's knowledge of C64 6502 CPU op-codes, has translated the raw memory into known commands. Fortunately, it also has information on the mapping of the memory and therefore does a very good job of this translation. Unfortunately, there is no search command... so we will need to scour over this to find what we are looking for.

What are we looking for?

The million-dollar question. The serial test app that we've compiled for the C64 (and executed in VICE) is based on the sample code by Johan. It also includes his 2400-baud serial driver. Looking at his source code, we can see the routines that are called when we call the ser_get function from our main loop. If you browse over to his source, you'll find the relevant lines in driver/c64-up2400.s.

GET:   
        ldx #2
        jsr CHKIN
        jsr rshavedata
        beq GET_NO_DATA
        jsr BASIN
        ldx #$00
        sta (ptr1,x)
        jsr CLRCH 
        lda #<SER_ERR_OK
        tax 
        rts

The snippet above was copied on the 24th of June, 2016. This code has hopefully been updated by now, but what you see above is the function, as it was back at that time, which reads data in from the RS-232 Serial port. I can't fully explain each line, but what we can do is find this chunk of code in the disassembly window. Unfortunately, we'll be doing this by hand as I don't know a better method to search for it.

After a lot of scrolling... this code has been found in memory location $238A. ldx #2 is seen as LDX #$02. The further jumps are visible, but their happy names are gone. Instead we see the memory addresses of each of those functions. You can see that I've also clicked the row in question. This sets a breakpoint, represented as a red highlight on the row.

Triggering the debugger

From here, we can actually close the debugger. Make sure that the red row is set prior to closing the main Monitor window. If you just did this, then you'll find that the debugger window popped back up straight away and now our red breakpoint is a lovely shade of teal. Teal? What happened to Red? In the disassembly list, a blue line indicates the location of the next line to be executed. Ours is teal because it's a set breakpoint AND it's the next line to execute. If you're bored, click the line and it'll turn blue. Click it again and it should return to teal.

Ok, we've triggered it... but why? No data was sent down the channel! It turns out that, if you look at the source above, there's a hint on the line with GET_NO_DATA. The ser_get is called from our code on each program loop; it returns a SER_ERR_NO_DATA when there is no data... so this line we've broken on is too early in the driver code. We really want to set a breakpoint at the first line executed when there is data.

So step through the code. To do this, press the 8th button on the toolbar in the Monitor window. Press it a few more times and watch the execution. We're currently 'stepping over' functions; the two JSRs might well send the program counter off into other parts of memory and execute code, but we don't care. Once you're on the BEQ GET_NO_DATA line, press it again. Note that we now jump down to the location of GET_NO_DATA and exit out. We can assume that this jump is the significant point where, when data exists, the execution continues straight through and does not jump.

Now that we know where this junction is, we want to clear our previous breakpoint and set one on the line directly after BEQ. Set the breakpoint and close the Monitor. It should not pop back up this time.

Viewing the data coming in via Serial

Ok, we've set our breakpoint in the driver and it hasn't triggered yet; let's hope it does when data is received. To test this theory, we'll need to use the TCP Server from above. If it's not already running, close VICE first and then start the Server. You need to have the TCP Server running prior to VICE for the connection to be correctly established. Once everything is running, ensure that the server has reported that the connection is established. Confirm your breakpoint is still in place; VICE doesn't save these, so you'll need to go hunting again if you lost the position. Once everything is configured, make sure you close the Monitor window to allow the C64 emulation to proceed. When the Monitor window is open, the CPU emulation is halted!

Once ready, focus on the TCP Server window and press the a key. Make sure it reports back that the byte is set to 97, which is ASCII. Press the enter key and you should trigger the breakpoint in VICE.

Ok, we've got it, our breakpoint was hit and we presumably have data somewhere? Our breakpoint is actually on a JSR through to the address $FFCF. At this point in time, this address is unfamiliar. All our previous driver code has been around the $2300 mark. Due to this, we're going to step over it. The next call is an LDA which uses a pointer + offset to load a value into register A. Stepping over this, we see the following in the Monitor window:

(C:$2394) n
.C:2397  A2 00       LDX #$00       - A:61 X:03 Y:00 SP:f2 ..-.....    6728381

Interesting... A:61 hey? That's the ASCII HEX value for a lower-case a! So there it is, the value has come through and we can see it in memory. You can close the debugger now and play around with data on the TCP server to see how it arrives on the C64.

Note that as soon as you try to send a zero from the TCP Server, it'll appear as an 0x0d. At this point, we've honed in on the location where the char is read in, but we don't know what code is hosted up in the $FF00 range?

C64 Memory Mapping

Time to go deeper. Thank god we're working on a seriously old system that has been thoroughly documented. You'll find the memory map of the Commodore 64 here. If you scroll down the page, you'll find that the area in question contains the Kernal (Kernel?). At this point... we might as well retire.

Retire? Yes. The KERNAL is ROM, you know... READ-ONLY MEMORY. It's the raw system code burnt onto chips on the motherboard and it is a fixed entity. Those bytes can't be hacked... anything we try to do will require reproduction of kernal commands in the 'user' area. Should we try this? Sure... but first it'd be nice to know what's going on.

C64 Kernal Disassembly

Again, thanks to the age of this system, there's a full disassembly of the C64 Kernal here. We know that we're jumping in to location $FFCF, so browse down to that area.

FFCF   6C 24 03   JMP ($0324)   ; (F157) input char on current device

Oh look, it's just another JMP. Fortunately it's commented and we know to browse through to $F157. Now we are in murky waters... not many comments here. We get a heading of ; input a character, but then not much else. The assembly, to the naked eye, looks like a switch statement. It seems to be going through checking which device to read from. In our case, we can actually step through it in the debugger. If you step into the JSR #$FFCF then you'll be able to watch it jump as you send in characters from the TCP Server.

The basic trail jumps through: $FFCF -> $F157 -> $F166 -> $F173 -> $F1B8 (a.k.a. read a byte from RS-232 bus) via F177. At $F1BD, there's a comparison of the incoming byte to 0x00. If there's no match, then the code jumps out to $F1B3 and returns the value. If there is a zero, then the following occurs:

F1C1   AD 97 02   LDA $0297
F1C4   29 60      AND #$60
F1C6   D0 E9      BNE $F1B1

If the AND fails to compare, then the jump happens at $F1C6 to $F1B1. Looking at $F1B1 made me cry. The code implicitly inserts an 0x0d, overwriting the byte that was read into the buffer and then returns it.

I don't quite know what the LDA from $0297 is and why it is AND'd with #$60. I'm sure there's some RFC or some prehistoric rule back in the late 1980s that said if a modem or other serial device returned a zero, then it actually meant it to be a carriage return. Maybe it was a BBS thing? I'll continue digging and attempt to comment this area of code, but for now... we know that it's futile. The KERNAL ROM is fighting us and thinks it knows better!

A valid workaround

Righto... what do we do here? I initially thought it was a complete loss and gave up. Further Sapporo made me realise that this call was made from our custom driver. What if we specifically mention that our custom driver is built to handle 'zero' byte data and implement a work-around? If we copy out the code from the Kernal and re-produce it in the driver, then we can effectively resolve this (I wont say bug, I'm sure they had their reasons!) issue.

So, the trick here is to grab the portion of code from the full disassembly of the C64 Kernal and build it into a function in Johan's driver.

We know the entry point is $FFCF. This is a JMP to the switch statement which chooses the device. We know that this is the RS-232 driver, so we can skip that part and copy the code from $F1B8 to $F1C8. I've pasted this in below.

; read a byte from RS-232 bus

F1B8   20 4E F1   JSR $F14E
F1BB   B0 F7      BCS $F1B4
F1BD   C9 00      CMP #$00
F1BF   D0 F2      BNE $F1B3
F1C1   AD 97 02   LDA $0297
F1C4   29 60      AND #$60
F1C6   D0 E9      BNE $F1B1
F1C8   F0 EE      BEQ $F1B8

That CMP #$00 is the pain. Let's just jump to $F1B3 all the time. Actually... $F1B3 is just CLC and RTS. Let's just write that. We also can't directly BCS to $F1B4, so we'll need to JSR to a closer function and then call JMP. If we JMP directly then we'll lose our position in the stack.

F1B8   20 4E F1   JSR $F14E
F1BB   B0 F7      BCS $F1B4   ;re-write this to a BCS and JMP
...    ...        CLC
...    ...        RTS

With my patch above, I've removed (what seems to be) a re-try loop in the code. If it falls all the way through to $F1C8 then it returns to $F1B8 and tries to read a character again. I haven't seen this state occur in real life, but I'll keep an eye out and try and work out when this actually occurs. It seems that the AND #$60 must check for an error state which I'm yet to encounter.

I don't actually know the assembled opcodes off-hand. We will write this as standard assembly into the c64-up2400.S driver source file and then it'll write the opcodes on compilation. So, from line 139 we slap in:

;----------------------------------------------------------------------------
; OTHERFUNC: Shortcut to Kernal code
;

OTHERFUNC:
		jmp $F1B4
		
;----------------------------------------------------------------------------
; OURBASIN: This is a minimised call to get the character from the buffer.
; The Kernal code does not allow zero bytes (0x00)... this does.
;

OURBASIN:
		jsr $F14E
		bcs OTHERFUNC
		clc
		rts
		
;----------------------------------------------------------------------------
; GET: Will fetch a character from the receive buffer and store it into the
; variable pointer to by ptr1. If no data is available, SER_ERR_NO_DATA is
; return.
;

GET:   
        ldx #2
        jsr CHKIN
        jsr rshavedata
        beq GET_NO_DATA
        jsr OURBASIN
        ldx #$00
        sta (ptr1,x)
        jsr CLRCH 
        lda #<SER_ERR_OK
        tax 
        rts

I'll give the kernal function a real name soon. Right now the basic point is that we write our own BASIN function that is just a tiny subset of the greater procedure and then skip the part where it inserts that shitty little 0x0d.

Either way... compiling this (see notes on that here) saw the bloody thing work! I'm going to get in touch with Johan now and determine what needs to be tidied up to get this trick included in the trunk.

That was fun!