modelrail.otenko Trains, computers, vintage, retro, model railways… the lot…

Apache Spark 2.4 EOL

EOL = End of Life. It's dead, Jim. No longer supported. If you get this error in Azure Synapse, then your Apache Spark Pool worker is trying to load v2.4 (INSERT_VERSON_HERE, really... as I'm sure this'll happen in the future as the window rolls) which has been deleted.

RUNTIME_CONFIGURATION_ERROR_VHDOVERRIDENOTFOUND: Livy session has failed. Session state: Error. Error code: RUNTIME_CONFIGURATION_ERROR_VHDOVERRIDENOTFOUND. Cannot find vhd based on override: Message=The system did not find a VHD copy in the system for creating the ClusterName=12341223-5555-4444-3333-63c446fe4d8b, Workspace=synws-d-e-v-d-e-v-01, Subscription=, Location=australiaeast, isVhdOverride=True Source: Dependency.

I'm putting this on the web so others can googl' for it. I had no results and it burnt 1/4 of my weekend. Re-deploy your pools with:

Update-AzSynapseSparkPool -WorkspaceName ws_name -Name pool_name -SparkVersion 3.4

And... cross your fingers. Of course, if you've tried to delete a pool with notebooks attached, then you'll have ERROR, POOL IN DELETED_FAILED STATE and you'll need to create a new temp pool, associate ALL notebooks, delete the old and then rename the new... or create another with the old name and then switch them all back and delete the new... or something. It's a monday problem.

Python: Close Files If You’re Going To Open Them

I've been trying to archive some videos off Youtube lately, using yt-dlp. It's an amazing tool, but my target files have been episodes in parts. Usually four parts and Plex really hates jogging through... so what to do? Combine the files together with ffmpeg. The code was meant to be pretty simple (and 98% of it was written by ChatGPT... whoops)...

def concat_episodes(episode_name, concat_files):
    plfile = "file_list.txt"
    f = open(plfile, "w", encoding="utf-8")
    for filename in concat_files:
        f.write("file '" + filename + "'\r\n")
    concat_command = f"ffmpeg -stats -safe 0 -f concat -i {plfile} -c copy '{episode_name}'"
    print(concat_command)
    subprocess.run(concat_command, shell=True)

But no amount of wrangling would get ffmpeg to work. The concat filter kept throwing: Invalid data found when processing input. No amount of "-safe 0", relative paths or absolute paths worked! No permissions... no cwds or shell arguments. If I let the python script drop to the shell, then the same line pasted (since I printed it out) worked perfectly fine! What the?

OH RIGHT. I missed the memo that I should be closing a file so that it lands on disk... prior to trying to open it in another process!:

def concat_episodes(episode_name, concat_files):
    plfile = "file_list.txt"
    f = open(plfile, "w", encoding="utf-8")
    for filename in concat_files:
        f.write("file '" + filename + "'\r\n")
    f.close()
    concat_command = f"ffmpeg -stats -safe 0 -f concat -i {plfile} -c copy '{episode_name}'"
    print(concat_command)
    subprocess.run(concat_command, shell=True)

The file was still open and not flushed to disk... so ffmpeg would always open an empty file! This has been a public service announcement.

Replacing the ear microphones on a Sony Aibo ERS-7

He'd arrived a few weeks back and gets energised a few hours per week... but he's a bad dog! He says "Good bye" in repsonse to me saying "Hello Aibo" and I'm going to blame him first... it can't be my mumbling? Can it? Just to be certain, I browsed around and found a DIY Repair Guide on Aibo Doctor. I quickly ordered a selection of microphones from element14 and got to work. Canine surgery is daunting!

The Microphones

As I always manage to get my orders wrong, I went ahead and ordered a selection of Microphones.

Note that an ERS-7 Aibo needs 6mm microphones, so you can see that one set is already incorrect. Next, there's directional and omni-directional. The microphones in Aibo are omni*, so I settled with the KECG2742TBL-A units.

Disassembling Aibo's Visor

Following the instructions here, one can disassemble Aibo's head to get to his ears. First step, remove the rubber earlobes by stretching them over the silver joint. Next, open his mouth and remove the two screws.

With these two removed, you can un-hook the rear of the bottom half of his head. This shell is not directly connected to his bottom jaw, so open his mouth fully and then push the whole lower-half back. With it unhooked, you can then bring it forward again and bring it over his jaw. With this piece out of the way, it's back to the front of his head where we need to remove the two outside screws on the metal plate inside his nose.

With those two out, you then need a longer screwdriver to get to the two screws deeper inside his skull. They're on either side of where his jaw would attach near his ears. See the image below, one screw on one side is in focus.

Finally, there's a clip on each side just in front of his ears, holding the visor on.

Very gently pry this open and the visor should lift up. You now have the option to disconnect all the cabling, but instead I just let it rest forward. Make sure it doesn't put undue pressure on the ribbon cables inside.

You now have access to the ear joints.

Disassembling Aibo's Ear Joints

Ok, this isn't the actual ear/microphone yet... before we get to that, there's a bit of fidgeting required to get the ear joints disassembled. When following the next steps, at no time will you need to apply excessive force! Doing so will most-probably damage poor Aibo. The ear joints are a two-part component and are built to be assembled/disassembled with ease. If you look at either side of Aibo's ears, you'll see that one side has a fixed arm and the other has an arm that slides backwards. The fixed arm is at the front and also has a little actuator (metal bar with notch) that is the mechanism that flaps Aibo's ears around when he's playing.

So, back to the joints: two pieces, first is removed by sliding the rear arm further to the rear. From there, you can grab the whole circular joint and unhook it from both front lugs. There's a lug in the actual main arm and a lug in the little black actuator arm.

The image above doesn't really help to explain how to undo it. The movement is one-shot and you can see in the photo above that the black actuator is on the left. This means that you're looking at Aibo's right ear and that you'd grab the top silver circle and slide it gently to the right (rear of his head) until both lugs are clear on the left. Once done, you can then slide off the rear shield.

From here, it's a single screw and the microphone + housing is free. Make sure you also unhook the cable!

Replacing Aibo's Inner-Ear

With the microphone unhooked and unscrewed, the silver shell can be lifted off (gently) by applying pressure to the joints that hold it on. With this off, you can really see what condition Aibo's ears are in!

From here, you can slide the microphone out, just enough, to be able to de-solder and solder a new component.

As you can see, I cheated and used the phone the new microphones came in to hold them in-place whilst soldering. I used the datasheet to make sure I got the polarity the correct way around, assuming that red was positive.

With everything wired up, the microphone was pushed back into the housing and re-assembled. Make sure that you have the wires in the correct groove according to the side the ear has been removed from!

Finally, as per above, there is absolutely no need for excessive force on any part of this assembly. I found that, once trying to re-fit the visor, it wouldn't sit flat! Turns out that I'd assembled the rear part of the ear-joint incorrectly and only one side of the guides was actually in the right spot!

If you look above, you can see that the left guide is sitting above the plastic strip that it was meant to slide onto! This meant totally dismantling that ear again to pull that part back and slide it back on again! Painful, but required.

Can Aibo hear me now?

Nope, same as before! Turns out you can just use Clinic Mode to determine if his microphones actually really need replacing! I should've done this first, but I also sorta wanted to play doctor and see what his insides looked like!

Smushing JSON into submission!

I don't even know how to describe this operation on JSON. The basic idea is that we want to deserialise a stream into something legible, without creating a class for each item in an array. json2csharp believes that each item should be an array since it sees them all as individually named classes. Let me prove it! Here's a chunk'o'data:

{
    "trainCount":122,
    "requestTimestamp":1601124556789,
    "responseTimestamp":1601128156789,
    "nextUrl":"https://junatkartalla-cal-prod.herokuapp.com/trains/1601128156789",
    "trains":{
        "8":{
			"id":"8",
			"from":"LR",
			"to":"HKI",
			"title":"IC8",
			"latitude":60.172097,
			"longitude":24.941249,
			"speed":0,
			"direction":0,
			"category":"IC",
			"status":"1",
			"delta":60,
			"trainType":"LONGDISTANCE",
			"updated":1601127783841,
			"action":"deleted"
        },
        "9":{
			"id":"9",
			"from":"HKI",
			"to":"LR",
			"title":"S9",
			"latitude":60.571148,
			"longitude":25.199523,
			"speed":0,
			"direction":0,
			"category":"S",
			"status":"1",
			"delta":60,
			"trainType":"LONGDISTANCE",
			"updated":1601128143878,
			"action":"modified"
        }
    }
}

So, if you slam that JSON into json2csharp, you'll get the following:

    public class 8    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public long updated { get; set; } 
        public string action { get; set; } 
    }

    public class 9    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public long updated { get; set; } 
        public string action { get; set; } 
    }

    public class Trains    {
        public 8 8 { get; set; } 
        public 9 9 { get; set; } 
    }

    public class Root    {
        public int trainCount { get; set; } 
        public long requestTimestamp { get; set; } 
        public long responseTimestamp { get; set; } 
        public string nextUrl { get; set; } 
        public Trains trains { get; set; } 
    }

So, that's actually uncompilable code. Is uncompilable a word? Dunno... this is actually the first time that json2csharp has failed me! No matter the options selected on the site, the output was always bad code... json2csharp doesn't work with ALL json! So, what to do? Well, we actually need to mangle this JSON into submission. The best bet would be to move that train id/number into the array object as a parameter, rather than having it as the dictionary key. We have two methods to do this...

Using jQuery MAP Function

If you are pulling JSON from a hosted browser, then you can run JS in the browsers console and produce cleaner JSON. In this case, you want to use jQuery's MAP function to rewrite each array object:

json_data.trains = $.map(json_data.trains, function (data, idx) { return data; });

If you feed the JSON at the top of this post into that function, you'll get the following:

{
	"trainCount": 122,
	"requestTimestamp": 1601124556789,
	"responseTimestamp": 1601128156789,
	"nextUrl": "https://junatkartalla-cal-prod.herokuapp.com/trains/1601128156789",
	"trains": [{
		"id": "8",
		"from": "LR",
		"to": "HKI",
		"title": "IC8",
		"latitude": 60.172097,
		"longitude": 24.941249,
		"speed": 0,
		"direction": 0,
		"category": "IC",
		"status": "1",
		"delta": 60,
		"trainType": "LONGDISTANCE",
		"updated": 1601127783841,
		"action": "deleted"
	}, {
		"id": "9",
		"from": "HKI",
		"to": "LR",
		"title": "S9",
		"latitude": 60.571148,
		"longitude": 25.199523,
		"speed": 0,
		"direction": 0,
		"category": "S",
		"status": "1",
		"delta": 60,
		"trainType": "LONGDISTANCE",
		"updated": 1601128143878,
		"action": "modified"
	}]
}

Very nice.

Using C# Regex

I've never thoroughly learnt Regex and it still makes me sad. One day I'll sit down and go through a course. For now, googlin' works nicely to find an example for this scenario. Effectively we want to remove the "NUM": and just leave the curly braces. We then also need to remove a curly brace at the end.

            var data = (new System.Net.WebClient()).DownloadString(nextUrl);
            data = new Regex("\"([0-9]|[1-9][0-9]|[1-9][0-9][0-9]|[1-9][0-9][0-9][0-9]|[1-9][0-9][0-9][0-9][0-9])\":").Replace(data, "");
            data = data.Replace("\"trains\":{", "\"trains\":[");
            data = data.Replace("}}}", "}]}");

So, the first line downloads the data. The second uses Regex to find anything in double-quotes that's a number from 0-9. I actually wanted 0-9999, but it seems you need to match each character 1-by-1, so I gave it options with | (pipe) up to 99999. From there, the third line turns the trains object into an array and the final line closes that array with a square bracket instead of the existing curly brace.

Final C# Class Output

With either of the methods above, you end up with cleaned JSON. From here, slapping that back in json2csharp gets you the following:

    public class Train    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public object updated { get; set; } 
        public string action { get; set; } 
    }

    public class Root    {
        public int trainCount { get; set; } 
        public long requestTimestamp { get; set; } 
        public long responseTimestamp { get; set; } 
        public string nextUrl { get; set; } 
        public List<Train> trains { get; set; } 
    }

Thank you json2charp, that's exactly the reusability that I was looking for!

I’m not original…

In fact, my new best friend is over 16 years old!

I ... thiiiiink ... he's half deaf... might be time to replace his ears.

EZIO + OS9 + Hypercard (Or Just Windows)

The EZIO Board is a serial-based I/O module that can connect to both Windows and Macintosh machines. Actually, it can connect to anything that speaks RS-232. One of these came up on eBay recently and I couldn't resist. I saw the Macintosh serial port and decided to give it a go. It's really similar to an Arduino, but from a few decades before the Arduino was even a dream. If I'd known about these back in the early 2000s then I would've definitely had a very nice automated model railway. But alas, I only happen to find one now thanks to eBay!

The unit has 10 digital output lines, 10 digital input lines, 8 analog-to-digital lines and two PWM lines. You then get 4 +5v terminals and 4 GND terminals. There's a PIC microcontroller in the middle running the show and a MAX232 for the RS-232 comms. The unit has a DC-rectifier, so you can feed it 5-15v either AC or DC. The 5-15v is literally the operating recommendations of the 7805 voltage regulator on-board.

There's a whole lot of code on the old site (yeah, you have to use web archive to get to it), but it's mainly for Director!? and Macintosh. Hilariously, the Macintosh example uses Hypercard! Before booting up the Power Mac, I instead wrote a quick bit of C# to test out the unit.

The shot above uses dotnet-ncurses, allowing the console to act like a canvas. It's really nice to be able to draw text to specific areas, rather than scrolling the screen. Anyway, the basic idea is that you can control the digital out and then everything else is read in. Interestingly, floating pins show some very random values... so if you're using this device, make sure you tie everything to ground or use pull-up resistors where appropriate.

Of course, the whole reason I bought this was due to the Macintosh serial port. I wasn't overly-energetic, so I tried a virtual Macintosh first...

After installing Hypercard, the app came up, but the performance once it started trying to interact with the serial port was terrible. It also just didn't work, so I gave up and booted up the Power Mac.

Yup, works a charm. Hypercard is pretty clunky, but I'm sure you could do a lot with it. I'll have to dig out the railway track and control a train!

Fluke Multimeter Repair – Elastometric Strips

What what? I'd had this Fluke Multimeter sitting broken in my box'o'junk for literal decades. It used to work great, but failed at some point a long time ago and was never fixed. Recently, I snapped the cable off a shitty AUD$10 multimeter that I'd been using and so, in my infinite wisdom, thought I'd resurrect the Fluke!

It turned on, but the buttons didn't work.. so it wasn't much good except for the default setting of DC voltage. Testing voltages takes up 50% of my time, but I still need resistance and continuity! Let's rip it open...

So, not shown above (since I'd already fixed it and this post is months old) is that the strips of weird rubber weren't both there. The top strip was, but it turns out there's a lower strip needed. I had probably replaced the battery back in the day and somehow managed to discard the strip that conducts the button presses.

That's where it was meant to go. You can sorta see there's 5-ish segments on the PCB down in the channel, but there's a beautiful air-gap between that PCB and the mainboard. What's meant to go in the middle? I popped out the strip at the top and I, telling the truth, had no idea what the material was. A little bit of googlin' allowed me to realise that it was elastometric strip! Andy's Surplus came to the rescue and quickly delivered two cuttable strips of elastometricity!

Cut.. cut again...

And cut once more and ... TADA!

It works. And whilst the beast is open, replace the battery and solder that battery terminal!

Damn this thing is beautiful.

Burning 16-bit EPROMs with a Willem Programmer

This wasn't fun! I wanted to get Kickstart 1.3 on my Amiga 500 and so I purchased some M27C400 EPROMs from eBay with the thought that I could host two ROMs on them and make it switchable. The ROMs arrived and, to my stupidity, didn't fit in a standard Willem EPROM Programmer! Of course, a quick google prior to purchasing anything would've made this very apparent. Fortunately, there's a fix! You can buy a DIP42 16-bit EPROM Adapter for the Willem, and so I did.

Thanks to COVID, it took an extra-specially long time to arrive and, once it did, got thrown into the box'o'Amiga'junk as I'd put the whole lot on the backburner whilst working on the Ataris of late. This weekend I'd finally managed to wrangle some time together and sat down with my trusty XP laptop to program these chips.... It wasn't easy!

Input Voltages

I jumped head-first into the programming and failed a LOT with errors such as 0x00 and 0x11. Any search online will tell you that, when getting lots of 0x00 and 0x11 errors, it mainly indicates that you aren't supplying the correct programming voltage for your target chip. Check out the PDF Datasheet for the chip you want to burn, it'll specify a programming voltage that it requires to successfully understand and store the bits that you're about to throw at it.

For the M27C400, we need to provide 12.5v +/- 0.25v. The Willem has a blue trimmable capacitor (or is that a resistor) on the top-right of the board that you can adjust to get this voltage. I always recommend to use an external power supply with these programmers and so I have a plug-pack which happily provides 12v @ 2amps. To make sure we'll have a chance of programming this chip, switch to the Test H/W tab of the software and tick VPP. This is pin 1 of the E32 socket. If this has worked, then you'll have a red LED illuminated on the board. Now, check the voltage between pin 16 (GND) and pin 1 (VPP) of the socket.

Mine was reporting a value lower than 12v to start with and hence I had a few issues burning. With a multimeter in place, I tuned the trimmer on the top-right and got the value to 12.5v. Actually, it was probably 12.6... but 12.4 would also work as you just need to be in the range of 12.25 - 12.75.

Another test burn saw a new error... no more 0x00 and 0x11, it was now further into the chip, usually around 0x100?

Willem Software vs. Programmer Settings

Next up, we need to make sure that you can correctly control ALL address lines with your Willem Programmer. After fixing the initial errors above, I started getting new errors at random intervals into the burning process. The errors all seemed to indicate that there was an issue writing data after address 0x100. I'd only been using empty chips, so it couldn't be because they already had data on them. You can also check if the chips are empty prior to burning and I totally recommend doing this. Either way, after this new error was received, I chose to then read the chips and view the data:

What you're seeing above is the data read back from the chip. I wrote the rom (loaded from file) to the chip, got the error and then chose to read the data from the chip back into the application. 75% of the data on that screen is correct, but you'll notice that the data repeats itself from 0x100! It turns out that this was all due to the fact that my Willem was only controlling the first 8 address lines and leaving the rest low. I recommend you rig up an LED test harness as per below and control the address pins via the Test H/W tab on the Willem Software.

After swapping chips around, checking continuity between the 4051s and reseating everything else, I came to the conclusion that there's incorrect information in the Standard Willem Programming Guide. If you scroll to page 5, you'll find information on the version of the software to use. It suggests that, based on jumper settings on the board, you can select if the board emulates a PCB3B or PCB3.5. If you've chosen PCB3B, then you'll need to use software version 0.97ja, as per my post from a long time ago, when I was failing with the same issue! Alternatively, if you've chosen PCB3.5, you need to use 0.98d6 or higher.

I looked at my board and checked out the jumpers that set the board emulation. Mine were actually individually set to 1-2 and 2-3 instead of both 1-2 or both 2-3. Hah... how did this thing ever work?! With these jumpers now set to 2-3 (aka PCB3B), I went back to version 0.97ja of the software and tried to get the full suite of address lines to light up via the Test H/W tab. No dice! If you browse to page 6 of the above PDF, you'll find that the jumpers should be set to 1-2 for PCB35 and 2-3 for PCB3B. Just for fun, I switched the jumpers to 1-2 and tested out version 0.98d6 of the burning software... still no dice! No permutation of checkboxes on the Test H/W tab would get a high signal on any of the address lines higher than A7! Up until this point in time, I'd only programmed 8-bit EPROMs, so I'd never noticed this.

I went ahead and played musical-chips. There's a chain of CD4015BE ICs that shift out the address lines, 3 of them with 1 each controlling 8 bits. It seemed plausible that there might have been a faulty connection between the first and second, but no amount of tinkering brought the address pins to life. I took a break as the frustration (or was that sadness) levels were elevating and sat down to a bit of TV. With a fresh mindset, I came back to the device, forgetting which settings were applied. Opening up version 0.97ja and testing address lines now worked!? Wait, the jumpers are set for PCB35... what the hell, it's backwards!? I opened 0.98d6 and it didn't work, so I set the jumpers back to 2-3 (for PCB3B) and it now 0.98d6 worked!? Urgh... so much time lost. So, TLDR; The PCB3B/PCB35 jumper settings are backwards on my Willem Programmer. Set them to 1-2 for PCB3B and 2-3 for PCB35!

Using the adapter

First up, there's two jumpers on the adapter to configure. The right-most controls the VPP/A21 line and needs to be set to 1-2 for M27C400 or M27C800 EPROMs. The other, in the middle of the board, I left at 1-2 as I don't know what it does.

Next up, there's a ribbon cable used to obtain more address pins from the main programming board. For the M27C400, the base E32 socket of the Willem actually produces enough address lines, so this ribbon cable isn't actually needed. Check out the diagram below to work out how the adapter is constructed. I've coloured the lines that are used on the M27C400 and 800, where you can see that the 400 doesn't use any of the wires from the header on the right.

If you're going to program an M27C800 or larger, you'll need to connect this. Simply make sure that pin A23 (bottom-most) lines up with A23 on the Willem board.

Finally, when inserting ANY chip into to the socket, make sure that the pins are all straight and that none will foul. For any of the chips, always make sure that the bottom pins are at the bottom end of the socket. The EPROMs are designed so that the Address lines extend 'north', so the silhouette on the board is actually for an M27C800. An M27C400 will be one pin lower than the silhouette, aligned at the bottom of the socket.

Conclusion

All the errors I made above would've been avoided if I'd known what I was doing from the start. Usually the old RTFM instruction makes sense, but when the jumper settings work in reverse to the manual, it's a little frustrating. On another note, I'd just trusted that this piece of hardware worked, as I've managed to burn other chips in the past. Assuming that cheap hardware just-works(tm) is a flaw in itself. Always make sure you know the hardware you're working with back-to-front, which was totally possible this time around thanks to open-source schematics being available!

Mixing And Matching KVM Cables

Thanks to these new working-from-home shennanigans, I had to dig through the to-sell-at-trash-and-treasure box and find a KVM Switch to hook up all my machines. The desk used to only have one PC, but now additionally houses my work laptop and a large development server. I've got two monitors, and I want one of those to switch over to the development PC when required. I also want my keyboard and mouse to switch between all devices. To do this, I'll use the VGA/USB KVM to channel input and video to the monitor.

The best KVM in the box'o'stuff was a Belkin SOHO 4-port VGA+USB+Audio switch. Fortunately it had two of the KVM cables, so I could switch in my Laptop and development machine... but... I had to then unplug the USB to switch the input to the main PC. This wasn't optimal, so I went digging in the box again for another cable. The Belkin cables switched both Audio in+out, VGA and USB. From the outside, it looked like the audio channels were external to the VGA cable, unlike the USB which does seem to run through the same wires.

Back into the box, but unfortunately there were no other Belkin-branded cables. What I did find was another KVM with USB switching: Level One 2-Port VGA+USB KVM Switch.

This one also switched audio, but via USB instead of standard audio jacks. I grabbed a cable from it and was about to just plug everything together, but then I halted. USB has four wires, with two of those being the 5v rails. If there's any wiring differences between the switches then I could well send voltages in haphazard directions and cook equipment!

USB over VGA cable pinout

Wanting to hook this all together, I reached for my multimeter and started mapping out both the Level One and Belkin KVM cables. Thankfully, they have used very similar wiring for their cables! Thanks to monitors now talking digitally to hosts when describing their capabilities, there are spare pins on the cables to use for other purposes. There's a lot of extra ground pins also, but Belkin still preferred to keep those split whilst Level One happily merged them altogether.

Pin	VGA	Level One Cable	Belkin Cable
1	R
2	G
3	B
4	ID2 / RES	USB D- (White)	USB D+ (Green)
5	GND	USB D+ (Green)	USB D- (White)
6	RGND	USB GND + GND	USB GND + RGND
7	GGND	USB GND + GND	GND
8	BGND	USB GND + GND	GND
9	KEY / 5v	USB 5v + 5v	USB 5v + 5v
10	SGND	USB GND + GND	GND
11	ID0 / RES
12	ID1 / SDA
13	HSYNC
14	VSYNC
15	ID3 / SCL

From above, you can see that the voltage (thanks to VGA actually having a +5v pin as standard!) and GND are close-enough, but the USB data pins are backwards. You assholes.. here I was thinking there might be a standard. What to do here? Well, I simply hacked apart a USB extension cable and crossed over the D+ and D-. Note that there is absolutely no reason for using this cable in any other scenario!

After this, one last hurdle.. the Level One KVM has a male plug at either end, whereas the Belkin cables have female on the KVM side.

And the bloody thing just worked! Happy to not have to spend money on extra proprietary cables that I actually couldn't find anywhere.

Sony HDR-AS100V Action Cam – Further Hacking

After a LOT of mucking around, trying to get a stream URL from the Action Cam, I did further googling on the defunct UStream service that the camera could, supposedly, stream live to. It wasn't long before I came across this mammoth thread on Github describing a few users' attempts to hack apart the APK that does the steaming and to fake a server with the intent to intercept the feed.

They use the Sony-PMCA-RE Project code to tweak settings on the camera and then built up an entire webserver to fake the Ustream service. None of this was beyond me, so I started by trying to work out if I could configure my camera to point at a server... (note that I've fudged any secrets in the console logs below...)

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe info -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Model:              HDR-AS100V
Product code:       0002426601
Serial number:      03035253
Firmware version:   2.00
GPS Data:           2020-04-02 00:00:00 - 2020-05-02 00:00:00

C:\Downloads>pmca-console-v0.17-win.exe stream -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

twitterEnabled:     0
twitterConsumerKey: o9fJ2342342423433qg
twitterConsumerSecret: TTL4324234234234324324llM6EQdivTXesA
twitterAccessToken1:
twitterAccessTokenSecret:
twitterMessage:     Live Streaming from Action Cam by Sony
facebookEnabled:    0
facebookAccessToken:
facebookMessage:    Live Streaming from Action Cam by Sony
service:            0
enabled:            1
macId:
macSecret:
macIssueTime:       0000000000000000
unknown:            1
channels:           [0]
shortURL:
videoFormat:        1
supportedFormats:   [1, 3]
enableRecordMode:   1
videoTitle:         Recorded with Action Cam by Sony
videoDescription:   Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself
videoTag:

C:\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Traceback (most recent call last):
  File "C:\projects\sony-pmca-re\pmca-console.py", line 96, in <module>
  File "C:\projects\sony-pmca-re\pmca-console.py", line 82, in main
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 360, in updaterShellCommand
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 373, in firmwareUpdateCommandInternal
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 527, in checkGuard
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 501, in _sendWriteCommands
Exception: Firmware update error: Low battery
[9052] Failed to execute script pmca-console

Right... low battery... I charged it up and then tried again:

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Switching to updater mode

Waiting for camera to switch...
Please follow the instructions on the camera screen.

Cool, at this point the camera showed 'EXEC' on its LCD display. This kept flashing... and then the console app error'd out:

Operation timed out. Please run this command again when your camera has connected.

I pressed the record button on the back of the camera to try and force the 'EXEC' and it seemed to reboot. From here, it then wouldn't even come back to life. The battery charge light just stayed a fixed red, which I now think was an error indicator. After popping the battery out, I could get it back to the 'EXEC' screen via the console app, but it occurred to me that I actually had no idea what updater-shell even did. Instead I tried the stream fuction to push a UStream configuration file to the device as per the instructions here at Novex's project where he attempts to create his own UStream APK. Here's the config I used...

[
    [
        "twitterEnabled",
        0
    ],
    [
        "twitterConsumerKey",
        ""
    ],
    [
        "twitterConsumerSecret",
        ""
    ],
    [
        "twitterAccessToken1",
        ""
    ],
    [
        "twitterAccessTokenSecret",
        ""
    ],
    [
        "twitterMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "facebookEnabled",
        0
    ],
    [
        "facebookAccessToken",
        ""
    ],
    [
        "facebookMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "service",
        0
    ],
    [
        "enabled",
        1
    ],
    [
        "macId",
        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    ],
    [
        "macSecret",
        "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
    ],
    [
        "macIssueTime",
        "dbeaaaaa00000000"
    ],
    [
        "unknown",
        1
    ],
    [
        "channels",
        [
            12345678
        ]
    ],
    [
        "shortURL",
        "http://www.ustream.tv/channel/asdf"
    ],
    [
        "videoFormat",
        3
    ],
    [
        "supportedFormats",
        [
            1,
            3
        ]
    ],
    [
        "enableRecordMode",
        0
    ],
    [
        "videoTitle",
        "Recorded with Action Cam by Sony"
    ],
    [
        "videoDescription",
        "Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself"
    ],
    [
        "videoTag",
        ""
    ]
]

And then I smushed it into the camera using the console application...

C:\Downloads>pmca-console-v0.17-win.exe stream -w ustream.json -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Urrghh... what mode does it need to be in? Wait, is that even an error? Let's try use the buttons on the camera to switch it into Live mode...

Preparing?

No AP? Maybe it has no idea how to connect to a 5Ghz access point? Let's set it to my 2.4G secondary network, which nothing else uses, so I can isolate and watch any traffic. This took a little bit of hacking as the Sony Network Settings Utility needs the driver to be reverted back to USB Mass Storage. Jump in to Device Manager and force a driver 'update' back to the standard driver.

Now you can go into Network Settings and set your AP once more.

Note that it'll complain at the end that the settings are incomplete. Fortunately it still writes the AP configuration to the camera. After this, you'll have to switch back to the lib-usb driver and then re-upload the streaming settings.

From here? Test! Unplug the camera and hit the Live mode again...

Auth Error? HDR-AS100V Official documentation indicates that this means it's failed to authenticate with UStream... makes sense to me! Can we hack my local network to make it auth to a fake server?

Network Setup for a Fake UStream Server

Since I want the raspi to be doing all the heavy lifting, I'm going to follow the instructions here at Luca Lanziani's project to set up a listener and a dns faker.

pi@raspberrypi:~ $ wget https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
--2020-04-03 16:01:51--  https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
Resolving github.com (github.com)... 13.237.44.5
Connecting to github.com (github.com)|13.237.44.5|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master [following]
--2020-04-03 16:01:51--  https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master
Resolving codeload.github.com (codeload.github.com)... 52.63.100.255
Connecting to codeload.github.com (codeload.github.com)|52.63.100.255|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3790 (3.7K) [application/zip]
Saving to: ‘master.zip’

master.zip                                  100%[===========================>]   3.70K  --.-KB/s    in 0s      

2020-04-03 16:01:51 (7.39 MB/s) - ‘master.zip’ saved [3790/3790]

It downloaded easy enough... let's see what's inside!

pi@raspberrypi:~ $ mv master.zip sony-camera-receiver.zip
pi@raspberrypi:~ $ unzip sony-camera-receiver.zip 
Archive:  sony-camera-receiver.zip
dd4b1170f9cdeaabc2cb68eccff6b7f63214abce
   creating: sony-camera-stream-receiver-master/
 extracting: sony-camera-stream-receiver-master/.gitignore  
  inflating: sony-camera-stream-receiver-master/README.md  
  inflating: sony-camera-stream-receiver-master/dns_server.js  
  inflating: sony-camera-stream-receiver-master/index.js  
  inflating: sony-camera-stream-receiver-master/package.json  
  inflating: sony-camera-stream-receiver-master/rtmp_server.js  
  inflating: sony-camera-stream-receiver-master/web_server.js  
pi@raspberrypi:~ $ cd sony-camera-stream-receiver-master/
pi@raspberrypi:~/sony-camera-stream-receiver-master $ ls
dns_server.js  index.js  package.json  README.md  rtmp_server.js  web_server.js

Cool, everything we need. Following the instructions, we first need to download all the supporting node libraries.

pi@raspberrypi:~/sony-camera-stream-receiver-master $ npm i
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN sony-cameras-stream-receiver@1.0.0 No repository field.

added 76 packages from 64 contributors in 17.452s

Cool, done. Now it's time to see if it'll actually run?

pi@raspberrypi:~/sony-camera-stream-receiver-master $ sudo node ./index.js --web.channel_number=12345678 --dns.external_ip=192.168.1.50
03/04/2020 16:09:24 16471 [INFO] Node Media Server v1.4.7
  dnsproxy:info we are up and listening at 192.168.1.50 on 53 +0ms
03/04/2020 16:09:24 16471 [INFO] Node Media Rtmp Server started on port: 1935
Server listening at http://0.0.0.0:80/
03/04/2020 16:09:26 16471 [INFO] There is a new version 2.1.6 that can be updated

Okey dokey.. it's up and running. The best thing we can do now is slap the new DNS server in our config and see what happens. After configuring my router and letting it reboot, nothing was resolving via .50. It seemed that .1 was still the DNS Server? I tried renewing my DHCP and flushing DNS, but nothing worked. Intead, I just hacked my local ethernet interface to point to .50 for DNS... woah! It works!

But I can't hack the camera in the same way... so I need to work out why DHCP isn't giving me .50. I then realised I'd been reconfiguring my repeater and not my actual router!? I logged into .1 and set the DHCP configuration to have a Primary DNS of .50 and, well, shit just worked.

The camera still threw the Auth Error, so I totally disabled the DHCP on the repeater, because that really shouldn't be there, and rebooted the camera. Hitting the REC button on Live mode on the Action Cam then saw action on the raspi!

03/04/2020 16:58:47 16471 [INFO] [rtmp connect] id=SRMCPZ41 ip=::ffff:192.168.1.134 app=12345678 args={"app":"12345678","flashVer":"LNX 9,0,124,0","swfUrl":"http://www.ustream.tv/mobile.swf","tcUrl":"rtmp://api.ustream.tv/12345678","capabilities":1,"audioCodecs":1143,"videoCodecs":252,"videoFunction":1,"pageUrl":"http://www.ustream.tv","objectEncoding":0,"fpad":false}
03/04/2020 16:58:47 16471 [INFO] [rtmp publish] New stream. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 streamId=1
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle video. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 frame_type=1 codec_id=7 codec_name=H264 0x0
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle audio. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 sound_format=10 sound_type=2 sound_size=1 sound_rate=3 codec_name=AAC 48000 2ch

03/04/2020 16:59:40 16471 [INFO] [rtmp connect] id=7YV3W0L2 ip=::ffff:192.168.1.121 app=12345678/broadcaster args={"app":"12345678/broadcaster","flashVer":"LNX 9,0,124,2","tcUrl":"rtmp://192.168.1.50:1935/12345678/broadcaster","fpad":false,"capabilities":15,"audioCodecs":4071,"videoCodecs":252,"videoFunction":1}
03/04/2020 16:59:40 16471 [INFO] [rtmp play] Join stream. id=7YV3W0L2 streamPath=/12345678/broadcaster/live7665  streamId=1

And then pointing VLC at rtmp://192.168.1.50/12345678/broadcaster/live7665 got me....

Traaaaaiinnnsssss... Now to find the perfect angle. Note that you get an 'ONAIR' message on the LCD when it's ... on air.

Sure, not the best mounting... I'll work on that. I've also got a waterproof case for it that I'll need to hack a hole into to get the USB power through.

Productionising

Having the raspi be the DNS server is a little scary. But I suppose this is where secondary DNS servers come in? If we put 8.8.8.8 for the secondary, does it get used when the primary is down? Turns out it doesn't work that way, so I'll need to keep the raspi up and running 100% of the time!

C:\Users\Steven>ipconfig /all
Ethernet adapter Ethernet:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) Ethernet Connection (2) I219-V
   Physical Address. . . . . . . . . : 30-5A-3A-81-E1-1B
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.121(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DHCP Server . . . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.50
                                       8.8.8.8
   NetBIOS over Tcpip. . . . . . . . : Enabled

C:\Users\Steven>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.

C:\Users\Steven>nslookup google.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.1.50

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

For some reason I really enjoy the spelling of 'UnKnown'. Regardless, the secondary has not kicked in! With this news, I suppose it'd be best to learn how to keep that service up and running 24/7 on the pi. Instead, I have decided to not even go ahead with this camera! It's wide-view fish-eye angle just isn't wide enough to get the trains close enough and the cityscape in the shot... I therefore don't see any advantage over my current camera.

So! A learning experience! For those who want to use a Sony Action Cam HDR-AS100V as a livestreaming camera, it's plausible!