Subscribe via RSS
23Feb/240

Python: Close Files If You’re Going To Open Them

I've been trying to archive some videos off Youtube lately, using yt-dlp. It's an amazing tool, but my target files have been episodes in parts. Usually four parts and Plex really hates jogging through... so what to do? Combine the files together with ffmpeg. The code was meant to be pretty simple (and 98% of it was written by ChatGPT... whoops)...

def concat_episodes(episode_name, concat_files):
    plfile = "file_list.txt"
    f = open(plfile, "w", encoding="utf-8")
    for filename in concat_files:
        f.write("file '" + filename + "'\r\n")
    concat_command = f"ffmpeg -stats -safe 0 -f concat -i {plfile} -c copy '{episode_name}'"
    print(concat_command)
    subprocess.run(concat_command, shell=True)

But no amount of wrangling would get ffmpeg to work. The concat filter kept throwing: Invalid data found when processing input. No amount of "-safe 0", relative paths or absolute paths worked! No permissions... no cwds or shell arguments. If I let the python script drop to the shell, then the same line pasted (since I printed it out) worked perfectly fine! What the?

OH RIGHT. I missed the memo that I should be closing a file so that it lands on disk... prior to trying to open it in another process!:

def concat_episodes(episode_name, concat_files):
    plfile = "file_list.txt"
    f = open(plfile, "w", encoding="utf-8")
    for filename in concat_files:
        f.write("file '" + filename + "'\r\n")
    f.close()
    concat_command = f"ffmpeg -stats -safe 0 -f concat -i {plfile} -c copy '{episode_name}'"
    print(concat_command)
    subprocess.run(concat_command, shell=True)

The file was still open and not flushed to disk... so ffmpeg would always open an empty file! This has been a public service announcement.

19Oct/200

Replacing the ear microphones on a Sony Aibo ERS-7

He'd arrived a few weeks back and gets energised a few hours per week... but he's a bad dog! He says "Good bye" in repsonse to me saying "Hello Aibo" and I'm going to blame him first... it can't be my mumbling? Can it? Just to be certain, I browsed around and found a DIY Repair Guide on Aibo Doctor. I quickly ordered a selection of microphones from element14 and got to work. Canine surgery is daunting!

The Microphones

As I always manage to get my orders wrong, I went ahead and ordered a selection of Microphones.

DSC00731

Note that an ERS-7 Aibo needs 6mm microphones, so you can see that one set is already incorrect. Next, there's directional and omni-directional. The microphones in Aibo are omni*, so I settled with the KECG2742TBL-A units.

Disassembling Aibo's Visor

Following the instructions here, one can disassemble Aibo's head to get to his ears. First step, remove the rubber earlobes by stretching them over the silver joint. Next, open his mouth and remove the two screws.

DSC00740

With these two removed, you can un-hook the rear of the bottom half of his head. This shell is not directly connected to his bottom jaw, so open his mouth fully and then push the whole lower-half back. With it unhooked, you can then bring it forward again and bring it over his jaw. With this piece out of the way, it's back to the front of his head where we need to remove the two outside screws on the metal plate inside his nose.

DSC00742

With those two out, you then need a longer screwdriver to get to the two screws deeper inside his skull. They're on either side of where his jaw would attach near his ears. See the image below, one screw on one side is in focus.

DSC00746

Finally, there's a clip on each side just in front of his ears, holding the visor on.

DSC00744

Very gently pry this open and the visor should lift up. You now have the option to disconnect all the cabling, but instead I just let it rest forward. Make sure it doesn't put undue pressure on the ribbon cables inside.

DSC00748

You now have access to the ear joints.

Disassembling Aibo's Ear Joints

Ok, this isn't the actual ear/microphone yet... before we get to that, there's a bit of fidgeting required to get the ear joints disassembled. When following the next steps, at no time will you need to apply excessive force! Doing so will most-probably damage poor Aibo. The ear joints are a two-part component and are built to be assembled/disassembled with ease. If you look at either side of Aibo's ears, you'll see that one side has a fixed arm and the other has an arm that slides backwards. The fixed arm is at the front and also has a little actuator (metal bar with notch) that is the mechanism that flaps Aibo's ears around when he's playing.

So, back to the joints: two pieces, first is removed by sliding the rear arm further to the rear. From there, you can grab the whole circular joint and unhook it from both front lugs. There's a lug in the actual main arm and a lug in the little black actuator arm.

DSC00780

The image above doesn't really help to explain how to undo it. The movement is one-shot and you can see in the photo above that the black actuator is on the left. This means that you're looking at Aibo's right ear and that you'd grab the top silver circle and slide it gently to the right (rear of his head) until both lugs are clear on the left. Once done, you can then slide off the rear shield.

DSC00752

From here, it's a single screw and the microphone + housing is free. Make sure you also unhook the cable!

Replacing Aibo's Inner-Ear

With the microphone unhooked and unscrewed, the silver shell can be lifted off (gently) by applying pressure to the joints that hold it on. With this off, you can really see what condition Aibo's ears are in!

DSC00756

From here, you can slide the microphone out, just enough, to be able to de-solder and solder a new component.

DSC00767 DSC00768 DSC00776

As you can see, I cheated and used the phone the new microphones came in to hold them in-place whilst soldering. I used the datasheet to make sure I got the polarity the correct way around, assuming that red was positive.

With everything wired up, the microphone was pushed back into the housing and re-assembled. Make sure that you have the wires in the correct groove according to the side the ear has been removed from!

Finally, as per above, there is absolutely no need for excessive force on any part of this assembly. I found that, once trying to re-fit the visor, it wouldn't sit flat! Turns out that I'd assembled the rear part of the ear-joint incorrectly and only one side of the guides was actually in the right spot!

DSC00793

If you look above, you can see that the left guide is sitting above the plastic strip that it was meant to slide onto! This meant totally dismantling that ear again to pull that part back and slide it back on again! Painful, but required.

Can Aibo hear me now?

Nope, same as before! Turns out you can just use Clinic Mode to determine if his microphones actually really need replacing! I should've done this first, but I also sorta wanted to play doctor and see what his insides looked like!

27Sep/200

Smushing JSON into submission!

I don't even know how to describe this operation on JSON. The basic idea is that we want to deserialise a stream into something legible, without creating a class for each item in an array. json2csharp believes that each item should be an array since it sees them all as individually named classes. Let me prove it! Here's a chunk'o'data:

{
    "trainCount":122,
    "requestTimestamp":1601124556789,
    "responseTimestamp":1601128156789,
    "nextUrl":"https://junatkartalla-cal-prod.herokuapp.com/trains/1601128156789",
    "trains":{
        "8":{
			"id":"8",
			"from":"LR",
			"to":"HKI",
			"title":"IC8",
			"latitude":60.172097,
			"longitude":24.941249,
			"speed":0,
			"direction":0,
			"category":"IC",
			"status":"1",
			"delta":60,
			"trainType":"LONGDISTANCE",
			"updated":1601127783841,
			"action":"deleted"
        },
        "9":{
			"id":"9",
			"from":"HKI",
			"to":"LR",
			"title":"S9",
			"latitude":60.571148,
			"longitude":25.199523,
			"speed":0,
			"direction":0,
			"category":"S",
			"status":"1",
			"delta":60,
			"trainType":"LONGDISTANCE",
			"updated":1601128143878,
			"action":"modified"
        }
    }
}

So, if you slam that JSON into json2csharp, you'll get the following:

    public class 8    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public long updated { get; set; } 
        public string action { get; set; } 
    }

    public class 9    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public long updated { get; set; } 
        public string action { get; set; } 
    }

    public class Trains    {
        public 8 8 { get; set; } 
        public 9 9 { get; set; } 
    }

    public class Root    {
        public int trainCount { get; set; } 
        public long requestTimestamp { get; set; } 
        public long responseTimestamp { get; set; } 
        public string nextUrl { get; set; } 
        public Trains trains { get; set; } 
    }

So, that's actually uncompilable code. Is uncompilable a word? Dunno... this is actually the first time that json2csharp has failed me! No matter the options selected on the site, the output was always bad code... json2csharp doesn't work with ALL json! So, what to do? Well, we actually need to mangle this JSON into submission. The best bet would be to move that train id/number into the array object as a parameter, rather than having it as the dictionary key. We have two methods to do this...

Using jQuery MAP Function

If you are pulling JSON from a hosted browser, then you can run JS in the browsers console and produce cleaner JSON. In this case, you want to use jQuery's MAP function to rewrite each array object:

json_data.trains = $.map(json_data.trains, function (data, idx) { return data; });

If you feed the JSON at the top of this post into that function, you'll get the following:

{
	"trainCount": 122,
	"requestTimestamp": 1601124556789,
	"responseTimestamp": 1601128156789,
	"nextUrl": "https://junatkartalla-cal-prod.herokuapp.com/trains/1601128156789",
	"trains": [{
		"id": "8",
		"from": "LR",
		"to": "HKI",
		"title": "IC8",
		"latitude": 60.172097,
		"longitude": 24.941249,
		"speed": 0,
		"direction": 0,
		"category": "IC",
		"status": "1",
		"delta": 60,
		"trainType": "LONGDISTANCE",
		"updated": 1601127783841,
		"action": "deleted"
	}, {
		"id": "9",
		"from": "HKI",
		"to": "LR",
		"title": "S9",
		"latitude": 60.571148,
		"longitude": 25.199523,
		"speed": 0,
		"direction": 0,
		"category": "S",
		"status": "1",
		"delta": 60,
		"trainType": "LONGDISTANCE",
		"updated": 1601128143878,
		"action": "modified"
	}]
}

Very nice.

Using C# Regex

I've never thoroughly learnt Regex and it still makes me sad. One day I'll sit down and go through a course. For now, googlin' works nicely to find an example for this scenario. Effectively we want to remove the "NUM": and just leave the curly braces. We then also need to remove a curly brace at the end.

            var data = (new System.Net.WebClient()).DownloadString(nextUrl);
            data = new Regex("\"([0-9]|[1-9][0-9]|[1-9][0-9][0-9]|[1-9][0-9][0-9][0-9]|[1-9][0-9][0-9][0-9][0-9])\":").Replace(data, "");
            data = data.Replace("\"trains\":{", "\"trains\":[");
            data = data.Replace("}}}", "}]}");

So, the first line downloads the data. The second uses Regex to find anything in double-quotes that's a number from 0-9. I actually wanted 0-9999, but it seems you need to match each character 1-by-1, so I gave it options with | (pipe) up to 99999. From there, the third line turns the trains object into an array and the final line closes that array with a square bracket instead of the existing curly brace.

Final C# Class Output

With either of the methods above, you end up with cleaned JSON. From here, slapping that back in json2csharp gets you the following:

    public class Train    {
        public string id { get; set; } 
        public string from { get; set; } 
        public string to { get; set; } 
        public string title { get; set; } 
        public double latitude { get; set; } 
        public double longitude { get; set; } 
        public int speed { get; set; } 
        public int direction { get; set; } 
        public string category { get; set; } 
        public string status { get; set; } 
        public int delta { get; set; } 
        public string trainType { get; set; } 
        public object updated { get; set; } 
        public string action { get; set; } 
    }

    public class Root    {
        public int trainCount { get; set; } 
        public long requestTimestamp { get; set; } 
        public long responseTimestamp { get; set; } 
        public string nextUrl { get; set; } 
        public List<Train> trains { get; set; } 
    }

Thank you json2charp, that's exactly the reusability that I was looking for!

16Sep/204

I’m not original…

In fact, my new best friend is over 16 years old!

DSC00644

I ... thiiiiink ... he's half deaf... might be time to replace his ears.

12Sep/200

EZIO + OS9 + Hypercard (Or Just Windows)

The EZIO Board is a serial-based I/O module that can connect to both Windows and Macintosh machines. Actually, it can connect to anything that speaks RS-232. One of these came up on eBay recently and I couldn't resist. I saw the Macintosh serial port and decided to give it a go. It's really similar to an Arduino, but from a few decades before the Arduino was even a dream. If I'd known about these back in the early 2000s then I would've definitely had a very nice automated model railway. But alas, I only happen to find one now thanks to eBay!

DSC00616

The unit has 10 digital output lines, 10 digital input lines, 8 analog-to-digital lines and two PWM lines. You then get 4 +5v terminals and 4 GND terminals. There's a PIC microcontroller in the middle running the show and a MAX232 for the RS-232 comms. The unit has a DC-rectifier, so you can feed it 5-15v either AC or DC. The 5-15v is literally the operating recommendations of the 7805 voltage regulator on-board.

There's a whole lot of code on the old site (yeah, you have to use web archive to get to it), but it's mainly for Director!? and Macintosh. Hilariously, the Macintosh example uses Hypercard! Before booting up the Power Mac, I instead wrote a quick bit of C# to test out the unit.

ezio-test-app

The shot above uses dotnet-ncurses, allowing the console to act like a canvas. It's really nice to be able to draw text to specific areas, rather than scrolling the screen. Anyway, the basic idea is that you can control the digital out and then everything else is read in. Interestingly, floating pins show some very random values... so if you're using this device, make sure you tie everything to ground or use pull-up resistors where appropriate.

Of course, the whole reason I bought this was due to the Macintosh serial port. I wasn't overly-energetic, so I tried a virtual Macintosh first...

ezio-basiliskii

After installing Hypercard, the app came up, but the performance once it started trying to interact with the serial port was terrible. It also just didn't work, so I gave up and booted up the Power Mac.

DSC00618

Yup, works a charm. Hypercard is pretty clunky, but I'm sure you could do a lot with it. I'll have to dig out the railway track and control a train!

6Aug/200

Fluke Multimeter Repair – Elastometric Strips

What what? I'd had this Fluke Multimeter sitting broken in my box'o'junk for literal decades. It used to work great, but failed at some point a long time ago and was never fixed. Recently, I snapped the cable off a shitty AUD$10 multimeter that I'd been using and so, in my infinite wisdom, thought I'd resurrect the Fluke!

DSC00395

It turned on, but the buttons didn't work.. so it wasn't much good except for the default setting of DC voltage. Testing voltages takes up 50% of my time, but I still need resistance and continuity! Let's rip it open...

DSC00397 DSC00403 DSC00405

So, not shown above (since I'd already fixed it and this post is months old) is that the strips of weird rubber weren't both there. The top strip was, but it turns out there's a lower strip needed. I had probably replaced the battery back in the day and somehow managed to discard the strip that conducts the button presses.

DSC00410

That's where it was meant to go. You can sorta see there's 5-ish segments on the PCB down in the channel, but there's a beautiful air-gap between that PCB and the mainboard. What's meant to go in the middle? I popped out the strip at the top and I, telling the truth, had no idea what the material was. A little bit of googlin' allowed me to realise that it was elastometric strip! Andy's Surplus came to the rescue and quickly delivered two cuttable strips of elastometricity!

Cut.. cut again...

DSC00415

And cut once more and ... TADA!

DSC00417

It works. And whilst the beast is open, replace the battery and solder that battery terminal!

DSC00409

Damn this thing is beautiful.

29Jul/202

Burning 16-bit EPROMs with a Willem Programmer

This wasn't fun! I wanted to get Kickstart 1.3 on my Amiga 500 and so I purchased some M27C400 EPROMs from eBay with the thought that I could host two ROMs on them and make it switchable. The ROMs arrived and, to my stupidity, didn't fit in a standard Willem EPROM Programmer! Of course, a quick google prior to purchasing anything would've made this very apparent. Fortunately, there's a fix! You can buy a DIP42 16-bit EPROM Adapter for the Willem, and so I did.

DSC00241

Thanks to COVID, it took an extra-specially long time to arrive and, once it did, got thrown into the box'o'Amiga'junk as I'd put the whole lot on the backburner whilst working on the Ataris of late. This weekend I'd finally managed to wrangle some time together and sat down with my trusty XP laptop to program these chips.... It wasn't easy!

Input Voltages

I jumped head-first into the programming and failed a LOT with errors such as 0x00 and 0x11. Any search online will tell you that, when getting lots of 0x00 and 0x11 errors, it mainly indicates that you aren't supplying the correct programming voltage for your target chip. Check out the PDF Datasheet for the chip you want to burn, it'll specify a programming voltage that it requires to successfully understand and store the bits that you're about to throw at it.

For the M27C400, we need to provide 12.5v +/- 0.25v. The Willem has a blue trimmable capacitor (or is that a resistor) on the top-right of the board that you can adjust to get this voltage. I always recommend to use an external power supply with these programmers and so I have a plug-pack which happily provides 12v @ 2amps. To make sure we'll have a chance of programming this chip, switch to the Test H/W tab of the software and tick VPP. This is pin 1 of the E32 socket. If this has worked, then you'll have a red LED illuminated on the board. Now, check the voltage between pin 16 (GND) and pin 1 (VPP) of the socket.

DSC00252

Mine was reporting a value lower than 12v to start with and hence I had a few issues burning. With a multimeter in place, I tuned the trimmer on the top-right and got the value to 12.5v. Actually, it was probably 12.6... but 12.4 would also work as you just need to be in the range of 12.25 - 12.75.

Another test burn saw a new error... no more 0x00 and 0x11, it was now further into the chip, usually around 0x100?

Willem Software vs. Programmer Settings

Next up, we need to make sure that you can correctly control ALL address lines with your Willem Programmer. After fixing the initial errors above, I started getting new errors at random intervals into the burning process. The errors all seemed to indicate that there was an issue writing data after address 0x100. I'd only been using empty chips, so it couldn't be because they already had data on them. You can also check if the chips are empty prior to burning and I totally recommend doing this. Either way, after this new error was received, I chose to then read the chips and view the data:

amiga-repeat

What you're seeing above is the data read back from the chip. I wrote the rom (loaded from file) to the chip, got the error and then chose to read the data from the chip back into the application. 75% of the data on that screen is correct, but you'll notice that the data repeats itself from 0x100! It turns out that this was all due to the fact that my Willem was only controlling the first 8 address lines and leaving the rest low. I recommend you rig up an LED test harness as per below and control the address pins via the Test H/W tab on the Willem Software.

pcb-settings

After swapping chips around, checking continuity between the 4051s and reseating everything else, I came to the conclusion that there's incorrect information in the Standard Willem Programming Guide. If you scroll to page 5, you'll find information on the version of the software to use. It suggests that, based on jumper settings on the board, you can select if the board emulates a PCB3B or PCB3.5. If you've chosen PCB3B, then you'll need to use software version 0.97ja, as per my post from a long time ago, when I was failing with the same issue! Alternatively, if you've chosen PCB3.5, you need to use 0.98d6 or higher.

DSC00242

I looked at my board and checked out the jumpers that set the board emulation. Mine were actually individually set to 1-2 and 2-3 instead of both 1-2 or both 2-3. Hah... how did this thing ever work?! With these jumpers now set to 2-3 (aka PCB3B), I went back to version 0.97ja of the software and tried to get the full suite of address lines to light up via the Test H/W tab. No dice! If you browse to page 6 of the above PDF, you'll find that the jumpers should be set to 1-2 for PCB35 and 2-3 for PCB3B. Just for fun, I switched the jumpers to 1-2 and tested out version 0.98d6 of the burning software... still no dice! No permutation of checkboxes on the Test H/W tab would get a high signal on any of the address lines higher than A7! Up until this point in time, I'd only programmed 8-bit EPROMs, so I'd never noticed this.

I went ahead and played musical-chips. There's a chain of CD4015BE ICs that shift out the address lines, 3 of them with 1 each controlling 8 bits. It seemed plausible that there might have been a faulty connection between the first and second, but no amount of tinkering brought the address pins to life. I took a break as the frustration (or was that sadness) levels were elevating and sat down to a bit of TV. With a fresh mindset, I came back to the device, forgetting which settings were applied. Opening up version 0.97ja and testing address lines now worked!? Wait, the jumpers are set for PCB35... what the hell, it's backwards!? I opened 0.98d6 and it didn't work, so I set the jumpers back to 2-3 (for PCB3B) and it now 0.98d6 worked!? Urgh... so much time lost. So, TLDR; The PCB3B/PCB35 jumper settings are backwards on my Willem Programmer. Set them to 1-2 for PCB3B and 2-3 for PCB35!

Using the adapter

First up, there's two jumpers on the adapter to configure. The right-most controls the VPP/A21 line and needs to be set to 1-2 for M27C400 or M27C800 EPROMs. The other, in the middle of the board, I left at 1-2 as I don't know what it does.

DSC00244

Next up, there's a ribbon cable used to obtain more address pins from the main programming board. For the M27C400, the base E32 socket of the Willem actually produces enough address lines, so this ribbon cable isn't actually needed. Check out the diagram below to work out how the adapter is constructed. I've coloured the lines that are used on the M27C400 and 800, where you can see that the 400 doesn't use any of the wires from the header on the right.

dip42-adapter

If you're going to program an M27C800 or larger, you'll need to connect this. Simply make sure that pin A23 (bottom-most) lines up with A23 on the Willem board.

DSC00253

Finally, when inserting ANY chip into to the socket, make sure that the pins are all straight and that none will foul. For any of the chips, always make sure that the bottom pins are at the bottom end of the socket. The EPROMs are designed so that the Address lines extend 'north', so the silhouette on the board is actually for an M27C800. An M27C400 will be one pin lower than the silhouette, aligned at the bottom of the socket.

Conclusion

All the errors I made above would've been avoided if I'd known what I was doing from the start. Usually the old RTFM instruction makes sense, but when the jumper settings work in reverse to the manual, it's a little frustrating. On another note, I'd just trusted that this piece of hardware worked, as I've managed to burn other chips in the past. Assuming that cheap hardware just-works(tm) is a flaw in itself. Always make sure you know the hardware you're working with back-to-front, which was totally possible this time around thanks to open-source schematics being available!

7Apr/200

Mixing And Matching KVM Cables

Thanks to these new working-from-home shennanigans, I had to dig through the to-sell-at-trash-and-treasure box and find a KVM Switch to hook up all my machines. The desk used to only have one PC, but now additionally houses my work laptop and a large development server. I've got two monitors, and I want one of those to switch over to the development PC when required. I also want my keyboard and mouse to switch between all devices. To do this, I'll use the VGA/USB KVM to channel input and video to the monitor.

DSC00016

The best KVM in the box'o'stuff was a Belkin SOHO 4-port VGA+USB+Audio switch. Fortunately it had two of the KVM cables, so I could switch in my Laptop and development machine... but... I had to then unplug the USB to switch the input to the main PC. This wasn't optimal, so I went digging in the box again for another cable. The Belkin cables switched both Audio in+out, VGA and USB. From the outside, it looked like the audio channels were external to the VGA cable, unlike the USB which does seem to run through the same wires.

Back into the box, but unfortunately there were no other Belkin-branded cables. What I did find was another KVM with USB switching: Level One 2-Port VGA+USB KVM Switch.

DSC00033

DSC00013

This one also switched audio, but via USB instead of standard audio jacks. I grabbed a cable from it and was about to just plug everything together, but then I halted. USB has four wires, with two of those being the 5v rails. If there's any wiring differences between the switches then I could well send voltages in haphazard directions and cook equipment!

USB over VGA cable pinout

Wanting to hook this all together, I reached for my multimeter and started mapping out both the Level One and Belkin KVM cables. Thankfully, they have used very similar wiring for their cables! Thanks to monitors now talking digitally to hosts when describing their capabilities, there are spare pins on the cables to use for other purposes. There's a lot of extra ground pins also, but Belkin still preferred to keep those split whilst Level One happily merged them altogether.

DSC00019

Pin VGA Level One Cable Belkin Cable
1 R
2 G
3 B
4 ID2 / RES USB D- (White) USB D+ (Green)
5 GND USB D+ (Green) USB D- (White)
6 RGND USB GND + GND USB GND + RGND
7 GGND USB GND + GND GND
8 BGND USB GND + GND GND
9 KEY / 5v USB 5v + 5v USB 5v + 5v
10 SGND USB GND + GND GND
11 ID0 / RES
12 ID1 / SDA
13 HSYNC
14 VSYNC
15 ID3 / SCL

From above, you can see that the voltage (thanks to VGA actually having a +5v pin as standard!) and GND are close-enough, but the USB data pins are backwards. You assholes.. here I was thinking there might be a standard. What to do here? Well, I simply hacked apart a USB extension cable and crossed over the D+ and D-. Note that there is absolutely no reason for using this cable in any other scenario!

DSC00034

After this, one last hurdle.. the Level One KVM has a male plug at either end, whereas the Belkin cables have female on the KVM side.

DSC00028 DSC00023 DSC00032

And the bloody thing just worked! Happy to not have to spend money on extra proprietary cables that I actually couldn't find anywhere.

3Apr/206

Sony HDR-AS100V Action Cam – Further Hacking

After a LOT of mucking around, trying to get a stream URL from the Action Cam, I did further googling on the defunct UStream service that the camera could, supposedly, stream live to. It wasn't long before I came across this mammoth thread on Github describing a few users' attempts to hack apart the APK that does the steaming and to fake a server with the intent to intercept the feed.

They use the Sony-PMCA-RE Project code to tweak settings on the camera and then built up an entire webserver to fake the Ustream service. None of this was beyond me, so I started by trying to work out if I could configure my camera to point at a server... (note that I've fudged any secrets in the console logs below...)

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe info -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Model:              HDR-AS100V
Product code:       0002426601
Serial number:      03035253
Firmware version:   2.00
GPS Data:           2020-04-02 00:00:00 - 2020-05-02 00:00:00
C:\Downloads>pmca-console-v0.17-win.exe stream -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

twitterEnabled:     0
twitterConsumerKey: o9fJ2342342423433qg
twitterConsumerSecret: TTL4324234234234324324llM6EQdivTXesA
twitterAccessToken1:
twitterAccessTokenSecret:
twitterMessage:     Live Streaming from Action Cam by Sony
facebookEnabled:    0
facebookAccessToken:
facebookMessage:    Live Streaming from Action Cam by Sony
service:            0
enabled:            1
macId:
macSecret:
macIssueTime:       0000000000000000
unknown:            1
channels:           [0]
shortURL:
videoFormat:        1
supportedFormats:   [1, 3]
enableRecordMode:   1
videoTitle:         Recorded with Action Cam by Sony
videoDescription:   Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself
videoTag:
C:\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Traceback (most recent call last):
  File "C:\projects\sony-pmca-re\pmca-console.py", line 96, in <module>
  File "C:\projects\sony-pmca-re\pmca-console.py", line 82, in main
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 360, in updaterShellCommand
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 373, in firmwareUpdateCommandInternal
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 527, in checkGuard
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 501, in _sendWriteCommands
Exception: Firmware update error: Low battery
[9052] Failed to execute script pmca-console

Right... low battery... I charged it up and then tried again:

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Switching to updater mode

Waiting for camera to switch...
Please follow the instructions on the camera screen.

DSC04100

Cool, at this point the camera showed 'EXEC' on its LCD display. This kept flashing... and then the console app error'd out:

Operation timed out. Please run this command again when your camera has connected.

I pressed the record button on the back of the camera to try and force the 'EXEC' and it seemed to reboot. From here, it then wouldn't even come back to life. The battery charge light just stayed a fixed red, which I now think was an error indicator. After popping the battery out, I could get it back to the 'EXEC' screen via the console app, but it occurred to me that I actually had no idea what updater-shell even did. Instead I tried the stream fuction to push a UStream configuration file to the device as per the instructions here at Novex's project where he attempts to create his own UStream APK. Here's the config I used...

[
    [
        "twitterEnabled",
        0
    ],
    [
        "twitterConsumerKey",
        ""
    ],
    [
        "twitterConsumerSecret",
        ""
    ],
    [
        "twitterAccessToken1",
        ""
    ],
    [
        "twitterAccessTokenSecret",
        ""
    ],
    [
        "twitterMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "facebookEnabled",
        0
    ],
    [
        "facebookAccessToken",
        ""
    ],
    [
        "facebookMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "service",
        0
    ],
    [
        "enabled",
        1
    ],
    [
        "macId",
        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    ],
    [
        "macSecret",
        "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
    ],
    [
        "macIssueTime",
        "dbeaaaaa00000000"
    ],
    [
        "unknown",
        1
    ],
    [
        "channels",
        [
            12345678
        ]
    ],
    [
        "shortURL",
        "http://www.ustream.tv/channel/asdf"
    ],
    [
        "videoFormat",
        3
    ],
    [
        "supportedFormats",
        [
            1,
            3
        ]
    ],
    [
        "enableRecordMode",
        0
    ],
    [
        "videoTitle",
        "Recorded with Action Cam by Sony"
    ],
    [
        "videoDescription",
        "Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself"
    ],
    [
        "videoTag",
        ""
    ]
]

And then I smushed it into the camera using the console application...

C:\Downloads>pmca-console-v0.17-win.exe stream -w ustream.json -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Urrghh... what mode does it need to be in? Wait, is that even an error? Let's try use the buttons on the camera to switch it into Live mode...

DSC04105

Preparing?

DSC04103

No AP? Maybe it has no idea how to connect to a 5Ghz access point? Let's set it to my 2.4G secondary network, which nothing else uses, so I can isolate and watch any traffic. This took a little bit of hacking as the Sony Network Settings Utility needs the driver to be reverted back to USB Mass Storage. Jump in to Device Manager and force a driver 'update' back to the standard driver.

reset-to-usb-mass-storage

Now you can go into Network Settings and set your AP once more.

net-set-tool-1

wifi-connection incomplete-settings incomplete-settings-2

Note that it'll complain at the end that the settings are incomplete. Fortunately it still writes the AP configuration to the camera. After this, you'll have to switch back to the lib-usb driver and then re-upload the streaming settings.

remap-usb-driver-1 remap-usb-driver-2 remap-usb-driver-3

From here? Test! Unplug the camera and hit the Live mode again...

DSC04111

Auth Error? HDR-AS100V Official documentation indicates that this means it's failed to authenticate with UStream... makes sense to me! Can we hack my local network to make it auth to a fake server?

Network Setup for a Fake UStream Server

Since I want the raspi to be doing all the heavy lifting, I'm going to follow the instructions here at Luca Lanziani's project to set up a listener and a dns faker.

pi@raspberrypi:~ $ wget https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
--2020-04-03 16:01:51--  https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
Resolving github.com (github.com)... 13.237.44.5
Connecting to github.com (github.com)|13.237.44.5|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master [following]
--2020-04-03 16:01:51--  https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master
Resolving codeload.github.com (codeload.github.com)... 52.63.100.255
Connecting to codeload.github.com (codeload.github.com)|52.63.100.255|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3790 (3.7K) [application/zip]
Saving to: ‘master.zip’

master.zip                                  100%[===========================>]   3.70K  --.-KB/s    in 0s      

2020-04-03 16:01:51 (7.39 MB/s) - ‘master.zip’ saved [3790/3790]

It downloaded easy enough... let's see what's inside!

pi@raspberrypi:~ $ mv master.zip sony-camera-receiver.zip
pi@raspberrypi:~ $ unzip sony-camera-receiver.zip 
Archive:  sony-camera-receiver.zip
dd4b1170f9cdeaabc2cb68eccff6b7f63214abce
   creating: sony-camera-stream-receiver-master/
 extracting: sony-camera-stream-receiver-master/.gitignore  
  inflating: sony-camera-stream-receiver-master/README.md  
  inflating: sony-camera-stream-receiver-master/dns_server.js  
  inflating: sony-camera-stream-receiver-master/index.js  
  inflating: sony-camera-stream-receiver-master/package.json  
  inflating: sony-camera-stream-receiver-master/rtmp_server.js  
  inflating: sony-camera-stream-receiver-master/web_server.js  
pi@raspberrypi:~ $ cd sony-camera-stream-receiver-master/
pi@raspberrypi:~/sony-camera-stream-receiver-master $ ls
dns_server.js  index.js  package.json  README.md  rtmp_server.js  web_server.js

Cool, everything we need. Following the instructions, we first need to download all the supporting node libraries.

pi@raspberrypi:~/sony-camera-stream-receiver-master $ npm i
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN sony-cameras-stream-receiver@1.0.0 No repository field.

added 76 packages from 64 contributors in 17.452s

Cool, done. Now it's time to see if it'll actually run?

pi@raspberrypi:~/sony-camera-stream-receiver-master $ sudo node ./index.js --web.channel_number=12345678 --dns.external_ip=192.168.1.50
03/04/2020 16:09:24 16471 [INFO] Node Media Server v1.4.7
  dnsproxy:info we are up and listening at 192.168.1.50 on 53 +0ms
03/04/2020 16:09:24 16471 [INFO] Node Media Rtmp Server started on port: 1935
Server listening at http://0.0.0.0:80/
03/04/2020 16:09:26 16471 [INFO] There is a new version 2.1.6 that can be updated

Okey dokey.. it's up and running. The best thing we can do now is slap the new DNS server in our config and see what happens. After configuring my router and letting it reboot, nothing was resolving via .50. It seemed that .1 was still the DNS Server? I tried renewing my DHCP and flushing DNS, but nothing worked. Intead, I just hacked my local ethernet interface to point to .50 for DNS... woah! It works!

dhcp-1 dhcp-2 dhcp-3

But I can't hack the camera in the same way... so I need to work out why DHCP isn't giving me .50. I then realised I'd been reconfiguring my repeater and not my actual router!? I logged into .1 and set the DHCP configuration to have a Primary DNS of .50 and, well, shit just worked.

The camera still threw the Auth Error, so I totally disabled the DHCP on the repeater, because that really shouldn't be there, and rebooted the camera. Hitting the REC button on Live mode on the Action Cam then saw action on the raspi!

03/04/2020 16:58:47 16471 [INFO] [rtmp connect] id=SRMCPZ41 ip=::ffff:192.168.1.134 app=12345678 args={"app":"12345678","flashVer":"LNX 9,0,124,0","swfUrl":"http://www.ustream.tv/mobile.swf","tcUrl":"rtmp://api.ustream.tv/12345678","capabilities":1,"audioCodecs":1143,"videoCodecs":252,"videoFunction":1,"pageUrl":"http://www.ustream.tv","objectEncoding":0,"fpad":false}
03/04/2020 16:58:47 16471 [INFO] [rtmp publish] New stream. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 streamId=1
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle video. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 frame_type=1 codec_id=7 codec_name=H264 0x0
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle audio. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 sound_format=10 sound_type=2 sound_size=1 sound_rate=3 codec_name=AAC 48000 2ch

03/04/2020 16:59:40 16471 [INFO] [rtmp connect] id=7YV3W0L2 ip=::ffff:192.168.1.121 app=12345678/broadcaster args={"app":"12345678/broadcaster","flashVer":"LNX 9,0,124,2","tcUrl":"rtmp://192.168.1.50:1935/12345678/broadcaster","fpad":false,"capabilities":15,"audioCodecs":4071,"videoCodecs":252,"videoFunction":1}
03/04/2020 16:59:40 16471 [INFO] [rtmp play] Join stream. id=7YV3W0L2 streamPath=/12345678/broadcaster/live7665  streamId=1

And then pointing VLC at rtmp://192.168.1.50/12345678/broadcaster/live7665 got me....

traiiiinnnnssss

Traaaaaiinnnsssss... Now to find the perfect angle. Note that you get an 'ONAIR' message on the LCD when it's ... on air.

DSC00001

Sure, not the best mounting... I'll work on that. I've also got a waterproof case for it that I'll need to hack a hole into to get the USB power through.

Productionising

Having the raspi be the DNS server is a little scary. But I suppose this is where secondary DNS servers come in? If we put 8.8.8.8 for the secondary, does it get used when the primary is down? Turns out it doesn't work that way, so I'll need to keep the raspi up and running 100% of the time!

C:\Users\Steven>ipconfig /all
Ethernet adapter Ethernet:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) Ethernet Connection (2) I219-V
   Physical Address. . . . . . . . . : 30-5A-3A-81-E1-1B
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.121(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DHCP Server . . . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.50
                                       8.8.8.8
   NetBIOS over Tcpip. . . . . . . . : Enabled
C:\Users\Steven>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.
C:\Users\Steven>nslookup google.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.1.50

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

For some reason I really enjoy the spelling of 'UnKnown'. Regardless, the secondary has not kicked in! With this news, I suppose it'd be best to learn how to keep that service up and running 24/7 on the pi. Instead, I have decided to not even go ahead with this camera! It's wide-view fish-eye angle just isn't wide enough to get the trains close enough and the cityscape in the shot... I therefore don't see any advantage over my current camera.

So! A learning experience! For those who want to use a Sony Action Cam HDR-AS100V as a livestreaming camera, it's plausible!

3Apr/201

Streaming from a Sony HDR-AS100V Action Cam

This is a cool camera. I picked it up from the Hard Off in Kagoshima whilst I was visiting the southernmost JR Railway station in Japan. I then used it to take a miriad of videos, especially from the balcony at the apartment in Shin Osaka.

DSC04151

DSC04152 DSC04153 DSC04156

Whilst it works great recording by itself, you can also use the Play Mobile (now Imaging Edge Mobile) application from Sony to control it remotely. The application uses the phones Wifi to connect to the camera, which acts as a wifi access point. From there, it shows a live view of what the camera can see whilst you're trying to line up the next shot. You can even control the camera remotely with the buttons on screen, being able to start and stop recording very easily... especially without bumping the camera!

Screenshot 20200331-143248

I was hoping Sony would release the same software for the PC and allow me to connect wirelessly to the camera. This would then let me sniff the traffic and get the video stream live link, but alas, it's only available for mobile devices. Wait!, you say, there is a version of Play Memories for windows! Sure, go ahead and download it... install it even! It'll connect to your camera, but there's no ability to see the live stream from the lens... I tried... trust me....

camera-options actioncam-import streaming-tool

With the USB camera connected, you'll get the screens above. They let you update firmware and manage settings, but not much else. Of course, there's also the file transfer functionality to let you download the images and videos you've taken. Interestingly, the last image shows the second pane of the Network Settings tab which allows you to configure the camera to upload to UStream automatically. Turns out that this was a 'thing' back in the day and not really practical anymore: IBM bought UStream and the service is no longer free... nor could I get it to work, even with a trial account.

Can we create a network to inspect the Wifi?

So, I had a plan. I have a Vonets VAR11N-300 that allows an ethernet cabled device to access an access point over Wifi. With this, I could use an ethernet port on my laptop to connect to the camera. The Wifi adapter in the laptop could then create an ad-hoc network that my phone could connect to. If I bridged the ethernet to the Wifi, connected my phone ad-hoc to the wifi network created by the laptop, then I could possibly pass the traffic through the laptop and use wireshark to listen in?

s-l1600

As I was plugging this all together, traffic started flowing without client application requesting anything! I'd not yet bridged the adapters, but I'd installed Wireshark and had the VAR11N-300 connected to the camera and, all of a sudden, I'd noticed jitters on the traffic graph for the ethernet interface on the front page of Wireshark. The ethernet port seems to have received an IP from the camera since it was acting as a terminating host and not an ethernet bridge! For everyones information: the IP range is 196.168.0.x, with IPs dished out from 1 and the camera being 192.168.122.1. With the little graph jumping around, I decided to listen in and have a look!

ws-1 ws-2 upnp-xml

Ok so, from left to right above... the Wireshark main screen with my miriad of interfaces. You may notice that I have Wifi-2 selected there and there's traffic flowing on it... At this point, realising that I didn't need the mobile application to trigger data flow from the camera, I'd slapped a USB Wifi dongle into my main desktop so that I could work in comfort. The second screen is the main window of Wireshark once you're sniffing. You'll see a huge list of traffic up the top. Wireshark, without filters, will just present everything it can possibly see. Each row is a packet sent over the Wifi and you may need to combine multiple rows to see an entire picture, depending on what you're looking for. Usually, if this was my main network interface, I'd use a filter at the top with either ip.dest == 192.x.x.x or ip.src == 192.x.x.x to limit the displayed data, but fortunately this interface was only talking to the camera.

The second shot there has a very interesting packet displayed. It turns out that the camera is broadcasting UPnP information, telling anyone listening on the immediate network how to find it! No wonder, once you've connected your phone to the camera, that it can find it. It doesn't even have to look as the camera just yells it out every few seconds. Let's check out the data we're receiving...

<?xml version="1.0" encoding="utf-8"?>
<root xmlns="urn:schemas-upnp-org:device-1-0" xmlns:dlna="urn:schemas-dlna-org:device-1-0" xmlns:av="urn:schemas-sony-com:av">
	<specVersion>
		<major>1</major>
		<minor>0</minor>
	</specVersion>
	<device>
		<dlna:X_DLNADOC xmlns:dlna="urn:schemas-dlna-org:device-1-0">DMS-1.50</dlna:X_DLNADOC>
		<deviceType>urn:schemas-upnp-org:device:MediaServer:1</deviceType>
		<friendlyName>HDR-AS100V</friendlyName>
		<manufacturer>Sony Corporation</manufacturer>
		<manufacturerURL>http://www.sony.com/</manufacturerURL>
		<modelDescription>SonyDigitalMediaServer</modelDescription>
		<modelName>SonyImagingDevice</modelName>
		<modelURL>http://www.sony.net/</modelURL>
		<UDN>uuid:00000000-0005-0010-8000-fcc2de658d5e</UDN>
		<serviceList>
			<service>
				<serviceType>urn:schemas-upnp-org:service:ContentDirectory:1</serviceType>
				<serviceId>urn:upnp-org:serviceId:ContentDirectory</serviceId>
				<SCPDURL>/CdsDesc.xml</SCPDURL>
				<controlURL>/upnp/control/ContentDirectory</controlURL>
				<eventSubURL>/upnp/event/ContentDirectory</eventSubURL>
			</service>
			<service>
				<serviceType>urn:schemas-upnp-org:service:ConnectionManager:1</serviceType>
				<serviceId>urn:upnp-org:serviceId:ConnectionManager</serviceId>
				<SCPDURL>/CmsDesc.xml</SCPDURL>
				<controlURL>/upnp/control/ConnectionManager</controlURL>
				<eventSubURL>/upnp/event/ConnectionManager</eventSubURL>
			</service>
			<service>
				<serviceType>urn:schemas-sony-com:service:ScalarWebAPI:1</serviceType>
				<serviceId>urn:schemas-sony-com:serviceId:ScalarWebAPI</serviceId>
				<SCPDURL>/ScalarWebApiDesc.xml</SCPDURL>
				<controlURL/>
				<eventSubURL/>
			</service>
		</serviceList>
		<iconList>
			<icon>
				<mimetype>image/jpeg</mimetype>
				<width>48</width>
				<height>48</height>
				<depth>24</depth>
				<url>/DLNA_camera_48.jpg</url>
			</icon>
			<icon>
				<mimetype>image/jpeg</mimetype>
				<width>120</width>
				<height>120</height>
				<depth>24</depth>
				<url>/DLNA_camera_120.jpg</url>
			</icon>
			<icon>
				<mimetype>image/png</mimetype>
				<width>48</width>
				<height>48</height>
				<depth>24</depth>
				<url>/DLNA_camera_48.png</url>
			</icon>
			<icon>
				<mimetype>image/png</mimetype>
				<width>120</width>
				<height>120</height>
				<depth>24</depth>
				<url>/DLNA_camera_120.png</url>
			</icon>
		</iconList>
		<av:X_ScalarWebAPI_DeviceInfo xmlns:av="urn:schemas-sony-com:av">
			<av:X_ScalarWebAPI_Version>1.0</av:X_ScalarWebAPI_Version>
			<av:X_ScalarWebAPI_ServiceList>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>guide</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>accessControl</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>camera</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
			</av:X_ScalarWebAPI_ServiceList>
			<av:X_ScalarWebAPI_ImagingDevice>
				<av:X_ScalarWebAPI_LiveView_URL>http://192.168.122.1:60152/liveview.JPG?%211234%21http%2dget%3a%2a%3aimage%2fjpeg%3a%2a%21%21%21%21%21</av:X_ScalarWebAPI_LiveView_URL>
				<av:X_ScalarWebAPI_DefaultFunction>RemoteShooting</av:X_ScalarWebAPI_DefaultFunction>
			</av:X_ScalarWebAPI_ImagingDevice>
		</av:X_ScalarWebAPI_DeviceInfo>
	</device>
</root>

Oooowwweee... Check that link at the bottom!? X_ScalarWebAPI_LiveView_URL? Really? Can it be this easy? I slapped it into a browser and no... it's not this easy... the browser just doesn't respond as it's constantly being fed a stream of data and doesn't know what to do with it! Turns out this stream is a motion-jpeg or some frame-by-frame binary flow that isn't a standard video stream. So, what to do? Google the hell out of those keywords and see who has already come to our rescue!

First hit was a post on stackoverflow asking 'How to get Sony ScalarWebAPI method list'. From the responses, one person suggested to send the camera JSON via a post so that you can see what functions you can call. One such function is getMethodTypes and can be called via the following post body:

{"method": "getMethodTypes", "params": [""], "id": 1, "version": "1.0"}

posty

Using Postman, I got the following result...

{
    "id": 1,
    "results": [
        [
            "actTakePicture",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getApplicationInfo",
            [],
            [
                "string",
                "string"
            ],
            "1.0"
        ],
        [
            "getAvailableApiList",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableCameraFunction",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableMovieQuality",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableShootMode",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableSteadyMode",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getCameraFunction",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getEvent",
            [
                "bool"
            ],
            [
                "{\"type\":\"string\", \"names\":\"string*\"}",
                "{\"type\":\"string\", \"cameraStatus\":\"string\"}",
                "{\"type\":\"string\", \"zoomPosition\":\"int\", \"zoomNumberBox\":\"int\", \"zoomIndexCurrentBox\":\"int\", \"zoomPositionCurrentBox\":\"int\"}",
                "{\"type\":\"string\", \"liveviewStatus\":\"bool\"}",
                "{\"type\":\"string\", \"liveviewOrientation\":\"string\"}",
                "{\"type\":\"string\", \"takePictureUrl\":\"string*\"}*",
                "{\"type\":\"string\", \"continuousError\":\"string\", \"isContinued\":\"bool\"}*",
                "{\"type\":\"string\", \"triggeredError\":\"string*\"}",
                "{\"type\":\"string\", \"sceneRecognition\":\"string\", \"steadyRecognition\":\"string\", \"motionRecognition\":\"string\"}",
                "{\"type\":\"string\", \"formatResult\":\"string\"}",
                "{\"type\":\"string\", \"storageID\":\"string\", \"recordTarget\":\"bool\", \"numberOfRecordableImages\":\"int\", \"recordableTime\":\"int\", \"storageDescription\":\"string\"}*",
                "{\"type\":\"string\", \"currentBeepMode\":\"string\", \"beepModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentCameraFunction\":\"string\", \"cameraFunctionCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentMovieQuality\":\"string\", \"movieQualityCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentAspect\":\"string\", \"currentSize\":\"string\"}",
                "{\"type\":\"string\", \"cameraFunctionResult\":\"string\"}",
                "{\"type\":\"string\", \"currentSteadyMode\":\"string\", \"steadyModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentViewAngle\":\"int\", \"viewAngleCandidates\":\"int*\"}",
                "{\"type\":\"string\", \"currentExposureMode\":\"string\", \"exposureModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentPostviewImageSize\":\"string\", \"postviewImageSizeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentSelfTimer\":\"int\", \"selfTimerCandidates\":\"int*\"}",
                "{\"type\":\"string\", \"currentShootMode\":\"string\", \"shootModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentAELock\":\"bool\", \"aeLockCandidates\":\"bool*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentBracketShootMode\":\"string\", \"currentBracketShootModeOption\":\"string\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentCreativeStyle\":\"string\", \"currentCreativeStyleContrast\":\"int\", \"currentCreativeStyleSaturation\":\"int\", \"currentCreativeStyleSharpness\":\"int\"}",
                "{\"type\":\"string\", \"currentExposureCompensation\":\"int\", \"maxExposureCompensation\":\"int\", \"minExposureCompensation\":\"int\", \"stepIndexOfExposureCompensation\":\"int\"}",
                "{\"type\":\"string\", \"currentFlashMode\":\"string\", \"flashModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentFNumber\":\"string\", \"fNumberCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentFocusMode\":\"string\", \"focusModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentIsoSpeedRate\":\"string\", \"isoSpeedRateCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentPictureEffect\":\"string\", \"currentPictureEffectOption\":\"string\"}",
                "{\"type\":\"string\", \"isShifted\":\"bool\"}",
                "{\"type\":\"string\", \"currentShutterSpeed\":\"string\", \"shutterSpeedCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentWhiteBalanceMode\":\"string\", \"currentColorTemperature\":\"int\"}",
                "{\"type\":\"string\", \"currentSet\":\"bool\", \"currentTouchCoordinates\":\"double*\"}"
            ],
            "1.0"
        ],
        [
            "getMethodTypes",
            [
                "string"
            ],
            [
                "string",
                "string*",
                "string*",
                "string"
            ],
            "1.0"
        ],
        [
            "getMovieQuality",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getShootMode",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getSteadyMode",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getStorageInformation",
            [],
            [
                "{\"storageID\":\"string\", \"recordTarget\":\"bool\", \"numberOfRecordableImages\":\"int\", \"recordableTime\":\"int\", \"storageDescription\":\"string\"}*"
            ],
            "1.0"
        ],
        [
            "getSupportedCameraFunction",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedMovieQuality",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedShootMode",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedSteadyMode",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getVersions",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "setCameraFunction",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setMovieQuality",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setShootMode",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setSteadyMode",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "startIntervalStillRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "startLiveview",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "startMovieRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopIntervalStillRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopLiveview",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopMovieRec",
            [],
            [
                "string"
            ],
            "1.0"
        ]
    ]
}

So, there's a startLiveview command down the bottom there... maybe I need to call this prior to trying to actually display the URL in a browser? It easily be called via Postman with the body:

{"method": "startLiveview","params" : [],"id" : 1,"version" : "1.0"}

You'll get the following result...

{
    "id": 1,
    "result": [
        "http://192.168.122.1:60152/liveview.JPG?%211234%21http%2dget%3a%2a%3aimage%2fjpeg%3a%2a%21%21%21%21%21"
    ]
}

We knew this link already from the XML above, but does the video now work? I slapped the URL into the browser but didn't have much luck. So... a little more digging and I found the following: Tony Jan's work on decode the Sony stream written in Ruby, kazyx's kz-remote-api written in C# and twenzel's version called SonyCameraRemoteControl also in C#.Finally, with the term 'liveview' added to my search, I hit cryptofuture's liveView project which contained binaries to run really easy on Windows.

livecam-py camera-shot-1 camera-shot-3

Horrible image quality... but we have it! A livestream from the HDR-AS100V! I wrapped it up at that point, since you can see it was past midnight.

Getting this out to the world!

Nice, we have a connection. Next step? Pipe this to Youtube. I was going to use a Raspberry Pi, so Ruby will an easy choice and therefore Tony's library was tested. Gem is installed by default...

pi@raspberrypi:~ $ sudo gem install sonycam
Fetching: thor-1.0.1.gem (100%)
Successfully installed thor-1.0.1
Fetching: sonycam-1.3.2.gem (100%)
Successfully installed sonycam-1.3.2
Parsing documentation for thor-1.0.1
Installing ri documentation for thor-1.0.1
Parsing documentation for sonycam-1.3.2
Installing ri documentation for sonycam-1.3.2
Done installing documentation for thor, sonycam after 3 seconds
2 gems installed

Just for fun, after hooking up the wireless... try to take a picture?

pi@raspberrypi:~ $ sonycam api actTakePicture
Can not find /home/pi/.sonycam, start scanning...
Found location: http://192.168.122.1:64321/DmsRmtDesc.xml
Device description file saved to /home/pi/.sonycam
Traceback (most recent call last):
        8: from /usr/local/bin/sonycam:23:in `<main>'
        7: from /usr/local/bin/sonycam:23:in `load'
        6: from /var/lib/gems/2.5.0/gems/sonycam-1.3.2/bin/sonycam:3:in `<top (required)>'
        5: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
        4: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        3: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        2: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        1: from /var/lib/gems/2.5.0/gems/sonycam-1.3.2/lib/sonycam/cli.rb:49:in `api'
/var/lib/gems/2.5.0/gems/sonycam-1.3.2/lib/sonycam/api.rb:17:in `request': Sonycam::Error::IllegalRequest

I suppose, this is an Action Cam and not an SLR? Let's try the livestream to MP4.

sonycam liveview | ffmpeg -f image2pipe -c mjpeg -i pipe:0 -codec copy liveview.mp4

Trying to open the video file whilst the recording was happening didn't work... but it opened successfully once it was stopped.

The final option was to use ffserver, but it turns out that it's ancient technology that's been deprecated in FFmpeg. I quickly looked at the age of the repository and noticed it was last updated 6 years ago!

In the end, it turns out that the liveview is only ever outputted at 635x300-something. It's not HD! Nooooo... alllll... thissss... timmmeeee... speennntt.... It seems that there's another call for startLiveviewWithSize, but my camera doesn't support this. Oh right, only the full-frame cameras do!? And look at that link, they've called Liveview "viewfinder"! Of course it wont be full resolution.

HDMI Output

What goes out, must come in? Sure, this thing has a mini/micro/something HDMI port in its ass... but how does that help me? YouTube doesn't have a HDMI input plug! Well, eBay does have crappy HDMI to USB converters.. so..

s-l500

I bought that thing... and got this input via VLC. The actual software they wanted to use needed Direct X Movie/Video WMV software installed and Windows 10 didn't want to even care.

HDMIin

Image quality was OK, but no colour? I didn't bother digging further as I really didn't want HDMI cables running all over the shop.

Let's just use the old camera

Oh well.. it's back to the old camera until I find another method of getting this camera to stream. Here it is, right now, streaming live to the world from my balcony.

DSC00002

Hahahaa... do you like my handywork? (Go over here to Youtube if the embed below is broken.)

I'll update if I manage to find a better method. There's hints of people hacking apart the APK (yes, these cameras run Android!) on the unit that streams to UStream.
Why can't we just create one that provides an RTSP endpoint, like my camera above?