Subscribe via RSS
8Apr/200

Restoring an Azure DB Bacpac into SQL Express

Every so-often, one receives a request out of the blue to revive things that one has totally forgotten about. In most cases, it's comments on pages on this hideous blog asking questions about things that I've gotten rid of, in one form or another, and can't answer! Fortuantely, this time I could: it was a request to revive a project which had been on ice for a long time, but had been well-backed-up... or so I thought.

Firstly, it took me two days to find the files... not cool. If you shut down a project, get EVERYTHING and put it in ONE LOCATION. The system comprised of a zip of the IIS inetpub folder, a zip of the data drive and finally, a database backup from the SQL-as-a-service database. This, thanks to Azure not providing direct access to the server, takes the form of a bacpac file.

Back in the day, when developing, the preprod server was located on a HP something-something with 44gb of RAM which hosted multiple VMs. It made a fantastic noise when it first booted, making sure the fans could hit the required RPMs, and so it was left on most of the time. It was quite the powerhouse and did everything we needed. To save cost, the preprod server only ran SQL Express, so whenever we needed to back-port the production database to diagnose and squash bugs, we needed to manipulate the backups to make them fit.

What do I mean by this? Well SQL Express (or, at least the old 2014 version running in the VM) isn't as capable as Azure DBs and therefore you needed modify the backups so that they wouldn't throw errors when they were being restored. Below is a perfect example (this happened tonight) of such errors...

database-1 database-2 database-3

All looking OK, taking large amounts of time, and then...

database-4

Bang! With is not a valid operator... what's it doing embedded in a SQL script? Let's find out! Did you know that a bacpac file is just a pkzip archive? Install 7-zip, right click on the bacpac file and choose Open Archive.

database-5

Nice, contents. You'll find the database tables stored as BCP files in the Data folder and actual entire structure and scripts in the model.xml file. If you have your favourite editor installed correctly, then you should just be able to double-click model.xml and edit the file inline.

We know that the error is about shared access keys, so I CTRL-F'd the relevant keyword and the result was at the very bottom of the file. It's trying to store the logging and auditing on Azure shared storage... which is obviously not a good idea if we're running a private server. It's also totally incompatible with SQL Express, so lets rip it out.

<DataSchemaModel>
	<Model>
		...
		<Element Type="SqlDatabaseCredential" Name="[https://azure.net/sqldbauditlogs]">
			<property Name="Identity" Value="SHARED ACCESS SIGNATURE">
		</Element>
		<Element Type="SqlDatabaseCredential" Name="[https://azure.net/sqldbtdlogs]">
			<Property Name="Identity" Value="SHARED ACCESS SIGNATURE" />
		</Element>
		<Element Type="SqlFullTextCatalog" Name="[SnomedCatalog]">
			<Property Name="IsAccentSensitive" Value="True" />
			<Relationship Name="Authorizer">
				<Entry>
					<References ExternalSource="BuiltIns" Name="[dbo]" />
				</Entry>
			</Relationship>
		</Element>
	</Model>
</DataSchemaModel>

I also ripped out the Full-Text Searching block above, as shown, and another block much further above that was also referencing this block. Once I'd made all the changes to the zip file, I closed my editor and went back to 7-zip.

database-6

Yes, 7-zip, I very muchly would love to commit the changes back into this 1gb archive, thank you. From here, I attempted to deploy the bacpac to the server...

Ok, right, by editing a file inside this archive, we've trashed the thumbprint. A checksum is a magical calculation of a chunk of binary data that, somehow, resolves it down to a unique number that indicates the bits. If a bit changes, the checksum number changes and this allows software to know if things have been tampered with. That's what has occurred here... we've actually tampered with it and it's changed the calculation.

The SQL Server has run a calculation on the file and the result has mismatched with something, but with what? Turns out that in the original.xml file is a checksum value! We need to change this value to the new checksum, but how do we calculate that? Don't even bother thinking, just copy the following script into a file called checksum.ps1 and store it in your home directory.

$modelXmlPath = Read-Host "model.xml file path"
$hasher = [System.Security.Cryptography.HashAlgorithm]::Create("System.Security.Cryptography.SHA256CryptoServiceProvider")
$fileStream = new-object System.IO.FileStream ` -ArgumentList @($modelXmlPath, [System.IO.FileMode]::Open)
$hash = $hasher.ComputeHash($fileStream)
$hashString = ""
Foreach ($b in $hash) { $hashString += $b.ToString("X2") }
$fileStream.Close()
$hashString

If you've never run scripts in Powershell before, then you'll need to run the following command first if you get an error like 'checksum.ps1 cannot be loaded because running scripts is disabled on this system.'

set-executionpolicy remotesigned

Once sorted, run the following:

PS D:\> checksum.ps1
model.xml file path: d:\model.xml
898D9323A53F0C52B782867B9E8343D8371059A1192849F793CEA168FEAC7AA0

That final number needs to be pasted into original.xml in the archive. Save it back into the archive and deploy once more. Note that, before you try to deploy again, you'll need to detach the half-deployed database and delete the mdf/ldf files that were half-stored in the deployment folders that you set when you tried previously.

I then hit another error, a long way into deployment where the server said that AT was a shit command. Turns out one of my previous developers had tried to cast a datetime into a timezone using the query getdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Aus Eastern Standard Time'. Neat! But SQL Express hates that... so... open the archive, edit model.xml, save it, copy it out of the archive, checksum it, edit original.xml and deploy again...

database-7

Fiinnnnaaalllyyyy....!!! IT'S ALIVE! ... Hah... it's not...

database-8

dbo.Sort Failed? What dat? Google! There's always someone else that has had your error before... tempdb full? Let's add a new 100gb disk and slap tempdb over there. You can do this by running the following in a new query editor against master...

USE master;
GO

ALTER DATABASE tempdb 
MODIFY FILE (NAME = tempdev, FILENAME = 'T:\MSSQL\DATA\tempdb.mdf');
GO

ALTER DATABASE tempdb 
MODIFY FILE (NAME = templog, FILENAME = 'T:\MSSQL\DATA\templog.ldf');
GO

And you'll need to restart SQL before it applies...

The file "tempdev" has been modified in the system catalog. The new path will be used the next time the database is started.
The file "templog" has been modified in the system catalog. The new path will be used the next time the database is started.

Completion time: 2020-04-08T22:55:04.1929547+10:00

Of course, you'll need to change T drive to something else! Enjoy.

Filed under: Retro No Comments
7Apr/200

Mixing And Matching KVM Cables

Thanks to these new working-from-home shennanigans, I had to dig through the to-sell-at-trash-and-treasure box and find a KVM Switch to hook up all my machines. The desk used to only have one PC, but now additionally houses my work laptop and a large development server. I've got two monitors, and I want one of those to switch over to the development PC when required. I also want my keyboard and mouse to switch between all devices. To do this, I'll use the VGA/USB KVM to channel input and video to the monitor.

DSC00016

The best KVM in the box'o'stuff was a Belkin SOHO 4-port VGA+USB+Audio switch. Fortunately it had two of the KVM cables, so I could switch in my Laptop and development machine... but... I had to then unplug the USB to switch the input to the main PC. This wasn't optimal, so I went digging in the box again for another cable. The Belkin cables switched both Audio in+out, VGA and USB. From the outside, it looked like the audio channels were external to the VGA cable, unlike the USB which does seem to run through the same wires.

Back into the box, but unfortunately there were no other Belkin-branded cables. What I did find was another KVM with USB switching: Level One 2-Port VGA+USB KVM Switch.

DSC00033

DSC00013

This one also switched audio, but via USB instead of standard audio jacks. I grabbed a cable from it and was about to just plug everything together, but then I halted. USB has four wires, with two of those being the 5v rails. If there's any wiring differences between the switches then I could well send voltages in haphazard directions and cook equipment!

USB over VGA cable pinout

Wanting to hook this all together, I reached for my multimeter and started mapping out both the Level One and Belkin KVM cables. Thankfully, they have used very similar wiring for their cables! Thanks to monitors now talking digitally to hosts when describing their capabilities, there are spare pins on the cables to use for other purposes. There's a lot of extra ground pins also, but Belkin still preferred to keep those split whilst Level One happily merged them altogether.

DSC00019

Pin VGA Level One Cable Belkin Cable
1 R
2 G
3 B
4 ID2 / RES USB D- (White) USB D+ (Green)
5 GND USB D+ (Green) USB D- (White)
6 RGND USB GND + GND USB GND + RGND
7 GGND USB GND + GND GND
8 BGND USB GND + GND GND
9 KEY / 5v USB 5v + 5v USB 5v + 5v
10 SGND USB GND + GND GND
11 ID0 / RES
12 ID1 / SDA
13 HSYNC
14 VSYNC
15 ID3 / SCL

From above, you can see that the voltage (thanks to VGA actually having a +5v pin as standard!) and GND are close-enough, but the USB data pins are backwards. You assholes.. here I was thinking there might be a standard. What to do here? Well, I simply hacked apart a USB extension cable and crossed over the D+ and D-. Note that there is absolutely no reason for using this cable in any other scenario!

DSC00034

After this, one last hurdle.. the Level One KVM has a male plug at either end, whereas the Belkin cables have female on the KVM side.

DSC00028 DSC00023 DSC00032

And the bloody thing just worked! Happy to not have to spend money on extra proprietary cables that I actually couldn't find anywhere.

3Apr/200

Sony HDR-AS100V Action Cam – Further Hacking

After a LOT of mucking around, trying to get a stream URL from the Action Cam, I did further googling on the defunct UStream service that the camera could, supposedly, stream live to. It wasn't long before I came across this mammoth thread on Github describing a few users' attempts to hack apart the APK that does the steaming and to fake a server with the intent to intercept the feed.

They use the Sony-PMCA-RE Project code to tweak settings on the camera and then built up an entire webserver to fake the Ustream service. None of this was beyond me, so I started by trying to work out if I could configure my camera to point at a server... (note that I've fudged any secrets in the console logs below...)

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe info -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Model:              HDR-AS100V
Product code:       0002426601
Serial number:      03035253
Firmware version:   2.00
GPS Data:           2020-04-02 00:00:00 - 2020-05-02 00:00:00
C:\Downloads>pmca-console-v0.17-win.exe stream -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

twitterEnabled:     0
twitterConsumerKey: o9fJ2342342423433qg
twitterConsumerSecret: TTL4324234234234324324llM6EQdivTXesA
twitterAccessToken1:
twitterAccessTokenSecret:
twitterMessage:     Live Streaming from Action Cam by Sony
facebookEnabled:    0
facebookAccessToken:
facebookMessage:    Live Streaming from Action Cam by Sony
service:            0
enabled:            1
macId:
macSecret:
macIssueTime:       0000000000000000
unknown:            1
channels:           [0]
shortURL:
videoFormat:        1
supportedFormats:   [1, 3]
enableRecordMode:   1
videoTitle:         Recorded with Action Cam by Sony
videoDescription:   Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself
videoTag:
C:\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Traceback (most recent call last):
  File "C:\projects\sony-pmca-re\pmca-console.py", line 96, in <module>
  File "C:\projects\sony-pmca-re\pmca-console.py", line 82, in main
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 360, in updaterShellCommand
  File "C:\projects\sony-pmca-re\pmca\commands\usb.py", line 373, in firmwareUpdateCommandInternal
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 527, in checkGuard
  File "C:\projects\sony-pmca-re\pmca\usb\sony.py", line 501, in _sendWriteCommands
Exception: Firmware update error: Low battery
[9052] Failed to execute script pmca-console

Right... low battery... I charged it up and then tried again:

C:\Users\Steven\Downloads>pmca-console-v0.17-win.exe updatershell -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Getting device info
Using firmware for model HDR-AS100V

Initializing firmware update
Switching to updater mode

Waiting for camera to switch...
Please follow the instructions on the camera screen.

DSC04100

Cool, at this point the camera showed 'EXEC' on its LCD display. This kept flashing... and then the console app error'd out:

Operation timed out. Please run this command again when your camera has connected.

I pressed the record button on the back of the camera to try and force the 'EXEC' and it seemed to reboot. From here, it then wouldn't even come back to life. The battery charge light just stayed a fixed red, which I now think was an error indicator. After popping the battery out, I could get it back to the 'EXEC' screen via the console appm but it occurred to me that I actually had no idea what updater-shell even did. Instead I tried the stream fuction to push a UStream configuration file to the device as per the instructions here at Novex's project where he attempts to create his own UStream APK. Here's the config I used...

[
    [
        "twitterEnabled",
        0
    ],
    [
        "twitterConsumerKey",
        ""
    ],
    [
        "twitterConsumerSecret",
        ""
    ],
    [
        "twitterAccessToken1",
        ""
    ],
    [
        "twitterAccessTokenSecret",
        ""
    ],
    [
        "twitterMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "facebookEnabled",
        0
    ],
    [
        "facebookAccessToken",
        ""
    ],
    [
        "facebookMessage",
        "Live Streaming from Action Cam by Sony"
    ],
    [
        "service",
        0
    ],
    [
        "enabled",
        1
    ],
    [
        "macId",
        "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
    ],
    [
        "macSecret",
        "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
    ],
    [
        "macIssueTime",
        "dbeaaaaa00000000"
    ],
    [
        "unknown",
        1
    ],
    [
        "channels",
        [
            12345678
        ]
    ],
    [
        "shortURL",
        "http://www.ustream.tv/channel/asdf"
    ],
    [
        "videoFormat",
        3
    ],
    [
        "supportedFormats",
        [
            1,
            3
        ]
    ],
    [
        "enableRecordMode",
        0
    ],
    [
        "videoTitle",
        "Recorded with Action Cam by Sony"
    ],
    [
        "videoDescription",
        "Shot 100% with Sony's Action Cam #SonyActionCam #ProveYourself"
    ],
    [
        "videoTag",
        ""
    ]
]

And then I smushed it into the camera using the console application...

C:\Downloads>pmca-console-v0.17-win.exe stream -w ustream.json -d libusb
Using drivers libusb-MSC, libusb-MTP
Looking for Sony devices

Querying mass storage device
Sony Camcorder is a camera in mass storage mode

Urrghh... what mode does it need to be in? Wait, is that even an error? Let's try use the buttons on the camera to switch it into Live mode...

DSC04105

Preparing?

DSC04103

No AP? Maybe it has no idea how to connect to a 5Ghz access point? Let's set it to my 2.4G secondary network, which nothing else uses, so I can isolate and watch any traffic. This took a little bit of hacking as the Sony Network Settings Utility needs the driver to be reverted back to USB Mass Storage. Jump in to Device Manager and force a driver 'update' back to the standard driver.

reset-to-usb-mass-storage

Now you can go into Network Settings and set your AP once more.

net-set-tool-1

wifi-connection incomplete-settings incomplete-settings-2

Note that it'll complain at the end that the settings are incomplete. Fortunately it still writes the AP configuration to the camera. After this, you'll have to switch back to the lib-usb driver and then re-upload the streaming settings.

remap-usb-driver-1 remap-usb-driver-2 remap-usb-driver-3

From here? Test! Unplug the camera and hit the Live mode again...

DSC04111

Auth Error? HDR-AS100V Official documentation indicates that this means it's failed to authenticate with UStream... makes sense to me! Can we hack my local network to make it auth to a fake server?

Network Setup for a Fake UStream Server

Since I want the raspi to be doing all the heavy lifting, I'm going to follow the instructions here at Luca Lanziani's project to set up a listener and a dns faker.

pi@raspberrypi:~ $ wget https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
--2020-04-03 16:01:51--  https://github.com/LucaLanziani/sony-camera-stream-receiver/archive/master.zip
Resolving github.com (github.com)... 13.237.44.5
Connecting to github.com (github.com)|13.237.44.5|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master [following]
--2020-04-03 16:01:51--  https://codeload.github.com/LucaLanziani/sony-camera-stream-receiver/zip/master
Resolving codeload.github.com (codeload.github.com)... 52.63.100.255
Connecting to codeload.github.com (codeload.github.com)|52.63.100.255|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3790 (3.7K) [application/zip]
Saving to: ‘master.zip’

master.zip                                  100%[===========================>]   3.70K  --.-KB/s    in 0s      

2020-04-03 16:01:51 (7.39 MB/s) - ‘master.zip’ saved [3790/3790]

It downloaded easy enough... let's see what's inside!

pi@raspberrypi:~ $ mv master.zip sony-camera-receiver.zip
pi@raspberrypi:~ $ unzip sony-camera-receiver.zip 
Archive:  sony-camera-receiver.zip
dd4b1170f9cdeaabc2cb68eccff6b7f63214abce
   creating: sony-camera-stream-receiver-master/
 extracting: sony-camera-stream-receiver-master/.gitignore  
  inflating: sony-camera-stream-receiver-master/README.md  
  inflating: sony-camera-stream-receiver-master/dns_server.js  
  inflating: sony-camera-stream-receiver-master/index.js  
  inflating: sony-camera-stream-receiver-master/package.json  
  inflating: sony-camera-stream-receiver-master/rtmp_server.js  
  inflating: sony-camera-stream-receiver-master/web_server.js  
pi@raspberrypi:~ $ cd sony-camera-stream-receiver-master/
pi@raspberrypi:~/sony-camera-stream-receiver-master $ ls
dns_server.js  index.js  package.json  README.md  rtmp_server.js  web_server.js

Cool, everything we need. Following the instructions, we first need to download all the supporting node libraries.

pi@raspberrypi:~/sony-camera-stream-receiver-master $ npm i
npm WARN npm npm does not support Node.js v10.15.2
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN sony-cameras-stream-receiver@1.0.0 No repository field.

added 76 packages from 64 contributors in 17.452s

Cool, done. Now it's time to see if it'll actually run?

pi@raspberrypi:~/sony-camera-stream-receiver-master $ sudo node ./index.js --web.channel_number=12345678 --dns.external_ip=192.168.1.50
03/04/2020 16:09:24 16471 [INFO] Node Media Server v1.4.7
  dnsproxy:info we are up and listening at 192.168.1.50 on 53 +0ms
03/04/2020 16:09:24 16471 [INFO] Node Media Rtmp Server started on port: 1935
Server listening at http://0.0.0.0:80/
03/04/2020 16:09:26 16471 [INFO] There is a new version 2.1.6 that can be updated

Okey dokey.. it's up and running. The best thing we can do now is slap the new DNS server in our config and see what happens. After configuring my router and letting it reboot, nothing was resolving via .50. It seemed that .1 was still the DNS Server? I tried renewing my DHCP and flushing DNS, but nothing worked. Intead, I just hacked my local ethernet interface to point to .50 for DNS... woah! It works!

dhcp-1 dhcp-2 dhcp-3

But I can't hack the camera in the same way... so I need to work out why DHCP isn't giving me .50. I then realised I'd been reconfiguring my repeater and not my actual router!? I logged into .1 and set the DHCP configuration to have a Primary DNS of .50 and, well, shit just worked.

The camera still threw the Auth Error, so I totally disabled the DHCP on the repeater, because that really shouldn't be there, and rebooted the camera. Hitting the REC button on Live mode on the Action Cam then saw action on the raspi!

03/04/2020 16:58:47 16471 [INFO] [rtmp connect] id=SRMCPZ41 ip=::ffff:192.168.1.134 app=12345678 args={"app":"12345678","flashVer":"LNX 9,0,124,0","swfUrl":"http://www.ustream.tv/mobile.swf","tcUrl":"rtmp://api.ustream.tv/12345678","capabilities":1,"audioCodecs":1143,"videoCodecs":252,"videoFunction":1,"pageUrl":"http://www.ustream.tv","objectEncoding":0,"fpad":false}
03/04/2020 16:58:47 16471 [INFO] [rtmp publish] New stream. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 streamId=1
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle video. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 frame_type=1 codec_id=7 codec_name=H264 0x0
03/04/2020 16:58:48 16471 [INFO] [rtmp publish] Handle audio. id=SRMCPZ41 streamPath=/12345678/broadcaster/live7665 sound_format=10 sound_type=2 sound_size=1 sound_rate=3 codec_name=AAC 48000 2ch

03/04/2020 16:59:40 16471 [INFO] [rtmp connect] id=7YV3W0L2 ip=::ffff:192.168.1.121 app=12345678/broadcaster args={"app":"12345678/broadcaster","flashVer":"LNX 9,0,124,2","tcUrl":"rtmp://192.168.1.50:1935/12345678/broadcaster","fpad":false,"capabilities":15,"audioCodecs":4071,"videoCodecs":252,"videoFunction":1}
03/04/2020 16:59:40 16471 [INFO] [rtmp play] Join stream. id=7YV3W0L2 streamPath=/12345678/broadcaster/live7665  streamId=1

And then pointing VLC at rtmp://192.168.1.50/12345678/broadcaster/live7665 got me....

traiiiinnnnssss

Traaaaaiinnnsssss... Now to find the perfect angle. Note that you get an 'ONAIR' message on the LCD when it's ... on air.

DSC00001

Sure, not the best mounting... I'll work on that. I've also got a waterproof case for it that I'll need to hack a hole into to get the USB power through.

Productionising

Having the raspi be the DNS server is a little scary. But I suppose this is where secondary DNS servers come in? If we put 8.8.8.8 for the secondary, does it get used when the primary is down? Turns out it doesn't work that way, so I'll need to keep the raspi up and running 100% of the time!

C:\Users\Steven>ipconfig /all
Ethernet adapter Ethernet:
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) Ethernet Connection (2) I219-V
   Physical Address. . . . . . . . . : 30-5A-3A-81-E1-1B
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 192.168.1.121(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
   DHCP Server . . . . . . . . . . . : 192.168.1.1
   DNS Servers . . . . . . . . . . . : 192.168.1.50
                                       8.8.8.8
   NetBIOS over Tcpip. . . . . . . . : Enabled
C:\Users\Steven>ipconfig /flushdns
Windows IP Configuration
Successfully flushed the DNS Resolver Cache.
C:\Users\Steven>nslookup google.com
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.1.50

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

For some reason I really enjoy the spelling of 'UnKnown'. Regardless, the secondary has not kicked in! With this news, I suppose it'd be best to learn how to keep that service up and running 24/7 on the pi. Instead, I have decided to not even go ahead with this camera! It's wide-view fish-eye angle just isn't wide enough to get the trains close enough and the cityscape in the shot... I therefore don't see any advantage over my current camera.

So! A learning experience! For those who want to use a Sony Action Cam HDR-AS100V as a livestreaming camera, it's plausible!

3Apr/200

Streaming from a Sony HDR-AS100V Action Cam

This is a cool camera. I picked it up from the Hard Off in Kagoshima whilst I was visiting the southernmost JR Railway station in Japan. I then used it to take a miriad of videos, especially from the balcony at the apartment in Shin Osaka.

DSC04151

DSC04152 DSC04153 DSC04156

Whilst it works great recording by itself, you can also use the Play Mobile (now Imaging Edge Mobile) application from Sony to control it remotely. The application uses the phones Wifi to connect to the camera, which acts as a wifi access point. From there, it shows a live view of what the camera can see whilst you're trying to line up the next shot. You can even control the camera remotely with the buttons on screen, being able to start and stop recording very easily... especially without bumping the camera!

Screenshot 20200331-143248

I was hoping Sony would release the same software for the PC and allow me to connect wirelessly to the camera. This would then let me sniff the traffic and get the video stream live link, but alas, it's only available for mobile devices. Wait!, you say, there is a version of Play Memories for windows! Sure, go ahead and download it... install it even! It'll connect to your camera, but there's no ability to see the live stream from the lens... I tried... trust me....

camera-options actioncam-import streaming-tool

With the USB camera connected, you'll get the screens above. They let you update firmware and manage settings, but not much else. Of course, there's also the file transfer functionality to let you download the images and videos you've taken. Interestingly, the last image shows the second pane of the Network Settings tab which allows you to configure the camera to upload to UStream automatically. Turns out that this was a 'thing' back in the day and not really practical anymore: IBM bought UStream and the service is no longer free... nor could I get it to work, even with a trial account.

Can we create a network to inspect the Wifi?

So, I had a plan. I have a Vonets VAR11N-300 that allows an ethernet cabled device to access an access point over Wifi. With this, I could use an ethernet port on my laptop to connect to the camera. The Wifi adapter in the laptop could then create an ad-hoc network that my phone could connect to. If I bridged the ethernet to the Wifi, connected my phone ad-hoc to the wifi network created by the laptop, then I could possibly pass the traffic through the laptop and use wireshark to listen in?

s-l1600

As I was plugging this all together, traffic started flowing without client application requesting anything! I'd not yet bridged the adapters, but I'd installed Wireshark and had the VAR11N-300 connected to the camera and, all of a sudden, I'd noticed jitters on the traffic graph for the ethernet interface on the front page of Wireshark. The ethernet port seems to have received an IP from the camera since it was acting as a terminating host and not an ethernet bridge! For everyones information: the IP range is 196.168.0.x, with IPs dished out from 1 and the camera being 192.168.122.1. With the little graph jumping around, I decided to listen in and have a look!

ws-1 ws-2 upnp-xml

Ok so, from left to right above... the Wireshark main screen with my miriad of interfaces. You may notice that I have Wifi-2 selected there and there's traffic flowing on it... At this point, realising that I didn't need the mobile application to trigger data flow from the camera, I'd slapped a USB Wifi dongle into my main desktop so that I could work in comfort. The second screen is the main window of Wireshark once you're sniffing. You'll see a huge list of traffic up the top. Wireshark, without filters, will just present everything it can possibly see. Each row is a packet sent over the Wifi and you may need to combine multiple rows to see an entire picture, depending on what you're looking for. Usually, if this was my main network interface, I'd use a filter at the top with either ip.dest == 192.x.x.x or ip.src == 192.x.x.x to limit the displayed data, but fortunately this interface was only talking to the camera.

The second shot there has a very interesting packet displayed. It turns out that the camera is broadcasting UPnP information, telling anyone listening on the immediate network how to find it! No wonder, once you've connected your phone to the camera, that it can find it. It doesn't even have to look as the camera just yells it out every few seconds. Let's check out the data we're receiving...

<?xml version="1.0" encoding="utf-8"?>
<root xmlns="urn:schemas-upnp-org:device-1-0" xmlns:dlna="urn:schemas-dlna-org:device-1-0" xmlns:av="urn:schemas-sony-com:av">
	<specVersion>
		<major>1</major>
		<minor>0</minor>
	</specVersion>
	<device>
		<dlna:X_DLNADOC xmlns:dlna="urn:schemas-dlna-org:device-1-0">DMS-1.50</dlna:X_DLNADOC>
		<deviceType>urn:schemas-upnp-org:device:MediaServer:1</deviceType>
		<friendlyName>HDR-AS100V</friendlyName>
		<manufacturer>Sony Corporation</manufacturer>
		<manufacturerURL>http://www.sony.com/</manufacturerURL>
		<modelDescription>SonyDigitalMediaServer</modelDescription>
		<modelName>SonyImagingDevice</modelName>
		<modelURL>http://www.sony.net/</modelURL>
		<UDN>uuid:00000000-0005-0010-8000-fcc2de658d5e</UDN>
		<serviceList>
			<service>
				<serviceType>urn:schemas-upnp-org:service:ContentDirectory:1</serviceType>
				<serviceId>urn:upnp-org:serviceId:ContentDirectory</serviceId>
				<SCPDURL>/CdsDesc.xml</SCPDURL>
				<controlURL>/upnp/control/ContentDirectory</controlURL>
				<eventSubURL>/upnp/event/ContentDirectory</eventSubURL>
			</service>
			<service>
				<serviceType>urn:schemas-upnp-org:service:ConnectionManager:1</serviceType>
				<serviceId>urn:upnp-org:serviceId:ConnectionManager</serviceId>
				<SCPDURL>/CmsDesc.xml</SCPDURL>
				<controlURL>/upnp/control/ConnectionManager</controlURL>
				<eventSubURL>/upnp/event/ConnectionManager</eventSubURL>
			</service>
			<service>
				<serviceType>urn:schemas-sony-com:service:ScalarWebAPI:1</serviceType>
				<serviceId>urn:schemas-sony-com:serviceId:ScalarWebAPI</serviceId>
				<SCPDURL>/ScalarWebApiDesc.xml</SCPDURL>
				<controlURL/>
				<eventSubURL/>
			</service>
		</serviceList>
		<iconList>
			<icon>
				<mimetype>image/jpeg</mimetype>
				<width>48</width>
				<height>48</height>
				<depth>24</depth>
				<url>/DLNA_camera_48.jpg</url>
			</icon>
			<icon>
				<mimetype>image/jpeg</mimetype>
				<width>120</width>
				<height>120</height>
				<depth>24</depth>
				<url>/DLNA_camera_120.jpg</url>
			</icon>
			<icon>
				<mimetype>image/png</mimetype>
				<width>48</width>
				<height>48</height>
				<depth>24</depth>
				<url>/DLNA_camera_48.png</url>
			</icon>
			<icon>
				<mimetype>image/png</mimetype>
				<width>120</width>
				<height>120</height>
				<depth>24</depth>
				<url>/DLNA_camera_120.png</url>
			</icon>
		</iconList>
		<av:X_ScalarWebAPI_DeviceInfo xmlns:av="urn:schemas-sony-com:av">
			<av:X_ScalarWebAPI_Version>1.0</av:X_ScalarWebAPI_Version>
			<av:X_ScalarWebAPI_ServiceList>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>guide</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>accessControl</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
				<av:X_ScalarWebAPI_Service>
					<av:X_ScalarWebAPI_ServiceType>camera</av:X_ScalarWebAPI_ServiceType>
					<av:X_ScalarWebAPI_ActionList_URL>http://192.168.122.1:10000/sony</av:X_ScalarWebAPI_ActionList_URL>
					<av:X_ScalarWebAPI_AccessType />
				</av:X_ScalarWebAPI_Service>
			</av:X_ScalarWebAPI_ServiceList>
			<av:X_ScalarWebAPI_ImagingDevice>
				<av:X_ScalarWebAPI_LiveView_URL>http://192.168.122.1:60152/liveview.JPG?%211234%21http%2dget%3a%2a%3aimage%2fjpeg%3a%2a%21%21%21%21%21</av:X_ScalarWebAPI_LiveView_URL>
				<av:X_ScalarWebAPI_DefaultFunction>RemoteShooting</av:X_ScalarWebAPI_DefaultFunction>
			</av:X_ScalarWebAPI_ImagingDevice>
		</av:X_ScalarWebAPI_DeviceInfo>
	</device>
</root>

Oooowwweee... Check that link at the bottom!? X_ScalarWebAPI_LiveView_URL? Really? Can it be this easy? I slapped it into a browser and no... it's not this easy... the browser just doesn't respond as it's constantly being fed a stream of data and doesn't know what to do with it! Turns out this stream is a motion-jpeg or some frame-by-frame binary flow that isn't a standard video stream. So, what to do? Google the hell out of those keywords and see who has already come to our rescue!

First hit was a post on stackoverflow asking 'How to get Sony ScalarWebAPI method list'. From the responses, one person suggested to send the camera JSON via a post so that you can see what functions you can call. One such function is getMethodTypes and can be called via the following post body:

{"method": "getMethodTypes", "params": [""], "id": 1, "version": "1.0"}

posty

Using Postman, I got the following result...

{
    "id": 1,
    "results": [
        [
            "actTakePicture",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getApplicationInfo",
            [],
            [
                "string",
                "string"
            ],
            "1.0"
        ],
        [
            "getAvailableApiList",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableCameraFunction",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableMovieQuality",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableShootMode",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getAvailableSteadyMode",
            [],
            [
                "string",
                "string*"
            ],
            "1.0"
        ],
        [
            "getCameraFunction",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getEvent",
            [
                "bool"
            ],
            [
                "{\"type\":\"string\", \"names\":\"string*\"}",
                "{\"type\":\"string\", \"cameraStatus\":\"string\"}",
                "{\"type\":\"string\", \"zoomPosition\":\"int\", \"zoomNumberBox\":\"int\", \"zoomIndexCurrentBox\":\"int\", \"zoomPositionCurrentBox\":\"int\"}",
                "{\"type\":\"string\", \"liveviewStatus\":\"bool\"}",
                "{\"type\":\"string\", \"liveviewOrientation\":\"string\"}",
                "{\"type\":\"string\", \"takePictureUrl\":\"string*\"}*",
                "{\"type\":\"string\", \"continuousError\":\"string\", \"isContinued\":\"bool\"}*",
                "{\"type\":\"string\", \"triggeredError\":\"string*\"}",
                "{\"type\":\"string\", \"sceneRecognition\":\"string\", \"steadyRecognition\":\"string\", \"motionRecognition\":\"string\"}",
                "{\"type\":\"string\", \"formatResult\":\"string\"}",
                "{\"type\":\"string\", \"storageID\":\"string\", \"recordTarget\":\"bool\", \"numberOfRecordableImages\":\"int\", \"recordableTime\":\"int\", \"storageDescription\":\"string\"}*",
                "{\"type\":\"string\", \"currentBeepMode\":\"string\", \"beepModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentCameraFunction\":\"string\", \"cameraFunctionCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentMovieQuality\":\"string\", \"movieQualityCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentAspect\":\"string\", \"currentSize\":\"string\"}",
                "{\"type\":\"string\", \"cameraFunctionResult\":\"string\"}",
                "{\"type\":\"string\", \"currentSteadyMode\":\"string\", \"steadyModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentViewAngle\":\"int\", \"viewAngleCandidates\":\"int*\"}",
                "{\"type\":\"string\", \"currentExposureMode\":\"string\", \"exposureModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentPostviewImageSize\":\"string\", \"postviewImageSizeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentSelfTimer\":\"int\", \"selfTimerCandidates\":\"int*\"}",
                "{\"type\":\"string\", \"currentShootMode\":\"string\", \"shootModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentAELock\":\"bool\", \"aeLockCandidates\":\"bool*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentBracketShootMode\":\"string\", \"currentBracketShootModeOption\":\"string\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentCreativeStyle\":\"string\", \"currentCreativeStyleContrast\":\"int\", \"currentCreativeStyleSaturation\":\"int\", \"currentCreativeStyleSharpness\":\"int\"}",
                "{\"type\":\"string\", \"currentExposureCompensation\":\"int\", \"maxExposureCompensation\":\"int\", \"minExposureCompensation\":\"int\", \"stepIndexOfExposureCompensation\":\"int\"}",
                "{\"type\":\"string\", \"currentFlashMode\":\"string\", \"flashModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentFNumber\":\"string\", \"fNumberCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentFocusMode\":\"string\", \"focusModeCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"currentIsoSpeedRate\":\"string\", \"isoSpeedRateCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentPictureEffect\":\"string\", \"currentPictureEffectOption\":\"string\"}",
                "{\"type\":\"string\", \"isShifted\":\"bool\"}",
                "{\"type\":\"string\", \"currentShutterSpeed\":\"string\", \"shutterSpeedCandidates\":\"string*\"}",
                "{\"type\":\"string\", \"checkAvailability\":\"bool\", \"currentWhiteBalanceMode\":\"string\", \"currentColorTemperature\":\"int\"}",
                "{\"type\":\"string\", \"currentSet\":\"bool\", \"currentTouchCoordinates\":\"double*\"}"
            ],
            "1.0"
        ],
        [
            "getMethodTypes",
            [
                "string"
            ],
            [
                "string",
                "string*",
                "string*",
                "string"
            ],
            "1.0"
        ],
        [
            "getMovieQuality",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getShootMode",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getSteadyMode",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "getStorageInformation",
            [],
            [
                "{\"storageID\":\"string\", \"recordTarget\":\"bool\", \"numberOfRecordableImages\":\"int\", \"recordableTime\":\"int\", \"storageDescription\":\"string\"}*"
            ],
            "1.0"
        ],
        [
            "getSupportedCameraFunction",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedMovieQuality",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedShootMode",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getSupportedSteadyMode",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "getVersions",
            [],
            [
                "string*"
            ],
            "1.0"
        ],
        [
            "setCameraFunction",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setMovieQuality",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setShootMode",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "setSteadyMode",
            [
                "string"
            ],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "startIntervalStillRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "startLiveview",
            [],
            [
                "string"
            ],
            "1.0"
        ],
        [
            "startMovieRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopIntervalStillRec",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopLiveview",
            [],
            [
                "int"
            ],
            "1.0"
        ],
        [
            "stopMovieRec",
            [],
            [
                "string"
            ],
            "1.0"
        ]
    ]
}

So, there's a startLiveview command down the bottom there... maybe I need to call this prior to trying to actually display the URL in a browser? It easily be called via Postman with the body:

{"method": "startLiveview","params" : [],"id" : 1,"version" : "1.0"}

You'll get the following result...

{
    "id": 1,
    "result": [
        "http://192.168.122.1:60152/liveview.JPG?%211234%21http%2dget%3a%2a%3aimage%2fjpeg%3a%2a%21%21%21%21%21"
    ]
}

We knew this link already from the XML above, but does the video now work? I slapped the URL into the browser but didn't have much luck. So... a little more digging and I found the following: Tony Jan's work on decode the Sony stream written in Ruby, kazyx's kz-remote-api written in C# and twenzel's version called SonyCameraRemoteControl also in C#.Finally, with the term 'liveview' added to my search, I hit cryptofuture's liveView project which contained binaries to run really easy on Windows.

livecam-py camera-shot-1 camera-shot-3

Horrible image quality... but we have it! A livestream from the HDR-AS100V! I wrapped it up at that point, since you can see it was past midnight.

Getting this out to the world!

Nice, we have a connection. Next step? Pipe this to Youtube. I was going to use a Raspberry Pi, so Ruby will an easy choice and therefore Tony's library was tested. Gem is installed by default...

pi@raspberrypi:~ $ sudo gem install sonycam
Fetching: thor-1.0.1.gem (100%)
Successfully installed thor-1.0.1
Fetching: sonycam-1.3.2.gem (100%)
Successfully installed sonycam-1.3.2
Parsing documentation for thor-1.0.1
Installing ri documentation for thor-1.0.1
Parsing documentation for sonycam-1.3.2
Installing ri documentation for sonycam-1.3.2
Done installing documentation for thor, sonycam after 3 seconds
2 gems installed

Just for fun, after hooking up the wireless... try to take a picture?

pi@raspberrypi:~ $ sonycam api actTakePicture
Can not find /home/pi/.sonycam, start scanning...
Found location: http://192.168.122.1:64321/DmsRmtDesc.xml
Device description file saved to /home/pi/.sonycam
Traceback (most recent call last):
        8: from /usr/local/bin/sonycam:23:in `<main>'
        7: from /usr/local/bin/sonycam:23:in `load'
        6: from /var/lib/gems/2.5.0/gems/sonycam-1.3.2/bin/sonycam:3:in `<top (required)>'
        5: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
        4: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        3: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        2: from /var/lib/gems/2.5.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        1: from /var/lib/gems/2.5.0/gems/sonycam-1.3.2/lib/sonycam/cli.rb:49:in `api'
/var/lib/gems/2.5.0/gems/sonycam-1.3.2/lib/sonycam/api.rb:17:in `request': Sonycam::Error::IllegalRequest

I suppose, this is an Action Cam and not an SLR? Let's try the livestream to MP4.

sonycam liveview | ffmpeg -f image2pipe -c mjpeg -i pipe:0 -codec copy liveview.mp4

Trying to open the video file whilst the recording was happening didn't work... but it opened successfully once it was stopped.

The final option was to use ffserver, but it turns out that it's ancient technology that's been deprecated in FFmpeg. I quickly looked at the age of the repository and noticed it was last updated 6 years ago!

In the end, it turns out that the liveview is only ever outputted at 635x300-something. It's not HD! Nooooo... alllll... thissss... timmmeeee... speennntt.... It seems that there's another call for startLiveviewWithSize, but my camera doesn't support this. Oh right, only the full-frame cameras do!? And look at that link, they've called Liveview "viewfinder"! Of course it wont be full resolution.

HDMI Output

What goes out, must come in? Sure, this thing has a mini/micro/something HDMI port in its ass... but how does that help me? YouTube doesn't have a HDMI input plug! Well, eBay does have crappy HDMI to USB converters.. so..

s-l500

I bought that thing... and got this input via VLC. The actual software they wanted to use needed Direct X Movie/Video WMV software installed and Windows 10 didn't want to even care.

HDMIin

Image quality was OK, but no colour? I didn't bother digging further as I really didn't want HDMI cables running all over the shop.

Let's just use the old camera

Oh well.. it's back to the old camera until I find another method of getting this camera to stream. Here it is, right now, streaming live to the world from my balcony.

DSC00002

Hahahaa... do you like my handywork?

I'll update if I manage to find a better method. There's hints of people hacking apart the APK (yes, these cameras run Android!) on the unit that streams to UStream.
Why can't we just create one that provides an RTSP endpoint, like my camera above?

27Mar/200

Amiga 500 – Internal IDE Port

Whilst looking for side-car Amiga A590 Hard drives for the Amiga 500, I found this deisgn at PCBWay of an internal IDE interface. The design seemed pretty straight-forward, but it did have its caveats. The main one was that this device does not Auto-Configure and therefore needs a boot floppy to use it. This wouldn't have been such a bad problem, if it wasn't for the fact that I wasn't able to write a bootable floppy disk. I therefore got my Goteks working in the A500 first.

In case you didn't know, PCBWay is a fully-fledged PCB design and fabrication site. They even provide a community side (similar to Thingyverse for 3D printing) where users can upload their designs and anyone can have them made! After finding this design on their site, I followed the checkout and a few of boards created. I wanted to save money, so I had two other designs created at the same time... but I'll talk about them in later posts.

DSC04068

With the PCBs now in my hand, it was time to fill them up and test the devices. Using the shopping list of parts provided at the original site, I purchased everything I needed from either Digi-Key or Jaycar.

This all arrived at around the same time and, thanks to being locked up at home due to a bug floating around the entire world, I now had time to get this thing built! First-thing's-first... take your bloody time doing this, otherwise ... or, as I did on the very first build ... you'll solder items in backwards. I quickly put that first attempt aside and started again.

DSC04071

Jaycar only had horizontal 50-pin IDC headers, so I cut it down to fit... not the prettiest!

DSC04072 DSC04080 DSC04081

I also purchased pin strips of both male and female thinking that the board had two individual areas for the male and female pins that the CPU plugs through.

DSC04083

Turns out that was totally wrong and you need to make sure you get the 'triple-length' female pin strips! Here's the item at Jaycar.

DSC04095

They're not cheap, but you want absolute stability when you're dealing with a piggy-backed CPU!

DSC04096 DSC04099 DSC04103

Note that if you look closely at that middle photo.. you'll see that I failed to solder two pins. Sure, the strength of the solder on the neighbouring pins will probably force contact... but... again... take your time!

The IDE header required a re-think back to my 386 upgrade. There's an issue here between providing a male or female pin header on the board. Officially, this is the 'host' of the IDE channel so, like any motherboard, it should provide male pins. You can then use a cable to provide a female IDC connector if your device has male pins. Also note that some units (like the transcend disk-on-a-stick in over here) are built to plug straight into motherboards and therefore will plug straight into this unit.

The only real thing you need to worry about is that if you're using a CF to IDE adapter, make sure it has a female header on it, i.e. on that's meant to go straight into a motherboard, or, if it has male pins, that you have a short IDE cable to go with it. Look over here for the logic on how to run the IDE cable... making sure that you get the pins in the correct order.

Mounting it

It was now finally built! Mounting it into the A500 I had open on the workbench took a little more effort than I expected. It turns out the legs of the piggy-back pin header are quite a bit thicker than the legs of the standard 68000 CPU. This meant that the whole unit required quite a shove to get it into the socket. It also made me wonder if the CPU would happily sit back in again afterwards if I ever had to revert this work. Firstly, anyway, I had to remove the cpu!

DSC04108 DSC04112 DSC04116

I'm pretty sure I mentioned something about taking your time with these things... After successfully removing the CPU from the motherboard, I also successfully bent two pins when mounting it onto my new PCB. Relax, breathe, grab a set of tweezers... realign the pins without snapping and try again!

DSC04120

It was in and, without a disk connected, the machine still powered up. Make sure you test this scenario as a milestone during your build!

DSC04175

Connecting the disk required a little trickery with the power cables. Thankfully there's a 5v header on the adapter and that's all this disk needs.

Testing it

In the zip file provided on the original site, you'll find bootable disk images that have the IDE device driver included. Or do they? It turns out Kickstart 1.3 also had the IDE driver inside... but 1.2 didn't? At first, I connected my Transcend disk-on-a-stick and booted the unit using the Boot_WB13.adf from the original site. I got the following:

DSC04124

Ok, c/assign can't be found. From what I've learnt, c is a folder on all Workbench installations with standard binary tools. Something like the DOS folder in MS-DOS or the /bin folder in Unix. It's trying to run the assign application which creates temporary symbolic links to paths on disks so that the shell knows what to run. The fact that it's failing isn't good, but let's try work out why.

Firstly, where is this line even being called from? It turns out it's the startup-sequence in the s folder on the boot: disk, which happens to be the disk image we downloaded and booted from. We can use the type command to work out what's in this file...

DSC04150

Ok, so it's tried to mount the HDD on DH0:. It's then changed to it, and succeeded? It's then tried to run the assign application from the c folder on the HDD. Uh, there's nothing on this HDD... actually, I have no idea what's on it! Quick way to find out...

DSC04177

Uh, yeah.. that'll never work! The delete command will come in handy here... I went ahead and whacked each of those useless files. Sorry DOS! Note that I could have also formatted the partition at this point... I had tried to do so during this adventure, but I couldn't work out how to format it as FAT (or FAT32) from the Amiga. Whenever I formatted it, it came up as an unknown DOS disk... so I slapped it into a Windows PC and formatted it there.

DSC04178 DSC04179 DSC04181

That last image above shows the Mountlist file located in the devs/ folder. This is super important as it tells mount how to understand what each partition is. It's similar to the /etc/fstab in Unix. Above you can see that it would try to use the fat95 library located in the L folder. Hence, when I formatted it and it became an Amiga partition, the fat95 driver would then throw an error indicating it wasn't a DOS drive. I would still love to know how to fix this Mountlist after a format... what parameters do I need to give the rest instead of just the driver being FastFileSystem?

Anyway, we're getting lost... leave it as FAT and then run copy boot: DH0: to copy the contents of the bootdisk to the HDD. Once done, reboot!

DSC04142

Hmm... still the c/assign error? Switching to the disk saw that only the root files copied... is there a -R switch? Turns out I should have run the following: copy boot: DH0: ALL...

DSC04146 DSC04148 DSC04149

Cool! The next error!? It's trying to map a fonts folder? Checking the bootdisk, there is no such folder. Oh, right, the bootdisk doesn't actually have WB1.3 on it! It's just a booter to get WB1.3 running off the HDD. Go and purchase the Floppy & Hard Disk Image Pack from Amiga Forever, download the ADFs and get them onto your USB stick. Boot the machine with the IDE bootdisk, swap the ADF via the Gotek change disk button to WB1.3 and run the same copy, but this time with the differing drive name: copy workbench1.3: DH0: ALL

DSC04151

And reboot!

DSC04157

Holy moly... my HDD is visible! Note that this was tested on both a Kickstart 1.2 and Kickstart 1.3 Amiga 500! I quickly booted up SysInfo to see what the system specs were.

DSC04170 DSC04172 DSC04171

Yup, it's still a lowly A500. Note, if you want one of these, please drop a comment below and I'll contact you!

Filed under: C64/Amiga No Comments
24Mar/200

Amiga 500 – Gotek

So, the goal was an internal IDE drive, but to do that I needed a boot disk as my A500 with Kickstart 1.2 would not auto-boot. To make it boot and find a HDD, I needed a floppy image written to a floppy disk. Of course, you can't write this with a standard PC drive; instead I either needed a bootable Workbench disk and serial magic (like what I did to the Apple II back in the day) to copy over floppy-copying-software, or just use a Gotek!

I happened to have a spare 'cheap' gotek that I've complained about before which deserved to be used for this hack and slashery.

DSC04004

Could I flash Amiga firmware to it? I don't have an FTDI USB device, but I have plenty of Arduinos! Firstly though, we'll need to allow data access to the Gotek. This means soldering on some pin-headers to the holes right behind the power plug.

DSC04006 DSC04011 DSC04013

Regarding using an Arduino, it's the internet, so someone has already done it. Wiring it up was pretty straight-forward: dedicate two pins to SoftSerial and then make sure the transmission line has a few resistors on it. The Gotek is TTL level, which means 3.3v data signals. As the Arduino puts out 5v, we need a 4.7k resistor between the TX pin and the gotek. Where the resistor meets your wire, also put a 10k resistor to ground.

DSC04020

Yup, you can't see shit because of my 'mood lighting'. The sketch on the Arduino was as follows:

#include <SoftwareSerial.h>
SoftwareSerial mySerial(10, 11); // RX, TX
void setup() {
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
  mySerial.begin(9600);
}

void loop() { // run over and over
  if (mySerial.available()) {
    Serial.write(mySerial.read());
  }
  if (Serial.available()) {
    mySerial.write(Serial.read());
  }
}

Very simply, the code brings up the hardware port to 9600 baud and the same with the softserial port on pins 10 and 11. Pin 11 is TX and had the resistors, as mentioned above. The Gotek then needed 5v and GND from the Arduino. From there, you need to install the STM software and grab the Cortex hex image to flash. It's all in this blog post, with the downloads being at the very bottom.

DSC04021 DSC04023 DSC04024

DSC04026 DSC04029 DSC04030

I then unmounted the existing drive in the Amiga and swapped in the Gotek on a 3d-printed base that a friend (thanks Shouchan!) had made. Unfortunately, it seems to be for a newer revision of the A500? Or I'm mounting it wrong. Regardless, it all went in and the access light just stayed on when I booted the machine. No dice, just the usual Kickstart 1.2 screen asking for a floppy.

DSC04045

At this point, I could probably write a few extra paragraphs here on how I then went and tried FlashFloppy, to have that fail too. I thought I should then try HxC which also, after paying for it, ended up with the same error. So... I slept.

Testing out my 'better' Gotek

I had another Gotek (with display and buttons) that I've previously used in my IBM PCs and didn't really want to hack apart. My assumption was that the 'fake' one above with no display or buttons couldn't handle the 3rd-party firmware and this just had to work!

DSC04047

I did the usual... soldered on the header and programmed it.

DSC04035 DSC04043 DSC04045

This time I used the USB Serial adapter from Jaycar that I'd bought above to get HxC to burn. It wouldn't work via the Arduino as the baud rate needs to be 115200 and the Arduino can't 'pipe' this through quick enough! Anyway, as you can see above, I flashed it and still got the standard 'insert disk' screen. What am I doing wrong!? I then looked at the actual Gotek, which now had a beautiful error on display...

DSC04049

Ah crap... remember back in the day with standard floppy drives... how when you had the ribbon cable backwards, they'd just stay lit up (or even make a horrid repetitive searching noise)!? Amiga drives do this too. Better yet, when a Gotek (with a display) has the ribbon cable backwards, it'll tell you with a rib warning!

All this time spent above was a total waste as I'd just had the cable backwards the whole time. To me, the cable was actually forwards, as pin 1 matched pin 1 on both ends... but it turns out you need to flip it on the Gotek side. I'm sure this information is written in a manual somewhere?

A quick flip of the cable and...

DSC04050

Hoy polloy... I've learnt a lot... have a bit of new usb-serial hardware, and two dismantled Goteks. So, I then purchased the original Workbench disks from Amiga Forever (USD$9.95 for just the floppy images!) and threw the ADFs on the USB drive.

DSC04058 DSC04059 DSC04062

DSC04064 DSC04065 DSC04066

Ah. Nice! Time to give this thing an internal IDE HDD.

Filed under: C64/Amiga No Comments
4Mar/200

Commodore 1084S Monitor Power Switch

As per usual, I saw an auction on eBay and thought I'd have a crack. It happened to be 3 Amiga 500s with a lot of paraphernalia. Included, if I picked it up personally, was a beautiful Commodore 1084S monitor, with a faulty power switch.

DSC03795

After a quick message to a good friend in the Amiga community (thanks Steve!), I had a link to a fellow Australian's anonymous blog on blogspot under the name Random Mayhem. Here's the link to the article, but it seems to have disappeared in the time that I've now come around to writing this post. I picked everything up mid-february, got around to purchasing the items soon after... but now two weeks later the blog is gone! Thankfully, here's the cached version from Google. I have no idea how long that'll be around.

Thankfully, the whole process is very straightforward. First up you'll need to buy a replacement switch from Wagner Electronics Super Store in Sydney. From here, we make minor mods to the switch, as it now has a secondary pole, and then swap it in.

Opening up the Monitor

CRT Monitor contain high voltages, which can be stored for long periods of time. BE CAREFUL! Yes, the usual warning. The large capacitors in CRT monitors will hold charge and need to be discharged if the monitor has been on recently. Grab an insulated flat-head screwdriver and bridge the pins on any capacitor. If you get a spark out of it, then you've probably just saved that spark from travelling through your finger.

DSC03801

There's around 6 screws to remove to take the main housing off. From here, don't pull it away with speed. There are two speakers in the housing and these are connected by relatively-loose cables that run to the front-left of the monitor, where the headphone jack is. You'll need to lift the housing around 5cm off the chassis and then unplug these two cables. Once done, you can remove the housing completely.

DSC03802

DSC03804 DSC03807 DSC03809

From here, give the internals a good clean-up... my unit sure did need it!

Removing the existing switch

The switch is held in with 6 pins: 4 electrical contacts and two for the support bracket. Removing it will involve removing the solder from the pins and then 'walking' the switch out. The basic idea of 'walking' is to heat up the pins on one side of the switch and prise it up as high as possible, only moving it millimetres at a time. From here, heat up and loosen the pins at the other end and prise that up. Once you've got one end clear of the solder pads, heating up the other end should see the whole switch come out.

DSC03811 DSC03816 DSC03823

With the switch out, we can now remove the support bracket. This is held in by two bent tabs on the top side. Grab a set of reverse pliers, the ones that open when squeezed, and slowly bend the top pins open. We want to re-use this support bracket, so try not to snap the tabs off! They should bend relatively easily. Test if you can wriggle the switch vertically out of the bracket each time to prevent opening up the tabs too much. Too much flexing of the tabs could weaken and see them tear off.

DSC03814 DSC03812 DSC03821

With everything apart, we can now install the new switch.

Modifying and installing the new switch

First up, you need to cut off the three pins on the new model. We don't need these on our old monitor. I grabbed my sharpest pair of pliers and cut the tabs off right at the base. I suppose you could use them for a mod if you wanted? LED lighting? an extra amp for the speakers? I had no use though.

DSC03817

Once done, you can slide the support bracket on and gently bend the tabs back into position. The whole switch will then slide perfectly into the previous switch's position. If not, just make sure all the solder holes are clean. Once in place, solder away!

Re-assembling the housing

Just a quick note here... make sure you run the speaker cables correctly. One runs internally around the entirety of the housing so as to keep it from electrical interference produced by the circuitry. I assume there's also heat concerns, so make sure it's not running too close the to main tube. There are actually little plastic tabs where you can push the cable into, to keep it in position when you push the housing onto the chassis.

DSC03817

Also, when plugging the cables back in, make sure you have them the right way around. You wrote down the order, didn't you? Or did you take a photo prior!?

Testing... or... please keep the magic smoke in!

I bit the bullet and just applied power and video data. Thankfully, I got a Kickstart 1.2 boot screen! I knew the Amiga already worked, as I'd tested the black-and-white composite out. I was very happy to see this crispy screen come to life.

DSC03825

DSC03827 DSC03830 DSC03831

As you can see... it was time to load A-Train, which I'd bought whilst I had the A1200, but which had arrived after I'd sold the A1200.

A-Train disks on the Amiga

A quick note here... if you look at the final photo above, you'll notice A-Train is asking for 'Disk 2'.

DSC03834

Uh... 'Disk 2'? I'd booted off 'Lo-res', so I slapped in 'Hi-res'. It worked, but the disk naming is terrible. Nowhere on those disks is the number 1 or 2?

DSC03837

DSC03835 DSC03836 DSC03840

Monitor built-in stand

Just in-case you didn't notice, there's a built-in stand in the base of the monitor. The stand consists of two plates of plastic that are hinged to provide a triangular stand, supporting the rear of the monitor. This is specifically to be used when the front half of the monitor is sitting on the rear edge of your Amiga.

DSC03845 DSC03843 DSC03842

It's actually very handy. Without it, the angle that the monitor sits at would put severe pressure on any cable plugged into the back.

Filed under: C64/Amiga No Comments
29Jan/200

Amsterdam + Koplopers – January, 2020

After being delayed on the return trip from Belgium, I'd chosen a hotel in Amsterdam that I'd always wanted to stay out. The Ibis Amsterdam City Centre Hotel is located both adjacent and on-top-of Amsterdam Centraal Station.

DSC00155

In the shot above you can see the main building to the left and the above-the-rails building to the right. Just like the Wuppertal Schwebebahn, the building is suspended above the platforms by large 45-degree legs. Note that one of the floors in the annex, as the extra building is called, is being refurbished.

DSC00145 DSC00143 DSC00144

..but don't let that stop you as there was absolutely no noise during the stay. You'll find views from all windows of the building as you transfer between lifts to get to the 'upper' floors in the annex.

DSC00139

DSC00141 DSC00146 DSC00147

And then the view from the room. Don't forget to ask for a room with a north-facing view!!

DSC00150 DSC00152 DSC00153

Almere Poort

During the entire trip, we'd gone to meet family in Groningen. The trip consisted of a fast-paced ride through the country-side on a Koploper. As this was my most-favourite NS train, I'd made an effort along the way to determine the easiest station to take a few videos from, where I could capture these trains at full-tilt. I also wanted to make sure it wasn't too far from the city, so I chose Almere Poort as the trains didn't stop when we passed through.

DSC00168

DSC00165 DSC00164 DSC00169

The weather was miserable, but it was the last day in the country and I was determined to check the place out. The station provided good visibility in both directions, but was a little 'urban' with lots of concrete and glass for framing.

DSC00170 DSC00176 DSC00178

Multiple services passed through with Sprinters, as above, stopping to pick up the college kids and residents. No one minded that I was there to take photos of trains. Actually... that's a good point: the NS staff even asked how my train photos were when they saw me with a camera. Anyway... here's the resulting video.

We'll declare it as historical, rather than a work of art.

Filed under: INTL No Comments
27Jan/200

Brussels, Belgium – January, 2020

A quick stopover was made in Brussels, Belguim after visiting York, UK. The main goal was to visit breweries, but a few side-attractions were also entertained. We stayed in the ibis Brussels Centre Gare Midi Hotel, right next to the Brussels Midi Station and had a view of the trains and trams!

DSC0078

The first night was spent with a wander up into town. Dinner was had at KFK Hope where both delicious beer and stoemp were served! I'd nearly demanded McDonald's when I saw the varied menu, but it was then a perfect fluke to find this bar/restaurant with a delicious menu.

The next day involved a lot of walking. Firstly, uphill in search of breakfast. It turns out Belgian's really only drink coffee, so a croissant was purchased from a small convenience-ish store as it looked neat with its layer of icing sugar on top. It happened to be full of apple! So good. Anyway, the goal was to actually wander to a model railway store, but we ended up crossing via two really random tram intersections.

Firstly, up the top of a rise, the intersections of Rue Theodore Verhaegen, Avenue du Parc, Chausee de Waterloo, Rue de l'Hotel des Monnaies, Chaussee d'Alsemberg and Avenue Paul Dejaer create a round-a-bout that includes a tram line! Usually these aren't anything special, as we even have them in Melbourne, but here, the rails will cut straight through the round-a-bout... not also use it as a junction!

DSC00102

The lighting wasn't the best... actually, the rain wouldn't hold off, but we still managed to watch a few trams run the gauntlet and dodge traffic. Of course, they can't dodge at all and so cars new full-well to give way. Another good point was that all cars were giving way to the right and letting cars in front of them enter the round-a-bout. This was very unexpected as here in Australia we also give way to the right and it therefore lends that these guys should've been giving way to the left? A quick search shows that it's known as priority-to-the-right and Belgium actually wants to get rid of the law!

DSC0083 DSC0085 DSC0086

DSC0090 DSC0091 DSC00104

DSC0097

After a while, a tram came through and halted in the intersection before the exit points closest to me. It turns out they also have cab-controls fail and have to switch points manually at times.

DSC00111

From here it was a nice dawdle downhill to find the model railway store. We ended up passing a church that the trams had also been built around.

DSC00115 DSC00116 DSC00117

DSC00119

Further fluking led us to the Marolles Flea Market. Here I successfully acquired a Philips CDI console with broken controller and game for $5. The rain really didn't help the sellers, but most were convinced to just let the wares deal with the elements.

DSC00122 DSC00123 DSC00125

DSC00124

From here, a short wander was had to Cantillon Brewery. This is a living museum, still producing beer, whilst allowing tours of the entire factory. Infact, you're allowed to wander around by yourself after a quick introduction and disclaimer. For a nominal fee, you also get a tasting included at the end. The beer was very wine-like... and super delicious. For someone like me who just devours things, actually taking time to appreciate the 'depth' took a lot more effort than I was expecting!

DSC00130

DSC00126 DSC00128 DSC00133

From there, I finally had my Royal O'Fish at McDonald's, proceeded by an afternoon session at The Belgian Beer Project. This was nice and quiet as we arrived at 2pm as it opened. It quickly changed as a work christmas party arrived. Many tasting trays were created for the 40-odd new guests, but most of them weren't interested... and so we happily received their left-overs.

Finally, the 1757 Amsterdam-bound Thalys was cancelled. There was no mention of it at all on the departure board and I had to line up to find out what was going on. The advice provided was "to take the next train. Choose any free seat after everyone else was seated." We fluked forward-facing seats and got to Amsterdam without much further trouble!

Filed under: INTL No Comments
24Jan/200

York & The Mallard – January, 2020

After getting up-close-and-personal with an EF58 (my favourite Japanese electric locomotive), I realised I needed to tick another box: inappropriately touch 4468 The Mallard. I'd already seen one running, so this task was to be simple enough as the Mallard had been relegated to show-pony duties in the National Railway Museum in York.

The museum is a located to the north-west of York's main railway station. You can easily access it on foot, via the southern entrance of the station. Entry is free, but a donation is highly recommended! You'll find two large halls full of memorabilia, with the main hall being to the right as you enter the museum.

DSC0002

I happened to head staight-ahead as I entered and viewed the smaller hall first. Straight-up I was confronted with a life-size version of my first HO Scale model train. But then I realised it wasn't. My first loco was a 6201 Princess Elizabeth and this happened to be a LMS Hughes Crab... close, but no cigar.

DSC0004

DSC0003 DSC0005 DSC0006

Before long, I was in the main hall and staring at a Eurostar. Although a relatively new engine, it's of historical significance, being one of the first consists to cross the channel on rails. I was perfectly surprised to then see a 0-series Shinkansen cab car!

DSC0007

DSC0008 DSC0009 DSC0010

DSC0011 DSC0013 DSC0012

From there, a quick wander gets you to the back of the Mallard... where the lighting was better than from the front angle. Arghh!

DSC0016

There was something like an up-close-and-personal going on with the group at the back, but I wasn't interested... I just wanted to see the behemoth in person. It lived up to my expectations!

DSC0017DSC0018 DSC0021

DSC0019

And that was that... the rest of the afternoon was spent checking out the town.

York Station

After a night of pub-hopping, it was an early start off to Belgium, via London. To get to the station, I took the extended walking path to the south first, over a railway bridge and back up the western side of the line. Unfortunately, for the entire path there was next-to-zero visibility of the rails. It was also just after 0830 and the sun hadn't even considered shining yet.

After picking up luggage and checking out, I bunkered down with a coffee on a bench at York Station. I was actually very happy to see quite a few cafes located throughout the station! Don't always go for the first Costa Coffee available (although it is located in the old signal box)!

DSC0039

DSC0040 DSC0071 DSC0045

DSC0041 DSC0042 DSC0050

The layout of the railway around the station seemed to indicate that there was a freight bypass to the south. Hence my earlier walk; I wanted to actually see freight through York prior to my departure. After the disappointing stroll, I'd sat at the station with the intent to just watch an Azuma pass through... but was pleasantly surprised when a coal train crawled past me!

DSC0029

DSC0031 DSC0032 DSC0033

After that, a myriad of passenger trains passed in every direction...

DSC0037 DSC0043 DSC0055

DSC0065 DSC0066 DSC0067

DSC0070 DSC0069 DSC0068

Interestingly, that purple metro above had a partner sitting next to it, as you'll see in shots further above. It departed as I was near it and the wheels sounded absolutely terrible! It wasn't quite the sound of flat-spots... or the sound of steel wheels turning a corner... it was a serious grinding sound! A quick look down at the rails showed what the actual cause was...

DSC0053

They've welded zig-zags on the rail-head! Woah... I assume that a train has come in at some point and slid on the DMU oil and hit the buffers. To prevent this, instead of sand or other cleaning, they've added friction-inducing rough-ness to the rail-head. Well played, but can't be good for passenger comfort or the wheel surfaces?

DSC0073

My non-Azuma London-bound LNER service arrived and that was it for York.

Filed under: INTL No Comments