The Quadra is a 68040-based Macintosh. Unfortunately, MacOS 8.5 and higher require a PowerPC processor. Regardless of the fact that you can 'add' a PowerPC Processor to the Quadra, MacOS 8.5 is still listed as unsupported on this model.
Impersonating a 'real' PPC
It seems that every model of Macintosh has a specific unique ID stored in the BIOS of the logic board. The MacOS installer will verify this against a known set of values before proceeding. If a correct value is not found, then you'll get the dreaded This program cannot run on your computer error.
There's numerous tutorials online on how to get around this. The Unofficial Turbo 601 Site has instructions and so does Running Mac OS 8.5 on A 68k Mac with A PPC Upgrade Card. The Mac OS 8.5 Special Report has an Install Hack page also describing how to do this.
Note: You don't need to go through the hacking process! The Presto PPC Software can do it all for you and I've described the process further below.
The main point behind all of this is that the hardware is there and ready to roll for MacOS 8.5. Apple, in their infinite (loop) wisdom, decided that they'd wipe the slate clean and only support the 'real' Macintosh PPC models. We can get around this and the following steps will show you how.
Tricking the installer
Drag both the extension and control panel at the same time to your startup system folder. Once copied, jump over to Control Panel and open Wish I Were. Set your Gestalt to Powermac 6100/60. You'll now need to reboot... Don't bother with the Quadra 940 with Upgrade, I thought this might work, but it's probably the original ID that the Quadra 950 CPU/BIOS is already returning.
If you have your installation disk ready to go (it is recommended to install to another physical disk!) then start the MacOS 8.5 installer now.
Note: Prior to even trying the install, make sure you have enabled your PPC Upgrade Card and have set Wish I Were to a relevant value!
Grab the software and burn it to a CD or mount the image on the desktop. Double-click the Mac OS Install application to get started. Installation itself is painless... lots of 'next, 'next', 'finish'. You can choose to customise; for some reason Quickdraw GX was unchecked for my machine, I checked it for fun.
The install will try to update harddisk drivers. Don't be worried if you get a warning that this cannot be done. It will usually only work for drives that have Apple firmware. The installer will continue regardless of a successful firmware update. Note that it will try to update firmware for all attached harddisks; not just the target disk.
Files started copying... (the estimate was 21 minutes) and then fail: "Get QuickTime Pro" could not be copied from Disk "". What? Blank disk label? I had been using the internal drive at this point and decided to switch to the slower PowerCD (see all about my optical drive illusions here.)
I inserted the original Mac OS 8.5 CD in the PowerCD and everything loaded. You could feel the icons refreshing in the finder folder and the screen refreshing the license agreements during the installation steps. The files started copying again and the estimate was 50 minutes! They were smart enough to judge the speed already, prior to any files copied and guessed that this drive would take more than double that of the internal drive.
The file copying continued without errors... was the slower PowerCD more capable of reading a damaged disk? The-slower-the-better with scratches? Either way, the 50 minutes quickly became 46, then 40 and then 30. It still did take a good amount of time. My last Windows 8.1 install felt like it took me only 15 minutes to get to the desktop!
Once the install has finished... make sure the startup disk you installed from is still handy (and remember the SCSI ID!) Rebooting at this point will make Mac OS 8.5 throw the usual "this startup disk will not work on this Macintosh model." .. feel free to reboot just to ensure this occurs.
If it does, hold down CMD-OPTION-SHIFT-DELETE-(SCSI ID) on your next reboot and make your Macintosh boot the harddisk of the associated SCSI ID. Note that 'Delete' needs to be the full 'Delete key, not 'Del'. The number also needs to be a number from across the top of the keyboard, not the numpad! This should be your previous startup disk, or another... either way, we just need an environment to ResEdit the Mac OS 8.5 partition.
Once you're back on a known desktop, download and install ResEdit. Open the 'System Suitcase' from inside the System Folder on your new MacOS 8.5 partition. Once in there, open the gusd resource and then open resource 1. Go to Find -> Find Hex and search for 001f001c, there'll be one occurrence. Highlight the block including 001f and type 0076 for the Quadra 950 (see other model IDs here.)
Quit out of ResEdit, saving your changes... as that you booted with the special shortcut, your startup disk will still be the new MacOS 8.5 partition... reboot your Macintosh. Now you get to watch it pass the hardware check and load to the desktop! Step through the Mac OS Setup Assistant if you're really bored. 'Checking network connections' in the setup assistant took a REALLY long time... minutes... and then never completed... I forced it closed, but Finder then never responded. Sorry Finder, you get a reboot. I then, for the first time in ages, saw the "Your machine was not shutdown/crashed" disk-check screen. I'd forgotten all about it!
Meanwhile, my Asante Card just worked! Note that TCP/IP is disabled by default. Go to the control panel and enable it. I loved it that my Ethernet slot 3 was already a selectable option.
Also, all the disk mounting issues from above had gone away. Blank floppies show up as 'PC' floppies but the PowerCD still wouldn't work with Audio CDs (data CDs were fine.)
Upgrading to MacOS 8.6
I acquired the update and burnt it to a blank CDR. Turns out the internal NEC drive just hates recordable media. Fortunately, the trusty PowerCD came through with the goods and got me started. At the 8.5 desktop, with the CD mounted, I double-clicked on "Upgrade to 8.6" and got the usual "you have an unsupported system." Of course I do... I'd been running peacefully in 8.5 for so long that I forgot that the Wish I Were hack was on the previous 8.1 disk. So, I scrounged around, extracted the extension/control panel from the other disk and dumped them on my 8.5 install. Prior to reboot I adjusted the machine back to a Powermac 6100/60.
After a reboot, I was running the installer. It is extremely streamlined... I get one option to update SCSI HD Drivers and that's about it. The installation proceeded (slowly on the PowerCD) without a hitch.
After the installation, I attempted to get back to the desktop. Unfortunately the installer wasn't going to have it... it kept demanding that a restart was in order. I had no choice... I didn't want to restart because I knew it'd throw an error. Luckily, my 8.1 partition was there, so option-command-shift-delete-0 (remember, that's delete, not del) saw that boot.
Once at the desktop I opened the new System Suitcase in ResEdit, made the hex change to gusd and saved everything. The system was now allowed to reboot.
The desktop came up and everything worked very well. My network wouldn't get a DHCP address to start with, but that ended up being my ethernet-over-power setup... a solid cable fixed it up.
Note that if you have any DHCP issues that you can supposedly update OpenTransport to the version included with MacOS 9. Sustainable Software's page on DHCP and Mac OS indicates that MacOS 8.6 uses a different technique to obtain DHCP addresses. Download OpenTransport 2.6 here. You will probably need to re-enable 'Wish I Were' to install it.
Sonnet Presto PPC 8.5 and a fresh MacOS 8.6 install
It seems to be little-known that this can all be automated. Sonnet, a maker of PPC upgrade cards, released their Presto PPC utilities to do everything listed above in a much easier fashion. Find the manual for an upgrade here. Download the application and install open the archive. You'll get an dialog that allows you to mount the disk or write it to a real floppy. I did the latter and, once successful, you get the Macintosh Text-To-Speech telling you it's complete! How quaint!
From here, I've installed a fresh version onto another SCSI HD. This drive is 36gb and needed to be partitioned. Using my previous install, I had a base to run partitioning software from. It turns out there's quite a few options... your mileage may vary! Check out this page for partitioning and the tools available.
Partitions ready, I copied over my Games, Apps and Development files. I then restarted with the Presto Floppy Disk. The floppy has a raw Apple CD Driver on it and it didn't find any of my CD-ROMs. I threw CD Sunrise in the system folder and then rebooted with the floppy... it tried to see the PowerCD but thought it was uninitialised!
I opened up the MacOS 8.1 partition and copied the ISO9660 File Access, AudioCD Access and High Sierra Access extensions to the floppy. I had to remove the Profiler patches to make this work, but I wont need them yet anyway. After a reboot, still no dice. I swapped the CD Sunrise extension out for the original Apple CD version, just in case that only worked with the access extensions... it still didn't work on reboot.
At this point I realised I wasn't going to be using the floppy to boot and install. So I booted into my 8.1 install and ran the installer from there. I chose to install a fresh version on my newly partitioned HD. Starting the installer got the usual "this software cannot be installed on your machine", so I resorted to the Presto PPC manual.
So, floppy disk inserted, you must first copy the Presto PPC 8.5 Enabler and Presto PPC ID files to your current system folder. Do this and restart your machine. Installation can now proceed as usual. I chose a clean install on the blank partition.
The install took so long that I gave up. Don't even bother using an Apple PowerCD for data... it's toooo slow. Single speed maybe? I booted up Cockatrice III on my Windows machine and mounted the 8.6 ISO. I then shared it via Chooser to the Quadra and installed it that way. Worked a lot quicker.
Rebooting got the usual This software is not designed for this Macintosh. This is expected... the fresh install did not have the PowerPC ID masking files. So, back to the manual, I booted from the floppy and failed ... The System version on the Presto PPC Floppy is too old and wont recognise my 4gb partition. It tells me there isn't enough space to copy the files over! I'd imagine it's an integer size issue.
cmd-opt-shift-del-0 WOULD NOT boot my original partition... it seems that this time around the system wanted to ignore my keypresses! Booting from the floppy and trying the short-cut combination wouldn't boot my other hard disk either... guess the hack? Boot the Presto PPC floppy to the desktop, open the system folder on your MacOS 8.6 install partition and browse to the Startup Disk Control Panel. Changing the startup disk there will actually change it for that entire drive. I then went Special -> Restart and Finder locked up. I forced a restart, the disk didn't eject and we rebooted back into the floppy's archaic desktop. A further restart got me back into MacOS 8.1!
So, now I could copy the extensions over. These are the Presto PPC 8.5 Enabler and the Power Macintosh Card Control Panel (yes, it says 601 Processor Upgrade in the doc... but on the disk that is not its name!) Drag them from the floppy to the 8.6 System Folder.
From MacOS 8.1 I then switched the Startup Disk back to MacOS 8.6... and... the friggen thing booted. Winner!
Black-screen-of-death (aka Sad Mac)
If you do anything to screw your PRAM, then it will default back to the 68k processor. As soon as you do this, you'll get the following error screen:
This error: Sad Mac 0000000F 0000000C indicates that MacOS has tried to boot and found invalid hardware. 8.6 only wants to boot on a real PowerPC, so it's found the 68k processor and failed.
The resolution is to boot from a disk that can boot on 68k (Presto Floppy as above? Older MacOS 8.1 boot disk? etc...) and then get to a control panel that'll let you switch the PRAM back to PowerPC mode. My answer was to boot the Presto disk, go to the System Folder on my 8.6 HDD and then toggle the PowerPC card.
The system then rebooted fine!