Remotely Controlling A DOS PC via Serial + VGA (with AI)
There's no reason, on earth, to do this... Just use a VM... but I asked AI if we could build a TSR to remotely operate DOOM on the machine... and it responded with:
Very doable for control—the only real bottleneck is video.
A Moxa NPort 5110 is just a standard RS-232 serial port (COM1 at 0x3F8, max ~115,200 bps = ~11.5 KB/s) bridged to a TCP socket. Here is how that stacks up for DOS + Doom on a 386:
1. Control (DOS + Doom Input): Easy
Doom installs its own hardware INT 09h keyboard handler (it ignores INT 16h), so a normal INT 16h key-stuffer won't work in Doom.
The trick: On an AT/386, writing command 0xD2 ("Write Keyboard Output Buffer") to port 0x64 followed by a scan code to port 0x60 makes the 8042 keyboard controller fire a real IRQ1 with that scan code. A tiny ~1–2 KB TSR hooked to COM1 (INT 0Ch / INT 08h) can inject real make/break scan codes that work in both DOS and Doom (and/or use Doom's -control external driver API).
2. Seeing the Screen: The Catch
100% Serial-only: Doom runs in 320×200×256 (64,000 bytes/frame). At 115,200 bps, a raw frame takes ~5.5 seconds. You can have a TSR sample a tiny 4-bit grayscale 64×40 or ASCII-art thumbnail from A000:0000 over serial at ~1–2 FPS, which makes it "turn-based Doom."
Best setup (if you have it): Plug the Trident VGA output into a VGA-to-HDMI + cheap USB capture dongle (or even a webcam aimed at the CRT) for vision, and use the Moxa 5110 TSR purely for keyboard/DOS-prompt control—then Doom runs at full 386 speed with zero serial bandwidth wasted.
(Side note: check how much RAM your AX386HD has—standard Doom needs 4 MB and a 386, though Doom 8088 / FastDoom runs better on slower 386s.)
If you want to build this, let me know:
Do you have a VGA capture / camera for video, or do you want 100% serial-only (text + downsampled framebuffer)?
Which COM port (COM1 / 0x3F8 or COM2 / 0x2F8) is the Moxa 5110 plugged into?
So, my set up? I have a MOXA nPort 5110 (that I found in a bargain bin in Hard-Off somewhere in Japan for 400Yen) and a VGA to HDMI el-cheapo dongle. That won't work with DOS resolutions, so I also am going to use a Kramer Tools VP-420 (another tip-shop purchase) which will upscale to a resolution that the VGA to HDMI adapters like.
Getting The MOXA nPort 5110 Going...
You'll need a Windows 11 PC on the same network (doesn't have to be the same IP subnet) and you'll need to download the Device Search Utility. With everything hooked up, hold down the reset button on the unit for 5 seconds so that it'll reset to it's 192.168.127.254 (docs say .100, but mine reset to .254) IP.
If you're in luck, it'll show in the utility and then you can reset the IP config to match your network. I went for DHCP and the new IP was then accessible in the web browser!
I actually didn't have enough 9-25 pin adapters to use my 25-pin NULL-MODEM unit, so I decided to just get two 9-pin female sockets and wire them together.
The wiring needs to be:
| Female DB9 A (MOXA) | Female DB9 B (PC) | |||
|---|---|---|---|---|
| RXD | Pin 2 | == | Pin 3 | TXD |
| TXD | Pin 3 | == | Pin 2 | RXD |
| GND | Pin 5 | == | Pin 5 | GND |
| RTS | Pin 7 | == | Pin 8 | CTS |
| CTS | Pin 8 | == | Pin 7 | RTS |
| DTR | Pin 4 | == | Pin 6 | DSR |
| DSR | Pin 6 | == | Pin 4 | DTR |
| DCD | Pin 1 | == | Pin 1 | DCD |
Officially just pins 5 and 1 are straight-through, the other pairs are crossed over.
From here, configure the Serial Port from Real COM Mode to TCP Server and save all settings. Also make sure to change the serial port to your expected BAUD rate. 9600,8,N,1 for max compatibility. Let the node save/reset and then open PuTTY, type in the IP @ port 4001 with Connection Type of Other: RAW ...
And then, from DOS:
C:\MODE COM1:9600,N,8,1 C:\ECHO HELLORLD, Let's play Railroad Tycoon! > COM1
And then... BAM:
Done!
Video Capture
This took a bit of mucking around. I had assumed that the VGA to HDMI dingle-dongle would just-work(tm), but it didn't. I then remembered I had an upscaler that I'd found at the tip shop which did VGA to VGA, so I slapped that in-between the machine and the VGA-to-HDMI and it worked!
I then needed to capture the HDMI, so I looked in the box'o'junk again and found a USB-based HDMI capture unit. Will it work?
It needed a crappy driver installed, and Windows 11 Memory Integrity turned off ...
but ...
Winner!... on Windows 11 also... VLC captures it.
Let's Rig AI Up To This...
I simply want to watch AI play a game of Railroad Tycoon. Can it do it? The first thing would be to type up a prompt that covers all of our requirements.
Dear AI, There is a DOS 6.22 machine sitting at the end of a Moxa nPort 5110 IP to Serial adapter located at 192.168.1.169 (port 4001). Can you please write up a TSR (which I'll copy over manually via floppy, unless you can send a byte-stream via serial?) which will open COM1 @ a specified bitrate and perform actions. This is expected to be keyboard/mouse movements to control the machine. On this machine, we'll build an application where you will connect to the nPort via the network and send bytes to the TSR to perform the actions on the machine. You will be able to see the machine's VGA output via video capture on this machine... of which you'll need to process the live image as it comes through to determine what actions to take. Can we start by creating a plan and choosing a language? I'm preferable to a C# windows app which will be able to handle the video stream directly.... but I don't know then how to pass that off to you for processing. This machine is capable of running an LLM locally, so feel free to use OpenCV or some other similar library to process the video locally first, rather than sending the entire stream to your servers? An initial goal would be to run a DIR in C:\ on the machine and read the contents of the folder via video parsing on this side. Of course, this is mildly pointless as you could probably send the screen text buffer via the serial port... Once we get to games though, you'll need to process the video stream quickly. Once this app is running on this machine, I'd then want you to use it to play a game of Railroad Tycoon on that DOS machine. Would you expect to be prompted to carry out steps? Or do you think you can win the easiest scenario without help?
I fed this into Antigravity, choosing to use the latest Google Hydrogen model.
That is a very fun setup—and Railroad Tycoon (1990) is a much better target than Doom for an AI over serial + capture, because you can pause the game while thinking.
Here is the recommended stack and plan.
1. Language & Stack Choice
DOS 6.22 Side (TSR): 16-bit C + inline Assembly (Open Watcom wcl -bt=dos)
Keeps the TSR tiny (~2–4 KB resident), easy to maintain, and compiles on your Windows machine.
Transfer: Copying the compiled .COM/.EXE via floppy once is cleanest (you can bootstrap via CTTY COM1 + DEBUG, but floppy is far less error-prone).
Windows Host Side: C# (.NET 8) + OpenCvSharp4
C# handles System.Net.Sockets.TcpClient (to 192.168.1.169:4001), DirectShow/MediaFoundation/UVC video capture via OpenCvSharp4, and local image pre-processing (frame diffing, nearest-neighbor upscaling of 320×200 DOS video so pixel text is crisp, and optional local OCR).
2. How the Video Stream Connects to the AI
Sending raw video to a remote server isn't needed (and is too slow). Instead, the C# app acts as a local smart bridge:
text
[Sharp AX386HD: DOS 6.22 + TSR]
│ ▲
│ └──── COM1 (RS-232) ◄─── Moxa NPort 5110 (192.168.1.169:4001) ◄───┐
│ │
└─── VGA Out ──► USB Capture Card ──► [C# App + OpenCvSharp4] ───────┘
│
(Only on screen change / on demand:
upscaled PNG + diff/OCR metadata)
│
┌──────────────────────┴──────────────────────┐
▼ ▼
[Option A: Local HTTP/CLI Bridge] [Option B: Autonomous C# Loop]
(I inspect PNG + send key/mouse (Calls your local Vision LLM
commands from this workspace) via Ollama/OpenAI-compat API)
Local OpenCV pre-processing: Only emit/process a frame when pixels actually change (frame diff), crop/upscale 320×200 → 960×600 (nearest-neighbor, zero blur), and optionally run local OCR for text-mode DIR output.
3. Phased Plan
Phase 1 — DOS 6.22 Serial Control TSR (SERCTRL.COM)
COM1 (0x3F8, IRQ4 / INT 0Ch): Opens COM1 at a configurable baud rate (e.g., 19200 or 38400—reliable on a 386 without 16550 FIFO drops) with a small ring buffer.
Simple binary protocol (3-byte packets):
K : Injects hardware scan codes via the 8042 keyboard controller (out 0x64, 0xD2; out 0x60, scancode), so both DOS and games (which hook INT 09h) see real key presses/releases.
M : Hooks INT 33h (Microsoft Mouse driver API) directly in the TSR—so even with COM1 occupied by the Moxa, games like Railroad Tycoon get full mouse support without needing a second serial port on COM2.
Phase 2 — C# Host App (DosBridge)
Connects to 192.168.1.169:4001 via TCP.
Captures the USB VGA feed with OpenCvSharp4, auto-detects active DOS resolution (640×400 text vs 320×200 VGA), and exposes a tiny local REST/CLI interface:
GET /frame.png (upscaled current frame)
POST /key (press/release/type string)
POST /mouse (move/click)
Phase 3 — Milestone 1: DIR C:\ via Video Parsing
Send dir c:\ over TCP → TSR injects keystrokes → C# captures VGA frame → OpenCV crops/upscales → read directory listing purely from the captured image.
Phase 4 — Milestone 2: Railroad Tycoon
Calibrate mouse coordinate scaling (320×200 game space vs capture frame) and verify INT 33h mouse + keyboard shortcuts (Space to pause, F1–F10, Alt menus).
4. Can I Play & Win Railroad Tycoon Without Help?
If you run it through this chat UI: I only act when a turn/tool wakes me up, so I would either run a multi-step tool loop (reading frame.png → sending keys/mouse → sleeping/waiting for screen change) or you'd prompt me to continue periodically.
If the C# app runs an autonomous loop (calling a Vision LLM—either your local one or an API—every 1–2 seconds while pausing the game with Space when needed): No human prompting is required.
Can it win the easiest scenario (Financier / 1830s East Coast US) unassisted?
Likely yes, once calibrated, because the winning opening is straightforward: pause (Space), build a short double-city connection (e.g., New York ↔ Philadelphia), place stations, buy one locomotive with passenger + mail cars, unpause, and reinvest profits.
The main thing that trips up Vision LLMs in 1990 Railroad Tycoon is tiny 320×200 pixel fonts and small UI hitboxes—which is why OpenCV nearest-neighbor upscaling + exact INT 33h mouse positioning in Phase 2 makes all the difference.
Want to start with Phase 1 (writing the DOS TSR), or Phase 2 (setting up the C# + OpenCvSharp4 capture/TCP skeleton) first?
The Result
Hilarity! The code went through a few revisions and ... worked! There were a few things missing from the start, like modifier keys and special keys. For example, numpad keys are needed to build track in Railroad Tycoon, once Num Lock is enabled. This took a few loops, but then some AIs still just couldn't work it out!
Other times the AIs just sent enough Keypresses to crash DOS. Funny. I might continue this project... but just use a VM instead of thrashing a poor old 386!


Melbourne BG SCS Train Timetable 


