modelrail.otenko Trains, computers, vintage, retro, model railways… the lot…

Remote Garage Door Sensor/Opener

So, I live on the top floor of a three-storey building. My garage has an electric door and is vertically below my aparment on the 'basement' level. The following post details my efforts to be able to know if the door is open, and to possibly control it remotely. In the past, I've accidently left the door open and I want to be able to prevent this! I also want to be able to access the garage when the keys aren't nearby.

I've already tried a crappy 433mhz universal remote from eBay, which did train to the garage remote. I then went to test it and 1-in-50 key-presses worked. The purchase ended up being a quick and cheap way to work out that the Vicway V-380G has rolling codes!

Fortunately, this model garage-door-opener also has terminal blocks with a screw-terminal named DOOR. I asked the manufacturer if I could just short that terminal with the one next to it named GND and they confirmed that doing so will operate the door! What should I use to do this? Arduino? Raspi? Any of the above? Sure... but how do I communicate with any of them from three concrete storeys above?

• WIRELESS: Does not work... the signal doesn't reach. My mobile phone gets no wifi signal when in the garage. It does work out in the carpark though.
• ETHERNET: Ok sure... run a cable down the side of the building? Is there an inner cable cavity between solid concrete floors? No, but, ethernet-over-power does work! Do I want my LAN exposed to the entire building's power grid? I dunno.
• LORA: Will this be able to communicate through three layers of solid concrete? Wifi couldn't. I'm still going to give it a bash. Will the signal be encrypted? Can any hacker just muck around with my garage door if they wanted to?

POC #1: Ethernet

I have two pairs of ethernet-over-power devices, and I tested a Netcomm set first. Plugging it in saw the home link light mainly solid, but it was probably blinking more than it usually should. I assume the connection through the building's power grid is noisy-AF and, well, I didn't really think I expected anything less. Regardless... I wasn't here to stream 4K60, I just wanted a crappy website hosted on the Arduino with a button and a bit of status information.

An Arduino Ethernet Shield was purchased from Jaycar, along with a tiny relay board. I have a few boards with 4 or 8 relays... but I didn't want to waste them. The example webserver sketch was uploaded to a UNO with the Ethernet shield attached and the unit was moved to the garage with the ethernet-over-power in place.

And ... nothing. Link, but no data. I don't know what I was expecting... ethernet-over-power was not going to work.

POC #2: LoRa with MQTT

There are a lot of great blogs on the internet... and whenever I think of a cool idea for a project... I know someone has already done it... somewhere... somehow. What I wasn't expecting was to find someone in NZ that was considering the exact same setup of components from jaycar. Jon, thank you for the work you put in to documenting the caveats of these older-style components. I went ahead and purchased both the LoRa Sheild (I already had enough spare Arduinos at home) and LoRa Gateway.

The LoRa Gateway is an all-in-one LoRa radio + Arduino + Linux SOC. The SOC talks to the Arduino via SPI and uses Yun's Bridge Protocols. If you search for any of this, then you'll quickly find that all of this tech is OLD. LoRaWAN is the new standard and this is all LoRa-only.

I had plans to host a webpage on the gateway and send data directly to the LoRa node, but I quickly realised that extending the gateway to do my bidding was going to be difficult. It uses flash disk with most files read-only and the website is hosted from ROM! Instead, I'd have to have the gateway actually be a gateway and transmit data to some other server on my LAN.

The setup would therefore be configured as follows...

Garage		Gateway				NAS
Arduino (Read Sensors)	–>	LORA Gateway (Arduino)	–>	LORA Gateway (Linux SOC)	–>	MQTT Broker	<->	Website
Arduino (Operate Relay)	<–	LORA Gateway (Arduino)	<–	LORA Gateway (Linux SOC)	<–		–>	SQL Database

I chose a local MQTT Broker, as I didn't really want to sign up to another external webservice (they seem to use Thingspeak), to only then have to bring the data back down to the local network once again. The local broker wouldn't persist data, so I would also have to work out a mechanism to store in a DB somewhere. Fortunately (hah, or so I thought) I already has MS SQL running in docker on the NAS.

Setting up the Arduino IDE

Following the actual Dragino documentation with a newer version of Arduino will end in misery. I installed the latest 2.3+ version and got no end of errors of missing libraries. So, what to do? Spin up Win7 on my NAS and set up Arduino 1.8.6.

Once installed, add the following url to the Additional Boards Manager URLs: http://www.dragino.com/downloads/downloads/YunShield/package_dragino_yun_test_index.json

Now go to Board Manager and install the Dragino Yun set of boards:

Next, download the RadioHead Library and extract it into Documents\Arduino\libraries.

Finally, select the board, the port and load up the LoRa_Simple_Server_Yun sketch from the Dragino examples.

If it compiles... try an upload! You'll need to enter your LG01's password...

... and then ...

Seamless! Amazing actually. The code has passed through a Linux SOC and then been transmitted over SPI into the Atmel co-CPU? If this didn't work, then write a comment below and tell me what happened. Next up, set up your first node. I used a spare Leonardo with a LoRa Shield.

Download this sketch (or Jon's version), modify the node_id from 12345 to something relevant (leaving the less-than and greater-than symbols in-place) and check the serial monitor on both sides...

It... works! RSSI of -21 is very good... since the nodes are on the workbench, right next to eachother. Let's see what happens when I shift the node to the garage.

Integrate the Garage

The garage door motor has a terminal block wth DOOR and GND. These need to be bridged to activate (open or close) the door... so I bought a relay module and wired it across. Next up, I wanted to know if the door was open or closed, so I bought two magnetic reed switches and installed them at either end of the door chain traversal. I also threw on a temperature and humidity sensor.

With this all wired up, the Arduino was ready to transmit statistics and react to commands.

Provide basic feedback from the Gateway

I was initially going to use an Arduino Buzzer from Jaycar and the Buzzer Library (who doesn't want bits of the Super Mario tune playing!?), so I could get audio feedback to know what the garage door is doing. Unfortunately, including and operating the buzzer interfered with the Yun Bridge Library communication when writing/reading files from the Linux SOC.

It seems that Timer or PWM operation trashes the Process call and no amount of asynchronous or shell-command tinkering worked. Even the FileSystem write command spewed out empty files. I didn't even bother begging for help on the forums as the library is already deprecated. One note though... The official dragino repo has Arduino 1.6.9 for download and I wonder if this is the final 'supported' version with this gateway. Using 1.8.6 might cause these bugs?

Instead, I purchased a self-contained piezo buzzer and wired it up via my transistor power method. I wasn't sure if the digital pins could deal with the current and therefore used the transistor as a switch. I also grabbed a large flashing LED for consistent 'door is open' statii notification.

Attaching all this to the Gateway was easy enough as they provide a really nice screw-terminal block. The pinout is as follows:

1		2		3		4		5		6		7
+	-	+	-	+	-	+	-	+	-	+	-	a	b
+5v	GND	A0	A1	GND	A3	A4	A5	A6	A7	N/A	GND	D3	D4

I chose to integrate with the Digital pins 3 and 4 for my buzzer and LED respectively.

MQTT Broker Installation

This MQTT Server and Node Tutorial helped a lot, but I still didn't want to use Thingspeak. Instead, I overloaded my NAS further with the Eclipse Mosquitto Docker Image.

I created a base folder on the NAS for the home of Mosquitto. In here, I downloaded the default configuration file and created a data and log directory. The default configuration only allows local connections, so we'll need to edit it to allow external nodes to report to it. There's a great guide here, but the basic idea is to add the following line to the configuration file:

# listener port-number [ip address/host name/unix socket path]
listener 1883 0.0.0.0
listener 9001 0.0.0.0
protocol websockets

Search for #listener and replace that chunk with the bit above. Now we need to create the server so that we can shell in and create a password file. Port 9001 will be used for WebSocket connections and port 1883 will be for standard TCP connections. I've called it mosqii here, but you can call it whatever you like. Just remember to use the new name in each subsequent command.

#docker create --name mosqii -it -p 1883:1883 -p 9001:9001 -v /volume2/SSD/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf -v /volume2/SSD/mosquitto/data:/mosquitto/data -v /volume2/SSD/mosquitto/log:/mosquitto/log eclipse-mosquitto

Once created, start it with:

$ docker start mosqii

With it up and running, use the following command to connect via sh and create a password file with a username and password of your choice:

$ docker exec -it mosqii sh
/ # cd mosquitto/config
/mosquitto/config # mosquitto_passwd -c garage_door_auth garage_door
Password:
Reenter password:
/mosquitto/config #

Type exit and get out of docker. Stop the docker container with:

$ docker stop mosqii

Edit mosquitto.conf once again, adding the newly-created password file to our listener configuration. We couldn't do this at the start as the server would crash if the file wasn't found.

# listener port-number [ip address/host name/unix socket path]
listener 1883 0.0.0.0
listener 9001 0.0.0.0
protocol websockets
password_file /mosquitto/config/garage_door_auth

Download MQTT Explorer and connect to the server to test your settings...

Data!

Configure the Gateway for MQTT

With the Gateway connected to my local LAN, I browsed to my router and checked the DHCP listing. duinotech-xxxxx was listed at IP 192.168.1.156 and a quick browse via chrome brought up the internal website. If you've bought one from Jaycar, then the initial password is duinotech.

The unit was quickly upgraded with firmware v4.3.7 and the root password then changed to dragino.

Let it reboot and do its thing... then browse to the Servers and select MQTT with debugging enabled.

Next head to the MQTT Settings page and fill out the private server details. Add a row to the channels table so that we have something to match on when the LoRa node sends data.

With it all configured, return to the system logging and see what's going on. Thankfully the logging is pretty self-explanatory!

Note that initially I had no data flowing to the newly created server. I actually had to SSH into the Gateway to work out what was going on, but even getting in wasn't easy...

$ ssh -l root 192.168.1.156
Unable to negotiate with 192.168.1.156 port 22: no matching key exchange method found. Their offer: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,kexguess2@matt.ucc.asn.au

Turns out it's using old cipher methods and you need to overload your ssh client to force it to connect...

$ ssh -oKexAlgorithms=+diffie-hellman-group1-sha1 -oHostKeyAlgorithms=+ssh-rsa -l root 192.168.1.156
root@192.168.1.156's password:


BusyBox v1.23.2 (2019-01-10 15:05:04 CST) built-in shell (ash)

 ____  ____      _    ____ ___ _   _  ___
|  _ \|  _ \    / \  / ___|_ _| \ | |/ _ \
| | | | |_) |  / _ \| |  _ | ||  \| | | | |
| |_| |  _ <  / ___ \ |_| || || |\  | |_| |
|____/|_| \_\/_/   \_\____|___|_| \_|\___/

W i F i, L i n u x, M C U, E m b e d d e d

OpenWRT Chaos Calmer 15.05
Version: Dragino-v2 IoT-4.3.7
Build Wed Sep 11 22:30:26 CST 2019

www.dragino.com
----------------------------------------------------

root@dragino-17b9d2:~#

All scripts are in /etc/iot/scripts/. I was going to inspect mqtt_process.sh, but decided to just follow Jon's instructions and replace the existing file.

root@dragino-17b9d2:~# cd /etc/iot/scripts/
root@dragino-17b9d2:/etc/iot/scripts# ls
lg01_pkt_fwd         mqtt_process.sh      mqtt_process_old.sh  mqtt_sub.sh          polish_mqtt_config   tcp_client           tcp_client.lua       xively_routine.lua
root@dragino-17b9d2:/etc/iot/scripts#

The replacement script is here, but that also caused issues for me. I could see in the web logs that it was complaining that "12345/" didn't match any known configuration keys. It turns out that a trailing slash was causing the issue. This can be fixed on line 66 of Jon's mqt_process.sh by removing the trailing slash so that it reads:

CID=`ls /var/iot/channels/`

With this fix done, the data was flowing to the MQTT Broker and visible in MQTT Explorer!

Web remote

A quick website was spun up in a folder on my Windows machine. It uses MQTT.js and reports the data from the garage.

<html>
	<head>
		<script
		  src="https://code.jquery.com/jquery-3.7.1.min.js"
		  integrity="sha256-/JqT3SQfawRcv/BIHPThkBvs0OEvtFFmqPF/lYI/Cxo="
		  crossorigin="anonymous"></script>
		<script src="https://unpkg.com/mqtt/dist/mqtt.min.js"></script>
		<script>
		  const url = 'mqtt://someserver.somewhere.org:9001'
		  const options = {
			  // Clean session
			  clean: true,
			  connectTimeout: 4000,
			  // Authentication
			  clientId: 'garage_door_ctrl',
			  username: 'garage_door',
			  password: 'some_password_here',
			}
			const client  = mqtt.connect(url, options)
			client.on('connect', function () {
			  console.log('Connected')
			  // Subscribe to a topic
			  client.subscribe('channels/garage_door/publish/this_is_a_key', function (err) {
				console.log('ERROR: ' + err);
			  })
			})
			
			
			client.on("message", (topic, message) => {
			  // message is Buffer
			  //console.log(topic.toString() + ": " + message.toString());
			  var splitted = message.toString().split('&');
			  $("#temp").text(splitted[0].split("=")[1]);
			  $("#humidity").text(splitted[1].split("=")[1]);
			  
			if (splitted.length > 3) {
				op = splitted[2].split("=")[1];
				cl = splitted[3].split("=")[1]
				
				$("#open_d").text(op);
				$("#closed_d").text(cl);
				
				if (op == "0") {
					$("#open_close_btn").text("Close Door");
					$("#open_close_btn").prop("disabled", "");
				} else if (cl == "0") {
					$("#open_close_btn").text("Open Door");
					$("#open_close_btn").prop("disabled", "");
				} else {
					$("#open_close_btn").text("...moving...");
					$("#open_close_btn").prop("disabled", "disabled");
				}
			}
			});
			
			function open_close() {
				client.publish('channels/garage_commands', 'open_door');
				$("#open_close_btn").prop("disabled", "disabled");
			}
		</script>
	</head>
	<body>
		<center>
			<button onclick="open_close();" id="open_close_btn" 
				disabled="disabled"
				style="width:90%;height:200px;font-family:tahoma; font-size:32pt;">...determining position...</button>
			<table style="font-family:tahoma; font-size:32pt;">
				<tr>
					<td>T:<span id=temp>?</span></td>
					<td>/</td>
					<td>H:<span id=humidity>?</span></td>
					<td>/</td>
					<td>O:<span id=open_d>?</span></td>
					<td>/</td>
					<td>C:<span id=closed_d>?</span></td>
				</tr>
				<tr>
					<td colspan=4>
						<center id=message></center>
					</td>
				</tr>
			</table>
		</center>
	</body>
</html>

It also sends an "open_door" message to the MQTT server on channels/garage_commands when you hit the button. Note that you need to have enabled websockets on 9001 to be able to use the MQTT.js library!

Sending data back to the Garage

One-day data is great, but how do we get the data flowing in the other direction? Turns out we actually need to subscribe to the MQTT Broker on the LG01 and act when data changes. Unfortunately, the LG01-S doesn't support configuring MQTT subscription settings in the Web UI? The documentation here indicates there should be an MQTT Subscribe configuration area, but we don't have it. Fortunately it does tell us the shell commands!

mosquitto_sub -h 192.168.199.148 -p 1883 -i dragino-1b7060 -t command

So, we can use mosquitto_sub to get commands from the MQTT broker... but how do we then get them into the MCU to send over LoRa? Turns out there's a 'talkback' server demo in the github which makes use of the Yun Process Library once again.

The goal would be to store the commands received from MQTT in a file somewhere on the Linux SOC, without bashing the flash too much and causing it to wear out, and then read the file back into the MCU. The Linux SOC offers a /tmp folder which is actually hosted in RAM, so this'll work for us. We can then read the file via the Yun Process Library on the MCU, and, if the content of the file matches a command that we want to react to, we can then send a command via the LoRa network to the node in the garage and clear the local file. Actually, we should clear the local file regardless, as there are no other consumers and we want to know if/when another MQTT message comes in.

So, on the Gateway MCU side I created an mqtt_sub.sh script in the /etc/iot/scripts/ folder:

#!/bin/sh
touch /var/last_command
while true
do
        echo "waiting..."
        mosquitto_sub -C 1 -u garage_door -P l0r@l0r@ -h 192.168.1.61 -p 1883 -t channels/garage_commands > /var/last_command_tmp
        mv /var/last_command_tmp /var/last_command
        cat /var/last_command
done

Since we now have a file to read, I added the following chunk to the gateway node so that it could send out a message over LoRa:

void checkForCommands() {
  String lastCommand = "";

  Process pDown;
  pDown.begin("cat");
  pDown.addParameter("/tmp/last_command");
  pDown.run();    // Run the process and wait for its termination

  while (pDown.available() > 0)
  {
    char c = pDown.read();
    if (c != '\n' && c != '\r') lastCommand += c;
  }
  
  if (lastCommand != "") {
    if (lastCommand == "open_door") {
      Console.println("Opening door!...");
      uint8_t data[] = "OPEN_DOOR";
      rf95.send(data, sizeof(data));
      rf95.waitPacketSent();
    } else {
      Console.print("Unknown command: ");
      Console.println(lastCommand);
    }
    
    File last_cmd = FileSystem.open("/tmp/last_command", FILE_WRITE);
    last_cmd.println("");
    last_cmd.close();
  }
}

And it worked! The message was sent to the LoRa node and the arduino code was updated accordingly to watch for messages and act!

The only note above is that the /var folder is actually a symlink to /tmp, so they can be used interchangeably. Also note that the step to write from the temporary file to the real command file is essential. If you try to pipe out from mosquitto_sub directly then the file will be empty until a message comes in.

I then scheduled this in /etc/rc.local so that it started on boot of the gateway. Make sure you keep the ampersand at the end!

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

/etc/iot/scripts/mqtt_sub.sh &

exit 0

With this, the unit was listening consistently to the MQTT broker for commands.

Storing the data long-term!

For data persistence... any sane person would use MySql. Unfortunately, I have MS SQL running in docker... because I could... as I use it for other application development. Instead of runnign yet another DB on the NAS, it'd be nice to store this data in there also! Of course, it's never that easy! There's a great example here using python, I need ODBC from Microsoft... and I'm running this raw on the NAS... so no package manager!

So, yeah, this going to have to be ANOTHER docker container? Do I have to? Can this NAS even handle all of this? Can't I run a plugin on the MS SQL docker container? Let's see what's running on the shell inside the SQL container and if I can just run an extra process.

$ docker exec -u 0 -it sql1 /bin/bash
root@sql1:/# python3
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
root@sql1:/# pip
bash: pip: command not found
root@sql1:/# apt-get install pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package pip
root@sql1:/# apt-get update
Get:1 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease [3632 B]
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Get:3 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Get:4 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main arm64 Packages [40.1 kB]
Get:5 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages [164 kB]
Get:6 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main all Packages [1035 B]
Get:7 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main armhf Packages [14.6 kB]
Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2771 kB]
Get:9 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Get:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages [1792 kB]
Get:12 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [44.7 kB]
Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1129 kB]
Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2104 kB]
Get:15 http://archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [266 kB]
Get:16 http://archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages [164 kB]
Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [17.5 MB]
Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2858 kB]
Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [51.8 kB]
Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1420 kB]
Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2378 kB]
Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [81.0 kB]
Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [33.7 kB]
Fetched 33.4 MB in 14s (2340 kB/s)
Reading package lists... Done
root@sql1:/# apt-get install pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'python3-pip' instead of 'pip'
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-11 dirmngr dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base gnupg gnupg-l10n
  gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libassuan0
  libbinutils libc-dev-bin libc-devtools libc6 libc6-dbg libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl
  libfontconfig1 libfreetype6 libgcc-11-dev libgd3 libgdbm-compat4 libgdbm6 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 libjs-jquery libjs-sphinxdoc libjs-underscore libksba8
  liblocale-gettext-perl liblsan0 libmpc3 libnpth0 libnsl-dev libperl5.34 libpng16-16 libpython3-dev libpython3.10 libpython3.10-dev libpython3.10-minimal libpython3.10-stdlib libquadmath0
  libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1 libwebp7 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev lto-disabled-list make manpages manpages-dev netbase
  patch perl perl-modules-5.34 pinentry-curses python3-dev python3-distutils python3-lib2to3 python3-pkg-resources python3-setuptools python3-wheel python3.10 python3.10-dev python3.10-minimal
  rpcsvc-proto xz-utils zlib1g-dev
Suggested packages:
  binutils-doc cpp-doc gcc-11-locales dbus-user-session libpam-systemd pinentry-gnome3 tor debian-keyring g++-multilib g++-11-multilib gcc-11-doc gcc-multilib autoconf automake libtool flex
  bison gcc-doc gcc-11-multilib parcimonie xloadimage scdaemon apache2 | lighttpd | httpd glibc-doc git bzr libgd-tools gdbm-l10n libstdc++-11-doc make-doc man-browser ed diffutils-doc perl-doc
  libterm-readline-gnu-perl | libterm-readline-perl-perl libtap-harness-archive-perl pinentry-doc python-setuptools-doc python3.10-venv python3.10-doc binfmt-support
Recommended packages:
  libnss-nis libnss-nisplus
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential cpp cpp-11 dirmngr dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base gnupg gnupg-l10n
  gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm javascript-common libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libassuan0
  libbinutils libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libctf-nobfd0 libctf0 libdeflate0 libdpkg-perl libexpat1-dev libfakeroot libfile-fcntllock-perl libfontconfig1
  libfreetype6 libgcc-11-dev libgd3 libgdbm-compat4 libgdbm6 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 libjs-jquery libjs-sphinxdoc libjs-underscore libksba8
  liblocale-gettext-perl liblsan0 libmpc3 libnpth0 libnsl-dev libperl5.34 libpng16-16 libpython3-dev libpython3.10-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev libtsan0 libubsan1
  libwebp7 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxpm4 linux-libc-dev lto-disabled-list make manpages manpages-dev netbase patch perl perl-modules-5.34 pinentry-curses python3-dev
  python3-distutils python3-lib2to3 python3-pip python3-pkg-resources python3-setuptools python3-wheel python3.10-dev rpcsvc-proto xz-utils zlib1g-dev
The following packages will be upgraded:
  libc6 libc6-dbg libpython3.10 libpython3.10-minimal libpython3.10-stdlib python3.10 python3.10-minimal
7 upgraded, 103 newly installed, 0 to remove and 14 not upgraded.
Need to get 113 MB of archives.
After this operation, 318 MB of additional disk space will be used.
Do you want to continue? [Y/n]

That's a positive shiteload of dependencies... but we're on the NAS.. I have ~25tb free... so let's goooooo....

Get:106 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-distutils all 3.10.8-1~22.04 [139 kB]
Get:107 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-dev amd64 3.10.6-1~22.04 [26.0 kB]
Get:108 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-setuptools all 59.6.0-1.2ubuntu0.22.04.1 [339 kB]
Get:109 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-wheel all 0.37.1-2ubuntu0.22.04.1 [32.0 kB]
Get:110 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-pip all 22.0.2+dfsg-1ubuntu0.4 [1305 kB]
Fetched 113 MB in 38s (3007 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 9933 files and directories currently installed.)
Preparing to unpack .../libc6-dbg_2.35-0ubuntu3.8_amd64.deb ...
Unpacking libc6-dbg:amd64 (2.35-0ubuntu3.8) over (2.35-0ubuntu3.7) ...
Preparing to unpack .../libc6_2.35-0ubuntu3.8_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.34.0 /usr/local/share/perl/5.34.0 /usr/lib/x86_64-linux-gnu/perl5/5.34 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.34 /usr/share/perl/5.34 /usr/local/lib/site_perl) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
/var/lib/dpkg/tmp.ci/preinst: 9: arithmetic expression: expecting primary: "5 * 10000 + 13 * 100 + "
dpkg: error processing archive /var/cache/apt/archives/libc6_2.35-0ubuntu3.8_amd64.deb (--unpack):
 new libc6:amd64 package pre-installation script subprocess returned error exit status 2
Errors were encountered while processing:
 /var/cache/apt/archives/libc6_2.35-0ubuntu3.8_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@sql1:/#

Short-lived excitement. Seems it wants to run a configuration screen but it can't work out how to display a text GUI? Anyway, this is the internet... someone has had this error before. So, remove the half-installed , set up apt-utils and go:

apt install apt-utils
echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections

There's actually a second error above regarding arithmetic. It's trying to calculate a version number, but the NAS is reporting an x for the minor version via uname. We need to mask /bin/uname and make it provide the information we need (thanks to the help over here):

root@sql1:/bin# apt-get install nano
...
root@sql1:/bin# mv /bin/uname /bin/uname_old
root@sql1:/bin# nano /bin/uname

Paste in the following, replacing the version number appropriately:

#!/bin/sh
case "$1" in
"-r")
echo 5.13.0
;;
*)
/bin/uname_old $1
esac

Don't forget to chmod a+x it. You should then get:

root@sql1:/bin# uname -r
5.13.0
root@sql1:/bin# uname -a
Linux sql1 5.13.x #1 SMP Wed Jun 12 00:11:51 CST 2024 x86_64 x86_64 x86_64 GNU/Linux

And then, try try try again...

root@sql1:/usr/bin# apt-get install pip
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'python3-pip' instead of 'pip'
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential...
Suggested packages:
  binutils-doc cpp-doc gcc-11-locales...
Recommended packages:
  libnss-nis libnss-nisplus
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu build-essential...
The following packages will be upgraded:
  libc6 libpython3.10 libpython3.10-minimal libpython3.10-stdlib python3.10 python3.10-minimal
6 upgraded, 103 newly installed, 0 to remove and 14 not upgraded.
Need to get 99.4 MB of archives.
After this operation, 318 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
...
Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 fontconfig-config all 2.13.1-4.2ubuntu5 [29.1 kB]
Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-l10n all 2.2.27-3ubuntu2.1 [54.4 kB]
Get:71 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnupg-utils amd64 2.2.27-3ubuntu2.1 [308 kB]
Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg amd64 2.2.27-3ubuntu2.1 [519 kB]
Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 pinentry-curses amd64 1.1.1-1build2 [34.4 kB]
Get:74 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 gpg-agent amd64 2.2.27-3ubuntu2.1 [209 kB]
...
Fetched 99.4 MB in 46s (2182 kB/s)
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 9572 files and directories currently installed.)
Preparing to unpack .../libc6_2.35-0ubuntu3.8_amd64.deb ...
Unpacking libc6:amd64 (2.35-0ubuntu3.8) over (2.35-0ubuntu3.7) ...
...
Setting up libalgorithm-diff-xs-perl (0.04-6build3) ...
Setting up libalgorithm-merge-perl (0.08-3) ...
Setting up libpython3-dev:amd64 (3.10.6-1~22.04) ...
Setting up python3-dev (3.10.6-1~22.04) ...
Processing triggers for libc-bin (2.35-0ubuntu3.7) ...
/sbin/ldconfig.real: /opt/mssql/lib/libc++.so.1 is not a symbolic link

/sbin/ldconfig.real: /opt/mssql/lib/libc++abi.so.1 is not a symbolic link

root@sql1:/# pip install paho-mqtt
Collecting paho-mqtt
  Downloading paho_mqtt-2.1.0-py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.2/67.2 KB 40.3 kB/s eta 0:00:00
Installing collected packages: paho-mqtt
Successfully installed paho-mqtt-2.1.0
root@sql1:/#

Errors? But, yey, paho-mqtt is installed! Install pyodbc also, and then grab the scripts and follow the instructions for the MS SQL driver over here. I copied the ubuntu scripts, removed sudo as it's not installed, installed curl as it's required and then kicked it off.

if ! [[ "16.04 18.04 20.04 22.04" == *"$(lsb_release -rs)"* ]];
then
    echo "Ubuntu $(lsb_release -rs) is not currently supported.";
    exit;
fi

curl https://packages.microsoft.com/keys/microsoft.asc | tee /etc/apt/trusted.gpg.d/microsoft.asc
curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | tee /etc/apt/sources.list.d/mssql-release.list

apt-get update
ACCEPT_EULA=Y apt-get install -y msodbcsql17
# optional: for bcp and sqlcmd
ACCEPT_EULA=Y apt-get install -y mssql-tools
echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
source ~/.bashrc
# optional: for unixODBC development headers
apt-get install -y unixodbc-dev

root@sql1:/database_tools# ./inst_ms.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   983  100   983    0     0    231      0  0:00:04  0:00:04 --:--:--   240
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    88  100    88    0     0     27      0  0:00:03  0:00:03 --:--:--    27
Hit:1 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease
Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:4 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Fetched 257 kB in 9s (29.6 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
msodbcsql17 is already the newest version (17.10.6.1-1).
msodbcsql17 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
mssql-tools is already the newest version (17.10.1.1-1).
0 upgraded, 0 newly installed, 0 to remove and 14 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libodbccr2
The following NEW packages will be installed:
  libodbccr2 unixodbc-dev
0 upgraded, 2 newly installed, 0 to remove and 14 not upgraded.
Need to get 264 kB of archives.
After this operation, 1895 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libodbccr2 amd64 2.3.9-5ubuntu0.1 [16.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unixodbc-dev amd64 2.3.9-5ubuntu0.1 [248 kB]
Fetched 264 kB in 8s (33.1 kB/s)
Selecting previously unselected package libodbccr2:amd64.
(Reading database ... 19952 files and directories currently installed.)
Preparing to unpack .../libodbccr2_2.3.9-5ubuntu0.1_amd64.deb ...
Unpacking libodbccr2:amd64 (2.3.9-5ubuntu0.1) ...
Selecting previously unselected package unixodbc-dev:amd64.
Preparing to unpack .../unixodbc-dev_2.3.9-5ubuntu0.1_amd64.deb ...
Unpacking unixodbc-dev:amd64 (2.3.9-5ubuntu0.1) ...
Setting up libodbccr2:amd64 (2.3.9-5ubuntu0.1) ...
Setting up unixodbc-dev:amd64 (2.3.9-5ubuntu0.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.7) ...
/sbin/ldconfig.real: /opt/mssql/lib/libc++.so.1 is not a symbolic link

/sbin/ldconfig.real: /opt/mssql/lib/libc++abi.so.1 is not a symbolic link

root@sql1:/database_tools#

Before-long, SQL drivers were installed. Now, the meat. We need a python script to watch MQTT. Using chunks from this tutorial, I came up with the following:

#!/usr/bin/python3
import paho.mqtt.client as mqtt
import pyodbc
from datetime import datetime

#MQTT Settings
brokerAddress = "localhost"
userName = "garage_door"
passWord = "password_here"
subscribeTopic = "garage_door/publish/this_is_a_key"

# Connect to Database
driver = "{ODBC Driver 17 for SQL Server}"
server = "localhost"
database = "MQTT_DATA"
username = "mqtt_user"
password = "password_here"
connectionString = "DRIVER=" + driver + ";SERVER=" + server + ";DATABASE=" + database + ";UID=" + username + ";PWD=" + password
conn = pyodbc.connect(connectionString)
cursor = conn.cursor()

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
        if rc == 0:
                print("Connected successfully")
        else:
                print("Connect returned result code: " + str(rc))

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
        topic = msg.topic
        measurementValue = msg.payload.decode("utf-8")
        SaveToDatabase(topic, measurementValue)

def SaveToDatabase(topic, measurementValue):
        print(topic + " " + measurementValue)
        #Find Date and Time
        now = datetime.now()
        datetimeformat = "%Y-%m-%d %H:%M:%S"
        measurementDateTime = now.strftime(datetimeformat)
        # Insert Data into Database
        query = "INSERT INTO MEASUREMENTDATA (SensorName, MeasurementValue, MeasurementDateTime) VALUES (?,?,?)"
        sensorName = topic
        parameters = sensorName, measurementValue, measurementDateTime
        cursor.execute(query, parameters)
        cursor.commit()

# Create the MQTT client
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
#client.tls_set(tls_version=mqtt.ssl.PROTOCOL_TLS) #this may cause issues if not set up.
client.username_pw_set(userName, passWord)
client.connect(brokerAddress, 1883)
client.subscribe(subscribeTopic)
client.loop_forever()

With all the correct drivers installed, I got the following:

root@sql1:/database_tools# ./mqtt_watcher.py
Traceback (most recent call last):
  File "/database_tools/./mqtt_watcher.py", line 20, in 
    conn = pyodbc.connect(connectionString)
pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'mqtt_user'. (18456) (SQLDriverConnect)")

HAHA! Yes. It hates me.. and it should... none of those credentials are correct. I went and corrected the auth, the table and the insert schema and the data was logged!

More (Bit)Bang For Your Buck

Arduinos are a great hobbyist platform for projects, but you'll often encounter speed and memory issues when you try to squeeze in too many features. Fortunately, being an open source system, one can easily bypass the standard libraries and push these little units as hard as possible. Of course, once you're past the warranties and disclaimers, you're out on your own.

I've previously had to jump out of the IDE and use 'faster' code whilst using a Dreamcast Controller for my Model Railway, but that was a little more advanced than what we're about to do here. Today's goal is to bitbang the data stream to the Red LED Sign as quickly as possible. If you check that post, you'll see I'm already using digitalWriteFast, but it seems that we can go even quicker with the SPI library.

What's SPI, you ask? To steal from Arduino: Serial Peripheral Interface (SPI) is a synchronous serial data protocol used by microcontrollers for communicating with one or more peripheral devices quickly over short distances. As it turns out, that's exactly what we want to do! We want the display to be a slave and, well, we're just going to hack the right RX/TX lines together to get the data to flow. A fellow Arduino-er has done this here whilst interfacing with a 74HC595 and provides a few hints as to problems we might run in to.

Actually, now that I think of it, the data is to be sent out in bytes, but my screen is no where near the order of a clean multiplication of bytes... will have to work on that. Also... the data I read is the top 'line' of each letter, it's not a clean char in any memory array. Hmmm...

Test Patterns

Without pulling apart my currently-hacked-together-and-functional-display, I used another Uno I had lying around and hooked it up to the two spare LED panels. The whole display came with 5 panels, but I wasn't able to smoothly run them all at once, so these two were not in use.

The hook-up is pretty straight forward. The Wikipedia article on SPI gives a little more information on the bus and, more specifically, dictates which wires we need to use. We'll need the data output wire and the clock, known as MOSI, or Master Out-Slave In, and SCLK, the serial clock. Note that some devices read the data pin when the clock transitions from LOW to HIGH, and some vice-versa. Fortunately, this can all be configured via the SPI library.

With the following code, I managed to get a test pattern on the display...

#include <SPI.h>
void setup() {
  for (int p = 0; p <= 12; p++) {
    pinMode(p, OUTPUT);
    digitalWrite(p, LOW);
  }
  SPI.begin();
  SPI.setDataMode(SPI_MODE2);
  SPI.setBitOrder(LSBFIRST);
  SPI.setClockDivider(SPI_CLOCK_DIV128);
}

bool isEvenRow = false;
void loop() {
  isEvenRow = false;
  for (int row = 0; row < 7; row++) 
  {
    if (isEvenRow) {
      SPI.transfer(B01010101);
      SPI.transfer(B01010101);
      SPI.transfer(B01010101);
      SPI.transfer(B01010101);
    } else {
      SPI.transfer(B10101010);
      SPI.transfer(B10101010);
      SPI.transfer(B10101010);
      SPI.transfer(B10101010);
    }
    digitalWrite(row + 3, HIGH);
    delayMicroseconds(150);
    digitalWrite(row + 3, LOW);
    isEvenRow = !isEvenRow;
  }
}

Well, neat! It worked? Playing with the SPI_CLOCK_DIv variable made the loop even quicker, down to the point where I had to up the delay at the end so that the LEDs would illuminate for a longer timeframe. SPI_CLOCK_DIV determines the speed at which the data is sent out, dividing the main 16mhz clock on the Arduino. DIV2 halves it (8mhz), DIV4 = 1/4 = 4mhz, DIV8 = 1/8 = 2mhz, DIV16 = 1/16 = 1mhz, etc...

Anyway, the test pattern was great... but I needed to output text, in rows of bytes!?

What About Text?

So this is where it gets interesting. My font array has 7 rows of 5 pixels per font. I then have up to 6 characters per LED display unit, equalling a total of 30 bits each. Of course, there's 8 bits to a byte, so the neatly rounded-up value would be 32 bits. I could therefore send out 4 bytes to get each row transmitted over the wire. But do be warned, if you start the data on the first bit out, it'll 'run off the edge' of the screen as the screen is only 30 bits wide...

Bytes	00								01								02								03
Pixels	--	--	00	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29
Row 0	00	00	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	00	00	00
Row 1	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 2	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 3	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 4	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	00	00	00	00
Row 5	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 6	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00

As you can see, the text to be displayed is "TEST!" in the cute Arduino 5x7 font. Our display is 30 bits wide, so there's two columns of blank to the left. After the 2 pixel offset, we start throwing out each row of data. This won't be overly-easy either, as we need to get the rows of bits per character per font.

So, based on the current pixel to be pushed out, firstly get the character from the string array. We then need to look up the font pixel array for this character, only getting the relevant row that we're trying to display. Once we have that, we can then just send the bits out to a temporary array of bytes. As that we'll only have 5 bits per font line row, we'll actually need to merge multiple characters into each byte of the array. If we then want to scroll text, we'll have to put a little more effort in. This isn't overly different from when we were sending the data out bit-by-bit; we're just stuffing it into bytes now, making sure we get everything into the correct byte in the output array.

I jammed this calculation into one row below and the output was actually quite surprising! Mainly in the fact that it actually worked!

#include <SPI.h>
#include "myfont.h"

String serialString = "TEST! ";
const int BUFFER_LENGTH = 4;
const int DISPLAY_LENGTH = 30;
byte* datarow = new byte[BUFFER_LENGTH];

void setup() {
  for (int p = 0; p <= 12; p++) {
    pinMode(p, OUTPUT);
    digitalWrite(p, LOW);
  }
  SPI.begin();
  SPI.setDataMode(SPI_MODE2);
  SPI.setBitOrder(LSBFIRST);
  SPI.setClockDivider(SPI_CLOCK_DIV128);
}

void loop() {
  for (int row = 0; row < 7; row++) 
  {
    for (int col = 0; col < DISPLAY_LENGTH; col++) {
      datarow[((col + ((BUFFER_LENGTH * 8) - DISPLAY_LENGTH)) / 8)] |= 
        ((font[serialString[col / 5]][col % 5] & (1 << (6 - row))) 
          ? 1 : 0) << ((col) % 8);
    }
    for (int col = 0; col < BUFFER_LENGTH; col++) SPI.transfer(datarow[col]);
    digitalWrite(row + 3, HIGH);
    delayMicroseconds(500);
    digitalWrite(row + 3, LOW);
  }
}

Everything was actually working nicely... but then I tried to extend it out to 4 displays. 120 pixels out, even at the fastest SPI, caused severe flickering. It dawned on me that I was doing some pretty hefty lifting in my display loop, presumably slowing down the drawing. It's actually a critical loop and it seems that any processing will slow down the line drawing, causing the flickering I was seeing.

With this problem in mind, it became apparent that I'd need to move the 'buffer' calculation out of the main loop. I quickly tested this theory by moving it to a once-off in the setup procedure.

#include <SPI.h>
#include "myfont.h"

String serialString = "TEST! ";
const int BUFFER_LENGTH = 4;
const int DISPLAY_LENGTH = 30;
const int DISPLAY_HEIGHT = 7;
byte* datarow = new byte[DISPLAY_HEIGHT][BUFFER_LENGTH];
void setup() {
  for (int p = 0; p <= 12; p++) {
    pinMode(p, OUTPUT);
    digitalWrite(p, LOW);
  }
  SPI.begin();
  SPI.setDataMode(SPI_MODE2);
  SPI.setBitOrder(LSBFIRST);
  SPI.setClockDivider(SPI_CLOCK_DIV128);

  for (int row = 0; row < DISPLAY_HEIGHT; row++) 
  {
    for (int col = 0; col < DISPLAY_LENGTH; col++) {
      datarow[row][((col + ((BUFFER_LENGTH * 8) - DISPLAY_LENGTH)) / 8)] |= 
        ((font[serialString[col / 5]][col % 5] & (1 << ((DISPLAY_HEIGHT - 1) - row))) 
          ? 1 : 0) << ((col) % 8);
    }
  }
}

void loop() {
  for (int row = 0; row < DISPLAY_HEIGHT; row++) 
  {
    for (int col = 0; col < BUFFER_LENGTH; col++) SPI.transfer(datarow[row][col]);
    digitalWrite(row + 3, HIGH);
    delayMicroseconds(500);
    digitalWrite(row + 3, LOW);
  }
}

My text was static, so I now only built the buffer once. Would you believe it? The rendering was beautiful! I couldn't even notice a flicker. This was then short-lived when I realised I needed to scroll text. Again, there was no need to rebuild the buffer on each loop, so I wrote a procedure and only called it when there was text to move; defined by a delayed scroll rate.

This worked nicely, but there was an obvious jarring delay as the text shifted along. You could feel it re-processing in-between the character movements. I was still happy with it! I went ahead and reconfigured my initial display setup to use two Arduinos: one for the remocon+wifi and the other to receive serial data and light this dispalay.

But Wait, Why Don't I Build The Entire Text Buffer?

Whilst writing this up, it occurred to me that I was only rebuilding the buffer for the text that was visible on the display, not the entire string. If I had the full text buffer already created somewhere in memory, I then would only need to get SPI to send out the data from a specific bit offset. But, how do you start sending data via SPI from half-way through a byte? My fonts/characters were 5 bits wide and each byte had 1.5ish characters in them! I considered just shifting the pointer along the bytes, but the scrolling would be horrible.

So, I want to send out data from half way through a byte? Let's just borrow that table from above again:

Bytes	00								01								02								03
Pixels	--	--	00	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29
Row 0	00	00	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	00	00	00
Row 1	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 2	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 3	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 4	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	00	00	00	00
Row 5	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 6	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00

Officially, on the 3rd loop of scrolling, I would want to send out the green bits. I'd also then want to send out five zeroes at the other end so that the scrolling text trails off and doesn't wrap around. I can't tell SPI to start at the 4th bit of a byte? So I'll need to create a new buffer and populate this as quickly as possible, line by line, just as we're about to render it. Thanks to the glory of the internet, there's always someone else who wants to do something similar. Effectively we have an array of bytes (unsigned chars, meaning we get all 8 bits) and we'll need to shift the first one and append the next one. It'll be very much like a catepillar crawling along.

So, if we build the full text buffer array when the string comes in, we can then push the relevant chunk into the display buffer. There'll be no font calculations, just bit for bit copying, and this will hopefully still be fast enough. We can even make sure we only do it line by line, and only as required... if we're on a byte boundary, then we don't need to shift anything!

I mentioned shifting above, and that's what we'll need to do to get the chunks of bytes into the final display buffer. If you look at the table above, you'll notice I coloured the cells in light green and dark green. The first 3 bits are light green and come from the 'end' of the first byte of the text buffer. The next 5 bits are from the 'start' of the second byte in the text buffer. This chain then continues on as we shift the bits. The >> and << C operators will assist us with this task as that's exactly what they're designed to do: shift bits in a variable in a certain direction by a certain amount. Let's try some pseudo-code first....

define a string buffer that has the string to display: "TEST!";
build a buffer for this string: stringBitBuffer[7][4];
define a display buffer that is enough for 1 panel: displayBuffer = byte[7][4];
if our scroll offset is a multiple of 8
  foreach stringBitBuffer[scrolloffset /8] SPI.writeRow();
  write extra zeroes at the end.
else if our scroll offset is NOT a multiple of 8
  foreach byte in stringBitBuffer starting from (scrollOffset/8)
    shift the first block of the text buffer right scrollOffset times
    make anything to the left of the text in that byte zero.
    move the resulting value into the current displayBuffer byte.
    get the next text buffer byte. 
    take the first (8 - scrollOffset) bits and add them to the current displayBuffer byte
  next
end if

Here's an animation showing what bits we want where...

Bytes	00								01								02								03
Pixels	--	--	00	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29
Row 0	00	00	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	00	00	00
Row 1	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 2	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 3	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 4	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	00	00	00	00
Row 5	00	00	00	00	01	00	00	01	00	00	00	00	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
Row 6	00	00	00	00	01	00	00	01	01	01	01	01	01	01	01	01	01	00	00	01	00	00	00	00	01	00	00	00	00	00	00	00
TB	00	00	00	00	00	00	00	00	01	01	01	01	01	01	01	01	02	02	02	02	02	02	02	02	03	03	03	03	03	03	03	03
Bytes	00								01								02								03

The table above shows the 8 frames that need to be copied over to the framebuffer between bytes. Once the data has shifted a whole byte to the left, then the drawing routine just needs to start from the next byte, there's no need to send out the initial bytes if they're offscreen. We need to copy the bits in the main area of the table into the buffer bytes listed on top. I've added a new row at the bottom which indicates the first bit/byte of the text buffer (TB). With the duplicated Byte row underneath, you can see how we'll need a part of one text buffer and the rest of the next to fill the gaps.

Back to that forum post I mentioned above, and the description I gave about bit shifting, we can build up our buffer. We need to, based on the scroll offset, take the first (8 - offset) bits from every byte and then the 'offset' bits from the following byte. i.e. if the offset was 3, we want the last 5 bits of byte 1 and the first three of byte 2. If we're on the last byte, then we just fill the end with zeroes. The AND masks for row 2 are shown below. Row 1 is nearly all bits set... so wouldn't be a good test case.

Bytes	00								01
Pixels	00	01	02	03	04	05	06	07	08	09	10	11	12	13	14	15
Row 0	00	00	01	01	01	01	01	01	01	01	01	01	01	01	01	01
Row 1	00	00	00	00	01	00	00	01	00	00	00	00	01	00	00	00
AND #1	00	00	00	01	01	01	01	01	00	00	00	00	00	00	00	00
AND #2	00	00	00	00	00	00	00	00	01	01	01	00	00	00	00	00

And then we just need to shift the first 5 bits to the left... as we're stored in MSB.

#include <SPI.h>
#include "myfont.h"

int ccol = 0;
byte datarow[7][60];
int displayLength = 120;
int bytesRequired = (displayLength + (displayLength % 8) + 8) / 8;
int totalPixels = bytesRequired * 8;
byte frameBuffer[7][16];

unsigned long last_time = millis();
unsigned long scroll_time = millis();
unsigned long scroll_pause = millis();
unsigned long next_text_time = millis();
int charOffset = 0, loop_count = 0, charcol = 0;
String displayString;

void buildTextBuffer() {
  bytesRequired = ((displayString.length() * 5) / 8) + ((displayString.length() * 5) % 8 == 0 ? 0 : 1);
  totalPixels = bytesRequired  * 8;
  for (int row = 0; row < 7; row++) 
  {
    for (int b = 0; b < bytesRequired; b++) datarow[row][b] = 0;
    for (int col = 0; col < totalPixels; col++) {
      ccol = col + 8;
      datarow[row][ccol / 8] |= 
        ((font[displayString[col / 5]][col % 5] & (1 << (6 - row))) 
          ? 1 : 0) << (ccol % 8);
    }
  }
}

void buildFrameBuffer(int offset = 0) {
  int tOffset = offset / 8;
  int cOffset = offset % 8;
  int bb = 0;
  for (int row = 0; row < 7; row++) 
  {
    for (int b = 0; b < 16; b++) {
      bb = b+tOffset;
      frameBuffer[row][b] = (datarow[row][bb] >> cOffset);
      if (cOffset > 0) {
        if (bb <= bytesRequired) {
          frameBuffer[row][b] |= (datarow[row][bb + 1] << (8 - cOffset));
        }
      }
    }
  }
}

void setup() {
  Serial.begin(9600);
        
  for (int p = 0; p <= 12; p++) {
    pinMode(p, OUTPUT);
    digitalWrite(p, LOW);
  }
  
  SPI.begin();
  SPI.setDataMode(SPI_MODE3);
  SPI.setBitOrder(LSBFIRST);
  SPI.setClockDivider(SPI_CLOCK_DIV64);

  displayString = "Warming up ... ";
  buildTextBuffer();
  buildFrameBuffer(0);
}

String newString = "";
void getNextIndex() {
  for (int i = 0; i < 7; i++)
    for (int x = 0; x < 60; x++)
      datarow[i][x] = 0;
      
  if (Serial.available()) {
    newString = Serial.readStringUntil('\n');
    if (!newString.equals(displayString)) {
      displayString = newString;
      next_text_time = millis();
    }
  } else {
    displayString = "";
  }
  if (displayString.length() > 0) buildTextBuffer();
  buildFrameBuffer(0);
}

void loop() {  
  for (int row = 0; row < 7; row++) 
  {
    for (int col = 0; col < 16; col++) SPI.transfer(frameBuffer[row][col]);
    digitalWrite(row + 3, HIGH);
    delayMicroseconds(150);
    digitalWrite(row + 3, LOW);
  }

  if ((millis() - scroll_pause > 2500)) {
    if ((displayString.length() > (displayLength / 5))) {
      if (millis() - scroll_time > 30) {
        charOffset++;
        if ((charOffset / 5) > displayString.length()) {
          charOffset = 0;
          loop_count++;
          scroll_pause = millis();
          if (millis() - next_text_time > 5000) getNextIndex();
        }
        scroll_time = millis();
        buildFrameBuffer(charOffset);
      }
    } else {
      loop_count++;
      if (millis() - next_text_time > 5000) {
        getNextIndex();
        loop_count = 0;
      }
      charOffset = 0;
    }
  }
}

And it all actually worked... beautiful scrolling.

Further Possible Optimisations

We're sending the text into this Arduino from another Arduino over SoftSerial. Why send text when I could send the framebuffer bits instead? I wonder if sending (string-length * 5 * 7 * 8) bits would be quicker than the initial string and calculations on the secondary Arduino's side? Should I test it? Maybe later...

Arduino Uno WiFi + TV Remote + LED Sign

Over the weekend, whilst watching TV, I realised that I needed some kind of sign to work out when trains were passing my apartment. There's a railway line right next to me and around four-or-more services pass daily. I'd prefer to be notified in advance and, since I have data sources to tell me when trains are about to come by, I want to make the most of it. Currently that's done by RSS both on my phone and PC, but the whole point is that I'm watching TV and don't want to have to be checking my phone every 2 seconds.

After getting the red LED marquee to work on the old ISA I/O card, I decided it was time to get it plugged into the Arduino. I actually already have an Arduino Uno Wifi next to my TV, which I've recently been using to translate my TV remote control codes (similar to what I did with the old B&O TV) into Yamaha remote control codes for control of volume with one remote. Adding the LED marquee would let me know when a button was successfully pressed ... and it would also tell me when trains were coming past!

I hadn't actually worked with the WiFi on the Arduino yet, so that was a bit of an experience. The RF side of things was working fine and I had already worked out the wiring requirements for the LED display thanks to the old I/O card.

Getting the ESP8266 Online

The Arduino Uno Wifi is an Arduino Uno board with an embedded ESP8299 microcontroller. The basic idea is that the two units talk serial to each other, when configured to do so. If you want to program one, or the other, you have to adjust the serial communications path on the board via a tiny row of switches. Jaycar has a minimal instruction sheet here, but it's enough to get you started. From the information, please set your Arduino Uno Wifi's tiny switch row to: 1:OFF,2:OFF,3:OFF,4:OFF,5:ON,6:ON,7:ON,8:OFF.

I would want this unit to act by itself as both a web server and web requestor, so I needed to find code for both. Before this we need to set up the Arduino IDE for use with the ESP8266. I found a great tutorial here, but otherwise, the steps are pretty simple. Open the Arduino IDE, go to Preferences and then paste this URL into to the additional boards list: http://arduino.esp8266.com/stable/package_esp8266com_index.json.

Once done, go to the board manager and search for ESP8266...

Install the relevant row and then restart the Arduino IDE. From here, you should have a new selection list in the boards drop-down...

...and then, once selected, a hideous amount of new options to adjust the programming!

The main advice is to not touch any of them, apart from the serial port. You'll need to work out which one your Arduino is connected to, but depending on if you even had serial ports in the first place, it should be pretty obvious. I'm working on a Mac Mini and my Arduino was connected via cu.usbserial-1460. From here, open the Serial Monitor and hit reset on the Arduino.

What garbage is that? I then started from the slowest and tested each baud... turns out if you set to 74880 then you'll get the following...

What a weird port number? It seems that just the initial header information is at that BAUD rate. Afterwards, it'll be at whatever you've set in the code you upload.

Running A Web Server

So, where were we? We have a connected device, so let's grab the first example from here and compile it. After installing the ESP8266 libraries, this code should compile with zero issues. If you happen to have any, then write a comment below, providing as much detail as possible. If it's worked, then check the code and adjust the Wifi SSID and password to match your local AP. Hit upload and cross your fingers and toes... hopefully you'll get the following output:

Yes! Now, you need to switch OFF switch 7 on the little switch header on the Arduino. This pin enables the programming mode and we now want it in the run mode. Note also that we've now set the baud to 115200, so change your serial monitor to that too... Do you get the following after hitting the reset key on the Arduino?

Associated! And with a nice IP! So what does the web browser say?

WIN!

HTTPS Web Requests with the ESP8266

The basic ESP8266 examples use WiFiClient, but this only works with HTTP requests. My server runs over HTTPS, so we need to use WifiSecureClient for secure web requests. To call an HTTPS service, you'll need to get the fingerprint from the SSL certificate. Browse to the site in a regular web browser and check out the properties of the lock next to the URL in the address bar. Somewhere in there you'll find the required data.

And from there, edit the WifiClient Example to carry out an SSL request using these instructions:

#include <esp8266wifi .h>

const char* ssid = "********";
const char* password = "********";
const char* host = "api.github.com";
const char* fingerprint = "CF 05 98 89 CA FF 8E D8 5E 5C E0 C2 E4 F7 E6 C3 C7 50 DD 5C";

void setup()
{
  Serial.begin(115200);
  Serial.println();

  Serial.printf("Connecting to %s ", ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }
  Serial.println(" connected");
}


void loop()
{
  WiFiClientSecure client;
  Serial.print("[connecting to ");
  Serial.println(host);
  if (client.connect(host, 443)) {
    Serial.println("connected]");
    if (client.verify(fingerprint, host)) {
      Serial.println("[certificate matches]");
    } else {
      Serial.println("[certificate doesn't match]");
    }
    Serial.println("[Sending a request]");
    client.print(String("GET /") + " HTTP/1.1\r\n" +
                 "Host: " + host + "\r\n" +
                 "Connection: close\r\n" +
                 "\r\n");
    Serial.println("[Response:]");
    while (client.connected() || client.available())
    {
      if (client.available())
      {
        String line = client.readStringUntil('\n');
        Serial.println(line);
      }
    }
    client.stop();
    Serial.println("\n[Disconnected]");
  }
  else
  {
    Serial.println("connection failed!]");
    client.stop();
  }
  delay(5000);
}

And hopefully, you'll get something similar to the following in your serial terminal.

{ll⸮⸮|⸮d⸮|⸮d⸮b|⸮⸮⸮⸮s⸮b⸮c⸮⸮'o⸮dg'⸮⸮⸮cp⸮⸮dsdrlx⸮o⸮⸮d⸮⸮co⸮|l⸮⸮c⸮⸮go⸮d⸮⸮$`⸮'o$`gs⸮⸮⸮ob⸮$;⸮⸮gc⸮l⸮⸮dl⸮⸮d`⸮⸮'⸮

Connecting ...
scandone
........scandone
.scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 1
cnt 

connected with COMEGETSOME, channel 6
dhcp client start...
ip:192.168.1.121,mask:255.255.255.0,gw:192.168.1.1

Connected to COMEGETSOME
IP address:	192.168.1.121
mDNS responder started

connecting to api.github.com
Using fingerprint '59 74 61 88 13 CA 12 34 15 4D 11 0A C1 7F E6 67 07 69 42 F5'
requesting URL: /repos/esp8266/Arduino/commits/master/status
request sent
headers received
esp8266/Arduino CI has failed
reply was:
==========
{"state":"pending","statuses":[],"sha":"1e17dddd895883806c9bfd3dd7b7042aa813d94f","total_count":0,"repository":{"id":32969220,"node_id":"MDEwOlJlcG9zaXRvcnkzMjk2OTIyMA==","name":"Arduino..."}
==========
closing connection

Nice, we can successfully pull data from a HTTPS source. Let's rig up a polling system to get the data each minute.

Polling for Data + Extra Buttons

So, we're able to pull data from the internet, but I also want this unit to be a mini webserver. If I used Delay() to spread out the web-polling calls, then I'd halt the entire unit. We can't have it wait when it needs to be handling button presses from the web interface. Instead, we'll take a snapshot of the current internal millisecond counter using millis() and then check how many milliseconds have passed to work out if we should try and grab new data.

#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WiFiMulti.h> 
#include <ESP8266mDNS.h>
#include <ESP8266WebServer.h>   // Include the WebServer library

ESP8266WiFiMulti wifiMulti;     // Create an instance of the ESP8266WiFiMulti class, called 'wifiMulti'
ESP8266WebServer server(80);    // Create a webserver object that listens for HTTP request on port 80

void handleRoot();              // function prototypes for HTTP handlers
void handleNotFound();
void handleRemote();
unsigned long lastMillis;
const char* host = "trainfinder.otenko.com";
const int httpsPort = 443;
const char fingerprint[] PROGMEM = "BC 9C 56 D5 8E 7A FE CC 0C F7 A2 21 1F C5 7B 9A C0 FA 15 41";
String last_line = "";

void setup(void) {
  Serial.begin(9600);
  delay(100);
  Serial.println("\n!START");
  wifiMulti.addAP("COMEGETSOME", "blahblahblah");
  while (wifiMulti.run() != WL_CONNECTED) { delay(250); }
  Serial.println(WiFi.SSID());
  Serial.println(WiFi.localIP());

  server.on("/", HTTP_GET, handleRoot);
  server.on("/buttons", HTTP_POST, handleRemote);
  server.onNotFound(handleNotFound);
  server.begin();
  
  lastMillis = millis();
}

void loop(void){
  server.handleClient();
  if ((millis() - lastMillis) > (60 * 1000)) {
    WiFiClientSecure client;
    client.setFingerprint(fingerprint);
  
    if (!client.connect(host, httpsPort)) {
      Serial.println("!FAILED");
    } else {
      String url = "/some/cool/url";    
      client.print(String("GET ") + url + " HTTP/1.1\r\nHost: " + host + "\r\nConnection: close\r\n\r\n");
      while (client.connected()) {
        String line = client.readStringUntil('\n');
        if (line == "\r") break;
      }
      bool foundLastLine = false;
      String line = "";
      while(client.available()) {   
        line = client.readStringUntil('\n');
        if (last_line.equals("") || foundLastLine) {
          Serial.println(line);
          last_line = String(line);
          delay(6500); //space out the data to the Arduino
        }
        if (last_line.equals(line)) {
          foundLastLine = true;
        }
      }
      if (!foundLastLine) last_line = line;
    }
    lastMillis = millis();
  }
}

void handleRoot() {
  server.send(200, "text/html", "<form action='/buttons' method='POST'>" \
      "<input type='submit' name='pwr_plz' value='PWR!'/>" \
      "<input type='submit' name='vol_dn' value='VOL-'/>" \
      "<input type='submit' name='vol_up' value='VOL+'/>" \ 
    "</form><br/><br/>Last Line: <pre>" + last_line + "</pre>");   
}

void handleRemote() {
  if (server.hasArg("vol_up")) {
    Serial.println("!VOLUP");
  } else if (server.hasArg("vol_dn")) {
    Serial.println("!VOLDN");
  } else if (server.hasArg("pwr_plz")) {
    Serial.println("!POWER");
  }
  server.sendHeader("Location","/");
  server.send(303);
}

void handleNotFound(){
  server.send(404, "text/plain", "404: Not found");
}

Just for fun, here's the first shot of me writing the code. Instead of 'text/html', I was still outputting 'text/plain' and, well, as expected, got the following...

After fixing that, the screenshot below was taken. Of course, it doesn't match the code above, but you get the idea.

The basic idea is to print out the raw html to the browser. From here there's a form that has the buttons which post back to the ESP8266. The controller then sees the post, sends the command via Serial to the Arduino and then redirects the user back to the main form.

Now to get the Arduino to consume it.

ESP8266 to Arduino

We've currently had the ESP8266 piped directly out the USB Serial port. Now we want it to talk to the Arduino internally. To do this, the tiny onboard switch needs to have all switches off except for 1 and 2. This then routes the RX/TX directly from the ESP8266 to the Arduino. From here though, the unit becomes a black-box. We have no idea what the ESP8266 is saying to the Arduino, and vice-versa, as the connection via the USB serial port has been severed. We can only hope that the data we're sending through is what we want!

If you check the shots above of the data the ESP8266 sends, you can see that we want to skip a bit of the earlier stuff. To do this, I defined a "!START" command which the Arduino should look for before bothering to deal with any of the data from the serial channel. From there, anything starting with a "!" is a command, whereas anything else is to be printed to the LED Display.

Oh yeah, that TV Remote bit

I previously had an amplifier that ran over ARC over HDMI and also received controls... but half the time it'd switch on to Optical and just would not switch back to ARC without a lot of screwing around. No amount of power cycling, cable switching or button pressing would get it back. So it went to the farm. Instead, I purchased a beautiful old Yamaha amp for AUD$5 and then realised I had to get up and control it by hand. Not optimal, so I search google for the remote codes.... they didn't exist, so I search eBay for a spare remote... they didn't exist... but one guy in Spain had a company that programmed third-party remotes and had one for this amp!

It arrived, and I plugged it through the same code I used for the B&O TV. Over the serial, the codes and bits were reported, showing me that it used the "NEC" protocol. I recorded the codes for the buttons I wanted. I then did the same with my Sony TV Remote for buttons that I wanted to re-purpose. The goal was to have the TV Volume buttons also trigger the amp volume. This worked nicely, via an external IR LED that sits in front of the amp.

Of course, the TV volume still shifts, so I made the amp volume shift 4 times per button press... meaning the surround is always louder than the TV. Doesn't hurt to have a little bit of tinny sound from the TV also... a second center channel?

To make everything compile, install Ken Shirrif's IRremote library, available via the Arduino GUI...

The code is in the next segment.

Hooking up the LED Sign

I've used a font found online, but it seems that Arduino has one built-in? 5x7 font. Either way, the 3 data lines and 7 row-enable pins need to find homes in the Arduino digital IO pins. Make sure you don't use digital pin 0 or 1 as that's the serial channel to the ESP8266. I found this out later below...

Once hooked up, just use a similar loop process as per any LED display: Disable all rows, send out the columns to be lit for the first row, enable that row and quickly disable it. Then do the same for the next 6 rows. As quickly as possible! More information on lighting this sign is over here.

#include <IRremote.h>
#include "myfont.h"
#include "digitalWriteFast.h"

int RECV_PIN = 13;
IRrecv irrecv(RECV_PIN);
decode_results results;
IRsend irsend;

void setup()
{
  Serial.begin(9600);
  irrecv.enableIRIn(); // Start the receiver

  pinModeFast(0, OUTPUT);
  digitalWriteFast(0, LOW);
  pinModeFast(2, OUTPUT);
  digitalWriteFast(2, LOW);
  pinModeFast(12, OUTPUT);
  digitalWriteFast(12, LOW);

//matrix led row drivers
  for (int p = 0; p <= 11; p++) {
    pinModeFast(p, OUTPUT);
    digitalWriteFast(p, LOW);
  }
}

unsigned long last_time = millis();
unsigned long scroll_time = millis();
unsigned long scroll_pause = millis();
bool is_off = true;
int loop_count = 0;
String serialString = " Warming up..";
bool found_start = false;
int charOffset = 0;
int ccol;

void loop() {
  if (irrecv.decode(&results)) {    
    loop_count = 0;
    switch (results.value) {
      case 4841:
      case 0xA90:
        irsend.sendNEC(0xDE21807F, 32);
        if (is_off) {
          serialString = " Power ON!";
          is_off = false;
        } else {
          serialString = " GOOD BYE!";
          is_off = true;
        }
        break;
      case 0x490:
        irsend.sendNEC(0xDE21B04F, 32);
        delay(50);
        irsend.sendNEC(0xDE21B04F, 32);
        delay(50);
        irsend.sendNEC(0xDE21B04F, 32);
        serialString = " VOLUME UP!";
        break;
      case 0xC90:
        irsend.sendNEC(0xDE21708F, 32);
        delay(50);
        irsend.sendNEC(0xDE21708F, 32);
        delay(50);
        irsend.sendNEC(0xDE21708F, 32);
        serialString = " VOLUME DOWN!";
        break;
      case 0xFFFFFFFF:
        irsend.sendNEC(0xFFFFFFFF, 32);
        break;
        //sony teletext red
      case 21225:
        irsend.sendNEC(3726721215, 32);
        serialString = " INPUT!";
        //sony teletext green
      case 13033:
        irsend.sendNEC(3726717135, 32);
        serialString = " MODE UP!";
        break;
        //sony teletext yellow
      case 29417:
        irsend.sendNEC(3726735495, 32);
        serialString = " MODE DOWN!";
        break;
      default:
        Serial.println(results.value);
        Serial.println(results.bits);
        break;
    }
    delay(250);
    irrecv.enableIRIn();
    irrecv.resume(); // Receive the next value
  }

  if (charOffset == 0 && Serial.available() && ((millis() - last_time) > 4000)) {
    last_time = millis();
    serialString = Serial.readStringUntil('\n') + '\0';
    loop_count = 0;
    scroll_pause = millis();
  }
  
  //draw LEDs
  for (int row = 0; row < 7; row++) 
  {
    for (int col = 0; col < 90; col++) {
      if (((col / 5) + charOffset) < 60) {
        ccol = (charOffset * 5) + col;
        digitalWriteFast(2, ((font[serialString[ccol / 5]][ccol % 5] & (1 << (6 - row))) > 0) ? HIGH : LOW);
      } else {
        digitalWriteFast(2, LOW);
      }
      digitalWriteFast(12, HIGH);
      delayMicroseconds(5);
      digitalWriteFast(12, LOW);
    }
    digitalWriteFast(row + 5, HIGH);
    delayMicroseconds(75);
    digitalWriteFast(row + 5, LOW);
  }
  
  if ((millis() - scroll_pause > 2500)) {
    if ((serialString.length() > 18)) {
      if (millis() - scroll_time > 180) {
        charOffset++;
        if (charOffset > serialString.length()) {
          charOffset = 0;
          loop_count++;
          scroll_pause = millis();
          if (loop_count > 10) serialString = "";
        }
        scroll_time = millis();
      }
    } else {
      loop_count++;
      if (loop_count > 100) serialString = "";
      charOffset = 0;
    }
  }
}

Overloading Arduino Pins

Anyone watching above, or in the previous post where I hooked the display up to the old PC, was probably screaming in their seats. It's well known that you should NOT draw too much current through the pins on any microprocessor. As soon as I started hooking up more modules onto this unit, things started going haywire and it quickly dawned on me. My seven line-enable pins were being brought LOW to enable the rows... but these were actually working as the single GND conduit for the entire bloody LED display!

I quickly referred to the previous module that ran this sign and saw there was a 74LS145 binary counter in between the Atmel Microcontroller and the 7 enable rows. A quick review of the datasheet shows that this chip can 'sink' up to 15v and is 'great for driving LEDs'. Right... they were properly routing the GND wires away from the microcontroller and using the 145 as a set of transistors. They even saved 3 data lines in the process! I only had a ULN2003 (which would still use 7 lines) and there were no 74LS145 to be found at Jaycar!

This now directed the current away from the microcontroller... but didn't enhance the brightness.

Speeding up the Refresh Rate

It occurred to me that my refresh rates weren't anywhere near as good as the original controller that the sign contained. My loop is busy checking clocks, serial ports and mucking around, so it was never going to actually go fast enough. The code above could run a single segment OK, but started to scale worse as the bit shifting got longer. One trick, which is actually already implemented in the source above is to use an external library to make the digital pin manipulation faster.

With this implemented, there was less flickering and I found that three segments (90 LEDs per row) looked good. One final note is that you shouldn't really use pins 0 or 1 if you've got the serial enabled. They actually present the TX/RX lines and the serial data and the serial data to the Arduino is visible on them. In this case, it was the serial data to/from the ESP8266. In the case below, I had the three data lines required to drive the display on digital pins 0,1,2...

Shifting those to spare pins higher...

Yessssss! It works really nicely.

Arduino – How to save power (and control 5v+)

For the first time ever, I get to play with Arduino at work! We're doing a little bit of work on tracking and the goal is to acquire as much data as possible and report it to a webserver. Therefore I've got sensors, GPS and GSM. As one can imagine, this consumes a lot of power, and as one can also imagine: the goal will be to obtain as long of battery life as possible.

Each of the modules requires either 3.3v or 5v and these are usually hardwired into the power supply. They also require anywhere from 50mA to 2A (GSM when sending data or searching for signals) and therefore can't be controlled directly from pins on the Arduino. Due to this I'll explain a proper mechanism to switch their power supplies on and off. Also, we'll then want to dig into power saving on the Arduino itself, so stick around.

Controlling modules from Arduino digital pins

Digital pins on the Arduino are only good for 30mA maximum. In fact, you shouldn't even be going anywhere near this. Hence, using one to provide +5v or GND to a module will cause that module's current needs to flow through the pin. For example, the GPS module I bought from Jaycar needs anywhere from 0-70mA. Although low, this is still too high for a digital pin.

The goal therefore is to use an electronic switch. You might be thinking 'relay', but you'll also need to note that they require current which can peak above the digital pin tolerance! You'll find that relay modules don't connect the relay actuator to the input.

The answer here is to use a transistor to allow current to flow to your modules. Depending on your current requirement, I can either recommend a 2N2222 (for anything up to 600mA) or a TIP31 (good for 3A.) I also want to switch my GSM module, and that needs 2A, so I bought a handful of TIP31s.

Using them is very straight-forward. Connect your VCCs up to whatever power source is needed and embed the transistors in the ground path. From here, switching the digital pin high will let current flow and complete the circuit for the module. Make sure you set your digital pinMode to OUTPUT.

Ideas for saving power via code

The basic idea is to put the Arduino to sleep whenever there's nothing to do. My device only needed to report every 15 minutes, so in the meantime I tried to make it snooze. It turns out that, based on internal timers and interrupts, you can only get a maximum of 8 seconds power down out-of-the-box. This wasn't quite the 15 minutes I was looking for, but numerous sources online said to just keep putting the unit to sleep for 8 seconds over-and-over.

There's some great reading here by Nick Gammon on power saving. Anything from lower frequencies to shutting down certain parts of your project. You'll also find a blog post here at the Arduino Playground describing other options. Just make sure you read that entire second article as other contributors have made corrections or other important points. Tis great to see the community involved.

...I'll come back to power saving once the project gets started up again. The proof-of-concept worked and now we need to convince the powers-that-be to let us continue.

Arduino – Don’t buy the SIM900A for Australia!

tl;dr = This is a 2G-only device. There's only one network left, which shuts down on the 30th of June, 2018!

...but I needed a connection for work and I bought this unit from a seller on eBay without researching... so... compounding the error, I then also bought a SIM card for the network that'll still work: Vodafone. Thanks to Happymacer on Instructables who posted an article discussing how to use this device in Oz!

Note that I'd previously tried to use both Optus and Telstra SIM cards and got the following output when running the COPS command.

AT+COPS=?
+COPS: (3,"VODAFONE","voda AU","50503"),,(0-4),(0-2)
OK

Yey! Vodafone! But I'm on Telstra/Optus? So I can see a Vodafone tower, but it's prepended with 3 which actually means I'm not allowed to use it. Sorta makes sense as Optus and Telstra switched off their 2G networks ages ago and Vodafone must have decided that lingering devices on other companies' plans aren't allowed to use their towers.

I bought a Vodafone SIM and ran the commands again...

AT+COPS=?
+COPS: (2,"VODAFONE","voda AU","50503"),,(0-4),(0-2)
OK

Woweee! It's there. 2 means Roaming... which is a little weird, but it must be trying to hint to my device that 2G isn't the best method and that there should be a 3G tower around somewhere. Of course, even if there was, we couldn't connect to it. So I forced the connection with the following command...

AT+COPS=1,2,"50503"
+CREG: 2
+CREG: 1
OK

Those CREGs afterwards were output because I had previously run AT+CREG=1. This command tells the device to report any changes to the carrier connection. What you can see is it connecting to the tower in roaming mode (aka 2) and then connecting correctly in local mode (aka 1.) We're on!

Testing SMS

If you wanna test an SMS, then you can do so pretty easily. Rig up a SoftwareSerial example in Arduino and make sure you can send commands to the SIM900A via the Serial Monitor. From here, you can use the following script to send an SMS.

AT+CMGF=1
OK
AT+CMGS="+61xxxxxxxxx"
>

Note above ... you'll get the > prompt to enter the message. Once you're done, the unit requires a CTRL-Z control character to realise it's the end of the message. To do this, you'll need to open your favourite text editor, type ALT+026 (hold down ALT and type 0 then 2 then 6) and then copy the resulting character. It will be a black SUB if you use Notepad++. Now paste this into the text entry field in the Serial Monitor and hit enter.

Sending Data

From here, Jens Christoffersen at All About Circuits has helped us greatly with his article: Using a SIM900A to Send Sensor Data to a Website. Scroll down far enough and you'll find his code to send data to a server. He uses printf statements, as he's using a PIC and talking direct to the serial port... we'll have to use SoftwareSerial and print(ln).

AT+CGATT=1
OK
AT+SAPBR=1,1
OK
AT+SAPBR=2,1
+SAPBR: 1,1,"100.84.75.204"
OK
AT+HTTPINIT
OK
AT+HTTPPARA="URL","http://www.website.com/index.php?data1=1&data2=4"
OK
AT+HTTPPARA="CID",1
OK
AT+HTTPACTION=0
OK
+HTTPACTION: 0,200,0
AT+HTTPTERM
OK

Above is an example chat with the unit to send data. Paste each AT line into the Serial Monitor first to make sure you get the expected responses. The URL above is rubbish, so put one in that'll work properly instead. The 200 in the HTTPACTION response is the happy HTTP code.

#include <SoftwareSerial.h>
SoftwareSerial mySerial(8,9); // RX, TX

void setup() {
	// set the data rate for the SoftwareSerial port
	mySerial.begin(2400);
}

...

void processCommand() {
	int x = 0;
	switch (serial_buff[0])	{
		case 'T':
			setThrottle(serial_buff[1], serial_buff[2]);
			break;
		case 'D':
			changeDir(serial_buff[1], serial_buff[2]);
			break;
		case 'P':
			//adjust point
			break;	
		case 'S':
			//read sensors and return the data.
			break;
						
	}
	
	for (int x = 0; x < SERIAL_MAX; x++) serial_buff[x] = '\0';
	last_pos = 0;
}

...

void loop() {
	//if there's data and we've not read the end of the current message.
	if (mySerial.available() && (last_pos == 0 || last_char != '!')) {
		last_char = mySerial.read();
		serial_buff[last_pos] = last_char;
		last_pos++;
		if (last_pos > SERIAL_MAX) {
			//then we need to do something drastic
		}
		if (last_char == '!') processCommand();
	}
}

void sendCommand(unsigned char c, unsigned char t, unsigned char v) {
	ser_put(c);
	ser_put(t);
	ser_put(v);
	ser_put(33);
}

...

void main() {
	...
	sendCommand('d', 1, current_direction);
	sendCommand('t', 1, current_throttle);
	...
}

cl65 -t c64 --module -o c64-up2400.ser c64-up2400.s

#include <stdlib.h>
#include <stdio.h>
#include <conio.h>
#include <serial.h>
#define DRIVERNAME  "c64-up2400.ser"

static const struct ser_params serialParams = {
    SER_BAUD_2400,      /* Baudrate */
    SER_BITS_8,         /* Number of data bits */
    SER_STOP_1,         /* Number of stop bits */
    SER_PAR_NONE,       /* Parity setting */
    SER_HS_NONE         /* Type of handshake to use */
};

int main (void)
{
  int xx;
  
  clrscr();
  puts("C64 serial ...");

  // Load driver
  ser_load_driver(DRIVERNAME);

  // Open port
  ser_open(&serialParams);

  // Enable serial
  ser_ioctl(1, NULL);

  for (xx = 0; xx < 10; xx++) {
	ser_put('C'); 
	ser_put('6'); 
	ser_put('4');
	ser_put('.');
	ser_put('.');
	ser_put('.');
  }
 
  return EXIT_SUCCESS;
}

cl65 -t c64 -O -o trainctl2 trainctl2.c

  if (CAN.buffer0DataWaiting()) {
    CAN.readDATA_ff_0(&length,frame_data,&frame_id, &ext, &filter);
    Serial.write("X");
    Serial.write(frame_data[0]);
    Serial.write(frame_data[1]);
    Serial.write(frame_data[2]);
  }
  
  while (Serial.available() >= 9) {
    // read the incoming byte:
    b = Serial.read();
    if (b == 'L') {
      for (int i = 0; i < 8; i++) frame_data[i] = Serial.read();
      frame_id = 0x222;
      CAN.load_ff_0(8, &frame_id, frame_data, false);
    } else if (b == 'D') {
      for (int i = 0; i < 8; i++) frame_data[i] = Serial.read();
      frame_id = 0x666;
      CAN.load_ff_0(8, &frame_id, frame_data, false);
      //printBuf(frame_id, frame_data);
    } else if (b == 'R') {
      for (int i = 0; i < 8; i++) frame_data[i] = Serial.read();
      frame_id = 0x333;
      CAN.load_ff_0(8, &frame_id, frame_data, false);      
    } else if (b == 'P') {
      for (int i = 0; i < 8; i++) frame_data[i] = Serial.read();
      frame_id = 0x777;
      CAN.load_ff_0(8, &frame_id, frame_data, false);            
    }
  }

void loop() {	
  send_data[0] = 0;
  send_data[1] = 0;
  send_data[2] = 0;
  for (sens = 0; sens < 8; sens++) {
    setOutputBit(sens);
    sensor_read = readInput(0);
    if (sensor_read < sensor[sens]) sensor[sens] = sensor_read;
    send_data[0] |= (((20 + sensor[sens])      <= sensor_read) << sens);
    sensor_read = readInput(1);
    if (sensor_read < sensor[sens]) sensor[sens] = sensor_read;
    send_data[1] |= (((20 + sensor[sens + 8])  <= sensor_read) << sens);
    sensor_read = readInput(2);
    if (sensor_read < sensor[sens]) sensor[sens] = sensor_read;
    send_data[2] |= (((20 + sensor[sens + 16]) <= sensor_read) << sens);
  }
  frame_id = 0x111;
  CAN.load_ff_0(3, &frame_id, send_data, false);

  frame_data[0] = 0;

  data1 = CAN.buffer0DataWaiting();
  data2 = CAN.buffer1DataWaiting();

  if (data1 || data2) {
    if (data1) CAN.readDATA_ff_0(&length, frame_data, &frame_id, &ext, &filter);
    if (data2) CAN.readDATA_ff_1(&length, frame_data, &frame_id, &ext, &filter);

    if (frame_id == 0x222) {
      for (int i = 0; i < 8; i++) lcl.setRow(0, i, frame_data[i]);
    } else if (frame_id == 0x333) {
      for (sens = 0; sens < 8; sens++) {
        setOutputBit(sens);
        sensor[sens] = readInput(0);
        sensor[sens + 8] = readInput(1);
        sensor[sens + 16] = readInput(2);
      }   
    }
 }
 delay(100);
}

    using System.IO.Ports;
    SerialPort _serialPort;

    private void SetupComPort() {
        _serialPort = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
        _serialPort.Handshake = Handshake.None;
        _serialPort.DataReceived += new SerialDataReceivedEventHandler(HandleData);
        _serialPort.Open();
    }

    private void cb_CheckedChanged(object sender, EventArgs e)
    {
        Byte[] bytesToSend = new Byte[9];
        bytesToSend[0] = (byte)'L';
        for (int i = 0; i < 8; i++) {
            for (int z = 0; z < 8; z++) bytesToSend[i+1] |= (byte)((LEDs[(i*8) + z].Checked ? 1 : 0) << z);
        }
        _serialPort.Write(bytesToSend, 0, 9);
    }

    private void UpdateSpeed()
    {
        Byte[] bytesToSend = new Byte[9];
        bytesToSend[0] = (byte)'D';
        bytesToSend[1] = (byte)(direction ? 1 : 0);
        bytesToSend[2] = (byte)hSpeed.Value;
        bytesToSend[3] = (byte)points[0]; //for loop? probably should.
        bytesToSend[4] = (byte)points[1];
        bytesToSend[5] = (byte)points[2];
        bytesToSend[6] = (byte)points[3];
        bytesToSend[7] = (byte)points[4];
        bytesToSend[8] = (byte)points[5];
        _serialPort.Write(bytesToSend, 0, 9);
        Console.WriteLine("Sent: D" + bytesToSend[1] + "-" + bytesToSend[2]);
    }