Your own Cloud-IoT DIY project. Part 2: On Premises.
This is the second part of “Your own Cloud-IoT DIY project”. First part can be found here.
TL;DR: We’ll be using SparkFun Thing Plus — ESP32 WROOM (USB-C), I2C sensors, 3D printed cases, VSCode with PlatformIO extension, convenient and semi-automated provisioning process with BLE and local Web Server functionality. Code for tools, case and firmware is available on GitHub.
Hardware
Before discussing decisions made for the hardware, let’s emphasize again — project objective is to create a framework for DIY ! Professional IoT hardware obviously can and probably will have different requirements.
Therefore we’ll limit all hardware decisions to
- Widely available inexpensive microcontrollers, sensors, power supply, and other components
- Simple yet powerful IDE for the firmware development
- Widely available or DIY cases
Microcontroller choice
We’ll use SparkFun Thing Plus — ESP32 WROOM (USB-C) as it provides multiple features (WiFI, BLE, microSD card, etc.) and convenient interfaces including QWIIC. Moreover
- Sparkfun Thing family includes other microcontrollers so if you’ll want to play with
- Thing dimensions are the same as very popular Adafruit feather family so you’ll be able to easily add even more components to your project
Current cost of Sparkfun Thing is about $25 which is kind of expensive but it’s worth it. But if it’s too much you can switch to one of Adafruit ESP32 Wroom (about $20) or any other microcontroller with minimal changes in the code (downshifting can be complicated if one of dependencies will be incompatible, for example WiFiClientSecure is not available on ESP8266).
Sensors choice
You can use any QWIIC sensors if don’t want to bother with soldering. Otherwise you can use any I2C, SPI or any other sensor of your as ESP32 offers really large number of interfaces.
For the purpose of this article we’ll be using temperature sensor and environmental sensor on two different things.
Power supply choice
Things provides really large number of options. On the dev stage we’ll be powering our things from laptop USB or wall USB power adapter. On the last stage we’ll be using LiPo battery on one thing (charger is a part of the Thing) and 9v battery with buck adapter on another one.
IDE choice
For the sake of simplicity we’ll be using Arduino environment and libraries but with VSCode IDE. PlatformIO has a really good (and very popular) VSCode extension which makes developing for ES32 really comfortable even for not very experienced users. Details on “How To start with firmware” are available in the README.md file in the firmware section of repository.
There are a lot of materials on PlatformIO extension usage and “HowTo”.
Device case
While it might seem not important, the case plays crucial role if you want your thing to ever leave your table.
We’ll be using 3D printed parametrized stackable boxes as case. This approach makes possible combining different microcontrollers, sensors, power supply into one (maybe not ideal) but acceptably looking box.
Boxes are designed in the OpenSCAD (free and developer friendly 3D modeler) with sources and STL files available in the hardware section of the repository.
Note that our microcontrollers will be in the deep sleep most of the time so heat dissipation is not a big deal. However, if device will need to run more time, some update to the case may be required (like extra ventilation holes).
Device provisioning
In general, IoT Cloud provision consists of three steps:
- Connect your device to Internet
- Connect your device to Cloud backend
- Perform some additional configuration with data received from the backend
Connecting to Internet maybe not be applicable if your device uses Ethernet. However in most cases you’ll be using WiFi. And hardcoding your WiFi connection details in sources is not a good idea (that’s not secure and not flexible). Instead it’s better to provide WiFi details during provisioning and write it into the EEPROM or other non-volatile memory.
Our Provisioning process will be:
Connect your device to Internet
There are two main options — use WPS (not all routers and devices supports that) or use some communication channel (BLE, WiFi access point, serial, etc.).
As we are using ESP32 in this example, we’ll be using BLE connection for this initial step (code can easily be refactored to support other options).
So the firmware logic for the first promotion step will be:
- on boot we first check if button was pressed to force devices reconfiguration (if yes — wifi data will be cleaned from nvs)
- check if we don’t have WiFi connection info available in nvs
- start initial config step if needed which will start BLE Server, wait for the required info, write them down to nvs and stop BLE Server
- continue with connecting to WiFi using info from nvs
NOTE: for better security it’ll be better to use Blufi for secured WiFi credentials transfer. For now WiFi details are just transferred as open strings over BLE.
It’ll be nice to have dedicated mobile app for device configuration over BLE but for now we’ll be just using BLE Scanner. This is maybe not as beautiful but free and functional for our purposes.
Connect your device to Cloud backend
Device has to know your backend parameters (URLs, protocols, etc.) and to have some credentials to authenticate itself with the backend.
Backend parameters typically are not a sensitive information (it’s expected that you backend public but secured) so they can be hardcoded. However this makes your firmware less flexible. So we need a way to provide that parameters during the promotion. Other options will be:
- collect your backend parameters from the dedicated static API endpoint (not much different from the hardcoding)
- configure locally using web-interface locally available in the device (increase firmware size)
- add configuration to some detachable memory like SD card (limits devices choice)
Device credentials for the backend
It doesn’t matter what particular communication protocol (http, MQTT, etc.) and authentication protocol (OAuth, X509, etc) will be used — you need some security artifact on your device for authentication.
One of the most popular authentication standard for IoT devices is X509 certificates. You can find some benefits description here.
So we need somehow to deliver (at minimum) certificate (32Kb max) and private key to our device. And the requirements are similar to WiFi credentials so hardcoding certificate in source code is not an option.
For the purpose of this article we’ll be using Option 2 which requires more memory but offers a lot of flexibility including more than just a configuration form to that server.
So the firmware logic for the second promotion step (WiFi is already connected) will be:
- on boot check if we don’t have cloud backend info available in nvs OR button was pressed when boot
- start background config step if needed which will start BLE Server with information about IP address and Web Server with configuration html form, wait for the required info and write it down to nvm. Note that you’ll be able to access that web-server from your local network using any standard browser and device IP address which you can collect from BLE or find using any network scanner
- wait until all required cloud backend info is provided and write it down to nvm
- try to connect to backend and perform “provision by claim” (see below)
- on success — stop servers and proceed with boot
- on failure — clear info in the nvm and show the configuration form again
Important NOTE about this process: with “provision by claim” steps 1–3 can be almost skipped if “bootstrap cert/key” and “bootstrap MQTT endpoint” hardcoded as a part of firmware. While looks less secure this is one of the main method for production devices. In our case this can be achieved by just adding bootstrap certificate into the nvs (see Tools section and configuration over Serial) and it’ll be collected automatically. You’ll still need the to name your device (and provide other custom attributes) but certificate and key will be already on your device.
Provision device on the cloud side
Each device has to have a unique certificate to identify itself but backend also should know what certificates are legitimate (allowed). So in our scenario cloud backend will create a certificate for the new device and add it to the registry. But newly created certificate need to be delivered to the device then. We have two option:
- Collect newly created certificate and put it to the device with the help of some additional application. This process is typically named “Provisioning via a Trusted User”.
- Let the device connect to the backend with some intermediate credentials (typically referred as “bootstrap certificate”) and collect its new certificate by itself. This workflow is typically named “Provisioning by Claim”.
We’ll be using Option 2 with “bootstrap certificate” uploaded to device during the provisioning process (however trusted user flow is also available with provisioning tool discussed in further section).
There is a good AWS article describing the Provisioning by Claim workflow in details.
So in our scenario when device presents web server:
- we provide bootstrap cert (generated during project deployment) and “bootstrapping” MQTT endpoint (topic) as well as other thing data (like name, description and other thing attributes)
- device uses bootstrap cert to connect to backend with information about his specific configuration (including thing data model) and collect his “own” certificate as well as other communication details (MQTT topics for different planes). This process described in details in the next part (cloud<->device communications).
- device store all received information in the nvs
Provisioning tool
While it’s possible to upload certificate, key and other cloud backend parameters, the provisioning tool prov.py is a much better option (described in the Part 5). This tool (written in Python) not only pushes required data to the device but also performs some other useful operations.
Note that we’ve loosen security requirements again by transferring sensitive data (bootstrap certificate, etc.) to local server over plain http. While this maybe unacceptable for enterprise-level systems (despite the fact that bootstrapping certificate is often hardcode on device), it seems an acceptable simplification for DIY (at least for the first version). We can upgrade our system to use TLS but it’ll cost additional memory on the device and complications for provisioning subsystem.
Tools
Provisioning is one of the areas where tools can be extremely helpful.
- Convenient provisioning tool will be discussed in the Cloud part.
- Root certificates package is already included to the sources (which effectively enables https requests to any URL). However you may want to change that package (size issue) or to include some root certificates. Tool is available in the tools folder of the firmware section with usage details in the readme file.
- Clear NVS tool. This is not a tool but more of a code to clear up nvs of your thing. This is really important if you are changing NVS namespaces and/or keys. You need to clean up to avoid the NVS being filled with previous attempts’ garbage. The code is available in the tools folder of the firmware section with usage details in the readme file.
Firmware overview
Dependencies
One of the objective for this project was to limit number of dependencies.
So for now (in addition to base Espressif Arduino-ESP32) you’ll only need to add these libraries with PlatformIO Libraries Manager (configuration instructions are available in the firmware repo README.md):
- ESPAsyncWebServer-esphome (used to collect backend data)
This will automatically add some other dependencies — AsyncTCP-esphome and ESPAsyncTCP-esphome (for ESP8266 only)
AsyncWebServer is used to render backend configuration html form on the server root and support API endpoint — for html form response and for provisioning tool data upload.
- PubSubClient (MQTT client is used for all MQTT communications in conjunction with standard arduino-esp32 WiFiClientSecure for TCP connections with MTLS)
- ArduinoJSON (used for serializing and deserializing MQTT messages)
Software Design
It’s out of scope for this story to describe the firmware internal design in details. Anyone interested can take a look into the code. But it seems useful to provide some top-level software design diagram:
One important point is Thing declaration. Each Thing has
- limited set of properties defining the Thing which is used for initial thing provisioning. This set includes name, group, type, building and location ids
- model with definition of thing details. The model includes multiple Thing attributes, information about supported commands/status messages types and information about data points available
Important difference about this two part of the Thing declaration is that limited set of properties defines when/if thing should be provisioned/re-provisioned, but the model is populated from the Thing to the cloud to update current Thing info.
Another important point is how you define your own Thing. Typically you define functions to be executed as a response to command received, function used for data collection and your own thing class inherited from class TheThing. See code documentations, examples and comments for more details.