Your own Cloud-IoT DIY project. Part 5: Helper tools and project config.
This is the fifth part of “Your own Cloud-IoT DIY project”. Previous part can be found here.
TL;DR: Globally available configuration file is used as a source of truth. Multiple UI and command-line tools available for project bootstrapping, update deployments, MQTT testing, etc.
Project configuration
Our project consists of multiple cross-related parts — UI should know the URL of the backend API and IdP deployed; Firmware should know the URL of the MQTT endpoint, topics, MTLS API; Cloud Backend has a bunch of configuration parameters, etc.
There are multiple options to support configuration of such complicated projects. One can use naming convention to be sure that correct URLs is used. But this makes the project configuration fragile as one error can result in hard-to-find bug. Another option is to have ‘service discovery’ component which will be source of truth for any configuration parameter. This options is extremely powerful but require creation and management of extra service and extra component for every part of the project. This probably will be a good choice for enterprise solutions but may be an overkill for DIY.
We’ll be using an approach of global project configuration file and bootstrapping tool which will create/update every part of the project. We’ll be also using some naming convention component but its participation will be minimal.
Global project configuration file
Our global configuration file will serve as a source of truth for any component and will be updated during bootstrapping and backend deployment. We’ll rely on the requirement of particular project folders organization (if you clone the repo you’ll have exactly that structure):
As you can see every component of the project resides in its own subfolder and project configuration (project_config.json) is in the root accessible by every component and especially by bootstrapping tool (bootstrap.py) located in the Cloud_IoT_DIY_tools folder:
The structure of mentioned project_config.json seems complicated but in general it’s just set of configuration blocks for different project processes and components with some descriptions included. This file is source of truth for all components however some of them will have it’s own parts created/updated with data from it (like src/_projectData.hpp file in the firmware section and lib/_projectData.dart file in UI section).
While you can edit this file manually it’s highly recommended to use bootstrapping tool which will guide you through the configuration process, create/manage additional configuration filse and even collects some data for you from the cloud.
Project bootstrapping tool
While deployment of the cloud backend is literally executed with one command, it can be done only when the project is properly configured.
Additional complications come from the fact that project components (cloud-firmware-ui) are cross-dependent from configuration perspective.
So the overall project configuration is stored in the root project folder in the file project_config.json Data from this configuration file will be used for cloud backend deployment and cloud code. However, this is not enough as firmware/UI cannot use that config and some data cannot be placed into that config because it’ll be collected after the cloud deployment.
So in general the overall project provisioning process is:
As you can see this process looks is not that simple and error-prone (copy-pasting always is). So to simplify it project provisioning (bootstrapping) tool was created which will simplify and automated the process. Tool bootstrap.py is available in the Cloud_IoT_DIY_tools folder and became a mini-project itself. It provides a UI with cloud calls and backend deployment executed right from it. UI is based on PySimpleGUI with some MVC design on top of it. Example screenshots:
So in general — if your laptop is configured (prerequisites installed) the overall project bootstrapping will be done from the tool with graphical UI.
When the project correctly configured you can use just plane cdk deploy to update your project when/if needed (i.e. you don’t need to go thorugh the whole configuration process every time).
Bootstrapping tool internals
To support complicated workflows (like bootstrapping and device provisioning) mini-framewrok was created with design like this:
Note that device provisioning UI-enabled tool described below have similar internals.
Command line cloud backend tools
When working with cloud backend coded with CDK you typically use cdk cli (like cdk synth or cdk deploy). However this is not enough when you have complicated project with lambda functions and lambda layers (you’ll need to build/package them before the deployment), conditional predeployment scripts (some limited number of the cloud components can be scripted only), conditional post destroying scripts, etc.
So instead of using cdk cli we’ll be using custom tool named build.py for build, deploy and destroy the cloud backend:
Simple replacement command will be python lbuild.py --deploy instead of cdk deploy the result will be the same but all lambdas and layers code will be packaged for deployment. And you can still use plain cdk deploy if you just need to deploy some infrastructure update.
This lbuild.py will obviously rely on aws cli but also will use some other custom components including pre_deploy.py, _prepare_bootstrap.py and _remove_bootstraped.py. That tools can be also used separately.
Note that if you’ll use project bootstrapping tool described above the initial deployment of cloud backend (with help of lbuild.py) executed for you automatically!
MQTT helper tool
One of the challenge when creating complex IoT solutions is debugging MQTT communications and flows. There are multiple tools that can help you on the networking layer but application layer is typically unique for application and networking is not as convenient.
To test and debug our MQTT flows and especially ‘fleet provisioning’ special tool was created. This is in general just a flexible MQTT tests with UI and some specific flows coded.
In general, this tool gives you an ability to execute ‘fleet provisioning’ steps and see the results in UI (and in terminal logs).
This tool proves its usefulness when uncovering issues like ‘client_id’ is required, aws topics names are not accurate across documentation, etc.
This tool also helps to debug events flow (rules and lambdas execution) especially when used in combination with IoT Core logs in CloudWatch (configured with CDK to ‘WARN’ level by default but can be switched to ‘DEBUG’ in code or with console).
Tool is written on Python and relies on paho-mqtt and PySimpleGUI.
Device provisioning tool
In most cases you can provision your thing without this tool. However, there are situations when it can be if great help:
- You don’t want to use ‘fleet provisioning’ for your device (maybe you don’t want to share claim certificate/keys)
- Or you don’t want to code your thing ‘default’ name/type/location
- And you don’t want to use your phone to fins out Thing IP address and browser to provide required details (including certificate and keys)
If you’re Ok with using browser for configuration — you can skip this part.
So general idea of the tool to simplify first start (when device connected to WiFi but not provisioned yet).
Two options (independent tools) are available:
- Command-line — you just run Python script from the tools folder with options
python autoprov.py --ip <IPaddress> --name <thingName> --type <thingType> --building <buildingId> --location <locationId>(expecting that claim certificate and keys are available in the default location). This is simplest scenario and Thing will be ready for fleet provisioning as soon as script execution completed. You can also always request tool help by executingpython autorpov.py --help.
- UI-enabled tool — with UI you’ll be able to either provide ‘fleet provisioning’ options OR provision the device from the tool and transfer device cert/keys to the Thing (claim cert/keys are not shared with device in that scenario).
Just runpython prov.pyand follow the UI. Note that for smooth experience you need to be logged in with AWS —aws configto be executed earlier (this condition fulfilled automatically if you’ve used project bootstrapping tool described above).
Note that while Serial communication looks very attractive (reliable, secure, etc.) it has known issues including: wired connection to laptop is needed, serial connection is dependent on OS drivers (which may not be stable), serial connection may be noisy (signal/noise ratio not good) and unstable, establishing serial connection may (and in most cases will) require your device restart.
Comparing to Serial the IP connection is much more stable. The only current disadvantage is insecure communication (plain text http instead of https or other encrypted communication protocol) which will be addressed in the next versions.
There are two tabs (options) for Thing configuration/provisioning:
- FLEET is pushing claim cert/keys to the Thing. Further provisioning will happen directly from the Thing. This is the same functionality as command-line tool described above
- With THING option the device will be provisioned by the tool and collected Thing cert/keys will be pushed to the device
In both cases you’ll be able to see logs (different logs for different communication channel) right in the tool section.
