Your own Cloud-IoT DIY project. Part 1: Introduction.

5 min readAug 20, 2023

This story is about personal experience with DIY IoT-Cloud-Web-AI which hopefully would be helpful for other IoT enthusiasts. I want to have a system where I’ll be able to easily add newly created IoT device, have insights to device data from phone/laptop, have an opportunity to run some AI workflows on collected data, send commands to devices, etc. At the same time I want my system secure and my data not shared with some unknown parties.

Imagine you’ve created a DIY sensor or a system of sensors that you want to bring together in the cloud. For example, you made a self-watering system for your plants, with all the moisture detectors, watering systems etc. How do you operate it remotely? How do you collect the data in the cloud, and pass in commands based on that data? And what if you do not want to share it with third-parties or want to add some fancy cloud features?
While there are out of the box solutions on the Internet, they are either not secure (have access to your sensors), too expensive, or are limited and closed.

I’ve started to write the description of the system created and decisions made but quickly found that it’s too long for one post. So this will be a series as well as sources available in GitHub repo by request. For now it includes (not all of the stories are completed yet):

Your own Cloud-IoT DIY project. Part 1: Introduction.
Your own Cloud-IoT DIY project. Part 2: On Premises.
Your own Cloud-IoT DIY project. Part 3: Device-Cloud Communications.
Your own Cloud-IoT DIY project. Part 4: Cloud backend.
Your own Cloud-IoT DIY project. Part 5: Helper tools and project config.
Your own Cloud-IoT DIY project. Part 6: Data usage and visualization.
Your own Cloud-IoT DIY project. Part 7: AI processes and plugins.
Your own Cloud-IoT DIY project. Part 8: Complete step-by-step demo.
Your own Cloud-IoT DIY project. Part 9: Conclusion and thoughts.

While there are multiple freemium and even free solutions available to handle your IoT devices in the cloud sometimes it’s good to have your own one. This not only adds flexibility but also gives you an understanding how your data is used and who have an access to your IoT devices.

So the objective of this article is to demonstrate a solution which will be:

enterprise level from functionality perspective
easily manageable (DIY!) but also extensible
secure enough to be safely used for DIY projects
(notes for improving security to enterprise level to be available)
inexpensive (DIY!)

In simple words: I want the newly developed device to be securely and easily (within minutes) connected to cloud and information from that device available (securely) on my phone/laptop. I also want an opportunity to easily play with multiple cloud features (AI, analytics, etc.) and data collected from devices. And I’m not ready to pay more than $10/month.

System overview

System components

Obviously any IoT-Cloud solution consists of two big subparts — device and cloud plus some extra tools to be used for device provision, control, etc.

However this is over-simplified view because multiple technical details are not included into it. Some of the details are not that important while others are. In the next parts we’ll cover most of the details for every part and provide links to code which may be directly reused in your projects or used with minimal modification.

Cloud side

In the cloud part we’ll need:

solution for data ingestion (with proper AutheNtication and AuthoriZation — AuthN/Z) and transformation
solution for data storage
solution for secure access to collected data
solution for secure communication back from your UI to Cloud to device
multiple APIs for data access, device management, etc with proper AuthN/Z
preferably a UI to review your data easily but securely
preferably an opportunity to improve with some AI functionality, ability to send alerts to phones, etc.

NOTE: for enterprise-level solution there would be more requirements (reliability, data recovery, etc.) and some requirements will be more strict or complicated (security, AuthN/Z, etc.)

We’ll be using AWS as a cloud provider (Azure has very similar functionality available) and AWS CDK for infrastructure definition.
For the sake of simplicity (DIY) we’ll be using one AWS account and one “production” environment (no extra environments for development, QA, etc.). However more complex CICD could be added to the solution.

Device side

For this part we’ll need:

hardware design (targeting DIY we need to make design simple, available for enthuziasts, inexpensive yet flexible)
mechanical design(targeting modern DIY we should either rely on 3D-printed or widely available mechanical components)
firmware running on the device with comfortable provisioning

NOTE: for enterprise-level solution device-side design could be considerably different due to mass-production requirements

We’ll be using ESP32 (and specifically SparkFun Thing Plus — ESP32 WROOM (USB-C)) as microcontroller of choice for the purpose of this article. However most of design and implementation is portable to other popular DIY platforms.
For the sake of simplicity our IoT device will talk directly to the cloud (no IoT gateway will be used).