“Fetching or posting data to the internet is one of the core tasks of an IoT device. Doing so over HTTP is implemented quite well in the default ESP8266 Arduino libraries, but for HTTPS requests things are more difficult. In this post I will discuss the most common approaches used by the community, and develop my own method to do arbitrary HTTPS requests in a secure way. This method will not require any specific certificates or fingerprints to be manually coded in the application.
HTTPS if a method to do a HTTP request over a TLS (formerly SSL) connection. By doing so, the data that is sent back and forth between your computer and the server is encrypted and protected. The good news is that this protocol can be used with the ESP8266 with the WiFiClientSecure class. The bad news is that the common methods to do so have some big disadvantages.
First I will show the two most common approaches, and next I will describe a generic solution to their problems.
A word about certificates
Before diving into the details I will briefly explain the basic principles of secure HTTPS requests in layman’s terms.
Basically, every website has a certificate. This certificate is issued by somebody, who is called a certification authority (CA). Each CA certificate can be issued by another CA which leads to the so called certificate chain. In the picture the chain is 3 certificates long, but in reality the length of the chain can be anything.
The top level certificate is called a root certificate. This certificate is self-signed, which means that it can be trusted inherently. This is because only a few organisations can issue root certificates, and these are trusted to not offer fake or wrong certificates.
When you do a HTTPS request to a website from your browser, the browser will look at the certificate for the website, and validating if the certificate is indeed issued by its parent. This can be done because each certificate is signed with the private key of the upstream certificate. An explanation for dummies on public and private keys work can be found here .
When it is verified that the certificate is indeed issued by a trusted root CA issuer, it is verified that the domain in the certicate is the same as the actual domain. If that is true, we know that the server is who it claims to be, and a secure connection can be started 👍.
These trusted root certificates are actually stored as part of your browser to be able to validate all other certificates. Each OS or browser stores a slightly different set of roughly 100-200 root certificates, which it knows can be trusted. This is called a certificate store, and this is exactly what I will apply on the ESP8266 later in this article. But first, let’s start with the two most popular other approaches.
Fingerprints - secure but annoying
The method that is proposed in the official ESP8266 Arduino documentation is to extract the fingerprint of a site’s certificate and store this in the code. The fingerprint is a hash of the certificate. Because it is very unlikely that a second certificate exists with the same hash, we know the website can be trusted if the hash is the same as the one we store.”