“Train an artificial neural network directly on the Wio terminal with AIfES® to recognize simple gestures.
In this tutorial we would like to show you how to implement a simple AI based gesture recognition with AIfES® on the Wio Terminal. The special thing here is that everything is implemented directly on the Wio Terminal without the need for a PC. Three different gestures can be trained. As AI an artificial neural network (ANN) is used, which can be trained with the Adam optimizer in AIfES® directly on the device.
The internal 3-axis digital accelerometer (LIS3DHTR) is used for the gesture recognition. No additional hardware is needed.
The shown gesture recognition is also available for various Arduino boards. In the AIfES® for Arduino library you can find these in the examples.
For the following boards an example is available:
- Arduino Nano 33 IoT
- Arduino Nano BLE Sense
- Arduino Nano RP2040 Connect
Of course, other boards with an integrated or connected accelerometer will work as well. Just adapt the examples and get started.
More info about AIfES®:
- AIfES for Arduino - GitHub
- Arduino IDE
- Install AIfES® (search for aifes) with the Arduino library manager
- Install the Wio Terminal Board Library
- Install the “Grove - 3-Axis Digital Accelerometer ±2g to 16g (LIS3DHTR)” with the Arduino Library Manager (search for LIS3DHTR)
- Inspiration for this tutorial
- This tutorial was inspired by the work of Eloquent Arduino. After talking to him, the data / feature recording was taken from his project. He used support vector machines for the classification and trained them on the PC. We used an ANN, which we can train directly on the device using AIfES®.
- Here you can find his complete project, in which also the basics are explained nicely. You can find the code here.
- What kind of gestures can be trained?
Various gestures can be trained, such as:
- Movement to one side
A minimum acceleration must be reached for the data acquisition to start.
Data / features
If there is little or no movement, no data will be recorded. An acceleration threshold must be exceeded in order to record measured values.
A data / feature set consists of the acceleration values of the 3 axes (X, Y, Z). In the current configuration, 20 X, Y, Z acceleration samples are recorded one after the other and stored as one data set together with a label. Thus, a total of 60 inputs are passed to the neural network. The label is generated automatically by the training process.
The labels are created as follows:
- Gesture 1: 1, 0, 0
- Gesture 2: 0, 1, 0
- Gesture 3: 0, 0, 1
A data set for e.g. gesture 1 looks as follows:
X1, Y1, Z1, X2, Y2, Z2, …, X20, Y20, Z20, 1, 0, 0
This way of data acquisition is of course not a real feature extraction. We have avoided more complex pre-processing such as FFT etc. in order to better demonstrate the principle.
The artificial neural network
The ANN has a total of three layers and consists of the following structure:
- Input layer 60 Inputs
- Hidden layer 4 Neurons (sigmoid activation function)
- Output layer 3 Outputs (softmax activation function)
How did we find the ANN structure? In the concrete example, we did the development with AIfES® directly on the device. You can of course also use AIfES® on the PC or a Python framework like Keras, TensorFlow or PyTorch to find the optimal network structure. In this case you would then send the training data via UART to the PC and perform the training there. If you train in a Python framework, you can extract the weights afterwards and import them into AIfES®.
For the implementation we used AIfES-Express. The implementation is much easier and the ANN structure can be adapted with a few changes. If you want to change e.g. the number of neurons, activation function or number of layers, this is done quickly. Of course you can also increase the number of gestures, but then you have also to increase the number of neurons in the hidden layer. AIfES-Express is easier to use but since the model is built at runtime, it is slower than the normal AIfES® implementation. An advantage of AIfES-Express is that the ANN can be changed at runtime.”