Embedded Developers, System Architects and Product Managers, who plan to deploy an AI/ML model for an application in one or more of the following constrained edge device types: 64-bit Linux based Single Board Computers (SBCs) or 32-bit microcontrollers with or without Neural Network Accelerators. The course is also useful to get an understanding of transformer-based SOTA models such as vision language model (VLMs) and open-vocabulary segmentation models.  

Please note that the course does not delve into details of different types of neural network architecture.  These details are covered in the  Practical Deep Learning course.

• The foundation of a modern MLOps workflow

• Neural Network Basics – Model training, loss functions and APIs of Keras training framework

• Data planning and pre-processing techniques for deep learning training and inference

• Architectures of Fully Connected and Convolutional Neural Networks (CNNs) models

• Applying Transfer Learning to customize models to application specific datasets

• Classifying time series data using CNNs

• Converting models between different formats (ONNX, TFLite) including explainable classical ML algorithms

• Performing model inference using different runtimes and Python / C++ API’s

• Porting a trained model to a microcontroller using TinyML (LiteRT for Microcontrollers, aka TFLite Micro)

• Compiling a model to a neural network accelerator

• Performing Object Detection using Single Shot Detection (SSD) and YOLO (You Only Look Once) models

• Use of Transformer-based LLMs such as vision language model (VLM), automatic speech recognition (ASR) and open-vocabulary segmentation models

• Hosting AI models on an IoT Device and providing inference using MQTT or REST APIs

• Deploying AI models using Docker container

• Capstone project to implement an end-to-end edge AI application

Attendees should be familiar with the basic idea of an embedded AI application. Specifically, you should have:

• Working knowledge of a programming language like Python or C/C++.
• Some background knowledge about edge devices, such as Linux based single board computers (x64 or ARM), Neural Network Accelerators or 32-bit Microcontrollers (such as a Cortex-M4 based)

Please contact Doulos directly to discuss and assess your specific experience against the pre-requisites.

Doulos training materials are renowned for being the most comprehensive and user-friendly available. Their style, content and coverage are unique in the training world and have made them sought after resources. The materials include:

• Fully indexed class notes creating a complete reference manual
• Jupyter Notebooks containing complete working code for all the neural networks presented during the training class, which you can use after the class for revision or as the basis for your own networks.
• Source code on GitHub for exercises using Linux SBC, Neural Network Accelerator and Microcontroller.
• Supplementary material on Keras APIs, OpenCV, embedded build systems, and State-of-the-art edge AI models

Introduction

Supervised learning • Inference engines at the edge • Edge AI components • Edge AI applications Practicals:  Try out model training environments based on CPU and GPU

Neural Network Basics

Model training • Loss functions • Sequential and Functional API use and examples • Frequently used Keras APIs

Data Planning & Pre-processing

CRISP-DM Methodology & MLOps for Edge AI • Use case of audio feature pre-processing • Split audio file for training •

Conversion of audio segments to Spectrogram • Training data preparation

Practicals:  Use SoX (Sound eXchange) software to trim, filter, and plot Spectrograms • View Spectrograms before training •

Organize spectrograms into folders for training • Work with a subset of Environment Sound Classification (ESC) dataset

Creating Neural Network (NN) Models for Edge

Review of different kinds of neural networks • Convolutional Neural Network (CNN) using 2D and 1D convolution • Transfer Learning • Chaining NN models

Practicals:  Code (using Keras) simple Convolutional neural networks using small datasets • Write code to use pretrained models (such as MobileNet) for transfer learning

Edge AI Hardware

Constrained Inference platforms • Inference server •  Linux based SBC • 32 bit Microcontroller • NN Accelerator • GPU • FPGA

Practicals:  Take picture, play and record audio using SBC • Read sensor data using serial port from SBC and Microcontroller • Structure accelerometer data for supervised model training

Model Formats

Model Formats • Open Neural Network Exchange Format (ONNX)   •  TensorFlow Lite (TFLite) • Viewing Model Graph • Model Format Conversion • Model Quantization

Practicals:  Train and convert Scikit-Learn (ML) model to ONNX • Train and convert Keras Models to ONNX and TFLite

Model Inference

Inferencing steps • Input and Output Tensor Shape • ONNX runtime, TFLite Interpreter Python methods • TFLite C++ Classes Practicals: Convert Scikit-learn model to ONNX format.  Perform Inference of NN model using ONNX/TFLite formats in Python environment, Inference of TFLite and ONNX models using C++ classes

TinyML

TinyML Implementation frameworks • emlearn • TFLite Micro • Converting Keras Model • Setting up and using TFLite Micro Interpreter
Practicals:  Using emlearn • Quantize Fully Connected TFLite Model • Convert Model to C array for storing in MPU • Read sensor value and execute model using TinyML (TFLite Micro) on Cortex-M4 device

Neural Network Accelerator

Neural Network Accelerator platforms • Model compiler and workload partitioning • Executing Model on Accelerator • Working with multiple accelerators

Practicals:  Quantize and compile CNN Model for NN Accelerator • Run compiled model on accelerator and compare execution time with SBC

Object Detection

MobileNet architecture •Object Detection using MobileNet based SSD • Object Detector Output • Decoding and boxing detected output • Object Detection using You Only Look Once (YOLO) model

Practicals: Use object detection models to locate objects of interest. Employ post-processing on model output for NonMaximum Suppression (NMS)

Embeddings and Transformers

Word and audio embeddings • YAMNet model • Cosine similarity • Static and contextual embedding • Transformer architecture • Vision Transformer • Data requirements for Transformer based models

Practicals:  Determine cosine similarity of image and audio embeddings, compare CNN and Transformer based computer vision models

Transformer based SOTA Models

Contrastive Language-Image Pre-training (CLIP) •DEtection TRansformer (DETR) models • Segment Any Thing model (SAM) •Whisper / Moonshine Automatic Speech Recognition (ASR) models • Multi-Modal AI model – Gemma 3

Practicals: Perform inference on mobile / nano version of SOTA models • Retrain multi-modal model

Monitor Model Inference and Performance

Share Model Inference using MQTT and REST API • Monitor Model Drift 

Practicals: Communicate model inference output using MQTT and Flask webserver

Deploy Model to Edge Device

Enumerate Model Inference Dependencies • Create Dockerfile for Model Inference Container • Use Docker Compose • Push Model Inference Container to Edge Device • Use of Zephyr based device management services for TinyML devices

Practicals: Create ONNX/TFLite inference Dockerfiles • Use Dockerfile to create Docker container and test it on Edge device

Edge AI Capstone Project

Plan an end-to-end Edge AI application • Apply MLOps methodology to create an Edge AI solution • Two sample projects provided for reference

Supplementary content:

Appendix A. OpenCV

Classical computer vision algorithms • OpenCV pre-processing and post-processing functions • AI Model inferencing using

OpenCV

Appendix B. Build System

C++ application compilation • Use of CMake to build application • Use of precompiled shared object (.so) files of model runtime and OpenCV

Appendix C. Time Series

Terminology – univariate, multivariate, regression, classification • Pre-processing using Pandas • Time Series as supervised learning problem • Keras Time Series Generator API • Time Series classification using CNN • Time Series Database

Practicals: Set up time series dataset as supervised learning task • Perform CNN based fingerprint analysis of sections of time series data