Readme.md
1# Speech Recognition Example
2
3## Introduction
4This is a sample code showing automatic speech recognition using Arm NN public C++ API. The compiled application can take
5
6 * an audio file
7
8as input and produce
9 * recognised text to the console
10
11as output
12
13## Dependencies
14
15This example utilises `libsndfile`, `libasound` and `libsamplerate` libraries to capture the raw audio data from file, and to re-sample to the expected
16sample rate. Top level inference API is provided by Arm NN library.
17
18### Arm NN
19
20Speech Recognition example build system does not trigger Arm NN compilation. Thus, before building the application,
21please ensure that Arm NN libraries and header files are available on your build platform.
22The application executable binary dynamically links with the following Arm NN libraries:
23* libarmnn.so
24* libarmnnTfLiteParser.so
25
26The build script searches for available Arm NN libraries in the following order:
271. Inside custom user directory specified by ARMNN_LIB_DIR cmake option.
282. Inside the current Arm NN repository, assuming that Arm NN was built following [these instructions](../../BuildGuideCrossCompilation.md).
293. Inside default locations for system libraries, assuming Arm NN was installed from deb packages.
30
31Arm NN header files will be searched in parent directory of found libraries files under `include` directory, i.e.
32libraries found in `/usr/lib` or `/usr/lib64` and header files in `/usr/include` (or `${ARMNN_LIB_DIR}/include`).
33
34Please see [find_armnn.cmake](./cmake/find_armnn.cmake) for implementation details.
35
36## Building
37There is one flow for building this application:
38* native build on a host platform
39
40### Build Options
41* ARMNN_LIB_DIR - point to the custom location of the Arm NN libs and headers.
42* BUILD_UNIT_TESTS - set to `1` to build tests. Additionally to the main application, `speech-recognition-example-tests`
43unit tests executable will be created.
44
45### Native Build
46To build this application on a host platform, firstly ensure that required dependencies are installed:
47For example, for raspberry PI:
48```commandline
49sudo apt-get update
50sudo apt-get -yq install libsndfile1-dev
51sudo apt-get -yq install libasound2-dev
52sudo apt-get -yq install libsamplerate-dev
53```
54
55To build demo application, create a build directory:
56```commandline
57mkdir build
58cd build
59```
60If you have already installed Arm NN and and the required libraries:
61
62Inside build directory, run cmake and make commands:
63```commandline
64cmake ..
65make
66```
67This will build the following in bin directory:
68* `speech-recognition-example` - application executable
69
70If you have custom Arm NN location, use `ARMNN_LIB_DIR` options:
71```commandline
72cmake -DARMNN_LIB_DIR=/path/to/armnn ..
73make
74```
75## Executing
76
77Once the application executable is built, it can be executed with the following options:
78* --audio-file-path: Path to the audio file to run speech recognition on **[REQUIRED]**
79* --model-file-path: Path to the Speech Recognition model to use **[REQUIRED]**
80
81* --preferred-backends: Takes the preferred backends in preference order, separated by comma.
82 For example: `CpuAcc,GpuAcc,CpuRef`. Accepted options: [`CpuAcc`, `CpuRef`, `GpuAcc`].
83 Defaults to `CpuRef` **[OPTIONAL]**
84
85### Speech Recognition on a supplied audio file
86
87To run speech recognition on a supplied audio file and output the result to console:
88```commandline
89./speech-recognition-example --audio-file-path /path/to/audio/file --model-file-path /path/to/model/file
90```
91---
92
93# Application Overview
94This section provides a walkthrough of the application, explaining in detail the steps:
951. Initialisation
96 1. Reading from Audio Source
972. Creating a Network
98 1. Creating Parser and Importing Graph
99 3. Optimizing Graph for Compute Device
100 4. Creating Input and Output Binding Information
1013. Speech Recognition pipeline
102 1. Pre-processing the Captured Audio
103 2. Making Input and Output Tensors
104 3. Executing Inference
105 4. Postprocessing
106 5. Decoding and Processing Inference Output
107
108### Initialisation
109
110##### Reading from Audio Source
111After parsing user arguments, the chosen audio file is loaded into an AudioCapture object.
112We use [`AudioCapture`](./include/AudioCapture.hpp) in our main function to capture appropriately sized audio blocks from the source using the
113`Next()` function.
114
115The `AudioCapture` object also re-samples the audio input to a desired sample rate, and sets the number of channels used to one channel (i.e `mono`)
116
117### Creating a Network
118
119All operations with Arm NN and networks are encapsulated in [`ArmnnNetworkExecutor`](./include/ArmnnNetworkExecutor.hpp)
120class.
121
122##### Creating Parser and Importing Graph
123The first step with Arm NN SDK is to import a graph from file by using the appropriate parser.
124
125The Arm NN SDK provides parsers for reading graphs from a variety of model formats. In our application we specifically
126focus on `.tflite, .pb, .onnx` models.
127
128Based on the extension of the provided model file, the corresponding parser is created and the network file loaded with
129`CreateNetworkFromBinaryFile()` method. The parser will handle the creation of the underlying Arm NN graph.
130
131Current example accepts tflite format model files, we use `ITfLiteParser`:
132```c++
133#include "armnnTfLiteParser/ITfLiteParser.hpp"
134
135armnnTfLiteParser::ITfLiteParserPtr parser = armnnTfLiteParser::ITfLiteParser::Create();
136armnn::INetworkPtr network = parser->CreateNetworkFromBinaryFile(modelPath.c_str());
137```
138
139##### Optimizing Graph for Compute Device
140Arm NN supports optimized execution on multiple CPU and GPU devices. Prior to executing a graph, we must select the
141appropriate device context. We do this by creating a runtime context with default options with `IRuntime()`.
142
143For example:
144```c++
145#include "armnn/ArmNN.hpp"
146
147auto runtime = armnn::IRuntime::Create(armnn::IRuntime::CreationOptions());
148```
149
150We can optimize the imported graph by specifying a list of backends in order of preference and implement
151backend-specific optimizations. The backends are identified by a string unique to the backend,
152for example `CpuAcc, GpuAcc, CpuRef`.
153
154For example:
155```c++
156std::vector<armnn::BackendId> backends{"CpuAcc", "GpuAcc", "CpuRef"};
157```
158
159Internally and transparently, Arm NN splits the graph into subgraph based on backends, it calls a optimize subgraphs
160function on each of them and, if possible, substitutes the corresponding subgraph in the original graph with
161its optimized version.
162
163Using the `Optimize()` function we optimize the graph for inference and load the optimized network onto the compute
164device with `LoadNetwork()`. This function creates the backend-specific workloads
165for the layers and a backend specific workload factory which is called to create the workloads.
166
167For example:
168```c++
169armnn::IOptimizedNetworkPtr optNet = Optimize(*network,
170 backends,
171 m_Runtime->GetDeviceSpec(),
172 armnn::OptimizerOptions());
173std::string errorMessage;
174runtime->LoadNetwork(0, std::move(optNet), errorMessage));
175std::cerr << errorMessage << std::endl;
176```
177
178##### Creating Input and Output Binding Information
179Parsers can also be used to extract the input information for the network. By calling `GetSubgraphInputTensorNames`
180we extract all the input names and, with `GetNetworkInputBindingInfo`, bind the input points of the graph.
181For example:
182```c++
183std::vector<std::string> inputNames = parser->GetSubgraphInputTensorNames(0);
184auto inputBindingInfo = parser->GetNetworkInputBindingInfo(0, inputNames[0]);
185```
186The input binding information contains all the essential information about the input. It is a tuple consisting of
187integer identifiers for bindable layers (inputs, outputs) and the tensor info (data type, quantization information,
188number of dimensions, total number of elements).
189
190Similarly, we can get the output binding information for an output layer by using the parser to retrieve output
191tensor names and calling `GetNetworkOutputBindingInfo()`.
192
193### Speech Recognition pipeline
194
195The speech recognition pipeline has 3 steps to perform, data pre-processing, run inference and decode inference results
196in the post-processing step.
197
198See [`SpeechRecognitionPipeline`](include/SpeechRecognitionPipeline.hpp) for more details.
199
200#### Pre-processing the Audio Input
201Each frame captured from source is read and stored by the AudioCapture object.
202It's `Next()` function provides us with the correctly positioned window of data, sized appropriately for the given model, to pre-process before inference.
203
204```c++
205std::vector<float> audioBlock = capture.Next();
206...
207std::vector<int8_t> preprocessedData = asrPipeline->PreProcessing<float, int8_t>(audioBlock, preprocessor);
208```
209
210The `MFCC` class is then used to extract the Mel-frequency Cepstral Coefficients (MFCCs, [see Wikipedia](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum)) from each stored audio frame in the provided window of audio, to be used as features for the network. MFCCs are the result of computing the dot product of the Discrete Cosine Transform (DCT) Matrix and the log of the Mel energy.
211
212After all the MFCCs needed for an inference have been extracted from the audio data, we convolve them with 1-dimensional Savitzky-Golay filters to compute the first and second MFCC derivatives with respect to time. The MFCCs and the derivatives are concatenated to make the input tensor for the model
213
214
215#### Executing Inference
216```c++
217common::InferenceResults results;
218...
219asrPipeline->Inference<int8_t>(preprocessedData, results);
220```
221Inference step will call `ArmnnNetworkExecutor::Run` method that will prepare input tensors and execute inference.
222A compute device performs inference for the loaded network using the `EnqueueWorkload()` function of the runtime context.
223For example:
224```c++
225//const void* inputData = ...;
226//outputTensors were pre-allocated before
227
228armnn::InputTensors inputTensors = {{ inputBindingInfo.first,armnn::ConstTensor(inputBindingInfo.second, inputData)}};
229runtime->EnqueueWorkload(0, inputTensors, outputTensors);
230```
231We allocate memory for output data once and map it to output tensor objects. After successful inference, we read data
232from the pre-allocated output data buffer. See [`ArmnnNetworkExecutor::ArmnnNetworkExecutor`](./src/ArmnnNetworkExecutor.cpp)
233and [`ArmnnNetworkExecutor::Run`](./src/ArmnnNetworkExecutor.cpp) for more details.
234
235#### Postprocessing
236
237##### Decoding and Processing Inference Output
238The output from the inference must be decoded to obtain the recognised characters from the speech.
239A simple greedy decoder classifies the results by taking the highest element of the output as a key for the labels dictionary.
240The value returned is a character which is appended to a list, and the list is filtered to remove unwanted characters.
241
242```c++
243asrPipeline->PostProcessing<int8_t>(results, isFirstWindow, !capture.HasNext(), currentRContext);
244```
245The produced string is displayed on the console.