A simple and basic tutorial of tiny-dnn

Ever wondered how to do deep learning in C++? Ever want to do machine learning on one of the most used language that is not a script? tiny-dnn is your best solution.

I’ll assume that you are familiar with the basic concepts in a neural network and the (modern) C++ programming language.  Also you should be able to build and install library from source. I can’t teach how to do that here. Everyone’s setup is a bit different. It will take me forever to list all the possible methods of installing tiny-dnn.

Send me a mail or comment bellow if you need help with anything.

Solving the xor problem in tiny-dnn

To start with , let’s solve one of the most basic problem in Neural Networks – the xor problem.

The problem is simple. You have the input {{0,0}, {0,1}, {1,0}, {1,1}} that have to be mapped to the output {{0}, {1}, {1}, {0}}. Effectively making the neural network act like an XOR gate. Hence the name.  Let’s start coding.

First, we include the header and the namespaces.

#include "tiny_dnn/tiny_dnn.h"
using namespace tiny_dnn;
using namespace tiny_dnn::activation;
using namespace tiny_dnn::layers;

then construct and Neural Network

network<sequential> net;
net << fully_connected_layer(2,3) << sigmoid_layer()
  << fully_connected_layer(3,1) << sigmoid_layer();

Conveniently, all the weights are randomly initialized for us.

Then prepare the dataset.

std::vector<vec_t> trainIn = {{0,0}, {0,1}, {1,0}, {1,1}};
std::vector<vec_t> trainOut = {{0}, {1}, {1}, {0}};

Finally we can train our network with the gradient_descent optimizer (learning rate set to 0.53) set the loss function to MSE, batch size of 1 for 1000 epochs.

gradient_descent optimizer(0.53f);
net.fit<mse>(optimizer, trainIn, trainOut, 1, 1000);

Now .save the network for future use.

net.save("net"); //Load the network with net.load("net")

or use the trained network by calling  the predict method.

vec_t result = net.predict({1,1});
std::cout << result[0] << std::endl;

Something like 0.959534 should show up. That’s it. We just got an XOR network trained in tiny-dnn and saved it!

Full source code is available here:
https://gist.github.com/marty1885/dd648a1806348bf4cd2c2fd0feafae36

Note that you need a C++14 capable compiler to build any tiny-dnn code. Also you must link against the threading library on your system because tiny-dnn utilizes multi thread. For example to build the above code on clang: clang++ tiny-dnn-xor.cpp -O tiny-dnn-xor -std=c++14 -O3 -pthread.

Building an autoencoder

Lets do something more advanced with tiny-dnn. How about build an autoencoder to encode  raw audio signals. I’ll use adamstark’s AudioFile library to load and write WAV files.

It’s not useful but good enough for demonstration purpose.

We start by declaring our some parameters we need later on and a RNG to create training samples out of.

const int SAMPLE_NUM = 1024;
const int WINDOW_SIZE = 2048;
const int BATCH_SIZE = 256;
const int NUM_EPOCHS = 64;

vector<vec_t> samples(SAMPLE_NUM, vec_t(WINDOW_SIZE));
mt19937 rng;

Then load a audio file and create training samples.

AudioFile<float> audioFile;
audioFile.load("a.wav");
auto& data = audioFile.samples[0];
uniform_int_distribution<int> dist(0, data.size()-WINDOW_SIZE);

for(auto& v : samples)
{
  int offset = dist(rng);
  auto start = data.begin() + offset;
  copy(start, start+WINDOW_SIZE, v.begin());
}

Then create an autoencoder. Tran it on the dataset we just created and save it,

network<sequential> net;
net << fully_connected_layer(WINDOW_SIZE, 512) << tanh_layer()
  << fully_connected_layer(512, WINDOW_SIZE);

adagrad optimizer;
net.fit<absolute>(optimizer, samples, samples
  , BATCH_SIZE, NUM_EPOCHS);
net.save("net");

Last, we can use the network to generated encoded and decoded audio (code not shown). The resulting audio won’t sound great. But it sure sounds like the original audio. It works!

Full source source with some addition is available here:
https://gist.github.com/marty1885/4b4a0cfa1bec0eb5fc99cbafcfb00716

Advanced usage of tiny-dnn

Build flags/ configuration macros

There are several  configuration macros in tiny-dnn that you can put before the header. Utilizing them proerly can make your neural network run and train faster.

Most flags start with a CNN_ prefix.

  1. Parallelization flags. CNN_USE_OMP, CNN_USE_TBB and CNN_SINGLE_THREAD.
    • Selects what parallel framework tiny-dnn uses. If not specified, then C++ std::thread is used.
    • CNN_SINGLE_THREADdisables parallel computing.
  2. Backend flags. Ex: CNN_USE_AVX, CNN_USE_SSE, CNN_USE_AVX2, USE_OPENCL and CNN_USE_NNPACK, etc…
    • Enables the corresponding computing backend and set it as default. Otherwise a “internal” backend (written in C/C++ only) is used to ensure compatibility on most  computer.
    • Use CNN_USE_AVX if your are running on a modern x86. It is the best backend so far. The OpenCL backend is there but can’t do any training.

Optimizers

tiny-dnn provides a list of optimizers. Including gradient_descent, momentum, nesterov_momentum, adagrad, RMSProp, adamax and adam.

All hyper parameters are adjustable from their constructor. Have fun experimenting with them.

Accessing layers and weights

Assuming you have constructed a neural network called net. net[n] will let you access the N th layer in the network. layer->layer_type() will return a string that show what kind of the layers is. Ex:

network<sequential> net;
net << fully_connected(2,1) << tanh_layer();
for(int i=0;i<net.depth();i++)
  std::cout << net[i]->layer_type() << std::endl;

weights() returns a vector of flatten weights in the form of vec_t. For example, calling weights() of a fully connected layer will give you 2 set of weights. The first is the actual weight matrix, the other is the bias.

network<sequential> net;
net << fully_connected(2,2);
std::vector<vec_t*> weights = net[0]->weights();
std::cout << weights.size() << std::endl; //Prints 2

Other methods includes set_trainable(bool) sets if the layer’s weight is freezed. trainable() returns if the layer is trainable now.

tiny-dnn documents how to do more with it,
http://tiny-dnn.readthedocs.io/en/latest/how_tos/How-Tos.html

Advertisements

One thought on “A simple and basic tutorial of tiny-dnn

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Powered by WordPress.com.

Up ↑

%d bloggers like this: