I think these are Dancing:

I think these are Fighting:

The plots are interesting because there are many different ways that people dance or fight and our brains can usually see these, but can you describe the general difference.

I thought no but given the unreasonable effectiveness of neural networks to recognize patterns, maybe they could figure out what that difference was.

These plots show how the cases differ for those labelled by me as dancing and separately, to the right, as fighting. There really is nothing obvious here.

The spreadsheet generates stick figure couples with 10 dots per figure and I’ve laboriously picked out and labelled 600 cases where they appear to me to be either fighting or dancing. The spreadsheet contains two on-sheet fully connected neural networks with 2 layers of 15 and 10 neurons ending in a sigmoid binary classifier and looking at 24 XY coordinates in a flat 48-dimensional vector (there are 2 repeated dots; the neck and the pelvis – wasteful but I was lazy). The first Neural Net runs 5 batches of 100 cases each and learns the difference between Fighting and Dancing and the second runs stochastic gradient descent on a single case at a time using the learned parameters of the first but looking at a separate test dataset of 100 fresh, unseen labelled cases.

Excel lends itself well to building and understanding Neural Networks because it requires an explicit physical layout for the layers within the model, they are not hidden in virtual arrays defined by for-loops but displayed as actual blocks of numbers. The plot below shows the core of the model’s “brain”, the hidden layers and the parameter weights that, once trained, capture the function that we humans each develope to distinguish between aggression and joy. The Momentum and Gradient layers give you a view of how the spreadsheet learns. The gradients of the function are derived directly from a case (or 100 cases for this vectorized model) run through the current learned parameters. The gradients give the direction of travel required to update the parameters of each layer in order to reduce the loss function i.e. the difference between my classification of a particular case and the spreadsheet’s current understanding. If left to train for an hour or so with a small enough learning rate, the model will learn the training cases to an accuracy of 100%. However, this would be an example of overfitting and would not capture the essence of fighting or dancing and therefore not generalize well to new cases.

Excel has some real advantages for coding neural nets, beyond the fact that for me it’s the tool I know and I’m still Python semi-literate. I think these advantages are as follows:

• They lay the structure bare. All layers, all neurons and all connection weights are exposed, no searching or displaying the results of nested For loops, just 2D blocks of numbers.
• When coding you make fundamental wiring errors: This is not so helpful if you know where you’re going but at this stage, neural networks are still a black art. In all the training I’ve done to date the experts use the phrase “this works well but we don’t know why”.
• It’s slow to train and you see it train in minute step by step detail. This gives you time to think and ponder what’s going on, what’s wrong and what’s just plain peculiar.

If you’re interested in this, please let me know and I would be happy to share more with you. I’d certainly welcome any questions and queries you may have and will be releasing some of the base models on GitHub. Any expressions of interest prior to this would also be welcome.

I’ve uploaded a Youtube Video of the model running here: https://youtu.be/RsQp-5feqZQ and you can see more related Neural Network in Excel stuff on my channel here: https://www.youtube.com/

This videos and associated material if released are available to you under a Creative Commons Attribution-NonCommercial 4.0 International License the details of which can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode

The post A Neural Network in Excel learning the difference between fighting and dancing. appeared first on NN in XL.

I have been deeply involved with financial analysis and mathematical models for most of my career but began re-tooling with machine learning over the last two years.  I’m currently working through Andrew Ng’s brilliant Deep Learning course on Coursera and I’ve reached Course 4, Week 3. Throughout the course, I’ve been building out the neural net architectures he describes in Excel.  Excel is not “yet” the right medium to build convolutional neural nets for real-world applications. However, I know Excel and find it easier to construct these on a spreadsheet rather than in a new language. I certainly hope this state won’t last long.

Building in Excel may be slow, but it makes neural nets very transparent, in that you are visually confronted with the layers, neurons and their associated parameters and related calculations. Excel gives a less abstract view of a neural net than vectorised Python code and it helped me immensely in developing an understanding of these fantastic new tools.

Excel lets you see inside the machine but for me, an even more useful element was the bugs and miss wiring that I introduced. Neural networks are very robust to bugs, in fact, they often continue to learn but fail in odd and interesting ways. Figuring out the failure based on the learning behaviour really forced me to think about the maths and the structure. I would also argue that the speed of Excel gives you time to think as the failures manifest themselves. For a deeper understanding of Neural Nets go to Andrew Ng. My intuitions are still weak but certainly improving.

The classic beginners exercise for deep learning is to build an MNIST digit classifier. This relates to Yann LeCun’s data set of 60,000 handwritten digits (0 to 9) with an associated 10,000-digit test set. Yann has made this data available to all and there are plenty of higher level language examples.

My initial attempts at classifiers using plain vanilla fully connected neural networks are on my YouTube channel below but Mike Pallister was faster off the mark than me and is also worth a look.

Single Digit Classification https://www.youtube.com/watch?v=4P5r0tT7Hsc&t=33s

Batch Digit Classification https://www.youtube.com/watch?v=bJcv9vi4Gqg&t=6s

However, this blog relates to last week’s modelling of a convolutional neural net; again with ADAM optimisation and Batch capability. After several silly but informative mistakes, I completed the model below last night and would welcome your thoughts.

The model has very few layers, neurons and parameters, its tiny in comparison to real-world examples. By way of a full description, there are two convolutional layers with max pooling taking the images of the handwritten digits from 28h x 28w pixels to 24h x 24w x 4c (4 channels) and 12h x 12w x 4c after max pooling. Layer 2 condensed these to 8h x 8w x 8c and then 4h x 4w x 8c after max pooling. The final two fully connected layers had 15 and 10 neurons respectively. Giving a grand total of 936 convolutional parameters and 2,095 in the fully connected layers. This doesn’t sound like a lot of space to capture the vagaries of human handwriting, but it does. On its first 100k iterations I’m seeing 98.75% accuracy on the training data and 98% on the test data.  I would love to know what human accuracy levels on this data are but from my experience, it’s not much more than this.

I’d certainly welcome any questions and queries you may have and will be releasing some of the base models on GitHub. Any expressions of interest prior to this would also be welcome.

These videos and associated material if released are available to you under a Creative Commons Attribution-NonCommercial 4.0 International Licence the details of which can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode

The post Building Convolutional Neural Networks in Excel appeared first on NN in XL.