Fig: These GIFs stride back and forth over all 8 dimensions of the linear manifold of various digits while holding the other dimensions constant. They are from a Capsule Net built on-sheet in Excel which learns the linear manifold of the 10 MNIST handwritten digits and then uses these for categorisation of new Digits.

Since a turning point in 2012, neural networks, have become dominant in the field of machine learning and Artificial intelligence (AI). They are so named because they loosely model the structure of neurons in the brain. Nowadays, they pretty much form the default approach for computer vision, translation, speech recognition etc. Convolutional Neural Networks (ConvNets), are a sub-class of neural networks and our most powerful tool for image analysis but only now, after 20 years of incremental improvements and optimization. Capsule Networks (CapsNets) are new, different and significant. In their first incarnation in 2017, they hit or beat state of the art performance benchmarks in several areas. I, for one, think they represent the next step in humanity’s march toward Artificial General Intelligence (AGI).

CapsNets appear to behave more like the brain than ConvNets. Their inventor Geoff Hinton talks about these characteristics in this presentation https://youtu.be/rTawFwUvnLE , given shortly after he released the paper in late 2017.  Mimicking the human brain is a promising route to our understanding and developing a theory of intelligence. An analogy is the development of a theory of aerodynamics by initially studying birds. The current fruits of this initial approach are aircraft that can circle the earth in 6 hours or carry 500 passengers across the Atlantic. With a similar trajectory, we can only wonder what a theory of intelligence will yield.

Over the last two years I’ve been trying to master and implement Neural Nets in my work. To help me get up to speed, I’ve been building in Microsoft Excel. This is slow but gives me a different and intuitive way to see how they work, and given that some of these neural networks were considered almost magical and certainly state-of-the-art quite recently, to build in Excel is quite demystifying. I’ve built several relatively large neural Networks and posted quite a few online. This batch run ConvNet with Adam optimization hits recent benchmarks for recognizing human handwriting  https://youtu.be/OP7wi2MoSeM and should give you a flavour.

CapsNets are exciting and look to me like a massive development in AI that brings us closer to understanding how the human brain works or at least some of the key maths that play a part in human learning and intelligence. I say this because they learn, fail and succeed in far more human ways than do ConvNets.

• Visual twists, shifts, squeezes and expansions of objects (affine transforms) put a CapsNet off the scent far less than a ConvNet. Think of those verification “captchas” that websites present you with to prove you’re human. We are far better at recognizing distorted digit captures than ConvNets. For a ConvNet to do the same it would have had to be trained on some similar distortion before whereas a CapsNet can extrapolate along those transformations more easily. This also makes them far better at recognizing 3D objects from different viewpoints than ConvNets.
• Humans learn patterns that represent canonical objects with very few examples or rather instructions.  You don’t need to subject a child to 60,000 examples of handwritten digits before they know the numbers 0 to 9. CapsNets have been trained with as few as 25 interventions. We still need to show them a load of handwritten characters but not actually tell them what these are. The advantages they offer for unsupervised learning are already phenomenal.
• CapsNets effectively take bets on what they are seeing and seek information to confirm this. This is what the core routing by agreement algorithm does. As proof builds up they instantaneously prune densely connected layers to sparsely connected layers linking lower level features to specific higher-level features. This is analogous to our making an assumption on what we see, imposing a reference frame and seeking information to fill in the rest. This only becomes apparent when we get it wrong as we generally do with trick images like this shadow face: https://www.youtube.com/watch?v=sKa0eaKsdA0
• CapsNets suffer from the human visual problem known as crowding. This is where too many examples of the same object occur closely together and simply confuse our mind, an example being – how hard it is to count the separate lines here IIIIIIII- I can’t do this as easily as reading the word seven ????.

CapsNets are effectively a vectorized version of ConvNets. In ConvNets, each neuron in a layer gives the probability of the presence of a feature defined by its kernel. CapsNets do more, they convey not only the presence but the “pose” of the feature. By pose we could mean the scale, skew, rotation, viewpoint etc. The routing by agreement algorithm in a CapsNet assesses the match between the ”pose” of a lower level features and the features in the level above it, say bits of digits to a whole digit or components of a face to a whole face. When these agree e.g. the eyes, mouth and nose elements all correspond to a face of size X looking left, we get an indication that the higher-level feature is present. When the pose of many lower level features match a single higher-level feature we can be very certain that a higher level feature exists. If this is hard to digest have a look at the Hinton or Géron videos I reference later. These may help.

This blog post is about building a full capsule net in Excel for handwritten digit classification using MNIST data. MNIST is a data set of hand written digits provided by Yann LeCun and a staple in data science. Given the novelty of this new algorithm there is not so much information available on the net but the best I came across was Geoff Hinton’s original paper https://arxiv.org/pdf/1710.09829v2.pdf and Aurélien Géron’s Keras code and associated videos https://youtu.be/pPN8d0E3900 . I also found this talk by Dr Charles Martin helpful https://youtu.be/YqazfBLLV4U .

——- The remainder of this blog post improves, but is probably only of interest to nerds and insomniacs. ——-

CapsNets are exciting and potentially far more powerful than standard convolutional neural nets because:

• They don’t lose information via subsampling or max pooling which is the ConvNet way to introduce some invariance, CapsNets weights encode viewpoint-invariant knowledge and are equivariant.
• Through the above approach, they know the “pose” of parts and the whole which allows them to extrapolate their understanding of geometric relationships to radically new viewpoints (equivariance),
• They have built-in knowledge of the relationship of parts to a whole.
• They contain the notion of entities and those GIFs on the top of the blog represent movement along one dimension of the linear manifold of the top level entity that represents an eight.

The Hinton & Géron resources were superb for the forward model i.e. the algorithm that identifies categories based on trained parameters which is the true innovation of the CapsNet. However, information on the backward model, the mechanism by which it learns, was sparse and scattered. This is not surprising because back-propagation of the gradient of the loss function with respect to the parameters of each layer is mathematically so straightforward that the deep learning frameworks of TensorFlow and Keras do this automatically. However, wiring the chain rule backwards through the twists and turns of an Excel based CapsNet architecture was a challenge. This was largely because instead of reading through the theory first, I guessed, fiddled and played, until in exasperation, I looked harder for the “right” approach. I certainly learned a lot about how not to do it and it’s possible that I turned up some novel ideas, but above all, when I eventually “got it right’, the theory sunk in and meant a lot more to me.

How a capsule net works

I’ve uploaded a video walkthrough of the Excel model here: https://youtu.be/4uiFJZjw6fU It’s probably not for the casual reader but is a more visual way to see what’s happening and also covers a lot of the issues I’ve written about in this blog.

A big difference between CapsNets and standard neural networks is that CapsNets contain the notion of entities with pose parameters i.e. the network identifies component parts (lower level capsules) and determines if their pose parameters match those of the higher-level capsules where these parts are combined. Capsules require multiple dimensions to convey their pose and the diagram below shows where the additional dimensions appear:

FThe CapsNet I built is like the structure in Hinton’s paper but quite a bit smaller with 5×5 kernels for 4 & 8 channels in the first two layers and 8-dimensional Digit Capsules. This gives only 100 x 8D primary capsules and 10 x 8D digit capsules. The results for this are still impressive i.e. 98.7% accuracy rather than the 99.5% accuracy that we see with 1152 x 8D primary and 10 x 16D digit capsules in the paper. I chose this reduction after paring down the Keras model to a size that would be manageable in Excel without too much Excel build optimization. The structure of my forward CapsNet, or rather a screenshot of the actual CapsNet as it appears in my Excel spreadsheet, is below.

The backpropagation or learning mechanism is much bigger. In the figure below, I’ve put together several screen-shots of the entire spreadsheet model. This covers 1000 rows and 7500 columns. The bulk of the area relates to the decoder sections with their 784 neurons in the final layer and the Adam optimization I used to get the learning speed up. I’ve highlighted the big blue collection of layer 3 transform matrices on this to give you an indication of the size w.r.t. the above forward model, additional calculations, and complexity required for the backward pass.

The Process of Building.

Now I know what I’m doing I could probably mechanically build this in a couple of days or modify the size of a layer in an hour or so. However, if I built it in Keras it would take a couple of hours and to modify a layer would take seconds. Now I understand what I’m building but the initial build took several months and even understanding Aurélien Géron’s Keras code took days. 

My initial approach was to build only the forward model and feed this with pre-trained parameters from a modified version of Aurélien Géron’s code. My reduced spec (L1: 5x5K, 4c, L2: 5×5, 8c s2 L3: 8Dx10caps, Decoder 50, 50, 784) took the parameter count down from the 44,489,744 of the original paper that Aurélien had replicated to a more Excel manageable 127,943. Keras trials on this gave an MNIST test result of 98.69%, higher that I could regularly obtain with my Excel ConvNets but way below the 99.43% that the bigger model achieves.

Another important modification I made to get a clear comparison was to initially train and test the reduced size Keras model only on the 10k MNIST data set. This reached a 100% overfit after about 34 epochs. What I mean by this is that the model was able to learn the 10K data set to 100% accuracy i.e. the model could store sufficient information in its parameters to categorize all 10k MNIST digits correctly. This is useless as a generalizable model but gave me an easy test to see if the Keras trained parameters, when transferred to Excel would deliver the same result on the same test set.

I was on several steep learning curves throughout this process and was delighted when I eventually got a perfect match. However, as I added the backward model and learned from these already overfit parameters, the model’s precision collapsed to 30% or so and only then began to learn. I saw more failure modes than I can recall and given the slow speed at which the excel model learned had plenty of time to hypothesize the causes.

I began this process in August of 2018 and eventually confirmed that I had a working CapsNet in Excel on 31-December 2018. The closing stages of using this odd approach of matching to a 100% overfit are summarized below.

Once I had confirmation that it was eventually working, adding the full 60k digit training data, loading 50 epoch Keras trained parameters and running a 10k test in Excel that matched the 98.69% Keras Test result took no time. I headed out happy that night to celebrate the new year.

Interesting Learning

Triple Axel

One challenge that I faced was wiring the back propagation of the convolutions. Though I now know this to be straightforward, I went through the process without really thinking through the maths or approach and miraculously ended up with the right answer. On further research I found that this obscure transpose and flip of the kernel over its anti-diagonal is apparently called a Triple Axel named after a figure skater from the 19^th century. This is according to User1551 on math.stackexchange, though I can’t find any other evidence, I love the idea and am happy to propagate the meme.

In TensorFlow and as I understand it, basically all other code, the same approach is handled with a sparse weight-sharing matrix such that the reverse path through the matrix multiplication can simply be accomplished with a transpose of this matrix to get the same connectivity.

In Excel, the Triple Axel transformation of the kernel is much easier to code, use and audit, so makes for a nice approach.

Backpropagation through a stride

I tried many attempts at getting this to work before I began looking for a proper explanation. The best I came across was “A guide to convolution arithmetic for deep learning” by Vincent Dumoulin and Francesco Visin from Institut des algorithmes d’apprentissage de Montréa. I would check their paper out if you’re confused.

Ultimately the wiring in Excel for this is very straightforward and simply requires interspersing zeros in the post-stride channel to bring the size of the channel up to the pre-stride size as shown below. The re-shaping from the 8D capsule gradients was also straightforward and the figure below shows how I unrolled these capsules into the convolutional channels. Again, I tried all sorts of approaches to this simple unfurling of the channels and arrived at the correct one by chance.

Backprop through the affine transforms

I spent some time working through ways to build the derivative of the layer 2 output function dZ^[2] This comprises a sum of the matrix products of each transformation matrix by the dZ^[3] and routed via the derivative of the Layer 2 activation function i.e. only pass the gradient back if the U_i is greater than zero.

Margin loss & “Brim Loss?”

Generating a dZ^[3] with the right dimension (8D) to pass back through the affine transforms also caused me some issues. I tried various approaches but the one that mimicked TensorFlow, and one I therefore assumed to be correct, was simply to multiply the derivative of the loss function by the final digit capsule vectors i.e. after the vector nonlinearity or squash function.

I used the margin loss quoted in the paper but made some silly mistakes in calculating its derivative that negated the use of the max function and instead of ignoring gradients for activations greater than 90% and less than 10%, actually penalized high certainty above and below the thresholds.  This effectively optimized for uncertainty or specifically a 90% certainty of true and a 10% certainty of false. An interesting result of this was that the model trained up to the 100% overfit benchmark I was using faster. This approach also potentially introduces additional regularization at little cost in time and code. 

Because Excel is so slow I’ve stuck with this approach and until I find the correct name for it am calling it a “Brim” loss because the resulting loss curve looks like the brim of a hat. I explain this further in the figure below.

I ran 20 learning trials over 10 epochs for both the Margin Loss and the Brim loss, each with differing seeds for initialization. Multiple trials are the only way to get a rough measure of the advantage that the Brim loss may offer over the margin loss in this case. The trials below show the learning curves (as precision rather than loss) and the improvement is quite substantial. These were, of course run in Python as the process would have taken weeks in Excel.

The Next Steps

If I carry this further in Excel I think the next step will be to introduce an innate graphics model along the lines of “Extracting pose information by using a domain specific decoder” Navdeep Jaitly & Tijmen Tieleman. This will allow the model to run unsupervised learning to go from pixels to entities with poses and opens the ability to train on MNIST to with only a handful of supervised inputs.

I’m also keen to explore Matrix capsules, EM routing, running on the SmallNORB data set and of course optimizing Excel to run more quickly, perhaps making use of the iterative functions in Excel. I’ll update this blog as I make progress but would welcome any encouragement or tips and corrections.

The post Building a Capsule Net in Excel appeared first on NN in XL.

I think these are Dancing:

I think these are Fighting:

The plots are interesting because there are many different ways that people dance or fight and our brains can usually see these, but can you describe the general difference.

I thought no but given the unreasonable effectiveness of neural networks to recognize patterns, maybe they could figure out what that difference was.

These plots show how the cases differ for those labelled by me as dancing and separately, to the right, as fighting. There really is nothing obvious here.

The spreadsheet generates stick figure couples with 10 dots per figure and I’ve laboriously picked out and labelled 600 cases where they appear to me to be either fighting or dancing. The spreadsheet contains two on-sheet fully connected neural networks with 2 layers of 15 and 10 neurons ending in a sigmoid binary classifier and looking at 24 XY coordinates in a flat 48-dimensional vector (there are 2 repeated dots; the neck and the pelvis – wasteful but I was lazy). The first Neural Net runs 5 batches of 100 cases each and learns the difference between Fighting and Dancing and the second runs stochastic gradient descent on a single case at a time using the learned parameters of the first but looking at a separate test dataset of 100 fresh, unseen labelled cases.

Excel lends itself well to building and understanding Neural Networks because it requires an explicit physical layout for the layers within the model, they are not hidden in virtual arrays defined by for-loops but displayed as actual blocks of numbers. The plot below shows the core of the model’s “brain”, the hidden layers and the parameter weights that, once trained, capture the function that we humans each develope to distinguish between aggression and joy. The Momentum and Gradient layers give you a view of how the spreadsheet learns. The gradients of the function are derived directly from a case (or 100 cases for this vectorized model) run through the current learned parameters. The gradients give the direction of travel required to update the parameters of each layer in order to reduce the loss function i.e. the difference between my classification of a particular case and the spreadsheet’s current understanding. If left to train for an hour or so with a small enough learning rate, the model will learn the training cases to an accuracy of 100%. However, this would be an example of overfitting and would not capture the essence of fighting or dancing and therefore not generalize well to new cases.

Excel has some real advantages for coding neural nets, beyond the fact that for me it’s the tool I know and I’m still Python semi-literate. I think these advantages are as follows:

• They lay the structure bare. All layers, all neurons and all connection weights are exposed, no searching or displaying the results of nested For loops, just 2D blocks of numbers.
• When coding you make fundamental wiring errors: This is not so helpful if you know where you’re going but at this stage, neural networks are still a black art. In all the training I’ve done to date the experts use the phrase “this works well but we don’t know why”.
• It’s slow to train and you see it train in minute step by step detail. This gives you time to think and ponder what’s going on, what’s wrong and what’s just plain peculiar.

If you’re interested in this, please let me know and I would be happy to share more with you. I’d certainly welcome any questions and queries you may have and will be releasing some of the base models on GitHub. Any expressions of interest prior to this would also be welcome.

I’ve uploaded a Youtube Video of the model running here: https://youtu.be/RsQp-5feqZQ and you can see more related Neural Network in Excel stuff on my channel here: https://www.youtube.com/

This videos and associated material if released are available to you under a Creative Commons Attribution-NonCommercial 4.0 International License the details of which can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode

The post A Neural Network in Excel learning the difference between fighting and dancing. appeared first on NN in XL.

I have been deeply involved with financial analysis and mathematical models for most of my career but began re-tooling with machine learning over the last two years.  I’m currently working through Andrew Ng’s brilliant Deep Learning course on Coursera and I’ve reached Course 4, Week 3. Throughout the course, I’ve been building out the neural net architectures he describes in Excel.  Excel is not “yet” the right medium to build convolutional neural nets for real-world applications. However, I know Excel and find it easier to construct these on a spreadsheet rather than in a new language. I certainly hope this state won’t last long.

Building in Excel may be slow, but it makes neural nets very transparent, in that you are visually confronted with the layers, neurons and their associated parameters and related calculations. Excel gives a less abstract view of a neural net than vectorised Python code and it helped me immensely in developing an understanding of these fantastic new tools.

Excel lets you see inside the machine but for me, an even more useful element was the bugs and miss wiring that I introduced. Neural networks are very robust to bugs, in fact, they often continue to learn but fail in odd and interesting ways. Figuring out the failure based on the learning behaviour really forced me to think about the maths and the structure. I would also argue that the speed of Excel gives you time to think as the failures manifest themselves. For a deeper understanding of Neural Nets go to Andrew Ng. My intuitions are still weak but certainly improving.

The classic beginners exercise for deep learning is to build an MNIST digit classifier. This relates to Yann LeCun’s data set of 60,000 handwritten digits (0 to 9) with an associated 10,000-digit test set. Yann has made this data available to all and there are plenty of higher level language examples.

My initial attempts at classifiers using plain vanilla fully connected neural networks are on my YouTube channel below but Mike Pallister was faster off the mark than me and is also worth a look.

Single Digit Classification https://www.youtube.com/watch?v=4P5r0tT7Hsc&t=33s

Batch Digit Classification https://www.youtube.com/watch?v=bJcv9vi4Gqg&t=6s

However, this blog relates to last week’s modelling of a convolutional neural net; again with ADAM optimisation and Batch capability. After several silly but informative mistakes, I completed the model below last night and would welcome your thoughts.

The model has very few layers, neurons and parameters, its tiny in comparison to real-world examples. By way of a full description, there are two convolutional layers with max pooling taking the images of the handwritten digits from 28h x 28w pixels to 24h x 24w x 4c (4 channels) and 12h x 12w x 4c after max pooling. Layer 2 condensed these to 8h x 8w x 8c and then 4h x 4w x 8c after max pooling. The final two fully connected layers had 15 and 10 neurons respectively. Giving a grand total of 936 convolutional parameters and 2,095 in the fully connected layers. This doesn’t sound like a lot of space to capture the vagaries of human handwriting, but it does. On its first 100k iterations I’m seeing 98.75% accuracy on the training data and 98% on the test data.  I would love to know what human accuracy levels on this data are but from my experience, it’s not much more than this.

I’d certainly welcome any questions and queries you may have and will be releasing some of the base models on GitHub. Any expressions of interest prior to this would also be welcome.

These videos and associated material if released are available to you under a Creative Commons Attribution-NonCommercial 4.0 International Licence the details of which can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode

The post Building Convolutional Neural Networks in Excel appeared first on NN in XL.

The first programming assignment of Andrew’s second course “Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization” was to build a basic neural network to identify a function that would separate a scatter of red and blue dots based on their X, Y coordinates. For a human this is simple, the red dots formed a circle inside the black dots with a bit of random scatter but for an algorithm in Excel, well, before Andrew’s course it was not so easy.

The assignment required the Neural Network to find a boundary between red and black dots. This diagram shows the neural nets final output as shaded orange & blue areas.I had struggled to visualise what was happening inside the neural network and desperately wanted to see something in ‘my’ language, Excel. I searched the net for examples and came up with nothing other than some single neuron examples and one rough MNIST digit classifier. This didn’t make sense to me as I would have thought that Excel was an ideal teaching medium; lots of native speakers and a 2D layout ideal for exploring the dimensions of the inner hidden layers.

Could Excel expose the mystery of Neural Networks?

So, one sunny afternoon at home in India, I set off on a mission. The first step was to grab Andrew’s Data – nightmare, I couldn’t lift it out of Python, I was illiterate. After 2 futile hours, I did the obvious and constructed it myself. This took 5 minutes and opened the possibility of interesting patterns, say a doughnut or a letter. With the data in hand I began construction under the assumption that at some point I’d encounter insurmountable barriers on the way, but to my delight, there weren’t any. After a couple of hours, and with some luck, I’d built it and was ready to start the iterations and learn the function. I hit the macro button, watched for a while, however, nothing happened, I restarted a few times, checked the code then headed off for a coffee. Now the joy of a neural network is that it programs itself which in Excel takes luck and time. I came back a couple of hours later. Wow! I had iterating patterns and a learning curve that was headed in the right direction. The slow speed of my code was, to some extent, a plus and allowed me to see the function develop and ultimately segregate the dots. With the spreadsheet working and through the process of building it, the mystery of basic neural networks and back-propagation was finally clear and exposed.

The learning curve drops off at 400 iterations and by 1000 the neural network as learnt and represented the hidden function.

The first model was very slow but Andrew was covering momentum and regularisation that week in his course so I plugged the first in and attempted the second, regularisation, by making the random scatter change each iteration, this was probably more like data augmentation but it was all new to me. My thinking was that the base function behind the data was three circles and re-running the scatter on each iteration would help the model learn this underlying function rather than the training set – I got a huge increase in speed and some fascinating results on the boundary diagram.

During the learning process (left to right) the boundary function shows weird patterns as it closes in on the red dots.

The Neural Net architecture and equations were more or less straight out of Andrew’s course but placed into an Excel sheet. The layout shown below is very much orientated towards an Excel instantiation, it allowed me to update the model’s Weights and Momentum in a single line of VBA for each iteration namely:  Parameters (T) = Parameters (T+1) .value. I subsequently built a fully recursive version with no VBA that made use of Excels inbuilt iteration functionality but this is less satisfying to run.

You can view a video of the model in operation here: https://www.youtube.com/watch?v=mIpJu-I13cc

This is the basic layout of the on-sheet Excel. It required approximately 30 separate excel formula

The boundary plot i.e. the orange and blue pixels that cover the X plot and represent the boundary function discovered by the neural network required a second neural net without back propagation (the learning bit) that looks as follows:

The Neural network used for the boundary function was even simpler with only 8 separate formulae.

I subsequently recorded a construction video which runs for over an hour but gives a lot more detail on the Excel needed to get this up and running. You can view the video here: https://www.youtube.com/watch?v=suZhX6N5LAk&t=3042s

You can download the Excel file here: Download Excel

These videos and associated material if released are available to you under a Creative Commons Attribution-NonCommercial 4.0 International Licence the details of which can be found here: https://creativecommons.org/licenses/by-nc/4.0/legalcode

The post Neural Networks in Excel – Finding Andrew Ng’s Hidden Circle appeared first on NN in XL.