Page 1 of 2
Before the neural network algorithms in use today were devised, there was an alternative. It was invented in 1943 by neurophysiologist Warren McCulloch and logician Walter Pitts. Now networks of the McCulloch-Pitts type tend to be overlooked in favour of “gradient descent” type neural networks and this is a shame.
What is interesting about the McCulloch-Pitts model of a neural network is that it can be used as the components of computer-like systems. That is, where neural networks are commonly used to learn something, a McCulloch-Pitts neuron is constructed to do a particular job. OK, I have to admit that in many cases the same job can be done just as well using traditional Boolean components, but it is interesting to see how it all works using components which are closer to “biological” components.
The basic idea of a McCulloch-Pitts model is to use components which have some of the characteristics of real neurons. A real neuron has a number of inputs which are “excitatory” and some which are “inhibitory”. What the neuron does depends on the sum of inputs. The excitatory inputs tend to make the cell fire and the inhibitory inputs make is not fire – i.e. pass on the signal.
You can think of it as a sort of battle to control the neuron with the excitatory inputs pushing it to fire and the inhibitory inputs stopping it. There are lots of complications to this basic model – in particular a neuron fires a train of pulses rather than just changing its state - but the idea of inhibitory fighting excitatory seems to be the key behaviour.
How can we create an easier to use model of the neuron?
First let’s just have a cell that gives out a binary state – zero or one, on or off. The inputs then carry a binary signal and the only thing that matters is the number of “on” signals on the excitatory versus the inhibitory inputs. We have essentially a “threshold” gate. This turns out to be a bit too complicated to analyse so we make one more big simplification – the inhibitory input overrides the excitatory. That is, if an inhibitory input is on the cell cannot fire (i.e. it is off) no matter what the excitatory inputs are doing. With this small change things are made a lot easier and we can now define precisely what a McCulloch-Pitts neuron is:
- a cell which can output a 0 or a 1
- a number of excitatory inputs
- a single inhibitory input
- a threshold value
The rule for its operation is that at time t it looks at its excitatory inputs and counts up the number of ones present. If the count is equal to or greater than the threshold AND the inhibitory input is zero then at time t+1 the cell outputs a one, otherwise it outputs a zero.
Notice that this rule is more complicated than you might think at first and it does give rise to behaviours that are quite complicated. The key factor that you might just overlook is that we are using sequential logic that involves time. A clock is used to synchronise all of the cells in a network and they change state at each clock pulse. This is necessary to stop infinite period oscillation in some types of circuit.
I’m going to use the symbols and notation introduced by Marvin Minsky. Minsky uses a symbol that shows a cell as a circle with its threshold written in. It is very easy to invent equivalents of the standard Boolean gates. For example, it is very easy to work out how to make OR, AND and NOT.
The basic gates of Boolean logic
You should be able to see that the truth tables for each of these cells is equivalent to the corresponding Boolean gate.
For example, for the AND cell, one input at one doesn’t fire the cell but two and only two at one does fire the cell – i.e. an AND gate as promised. The only tricky one is the NOT gate and you can see here that a single inhibitory input (shown with a circle at the end) combined with a threshold of zero gives the desired result. When the inhibitor is off the threshold is equalled, i.e. zero excitatory inputs and the cell give out a one. If the inhibitor is on then the cell is obviously off – a NOT gate as promised.
More than Boolean?
Given we can make up the basic gates of Boolean logic it is clear that that McCulloch-Pitts networks are up to building anything that can be built with standard logic gates, but are they capable of anything more? The answer is yes and the reason is partly that they have time built into their specification and partly because they are threshold gates. For example, a delay gate can be built using a single excitatory input with a threshold of one. Stacking such gates up provides a delay of more than one time unit. Delaying a signal is the same as remembering it for n time periods, so cells have memory!
If you would like a clearer example of the memory inherent in McCulloch-Pitts cells have a look at the following simple latch.
The top cell passes the input to the output if it has a one from the other cell. You can see that the feedback loop in the lower cell keeps it switched on once it is on irrespective of the input on the Start line. Once on the only way to switch it off is to put a one on the Stop line which, being an inhibitor, turns everything off.
Divide by two
If you know some digital electronics you might well be tempted to try and expand this circuit from a latch to a divide by two circuit – something like a D-Type flipflop. Most people who try come up with a two-cell design something like that shown in the diagram:
A wrong divide by two network
You can see the basic idea. The threshold 2 cell acts as half of a flipflop because of the inhibitory self-feedback. If it is a one it changes back to a zero no matter what the input is. The threshold 1 cell is also inhibited in the same way and every two pulses the whole circuit switches output but there is a problem. If a pulse occurs while the cells are resetting then it is missed completely. In other words, this circuit divides by two but occasionally misses pulses. The correct solution, which doesn’t drop any pulses, needs an extra cell.
A correct divide by two network
The fact that it takes three cells to divide by two is something that surprises many a designer used to standard digital components.