4. Problems with Von Neumann (1)

The picture below shows the arrangement of a Von Neumann, stored program computer. This is the basis of modern sequential CPUs

This architecture is very successful but there is a subtle problem with it as well. Namely all the data and instructions have to go across the data bus in order to be handled by the CPU. But the transfer rate is far slower than the speed of the CPU. This problem is called the 'von neumann bottleneck'.

If nothing was done about it, then the performance of the computer is limited by the data bus. Increasing the speed of the CPU just means it sits around even longer waiting for instructions and data.

Partial answer:

A special kind of memory called a 'Cache'. This sits on the CPU side of the data bus.

Imagine the data bus as a bridge that can only carry so many instructions at a time. But what if we designed a 'holding area' on the CPU side of the bridge? Then we could store the most often-used instructions in the holding area instead of having to cross the bridge every time.

This holding area is called a 'cache' (pronounced 'cash'). If the software programmer is skilled enough, they will make it easier for the CPU to store the most-often used part of the code in the 'cache'. This activity is called 'code optimisation'.

To illustrate what is meant by 'often-used', consider the small piece of pseudo code below:

The instruction 'Add 1 to a variable' is repeated a 100 times in this loop, resulting in 100 transfers across the data bus. But if that instruction was stored in the cache, there is no need to use the slow data bus. Performance is improved significantly.

So the arrangement now looks like this:

This technique can be extended even further by arranging a hierarchy of caches within the CPU, each one faster than the one feeding it.

Thus there is a Level 1 cache (fastest) being serviced by a Level 2 cache (slower) and so on. The idea is to match the speed of instruction and data flow to the speed of the CPU to maximise performance.

There is no perfect cache design, as there are compromises. And so each CPU maker has its own ideas of the best arrangement.