Claude Shannon - Information Theory And More
Written by Historian   
Wednesday, 13 March 2013
Article Index
Claude Shannon - Information Theory And More
Bell Labs
The bit, the nit, the dit and the Hartley

 

That is:

Information in bits = Sum over all messages i of -Pilog2Pi

This is the measure of information that Shannon invented in 1948. He then went on to publish a series of papers - two in 1948 and one in 1949 - that presented the subject of information theory at a level of completeness that is surprising and an amazing achievement. Other people had been trying to find a measure of information for some time and Shannon goes and not only finds one but he then writes down its complete theory in only a couple of years.

You can go back to the original papers and they read like an authoritative text book on information theory - not research documents feeling their way towards an idea. He started the study of coding theory covering the effect of noise, bandwidth and power, optimal codes, error correcting codes, data compression and all things binary..

Shannon also solved the radio transmission problem described above by stating what is now known as the Shannon- Hartley law which says that given a bandwidth W and a signal to noise ratio of R then the fastest that you can reliably transmit is

 B=W log(1+R)

bits per second. In other words if you reduce the bandwidth or try to send data too quickly then you get an increasing number of errors - which is what anyone who has used a high speed modem over a telephone line or a digital phone over a weak connection will tell you! As the noise on the line gets worse the modem drops back to work at a slower speed and so reduce the number of errors.

The bit, the nit, the dit and the Hartley

Of course the unit of information that Shannon invented is universally called the "bit" standing for binary digit. However, there are other units of information.

If you take the log in Shannon's equation to the base e, i.e. a natural logarithm, then you get a measure of information called the nat or nit - I am completely serious here!

If you take the log to the base 10 then the unit is called the Hartley after R.V.L Hartley who tried to work it all out before Shannon but didn't have the benefit of thinking about binary numbers. The Hartley is also known as the dit.

You might also like to know that 1bit is .69 of a nat and .3 of a dit - fancy describing something as storing .69MegaNats or .3MegaDits.

All I can say is that I'm glad Shannon got there first!

Without Shannon it is doubtful we would still all be using decimal computers or be ignorant of the basic laws of data transmission and compression. Practical engineering has a way of dealing with issues as they arise  so someone would have come up with all the right ideas and theories, if only in retrospect. In this sense it is too strong to call Shannon the father of the bit but he told us all about it before we knew just how important it all was.

 

More Information

A Symbolic Analysis of Relay and Switching Circuits

Related Articles

The Logician and the Engineer

Coding Theory

Introduction to Boolean Logic

How Error Correcting Codes Work

Information Theory

 

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

 

blog comments powered by Disqus

 

Banner

 


ERNIE - A Random Number Generator

It is hard to believe that a whole computer could be needed just to generate some random numbers, but this was the first ERNIE's role in life and in 1957 it was a sophisticated giant.



Computer Languages by Committee - the 1960s

The 1960 saw the growth of interest in computer languages but, unlike today where successful languages are often designed by single-minded enthusiasts, this was the decade of the committee - language  [ ... ]


Other Articles

 <ASIN:0691151008>

<ASIN:0252725484>

<ASIN:0486665216>

<ASIN:0471241954>

 

 

 



Last Updated ( Sunday, 17 March 2013 )
 
 

   
RSS feed of all content
I Programmer - full contents
Copyright © 2014 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.