A private key in the context of Bitcoin is a secret number that allows Bitcoins to be spent. Every Bitcoin address has a matching private key, which is saved in the wallet file of the person who owns the balance. The private key is mathematically related to the Bitcoin address, and is designed so that the Bitcoin address can be calculated from the private key, but importantly, the same cannot be done in reverse.
Because the private key is the "ticket" that allows someone to spend Bitcoins, it is important that these are kept secure. Private keys can be kept on computer files, but they are also short enough that they can be printed on paper. An example of a utility that allows extraction of private keys from your wallet file for printing purposes is pywallet.
In Bitcoin, a private key is a 256-bit number, which can be represented one of several ways. Here is a private key in hexadecimal - 256 bits in hexadecimal is 32 bytes, or 64 characters in the range 0-9 or A-F.
E9 87 3D 79 C6 D8 7D C0 FB 6A 57 78 63 33 89 F4 45 32 13 30 3D A6 1F 20 BD 67 FC 23 3A A3 32 62
When we represent private keys in Bitcoin, however, we use a shorter format known as wallet import format, which offers a few advantages. The wallet import format is shorter, and includes built-in error checking codes so that typos can be automatically detected and/or corrected (which is impossible in hex format). Wallet import format is the most common way to represent private keys in Bitcoin. They are 51 characters and always start with the number 5. This is the same private key in wallet import format.
When a private key is imported, it always corresponds to exactly one Bitcoin address. Any utility which performs the conversion can display the matching Bitcoin address. The mathematical conversion is somewhat complex and best left to a computer, but it's notable that every private key will always correspond to the same address no matter which program is used to convert it.
The Bitcoin address corresponding to the sample above is: 1CC3X2gu58d6wXUWMffpuzN9JAfTUWu4Kj
Some applications use the mini private key format. Not every private key or Bitcoin address has a corresponding mini private key - they have to be generated a certain way in order to ensure a mini private key exists for an address. The mini private key is used for applications where space is critical, such as in QR codes and in physical bitcoins. The above example has a mini key, which is:
So in summary, any Bitcoins sent to the address 1CC3X2gu58d6wXUWMffpuzN9JAfTUWu4Kj can be spent by anybody who knows the private key in any of the three formats. That includes bitcoins presently at the address, as well as any bitcoins that are ever sent to it in the future. The private key is only needed to spend the balance, not necessarily to see it. The Bitcoin balance of the address can be determined by anybody with the public Block Explorer at http://www.blockexplorer.com/address/1CC3X2gu58d6wXUWMffpuzN9JAfTUWu4Kj - even if they don't have the private key.
If a private key with a Bitcoin balance is compromised or stolen, the bitcoin balance can only be protected if it is immediately spent to a different address whose private key is not compromised. Because bitcoins can only be spent once, when they are spent away from a private key, the private key is worthless unless more coins are sent to the address.