An off-chain transaction is the movement of value outside of the block chain. While an on-chain transaction - usually referred to as simply 'a transaction' - modifies the blockchain and depends on the blockchain to determine its validity an off-chain transaction relies on other methods to record and validate the transaction. Like on-chain transactions all parties must agree to accept the particular method by which the transaction occurs, the question then being, how can those parties be convinced that the movement of value has actually happened, will not be reversed, and can be exchanged in the future for something of value?
With an on-chain transaction those question are answered by the parties faith in the Bitcoin system as a whole. For instance a transaction (after some number of confirmations) can only be reversed if a majority of hashing power agrees to reverse the transaction. The parties to the transaction are trusting that the majority of hashing power in existence is controlled by "honest" parties who will not attempt to reverse the transaction.
On-chain transactions have disadvantages that make them unsuitable for some applications:
On-chain transactions take some time to accumulate enough confirmations to ensure that they can-not be reversed; accepting a transaction without any confirmations is potentially risky. Confirmations take time and the time they take to accumulate is random. Off-chain transaction systems can record that a transaction has happened immediately, and, subject to the guarantees of the system itself, immediately guarantee it won't be reversed.
All on-chain transactions are recorded publicly on the block chain; Bitcoin transactions are not inherently anonymous. It may be possible for a third-party to use the block chain transaction data to determine the source and/or destination of a transaction if they can gather enough information linking addresses to identities. Because off-chain transactions do not happen on the block chain they need not be public. Using cryptographic techniques such as chaum tokens it can be made impossible for even the operators of the system itself to determine who participated in a transaction.
Miners usually charge fees to confirm a transaction. While currently the demand for transactions is sufficiently low that fees are relatively small, and transactions can often be confirmed for free, for many applications even paying a few cents per transaction is unaffordable. In addition Bitcoin currently has a limit of 7 transactions per second, the blocksize limit. This limit is related to the scalability of the system as a whole, and one option to achieve higher transaction volumes is to keep the blocksize limit as is and use off-chain transactions for lower-value transactions; with higher volumes fees for transactions done on-chain will rise due to supply and demand.