Difference between revisions of "Libbitcoin Database"

From Bitcoin Wiki
Jump to: navigation, search
m (Example)
(History)
(6 intermediate revisions by the same user not shown)
Line 1: Line 1:
The [https://github.com/libbitcoin/libbitcoin-database libbitcoin-database ] library is a dependency of [[Libbitcoin_Blockchain|libbitcoin-blockchain]], [[Libbitcoin_Node|libbitcoin-node]] and [[Libbitcoin_Server|libbitcoin-server]]. It was originally contained within [[Libbitcoin_Blockchain|libbitcoin-blockchain]].
+
The [https://github.com/libbitcoin/libbitcoin-database libbitcoin-database] library is a dependency of [[Libbitcoin_Blockchain|libbitcoin-blockchain]], [[Libbitcoin_Node|libbitcoin-node]] and [[Libbitcoin_Server|libbitcoin-server]].
  
 
==Example==
 
==Example==
Line 21: Line 21:
  
 
==Design==
 
==Design==
The original implementation used [http://leveldb.org LevelDB]. Following a redesign in late 2014 by [[Amir_Taaki|Amir Taaki]] (genjix) the database was replaced by a [https://en.wikipedia.org/wiki/Memory-mapped_file memory-mapped file] implementation. Logical queries are performed using a set of hash tables. The number of hash buckets is optimized to minimize hash collisions, though collisions are accommodated. These changes resulted in a substantial performance increase, near constant time, for queries against the blockchain. Insert performance was not materially affected. The database uses sequence locking to avoid writer starvation while never blocking the reader. This is ideal for a high performance blockchain server as reads are significantly more frequent than writes and yet writes must proceed wtihout delay.
+
The database is built directly on the operating system's [https://en.wikipedia.org/wiki/Memory-mapped_file memory-mapped file] system. Logical queries are performed using a set of hash tables. The number of hash buckets is optimized to minimize hash collisions, though collisions are accommodated. These changes resulted in a substantial performance increase, near constant time, for queries against the blockchain. Insert performance was not materially affected. The database uses sequence locking to avoid blocking the writer. This is ideal for a high performance blockchain server as reads are significantly more frequent than writes and yet writes must proceed without delay.
  
 
[[Image:Libbitcoin-blockchain-gets.png|500px]] [[Image:Libbitcoin-blockchain-gets-normalised.png|500px]]
 
[[Image:Libbitcoin-blockchain-gets.png|500px]] [[Image:Libbitcoin-blockchain-gets-normalised.png|500px]]
  
 
==Database==
 
==Database==
The following files constitute the blockchain database non-volatile storage. As of height 350,000 the database consumes approximately 105 GB of disk space.
+
The following files constitute the blockchain database non-volatile storage for a node. As of height 440,000 these consume approximately 102 GB of disk space.
* blocks_lookup
+
* block_index
* blocks_rows
+
* block_table
* history_lookup
+
* transaction_table
 +
 
 +
The following files constitute the non-volatile storage for the server index.
 +
* history_table
 
* history_rows
 
* history_rows
* spends
 
* stealth_index
 
 
* stealth_rows
 
* stealth_rows
* txs
+
* spend_table
 +
 
 +
The following files are used for locking.
 +
* flush_lock
 +
* exclusive_lock
  
 
==Considerations==
 
==Considerations==
 
* There is no mechanical hard drive optimization. The implementation is intended for solid state drives ([http://en.wikipedia.org/wiki/Solid-state_drive SSD]).
 
* There is no mechanical hard drive optimization. The implementation is intended for solid state drives ([http://en.wikipedia.org/wiki/Solid-state_drive SSD]).
* There is a possibility of index corruption during hard shutdown. There is no means of detecting corruption aside from catastrophic fault. However given that the entire blockchain is a cache this is not considered significant. Repair can be accomplished by re-synchronizing the blockchain.
+
* There is a likelihood of index corruption during hard shutdown. In version 2 there is no means of detecting corruption aside from catastrophic fault. However given that the entire blockchain is a cache this is not considered significant. Repair can be accomplished by re-synchronizing the blockchain. Version 3 provides robust detection of corruption and the option to flush after each write, making corruption very unlikely (and still fully detectable) if enabled.
 
* Data files are append only, with logical deletions only. Therefore file size is not minimized following blockchain reorganization although the impact is typically small. Defragmentation can be accomplished by re-synchronizing the blockchain.
 
* Data files are append only, with logical deletions only. Therefore file size is not minimized following blockchain reorganization although the impact is typically small. Defragmentation can be accomplished by re-synchronizing the blockchain.
* The database is effectively locked during write operations. As these operations are anticipated on a period of approximately ten minutes this is not typically significant. However during a period of catch-up synchronizing the server can become continuously unresponsive to requests.
+
* The database is effectively locked during write operations. As these operations are anticipated on a period of approximately ten minutes this is not typically significant. However during a period of catch-up synchronizing a version 2 server can become continuously unresponsive to requests. This is not a problem for version 3.
 +
 
 +
==History==
 +
* The original implementation used [http://leveldb.org LevelDB].
 +
* In late 2014 by [[Amir_Taaki|Amir Taaki]] (genjix) replaced LevelDB with memory-mapped hash table storage and indexing.
 +
* In early 2016 [[Eric_Voskuil|Eric Voskuil]] separated the database from [[Libbitcoin_Blockchain|libbitcoin-blockchain]] into an independent library and used isolated locking to make the concurrent read-while-write safe and to enable concurrent writes.
  
 
==Dependencies==
 
==Dependencies==

Revision as of 17:47, 8 March 2017

The libbitcoin-database library is a dependency of libbitcoin-blockchain, libbitcoin-node and libbitcoin-server.

Example

#include <string>
#include <bitcoin/database.hpp>

// Create a new mainnet blockchain store.
int main(int argc, char* argv[])
{
    std::string prefix("blockchain");
    const auto genesis = bc::chain::block::genesis_mainnet();
    
    if (argc > 1)
        prefix = argv[1];
    
    if (!bc::database::data_base::initialize(prefix, genesis)
        return -1;
   
    return 0;
}

Design

The database is built directly on the operating system's memory-mapped file system. Logical queries are performed using a set of hash tables. The number of hash buckets is optimized to minimize hash collisions, though collisions are accommodated. These changes resulted in a substantial performance increase, near constant time, for queries against the blockchain. Insert performance was not materially affected. The database uses sequence locking to avoid blocking the writer. This is ideal for a high performance blockchain server as reads are significantly more frequent than writes and yet writes must proceed without delay.

Libbitcoin-blockchain-gets.png Libbitcoin-blockchain-gets-normalised.png

Database

The following files constitute the blockchain database non-volatile storage for a node. As of height 440,000 these consume approximately 102 GB of disk space.

  • block_index
  • block_table
  • transaction_table

The following files constitute the non-volatile storage for the server index.

  • history_table
  • history_rows
  • stealth_rows
  • spend_table

The following files are used for locking.

  • flush_lock
  • exclusive_lock

Considerations

  • There is no mechanical hard drive optimization. The implementation is intended for solid state drives (SSD).
  • There is a likelihood of index corruption during hard shutdown. In version 2 there is no means of detecting corruption aside from catastrophic fault. However given that the entire blockchain is a cache this is not considered significant. Repair can be accomplished by re-synchronizing the blockchain. Version 3 provides robust detection of corruption and the option to flush after each write, making corruption very unlikely (and still fully detectable) if enabled.
  • Data files are append only, with logical deletions only. Therefore file size is not minimized following blockchain reorganization although the impact is typically small. Defragmentation can be accomplished by re-synchronizing the blockchain.
  • The database is effectively locked during write operations. As these operations are anticipated on a period of approximately ten minutes this is not typically significant. However during a period of catch-up synchronizing a version 2 server can become continuously unresponsive to requests. This is not a problem for version 3.

History

  • The original implementation used LevelDB.
  • In late 2014 by Amir Taaki (genjix) replaced LevelDB with memory-mapped hash table storage and indexing.
  • In early 2016 Eric Voskuil separated the database from libbitcoin-blockchain into an independent library and used isolated locking to make the concurrent read-while-write safe and to enable concurrent writes.

Dependencies

See Also

References