Bitcoin Wiki - User contributions [en]

Satoshi Client Node Discovery

2017-11-22T00:15:42Z

Eklitzke: update dns seed list

==Overview==

The Satoshi client discovers the IP address and port of nodes in several different ways.

# Nodes discover their own external address by various methods.
# Nodes receive the callback address of remote nodes that connect to them.
# Nodes makes DNS request to receive IP addresses.
# Nodes can use addresses hard coded into the software.
# Nodes exchange addresses with other nodes.
# Nodes store addresses in a database and read that database on startup.
# Nodes can be provided addresses as command line arguments
# Nodes read addresses from a user provided text file on startup

A timestamp is kept for each address to keep track of when the node
address was last seen. The AddressCurrentlyConnected in [https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp net.cpp] handles
updating the timestamp whenever a message is received from a node.
Timestamps are only updated on an address and saved to the database
when the timestamp is over 20 minutes old.

See the Node Connectivity article for information on which type of
addresses take precedence when actually connecting to nodes.

In the first section we will cover how a node handles a request for
addresses via the "getaddr" message. By understanding the role of
timestamps, it will become more clear why timestamps are kept the way
they are for each of the different ways an address is discovered.

==Handling Message "getaddr"==

When a node receives a "getaddr" request, it first figures out how many
addresses it has that have a timestamp in the last 3 hours.
Then it sends those addresses, but if there are more than 2500 addresses
seen in the last 3 hours, it selects around 2500 out of the available
recent addresses by using random selection.

Now lets look at the ways a node finds out about node addresses.

==Discovery Methods==

===Local Client's External Address===
The client uses public web services which return the information to determine its own external, routable IP address.

The client runs a thread called ThreadGetMyExternalIP (in [https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp net.cpp])
which attempts to determine the client's IP address as seen from the outside
world.

First, it attempts to connect to 91.198.22.70 port 80, which should be
the checkip.dyndns.org server. If connection fails, a DNS request is made
for checkip.dyndns.org and a connection is attempted to that address.
Next, it attempts to connect to 74.208.43.192 port 80, which should be
the www.showmyip.com server. If connection fails, a DNS request is made
for www.showmyip.com and a connection is attempted to that address.

For each address attempted above, the client attempts to connect,
send a HTTP request, read the appropriate response line, and parse the
IP address from it.
If this succeeds, the IP address is returned, it is advertised to any connected
nodes, and then the thread finishes (without proceeding to the next
address).

===Connect Callback Address===
When a node receives an initial "version" message, and that node initiated the connection, then the node advertises its address to the remote so that it can connect back to the local node if it wants to.

After sending its own address, it sends a "getaddr" request message to the remote node to learn about more addresses, if the remote node version is recent or if the local node does not yet have 1000 addresses.

===IRC Addresses===

As of version 0.6.x the Bitcoin client no longer uses IRC bootstrapping by default, and as of version 0.8.2 support for IRC bootstrapping has been removed completely. This documentation below is accurate for most prior versions.

In addition to learning and sharing its own address, the node learned about other node addresses via an IRC channel. See [https://github.com/bitcoin/bitcoin/blob/847593228de8634bf6ef5933a474c7e63be59146/src/irc.cpp irc.cpp].

After learning its own address, a node encoded its own address into a string to be used as a nickname. Then, it randomly joined an IRC channel named between #bitcoin00 and #bitcoin99. Then it issued a WHO command. The thread read the lines as they appeared in the channel and decoded the IP addresses of other nodes in the channel. It did this in a loop, forever, until the node was shutdown.

When the client discovered an address from IRC, it set the timestamp on the address to the current time, but it used a "penalty" of 51 minutes, which means it looked like it was actually seen almost an hour earlier.

===DNS Addresses===
Upon startup, if peer node discovery is needed, the client then issues DNS requests to learn about the addresses of other peer nodes. The client includes a list of host names for DNS services that are seeded. As of November, 2017 the list (from chainparams.cpp<ref>[https://github.com/bitcoin/bitcoin/blob/master/src/chainparams.cpp#L139 bitcoin/src/chainparams.cpp at master]</ref>) includes:

* seed.bitcoin.sipa.be
* dnsseed.bluematt.me
* dnsseed.bitcoin.dashjr.org
* seed.bitcoinstats.com
* seed.bitcoin.jonasschnelli.ch
* seed.btc.petertodd.org

A DNS reply can contain multiple IP addresses for a requested name.

Addresses discovered via DNS are initially given a zero timestamp, therefore they are not advertised in response to a "getaddr" request.

===Hard Coded "Seed" Addresses===
The client contains hard coded IP addresses that represent bitcoin nodes.

These addresses are only used as a last resort, if no other method has produced any addresses at all. When the loop in the connection handling thread ThreadOpenConnections2() sees an empty address map, it uses the "seed" IP addresses as backup.

There is code is move away from seed nodes when possible. The presumption is that this is to avoid overloading those nodes. Once the local node has enough addresses (presumably learned from the seed nodes), the connection thread will close seed node connections.

Seed Addresses are initially given a zero timestamp,
therefore they are not advertised in response to a "getaddr" request.

===Ongoing "addr" advertisements===
Nodes may receive addresses in an "addr" message after having sent a "getaddr" request, or "addr" messages may arrive unsolicited, because nodes advertise addresses gratuitously when they relay addresses (see below), when they advertise their own address periodically, and when a connection is made.

If the address is from a really old version, it is ignored; if from a not-so-old version, it is ignored if we have 1000 addresses already.

If the sender sent over 1000 addresses, they are all ignored.

Addresses received from an "addr" message have a timestamp, but the timestamp is not necessarily honored directly.

For every address in the message:
# If the timestamp is too low or too high, it is set to 5 days ago.
# We subtract 2 hours from the timestamp and add the address.

Note that when any address is added, for any reason, the code that calls AddAddress() does not check to see if it already exists. The AddAddresss() function in [https://github.com/bitcoin/bitcoin/blob/master/src/net.cpp net.cpp] will do that, and if the address already exists, further processing is done to update the address record. If the advertised services of the address have changed, that is updated and stored.

If the address has been seen in the last 24 hours and the timestamp is currently over 60 minutes old, then it is updated to 60 minutes ago.

If the address has NOT been seen in the last 24 hours, and the timestamp is currently over 24 hours old, then it is updated to 24 hours ago.

====Address Relay====

Once addresses are added from an "addr" message (see above), they then may be relayed to the other nodes. First, the following criteria must be set [9]:

#The address timestamp, after processing, is within 60 minutes of the current time
#The "addr" message contains 10 addresses or less
#And fGetAddr is not set on the node. fGetAddr starts false, is set to true when we request addresses from a node, and it is cleared when we receive less than 1000 addresses from a node.
#The address must be routable.

For every address that meets the above criteria, the node hash the address, the current day (in the form of an integer), and a random 256 bit value (generated at client startup). The node takes the two addresses with the lowest hash values and relays "addr" messages to them. This ensures that each node only relays "addr" messages to two other clients at any given time, that the two other clients are randomly selected, and that the random selection starts over at least once every 24 hours.

====Self broadcast====

Every 24 hours, the node advertises its own address to all connected nodes.

It also clears the list of the addresses we think the remote node has, which will trigger a refresh of sends to nodes. This code is in SendMessages() in [https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp main.cpp].

====Old Address Cleanup====

In SendMessages() in [https://github.com/bitcoin/bitcoin/blob/master/src/main.cpp main.cpp], there is code to remove old addresses.

This is done every ten minutes, as long as there are 3 active connections.

The node erases messages that have not been used in 14 days as long as there are at least 1000 addresses in the map, and as long as the erasing process has not taken more than 20 seconds.

===Addresses stored in the Database===
Addresses are stored in the database when AddAddress() is called.

Addresses are read on startup when AppInit2() calls LoadAddresses(), which is located in [https://github.com/bitcoin/bitcoin/blob/master/src/db.cpp db.cpp].

Currently, it appears all addresses are stored all at once whenever any address is stored or updated<ref>[http://bitcointalk.org/index.php?topic=26436.0 Lots of disk activity on Bitcoin startup: easy fix?]</ref>. Indeed, AddAddress is seen to take over .01 seconds in various testing and is typically called tens of thousands of times in the initial 12 hours of running the client.

===Command Line Provided Addresses===

The user can specify nodes to connect to with the
-addnode <ip>
command line argument. Multiple nodes may be specified.

Addresses provided on the command line are initially given a zero timestamp, therefore they are not advertised in response to a "getaddr" request.

The user can also specify an address to connect to with the -connect <ip>
command line argument. Multiple nodes may be specified.

The -connect argument differs from -addnode in that -connect addresses are not added to the address database and when -connect is specified, only those addresses are used.

===Text File Provided Addresses===

The client will automatically read a file named "addr.txt" in the bitcoin data directory and will add any addresses it finds in there as node addresses. These nodes are given no special preference over other addresses. They are just added to the pool.

Addresses loaded from the text file are initially given a zero timestamp, therefore they are not advertised in response to a "getaddr" request.

==See Also==

* [[Network#Bootstrapping|Network - Boostrapping]]

==References==
<references />

[[Category:Developer]]
[[Category:Technical]]

User:Eklitzke

2017-11-19T20:18:10Z

Eklitzke: Created page with "I am Evan Klitzke. I have [https://eklitzke.org/ a website] and a [https://github.com/eklitzke a GitHub]."

I am Evan Klitzke. I have [https://eklitzke.org/ a website] and a [https://github.com/eklitzke a GitHub].

Data directory

2017-11-19T20:12:22Z

Eklitzke: /* chainstate subdirectory */

The data directory is the location where Bitcoin's data files are stored, including the [[Wallet|wallet]] data file.

==Default Location==
=== Windows ===

Go to Start -> Run (or press WinKey+R) and run this:

%APPDATA%\Bitcoin

Bitcoin's data folder will open. For most users, this is the following locations:

C:\Documents and Settings\YourUserName\Application data\Bitcoin (XP)

C:\Users\YourUserName\Appdata\Roaming\Bitcoin (Vista and 7)

"AppData" and "Application data" are hidden by default.

You can also store Bitcoin data files in any other drive or folder.

If you have already downloaded the data then you will have to move the data to the new folder.
If you want to store them in D:\BitcoinData then click on "Properties" of a shortcut to bitcoin-qt.exe and
add -datadir=D:\BitcoinData at the end as an example:

"C:\Program Files (x86)\Bitcoin\bitcoin-qt.exe" -datadir=d:\BitcoinData

Start Bitcoin, now you will see all the files are created in the new data directory.

=== Linux ===

By default Bitcoin will put its data here:

~/.bitcoin/

You need to do a "ls -a" to see directories that start with a dot.

If that's not it, you can do a search like this:

find / -name wallet.dat -print 2>/dev/null

=== Mac ===

By default Bitcoin will put its data here:

~/Library/Application Support/Bitcoin/

==Directory Contents==

===Files===

An overview of these is in [https://github.com/bitcoin/bitcoin/blob/master/doc/files.md files.md] in the Bitcoin Core documentation.

* .lock
** Bitcoin data directory lock file
* bitcoin.conf [optional]
**Contains [[Running_Bitcoin#Bitcoin.conf_Configuration_File|configuration options]].
* blk''xxxx''.dat [Versions prior to v0.8.0]
**Contains concatenated raw blocks. Stored are actual Bitcoin blocks, in network format, dumped to disk raw.
* blkindex.dat [Versions prior to v0.8.0]
**Indexing information used with blk''xxxx''.dat
* __db.''xxx''
**Used by BDB
* db.log
* debug.log
**Bitcoin's verbose log file. Automatically trimmed from time to time.
* wallet.dat
**Storage for keys, transactions, metadata, and options. <span style="color:red">Please be sure to make backups of this file. It contains the keys necessary for spending your bitcoins.</span>
* addr.dat [Versions prior to v0.7.0]
** Storage for ip addresses to make a reconnect easier
* peers.dat [Versions v0.7.0 and later]
** Storage for peer information to make a reconnect easier. This file uses a bitcoin-specific file format, unrelated to any database system<ref>[http://bitcointalk.org/index.php?topic=119525.msg1287284#msg1287284 Ultraprune merged in mainline]</ref>.
* fee_estimates.dat [Versions v0.10.0 and later]
** Statistics used to estimate fees and priorities. Saved just before program shutdown, and read in at startup.
The data, index and log files are used by Oracle [http://en.wikipedia.org/wiki/Berkeley_DB Berkeley DB], the embedded key/value data store that Bitcoin uses.

===database subdirectory===
Contains BDB journaling files

===testnet3 subdirectory===
Contains testnet versions of these files (if running with -testnet)

===blocks subdirectory===
[v0.8 and above] Contains blockchain data.

* blk*.dat
** Stored are actual Bitcoin blocks, in network format, dumped to disk raw. They are only needed for re-scanning missing transactions in a wallet, reorganizing to a different part of the chain, and serving the block data to other nodes that are synchronizing.

* blocks/index subdirectory
** [v0.8 and above] A LevelDB database that contains metadata about all known blocks, and where to find them on disk. Without this, finding a block would be very slow.

===chainstate subdirectory===
[v0.8 and above] A LevelDB database with a compact representation of all currently unspent transaction outputs and some metadata about the transactions they are from. The data here is necessary for validating new incoming blocks and transactions. It can theoretically be rebuilt from the block data (see the -reindex command line option), but this takes a rather long time. Without it, you could still theoretically do validation indeed, but it would mean a full scan through the blocks (150 GB as of Nov 2017) for every output being spent.

===locks subdirectory===
[v0.8 and above] Contains "undo" data.

* rev*.dat
You can see blocks as 'patches' to the chain state (they consume some unspent outputs, and produce new ones), and see the undo data as reverse patches. They are necessary for rolling back the chainstate, which is necessary in case of reorganizations.

===Personally identifiable data [v0.8 and above]===
This section may be of use to you if you wish to send a friend the blockchain, avoiding them a hefty download.

*wallet.dat
**Contains addresses and transactions linked to them. <span style="color:red">Please be sure to make backups of this file. It contains the keys necessary for spending your bitcoins.</span> You should not transfer this file to any third party or they may be able to access your bitcoins.
*db.log
**May contain information pertaining to your wallet. It may be safely deleted.
*debug.log
**May contain IP addresses and transaction ID's. It may be safely deleted.
*database/ folder
**This should only exist when bitcoin-qt is currently running. It contains information (BDB state) relating to your wallet.
*peers.dat
**Unknown whether this contains personally identifiable data. It may be safely deleted.

Other files and folders (blocks, blocks/index, chainstate) may be safely transferred/archived as they contain information pertaining only to the public blockchain.

==Transferability==

The database files in the "blocks" and "chainstate" directories are cross-platform, and can be copied between different installations. These files, known collectively as a node's "block database", represent all of the information downloaded by a node during the syncing process. In other words, if you copy installation A's block database into installation B, installation B will then have the same syncing percentage as installation A. This is usually ''far'' faster than doing the normal initial sync over again. However, when you copy someone's database in this way, you are trusting them '''absolutely'''. Bitcoin Core treats its block database files as 100% accurate and trustworthy, whereas during the normal initial sync it treats each block offered by a peer as invalid until proven otherwise. If an attacker is able to modify your block database files, then they can do all sorts of evil things which could cause you to lose bitcoins. Therefore, you should only copy block databases from Bitcoin installations under your personal control, and only over a secure connection.

Each node has a unique block database, and all of the files are highly connected. So if you copy just a few files from one installation's "blocks" or "chainstate" directories into another installation, this will almost certainly cause the second node to crash or get stuck at some random point in the future. If you want to copy a block database from one installation to another, you have to delete the old database and copy ''all'' of the files at once. Both nodes have to be shut down while copying.

Only the file with the highest number in the "blocks" directory is ever written to. The earlier files will never change. Also, when these blk*.dat files are accessed, they are usually accessed in a highly sequential manner. Therefore, it's possible to symlink the "blocks" directory or some subset of the blk*.dat files individually onto a magnetic storage drive without much loss in performance (see [[Splitting the data directory]]), and if two installations start out with identical block databases (due to the copying described previously), subsequent runs of rsync will be very efficient.

==See Also==

* [[Running Bitcoin]]
* [[Securing your wallet]]
* [[Splitting the data directory]]
* [http://bitcoin.stackexchange.com/a/11108/153 What is the database for?] Question on Bitcoin Stack Exchange

==References==
<references />

[[Category:Technical]]

[[es:Directorio de datos]]

{{Bitcoin Core documentation}}