Bitcoin Core 0.11 (ch 4): P2P Network

Bitcoin is a peer-to-peer network, so Bitcoin Core has code to discover peers and manage those connections.

Most of the network-handling code is in net.h/cpp.

Data structures to manage peers

At any given time, the node is connected with a set of other nodes, i.e. peers. By default the code connects to 8 outbound peers (nodes that our node goes out and finds) and allows up to 125 inbound peers (nodes that find us through the network).

The global variable vNodes (net.h) holds the set of peers. The variable is protected by cs_vNodes.

Each peer is represented by a CNode.

The CNode contains dozens of attributes, most of which have to do with low-level plumbing (sockets, byte streams, etc.)

Some of the key attributes of a CNode include:

CNode

Attribute	Description
nServices	Commonly referred to as "the service bits." This is a bitmap of what services the peer provides. As of 0.11 this is still just binary: either the node is NODE_NETWORK (it is a full node and does everything) or not (it is SPV). In future versions, these bits will convey more precise information about what the node can and can't do. For example, with Block Pruning a node may be able to serve recent blocks (say, the last week or two worth of blocks), but not the entire blockchain.
fClient	Whether this peer is a SPV node. (Here, the term "client" means "merely SPV, not a full node.")
fOneShot
fInbound	Whether this node is "inbound" or "outbound." Common sense suggests that a node that we discovered through the network (an outbound node) is less likely to attack us than a node that found us. So, for example, when the code looks for a peer to request historical blocks from, it prefers an outbound node, if possible.
fWhiteListed	A whitelisted node is not subject to being banned for bad behaviour.
setInventoryKnown
vSendMsg	Messages that we've queued up to send to the peer. This is type <CSerializeData> since we are going to send it over the network.
vRecvMsg	Messages that we've received from the peer. This is type <CNetMessage> because as soon as the data is received, it is deserialized and packed into a more useful format (an object).

Peer discovery & connectivity

Address manager

IP addresses of the node's peers are managed by the address manager (see addrman.h).

The code comment explains the address manager (edited here for conciseness):

Design goals:
  * Keep the address tables in-memory, and asynchronously dump the entire table to peers.dat.
  * Make sure no (localized) attacker can fill the entire table with his nodes/addresses.

To that end: 
  * Addresses are organized into buckets.
    * Addresses that have not yet been tried go into 1024 "new" buckets.
    * Addresses of nodes that are known to be accessible go into 256 "tried" buckets.
    * Bucket selection is based on cryptographic hashing, using a randomly-generated 256-bit key, which should not be observable by adversaries.
    * Several indexes are kept for high performance. Defining DEBUG_ADDRMAN will introduce frequent (and expensive) consistency checks for the entire data structure.

Timestamps

The address manager also keeps track of when each peer was last heard from. Timestamps are only updated on an address and saved to the database when the timestamp is over 20 minutes old. By understanding the role of timestamps, it will become more clear why timestamps are kept the way they are for each of the different ways an address is discovered.

Peer discovery

The program discovers the IP address and port of nodes in several different ways:

• Address database (peers.dat)
• User-specified (-addnode and -connect)
• DNS seeding
• Hard-coded seeds
• From other peers ("getaddr" and "addr" messages)

1) Address database (peers.dat)

Nodes store addresses in a database (peers.dat) which is read on startup, and loaded into the address manager.

This method does not work the first time the program is run, since it does not already know about any other nodes on the bitcoin network.

For details on when/how the code stores to the database, see the section above on the address manager.

2) User-specified on the command line

The user can specify nodes to connect to on the command line with -addnode <ip> or -connect <ip>.

Notes on user-specified IP addresses:

• Multiple nodes may be specified.
• Addresses are initially given a zero timestamp, therefore they are not advertised in response to a "getaddr" request.
• With -connect, the IP addresses will not be added to peers.dat and only the provided addresses will be used.
• With -addnode, the provided addresses will be used as a starting point, but the node will soon learn other peers.

3) DNS seeding

DNS seeding is only used if the peers.dat database is empty (as it would be when initiallly running the program) and the user has not specified any nodes with -addnode or -connect.

In this case, the node can issue DNS requests to discover IP addresses of other peers.

As of 0.11, there are six DNS servers hard-coded into the program - see chainparams.cpp.

A DNS reply can contain multiple IP addresses for a requested name.

Addresses discovered via DNS are initially given a zero timestamp, to avoid being advertised in response to a "getaddr" request.

4) Hard-coded nodes (last resort)

If DNS seeding fails, the client contains hard coded IP addresses that represent bitcoin nodes. See: chainparamsseeds.h

These addresses are only used as a last resort, and a log message will be printed: "Adding fixed seed nodes as DNS doesn't seem to be available." (net.cpp)

The idea is to move away from seed nodes as soon as possible, to avoid overloading those nodes.

Once the local node has enough addresses (presumably learned from the seed nodes), the connection thread will close any seed node connections.

Like the DNS seed addresses, the hard-coded seed addresses are also given a zero timestamp to avoid being advertised in response to a "getaddr" request.

5) From other nodes ("getaddr" and "addr" messages)

Nodes exchange IP addresses with other nodes via the "getaddr" and "addr" messages.

Usually, an addr" message is sent in response to a "getaddr".

However, the "addr" message may also arrive unsolicited, because nodes advertise addresses gratuitously when they:

• Relay addresses (see below)
• Advertise their own address periodically. (Every 24 hours, the node advertises its own address to all connected nodes.)
• When a connection is made (in response to an initial "version" message)

When does a node send a "getaddr" message?

• In response to a "version" message from an outbound node, if we don't yet have 1000 addresses.

Receiving an "addr" message:

• If the sending node is an old version and we have 1000 addresses already, it is ignored.
• If the sending node is a current version and is attempting to send us more than 1000 addresses, the peer is punished for misbehaving.
• If the address has been seen in the last 24 hours and the timestamp is currently over 60 minutes old, then it is updated to 60 minutes ago.
• If the address has NOT been seen in the last 24 hours, and the timestamp is currently over 24 hours old, then it is updated to 24 hours ago.

Responding to a "getaddr" message:

• The node figures out how many addresses it has that have a timestamp in the last 3 hours.
• It sends those addresses, but if there are more than 2500 addresses, it randomly selects 2500.
• It clears the list of the addresses we think the remote node has, which will trigger a refresh of sends to nodes. See SendMessages.

Address Relay

Once added, the newly received IP addresses may be relayed to other nodes if the following conditions are met:

• The address timestamp is recent (within 10 minutes of the current time)
• The "addr" message contained 10 addresses or less
• fGetAddr=false for the sending peer. (See the code for details.)
• The address must be routable.
• Code:

if (addr.nTime > nSince && !pfrom->fGetAddr && vAddr.size() <= 10 && addr.IsRoutable())

• If this test is passed, then the code's next step is (see main.cpp for details):

// Use deterministic randomness to send to the same nodes for 24 hours at a time so the addrKnowns of the chosen nodes prevent repeats.

Peer connectivity

The connection thread (ThreadOpenConnections) chooses among the available addresses and makes connections, and disconnects nodes when appropriate.

Use of CSemaphore for outbound connections

The code uses a semaphore to manage the number of outbound connections (usually 8).

Most of the code dealing with the semaphore is in net.cpp.

When a connection is opened, the semaphore grant is passed to the CNode data structure. This allows the socket thread to release the semaphore when the time comes, with:

pnode->grantOutbound.Release()  // see net.cpp

The code CSemaphore grant(*semOutbound) will wait until there is a connection available.

Inbound connections: accepting and disconnecting

Inbound connections can be up to 125 total.

ThreadSocketHandler has the code that accepts inbound connections.

The socket thread loop:

• 1) Disconnects sockets that have the fDisconnect flag set on them (and have empty buffers)
• 2) Prepares all sockets for "select"
• 3) Calls "select", which is a system call which waits for activity on a set of sockets.

When the select() call returns, the node accepts any new connections, receives and sends on any ready sockets, and marks any inactive sockets to be disconnected (whether inbound or outbound).

Sockets are disconnected if:

* they are 60 seconds old and have not sent or received data.
* they have not sent or received data in the last 20 minutes (TIMEOUT_INTERVAL = 20*60) (or 90 minutes if peer is an old version)
* the socket overfills the buffer (see CNode::ReceiveMsgBytes- "Oversized message from peer=%i, disconnecting\n" in net.cpp)

Sockets & Messages

Socket Thread

The socket thread operates at the TCP layer.

It goes through an endless loop, reading and writing the sockets. (see net.cpp).

Its loop involves three basic activities:

1) Administrative work: disconnecting unused sockets, checking which sockets have data, adding sockets for new connections.

2) Receiving data: It reads the sockets that have data using the recv() system call and places that data into the peer's queue of CNetMessages. A CNetMessage organizes the data into two data streams - the message header and the message data (vRecv). The socket thread reads the buffer until it has processed all the messages from this particular peer.

3) Sending data: The message thread queued up messages-to-be-sent as vSendMsg objects, so the socket thread deserializes these objects and sends them using send(). (send() is a syscall and any incompatibilities across different operating systems are handled in the compat.h file.)

Message Thread

This is the program's main thread.

It operates primarily at the "business logic" level - validating transactions, managing the blockchain, etc.

In a sense, all of the node's activities take the form of processing an inbound message or preparing an outbound message.

Like the socket thread, this thread consists of a while(true) loop, processing inbound messages and queuing up outbound messages. Once the program's initialization is complete, this loop (see net.cpp:ThreadMessageHandler) is the program's high-level point of control.

The loop uses signals to notify main.cpp that there are messages waiting to be processed. The signal is picked up by ProcessMessages().

The use of signals has nothing to do with multi-threading; the signal is sent and picked up in the same thread. The use of signals was introduced in version 0.9 for the purpose of decoupling net.cpp from main.cpp. In version 0.8, the loop simply called the ProcessMessages() function. By changing to signals, the net.cpp code no longer needs to be aware of the processing code. Removing that dependency allows the code to avoid circular includes (since main.cpp requires knowledge of net.h.

The pull request introducing signals is PR 2154.

The commit removing the "main.h" dependency is here.

ProcessMessages (main.cpp)

ProcessMessages() is the entry point in main.cpp for almost all of the code that processes and validates transactions and blocks, etc.

It attempts to find a message start signature in the vRecv stream. If it finds a message start, it deletes everything prior to the start. Then it reads the header, extracts the message type, and calls ProcessMessage on the message.

ProcessMessage() is basically a large "switch" which takes action based on what type of message it is dealing with.

Often, in the course of processing a message, the code will push messages to the outbound queue. For example, when processing an incoming "getdata" message, the node pushes the outbound data into the queue.

SendMessages (main.cpp)

SendMessages() creates messages and queues them up in the peer's vSendMsg queue (a double-ended queue, or "deque" in C++). The vSendMsg objects are basically just serialized data.

SendMessages goes through various data structures looking for work to do. When it produces a message it calls the CNode->PushMessage, which queues the outbound data. (Note that there are many other places in the code that produce messages and call CNode->PushMessage; SendMessages() doesn't have any kind of exclusive license on placing messages in the outbound queue.)

Once the data is queued up by PushMessage, it sits and waits for the socket thread to come along.

The socket thread and the message thread use a peer-specific lock (node->cs_vSend) to coordinate access to the socket.

Locks

The main locks associated with the P2P aspect of the code are:

• cs_vNodes controls access to the CNode objects.
• cs_vSend controls access to the node's send buffer.
• cs_vRecvMsg controls access to the node's receiving buffer.
• cs_inventory

Denial-of-Service Prevention

DoS prevention is implemented by keeping track of misbehaving peers, and if they misbehave, banning them.

The DoS prevention framework was introduced in 2011, in Pull 517.

As summarized there:

---

The big idea: if a peer is sending you obviously wrong information, punish it by maybe dropping your connection to it, and ban it's IP address so it cannot immediately re-connect.

The probability of dropping the connection, and the length of the ban, depend on how wrong, and how potentially wasteful/damaging, the peer is. So sending an extra 'version' message is a minor transgression that is usually tolerated, sending an more than MAX_BLOCK_SIZE block is a major transgression.

Detailed how-it-works, using "I got a version message I wasn't expecting" as the specific example:

Getting a version message from a peer increases that peer's 'misbehaving' score by 10, and (assuming that is the peer's first bad behavior) gives it a 10% of being disconnected. If it is disconnected, then that peer's IP address is banned from connecting for a couple of hours. If it is not disconnected, then nothing happens unless the peer misbehaves again; if it does, then its chances of being disconnected go up, and the length of time it will be banned increases.

Misbehavior/ban information is stored only in memory, and information about misbehaving peers is never broadcast. Also, peers that are disconnected/banned are just dropped, there is no warning or reason sent.

--Gavin Andresen

---

Source: https://github.com/bitcoin/bitcoin/pull/517

Banned nodes

The set of banned nodes is in setBanned in net.cpp.

By default, a node is banned for 24 hours, though this can be configured with -bantime option.

See also

Bitcoin Core 0.11 (Ch 1): Overview
Bitcoin Core 0.11 (Ch 2): Data Storage
Bitcoin Core 0.11 (Ch 3): Initialization and Startup