P2Pool code documentation

Ignore this page for the minute. Is just a scratch pad for documenting the p2pool code. Feel free to add or correct errors if you are familiar with code.

Will tidy up once initial pass done.

Overview

The code consists of a number of main tasks.

1. Communicate with bitcoind. This mainly gets work - i.e. the latest block and all transactions that need including in the next block. The other communication is for checking the payout address is OK, and publishing newly found blocks to the bitcoin network.
2. Store and track p2pool shares. We need to track what shares have been published by us and other users. We need this as to calculate the block generation transaction we need to know who created shares in the previous few days.
3. Communicate with miners. We need to respond the their get work requests with a block header to try to "solve". Also need to handle LongPolling which is a way to inform miners instantly when they need to stop solving the current block header and solve a new one.
4. Communicate with p2pool network. We need to connect to other members of the pool and publish and receive shares.
5. Web status pages. Web pages that give users details on status of pool.

Files

Files

Files Name	Description
p2pool/main.py	Main startup and initialisation code
p2pool/data.py	P2Pool data structures
p2pool/networks.py	Definitions of P2Pool networks (eg. Bitcoin, Bitcoin-testnet, Litecoin, ...)
p2pool/p2p.py	P2Pool P2P protocol implementation
p2pool/web.py	P2Pool web interface
p2pool/bitcoin/	Code related to Bitcoin and its clones. Contains nothing specific to P2Pool
p2pool/util/pack.py	Handling of over the wire data structures
p2pool/util/variable.py	Code to allow monitoring of when variables change and triggering events
p2pool/util/forest.py	Contains Tracker class and other classes to track shares and which share is head/tail.
p2pool/util/skiplist.py]	Base class for TrackerSkipList and DistanceSkipList.

p2pool/main.py

Makes extensive use of twisted.defer. This allows it to "yield" to allow long running network code to complete. Read up on Python Generators and this before progressing!

Contains main startup code.

Function run()

This is the initially executed function.

1. Parses arguments
2. Reads user/password from bitcoin config file
3. Sets up log file
4. Sets up logger that reports errors to http://u.forre.st/p2pool_error.cgi (If you are concerned this is a privacy issue add --no-bugreport to command line.)

Finally it adds the main function to the Twister Reactor and start the reactor. (i.e. runs the function main!).

Function main()

This does all the startup tasks.

1. Tests connection to bitcoind.
2. Prints hash of latest block to show bitcoind is up to date.
3. Tests connection to p2pool network.
4. Gets address to use for payout either from file or bitcoind.
5. Validates address and checks local bitcoind owns it.
6. Create a "tracker" and loads know shares from files in data/bitcoin/sharesX where X is a number.
7. poll_bitcoind then gets work from bitcoind (i.e. block header to hash). Does this by calling getwork function explained below.
8. The work_poller() function then polls bitcoind every 15 seconds for new work.
9. Check for work from peers. This is new code to try to reduce stales. It gets new block headers from peers if they arrive before they arrive from bitcoind.
10. Set up merged work for merged mining.
11. Sets up combined work.
12. Sets up Longpoll to trigger when current_work changes (transitions).
13. Creates Node class that handles connections to other p2pool nodes (see p2p.py also).
14. Read p2pool node address from addrs file else use bootstrap addresses.
15. Create node object and start it connecting/sending/receiving data.
16. Setup loop to save shares to disk every 60 seconds.
17. Create tunnel through routers using upnp if enabled.
18. Start listening for workers using WorkerBridge Class (e.g. cgminers).
19. Create web_root and start web server. This is the monitoring web pages. (see web.py)
20. Start IRC connection for announcing blocks.
21. Start Status process that output to screen data every 3 seconds.

Function main/submit_block

This is a critical function that watches for tracker.verified shares being added. If they meet the difficulty requirements it then submits the block to bitcoind using BOTH the p2p connection and the JSONRPC connection.

The _ function that calls it also send the share that contains the block solution out over the p2pool network to propagate the solution asap and so reduce orphans.

Class main/WorkerBridge

This is the main communication between p2pool process and the workers (mining processes running cgminer or similar).

get_user_details

This allows clients to connect. Also parses out if this client wants a higher than normal pseudoshare difficulty.

get_work

This is the main method. Creates a new share to solve from current data. Create a "BlockAttempt" which is a bitcoin header for the miner to find a nonce for. returns the BlockAttempt and a got_response function.

got_response is called when miner finds a solution. It checks that the solution is valid. If below block target we found a block. If below share target we found a share.

More debugging in here would be useful to find the "10%" issue

Function getWork()

This connects to the bitcoind process using the jsonrpc proxy. It calls the getmemorypool function (see Bitcoin json-rpc API document and getmemorypool document) This returns all the data needed to create a new block (except the nounce obviously!).

getwork then unpacks this into a dictionary containing the header info, the transactions , the merkel branch/root and the coinbase flags. This is everything that a miner needs to calculate a valid nonce/block.

Class Node

This also covers code in p2pool/p2p.py. It is dependant on p2pool/util/p2protocol.py which is the Twisted.protocol class that handles low level network communication and passes on messages to the handle_xxx methods of the Client and Server factories and Node class. This class is the main class that handles all the p2pool connections and message handling. It is the core of the p2p network for p2pool.

Is initialised with best_share_hash variable so it can update/monitor it, port, store of peer addresses.

Initially it starts up the client factory that connects to other nodes and a server factory that allows incoming connections.

Then it checks it has enough node addresses, if not asks random peers to send it 8 more addresses.

Method Node.handle_shares

Passes new shares onto tracker. Update peer_heads Calls compute_work if a new share.

p2pool/data.py

Contains the main data structures used in p2pool. These are:

hash_link_type

Serialized SHA256 engine state, used to prove that a coinbase transaction contains some data near the end without sending the entire transaction.

hash_link_type

Field	Type	Description
state	String(32)	?
extra_data	String(0)	Comments say this is a hack
length	VarInt	?

small_block_header

Bitcoin block header, excluding the merkle root. Included in shares, where the merkle root is computed implicitly from the coinbase transaction and the merkle branch.

small_block_header

Field	Type	Description
version	VarInt	?
previous_block	None or Int(256)	?
timestamp	Int(32)	?
bits	FloatingInteger(32)	?
nonce	Int(32)	?

share_data_type

Information contained within a share that is only relevant to P2Pool and that the client has control over (i.e. its value isn't fixed by the protocol rules).

share_data_type

Field	Type	Description
previous_share_hash	None or int(256)	?
coinbase	VarString	coinbase script
nonce	Int(32)	internal P2Pool nonce
pubkey_hash	Int(160)	pubkey hash of Bitcoin address that this share's payouts will go to
subsidy	Int(64)	total block value
donation	Int(16)	donation to authors. 0 = 0%, 65535 = 100%
stale_info	String(32)	Flag that tells whether the node that generated this share's last share was orphaned or dead. Used to compute pool statistics. Enum (orphan, dao, unk253, unk252...???
desired_version	VarInt	Vote for P2Pool version. Used to trigger upgrade warnings on other nodes

share_info_type

Information contained within a share that is only relevant to P2Pool

share_info_type

Field	Type	Description
share_data	share_data_type (see above)	?
far_share_hash	none or int(256)	Hash of 100th parent of this share. Not currently used for anything, but has applications in timestamping and more secure sharechain bootstrapping
max_bits	Float Int	Maximum share target allowed
bits	Float Int	Share target this share was mined at
timestamp	Int(32)	?

share_common_type

Data common to both of the following two types of shares.

share_common_type

Field	Type	Description
min_header	small_block_header_type	Block header of mined share
share_info	share_info_type	see above
ref_merkle_link	ComposedType(branch:List(Int), index:VarInt)	Merkle branch from hash(ref_type) to ref hash. Currently always empty in generated shares, but could be used by new merged mining implementations
hash_link	hash_link_type	Lets you compute the generation transaction hash as a function of this and the ref hash

share1a_type

This is a share that isn't a block solution.

share1a_type

Field	Type	Description
common	share_common_type	?
merkle_link	ComposedType(branch:List(Int), index:VarInt)	Merkle branch from generation transaction hash to block header's merkle root

share1b_type

This is a share that is a block solution.

share1b_type

Field	Type	Description
common	share_common_type	?
other_txs	List(bitcoin_data.tx_type)	List of transactions included in the block besides the generation transaction

ref_type

Internal data structure, hashed to determine the "reference hash" at the end of generation transaction.

ref_type

Field	Type	Description
identifier	str(4)	Value different for every P2Pool instance (Bitcoin, Bitcoin-testnet, Litecoin...)
share_info	share_info_type	see above

Class share

Class used to store shares.

Method generate_transaction

Returns the share_info and the generation transations. These are all the transactions that pay the other p2pool users the generated coins and fees. It include 2 special transactions

1. The donation sent to the developers.
2. A transaction that is not valid, but just has a value of zero and the hash of the share info. This is to prove that when a share is found it was whilst looking for a real p2pool block. This is computed from the ref_type data structure.

p2pool/util/pack.py

I think this handles all the binary data types used in the bitcoin protocol to send data over the network wire. These are nasty as very low level and many big endian/little endian complications. The p2pool network protocol uses these also. Do not think you need to really understand this unless making changes at this low level.

p2pool/__init__.py

At bottom has DEBUG flag. Change to true to get more output. (running p2pool with --debug does this) Other than that just returns version number from git if it can.