getblocktemplate

From Bitcoin Wiki
(Redirected from Libblkmaker)
Jump to navigation Jump to search

getblocktemplate is the new decentralized Bitcoin mining protocol, openly developed by the Bitcoin community over mid 2012. It supercedes the old getwork mining protocol.

Why change something that works?

Decentralization

The original getwork mining protocol simply issues block headers for a miner to solve. The miner is kept in the dark as to what is actually in this block, and has no influence over it. In effect, the authority of the miner to decide which transactions are accepted, etc, is all transferred blindly to the pool operator. A corrupt (or compromised) pool operator could use the combined hash power of all the miners to execute double spend attacks or other similar attacks.

getblocktemplate moves block creation to the miner, while giving pools a way to set down the rules for participation. While pools can do just as much as they could before by expressing it in these rules, miners can not be kept in the dark and are enabled to freely choose what they participate in mining. This improves the security of the Bitcoin network by making blocks decentralized again.

ASICs

The original getwork protocol only provides a single block header, which is sufficient for a total of about 4 GH of mining. With the "rollntime" extension, this can be extended to 4 GH *per second*, but even that is far from sufficient for the next generation of mining equipment (ASICs) which are capable of 1000 GH/s on the high end.

By moving block creation to the miners, they are enabled to create as much work as they need locally, thus overcoming this limitation.

Scalability

Due to scalability problems, bitcoind's JSON-RPC stack has not been able to keep up with the hashrates needed for solo mining today. Since getblocktemplate drastically reduces the load required to a single request per new block on the network, direct solo mining on bitcoind is again possible. Poolservers likewise benefit from having to meet much lower demands of miners who can make their own blocks.

Extensible

The original getwork protocol was designed in a way that was very incompatible with extensions. As a result, as new functionality was needed, extensions were "hacked in" out-of-band using HTTP headers. getblocktemplate is designed from the start to be flexible for future extensions, and the BIP 23 specification already covers how the established getwork extensions can be implemented cleanly, regardless of transport protocol.

How to use it

For miners

Mining software with GBT support:

To take advantage of getblocktemplate, you also need a compatible pool:

For pool operators

See also: Poolservers

If you are implementing your own pool server, see the section for poolserver development.

For developers

Mining software

Using libblkmaker (C library)

If your miner can include C libraries, you can harness libblkmaker to do all the GBT interpretation for you: all your miner needs to do then is handle the networking (libblkmaker can prepare the JSON for you) and ask libblkmaker for data (block headers to search). Note that libblkmaker does not provide a SHA256 implementation, and your miner needs to provide one for it to work. libblkmaker currently only supports the Jansson JSON library, but was designed such that it can easily be ported to others;

Using python-blkmaker

If your miner is written in Python, you can harness python-blkmaker, a native Python port of libblkmaker, to do all the GBT interpretation for you: all your miner needs to do then is handle the networking (python-blkmaker can prepare the JSON for you) and ask the blkmaker module for data (block headers to search). A short example Python script is included that finds a prepared share using CPU mining.

From scratch
Miner requests block template

To start participating, the miner contacts the pool and requests an initial template: 

{"id": 0, "method": "getblocktemplate", "params": [{"capabilities": ["coinbasetxn", "workid", "coinbase/append"]}]}

The server will respond with the full details needed to immediately begin mining blocks: 

{
 "error": null,
 "result": {
   "coinbasetxn": {
     "data": "0100000001000000000000000000000000000000000000000000000000000000
0000000000ffffffff1302955d0f00456c6967697573005047dc66085fffffffff02fff1052a01
0000001976a9144ebeb1cd26d6227635828d60d3e0ed7d0da248fb88ac01000000000000001976
a9147c866aee1fa2f3b3d5effad576df3dbf1f07475588ac00000000"
   },
   "previousblockhash": "000000004d424dec1c660a68456b8271d09628a80cc62583e5904f5894a2483c",
   "transactions": [],
   "expires": 120,
   "target": "00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
   "longpollid": "some gibberish",
   "height": 23957,
   "version": 2,
   "curtime": 1346886758,
   "mutable": ["coinbase/append"],
   "bits": "ffff001d"
 },
 "id": 0
}
How to build coinbase transaction

If the pool allows the "coinbase/append" mutation by including it in the "mutable" key, you can rebuild the coinbase transaction to append any data your miner wants, such as an extranonce - you can use as much space as you need so long as the coinbase data does not overflow the 100 byte hard limit. If the server does *not* allow "coinbase/append", or you don't care to change it, you can skip this step entirely :)

The coinbase data always begins after exactly 42 bytes of the coinbase transaction. The 42nd byte (that is, the byte immediately before the data) is the length of the data. So when/if you want to add on to the coinbase data, simply insert it 42+DataLen bytes into the transaction, and increment the 42nd byte by the length of your inserted data.

Python example: 

import binascii
coinbase = binascii.a2b_hex(template['coinbasetxn']['data'])
extradata = b'my block'
# The following can be done better in both Python 2 and Python 3, but this way works with both
origLen = ord(coinbase[41:42])
newLen = origLen + len(extradata)
coinbase = coinbase[0:41] + chr(newLen).encode('ascii') + coinbase[42:42 + origLen] + extradata + coinbase[42 + origLen:]
How to build merkle root

Collect your coinbase transaction (modified or not) at the front of the "transactions" list provided by the server. Apply a double-SHA256 hash to each transaction in the list.

Now, as long as the list has more than 1 hash remaining, go through each pair and hash them together. That is, concatenate the first two, double-SHA256 that, repeat for the next two, and so on. If you encounter an odd pair (that is, the hash list ends with a single item and no pairing), concatenate it with itself and hash that. Continue to do that until there is only one hash left: that is your merkle root.

Python example: 

import hashlib
def dblsha(data):
 	return hashlib.sha256(hashlib.sha256(data).digest()).digest()

txnlist = [coinbase] + [binascii.a2b_hex(a['data']) for a in template['transactions']]
merklehashes = [dblsha(t) for t in txnlist]
while len(merklehashes) > 1:
	 if len(merklehashes) % 2:
	  	merklehashes.append(merklehashes[-1])
 	merklehashes = [dblsha(merklehashes[i] + merklehashes[i + 1]) for i in range(0, len(merklehashes), 2)]
merkleroot = merklehashes[0]
How to build block header

Assemble the block header as laid out in the Bitcoin block hashing algorithm, using the data provided in the block template along with your very own merkle root. Note that miners are expected to check the "version" number, and should not create blocks with versions they do not understand unless the server instructs them to do so with the "version/force" or "version/reduce" mutations - you don't need to support those, but if you don't support the version the server has provided, understand that the server may reject submissions if they don't meet some unknown future rules.

Python example: 

import struct
blkheader = struct.pack('<I', template['version']) + \
            binascii.a2b_hex(template['previousblockhash']) + \
            merkleroot + \
            struct.pack('<I', template['curtime']) + \
            binascii.a2b_hex(template['bits']) + \
            b'NONC'
While you're mining

Since you're making all the blocks yourself, you really don't ever need to get another template... until it's invalid. Generally, it's a good idea to refresh more often anyway to get more transactions, but it's better to let the pool decide when to do that, since it knows what's changed in the meantime already.

If your template included a "longpollid" key, you can queue a request for a new template to be executed as soon as the pool decides you should change. This is the same as any other template request, except that you include the "longpollid" parameter that the pool provided in your request. If there is already a new template, the pool might respond immediately, so be sure not to depend on a delay!

So to amend the template request we used initially, now we use: 

{"id": 0, "method": "getblocktemplate", "params": [{
    "capabilities": ["coinbasetxn", "workid", "coinbase/append"],
    "longpollid": "some gibberish",
}]}
Submitting shares

When miner find the job which meets requested difficulty, it can submit the block to the server as a share: 

{"id": 0, "method": "submitblock", "params": [
     "020000003c48a294584f90e58325c60ca82896d071826b45680a661cec4d424d00000000
de6433d46c0c7f50d84a05aec77be0199176cdd47f77e344b6f50c84380fddba66dc47501d00ff
ff0000010001010000000100000000000000000000000000000000000000000000000000000000
00000000ffffffff1302955d0f00456c6967697573005047dc66085fffffffff02fff1052a0100
00001976a9144ebeb1cd26d6227635828d60d3e0ed7d0da248fb88ac01000000000000001976a9
147c866aee1fa2f3b3d5effad576df3dbf1f07475588ac00000000"
]}

To assemble the block data, simply concatenate your block header, number of transactions encoded in Bitcoin varint format, followed by each of the transactions in your block (beginning with the coinbase). If the server has listed "submit/coinbase" in its "mutable" key, you may opt to omit the transactions after the coinbase.

Python example: 

def varintEncode(n):
  if n < 0xfd:
    return struct.pack('<B', n)
  # NOTE: Technically, there are more encodings for numbers bigger than
  # 16-bit, but transaction counts can't be that high with version 2 Bitcoin
  # blocks
  return b'\xfd' + struct.pack('<H', n)
blkdata = blkheader + varintEncode(len(txnlist)) + coinbase
if 'submit/coinbase' not in template.get('mutable', ()):
  for txn in txnlist[1:]:
    blkdata += txn

Poolserver software

Recommended standards to start with (in order of importance):

Technical specifications

History

Getblocktemplate's origins trace back to forrestv's getmemorypool JSON-RPC method for bitcoind. He created it so that his pool (P2Pool) could piggy-back on bitcoind so as to avoid harming the Bitcoin network (up to this point, it mined only empty blocks that never confirmed transactions). Having been fighting with scalability problems in pushpool/bitcoind for months on his pool (Eligius), Luke-Jr set out to implement a fast pool server using getmemorypool to do its own block production (this became Eloipool, this first open source makes-its-own-blocks poolserver). Other poolservers also implemented block creation using getmemorypool over the months following.

At about the same time, interest in decentralizing pooled mining became a hot topic. While BitPenny had initially released its own decentralized mining proxy months prior, P2Pool's implementation became rapidly popular. Anyone involved in Bitcoin mining protocols could see the need to move control of block creation back into the hands of the miners. Unfortunately, both BitPenny and P2Pool had used very pool-specific proprietary protocols to implement their decentralization. On the other hand, getmemorypool was *almost* a perfect fit for the task - it just lacked support for the now-ubiquitous pooled mining extensions that had developed around the getwork protocol over time.

In February of 2012, Luke implemented and deployed a first draft of getmemorypool mining support in Eloipool (and on Eligius) along with a proof-of-concept getwork proxy (now known as gmp-proxy), adding revisions as needed to function as a general-purpose decentralized mining protocol. After he had confirmed it was working, he documented and proposed it on the Bitcoin development mailing list for review on February 28th, where discussion began on what was missing and what needed to be changed or clarified. During the following few months, a number of others, both developers and testers, provided constructive criticism and suggestions, which were integrated into the standard. Luke also actively encouraged participation in the development of the standard among pool operators and poolserver authors, especially as it became necessary to move forward into the ASIC "mining generation". Eventually, it was decided it would be best to rename it to the more appropriate "getblocktemplate" name and drop backward compatibility with getmemorypool for simplicity. The standard was split into two pieces and the technical specification can be found in BIP 22 and BIP 23.

See also


Bitcoin Core documentation
User documentation
Developer documentation
History & theory
Retrieved from "https://en.bitcoin.it/w/index.php?title=Getblocktemplate&oldid=66290#Using_libblkmaker_.28C_library.29"