BIP 0022

This page describes a BIP (Bitcoin Improvement Proposal). Please see BIP 2 for more information about BIPs and creating them. Please do not just create a wiki page.

  BIP: 22
  Title: getmemorypool
  Author: Luke Dashjr <luke+bip22@dashjr.org>
  Status: Draft
  Type: Standards Track
  Created: 28-02-2012

Abstract

This BIP describes a new JSON-RPC method for "smart" Bitcoin miners and proxies. Instead of sending a simple block header for hashing, the entire block structure is sent, and left to the miner to (optionally) customize and assemble.

Specification

JSON-RPC Method: getmemorypool

A JSON-RPC method is defined, called "getmemorypool". It accepts exactly one argument, which MUST be an Object of request parameters. If the request parameters include a "mode" key, that is used to explicitly select between "template", "proposal", and "submit" calls. Otherwise, it defaults to a submission if there is a "data" key (including the String-type argument mentioned above), and a template request if not.

Block Template Request

Block template creation can be influenced by various parameters:

template request
Key	Type	Description
capabilities	Array of Strings	SHOULD contain a list of the following, to indicate client-side support: "longpoll", "coinbasetxn", "coinbasevalue", "proposal", "serverlist", "workid", and any of the mutations
mode	String	MUST be "template" or omitted

getmemorypool MUST return a JSON Object containing the following keys:

template
Key	Type	Description
bits	String	the compressed difficulty in hexadecimal
curtime	Number	the current time as seen by the server (recommended for block time) - note this is not necessarily the system clock, and must fall within the mintime/maxtime rules
height	Number	the height of the block we are looking for
previousblockhash	String	the hash of the previous block, in big-endian hexadecimal
sigoplimit	Number	number of sigops allowed in blocks
sizelimit	Number	number of bytes allowed in blocks
transactions	Array of Objects	Objects containing information for Bitcoin transactions (excluding coinbase)
version	Number	always 1 (at least for bitcoin)
coinbaseaux	Object	data that SHOULD be included in the coinbase's scriptSig content. Only the values (hexadecimal byte-for-byte) in this Object should be included, not the keys.
coinbasetxn	String or Object	coinbase transaction, format specified by "tx:cb" parameter
coinbasevalue	Number	total funds available for the coinbase (in Satoshis)
workid	String	if provided, this value must be returned with results (see Block Submission)

Transactions Object Format

The Objects listed in the response's "transactions" key contains these keys:

template "transactions" element
Key	Type	Description
data	String	transaction data encoded in hexadecimal (byte-for-byte)
depends	Array of Numbers	other transactions (by 1-based index in "transactions" list) that must be present in the final block if this one is; if key is not present, dependencies are unknown and clients MUST NOT assume there aren't any
fee	Number	difference in value between transaction inputs and outputs (in Satoshis); for coinbase transactions, this is a negative Number of the total collected block fees (ie, not including the block subsidy); if key is not present, fee is unknown and clients MUST NOT assume there isn't one
hash	String	hash/id encoded in little-endian hexadecimal
required	Boolean	if provided and true, this transaction must be in the final block
sigops	Number	total number of SigOps, as counted for purposes of block limits; if key is not present, sigop count is unknown and clients MUST NOT assume there aren't any

Only the "data" key is required, but servers should provide the others if they are known.

Block Submission

When getmemorypool is called with a "data" key, and "mode" either set to "submit" or omitted, it is interpreted as a potential block (or share).

getmemorypool parameters
Key	Type	Description
data	String	MUST be hex-encoded block data
mode	String	MUST be "submit" or omitted
workid	String	if the server provided a workid, it MUST be included with submissions

This method MUST return either null (when a share is accepted), a String describing briefly the reason the share was rejected, or an Object of these with a key for each merged-mining chain the share was submitted to.

Optional: Long Polling

template request
Key	Type	Description
capabilities	Array of Strings	miners which support long polling SHOULD provide a list including the String "longpoll"
longpollid	String	"longpollid" of job to monitor for expiration; required and valid only for long poll requests

template
Key	Type	Description
longpollid	String	unique identifier for long poll request; MUST be omitted if the server does not support long polling
longpolluri	String	if provided, an alternate URI to use for longpoll requests
submitold	Boolean	only relevant for long poll responses: indicates if work received prior to this response remains potentially valid (default) and should have its shares submitted; if false, the miner may wish to discard its share queue

If the server supports long polling, it MUST include a "longpollid" key in block templates. It MAY supply the "longpolluri" key with a relative or absolute URI, which MAY specify a completely different resource than the original connection, including port number. If "longpolluri" is provided by the server, clients MUST only attempt to use that URI for longpoll requests.

Clients MAY start a longpoll request with a standard JSON-RPC request (in the case of HTTP transport, POST with data) and same authorization, setting the "longpollid" parameter in the request to the value provided by the server.

This request SHOULD NOT be processed nor answered by the server until it wishes to replace the current block data as identified by the "longpollid". Clients SHOULD make this request with a very long request timeout and MUST accept servers sending a partial response in advance (such as HTTP headers with "chunked" Transfer-Encoding), and only delaying the completion of the final JSON response until processing.

Upon receiving a completed response:

• Only if "submitold" is provided and false, the client MAY discard the results of past operations and MUST begin working on the new work immediately.
• The client SHOULD begin working on the new work received as soon as possible, if not immediately.
• The client SHOULD make a new request to the same long polling URI.

If a client receives an incomplete or invalid response, it SHOULD retry the request with an exponential backoff. Clients MAY implement this backoff with limitations (such as maximum backoff time) or any algorithm as deemed suitable. It is, however, forbidden to simply retry immediately with no delay after more than one failure. In the case of a "Forbidden" response (for example, HTTP 403), a client SHOULD NOT attempt to retry without user intervention.

Optional: Basic Pool Extensions

template request
Key	Type	Description
target	String	desired target for block template

template
Key	Type	Description
expires	Number	how many seconds (beginning from when the server sent the response) this work is valid for, at most
target	String	the number which valid results must be less than, in big-endian hexadecimal

Optional: Block Proposal

Servers may indicate support for proposing blocks by including a capability string in their original template:

template
Key	Type	Description
capabilities	Array of Strings	MAY contain "proposal" to indicate support for block proposal

If supported, a miner MAY propose a block to the server for general validation at any point before the job expires. This is accomplished by calling getmemorypool with two keys:

getmemorypool parameters
Key	Type	Description
data	String	MUST be hex-encoded block data
mode	String	MUST be "proposal"
workid	String	if the server provided a workid, it MUST be included with proposals

The block data MUST be validated and checked against the server's usual acceptance rules (excluding the check for a valid proof-of-work). If it is found to be in violation of any of these rules, the server MUST return a String giving the reason for the rejection (see example rejection reasons). If it is acceptable as-is, the server MAY return a true Boolean. Otherwise, the server may return a new block template (with or without changes); in this case, any missing keys are assumed to default to those in the proposed block or, if not applicable, the original block template it was based on.

It is RECOMMENDED that servers which merely need to track the proposed block for later share/* submissions, return a simple Object of the form:

{"workid":"new workid"}

Clients SHOULD assume their proposed block will remain valid if the only changes they make are to the portion of the coinbase scriptSig they themselves provided (if any) and the time header. Servers SHOULD NOT break this assumption without good cause.

Optional: Mutations

template request
Key	Type	Description
nonces	Number	size of nonce range the miner needs; if not provided, the server SHOULD assume the client requires 232

template
Key	Type	Description
fulltarget	String	the number which full results should be less than, in big-endian hexadecimal (see "share/*" mutations)
maxtime	Number	the maximum time allowed
maxtimeoff	Number	the maximum time allowed (as a moving offset from "curtime" - every second, the actual maxtime is incremented by 1; for example, "maxtimeoff":0 means "time" may be incremented by 1 every second)
mintime	Number	the minimum time allowed
mintimeoff	Number	the minimum time allowed (as a moving offset from "curtime")
mutable	Array of Strings	different manipulations that the server explicitly allows to be made (see Mutations)
noncerange	String	two 32-bit integers, concatenated in big-endian hexadecimal, which represent the valid ranges of nonces the miner may scan

If the block template contains a "mutable" key, it is a list of these to signify modifications the miner is allowed to make:

mutations
Value	Significance
coinbase/append	append the provided coinbase scriptSig
coinbase	provide their own coinbase; if one is provided, it may be replaced or modified (implied if "coinbasetxn" omitted)
generation	add or remove outputs from the coinbase/generation transaction (implied if "coinbasetxn" omitted)
time/increment	change the time header to a value after "time" (implied if "maxtime" or "maxtimeoff" are provided)
time/decrement	change the time header to a value before "time" (implied if "mintime" is provided)
time	modify the time header of the block
transactions/add (or "transactions")	add other valid transactions to the block (implied if "transactions" omitted from result)
prevblock	use the work with other previous-blocks; this implicitly allows removing transactions that are no longer valid, unless they are part of the "txrequired" count; it also implies adjusting "height" as necessary
Affecting the format of requests, but not the blocks themselves:
submit/hash	each transaction being sent in a request, that the client is certain the server knows about, may be replaced by its hash in little-endian hexadeciaml, prepended by a ":" character
submit/coinbase	if the "transactions" provided by the server are used as-is with no changes, only return the block header and coinbase transaction
submit/truncate	if the "coinbasetxn" and "transactions" provided by the server are used as-is with no changes, only return the block header
share/coinbase	same as "submit/coinbase", but only if the block hash is less than "target"
share/merkle	if the block hash is less than "target", but not less than "fulltarget", only return the block header, coinbase transaction, and merkle tree connecting that transaction to the root (in the form of repeated right-side SHA256 hashes)
share/truncate	same as "submit/truncate", but only if the block hash is less than "target"

Note that miners are NOT required to implement any of these mutations.

Format of Data for Merkle-Only Shares

The format used for submitting shares with the "share/merkle" mutation shall be the 80-byte block header, followed by the coinbase transaction, and then finally the little-endian SHA256 hashes of each link in the merkle chain connecting it to the merkle root.

Optional: Full Node Optimization

template request
Key	Type	Description
tx	String	format of response "transactions" key elements ("obj" per default; "hash" per full node optimization)

If the miner is running a full node, they may request to receive the "transactions" key in their template populated with transaction ids instead of the full data. This is done by specifying a "tx" key in the request set to "hash". Note that the "coinbasetxn" key in the template will remain in "obj" format.

FIXME: Describe gettransaction RPC method?

Optional: Logical Services

template
Key	Type	Description
serverlist	No	Array of Objects	list of servers in this single logical service

If the "serverlist" parameter is provided, clients MAY choose to intelligently treat the server as part of a larger single logical service.

Each host Object in the Array is comprised of the following fields:

serverlist element
Key	Type	Description
uri	String	URI of the individual server; if authentication information is omitted, the same authentication used for this request MUST be assumed
avoid	Number	number of seconds to avoid using this server
priority	Number	an integer priority of this host (default: 0)
sticky	Number	number of seconds to stick to this server when used
update	Boolean	whether this server may update the serverlist (default: true)
weight	Number	a relative weight for hosts with the same priority (default: 1)

When choosing which actual server to get the next job from, URIs MUST be tried in order of their "priority" key, lowest Number first. Where the priority of URIs is the same, they should be chosen from in random order, weighed by their "weight" key. Work proposals and submissions MUST be made to the same server that issued the job. Clients MAY attempt to submit to other servers if, and only if, the original server cannot be reached. If cross-server share submissions are desired, services SHOULD instead use the equivalent domain name system (DNS) features (RFCs 1794 and 2782).

Updates to the Logical Service server list may only be made by the original server, or servers listed with the "update" key missing or true. Clients MAY choose to advertise serverlist capability to servers with a false "update" key, but if so, MUST treat the server list provided as a subset of the current one, only considered in the context of this server. At least one server with "update" privilege MUST be attempted at least once daily.

If the "sticky" key is provided, then when that server is used, it should be used consistently for at least that many seconds, if possible.

A permanent change in server URI MAY be indicated with a simple "serverlist" parameter:

"serverlist":[{"uri": "http://newserver"}]

A temporary delegation to another server for 5 minutes MAY be indicated likewise:

"serverlist":[{"uri": "", avoid: 300}, {"uri": "http://newserver", "update": false}]

Optional: Template Tweaking

template request
Key	Type	Description
sigoplimit	Number or Boolean	maximum number of sigops to include in template
sizelimit	Number or Boolean	maximum number of bytes to use for the entire block

For "sigoplimit" and "sizelimit", negative values and zero are offset from the server-determined block maximum. If a Boolean is provided and true, the default limit is used; if false, the server is instructed not to use any limits on returned template. Servers SHOULD respect these desired maximums, but are NOT required to: clients SHOULD check that the returned template satisfies their requirements appropriately.

Optional: Backward Compatibility

Servers MAY choose to implement the following behaviours to remain backward compatible with pre-standard bitcoind 0.5 and 0.6 getmemorypool methods:

• If the parameters Object is omitted, it should be accepted as a template request, except that the format of the "transactions" key in the template changes. This same format can be requested explicitly by adding a "tx" key to the request set to "hex".
• If there is a single String argument instead of the parameters Object, it should be accepted as the "data" key for a submission. In this case, the response format is changed to either true (if a share is accepted) or false (if a share is rejected, regardless of reason).

template request
Key	Type	Description
tx	String	format of response "transactions" key elements ("obj" per default; "hex" per backward compat)

The format of the "transactions" key in block templates with the "hex" format should be:

template
Key	Type	Description
transactions ("hex" format)	Array of Strings	data for Bitcoin transactions (excluding coinbase) encoded in hexadecimal (byte-for-byte)

Appendix: Example Rejection Reasons

Possible reasons a share may be rejected include, but are not limited to:

share rejection reasons
Reason	Description
bad-cb-flag	the server detected a feature-signifying flag that it does not allow
bad-cb-length	the coinbase was too long (bitcoin limit is 100 bytes)
bad-cb-prefix	the server only allows appending to the coinbase, but it was modified beyond that
bad-diffbits	"bits" were changed
bad-prevblk	the previous-block is not the one the server intends to build on
bad-txnmrklroot	the block header's merkle root did not match the transaction merkle tree
bad-txns	the server didn't like something about the transactions in the block
bad-version	the version was wrong
duplicate	the server already processed this block data
high-hash	the block header did not hash to a value lower than the specified target
rejected	a generic rejection without details
stale-prevblk	the previous-block is no longer the one the server intends to build on
stale-work	the work this block was based on is no longer accepted
time-invalid	the time was not acceptable
time-too-new	the time was too far in the future
time-too-old	the time was too far in the past
unknown-user	the user submitting the block was not recognized
unknown-work	the template or workid could not be identified

Motivation

There is reasonable concerns about mining currently being too centralized on pools, and the amount of control these pools hold. By exposing the details of the block proposals to the miners, they are enabled to audit and possibly modify the block before hashing it.

To encourage widespread adoption, this BIP should be a complete superset of the existing centralized getwork protocol, so pools are not required to make substantial changes to adopt it.

Rationale

Why not just deal with transactions as hashes (txids)?

• Servers might not have access to the transaction database, or miners may wish to include transactions not broadcast to the network as a whole.
• Miners may opt not to do full transaction verification, and may not have access to the transaction database on their end.

Why allow servers to restrict the complete coinbase and nonce range?

• This is necessary to provide a complete superset of JSON-RPC getwork functionality, so that pools may opt to enable auditing without significantly changing or increasing the complexity of their share validation or mining policies.
• Since noncerange is optional (both for getwork and this BIP), neither clients nor servers are required to support it.

Why specify "time/*" mutations at all?

• In most cases, these are implied by the mintime/mintimecur/maxtime/maxtimecur keys, but there may be cases that there are no applicable minimums/maximums.

What is the purpose of the "prevblock" mutation?

• There are often cases where a miner has processed a new block before the server. If the server allows "prevblock" mutation, the miner may begin mining on the new block immediately, without waiting for a new template.

Why must both "mintime"/"maxtime" and "mintimeoff"/"maxtimeoff" keys be supported?

• In some cases, the limits may be unrelated to the current time (such as the Bitcoin network itself; the minimum is always a fixed median time)
• In other cases, the limits may be bounded by other rules (many pools limit the time header to within 5 minutes of when the share is submitted to them).

Is "target" really needed?

• Some pools work with higher targets, and should not be expected to waste bandwidth ignoring shares that don't meet it.
• Required to be a proper superset of getwork.
• As mining hashrates grow, some miners may need the ability to request a higher target from their pools to be able to manage their bandwidth use.

What is the purpose of "workid"?

• If servers allow all mutations, it may be hard to identify which job it is based on. While it may be possible to verify the submission by its content, it is much easier to compare it to the job issued. It is very easy for the miner to keep track of this. Therefore, using a "workid" is a very cheap solution to enable more mutations.

Why should "sigops" be provided for transactions?

• Due to the BIP 0016 changes regarding rules on block sigops, it is impossible to count sigops from the transactions themselves (the sigops in the scriptCheck must also be included in the count).

What is the purpose of the "hash" transaction list format?

• Non-mining tools may wish to simply get a list of memory pool transactions.
• Humans may wish to view their current memory pool.

What is the purpose of the "hex" transaction list format?

• Mainly backward compatibility with bitcoind 0.5 and 0.6.

To Be Resolved

• Since "data" is a key on the params Object now, perhaps "submit/hash" and "share/merkle" should use new keys for their non-block-format data?

Reference Implementation

• Eloipool
• bitcoind