Wallet protocol: Difference between revisions

From Bitcoin Wiki
Jump to navigation Jump to search
← Older edit
Casascius (talk | contribs)
Editors
618 edits
→‎Binary or plaintext: I would be supportive of either binary or plaintext, but this is an important consideration in favor of plaintext.
Luke-jr (talk | contribs)
Clerks, Administrators, Editors
3,074 edits
Undo revision 15211 by DataSurfer (talk)
 
(24 intermediate revisions by 8 users not shown)
Line 21: Line 21:
* Plaintext is easier for humans to debug, and avoids forming a dependency on a specific number of bits, endianness, or precision
* Plaintext is easier for humans to debug, and avoids forming a dependency on a specific number of bits, endianness, or precision
* Binary can include custom transactions as-is for transmission
* Binary can include custom transactions as-is for transmission
* Protobuf (binary) seems efficient and easy to use from all major languages
==== Binary wire formats ====
* JSON implementations tend to be buggy
* [http://code.google.com/p/protobuf/Protobuf Protobuf] seems efficient and easy to use from all major languages
 
==== Plaintext wire formats ====
* [http://www.json.org/ JSON] implementations tend to be buggy.


=== Cooperative mining ===
=== Cooperative mining ===
Line 28: Line 31:


=== UI bundling ===
=== UI bundling ===
For more secure use by ordinary end users, UIs and Wallets should support stream-based communication over standard I/O (stdio). This allows the UI to simply execute a dedicated Wallet, without requiring the user to setup Wallet-side authentication.  This also permits full compatibility with a hardware-based wallet located on a secured device (for example, a cryptographic hardware module, or a wallet device connected via RS232 or USB).
For more secure use by ordinary end users, UIs and Wallets should support stream-based communication over standard I/O (stdio). This allows the UI to simply execute a dedicated Wallet, without requiring the user to setup Wallet-side authentication.  This also permits full compatibility with a hardware-based wallet located on a secured device (for example, a cryptographic hardware module, or a wallet device connected via RS232 or USB). (Suggestion, if UI's could talk to any stream, including a child process's stdio, as well as a pipe or a device, the flexibility would be valuable)


= DRAFT 0 =
= DRAFT 0 =
Line 41: Line 44:
Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.
Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.


== Transport Layer ==
== Transport/Session Layer ==
Must support at least reliable in-order stream data.
Must support at least reliable in-order stream data.


Such as:
Such as:
* stdio (required for self-contained UI bundling)
* stdio (required for self-contained UI bundling)
* serial
* TCP
== Presentation Layer ==
=== Encryption ===
* TLS (recommended)
* TLS (recommended)
* TCP (NOT recommended; see TLS)
* SSH
* SSH


== Session Layer ==
=== Packetization ===
HTTP<ref name="RFC2616"/> or SPDY<ref name="SPDY"/>
Reliable, in-order packetization is required.
Packets should be request-oriented, while supporting:
* Fixed-length responses (MIME<ref name="RFC2045"/>-encoded)
* Persistent response streams (eg, HTTP chunked encoding)
* Optional: Full-duplex (two-way) communication (eg, WebSockets<ref name="WebSocket"/>)


== Presentation Layer ==
This supports:
HTTP-with-SSE<ref name="SSE"/> request semantics
* HTTP<ref name="RFC2616"/> with SSE<ref name="SSE"/> (and WebSockets<ref name="WebSocket"/>)
* GET for read-only access
* SPDY<ref name="SPDY"/>
* POST for methods (or complex read-only)
* 0MQ<ref name="0MQ-RFC2"/>, protobuf<ref name="protobuf"/>, and specialized .proto files
 
Request types:
* Optional: CONNECT for establishing a full-duplex stream
* GET for read-only access, always returning a fixed-length response
* POST for method calls
* PUT for uploads
* PUT for uploads
* HEAD for metadata
* HEAD for metadata


=== Formats ===
Note: Some protocols (eg, HTTP) do not allow complex structures in GET requests. An implementation using these protocols should allow silently upgrading GET to a POST with the same semantics.
 
Note: A logical CONNECT request may not map directly to the HTTP method by the same name. The current WebSocket specification uses a HTTP GET request.
 
=== MIME Formats ===
These content types may be specified in the Content-Type header, and Accept header.
These content types may be specified in the Content-Type header, and Accept header.
* application/x-www-form-urlencoded (required)
* application/x-www-form-urlencoded (required)
Line 72: Line 92:


== Application Layer ==
== Application Layer ==
=== Security ===
=== Authentication / security ===
Wallets may micro-manage security as they like, but the following are RECOMMENDED security levels:
Wallets may micro-manage security as they like, but the following are RECOMMENDED security levels:
{| class="wikitable"
{| class="wikitable"
| "Miner" || Can generate new addresses, get work, and report found blocks
| "Miner" || Can generate new addresses, get work, and report found blocks
|-
| Merchant || Can create a transaction (only with 'proposal' flag), but not send it
|-
|-
| Read-only || Can view accounts, addresses, and any public data, and combinations thereof (eg, balances)
| Read-only || Can view accounts, addresses, and any public data, and combinations thereof (eg, balances)
Line 85: Line 107:


Wallets MAY grant full access to username 'admin' with a null password, to requests over stdio.
Wallets MAY grant full access to username 'admin' with a null password, to requests over stdio.
Wallets SHOULD grant Merchant access to username 'merchant' will a null password, to requests made over direct (ie, physically local, like Bluetooth) connections.
User interfaces SHOULD monitor for new, unsigned transactions with the 'proposal' flag set, and prompt the user to sign or delete them.


=== Object/Query Schema ===
=== Object/Query Schema ===
Line 138: Line 163:
| {{Table Value Yes}} || {{Table Value No}} || {{Table Value No}}
| {{Table Value Yes}} || {{Table Value No}} || {{Table Value No}}
| {{Table Value Yes}}
| {{Table Value Yes}}
| -2 for rejected, -1 for unsent, 0 for broadcast, 1+ for number of confirms
|-
|-
| accepted
| accepted
Line 176: Line 202:
| {{Table Value Yes}}
| {{Table Value Yes}}
| 0xF9BEB4D9 for the main network; 0xFABFB5DA for testnet
| 0xF9BEB4D9 for the main network; 0xFABFB5DA for testnet
|-
| proposal
| {{Table Value No}} || {{Table Value Yes}} || {{Table Value No}}
| {{Table Value No}} || {{Table Value No}} || {{Table Value No}}
| {{Table Value No}}
|}
|}


Line 246: Line 277:


=== p2p protocol encapsulation ===
=== p2p protocol encapsulation ===
Optional: Establish a WebSocket connection to /p2p to speak direct p2p protocol
Optional: Establish a full-duplex connection to /p2p to speak direct p2p protocol


== Appendix A. Useful algorithms ==
== Appendix A. Useful algorithms ==
Line 284: Line 315:
== References ==
== References ==
<references>
<references>
<ref name="0MQ-RFC2">[http://rfc.zeromq.org/spec:2 The Size-Prefixed Blob Format], iMatix Corporation.</ref>
<ref name="protobuf">[http://code.google.com/apis/protocolbuffers/docs/encoding.html Protocol Buffers Encoding], Google.</ref>
<ref name="RFC2045">[http://tools.ietf.org/html/rfc2045 Multipurpose Internet Mail Extensions (MIME)], Freed & Borenstein, First Virtual.</ref>
<ref name="RFC2119">[http://tools.ietf.org/html/rfc2119 Key words for use in RFCs to Indicate Requirement Levels], S. Bradner. IETF.</ref>
<ref name="RFC2119">[http://tools.ietf.org/html/rfc2119 Key words for use in RFCs to Indicate Requirement Levels], S. Bradner. IETF.</ref>
<ref name="RFC2616">[http://tools.ietf.org/html/rfc2616 Hypertext Transfer Protocol -- HTTP/1.1], Fielding, et al, W3C.</ref>
<ref name="RFC2616">[http://tools.ietf.org/html/rfc2616 Hypertext Transfer Protocol -- HTTP/1.1], Fielding, et al, W3C.</ref>
Line 289: Line 323:
<ref name="SSE">[http://dev.w3.org/html5/eventsource/ Server-Sent Events], Ian Hickson, Google, Inc.</ref>
<ref name="SSE">[http://dev.w3.org/html5/eventsource/ Server-Sent Events], Ian Hickson, Google, Inc.</ref>
<ref name="SPDY">[http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2 SPDY Protocol], various. Google, Inc.</ref>
<ref name="SPDY">[http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2 SPDY Protocol], various. Google, Inc.</ref>
<ref name="WebSocket">[http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-06 The WebSocket protocol], Ian Hickson, Google, Inc.</ref>
<ref name="XML">[http://www.w3.org/TR/REC-xml Extensible Markup Language (XML) 1.0 (Second Edition)], Tim Bray, Jean Paoli, C. M. Sperberg-McQueen and Eve Maler, W3C.</ref>
<ref name="XML">[http://www.w3.org/TR/REC-xml Extensible Markup Language (XML) 1.0 (Second Edition)], Tim Bray, Jean Paoli, C. M. Sperberg-McQueen and Eve Maler, W3C.</ref>
</references>
</references>
[[Category:Developer]]
[[Category:Technical]]

Latest revision as of 03:35, 20 August 2011

This is a project to define a new wallet protocol addressing problems with the current JSON-RPC implementation in bitcoind. Please feel free to make edits. If you disagree with something, turn it into a "debate".

Explanation

This protocol aims to be a standard for communication between Wallets and User Interfaces. It can be used to access a Wallet remotely, or even locally, even if the Wallet and User Interface are produced by different or competing vendors. For example, you could use an Android User Interface from Google to control a QBitcoin Wallet.

Please see the Infrastructure page for an overview of where this protocol fits into the big picture.

Requirements

  • Never use human-formatted data (for example, use base bitcoin units, not Decimal BitCoins nor Tonal BitCoins, which should only be used in formatting for humans)
  • Don't require polling (for example, broadcast events for new transactions, or changes to current "work")
  • Cooperative mining (see below)
  • Allow calculating expected minimum fee for transactions, with either (wallet's choice) locking on the used inputs, or prebuilding the tx and error if it becomes invalid before sending
  • Easy to use in a RPC-like manner, for simple applications

Debates

Binary or plaintext

  • Binary is more efficient to parse and create
  • Plaintext is easier for humans to debug, and avoids forming a dependency on a specific number of bits, endianness, or precision
  • Binary can include custom transactions as-is for transmission

Binary wire formats

  • Protobuf seems efficient and easy to use from all major languages

Plaintext wire formats

  • JSON implementations tend to be buggy.

Cooperative mining

The protocol should support submitting completed blocks, such that it can get (just) addresses from an upstream pool to generate to, and send all completed blocks to that pool for verification and counting (and possibly submission to the network). This can reduce the pool's load.

UI bundling

For more secure use by ordinary end users, UIs and Wallets should support stream-based communication over standard I/O (stdio). This allows the UI to simply execute a dedicated Wallet, without requiring the user to setup Wallet-side authentication. This also permits full compatibility with a hardware-based wallet located on a secured device (for example, a cryptographic hardware module, or a wallet device connected via RS232 or USB). (Suggestion, if UI's could talk to any stream, including a child process's stdio, as well as a pipe or a device, the flexibility would be valuable)

DRAFT 0

Everything beyond this point is strictly DRAFT, should NOT be implemented, and is subject to being completely rewritten or modified!

Conformance requirements

All diagrams, examples, and notes in this specification are non-normative, as are all sections explicitly marked non-normative. Everything else in this specification is normative.

The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative parts of this document are to be interpreted as described in RFC2119[1]. For readability, these words do not appear in all uppercase letters in this specification.

Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("must", "should", "may", etc) used in introducing the algorithm.

Transport/Session Layer

Must support at least reliable in-order stream data.

Such as:

  • stdio (required for self-contained UI bundling)
  • serial
  • TCP

Presentation Layer

Encryption

  • TLS (recommended)
  • SSH

Packetization

Reliable, in-order packetization is required. Packets should be request-oriented, while supporting:

  • Fixed-length responses (MIME[2]-encoded)
  • Persistent response streams (eg, HTTP chunked encoding)
  • Optional: Full-duplex (two-way) communication (eg, WebSockets[3])

This supports:

  • HTTP[4] with SSE[5] (and WebSockets[3])
  • SPDY[6]
  • 0MQ[7], protobuf[8], and specialized .proto files

Request types:

  • Optional: CONNECT for establishing a full-duplex stream
  • GET for read-only access, always returning a fixed-length response
  • POST for method calls
  • PUT for uploads
  • HEAD for metadata

Note: Some protocols (eg, HTTP) do not allow complex structures in GET requests. An implementation using these protocols should allow silently upgrading GET to a POST with the same semantics.

Note: A logical CONNECT request may not map directly to the HTTP method by the same name. The current WebSocket specification uses a HTTP GET request.

MIME Formats

These content types may be specified in the Content-Type header, and Accept header.

  • application/x-www-form-urlencoded (required)
  • application/x-ripemd-160 (required)
  • application/x-sha (required)
  • multipart/mixed (required)
  • text/event-stream[5] (required for events over HTTP[4])
  • application/x-bitcoin (required wallet-side)
  • application/json[9] (recommended)
  • application/xml[10]

Application Layer

Authentication / security

Wallets may micro-manage security as they like, but the following are RECOMMENDED security levels:

"Miner" Can generate new addresses, get work, and report found blocks
Merchant Can create a transaction (only with 'proposal' flag), but not send it
Read-only Can view accounts, addresses, and any public data, and combinations thereof (eg, balances)
Financial Can create new transactions using the wallet's keys
Full/Admin Can administrate (download, destroy, etc) wallet keys

Wallets MAY grant full access to username 'admin' with a null password, to requests over stdio.

Wallets SHOULD grant Merchant access to username 'merchant' will a null password, to requests made over direct (ie, physically local, like Bluetooth) connections. User interfaces SHOULD monitor for new, unsigned transactions with the 'proposal' flag set, and prompt the user to sign or delete them.

Object/Query Schema

May be provided in multipart/mixed, JSON, XML, etc...

Field Name Blk Tx Accting Coin Key Acct Queryable Description
id Yes Yes Yes Yes Yes Yes Yes Unique data identifier; only guaranteed to remain the same for a single session
type Yes Yes Yes Yes Yes Yes Yes Type of data: 'key', 'tx', 'coin', 'block', 'account', 'accounting'
hash Yes Yes No No Yes No Yes
size Yes Yes ?? No Yes ?? Yes
time Yes Yes Yes No No No Yes
difficulty Yes No No No No No Yes
difficultyNext Yes No No No No No Yes
height Yes No No No No No Yes
confirmations Yes Yes ?? Yes No No Yes -2 for rejected, -1 for unsent, 0 for broadcast, 1+ for number of confirms
accepted ?? Yes ?? Yes No No Yes An unconfirmed, accepted tx, is part of the current mining merkle tree
related Yes Yes ?? ?? ?? ?? Yes Can have multiple entries (in JSON, an array value)
coinbase No Yes No Yes No No Yes
inputs No Yes ?? No No No Yes
outputs No Yes ?? No No No Yes
mine Yes Yes ?? Yes Yes ?? Yes
network Yes Yes ?? Yes Yes ?? Yes 0xF9BEB4D9 for the main network; 0xFABFB5DA for testnet
proposal No Yes No No No No No

Data

  • Arbitrary data can be accessed with the path /data/<id>
  • GET to fetch
  • To get only block header, use /data/<id>/header
  • To get only actual content (eg, no calculated values), use /data/<id>/raw
  • Optional: PUT to upload

Information

  • GET /state/<network>/connections
  • Mining info: (required for mining support)
    • GET /state/<network>/hashesPerSec (optional)
    • GET /state/<network>/merkle
  • GET /info/protocols
    • Returns a list of supported protocols/versions and their URIs

Requests

Address validation

  • GET /validate/address?network=<id>&address=<hash>

Calculate balance

  • GET /balance?<query-object>

Match transactions to be included in the balance. To ignore spends, use matching.

Data search

  • GET /dataSearch?<query-object>
  • POST /dataSearch allowed for more complex queries

NOTE: To get the id rather than the data itself, use a HEAD request and check the Location header.

Generate new address

  • POST /newAddress
    • Optional parameter: recycle (bool) -- allows wallet to reuse a spent change address; if not supported, wallet should silently ignore
    • Optional parameter: generate (bool) -- if true, make a new address on the spot, never use a preexisting one; if not supported, throw an error
    • Optional parameter: account -- associate new address funds with this account id

Monitor events

  • Always POST
New data available
  • Query

Move between accounts

  • POST /accounting/move
    • dest=<accountId>
    • query=<query-object> OR amount=<amount>
    • Optional: comment=<string>

Transaction creation

  • POST /tx/new
    • send (bool) -- If true, sign and broadcast the transaction immediately
    • fee (amount or range) -- If range, the wallet will try to calculate the minimum it expects the network to accept; if omitted, automatic fee range is unlimited unless bound by other factors (eg, input+strict)
    • Optional: fromAccount (account id)
    • Optional: input (multiple allowed of: coin id, OR coin id=>scriptSig)
    • output (multiple allowed of: address=>amount pairs, OR script=>amount)
    • strict (bool) -- If set, error rather than vary any parameters
    • Optional: comment=<string>

Transaction broadcast

  • POST /tx/send?id=<existing-tx-internal-id>

Shutdown wallet

  • POST /shutdown

Events

New Data Available

p2p protocol encapsulation

Optional: Establish a full-duplex connection to /p2p to speak direct p2p protocol

Appendix A. Useful algorithms

Counting confirmations

Because numerous (most) transactions change confirmation at the same event, the only way to do this efficiently, is to monitor for new block events. When a block arrives, increment the confirmation count of all transactions with at least 1 confirmation already. Then check the new block's introduced transactions (incrementing them from 0 to 1), and delete/invalidate any revoked transactions (eg, from a competing chain).

Appendix B. JSON-RPC equivalents

backupwallet

getaccount / getlabel

getaddressesbyaccount / getaccountaddress / getaddressesbylabel

getbalance

getblockcount / getblocknumber

getconnectioncount

getdifficulty

getgenerate

gethashespersec

getinfo

getnewaddress

getreceivedbyaccount / getreceivedbylabel

getreceivedbyaddress / getamountreceived

gettransaction

getwork

help

listaccounts

listreceivedbyaccount / listreceivedbylabel

listreceivedbyaddress / getallreceived

listtransactions

move

sendfrom

sendtoaddress

setaccount / setlabel

setgenerate

settxfee

stop

validateaddress

References

  1. ↑ Key words for use in RFCs to Indicate Requirement Levels, S. Bradner. IETF.
  2. ↑ Multipurpose Internet Mail Extensions (MIME), Freed & Borenstein, First Virtual.
  3. ↑ 3.0 3.1 The WebSocket protocol, Ian Hickson, Google, Inc.
  4. ↑ 4.0 4.1 Hypertext Transfer Protocol -- HTTP/1.1, Fielding, et al, W3C.
  5. ↑ 5.0 5.1 Server-Sent Events, Ian Hickson, Google, Inc.
  6. ↑ SPDY Protocol, various. Google, Inc.
  7. ↑ The Size-Prefixed Blob Format, iMatix Corporation.
  8. ↑ Protocol Buffers Encoding, Google.
  9. ↑ The application/json Media Type for JavaScript Object Notation (JSON), D. Crockford. JSON.org.
  10. ↑ Extensible Markup Language (XML) 1.0 (Second Edition), Tim Bray, Jean Paoli, C. M. Sperberg-McQueen and Eve Maler, W3C.
Retrieved from "https://en.bitcoin.it/w/index.php?title=Wallet_protocol&oldid=15297"