Clearing Up Misconceptions About Full Nodes: Difference between revisions
No edit summary |
m →How to run a full node as your wallet: Update link to joinmarket |
||
| (32 intermediate revisions by 3 users not shown) | |||
| Line 7: | Line 7: | ||
Problem is, not all nodes actually have open ports that can be probed. Either because they are behind firewalls or because their users have configured them to not listen for connections. | Problem is, not all nodes actually have open ports that can be probed. Either because they are behind firewalls or because their users have configured them to not listen for connections. | ||
Nobody knows how many [[full node|full nodes]] there are, since many people don't know how to forward ports behind a firewall, and bandwidth can be costly, its quite likely that the number of nodes with closed ports is at least another several thousand. | Nobody knows exactly how many [[full node|full nodes]] there are, since many people don't know how to forward ports behind a firewall, and bandwidth can be costly, its quite likely that the number of nodes with closed ports is at least another several thousand. | ||
Nodes with open ports are able to upload blocks to new full nodes. In all other ways they are the same as nodes with closed ports. But because open-port-nodes can be measured and closed-port-nodes cannot, some members of the [[bitcoin]] community have been mistaken into believing that open-port-nodes | Nodes with open ports are able to upload blocks to new full nodes. In all other ways they are the same as nodes with closed ports. But because open-port-nodes can be measured and closed-port-nodes cannot, some members of the [[bitcoin]] community have been mistaken into believing that open-port-nodes represent the full count. | ||
= | ====Very roughly estimating the total node count==== | ||
(Original research by theymos on June 6, 2016 18:30 UTC.) | |||
According to getaddr.bitnodes.io, there are 5834 listening full nodes. Assuming they're all using default Bitcoin Core settings, they'll each provide 117 connection slots to the network on net (125 provided minus 8 consumed). SPV nodes typically use 4 connection slots, and full nodes typically use 8. So the network can support a maximum of around 170,644 SPV nodes or 85,322 non-listening full nodes at one time. This is roughly the upper limit for the number of wallets that are online and connected to the Bitcoin network at any one time. (If there were more people online at once than that, people would start seeing various issues.) This doesn't include wallets that don't actually connect to the Bitcoin network, of course. | |||
If I look at my long-running listening full node, I currently have incoming connections from 40 full nodes and 10 SPV nodes via the Internet, and 11 full nodes via my Tor hidden service. So if I ignore the hidden service connections because they are somewhat unusual, and I assume that my node is typical otherwise, it seems reasonable to make a very rough guess that there are at this moment (40*5834)/8 = 29,170 full nodes (5834 listening) and (10*5834)/4 = 14,585 lightweight nodes connected to the network. | |||
====Luke's script==== | |||
=Myth: There is no incentive to run nodes | Luke-jr has a script which attempts to count all full nodes: http://luke.dashjr.org/programs/bitcoin/files/charts/software.html | ||
=Myth: The number of these nodes matters and/or is too low.= | |||
Nodes with open ports are useful to the [[Network|bitcoin network]] because they help [[Network#Initial_block_download|bootstrap new nodes]] by uploading historical blocks -- they are a measure of the number of redundant copies of the blockchain available for synchronizing with. There has been no shortage of bandwidth capacity for simply syncing wallets from the available nodes, and if there were, bandwidth might be added by renting cloud servers. | |||
Trust, security, and privacy are what matters right now. [[full node|Full nodes]] are able to check that all of bitcoin's rules are being followed. Rules like following the [[Controlled_supply|inflation schedule]], no spending the same coin twice, no spending of coins that don't belong to the holder of the private key and all the other rules required to make [[bitcoin]] work (e.g. [[difficulty]]) | |||
Full nodes are required to make [[bitcoin]] trustless. No longer do you have to trust a financial institution like a bank or paypal, you can simply run software on your own computer. ''To put simply, the only node that matters is the one you use''. | |||
=Myth: There is no incentive to run nodes so the network relies on altruism= | |||
It is very much in the individual bitcoin's users rational self interest to run a [[full node]] and use it as their [[wallet]]. | It is very much in the individual bitcoin's users rational self interest to run a [[full node]] and use it as their [[wallet]]. | ||
====Trustless==== | |||
[[File:Petertodd-screenshot-mit2016-talk-gullible-lite-clients.png|500px|thumb|alt=Petertodd alt text.|An example of a lightweight client happily accepting a fake transaction. If the attacker had some hashpower then this transaction could be made to have [[confirmation|confirmations]] too.]] | |||
Running a full node as your [[wallet]] is the only way to know for sure that none of bitcoin's rules have been broken. Rules like no coins were spent not belonging to the owner, that no coins were spent twice, that no inflation happens outside of the schedule and that all the rules needed to make the system work are followed (e.g. [[Difficulty]].) All other kinds of wallet involve trusting a third party server. | |||
If bitcoin is digital gold, then a full node wallet is your own personal goldsmith who checks for you that received payments are genuine. You wouldn't accept a cash banknote or gold coin without checking it is genuine, the same applies for bitcoin. | |||
====Security==== | |||
All these checks done by full nodes also increase the security. There are many attacks possible against lightweight wallets that do not affect full node wallets. | All these checks done by full nodes also increase the security. There are many attacks possible against lightweight wallets that do not affect full node wallets. | ||
This is not just mindless paranoia, there have been real world examples where full node users were unaffected by turmoil in the rest of the bitcoin ecosystem. The 4th July 2015 accidental chain fork effected many kinds of wallets. Here is the wiki page on this event [[July_2015_chain_forks#Wallet_Advice]] | This is not just mindless paranoia, there have been real world examples where full node users were unaffected by turmoil in the rest of the bitcoin ecosystem. The 4th July 2015 accidental chain fork effected many kinds of wallets. Here is the wiki page on this event [[July_2015_chain_forks#Wallet_Advice]] | ||
Notice how node software updated months ago was completely unaffected by the fork. All other wallets required either extra [[confirmations]] or checking that the third-party institution was running the correct version. | Notice how node software updated months ago was completely unaffected by the fork. All other wallets required either extra [[confirmation|confirmations]] or checking that the third-party institution was running the correct version. A bitcoin business, like an exchange, marketplace or online store should always use a full node for security. | ||
Full nodes wallets are also currently the most private way to use Bitcoin, with nobody else learning which bitcoin [[address|addresses]] belong to you. All other [[Thin_Client_Security|lightweight wallets]] leak information about which [[address|addresses]] are yours because they must query third-party servers. The Electrum servers will know which [[address|addresses]] belong to you and can link them together. Despite [[BIP_0037|bloom filtering]], lightweight wallets based on [[BitcoinJ]] do not provide much privacy against nodes who connected directly to the wallet or wiretappers.<ref>http://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/ Privacy in BitcoinJ</ref> | ====Privacy==== | ||
Full nodes wallets are also currently the most private way to use Bitcoin, with nobody else learning which bitcoin [[address|addresses]] belong to you. All other [[Thin_Client_Security|lightweight wallets]] leak information about which [[address|addresses]] are yours because they must query third-party servers. The Electrum servers will know which [[address|addresses]] belong to you and can link them together. Despite [[BIP_0037|bloom filtering]], lightweight wallets based on [[bitcoinj|BitcoinJ]] do not provide much privacy against nodes who connected directly to the wallet or wiretappers.<ref>http://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/ Privacy in BitcoinJ</ref> | |||
For some use cases, such privacy may not be required. But an important reason to run a full node and use it as a wallet is to get the full privacy benefits. | For some use cases, such privacy may not be required. But an important reason to run a full node and use it as a wallet is to get the full privacy benefits. | ||
| Line 41: | Line 59: | ||
To get the benefits of running a full node, you must use it as your wallet, preferably on hardware you control. | To get the benefits of running a full node, you must use it as your wallet, preferably on hardware you control. | ||
Most people who do this do not use a full node as their wallet. Unfortunately because Bitcoin has a similar name to Bittorrent, some people believe that upload capacity is the most important thing for a healthy network. As I've explained above: bandwidth and connections are not a problem today | Most people who do this do not use a full node as their wallet. Unfortunately because Bitcoin has a similar name to Bittorrent, some people believe that upload capacity is the most important thing for a healthy network. As I've explained above: bandwidth and connections are not a problem today; trust, security and privacy are. | ||
=Myth: Running a full node is not recommended, most people should use a lightweight client= | =Myth: Running a full node is not recommended, most people should use a lightweight client= | ||
| Line 47: | Line 65: | ||
This was common advice in 2012, but since then the full node software has vastly improved in terms of user experience. | This was common advice in 2012, but since then the full node software has vastly improved in terms of user experience. | ||
If you cannot spare the disk space to store the [[blockchain]], you can [https://bitcoin | If you cannot spare the disk space to store the [[blockchain]], you can [https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.12.0.md#wallet-pruning enable pruning]. Altogether this should require less than 900MB of hard disk space. | ||
If you cannot spare the bandwidth to upload blocks to other nodes, [https:// | If you cannot spare the bandwidth to upload blocks to other nodes, [https://btcinformation.org/en/full-node#reduce-traffic there are number of options to reduce or eliminate the bandwidth requirement]. These include limiting connections, bandwidth targeting and disabling listening. There is also the option [https://bitcointalk.org/index.php?topic=1377345.0 -blocksonly], where the node will not download unconfirmed transaction and only download new blocks. This more than halves the bandwidth usage at the expense of not seeing unconfirmed transactions. | ||
[[Network#Initial_block_download|Synchronizing the blockchain]] for a new node has improved since 2012 too. Features like [https://bitcoin | [[Network#Initial_block_download|Synchronizing the blockchain]] for a new node has improved since 2012 too. Features like [https://github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-0.10.0.md#faster-synchronization] and libsecp256k1 have greatly improved the initial synchronization time. | ||
It can be further improved by setting -dbcache=3000 which keeps more of the [[Data_directory#chainstate_subdirectory|UTXO set]] in memory. It reduces the amount of time reading from disk and therefore speeds up synchronization. Tests showed that the entire [[blockchain]] can now be synchronized in [https://github.com/bitcoin/bitcoin/pull/6954#issuecomment-154993958 less than **3 and a half hours**] (Note that you'll need Bitcoin Core 0.12 or later to get all these efficiency improvements) [https://www.botbot.me/freenode/bitcoin-core-dev/2016-04-13/?msg=64033258&page=1 Another example with 2h 25m] | It can be further improved by setting -dbcache=3000 which keeps more of the [[Data_directory#chainstate_subdirectory|UTXO set]] in memory. It reduces the amount of time reading from disk and therefore speeds up synchronization. Tests showed that the entire [[blockchain]] can now be synchronized in [https://github.com/bitcoin/bitcoin/pull/6954#issuecomment-154993958 less than **3 and a half hours**] (Note that you'll need Bitcoin Core 0.12 or later to get all these efficiency improvements) [https://www.botbot.me/freenode/bitcoin-core-dev/2016-04-13/?msg=64033258&page=1 Another example with 2h 25m] | ||
| Line 59: | Line 77: | ||
I think every moderate user of bitcoin would benefit by running a full node and using it as their wallet. There are several ways to do this. | I think every moderate user of bitcoin would benefit by running a full node and using it as their wallet. There are several ways to do this. | ||
* Run a [https:// | * Run a [https://bitcoincore.org/en/download/ bitcoin-qt full node]. | ||
* Use wallet software that is backed by a full node (e.g. [[Armory]], [ | * Use wallet software that is backed by a full node (e.g. [[Armory]], [[JoinMarket]] | ||
* Use a lightweight wallet that connects only to your full node (e.g. [[ | * Use a lightweight wallet that connects only to your full node (e.g. [[GreenAddress]] connecting only to your node running at home, [[Electrum]] connecting only to your own Electrum server) | ||
** [[Electrum#Electrum_Personal_Server|Electrum Personal Server]] is designed specifically for this use case. | |||
So what are you waiting for? The benefits are many, the downsides are not that bad. The more people do this, the more robust and healthy the bitcoin ecosystem is. | So what are you waiting for? The benefits are many, the downsides are not that bad. The more people do this, the more robust and healthy the bitcoin ecosystem is. | ||
| Line 71: | Line 90: | ||
[https://www.reddit.com/r/BitcoinBeginners/comments/3eq3y7/full_node_question/ctk4lnd Pieter Wuille's explaination of full node trustlessness] | [https://www.reddit.com/r/BitcoinBeginners/comments/3eq3y7/full_node_question/ctk4lnd Pieter Wuille's explaination of full node trustlessness] | ||
[https://bitcoin.org/en/bitcoin-core/features/validation bitcoin.org's article on validation] | |||
[https://www.reddit.com/r/Bitcoin/comments/44rx5k/psa_clearing_up_some_misconceptions_about_full/ Reddit comments] | |||
=References= | =References= | ||
| Line 76: | Line 99: | ||
[[Category:Technical]] | [[Category:Technical]] | ||
[[Category:Educational]] | |||
Latest revision as of 18:35, 28 January 2020
This article was written in March 2016 to clear up misconceptions common in the bitcoin community at the time.
Myth: There are only about 5500 full nodes worldwide
This number comes from sites like this and it measured by trying to connect to every nodes on their open ports.
Problem is, not all nodes actually have open ports that can be probed. Either because they are behind firewalls or because their users have configured them to not listen for connections.
Nobody knows exactly how many full nodes there are, since many people don't know how to forward ports behind a firewall, and bandwidth can be costly, its quite likely that the number of nodes with closed ports is at least another several thousand.
Nodes with open ports are able to upload blocks to new full nodes. In all other ways they are the same as nodes with closed ports. But because open-port-nodes can be measured and closed-port-nodes cannot, some members of the bitcoin community have been mistaken into believing that open-port-nodes represent the full count.
Very roughly estimating the total node count
(Original research by theymos on June 6, 2016 18:30 UTC.)
According to getaddr.bitnodes.io, there are 5834 listening full nodes. Assuming they're all using default Bitcoin Core settings, they'll each provide 117 connection slots to the network on net (125 provided minus 8 consumed). SPV nodes typically use 4 connection slots, and full nodes typically use 8. So the network can support a maximum of around 170,644 SPV nodes or 85,322 non-listening full nodes at one time. This is roughly the upper limit for the number of wallets that are online and connected to the Bitcoin network at any one time. (If there were more people online at once than that, people would start seeing various issues.) This doesn't include wallets that don't actually connect to the Bitcoin network, of course.
If I look at my long-running listening full node, I currently have incoming connections from 40 full nodes and 10 SPV nodes via the Internet, and 11 full nodes via my Tor hidden service. So if I ignore the hidden service connections because they are somewhat unusual, and I assume that my node is typical otherwise, it seems reasonable to make a very rough guess that there are at this moment (40*5834)/8 = 29,170 full nodes (5834 listening) and (10*5834)/4 = 14,585 lightweight nodes connected to the network.
Luke's script
Luke-jr has a script which attempts to count all full nodes: http://luke.dashjr.org/programs/bitcoin/files/charts/software.html
Myth: The number of these nodes matters and/or is too low.
Nodes with open ports are useful to the bitcoin network because they help bootstrap new nodes by uploading historical blocks -- they are a measure of the number of redundant copies of the blockchain available for synchronizing with. There has been no shortage of bandwidth capacity for simply syncing wallets from the available nodes, and if there were, bandwidth might be added by renting cloud servers.
Trust, security, and privacy are what matters right now. Full nodes are able to check that all of bitcoin's rules are being followed. Rules like following the inflation schedule, no spending the same coin twice, no spending of coins that don't belong to the holder of the private key and all the other rules required to make bitcoin work (e.g. difficulty)
Full nodes are required to make bitcoin trustless. No longer do you have to trust a financial institution like a bank or paypal, you can simply run software on your own computer. To put simply, the only node that matters is the one you use.
Myth: There is no incentive to run nodes so the network relies on altruism
It is very much in the individual bitcoin's users rational self interest to run a full node and use it as their wallet.
Trustless
Running a full node as your wallet is the only way to know for sure that none of bitcoin's rules have been broken. Rules like no coins were spent not belonging to the owner, that no coins were spent twice, that no inflation happens outside of the schedule and that all the rules needed to make the system work are followed (e.g. Difficulty.) All other kinds of wallet involve trusting a third party server.
If bitcoin is digital gold, then a full node wallet is your own personal goldsmith who checks for you that received payments are genuine. You wouldn't accept a cash banknote or gold coin without checking it is genuine, the same applies for bitcoin.
Security
All these checks done by full nodes also increase the security. There are many attacks possible against lightweight wallets that do not affect full node wallets.
This is not just mindless paranoia, there have been real world examples where full node users were unaffected by turmoil in the rest of the bitcoin ecosystem. The 4th July 2015 accidental chain fork effected many kinds of wallets. Here is the wiki page on this event July_2015_chain_forks#Wallet_Advice
Notice how node software updated months ago was completely unaffected by the fork. All other wallets required either extra confirmations or checking that the third-party institution was running the correct version. A bitcoin business, like an exchange, marketplace or online store should always use a full node for security.
Privacy
Full nodes wallets are also currently the most private way to use Bitcoin, with nobody else learning which bitcoin addresses belong to you. All other lightweight wallets leak information about which addresses are yours because they must query third-party servers. The Electrum servers will know which addresses belong to you and can link them together. Despite bloom filtering, lightweight wallets based on BitcoinJ do not provide much privacy against nodes who connected directly to the wallet or wiretappers.[1]
For some use cases, such privacy may not be required. But an important reason to run a full node and use it as a wallet is to get the full privacy benefits.
Myth: I can just set up a node on a cloud server instance and leave it
To get the benefits of running a full node, you must use it as your wallet, preferably on hardware you control.
Most people who do this do not use a full node as their wallet. Unfortunately because Bitcoin has a similar name to Bittorrent, some people believe that upload capacity is the most important thing for a healthy network. As I've explained above: bandwidth and connections are not a problem today; trust, security and privacy are.
Myth: Running a full node is not recommended, most people should use a lightweight client
This was common advice in 2012, but since then the full node software has vastly improved in terms of user experience.
If you cannot spare the disk space to store the blockchain, you can enable pruning. Altogether this should require less than 900MB of hard disk space.
If you cannot spare the bandwidth to upload blocks to other nodes, there are number of options to reduce or eliminate the bandwidth requirement. These include limiting connections, bandwidth targeting and disabling listening. There is also the option -blocksonly, where the node will not download unconfirmed transaction and only download new blocks. This more than halves the bandwidth usage at the expense of not seeing unconfirmed transactions.
Synchronizing the blockchain for a new node has improved since 2012 too. Features like [1] and libsecp256k1 have greatly improved the initial synchronization time.
It can be further improved by setting -dbcache=3000 which keeps more of the UTXO set in memory. It reduces the amount of time reading from disk and therefore speeds up synchronization. Tests showed that the entire blockchain can now be synchronized in less than **3 and a half hours** (Note that you'll need Bitcoin Core 0.12 or later to get all these efficiency improvements) Another example with 2h 25m
How to run a full node as your wallet
I think every moderate user of bitcoin would benefit by running a full node and using it as their wallet. There are several ways to do this.
- Run a bitcoin-qt full node.
- Use wallet software that is backed by a full node (e.g. Armory, JoinMarket
- Use a lightweight wallet that connects only to your full node (e.g. GreenAddress connecting only to your node running at home, Electrum connecting only to your own Electrum server)
- Electrum Personal Server is designed specifically for this use case.
So what are you waiting for? The benefits are many, the downsides are not that bad. The more people do this, the more robust and healthy the bitcoin ecosystem is.
Further Reading
Measuring Decentralization - Truthcoin
Pieter Wuille's explaination of full node trustlessness
bitcoin.org's article on validation
References
- ↑ http://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/ Privacy in BitcoinJ