This week’s newsletter requests testing for release candidates of Bitcoin Core and LND, describes a discussion about UTXO snapshots for fast initial syncing of nodes, and provides regular coverage of bech32 sending support and notable merges in popular Bitcoin infrastructure projects.
Help test Bitcoin Core 0.18.0 RC3: The third Release Candidate (RC) for the next major version of Bitcoin Core has been released. This may be the final test version, so we encourage organizations and expert users to test it promptly if they want to ensure any possible regressions are caught before release. Please use this issue for reporting feedback.
Help test LND 0.6-beta RC3: the first, second, and third RCs for the next major version of LND were released this week. Testing by organizations and experienced LN users is encouraged to catch any regressions or serious problems that could affect users of the final release. Open a new issue if you discover any problems.
Discussion about an assumed-valid mechanism for UTXO snapshots: When Bitcoin Core developers are preparing a new major release, one developer selects the hash of a recent block on the best block chain. Other well-known contributors check their personal nodes and ensure that hash is indeed part of the best block chain, and then add that hash to the software as the “assumed valid” block. When new users start Bitcoin Core for the first time, the program then defaults1 to skipping script evaluation in all transactions included in blocks before the assumed valid block. The program still keeps track of the bitcoin ownership changes produced by each transaction in a index called the Unspent Transaction Output (UTXO) set. Although reviewing each historic ownership change still takes time, simply skipping script checking reduces initial sync time by about 80% according to tests. Gregory Maxwell, who implemented the assumed valid feature, has argued that, “Because the validity of a chain history is a simple objective fact, it is [easy] to review this setting.”
This week James O’Beirne started a thread on the Bitcoin-Dev mailing list about taking a hash of the UTXO set at a particular block, having multiple well-known contributors verify they get the same hash, and then having freshly installed Bitcoin Core nodes default to using that hash to download the exact same UTXO set. This would allow a newly-started node to skip not only scripts but all block chain data before the assumed valid block, perhaps reducing the time requirements to start a node today by 95% or more (and certainly more as the block chain continues to grow). The verification of older blocks and transactions could then be done in the background after the user is already using their node, eventually giving them the same security as a user who disabled this feature. This is an old idea and is part of the motivation for research into other techniques such as fast updatable UTXO commitments and automatic levelDB backups.
Discussion mainly revolved around whether or not this is a good idea. Arguments in favor of it include it making starting a new node much easier and that it doesn’t seem to change the trust model for anyone who already trusts the peer review of their development team. Arguments against it include a fear that fast initial syncs with an assumed valid UTXO set would disguise the fact that block size increases make complete initial syncing from scratch much more expensive; if block sizes increased too much, it could become impossible for anyone of modest means to ever trustlessly verify Bitcoin’s UTXO state, forcing new users to trust existing users.
Bech32 sending support
Week 4 of 24. Until the second anniversary of the segwit soft fork lock-in on 24 August 2019, the Optech Newsletter will contain this weekly section that provides information to help developers and organizations implement bech32 sending support—the ability to pay native segwit addresses. This doesn’t require implementing segwit yourself, but it does allow the people you pay to access all of segwit’s multiple benefits.
The code is written using Node.js-style module inclusion syntax, so the first step is to compile it into code we can use in the browser. For that, we install a browserify tool:
sudo apt install node-browserify-lite
Then we compile it into a standalone file:
browserify-lite ./segwit_addr_ecc.js --outfile bech32-demo.js --standalone segwit_addr_ecc
Followed by including it in our HTML:
For convenience, we’ve included that file on the web
version of this newsletter, so you can follow along
with the rest of this example by simply opening the developer console in
your web browser. Let’s start by checking a valid address. Recall from
last week that we provide the network identifier when checking an
bc for Bitcoin mainnet):
>> segwit_addr_ecc.check('bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4', 'bc') error: null program: Array(20) [ 117, 30, 118, … ] version: 0
We see above that, just like last week, we get back the witness version and the witness program. The presence of the version field, plus the lack of an error, indicate that this program decoded without any checksum failure.
Now we replace one character in the above address with a typo and try checking that:
>> segwit_addr_ecc.check('bc1qw508d6qejxtdg4y5r4zarvary0c5xw7kv8f3t4', 'bc') error: "Invalid" pos: Array [ 21 ]
This time we get back the description of the error (the address is invalid because it doesn’t match its checksum) and a position. If we place the addresses above each other with each position marked, we see that this “21” identifies the location of the specific error:
1x 2x 0123456789012345678901 >> good='bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4' >> typo='bc1qw508d6qejxtdg4y5r4zarvary0c5xw7kv8f3t4' ^
What if we make an additional replacement to the typo address and try again?
>> segwit_addr_ecc.check('bc1qw508d6qejxtdg4y5r4zarvary0c5yw7kv8f3t4', 'bc') error: "Invalid" pos: Array [ 32, 21 ]
We get two locations. Once again, when we compare the addresses to each other, we see this has identified both incorrect characters:
1x 2x 3x 012345678901234567890123456789012 >> good='bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4' >> typo='bc1qw508d6qejxtdg4y5r4zarvary0c5yw7kv8f3t4' ^ ^
There’s a limit to how many errors the
check() function can specifically identify.
After that it can still tell that an address contains an error, but it
can’t identify where to look in the address for the error. In that
case, it’ll still return the address as invalid but it won’t return the
>> segwit_addr_ecc.check('bc1qw508z6qejxtdg4y5r4zarvary0c5yw7kv8f3t4', 'bc') error: "Invalid" pos: null
In the case where there are other problems with the address, the
field will be set to a more descriptive message that may or may not
include a position of the error. For example:
>> segwit_addr_ecc.check('bc1zw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4yolo', 'bc') error: "Invalid character" pos: Array [ 43 ]
You can review the source for a complete list of errors.
Although we spent a lot of time looking at errors in this mini tutorial, we’ve hopefully shown how easy it is to provide nice interactive feedback to users entering bech32 addresses on a web-based platform. We encourage you to play around with the interactive demo to get an idea of what your users might see if you make use of this bech32 address feature.
Notable code and documentation changes
Notable changes this week in Bitcoin Core, LND, C-Lightning, Eclair, libsecp256k1, and Bitcoin Improvement Proposals (BIPs). Note: all merges described for Bitcoin Core and LND are to their master development branches; some may also be backported to their pending releases.
Bitcoin Core #15596 updates the
sendmanyRPC to remove the
minconfparameter, which didn’t function the way people expected. Now the wallet defaults are always used. Those defaults are not to spend outputs received from other people until they are confirmed and to optionally allow spending unconfirmed change outputs from yourself depending on the
spendzeroconfchangeconfiguration setting. This is the same way the more commonly used
sendtoaddressRPC has always worked.
LND #2885 changes how LND attempts to reconnect to all of its peers when coming back online. Previously it attempted to open connections to all its persistent peers at once. Now it spreads the connections over a 30 second window to reduce peak memory usage by about 20%. This also means that messages that are sent on a regular interval, such as pings, do not happen at the same time for all peers.
LND #2740 implements a new gossiper subsystem which puts its peers into two buckets, active gossiper or passive gossiper. Active gossipers are peers communicating in the currently normal way of sharing all of their state with your node; passive gossipers are peers from which you will only request specific updates. Because most active gossipers will be sending you the same updates as all other gossipers, having more than a few of them is a waste of your bandwidth, so this code will ensure that you get a default of 3 active gossipers and then put any other gossipers into the passive category. Furthermore, the new code will try to only request updates from one active gossiper at a time in round-robin fashion to avoid syncing the same updates from different nodes. In one test described on the PR, this change reduced the amount of gossip data requested by 97.5%.
LND #2313 implements code and RPCs that allow LND nodes to use static channel backups. This is based on the Data Loss Protection (DLP) protocol implemented in LND #2370 to allow backing up a single file containing all of your current channel state at any point and then enabling restoring from that file at any later point to get your remote peer to help you to close any of those channels in their latest state (excluding unfinalized routed payments (HTLCs)). Note: despite the “static” in this feature’s name, this is not like an HD wallet one-time backup. It’s a backup that needs to be done at least as often as each time you open a new channel—but that’s much better than the current state where you may not be able to recover any funds from any of your channels if you lose data. Further improvements to backup robustness are mentioned in the PR’s description. See the description of LND #2370 in Newsletter #31 for more details on how DLP-based backup and recovery works. Getting this major improvement to backups merged was one of the major goals for upcoming LND version 0.6-beta.
The assumed valid mechanism is enabled by default but can be disabled by starting Bitcoin Core with the configuration parameter
noassumevalid). This is will allow your node to completely verify every transaction in the block chain to ensure it follows all consensus rules. Note that this will have no effect on blocks your node has already processed, so if you want to verify scripts in old blocks, you will need to have this option enabled from the first time you ever use your node or you will need to need to restart Bitcoin Core one time with the
reindex-chainstateconfiguration option. For pruned nodes, reindexing requires redownloading all pruned blocks. ↩