Soft fork activation

Also covering BIP8 and BIP9

Soft fork activation describes the moment when a Bitcoin full node begins to enforce one or more additional consensus rules. These transitions introduce a coordination risk between nodes on the network, so developers have devoted much effort over the years to creating and improving soft fork activation mechanisms that can minimize the chance a problem will occur.

Soft forks allow the overall network to transition to new consensus rules even if not every node has adopted those rules. However, anytime different nodes enforce different rules, there’s a risk that blocks violating the nonuniform rules will be accepted by some nodes and rejected by other nodes, causing a consensus failure (chain split) that can result in double spends of confirmed transactions and a loss of confidence in the safety of the Bitcoin system. This is the main problem that activation mechanisms attempt to mitigate.

History

Proposals for new soft fork activation mechanisms are often designed to avoid problems experienced during previous soft forks, so this section attempts to provide an overview of notable past activation attempts.

[2009] Hardcoded height: consensus nLockTime enforcement

The earliest known soft fork was implemented in Bitcoin 0.1.6 (released November 2009) and was hardcoded to activate at block 31,000, which occurred on 22 December 2009. This hardcoded height activation mechanism was used for at least one other early soft fork when most development was done by Satoshi Nakamoto.

[2011] Hardcoded time and manual intervention: the BIP12 `OP_EVAL` failure

After Nakamoto left development, the first soft forking code merged into Bitcoin was BIP12 OP_EVAL. This planned to use a combination of a hardcoded time activation method and manual intervention if less than 50% of the network hashrate was prepared to enforce the change. Quoting from BIP12:

[…] new clients and miners will be coded to interpret OP_EVAL as a no-op until February 1, 2012. Before then, miners will be asked to put the string “OP_EVAL” in blocks that they produce so that hashing power that supports the new opcode can be gauged. If less than 50% of miners accept the change as of January 15, 2012 the rollout will be postponed until more than 50% of hashing power supports OP_EVAL (the rollout will be rejected if it becomes clear that a majority of hashing power will not be achieved).

The manual intervention may have been needed in this case as a serious vulnerability in OP_EVAL was discovered after the activation code was merged but before it was released. Although that specific bug was fixed, some developers feared that there might be additional problems with such a powerful new opcode and the soft fork attempt was abandoned.

[2012] Hardcoded time and manual intervention tried again: BIP16 P2SH

Several simplified replacements for OP_EVAL were then proposed (see BIPs 13/16, 17, 18, and 19, among other ideas), with BIPs 13/16 Pay-to-Script-Hash (P2SH) gaining the most traction among developers. P2SH used the same activation mechanism as OP_EVAL. The first planned activation was 1 March 2012, but by the February 15th polling date, fewer than half of the last 1,000 blocks had indicated their miners were ready to enforce the BIP16 rules by March. This led to a “fairly long alternative chain fork” (chain split) where miners still enforcing the March 1st start date rejected blocks by the majority of miners not enforcing those rules. A second poll of the thousand most recent blocks occurred on March 15th; this had sufficient support and so developers released Bitcoin 0.6.0 on March 30th with an activation date of April 1st.

[2012] Hardcoded time: BIP30 rejecting duplicate txids

As P2SH activation was being worked on, it was discovered that multiple transactions could be created with the same txid. By itself, this bug would only destroy the funds of a user who attempted to exploit it, but it could be combined with an oddity in Bitcoin’s merkle tree construction to break consensus between nodes (see CVE-2012-2459). The first soft fork fix for this was BIP30, which simply marked as invalid transactions with the same txid as another transaction with an unspent output. The fix was non-controversial among the development team and was activated by a simple hardcoded time included in the same Bitcoin 0.6.0 release that contained P2SH’s activation parameters.

[2012] IsSuperMajority (ISM): BIP34 coinbase prefixes

Despite BIP30 fixing the immediate problem with duplicate txids, Bitcoin developers knew they didn’t want to have to search an index of every previous transaction with unspent outputs each time a new transaction was received, so a second solution was developed that would eliminate the weakness that made duplicating txids practical. This was BIP34. For this change, developers used something similar to the miner signaling method from BIP16 P2SH, but this time miners who were ready for BIP34 were asked to increment their block nVersion value. More notably, developers automated enforcement for the new rules in the Bitcoin code so they could release soft-fork ready software while still waiting for miners to upgrade. The rules from BIP34 were implemented in a function called IsSuperMajority() (ISM), which initially included a single activation threshold that would start enforcement of BIP34’s new consensus rule:

75% rule: If 750 of the last 1,000 blocks are version 2 or greater, reject invalid version 2 blocks.

During development of the PR, it was decided to add a second activation threshold that would actually fix the underlying problem by requiring BIP34 to be used:

95% rule: If 950 of the last 1,000 blocks are version 2 or greater, reject all version 1 blocks.

One known problem with the rule to reject old version blocks was that, until all miners had upgraded, up to several invalid blocks would be produced each day (a 5% chance per block assuming exactly 95% of miners upgraded). Upgraded nodes that enforced ISM’s rules would reject those blocks, but old nodes and lightweight clients that didn’t know about the rules would accept those blocks. This made the network was more reliant than usual on miners refusing to build on invalid blocks.

[2015] ISM and validationless mining: the BIP66 strict DER activation

In September 2014, Pieter Wuille discovered a divergence in the way OpenSSL handled DER-encoded signatures on different platforms. This could be used, for example, to create a block that would successfully validate on Linux but fail on Windows—making it possible for an attacker to create a targeted chain split. Wuille and a few other developers privately developed the patch for a soft fork that would ensure all signatures used the same format. BIP66 was created for the change and advertised publicly as a step towards removing Bitcoin’s dependency on OpenSSL (which was an actual goal and was finally achieved in 2019). After BIP66 gained adequate support from users and developers—including from many people who didn’t know about the security vulnerability—it was released using the same ISM activation mechanism as BIP34 used, incrementing the block version to v3 and requiring all v2 or lower blocks be rejected when the 95% threshold was reached.

The 75% threshold was reached at block height 359,753. On 4 July 2015, the 95% threshold was reached at block 363,725 and all nodes running Bitcoin Core version 0.10.0 or later (or a compatible implementation) began enforcing the new rules. However, at block height 363,731, a non-upgraded miner produced a block that didn’t contain the correct block version and so was not valid under the new ISM activation rules. Other miners built on top of this invalid block, ultimately producing a chain with six invalid blocks. This meant non-upgraded nodes and many lightweight clients treated the 96 transactions in the first invalid block as having six confirmations even though they hadn’t, at that time, been confirmed even once in a valid block. Eventually developers were able to contact pool operators and get them to manually reset their software to return to the valid chain. A second event of this type occurred the next day, giving transactions three false confirmations. Happily, all regular transactions from both the six-block and three-block invalid chains were later confirmed in valid blocks, meaning that no regular users lost money.

That initial invalid block at height 363,731 was one of the roughly 5% of blocks per day that was expected to be invalid for no reason other than using an old version number. The chance of the next block being produced by another non-upgraded miner was also 5%, or a cumulative probability of 5% * 5% = 0.25%. Under the premise that 95% of miners had upgraded, the cumulative probability of six blocks in a row all being produced by outdated miners was 0.000002%—but it wasn’t extraordinarily bad luck that was the culprit here. What hadn’t been considered was miners performing validationless mining, an efficiency optimization where miners would build upon the header of a new block before they had finished receiving and verifying that block. Although validationless mining software could easily deal with invalid block versions in theory, that feature had not been implemented in the software used by the miners who created the five blocks descended from #363,731. Eventually, enough miners improved their validationless mining software or upgraded their nodes and the accidental chain splits related to the BIP66 activation stopped occurring.

In response to the problems of these July 2015 chain forks, developers redoubled their efforts on reducing the need for validationless mining, leading to improvements such as BIP152 compact block relay and the FIBRE software. Developers also began working on an improved activation mechanism, which would become the BIP9 protocol described subsequently.

[2015] ISM one last time: the BIP65 `OP_CHECKLOCKTIMEVERIFY` activation

A soft fork to add a new OP_CHECKLOCKTIMEVERIFY (CLTV) opcode to Bitcoin was proposed prior to the BIP66 strict-DER soft fork but was delayed by the urgency of fixing the OpenSSL vulnerability. This illustrated another one of the weaknesses of the ISM mechanism using incrementing version numbers—a miner who signals readiness for the latest proposal (version n) also implicitly signals readiness for all previous proposals (e.g. version n-1). This limited the ability to use ISM to coordinate multiple upgrades in parallel.

However, despite the problems at the end of BIP66’s activation, ISM was again used to avoid delaying BIP65 further. There were no activation problems this time.

[2016] BIP9 versionbits: the BIP68/112/113 relative locktime activation

A new activation mechanism documented in BIP9 was proposed to fix several of the issues with ISM:

● Penalizing miners unnecessarily: when ISM activated and the minimum block version was incremented, any miners who failed to increment their version would produce invalid blocks even if those blocks didn’t violate any of the soft fork’s other rules. For example, in the block that triggered the 4 July 2015 chain split, all the transactions followed the new soft fork rules—the only reason miners suffered over $50,000 in losses was that the upgrade required a block header contain a 3 and a non-upgraded miner used a 2.
● Impractical parallelization: with ISM, developers felt they needed to wait for one fork to activate before a second fork could be started.
● Inability to fail: ISM attempts didn’t have expiration dates. Once node software was released that waited for the activation signal, all nodes from that point needed to continue monitoring for that signal until activation occurred. There was no way to decide the soft fork was no longer needed.
● Unpredictable activation times: Not knowing the exact activation time in advance meant it was difficult for protocol developers, merchant system administrators, and mining pool operators to be available in the hours shortly after activation occurred in case there were any problems that needed a quick response.

BIP9 versionbits attempted to solve these problems by using the block header version field as a bitfield. Bits in the field were used for signaling only—never to reject a block as invalid—and they could be set in parallel. Measurements were taken only once every 2,016 blocks, minimizing the window during which a small percentage of hashrate would need to be lucky to project a false impression of 95% signaling. Finally, when the 95% signaling threshold was reached, there would be a 2,016 block (roughly two week) “lock in” period until activation occurred to allow people to prepare. If the signaling threshold wasn’t reached by a timeout date, the attempt ended, allowing the unused code to be removed from later node software releases.

This activation method was first used with the soft fork that added BIP68 consensus-enforced sequence numbers, BIP112 OP_CHECKSEQUENCEVERIFY, and BIP113 nLockTime enforcement by median time past. The fork quickly made it to the locked-in phase, and then proceeded automatically to activation.

[2016-7] BIP9, BIP148, and BIP91: the BIP141/143 segwit activation

The segwit soft fork was released with BIP9 activation parameters. A few miners quickly began signaling readiness but support peaked far below the 95% threshold. Some Bitcoin users felt miners were illegitimately delaying a useful new feature and went on to develop the mandatory activation that became BIP148. The final form of BIP148 specified rejecting any blocks that didn’t signal support for segwit starting on a certain date.

With the introduction of software implementing BIP148, there were three sets of nodes on the network—non-upgraded, BIP9/141 nodes, and BIP148/141 nodes—increasing the number of situations that could end in consensus failure. If miners failed to signal support for segwit, and most users continued to consider those miners’ blocks as valid, users of BIP148 might accept bitcoins that would be invalid from the perspective of most other Bitcoin users. Alternatively, if most users of Bitcoin followed the BIP148 rules—but miners still produced many BIP148-invalid blocks—those users who didn’t enforce BIP148 could accept bitcoins that the BIP148 users would consider invalid. Only if all users followed the same rules, or most mining hashrate followed the BIP148 rules, would safety be assured.

One way to reduce this risk was to give more users time to upgrade to a node that would mandate activation of segwit. BIP148 couldn’t do this since its goal of triggering the existing BIP9 segwit deployment meant it had to force miner signaling to start long before segwit’s BIP9 timeout date. As an alternative in case BIP148 failed to gain sufficient support, BIP149 was proposed to give users another year to upgrade. BIP149 never gained much public support, but it was the first proposal to use BIP8, which would see additional discussion in subsequent years.

After BIP148 began receiving significant public support, several miners, exchanges, and other businesses signed their support for a two-step proposal that started with activation of segwit in a way that would remain in consensus with nodes supporting BIP148. This first stage was proposed in BIP91, which was a modification of the BIP9 rules. Miners used a BIP9 versionbit to signal whether they would enforce a temporary rule to reject any blocks not signaling the BIP141/143 segwit versionbit. Compared to BIP9, the BIP91 threshold was reduced from 95% to 80% and its monitoring and lockin period lengths were reduced from 2,016 blocks to 336 blocks.

BIP91 locked in and activated. Subsequently, BIP141/143 locked in and activated. BIP148 mandatory signaling expired when BIP141/143 locked in. The second stage of the proposal from miners, exchanges, and other businesses required a hard fork; spokespersons for the signers eventually withdrew their proposal after vocal opposition from a large number of individual users and several businesses.

It remains a subject of debate exactly how much each of the above events, and other events occurring around the same time, actually contributed to segwit being activated.

Emergency activations

In several cases where a serious vulnerability was found in consensus code, developers released a fix without any activation conditions. This risked a consensus failure but it also eliminated the immediate vulnerability for each node as soon as it upgraded. Notable occasions included:

● Replace height with chainwork (July 2010): Bitcoin was originally implemented to follow the valid chain with the most blocks (“longest chain”). If every block had the same difficulty, that would be the chain with the most Proof-of-Work (PoW). But different blocks can have different difficulty, so the chainwork soft fork was released in Bitcoin 0.3.3 to follow the valid chain with the greatest cumulative proof of work.
● Eliminate script bypass bug (July 2010): Bitcoin originally combined the script that spent a UTXO (the scriptSig) with the script that secured that UTXO (the scriptPubKey) and evaluated them at the same time. This could allow someone to cause the script to terminate in success before its locking mechanism had been evaluated (e.g. before running OP_CHECKSIG to verify a signature was valid). The bug was originally reported as a scriptSig of OP_TRUE OP_RETURN allowing spending anyone’s bitcoins. Initially this was fixed in Bitcoin 0.3.6 by causing the OP_RETURN opcode to always result in failure and by placing a number of other limits on scripts. Although all of those changes were soft fork changes, the same code revision also made a hard forking change that would allow removing some of those limitations later. Despite those significant changes, the underlying problem of scriptSigs being able to tamper with the execution of scriptPubKeys remained and so a second soft fork implemented in Bitcoin 0.3.8 began evaluating scriptSigs independently of scriptPubKeys.
● Fix value overflow bug (August 2010): someone created a transaction that spent 0.5 BTC to two outputs worth 92,233,720,368.54277039 BTC. Bitcoin requires that the output amount be less than or equal to the input amount, but this was checked by adding the output values into a single 64-bit integer which can only hold a maximum value of 9,223,372,036,854,776 sats (about 92 billion BTC) before rolling over to negative numbers, starting with -9,223,372,036,854,776 sats. This meant it looked like the transaction spent a total of -0.1 BTC (negative 0.1 BTC). This bypassed another rule that forbid individual negative outputs—but not negative aggregate amounts—since it assumed the sum of any set of positive values would also be positive. This gave the person who created that transaction 184 billion BTC. The trick could be repeated indefinitely at no cost, allowing production of an unlimited number of bitcoins. Within hours, a soft fork fix restricting outputs to 21 million bitcoins was released in Bitcoin 0.3.10. This did require abandoning the chain with the overflowed transaction—a deliberate consensus failure, but one required to ensure Bitcoin remained useful.
● Temporarily fix BDB locks issue (March 2013): in early 2012, Bitcoin developers became aware that trying to make too many changes to the UTXO database (chainstate) at the same time could lead to exceeding one of the default capacity limits of the chainstate’s database. With the small size of Bitcoin blocks at the time, this was only found to occur during block chain reorganizations where transactions from multiple blocks were all processed at the same time. A simple solution was implemented: during a reorganization, only process the transactions from a single block at a time. Later, some people began asking miners to raise the optional default block size from 250 kilobytes to something larger. On 12 March 2013, one of those miners produced an almost 1 megabyte block with over 1,700 transactions—by far the largest block on Bitcoin up to that date—and this was enough to exceed the database capacity on many nodes, causing them to consider the block invalid even though it followed all of Bitcoin’s explicit consensus rules. Complicating the situation, a new version of Bitcoin Core had been released which used a different database engine that didn’t contain this limit, and it accepted the large block without problems—producing a consensus failure between the two different node versions. After quickly analyzing the situation, developers encouraged users to temporarily downgrade to the old version of Bitcoin (the version that rejected the large block) and then to upgrade to an emergency release that implemented a soft fork with a temporary block size limit of 500,000 bytes, which was pre-programmed to expire several months later after everyone had a chance to upgrade to the new database engine.

Future activations

After the problems of the segwit activation, several people came to work on the BIP8 proposal that proponents claims resolves several issues with BIP9:

● Allows mandatory activation: BIP8 began as a generalization of BIP148, allowing miners to voluntarily signal for activation for most periods of the activation deployment but requiring them to signal during the final period or risk creating invalid blocks. Later, a parameter LockinOnTimeout (LOT) was created to toggle this behavior; nodes using LOT=true will require signaling in the final period before the timeout; nodes using LOT=false will not require signaling but will still enforce the new rules if enough blocks are signaling.
● Heights instead of times: BIP9 starts and stops monitoring for activation signals based on an average of the times encoded by miners into recent blocks. It’s possible for miners to manipulate these times, which can stymie a LOT=true activation, so the BIP8 proposal instead uses block heights.

BIP8’s flexibility allowed it to be used to compare a variety of activation proposals for the proposed taproot soft fork, although detractors have leveled several criticisms against aspects of it, such as configurations that allow miners to refuse activation of proposals that have wide community support, configurations that encourage one group to co-opt the signaling mechanism used by a different group, configurations that require miners to make ineffectual changes to the blocks they produce, configurations that seem to give developers authority over consensus rules, and configurations that increase the risk of consensus failure. As of this writing, discussion in the context of taproot activation remains ongoing.

Other activation ideas have also been discussed, including probabilistic soft fork activation (sporks), multi-stage (“modern”) soft fork activation (MSFA), decreasing threshold activation (decthresh), a return to hardcoded height or time activations (flag days), and a short signaling period followed by an activation delay (speedy trial).

Primary code and documentation

BIP9
BIP8

2022

2021

2020

2019