Participants
@Harry
@Ben Burgess
@Alec Thomas
@Lee Bousfield
@Joshua Colvin
@Nina Rong
Summary
- A bug in the batch poster caused it to post batches which reverted.
- The Sequencer was not affected or disrupted and continued to process transactions throughout.
- Reports stating that the Sequencer ran out of funds are incorrect. The Arbitrum Nitro funding mechanism consists of two wallets: the batch poster wallet and the "gas-refunder" wallet. The batch poster wallet is responsible for posting transactions on-chain and has a relatively low ETH balance that is programmatically refilled. The gas-refunder is a smart contract with a high ETH balance, and it automatically refills the batch poster wallet when it successfully posts batches.
Status
Resolved with followup actions
Root Cause (RC)
The batch poster upgrade to version v2.1.0-beta.1 introduced an issue from https://github.com/OffchainLabs/nitro/pull/1640. When the batch poster attempted to update its L1 state, it filled in the new L1 block number, but used the nonce as of the previous L1 block number which resulted in the two becoming desynchronized. As an even worse edge case, when first updating the L1 state on startup, the previous L1 block number was nil which resulted in the latest block being queried for the nonce.
Resolution (RS)
The problematic state was cleared from the Redis storage of the batch poster, and it was restarted using an older version that did not have the root cause issue.
Timeline (UTC)
- On June 7th of 2023 at 06:04:53 AM, due to a temporary issue with the L1 node, the batch poster failed to update its view of the L1 state. Due to the root cause issue, it continued trying to query for the state as of its previous L1 view's block number. This meant that even after the ephemeral L1 node issue resolved itself, the batch poster kept trying to query for the state as of the old L1 block number, whose state the L1 node no longer had since it is not an archive node.