Prysm attestation nodes suffered resource exhaustion.
During the periods that the event lasted, the validators were not able to work regularly.
From Prysm, an Ethereum consensus client, they revealed in a post-mortem analysis (after the event) the consequences of an incident that took place on December 4. This happened a day after the Fusaka update was deployed. One of these consequences was the loss of 382 ETH from rewards to Ethereum validators.
According to the report from Prysm, “almost all nodes [del cliente, no de la red Ethereum en general] experienced a resource exhaustion event when trying to process certain attestations. During this time, Prysm was unable to respond to validator requests in a timely mannerwhich resulted in the omission of blocks and attestations.
Facts about the Prysm incident
The incident with the Ethereum client lasted from the period (epoch) 411439 to 411480, adding a total of 42 periods.
The same report indicates that A total of 248 blocks were lost in the node registry of this client, which could not synchronize with the remaining nodes of the general network.
The specific problem was that the attestation validation logic in Prysm was compute intensive (reprocessing blocks and transitions from past epochs). This caused the depletion of resources. Under these conditions, many Prysm nodes fell behind or crashed, losing synchronization with the rest of the network.
However, this did not affect all Ethereum nodes, as other consensus clients continued to operate and kept the network running.
According to data, network participation during this event was as low as 75%. Validators lost approximately 382 ETH of attestation rewards by not being able to operate normally during the resource depletion event.
A bug identified on GitHub
Prysm gives a more complete technical explanation of what happened to the Ethereum validators. Clarifies that The bug was found in pull request 15965 from the repository.
Prysm beacon nodes received attestations from nodes that were possibly out of sync with the network. These attestations referred to the root of a block from the previous era. To validate them, Prysm attempted to recreate a beacon state compatible with the desynchronized node’s view of the chain. This resulted in multiple processing of blocks from previous epochs and costly epoch transition recalculations.
Security analysts at Prysm, Ethereum consensus client.
Ethereum was saved by customer diversity
He also clarified that the diversity of Ethereum validation clients reduced the impact of the incident: «Customer diversity prevented a notable impact on Ethereum users. A client with more than a third of the network would have caused a temporary loss of finality and more blocks lost. A client with errors and more than two thirds could end an invalid chain,” they said from Prysm.
However, they clarify that Lighthouse, another attestation client, could represent more than 56% of the network, and that this is dangerously close to the threshold at which A client error could terminate (definitively process on the blockchain) an invalid chain.
According to the incident documentation, client versions v7.0.1 and v7.1.0 contain a long-term fix for the bug that caused the loss of validator rewards.