Historical record of incidents for filecoin
Report: "F3 Mainnet Activation - Informational Update"
Last updateF3 Mainnet activation succeeded and there have been no observed adverse impacts on the network. We’re marking this as completed, but will certainly engage on any automated alerts or user reports in FIL Slack or GitHub.
F3 will be activated at epoch 4920480, which is 2025-04-29T10:00:00Z.This “Maintenance” is an FYI. Engagement is not needed from the community (but you are certainly welcome to follow along.)Learn more in the "F3 Comes to Mainnet - Launch Date and Preparation Guide": https://medium.com/@filoz/the-f3-journey-f3-comes-to-mainnet-launch-date-and-preparation-guide-2addc240aaf2, or follow along with activation steps in this Github Issue: https://github.com/filecoin-project/f3-activation-contract/issues/22
Report: "Filecoin Network Version 25 Teep Upgrade"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
The Mainnet Network Upgrade to nv25 has been delayed by four days due to a bug identified in a testnet. The upgrade is now scheduled for epoch 4878840, which corresponds to 2025-04-14T23:00:00Z.For more details on the required version updates, please refer to the community announcement: https://github.com/filecoin-project/community/discussions/74?sort=new#discussioncomment-12729231
The Filecoin Mainnet will be performing a Network Upgrade to nv25, codenamed Teep, at epoch 4867320, which is 2025-04-10T23:00:00ZPlease see the community announcement for more information: https://github.com/filecoin-project/community/discussions/74?sort=new#discussioncomment-11369195
Report: "Filecoin CalibrationNet nv25 - Patch Upgrade"
Last updateScheduled maintenance is currently in progress. We will provide updates as necessary.
An expedited patch network upgrade for the Calibration network is scheduled for epoch 2558014, corresponding to 2025-04-07T23:00:00Z. This upgrade is necessary due to a bug discovered following the nv25 network upgrade on Calibration.Please see the community announcement for more information: https://github.com/filecoin-project/community/discussions/74?sort=new#discussioncomment-12729231
Report: "Filecoin CalibrationNet Version 25 - Teep Upgrade"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The Filecoin CalibrationNet will undergo an upgrade to network version 25, codenamed Teep, at epoch 2523454, on 2025-03-26T23:00:00Z.Please see the community announcement for more information regarding implementation release candidates, updates, and more: https://github.com/filecoin-project/community/discussions/74?sort=new#discussioncomment-11369195
Report: "Node out of sync / partial network outage"
Last updateLotus patches are out now (https://github.com/filecoin-project/lotus/releases/tag/v1.28.3; https://github.com/filecoin-project/lotus/releases/tag/v1.29.1) and the impacted nodes are recovering. Implementers are considering this incident as resolved now, and we will keep monitoring the network. Thank you to the Filecoin node operators for all your support for a speedy recovery!
We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.
We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.
Many Filecoin nodes are not syncing due to "failed to verify beacon". Lotus implementers are investigating the issue with the Drand implementers, and we will provide updates in #fil-network-announcement and https://github.com/filecoin-project/lotus/issues/12467 as soon as we have more information.
Report: "Filecoin chain halted after height 336458"
Last updatePlease see this post on [**Resolving the Dec 19 Chain Halt: Cause, Impact, & Take Aways **](https://filecoin.io/blog/resolving-the-chain-halt/)for cause and impact of this outage. More specific post mortems WIP from the actors & infra teams to describe specific learnings and mitigations for the future.
If you’re just catching up: check the upgrade and recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) to get back in sync. Miners have upgraded & the chain is now recovering smoothly. 🎉🙏💪 We don’t expect further action is required, but will continue monitoring. Expect degraded chain quality. We should expect the network to be somewhat degraded for the next few hours as miners continue to upgrade and power recovers (currently ~34% upgraded). We recommend pausing all non-critical messages until chain capacity is restored to normal levels - BaseFee may be volatile as the network recovers. Full recovery ETA. Conservatively, we will deem the network to be fully recovered & stable 2x finality from when the chain resumed. This means 1,400 epochs (11.6 hrs) from 337120 (2020-12-19 23:20 UTC). So, conservatively, full recovery at epoch 338520 (2020-12-20 11:00 UTC). For custodians, exchanges, wallets: The full recovery ETA is very conservative, and the chain is expected to be fully functional long before. Our recommendation for you would be to keep transfers out disabled until that time or until you gain comfort with chain status -- at least for larger amounts. Your own internal transfers, trading, and other fully off-chain activity should be fine.
We are continuing to monitor for any further issues.
The network has restarted and progressed to the expected block height (337120 @ 3:20pm PT). As more power joins the upgraded chain, the network will continue to stabilize. Check back for an update here when >50% of power is in sync.
Lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) has been released. Please upgrade ASAP and see the recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) for more information/troubleshooting.
The Filecoin network is halted with most nodes stuck at height 336459 with mismatched state compute error due to indeterminate behavior. We have found the root cause and are testing the fix. Next Step: Miners will need to upgrade to lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) with the fix when it is released. Once miners upgrade, the chain will start to catch up.
Report: "This is an example incident"
Last updateWhen your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.
Empathize with those affected and let them know everything is operating as normal.
As you continue to work through the incident, update your customers frequently.
Let your users know once a fix is in place, and keep communication clear and precise.
Report: "Testing email integration for DRAND testnet"
Last updateWe are currently investigating this issue.