Is Helium Down Right Now? Check if there is a current outage ongoing.

The downstream storage provider has resolved their outage and we are closing this issue. That said, a new version of the Helium Wallet app will be released in the next 24 hours (assuming app store reviews are expedited) that should prevent this particular impact in the future.

Core developers have identified the downstream infrastructure provider and while they are working on a fix, we are also applying some mitigation. This will require an upgrade of the Helium Wallet app. We are starting the process to update metadata, the app, and submit it to the App Stores for review.

The core developers are investigating an outage in Hotspot rewards claiming in the Helium Wallet app. This appears to be due to a downstream dependency on an infrastructure provider that helps index Helium data for clients like the Helium Wallet.

The Helium Foundation has deployed additional Solana RPC infrastructure in order to support the growing Helium Wallet user base.

Due to Solana network congestion, transactions may fail or be delayed. For example, Hotspot reward claims may take several minutes to complete. Keep the wallet app open after hitting claim button or try again later.

This incident has been resolved.

Due to high Solana congestion, transactions may fail or be delayed. Please retry failed transactions.

We identified the root cause as the Foundation Console's router failing to reboot as expected after erroring. We have manually rebooted said router and packets are now correctly being received.

As reported by the community, The Foundation Console has been dropping device uplinks since approximately 1:30 UTC. We are currently looking into it.

We join the Solana community in thanking validator operators and core developers for their firefighting over the past few hours.

Block production has resumed on Solana. We will continue to monitor over the next few hours.

Solana validators have begun a coordinated restart with an updated release version. You can follow status here: https://stakewiz.com/restart

Engineers from across the Solana ecosystem are readying a new validator software release, which includes a patch to address an issue which caused the cluster to halt.

As noted on status.solana.com, the Solana network is currently experiencing a chain halt which may affect some activity for Helium network users. Notably, deployers will not be able to onboard or move Hotspots and token holders may not be able to perform transfers or participate in other on-chain activity. Rest assured that Solana validators and core developers are working on resuming block production and that Helium oracle activity is not affected.

The automated job that was supposed to end the November 13th epoch (19,674) at UTC midnight stalled due to a dropped transaction. The job continued to wait for confirmation on a transaction that was never picked up by Solana. The epoch was manually ended by Core Developers at 15:30 UTC. No further action is needed. The team will be improving alerting to ensure we are notified earlier when epochs fail to close. Additionally, they will be repairing the automated job to include a timeout on transaction confirmations.

This incident has been resolved.

A workaround has been provided by the RPC provider and Helium Wallet Application functionality has been restored. We will continue monitoring to ensure stability moving forward.

We have identified a major outage with Cloudflare causing downstream impact to the Solana RPC backing the Helium Wallet App. Wallet App operations are degraded at this time.

Based on the monitoring, all packet routers have fully stabilized.

The rollout is complete. The core team will monitor for 1 hour before the final status update.

The core team is rolling out a new packet router release • HPR utility for more debugging visibility on a route level • additional grpc stream management The expected downtime is 15-20 minutes.

The Packet Routers have been stable for the 2.5 hours. Marking this incident resolved and will follow up with post mortem.

Starting 8:18PM PT, the U.S. Packet Routers are offline again. The core team hasn't identified the root cause yet as all server metrics were healthy up til the crash, we are going to reboot and remove a load optimization constraint.

The Packet Routers have been stable for the past hour. The core team will keep monitoring the incident.

The U.S. Packet Routers have been rebooted. In the meantime, the core team is investigating the outage and will continue the update here.

The issue has been identified and a fix is being implemented.

Starting 5:19PM PT, the U.S. Packet Routers are offline. The core team is investigating the outage and will continue the update here.

The Helium Packet Routers started experiencing intermittent outage from 2:38PM PT due to misconfigured open file limit, the issue was resolved on 3:43 PT. The team is monitoring the incident as there still seem to be some loose ends. Will continue the update here.

This incident has been resolved.

The core team is performing load testing on the Singapore packet routers. We will be closely monitoring the infra metrics and data transfer health.

This incident has been resolved.

We believe we've identified and fixed the issue with the Configuration Service and will be monitoring it. Hotspots that have stopped Beaconing due to the outage should resume Beaconing within a couple of hours.

After continued monitoring we've determined that the Configuration Service, which tells Hotspots which LoRaWAN region they operate in, has a database issue and is not serving locations properly to some Hotspots. This will prevent these Hotspots from Beaconing, and in some cases, transferring data. We are investigating reloading the Configuration Service's data.

The core developers have noticed a drop off in Beaconing activity and believe that the cause is an earlier problem with the Configuration Service. The developers believe that Hotspots should resume their normal Beaconing some 4 hours after this notice and will continue to monitor the situation for changes.

This issue is now resolved. Realms and the Helium Wallet are back to normal operations.

In furtherance to the Solana RPC degraded performance, we have identified additional impacts affecting Realms and are continuing to coordinate with the third party provider on resolution.

We have identified an issue with the Solana RPC and are coordinating with the third party provider on resolution. Helium Wallet App performance may be degraded at this time.

Wallet developers have confirmed that this particular indexing issue was resolved overnight.

Wallet developers continue to monitor as we've heard some reports of improvements in the Claim Hotspot Rewards feature of the Helium Wallet App. As a reminder, your rewards can be safely claimed once the provider has fully returned.

Wallet developers have identified community reported issues with the Claim Hotspot Rewards feature of the Helium Wallet App. It is temporarily down for a large number of Hotspots due to an upstream RPC provider experiencing issues with indexing. Your rewards can be safely claimed once the provider is back online.

Based on the monitoring, all packet routers have fully stabilized.

The core team has also upgraded the U.S. packet routers, we will continue monitoring the performance.

The core team is going to patch the packet routers in U.S. with disk space optimization, and we will monitor the performance.

The issue was caused by the packet routers in U.S. running out of space due to multi-buy service logs. The core team is on it and we will provides updates here.

We are continuing to investigate this issue.

Due to a high amount of packet rate, the data transfer in U.S. may be experiencing degraded service. The core team is on it and we will provides updates here.

This incident has been resolved.

The fix appears to have addressed the issue but we will continue to monitor to ensure stability is maintained.

The issue has been identified and a potential fix has been deployed.

We are currently investigating an issue with the RPC with the third party provider. Explorer and Helium Wallet App performance may be degraded at this time.

This incident has been resolved.

Restarting the IOT PoC Verifier seems to have resolved the issue. The core developers will continue to monitor PoC in New Zealand for further issues.

The IOT PoC Verifier has been reset and is back up and running. The core developers are now monitoring the verifier's output reports to see if the number of invalidation events is decreasing.

The core developers are restarting the IOT PoC Verifier service in an attempt to rebuild its region parameter cache. This restart will be noticeable in activity dashboards for some tens of minutes as the service restarts, but it will not ultimately affect rewards as the service will catch up to queued beacon and witness events when it comes back up.

The core developers have noticed a rapid drop off of Beaconing and Witnessing events for the New Zealand after today's deployment of the new AS923-1C LoRaWAN region parameters to increase PoC power.

Based on the monitoring, all packet routers have fully stabilized.

The rollout is complete, the core team will closely monitor the packet routers.

In response to the previous degraded data transfer incidents, the core team is doing a rollout for memory optimizations on packet routers, the expected downtime is 15-20 minutes.

All packet routers have stabilized including those in Asia. Thanx.

All packet routers have stabilized except for those in Asia. We will be monitoring the situation and update once they are stabilized as well. Thanx.

The rollout is complete. We will make another announcement here when the packet routers fully stabilize. Thanx.

The hot fix has performed well. The core team is beginning a rollout for packet routers, which should fully stabilize in 1 hr. Thanx.

The core has deployed a hot fix on the packet routers, and our testing devices are now able to join successfully. We will be monitoring the situation along with the community. Thanx.

Based on preliminary investigations, the issue is related with memory usage on packet routers. The core team is working on a patch and we will provides updates here. Thanx.

Due to a high amount of packet rate, the data transfer in U.S. and Asia may be experiencing degraded service (e.g. join loop), which likely caused the issue above. The core team is investigating and we will provides updates here. Thanx.

The core developers have verified with the RPC provider that a bad configuration was inadvertently deployed. This configuration has been fixed and the Helium community should no longer see this particular issue.

We are seeing normal activity on Explorer and the Wallet app. Devs will continue to monitor the situation and close the issue after confirming recovery.

We are continuing to work on a fix for this issue.

The community and core developers have noticed that apps like the Explorer and Helium Wallet app are experiencing a degraded performance due to downstream RPC provider errors. Core developers are working with the provider to ensure service is restored soon. In the meantime, the apps may show incorrect data and transactions may not submit at this time. This should resolve as soon as the RPC provider is back online.

The upsizing is complete and the packet routers have fully stabilized.

Memory issues were identified, the core team is upsizing the packet routers in Asia accordingly. The upsizing should complete within 30 minutes and data transfer in Asia may be temporarily unstable during then. Thanx.

Due to a high amount of packet rate, the data transfer in Asia may be experiencing degraded service. The core team is on it and we will provides updates here. Thanx.

This incident has been resolved.

Oracle Data has completed processing its backlog and core developers will continue monitoring these services. Community applications may take some time to recover from the delay.

Core developers have identified an issue with an upstream data provider that is causing a delay in oracle data delivery (for the verified iot S3 buckets). Specifically, community applications that track near real-time IOT Proof-of-Coverage data may be out of date which may impact Hotspot owners being able to see recent PoC activity. All of the data is still being recorded and will resume once oracle services are restored.

This incident has been resolved.

The data provider has completed the migration process and Wallet app services have been restored. We will continue monitoring to ensure stability is maintained.

Core developers have identified an issue with a downstream data provider undergoing a migration process that is causing Helium Wallet app actions to be degraded. At this time, data in the wallet may be inaccurate or out of date and transactions may fail. We are working with this provider to ensure that services can be restored soon.

Based on the monitoring, all packet routers have fully stabilized and rollout is complete.

Based on the monitoring, the core team is rolling out the GRPC connection stability update to all packet routers.

The U.S. packet routers are back up. We are now evaluating the patch performance on GRPC connection stability in the selected few packet routers. After confirmation, we will rollout to the other packet routers.

The core team is doing a staged rollout for packet routers to optimize GRPC process leaks. We’ll start with a rollout on a few packet routers, monitor for a bit before releasing to all packet routers. The rollout is doing fine in other regions, but causing memory issues in U.S. The core team is upgrading the packet router instances in U.S., the expected downtime is 15-20 minutes.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

The Core Developers are investigating an issue that affects the Wallet App and the Wallet Widgets. Symptoms include: token balances not displaying, and the recommended websites tab is blank.

Based on the monitoring, all packet routers have fully stabilized.

The core team has been monitoring the rollback for almost 1hr and packet routers have stabilized. We will monitor them for 1 more hour before resolving the incident.

We are continuing to monitor for any further issues.

Based on the monitoring, the core team has begun rollback on the packet routers due to high memory usage.

The core team is rolling out packet routers across all regions to optimize GRPC process leaks. Data transfer may be temporarily unstable during the rollout. Should stabilize within 30 minutes.

The indexing fix made by the RPC provider appears to have resolved the UX issues that Wallet users were experiencing. Additional monitoring will be put into place to prevent an issue like this from happening again. Thank you for your patience.

A fix has been applied by the primary RPC provider that powers the Helium Wallet mobile app and we are seeing reports of normal activity resuming. Core developers will continue to monitor the situation for a few more minutes before closing this issue.

We have identified the issue in which Hotspots are not appearing in the Wallet App for some users, and are working with the RPC provider on a resolution. Rest assured that all Hotspots are still safely recorded on the blockchain.

We are investigating an issue in which Hotspots are not appearing in the Wallet App for some users, even though they are still safely recorded on the blockchain.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

It appears that the upstream RPC service which serves the wallet app has gone into a rate limiting mode. This appears to cause wallet balances to temporarily appear empty. We are investigating the cause.

This incident has been resolved.

The RPC service is back online. We will be monitoring its performance to ensure it stays stable.

The backend RPC service that the Helium Wallet App uses to claim network rewards and that the Helium Explorer uses to collect network statistics is under heavy load from a large Solana minting event. This unfortunately means that Helium Wallet App users will be unable to claim rewards until the load subsides. We are coordinating with the RPC service provider to improve service reliability.

A backend RPC service is experiencing a partial outage, which is causing errors in both the Helium Explorer and the Wallet App. This might affect users who are attempting to claim IOT and MOBILE rewards through the Wallet App.

A backend RPC service is experiencing a partial outage, which is causing errors in both the Helium Explorer and the Wallet App.

This incident has been resolved.

The core team has identified the issue, which is caused by a mismatch in data transfer oracle versioning and NOT caused by the Solana Migration. The core team is actively working on resolving it. The estimated downtime is 1 hour, we will provide updates throughout the day.

We are aware of an ongoing issue with data transfer. During this time you may experience unstable device communication. The core team is investigating the root cause and will provide updates throughout the day.

This incident has been resolved.

The Onboarding Service has started responding again, but we're unable to determine what went wrong. We will be monitoring the service for a while.

The Onboarding Service is currently down. We are investigating the issue. During this outage it will be impossible to add new Hotspots to the blockchain.

The Onboarding Service underwent Maintenance from 18:08 to 18:38 UTC on 2023-04-04. During this time it may have been impossible to onboard a new Hotspot on the Network. The maintenance has been completed and the Onboarding Service should be back to normal.

This incident has been resolved.

We are continuing to monitor for any further issues.

The Configuration Service has been updated. All AU/NZ Hotspots should be receiving the AS923_1 configuration now. Hotspots make take up to 30 minutes to receive the changes. Thereafter, Proof-of-Coverage Beacons and Data Transfer should resume as normal.

Unfortunately, many Hotspots in Australia and New Zealand do not appear to have the required firmware elements to support the new "AS923_1C" band plan being delivered by the Configuration Service. We will be rolling back the Configuration Service to deliver the older AS923_1 plan over the next several hours. Once this rollback is complete, we will ask the various Hotspot Makers to ensure that they have support for the AS923_1C plan and re-enable it.

The Core Developers have applied the fix to the Configuration Service. All Hotspots in the AU/NZ region should now be receiving the correct configuration, and if the Hotspot Maker has set up the firmware correctly, the Hotspot should see the correct configuration and adopt it within 30 minutes. We are seeing evidence, however, that some Makers are not prepared to use the information correctly when it is provided. We will update the community as we learn more.

The Core Developers believe that the issue stems from a configuration in the Configuration Service, which tells Hotspots which region plan to use. A fix to the Configuration Service is being tested.

We are investigating an issue where Hotspots in Australia appear to be listening and operating on the wrong frequencies after several Makers have updated their firmware to use the newer gateway-rs firmware. Affected Hotspots will not Beacon, Witness or transfer data correctly.

This incident has been resolved.

On further investigation, the Developers have concluded that the issue is also likely affecting Hotspots in AS923 regions, which broadly covers some regions in Asia, Australia and New Zealand.

The Core Developers have noticed that some Hotspots in Europe have stopped beaconing for roughly the last 24 hours. This is likely due to an issue that can be worked around in the IOT PoC Verifier Oracle. A fix is being tested.

This incident has been resolved.

A prolonged outage on the Solana blockchain has exposed a bug in the IoT Proof-of-Coverage Verifier Oracle. The outage caused the Verifier to unnecessarily rule that many Hotspot Beacons were not constructed properly. A fix has been deployed and is being monitored.

Core developers, validators, and other members of the community are waiting for more data on oracle performance and load shedding efforts that have been made. Block production resumed but election times are slightly longer than expected. We will keep monitoring.

After a few elections, block production and elections have resumed and Validator operators have reported that block times are more in line with expectations. Proof-of-Coverage has resumed and the IoT Oracle has been brought back online with the version that we planned to deploy early this week. It has been in devnet testing for the last 12 hours and we feel that it resolves some of the issues that have been pending since the mainnet launch of Oracled PoC. We will acknowledge the fixed issues and address any pending bugs. We will continue to monitor the chain for issues and look at increasing the transaction rate above 900 Proof-of-Coverage transactions per block. Stay tuned and thanks for being patient over the weekend. Next update should be in the next two hours..

This is placeholder comms for a decision made by the core devs and the Foundation a few minutes ago. We will come back with more specific communications on what we did, why it was necessary in an urgent manner, and what's next. Core developers, validator operators, wallet and exchange operators, and others in the community have noticed significantly slower block times in the last 12+ hours due to a large backlog of Oracled Proof-of-Coverage. No action is required by Hotspot Manufacturers, Validator operators, Data users, or Hotspot owners. We expect to resolve after we clear the backlog. This could be due to a rate limiter that hasn't been implemented yet that needs to be addressed but in order to continue processing state channels, payments, burns, Hotspot onboards, location changes, etc., we believe the correct response is to halt the oracle that is pushing transactions on chain and let it recover. We will provide more details as we execute the steps defined in the blockchain channel on Discord. Stay tuned and thanks for being patient over the weekend. We'll come back with a proper post-mortem and further steps during the week. Thanks!

This incident has been resolved.

During an upgrade to the latest database release, our service provider ran into an issue and had to stop the service that provides blockchain data to Explorer, Helium Vote, Helium App, and Wallet App. They're working on resolving the issue and hopefully have it back shortly.

The API is back and fully caught up. Thanks everyone for their patience. Happy Holidays!

Quick update - the primary database will take 5+ hours to repair in the recovery process. At this point, there will be no new updates until the morning (Pacific time) while we wait for the API to come back and sync to the head of the blockchain. If you have pending transactions, they will remain pending until the outage is resolved. Please do not resubmit transactions.

All - our API provider has let us know there's currently an API outage so data displayed on explorer, Helium Hotspot App, and Helium Wallet App are behind about 200 blocks. This also affects transactions recently submitted in the last 2 hours. If they have cleared, you're good to go. If they're still pending, please do not resubmit the transaction - they will automatically clear when the API is back up and fully synced. Our provider has rebooted the database and I'll provide another update in an hour or so. Your rewards are not affected - they will just show up when the API is back. Data transfer is not affected. While the main Helium API is still down, you might be able to do what you’re trying to do using community data sources such as: https://hotspotty.net/ https://hntscan.io/ https://www.heliumboard.com/ https://heliumtracker.io/ https://heliumgeek.com/

We have closed this incident and transactions are being processed by the Public API normally.

An ETL release (1.1.165) has been prepared by the core developers. Please upgrade your ETL in order to process block 1621472 and resume operations. https://github.com/helium/blockchain-etl/releases/tag/1.1.165 We will issue an all clear when the public API has resumed processing blocks and pending transactions can be resumed.

The public Helium API is currently not showing block and transaction data after block 1621471 due to a missing ETL patch for a new transaction type. The chain is still performing other activity normally and is not affected. Funds are safe and activity like Proof of Coverage and Data Transfers continue normally. The API, however, is behind the chain so apps that onboard and move Hotspots, transfer HNT and MOBILE through the API, etc. will be degraded until the API is able to process blocks again. Please do not resubmit transactions until patches are applied on public ETL and API instances. More information on these patches are being shared on Discord and will be published shortly.

The core team has resolved the issue with the help of the validator community. We appreciate your support and patience as the team worked to get the chain back on track.

Update: good news, the validator operator community has answered the call and block production has resumed. We'll continue to closely monitor status. Thanks again for your patience.

Update: Unfortunately, the chain remains halted because block production is still prevented by the corrupted cache issue. The decentralized nature of the network requires the core team to work with the validator operator community to make the necessary changes to resolve this issue. The core team has repeatedly communicated to operators to fix their cache so normal block production can resume. 🙏🏼 Thank you to those in the validator community that have timely responded to the call and made the updates, however, we’re not quite there yet. Once again we're sorry about this halt and appreciate your patience while we work to resolve.

Update: The core team is continuing to work with validator community to resolve the corrupted region cache issue which is preventing block production. All funds are safe, but pending transactions including rewards will be delayed until the chain can move forward again. Device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place. Please refrain from submitting duplicate transactions as they will all clear when the chain moves once again.

Validators with a corrupted region cache are preventing block production. The core team is working with operators to fix affected validators to resume block production. Sorry for the inconvenience and thanks for your patience. At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.

Explorer and the Helium Apps are now back. Thanks for everyone's patience during this extended maintenance period.

Explorer and Apps are syncing back up but still about 200 blocks behind.

GM. The API is still down. During the maintenance, a corrupt database affected how all services could be brought back up. As a result, over 170TB of data had to be rebuilt from scratch. The main database is up but in order to service all our users and services, we need several more instances.

The API is still down at this time. Databases are being brought up to restore services to the Apps, Explorer, and allow pending transactions first. The Public API will return after the other services are back up. We expect that this will be back online in the morning Pacific Time. An update will be provided then if anything changes.

The API is still down at this time. Databases are being brought up to restore services to the Apps, Explorer, and allow pending transactions first. The Public API will return after the other services are back up. Please also note that the chain is currently halted, and the Validator operators are working to get the chain moving again.

Quick update on the API: Services are slowly being brought back up. This will take time as there is a lot of data to move around. That said, our API provider expects all Helium Apps to be back up in the next 4-5 hours and pending transactions re-enabled in 4-5 hours as well. The public API will not return for about 8 more hours.

The Helium API Service is still offline due to unforeseen issues during the maintenance window. The team is working through a resolution and will update time of resolution as soon as possible. During this time: ▫️Submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App. (Transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots) ▫️ Information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable, out-of-date, or delayed. ▫️ Mining, Rewards, and data transfer are not affected, but their transactions will be delayed until the maintenance is over. While the main Helium API is still down, you might be able to do what you’re trying to do using community data sources such as: https://hotspotty.net/ https://hntscan.io/ https://www.heliumboard.com/ https://heliumtracker.io/ https://heliumgeek.com/ Give some of these tools developed by the Helium community a spin!

The chain has successfully elected a new group and we have resolved this incident.

A rescue block has been generated and issued by the core developers and the Helium Foundation in order to restart block production. A new group has been selected. Community validator operators continue to monitor the chain and block production. We will issue an all clear once we feel confident that the chain is stable.

The Validator Operators are continuing to work together to get agreement on the next block. The chain is still halted at this time. If you are a Validator Operator, please join us in #validator-ops and restart your node.

Quick update on this: We've requested all Validator Operators to restart their nodes to kickstart the consensus group so we can start producing blocks again.

The blockchain is currently stalled at block height 1619546 after activating the regional chain variables. The core team is working through the issue with the Validator Operators. We will provide an update as soon as we know more. Funds are safe.

The chain halt is resolved. Enough of the consensus members have agreed on blocks and elected several times. We will continue to monitor.

The chain has progressed a bit further but we're not out of the woods. We've yet to have a new Consensus Group election so rewards continue to be delayed. Proof-of-Coverage is degraded at this time, and pending transactions may take some time to clear.

The core team has a candidate Validator 1.15.8 release and will be available shortly. In the mean time, the current consensus group is having difficulty agreeing on a block and we are stuck again on 1,589,531. A reminder to the rest of community - pending transactions will not clear until we get a new block. Please refrain from submitting duplicate transactions as they will all clear when the chain moves once again.

The Consensus Group has agreed on a block and we're moving onto block 1,589,497. We're not in the clear yet, as we need to see through the next consensus group election. In the mean time, a Validator release 1.15.7 is available and in testing. This release includes a blessed snapshot and improvements to prevent Hotspot ddos. Run at your own risk as it's not been tagged GA yet. Final reminder: Validator Operators, please close port 8080. We will let you know when to reopen it.

The core team is actively monitoring the Consensus Group to see if they can agree on a new block. At this time @Validator if you have a validator in Consensus, please join us in validator-ops and remember to close your port 8080. The team is also preparing a new Validator release with the blessed snapshot and Hotspot ddos prevention.

The core developers and validator operators have noticed a chain halt at block 1,589,496. We are investigating the root cause. At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected. At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.

The chain has resumed normal operations. The core dev team is working on creating a blessed snapshot for a small number of Validators that continue to be stuck at block height 1,587,339 or prior to 1,587,313. If you're stuck on 1,587,313: - Nodes (ETL, routers, etc) - simply need to restart - Validators run the following command: miner eval 'blockchain_worker:sync().'

The core team is now trying to get the rest of the Validator fleet caught up to the new block height so it can gossip block information around and be eligible for a new consensus election.

A new consensus group has been assembled and there are now new blocks. We're now waiting to see if the other services including the ETL that serves data to the API (and explorer/apps) can catch up. At this time, please refrain from submitting new transactions until we can give the go-ahead, they may take a while to clear still.

The core team has assembled a new candidate Consensus Group. After the new group is elected, the core team will issue a rescue block to get us past the halt. For now, the chain is still halted. Funds are safe, data transfer is not affected.

We believe we have the root cause and will likely need to solve it with a rescue block. All Validator Operators should join us on Discord for further coordination. For the rest of the community, we remain halted at this time. Funds remain safe but transactions are still delayed until the halt is resolved.

The core developers and validators operators have noticed a chain halt where validators aren't producing blocks after 1,587,313. We are investigating the root cause. At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.

The core devs and the validator community have identified what we believe is the root cause of chain halt (and the prior one) and a new validator release is being prepared. In the meantime, a hotfix has been applied to some validator nodes in the Consensus Group and block production has resumed.

A fix has been implemented and we are monitoring the results.

The core team is investigating the chain halt that occurred at block height 1,575,955. The network appears to have hit the same issue that was cleared earlier and most likely the root cause of the initial outage. The core team and Validator operators in the community are working together to resolve this issue. Hotspots will not be participating in Proof of Coverage until block production resumes. All funds are safe, but pending transactions will be delayed until we can resolve this halt. Pending Transactions include adding a Hotspot, changing a Hotspot Location, transferring a Hotspot, payments, and token burns. At this time, data transfer is not affected by the chain halt.

This incident has been resolved.

Explorer Unavailable The core team is aware that explorer is not displaying accurate data at the moment. We've alerted our provider and they've started looking into it. Apologies for the inconvenience. Hotspots, tokens are safe. The blockchain is still moving. I'll be back in 30 minutes or less.

The ETL is now less than 20 blocks behind the blockchain and there are less than 100 pending transactions against the ETL. We are going to mark this issue as resolved. Thanks very much for your patience.

The ETL that feeds data to the Helium Public API is behind by about 130 blocks. This issue has been fixed but it will take a bit of time to be fully caught up to the blockchain.

Final update for the evening. The chain seems to be progressing. As more validators update and reload from a snapshot, we should continue to see them come back online. Core devs are moving to continuous monitoring of the Consensus Group and block production. Reminder, please see announcements in Discord channels if you operate a chain following node. Hotspot owners do not need to make any adjustments and PoC activity should return to the state it was prior to this halt after several hours. A postmortem report will be prepared and shared as soon as we can prepare it later this week.

Update: Blocks are moving again but the number of transactions in each block is low (about 10% of expected txns). We still have not seen a reward block, so we're continuing to monitor. Updates for ETL and Node are currently being tested. Validators have their update already.

Update: We have an emergency release going out to Validators (and another release for ETL, Node, and Router operators shortly). The emergency release includes a new snapshot that all Validators will sync to to realign ledgers (there was a ledger drift). Update again soon.

Update: Just setting some expectations here. We expect this to be a long chain halt (hours +) and there will be several emergency releases going out to Validators, Node, and ETL operators. Announcement on when that's available will be made separately. Chain is halted, not dead. Again, pending transactions will not go through, rewards are affected as the chain is not moving, but it's affecting everyone equally.

Update: We have halted again at 1,469,988 and getting more clarity at the root cause. We believe receipts in blocks cannot be absorbed. More details coming as soon as we have a plan.

Update: The core team has asked all Validators currently in the Consensus Group to restart their Validators in order to resume block production.

Update: We're still looking, but have a potential solution percolating. In the mean time, I want to also mention that device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.

Some nodes are able to progress to block 1469953 but not all. This issue may auto-resolve but the core team is still investigating.

The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable. All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward. Pending Transactions include: - Add Hotspot - Assert Location - Transfer Hotspot - Payment Transactions - Burn Transactions

A fix has been applied and the queue has cleared. Any pending transactions will be automatically cleared. Thanks for everyone's patience.

We've identified the issue that is backing up pending transactions. We will be updating and dropping a new tag of ETL shortly. We will then deploy the update. Thanks for your patience.

Transactions submitted to the public API or via Phone are seeing significant delays when being submitted to the blockchain. This issue is likely caused by an issue in today's new ETL release. We are currently investigating this issue and will remediate it as soon as possible. Keep in mind: 1. The blockchain is fine. 2. It's just taking longer than normal for the ETl to get new transactions into the blockchain. 3. The public api is online and up to date.

This incident has been resolved.

Last update on this outstanding issue: The core team has shifted focus on creating an off-chain PoC mechanism (a HIP will be proposed) that would allow Hotspots to beacon multiple times a day and allow more witnesses than before. The work that's on-chain today (increased challenges per block, activated chain variables, blockchain sync turned off) should help alleviate lack of PoC activity for the next few months. While there is room for improvement, the team believes the ultimate goal of off-chain PoC will satisfy the needs of Hotspot owners and the network overall.

The core team is prepared to activate 6 chain variables related to HIP 54: H3Dex. In addition to enabling HIP 54, the chain variables also include improvements to proof-of-coverage:  . Updates HIP15/17 values so that Hotspot in less-dense areas are not starved of rewards. Keep in mind that PoC selection is still random, so any improvements (in terms of earnings) can take a while (days++) to see.  . A fix that should see more PoC receipts coming in per block. Currently, 250 Challenges go out per block and about 170-180 receipts come back per block. At this rate, the average Hotspot beacons every 3-4 days. Once the fix is in, Hotspots may beacon more frequently than the average 3-4 days, but we will have to wait and see how well this works once activated. The goal is to get as close to 250 receipts back, instead of 170-180 we get today. Once these chain variables are activated, we expect to see a blockchain performance improvement and more Proof-of-Coverage receipts received, per block, over a period of several days.

Hotspot Release Available: Flatline Fix 

The core team has released a new Hotspot release to fix the flatline issue! Before explaining what the fix entails, I want set expectations that the flatline fix will not be *immediately* visible. A Hotspot Maker will need to deploy the update to their fleet of Hotspots (which can take several days! Keep an eye out on your maker’s discord or social media), then it can take 3-4 days for a Hotspot to beacon or witness and be rewarded. Flatlined Hotspots can take up to 7 days to come back to life with this bug fix. A summary of the flatline fix There were two issues have been causing various crashes and out of memory errors on Hotspots that have been preventing them from participating in network and potentially causing "flatline" reports in the community. Fix 1. The team identified an issue where Hotspots were talking to both Mainnet and Testnet causing a Hotspot to run out of memory and freeze/flatline. Fix 2. A Hotspot can crash due to a latent bug in the bluetooth stack Makers have been notified of this release and will send the update over-the-air to their Hotspot fleet. Hotspots connected to the internet will automatically download and install this update. Once the update is applied, flatlined Hotspots can take up to 7 days to come back to life. No user-action is required. If your Hotspot is all alone in a hex without any neighbors that can hear you beacon, the only rewards your Hotspot may earn is Data transfer rewards only. To resolve this, you’ll need more Hotspots in your area to participate in Proof-of-Coverage. **A note about Hotspots in less dense areas seeing less Proof-of-Coverage Activity**
 The fix for this is part of a Validator release that is still in testing and requires chain variable activation. There is no update at this time but the activation will not be this week, and will be no sooner than late next week.

Helium Community, The core team believes they have a good handle on the missing receipts and will be issuing several changes this week. 1️⃣ Things we found (with the help of technical community, thanks BigEnigma!) • The Proof-of-Coverage logic has versioning built in. BigEnigma found that the versioning (and thus challenger targeting) was mismatched between ledgers. This is also contributing to receipt timing issues and receipts coming in out of order. Receipts out of order and the versioning mismatch can cause a number of proof-of-coverage receipts to be dropped, Hotspots not credited for their work, and leads to lowered Hotspot earnings. This will be fixed in a Validator Release and is in review. • When a hex is targeted for a Proof-of-Coverage challenge, the logic is backwards. It is currently selecting 14 witnesses first, then filtering for inactive (aka offline) Hotspots, often leading to <14 Witnesses per challenge. The fix is to filter for inactive Hotspots first, then selecting 14 witnesses. • When will it be fix? These changes will be in a Validator Release, so once we have a release candidate and an activation date set, we’ll let the community know. 2️⃣ Light Hotspot Chain Syncing - When will chain sync be turned off? • The core team will be releasing a new Hotspot release this week for Makers that will re-enable mux. There is a small bug that the team is investigating when chain sync is turned off, but once this bug is squashed, we can turn off chain syncing. • Our target for this no later than next week. 3️⃣ Some Hotspots on the Denylist earning? What’s going on? • With the help of community, we've identified a Validator Operator with custom configuration overrides that is not able to pick up the Denylist list. The operator has been informed and will be fixing this in the next week or so. • Hotspots that use Validators serviced by this Validator Operator will not be able to exclude Denylist Hotspots participating in proof-of-coverage, leading to some Denylisted Hotspots earning. 4️⃣ Hotspot Flatlining • Core team has recently found a new cause of flatlines. We have an engineer assigned to dig a bit more and if there's a fix, I'll let community know.

Proof-of-Coverage Update: Hotspot Update Now Available for Makers The core team has made a new Hotspot update available for Makers to distribute to their fleet. This update will have additional fixes for “flatline”, fixes some of the timing with respect to submitting Proof-of-Coverage receipts, and picking a different Validator faster, if the first Validator is behind on blocks (Validators that are behind on blocks cannot properly challenge, affecting a Hotpot’s beacon activity). To see if your Hotspot’s maker has pushed the update, please check in with their Discord servers or on their social media accounts. Hotspots receive this update over-the-air and will automatically apply the update if it’s online and connected to the internet. No action required for Hotspot owners. Hotspot owners should see an overall improvement in Proof-of-Coverage over the next 48-72 hours after the update is installed on your Hotspot.

Since the last chain variable activation, the blockchain has stabilized enough for us to look at the data to identify further areas of improvement. The main data point the team is focused on is the number of PoC receipts coming back, per block. Current Variables: The target Challenges per block is 250. The number coming back has been consistently between 70-71% (around 170 receipts per block). There are 30% of receipts missing. Why? Missing Receipts 30% of receipts are missing which could be due to Hotspot flatlines, Validators being out of sync, or Validators being out of sync, then resyncing quickly. To address missing receipts, the core team will be putting together a Hotspot release to help reduce the 30% value and enable more Hotspots to participate in PoC. 1. Hotspot Flatline: there will be another HTTP2 fix in the upcoming Hotspot release. This is being tested on Testnet at the moment. We expect this to bring a few more Hotspots back to life and reduce the 30% number. 2. Validators being out of sync: Hotspots will ditch out of sync Validators sooner (instead of waiting 20 blocks, Hotspots will wait 10 blocks before looking for a new Validator). This provides a better chance for a Hotspot to connect to a good validator for PoC and should help reduce the 30% number. 3. Validators being out of sync for a bit, quickly syncing again, and the Hotspot’s proof-of-coverage challenge becomes invalid. The challenge should not be considered invalid. The fix will also be in this Hotspot release. We are aiming to push out this Hotspot release to Makers by Thursday 00:00 UTC. Further Investigations this week The team will be working with community this week to see if there is a correlation within the 30% of receipts missed in terms of maker type, region, or density by running the targeting logic. This may or may not yield meaningful results, but it’s worth the investigation.

The "Flatline" Fix has been well-received by Hotspots. About 36,000 more Hotspots beaconed in the last 24 hours (as noted in explorer.helium.com/hotspots) than before the fix. We will continue to monitor over the weekend.

The core team has tagged a new release for Hotspot Makers 2022.05.24.0. It is up to each Hotspot Manufacturer to send the software update over-the-air to their fleet.

GM 🌅 Helium! We have optimistic news today thanks to help provided by one of our own community members nmc. With their help, the core team was able to find a root cause of the “flatline” that some Hotspots had been seeing since activating Light Hotspots. To explain it simply: on a Hotspot, one of the lower-level processes (HTTP2 client) that is used when connecting to a challenger, stalls, and never returns since there is no timeout explicitly set. The fix is to add a default timeout and a monitor so it knows to “wake back up”. Next steps for the fix 1. The core team will send the fix to Testnet and Mainnet on alpha and beta Hotspots as soon as it is ready. 1-2 hours from now. Let it sit for an hour and confirm that everything works. 2. Tag the build for Hotspot Makers as “Urgent” and “Mandatory”. 3. Hotspot Makers will send the update down to their fleet.

Proof-of-Coverage Update 2 May 20 What to expect over the weekend and what core team is looking at next week: 1️⃣ Network Stability. We had some big chunky blocks today - all stemming from Proof-of-Challenge receipts when we had poc_challenge_rate ramped up to 700. How did this happen? We had a bunch of bugs that we fixed over the last 9 days + we activated a chain variable this morning that kept more Challenges active. This ultimately increased the number of valid receipts received on the chain. More receipts = more time spent validating each receipt = bigger block times = unstable network. To keep things moving, the team reduced challenges per block to 200 and now it’s at 300. Why 300 today is better than 700 last week About 20% of receipts per block were coming through last week (~140 receipts). With all the updates we made since, we’re seeing numbers much closer to the target of 300. This is better for you as a Hotspot owner at the end of the day. More receipts = more witnesses = better rewards. And with stable block times, there continues to be consistent rewards. 2️⃣ Validator Release with a bunch of improvements are going to be tested over the weekend. It will be available for Validator Operators shortly. Included in the Validator release will be improvements to Hotspot inactivity (when do you start selecting a Hotspot for Beaconing again; should be more often with this release), improvements to HIP17 targeting in less-dense Hotspot areas, witness receipt validation improvements (for network stability), and more.

Chain Variable Activation: poc_apply_gc_fix The team is activating the chain variable poc_apply_gc_fix that will greatly reduce the number of "invalid" or "expired" Proof-of-Coverage challenges. The outcome for Hotspot owners will be higher witness and beacon counts.

Proof-of-Coverage Update May 20 GM 🌅 Team is working on a handful of improvements today, including digging into the http2 Hotspot process crash and Validator-side changes. This morning, the team will activate a Chain Variable that will greatly reduce the number of "invalid" or "expired" Proof-of-Coverage challenges. We'll let you know when that goes out. The outcome for Hotspot owners will be higher witness and beacon counts. 1️⃣ Manufacturers are still pushing the update out from yesterday. We should continue to see improvements over the next 24 hours. 2️⃣ The team does not have a solution yet for the process crash on Hotspots (good chance it's related to the "flatlines"). Unscientifically, a Hotspot reboot may fix it 25% of the time but the core team does not recommend doing this at this time - do so at your own risk. 3️⃣ The Validator candidate release has a lot of fantastic fixes. Included are: - the HIP17 fix for less dense Hotspot areas* - improvements to witness validation (particularly useful to Validator performance and will stabilize block times)* - better identification of "active" Hotspots for targeting challenges* *All of these are also controlled by a chain variable, so we won't be able to see its benefits until we activate them next week.

Proof-of-Coverage Update 2 1️⃣ The core team has tagged a new release (2022.05.19.0) for Hotspot Makers to pick up and send down to their respective fleets. Original Helium Hotspots will pick up this update over the next 24 hours (that's how often it checks for a software update). We expect to see receipts grow over time as more Makers pick up this change. No action is required by Hotspot owners 2️⃣ In addition to a Hotspot release, the core team has identified a timing fix related to collecting invalid Proof-of-Coverage challenges too early that may recapture some challenges that should still be valid, but was marked invalid. 3️⃣ The Hotspot process crash is still being investigated. No new news. 4️⃣ A bug introduced in HIP17 related to hex targeting is causing less-dense Hotspot regions see their rewards diminish over time. The team is putting together a Validator-side fix that will be controlled by a chain variable. The expected activation will be next week. 5️⃣ As the core team improves on the receipt delivery on the chain, Validators will have more receipts to validate. As a result, block times will increase. To help mitigate this, the team is also working on receipt transaction improvements. This is mostly an FYI for the community as it likely has no bearing on your Hotspot performance.

Proof-of-Coverage Update GM 🌞 The team has been testing a fix to address the missing receipts issue overnight on mainnet test Hotspots and on Testnet. Results indicate a minor improvement but we think it will still make a difference in PoC receipts and rewards. An OTA is being prepared and will be made available for makers in a few hours. It is up to each individual maker to send the OTA update to their fleet of Hotspots. There may be action required for Hotspot owners, but check with your maker to be sure. Secondly, the core team is still working on fixing the dying process that I mentioned in point 2️⃣ above. This is likely also the reason why the average receipts over a long period of time seems to be trending down slowly.

Proof-of-Coverage Update I wanted to drop by with an update now that it's later in the afternoon on our current progress. 1️⃣ The fix is now in a miner image 05.18 on both Mainnet (on testing Hotspots) and Testnet. This is to fix the missing receipts issue. We're still monitoring the Hotspots but it is too early to tell. We will be back tomorrow to confirm that this is a release candidate and GA for Hotspot Makers. 2️⃣ This particular investigation will not be as fast as the others. There is quite a bit of surgery involved. The update here is.. we're still digging. Lastly, the AMA video should be out shortly!

Hello Helium Community: The core team is working on 2 things today. 1️⃣ The core team found an issue overnight and is testing a fix on how a Hotspot communicates to its Validator to make it more reliable. This should increase the number of proof-of-coverage receipts sent back to the Network and should improve rewards. This is getting testing time on Testnet as well as Mainnet on our Testing Hotspots. If this is a noticeable improvement, we will be issuing a Hotspot release (today or tomorrow). 2️⃣ The team, in collaboration with the community in the technical channels have also identified an interesting occurrence that likely manifests as a Hotspot “flatlining” or “doing nothing for days”: There is 1 process in the Hotspot that silently dies and never alerts anyone. This process is needed to communicate with the Validator the Hotspot is connected to. If this thing dies, the Hotspot is unable wake itself back up to respond to Proof-of-Coverage or even send Witness receipts; hence flatline. The core team is going to replace this process with another that already exists in the Validator code. Unscientifically, this may explain why sometimes Hotspots come back to life (about 25% of the time) after a reboot. This is not something the core team is recommending Hotspot owners do, but if you want to experiment, there’s not much to lose. Finally, the core team truly empathizes with Hotspot owners on the poor rewards. This is 100% the core team’s entire focus right now. It’s important not to point fingers at each other. We are in this together and in it for the long haul. If we can get these two issues fixed to bring back reward stability to the network, we can turn off chain syncing as early as next week.

GM 🌅 Helium Community. We are staying the course. After the updates yesterday, we saw Proof-of-Coverage receipts stay above our threshold of rollback. Overnight, we saw something happen on Testnet that we suspected, that is, a Hotspot was not able to tell its notifying Validator about its receipt - a likely cause of why some receipts are missing on mainnet. Today, the core team will do 4 things: 1. chase down the bug we saw on Testnet. If there is a fix, there's going to be a release. 2. understand why sometimes receipts come in out of order and if it affects performance. If it does, fix it. 3. performance improvements to Validator heartbeats and receipts 4. consider increasing the Challenges per block rate from 700. There will be a separate announcement if this happens. No action required for Hotspots, Makers, or Validators at this time.

Proof-of-Coverage receipts per block are trending in the right direction as Validators take the upgrade to 1.10.9.

Lowered PoC Rewards and Receipts Update 5 The core team is seeing very slow improvements to Proof-of-Coverage (and rewards) receipts since the Validator release. This slow change is likely due to Validators going offline to upgrade themselves, get back online and sync to the tip of the blockchain. We’re hopeful that this is the case for the slow improvements (about 30% are not upgraded/online) but we are prepared to issue a Light Hotspot rollback tomorrow morning Tuesday, May 17, 2022 6:00 AM PT We have identified another bug (see Discord #blockchain ) where the Proof-of-Coverage manager on Validators are crashing out every 50 blocks and purges the challenges. This is being reviewed and will be part of a new @Validator 1.10.9 release. This is not a silver-bullet bug fix, but rather looks to stabilize the receipts we already have from its current pattern so we can have a better view of the Proof-of-Coverage landscape tomorrow morning. We will roll back the Light Hotspot transition at Tuesday, May 17, 2022 6:00 AM PT if: 1. More than 90% of Validators upgraded (at least 1.10.8) AND 2. Proof-of-Coverage receipts per block is still consistently below 100 receipts. What will happen if we rollback: 1. Hotspots will challenge again. All Hotspots are still syncing the blockchain and there are consistent snapshots being produced to help those that are out of sync. No action required by Hotspot owners or Makers 2. Validators will continue performing Consensus Group duties and will no longer create Challenges. 3. We will continue our investigation on Testnet and will re-attempt activation at another time.

Lowered PoC Rewards and Receipts Update 4 About 66% of Validators have upgraded to the latest release. Once a Validator is updated, it will take 200 blocks (or approximately 3.5 hours in 🕐 time) for them to discard invalid proof-of-coverage challenges and start issuing the updated versions that will result in the graph going 📈 . TLDR: need more time for rewards to come back and stabilize as Validators upgrade and shed their old challenges. So if your Hotspot is connected to a Validator that just upgraded, it will be 3.5 hours before your Hotspot will be rewarded properly. The graph will not hockey stick. No action required for Hotspot owners.

Lowered PoC Rewards and Receipts Update 3 The Validator Upgrade has been available for ~1 hour and 269 Validators have upgraded. We need to see much more Validators update to 1.10.8 before we can see noticeable improvements to Proof-of-Coverage and rewards. The team is continuing to monitor dashboard.helium.com. No action is required for Hotspot owners.

Lowered PoC Rewards and Receipts Update 2 The core team has tagged a release for Validator Operators that includes the bug fix for the known Proof-of-Coverage issue. After the Validator update goes out, all Validators will be required to take this update ASAP. We will slowly start to see improvements (over the course of 2-3 hours). We are keeping the Challenges per block at 700 for now. No action required for Hotspot owners.

The core team is continuing to debug the root cause of lowered Proof-of-Coverage receipts (and consequently lowered rewards). We may have a lead and will update community as soon as it's confirmed and we have a fix. Also note that if you have altered your sys.config file on your Hotspot, you may have inadvertently broke your Hotspot's ability to participate in Proof-of-Coverage. Please check with your Hotspot Maker on what the correct configuration settings should be. If you have no idea what sys.config means, no action required - the likely cause of low rewards is what the devs are working on.

This incident has been resolved.

Block production has been restored. The core team will be monitoring the next few blocks and the next election for any further issues.

Consensus Group has stopped producing blocks at block height 1435692. The core team is investigating. At this time, submitting transactions to the blockchain will remain pending and will not clear. Transactions include: * Add Hotspot * Assert Location * Transfer Hotspot * Burn Transaction * Payment Transaction * Stake Validator * Transfer Stake Mining rewards are affected (no blocks, no PoC, no rewards), and affects everyone equally. Device Transfer is not affected. We will be back in 2 hours or sooner for an update.