Is Hologram Down Right Now? Check if there is a current outage ongoing.

We are currently investigating an incident affecting SIMs with the G2 profile (prefix 89357*). The issue is impacting Diameter and SS7 traffic, which may result in: • Failed or delayed SIM registrations • Loss of connectivity for some devices • Inability to establish new data sessions Our engineering team is actively investigating the root cause and working with our upstream partners to restore normal service. Updates will be posted as more information becomes available. We apologize for the disruption and appreciate your patience as we work to resolve the issue.

The issue has been resolved. G1 SIM Card Administrative operations are processing normally.

An upstream partner has identified an issue delaying SIM Card Administrative operations on G1 profiles. They have identified the cause and are working towards a resolution.

The issue has been resolved. G1 SIM Card Administrative operations are processing normally.

An upstream partner has identified an issue delaying SIM Card Administrative operations on G1 profiles. They have identified the cause and are working towards a resolution.

The instability has resolved, SIM Card Administrative Operations and Dashboard-originated SMS are processing as normal for SIMs with G2 profiles.

We are investigating instability over a connection to our G2 carrier partner which is causing intermittent delays in the processing of SIM Card Administrative Operations, and the delivery of device-terminated SMS sent from the Hologram Dashboard.

The issue affecting new G3 activations has been resolved.

We’ve identified the issue impacting new G3 SIM activations (ICCIDs starting with 89464*) where some SIMs were unable to establish data sessions. A temporary fix has been implemented. Impacted SIMs should now be able to open data sessions; however, there may still be a delay of a few hours after activation before full connectivity is established. This only affects a subset of new G3 activations — not all are impacted. Our team will continue working over the weekend to identify and implement a permanent fix. We appreciate your patience and will provide further updates as progress is made.

We’re currently investigating an issue affecting new G3 SIM activations (ICCIDs starting with 89464*), some SIMs might be unable to establish data sessions. This issue does not impact previously activated G3 SIMs or SIMs on other profiles. Our engineering team is actively working to identify the root cause and implement a resolution. We’ll provide an update as soon as more information is available.

The issue where some SIMs are being charged their monthly recurring cost (MRC) multiple times during their billing cycle renewal has been resolved. We will apply necessary billing corrections to impacted accounts.

We’ve deployed a fix for the issue that was causing some SIMs to be charged multiple times during their monthly renewal. Affected services have caught up and are now operating normally, including data usage reporting and billing cycle processing.

We’ve deployed a fix for the issue that was causing some SIMs to be charged multiple times during their monthly renewal. Affected services have caught up and are now operating normally, including data usage reporting and billing cycle processing. As a precaution, we are temporarily continuing to not suspend SIMs or organizations with negative balances on prepaid accounts, while we finish applying billing corrections.

We’ve identified the root cause of the issue that was causing some SIMs to be charged their monthly renewals multiple times. We are currently publishing a fix for this error. Until the update is deployed, SIMs will not pause when reaching their data limit, and customers on prepaid plans will not have their balance auto-refilled. Additionally, data usage reporting via our API will be delayed by several hours. We are actively working to restore the underlying services and will be applying necessary billing corrections to impacted accounts once the issue is resolved.

We’ve identified an issue where some SIMs are being charged their monthly recurring cost (MRC) multiple times during their billing cycle renewal. Our team is actively investigating the root cause and working to implement a fix. Once the issue has been resolved, we will apply any necessary billing corrections to impacted accounts. If you have any questions please reach out to support@hologram.io. Thank you for your patience and understanding.

This incident has been resolved.

We recently identified a delay in the processing of device usage and session data. This has been resolved, and the system is now recovering. Some customers may notice missing or delayed usage data in the Dashboard and API. This data is not lost — it will begin appearing gradually as the backlog is cleared. We expect full recovery over the next several hours. There is no impact to device connectivity.

The issue has been resolved.

We identified the issue and applied a fix, new sim activations should be able to open data sessions. We're monitoring.

We’re currently investigating an issue affecting new G3 SIM activations. Devices with ICCIDs starting with 89464 are unable to obtain an IP address, which prevents them from establishing data sessions. This issue does not impact previously activated G3 SIMs or SIMs on other profiles. Our engineering team is actively working to identify the root cause and implement a resolution. We’ll provide an update as soon as more information is available.

Issue is resolved.

A fix has been applied and backend services are recovering. We’re currently monitoring the stability of the API and Dashboard to ensure full functionality is restored. We’ll post a final update once the incident is fully resolved.

We’ve identified the source of the issue and we’re actively working with the responsible parties to mitigate and restore stability. We’ll provide further updates as resolution progresses.

Affected Services: • Dashboard • API • SIM management We are currently investigating an issue causing backend services (including the Hologram Dashboard and API) to be unavailable. Device connectivity is not affected at this time. Our team is actively working to identify the root cause and will provide updates as soon as we have more information. Thank you for your patience.

This incident has been resolved.

There is a backlog of SIM operations which is delaying activations, and as a result connectivity is affected for those specific SIMs. We are actively working on restoring service and will provide updates accordingly.

This incident has been resolved.

We’ve increased capacity on our PGW (Packet Gateway) network nodes and are already seeing significant improvement in performance. Our team is actively monitoring to ensure stability and will provide further updates as needed.

We are currently investigating a reported issue on our G3 SIMs/profiles (ICCID prefix 89464*). We are seeing high percentages of packet loss with both existing and new data sessions. We are actively working with our upstream partners on a resolution. We will provide updates as more information becomes available.

We’ve resolved the earlier disruption affecting a subset of G1 SIMs/profiles (ICCID prefix 8944*). The issue was related to an upstream IPX interconnect provider, which temporarily impacted inbound signaling traffic. Traffic on the IPX links has now returned to normal levels, and connectivity has stabilized across affected devices.

We're currently experiencing a partial outage on our cellular network specific to our G1 profiles (sims/profiles with ICCID prefix 8944*). We're working closely with our upstream partners to get this resolved as soon as possible.

This incident has been resolved.

We are currently experiencing delays in SIM-related operations due to an increased backlog. This may impact the following services: • SIM activations • State changes (e.g., suspensions, reactivations) • Plan changes Our team is actively working to process the backlog and restore normal operation times as soon as possible. We apologize for any inconvenience and appreciate your patience. For urgent requests or further assistance, please reach out to our support team.

This issue is now resolved. The Vodafone UK network is once again available for service. Hologram has monitored the network and confirmed successful registrations on Vodafone UK. We will continue to monitor the network throughout the day.

We are still experiencing ongoing issues. As a result, we have temporarily blocked Vodafone UK on our G1 profile (8944* prefix), this will allow devices to attach to other UK operators. Vodafone is actively investigating the issue. Our team will continue to monitor the situation and will reinstate Vodafone UK once they confirm a resolution and we determine it is safe to do so. We will provide further updates as soon as more information becomes available.

We are currently investigating a reported issue affecting Vodafone UK (GBRVF). Customers may experience difficulties with network registration and data sessions. Vodafone has confirmed an ongoing incident on their network and is actively working on a resolution. At this time, there are no confirmed issues with other UK networks. We will provide updates as more information becomes available.

This backlog has been cleared.

There is a currently a backlog of operations delaying SIM operations on our Global-1 and USA2 carrier. Backlog is processing and some customers may experience delays with some SIM operations. Cellular network and other API services are unaffected.

After monitoring, we determined all services and network performance levels are back to normal.

Our upstream networking partner has implemented a fix for the connectivity issues affecting sims with enabled G1 (89445*) profiles. We are now seeing significant improvements, and the network performance is on a path to full recovery. Current Status: • Data connections are stabilizing, and the frequency of failed sessions has reduced significantly. • Customer feedback indicates improving performance, though we continue to monitor for any anomalies. We will remain in the monitoring phase to ensure complete resolution. If you continue to experience issues or have any questions, please don’t hesitate to reach out to our support team for assistance.

We are currently experiencing a connectivity incident affecting sims with an active G1 (89445*) profile. The issue has been identified, and our upstream partner is working with high urgency to resolve it. Impact: • SIM cards may experience sporadic data issues, including interrupted sessions and inconsistent connectivity. We are actively monitoring the situation and will provide regular updates as more information becomes available. Thank you for your patience as we work to restore full service. For any questions or further assistance, please reach out to our support team.

Backlog has been cleared and SIM operations are back to normal.

We are currently experiencing delays in SIM-related operations due to an increased backlog. This may impact the following services: • SIM activations • State changes (e.g., suspensions, reactivations) • Plan changes Our team is actively working to process the backlog and restore normal operation times as soon as possible. We apologize for any inconvenience and appreciate your patience. For urgent requests or further assistance, please reach out to our support team.

This incident has been resolved.

Traffic has been at normal levels for several hours, and both new and existing sessions are working as expected. This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our connectivity partners have released a fix, however the large rush of returning traffic brought down a different part of their network. They are working on getting it resolved

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

Sims with the G2 profile are unable to create new sessions. Existing sessions are not impacted. Our connectivity partner is investigating the issue.

A proper fix has been put in place and over night monitoring has shown that all traffic is back to normal levels. This incident is now fully resolved.

We’ve identified the cause of the lingering connectivity problems for G2 SIMs with prefix 89357*. A temporary fix is now in place, and we’re working with our upstream partners on a permanent resolution. Thank you for your patience while we finalize these next steps.

We recently experienced an emergency maintenance event on one of our core network packet gateways, which impacted G2 SIMs with the ICCID prefix 89357* and led to issues with existing and new data sessions. The maintenance has now been fully resolved but we're still getting reports of sim cards that have been unable to reconnect. We're investigating and further updates will follow soon. Thank you for your patience and understanding.

This incident has been resolved.

We are seeing traffic levels are now back to normal, so we will be monitoring for some time. We will provide further updates as needed.

We're currently experiencing a partial outage on our cellular network specific to our G2 profiles (SIMs/profiles with ICCID prefix 89357*). Some devices are experiencing intermittent issues attaching, opening new connections or sending/receiving SMS messages, but most devices are currently unaffected. We're working closely with our upstream partners to get this resolved as soon as possible.

This incident has been resolved.

We are seeing traffic levels are now back to normal, so we will be monitoring for some time. We will provide further updates as needed.

We are seeing an issue with a network partner causing some SIM cards on the 89357* ICCID range to be unable to perform new registrations and new data sessions, but existing registrations and data sessions are not affected. We are investigating and will provide updates as we have more information.

This incident is now resolved.

We are seeing traffic is back to normal levels and our upstream partner has confirmed the issue is now resolved. We are closely monitoring for the time being and we'll provide another update shortly.

We're currently experiencing a partial outage on our cellular network specific to our G1 profiles (sims/profiles with ICCID prefix 89445*). We're working closely with our upstream partners to get this resolved as soon as possible.

This incident has been resolved.

Verizon confirmed that they have fully restored today's network disruption that impacted some customers, and their service has returned to normal levels.

Verizon has reported widespread outages, both roaming and direct access to the Verizon network might be impacted in our service. We're monitoring the situation and communicating with verizon.

This incident has been resolved. All delayed SIM statuses reported by the API and Dashboard are up to date, and new status changes will be reported without delay.

There is delayed status reporting for some sims that have recently completed state changes. In some cases a pending state might be reported even if a state change has already completed.

The issue was identified in a patch that was released earlier this morning and has been fixed.

We're investigating an issue preventing users to perform new sim activations using our Dashboard. API based activations are working.

All downstream services are operating normally. This incident has been resolved.

We have restored functionality of our internal queuing service, and are monitoring the downstream services as they recover. All features are operational. Preflight activations may take more time than usual to be reflected in Dashboard.

We have identified a problem with an internal queuing service which powers some event-based features of our platform. Connectivity for all SIMs is unaffected. Overage limits that pause SIMs for data usage are still being enforced. The following features are temporarily unavailable: Alerts: SMS & data usage thresholds State transitions: Preflight activations & pausing for SMS usage.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Usage reporting and billing of usage are delayed. Network connectivity and other services are unaffected. Hologram engineering is investigating the cause of the delay. Usage alerts are not working at this time.

Usage reporting and billing of usage are delayed. Network connectivity and other services are unaffected. Hologram engineering is investigating the cause of the delay.

On July 29, 2024, Hologram experienced an outage with our G-1 connectivity partner, which prevented some devices using G-1 SIMs/profiles from being able to establish new cellular data sessions during the incident, while devices with sessions already open and which did not close during the incident remained unaffected. During recovery, a shorter second period of unavailability affected new connection attempts, followed by degraded performance for establishing new connections. # Incident Summary and Impact Shortly after 23:00 UTC on July 29, 2024, one of Hologram's connectivity partners began experiencing degraded performance, followed by an outage affecting new connections. This incident impacted some devices that make use of the G-1 SIM/profile, where attempts to create new data sessions failed at times, followed by degraded performance for later attempts to open data sessions. As the outage was isolated to that single connectivity partner, customers using other global profiles \(G-2, G-3\) or non-global native profiles were unaffected. Unfortunately, many customers rely upon the affected partner and had devices that could not open new cellular data sessions during the incident. Customers with many devices attempting to connect during the incident were most affected. Devices with data sessions that were already open before the incident and remained open during the incident continued to function normally. Multiple actions were attempted to restore the ability to make new connections, and to address subsequent network congestion. The two most impactful actions were: \(1\) the ability to establish new connections was initially restored by manually rerouting connection requests to a secondary \(backup\) network authorization resource at 00:30 UTC on July 30, and \(2\) the subsequent resource exhaustion on the secondary authorization resource was rectified at 04:45 UTC on July 30 when the primary resource was restored. By 04:45 UTC on July 30, the authorization service was thereafter able to process new connection requests successfully, and most of Hologram's affected customers saw a steady recovery. By 06:00 UTC, both primary and secondary network authorization resources were processing new connection requests normally. Some devices reconnected right away during either the first or second stages of recovery, other devices reconnected more slowly due to congestion, and others waited to reconnect until device-specific retry timers automatically expired. The combined \(complete\) unavailability for new connections totaled approximately 115 minutes during this incident, comprised of each of an 85-minute and a 30-minute unavailability period, respectively. At the conclusion of the second unavailability period, congestion from a backlog of requests resulted in degraded performance for new connections, and this congestion was mostly resolved within 1 hour. With performance no longer degraded and device retry timers progressively expiring, numbers of new connection requests returned to pre-incident levels within approximately 6.5 hours. # Root Cause and Technical Description In order to implement network authorization and metering of usage, the G-1 connectivity partner operates redundant authorization services in different data centers. Redundancy for these authorization services is implemented in an active-standby configuration, in part to help reserve sufficient resources on the secondary systems in order to handle the primary systems' full load immediately following the failover period. In the event of a failure of the primary authorization resources, the secondary resources are intended to process new connection attempts after only a minimal failover period. During this incident, the primary service experienced exhaustion of some of its system resources, and alarms were triggered as resources approached exhaustion. While alarms were being investigated by the connectivity partner, further resource exhaustion resulted in failed new connection attempts and a drop in the success rate for processing new connection requests. Unfortunately, as implemented, the authorization service's redundancy failed to properly handle this failure mode for two reasons: \(1\) the primary resources did not fail completely and were still responding to some requests, which resulted in a longer-than-typical failover period, and \(2\) even after manual failover, the secondary resources were insufficient and were exhausted shortly afterward as well, like with the primary resources. Sharing load across the primary and secondary resources, once both were restored, increased the speed of recovery, but some devices experienced degraded performance while attempting to establish new data connections following service restoration, and other devices waited for some time before attempting to reconnect according to device-specific retry timers. Root cause analysis identified the following root causes to this incident: \(1\) insufficient system resource management by the G-1 connectivity partner, and \(2\) insufficient automated failover criteria for the G-1 connectivity partner's network authorization resources. # Timeline ## July 29, 2024 * **23:00 UTC:** G-1 connectivity partner observed primary authorization resources nearing exhaustion \(alarm\) * **23:05 UTC:** G-1 connectivity partner observed primary authorization processing success rate of <100% * **23:05 UTC:** Hologram number of new sessions reported on G-1 SIMs/profiles began decreasing, while some new sessions succeeded after delays or multiple attempts, which indicated possible degraded performance in session reporting or possible degraded performance in new connection success rate ## July 30, 2024 * **Around 00:00 UTC:** Hologram observed further decrease in new sessions reported on G-1 SIMs/profiles, with successful connections reported at a significantly lower rate than typical * **00:10 UTC:** G-1 connectivity partner disabled lower-priority services to try to mitigate * **Around 00:15 UTC:** Hologram confirmed complete unavailability of new connections on G-1 SIMs/profiles * **Around 00:30 UTC:** G-1 connectivity partner confirmed complete unavailability of new connections * **00:30 UTC:** G-1 connectivity partner manually rerouted new connection attempts to secondary resources * **00:30 UTC:** G-1 connectivity partner observed secondary authorization processing success rate of 100%, Hologram new connection requests succeeding and first recovery begins * **02:35 UTC:** G-1 connectivity partner re-enabled lower-priority services * **04:00 UTC:** G-1 connectivity partner restored primary authorization resources * **04:10 UTC:** G-1 connectivity partner observed secondary authorization resources nearing exhaustion \(alarm\), a reproduction of the trigger that had occurred on the primary authorization resources prior * **04:15 UTC:** G-1 connectivity partner observed secondary authorization processing success rate of <100% * **04:20 UTC:** G-1 connectivity partner disabled lower-priority services to try to mitigate * **04:45 UTC:** G-1 connectivity partner rerouted new connection attempts to primary resources * **04:45 UTC:** G-1 connectivity partner observed primary authorization processing success rate of 100%, Hologram new connection requests succeeding and second recovery begins * **Around 05:45 UTC:** G-1 connectivity partner observed significant drop in congestion \(which had been caused by backlog of new connection requests\) * **06:00 UTC:** G-1 connectivity partner restored secondary authorization resources * **07:00 UTC:** G-1 connectivity partner re-enabled lower-priority services * **10:45 UTC:** G-1 connectivity partner observed signalling \(including new connection requests\) at pre-incident levels # Remediation and Next Steps This incident had a widespread impact, and we take reliability very seriously. While Hologram and its partners implement systems and safeguards to maximize reliability, we have identified several areas of improvement and will continue to work on uncovering any other gaps to prevent a reoccurrence. **Insufficient system resource management:** Root cause analysis revealed that the G-1 connectivity partner did not budget sufficient extra resources and did not alarm soon enough to prevent the rapid exhaustion of the resources on their authorization service. The G-1 connectivity partner is addressing system resource management and alarm configuration to prevent this from reoccurring. **Insufficient automated failover criteria:** Root cause analysis also revealed that the G-1 connectivity partner's automated failover checks did not positively identify the emerging unhealthiness of the primary authorization resources due to the continued \(degraded\) availability of the service endpoint. The G-1 connectivity partner is investigating a more holistic health check to use for initiating failover to secondary resources. While the incident is resolved on our status page and new cellular data connections are succeeding, we are continuing our efforts to maximize reliability and address these areas of improvement. # Conclusion Our aim at Hologram is to provide the most reliable cellular connectivity product available, and we clearly fell short of our customer expectations with this incident that affected new connection requests on G-1 SIMs/profiles. Although the root cause was with an upstream partner, we are ultimately accountable and are deeply sorry for the disruption to our customers. We have started working on the changes outlined above and will continue our diligence to prevent this from reoccurring. Customers using other global profiles \(G-2, G-3\) and non-global native profiles were not affected. Additionally, users of Hologram Hyper\+ SIMs benefit from multi-core technology that provides outage protection and guaranteed uptime SLAs. Customers should reach out if interested in exploring these solutions.

Signaling congestion and connectivity are now back to pre-incident levels.

The issue has been identified and traffic has been rerouted. Connectivity is still impacted by congestion but as this eases the devices should naturally come back online.

We're currently experiencing a partial outage on our cellular network specific to our G1 profiles (sims/profiles with ICCID prefix 89445*).

This incident has been resolved.

We are monitoring delayed processing of Verizon Usage records for July 26-29 2024. These delayed records are in the queue for processing, and we will provide an update when all the records have been updated.

We are currently experiencing an issue with usage reporting for Verizon direct profiles. We are in contact with Verizon and working to determine the root cause. Dashboard and API users may observe lack of usage data for sims connecting directly through Verizon. There is no impact to connectivity.

This incident has been resolved.

There is a currently a backlog of operations delaying SIM operations on our Global-1 carrier. Backlog is processing and some customers may experience delays with some SIM operations. Cellular network and other API services are unaffected.

This incident has been resolved.

We're processing considerably larger amounts of API requests than usual, which is causing timeouts and delays on API responses, also impacting our Dashboard.

All our systems and global pingers are operational. The problems have been resolved.

We're still monitoring progress, this issue is impacting roaming capabilities of local carriers across the region which means it's affecting all roaming sim cards (not just the range mentioned previously). Most MVNOs and MNOs on the region are affected.

We're still monitoring the issue with connectivity and data transmissions affecting multiple network and operators in the areas of South and North America. It was caused by an IPX Failure on one of the larger IPX providers for the region, which continues to take actions to resolve the issue. We've been seeing continuous improvements and increased stability, but devices may still experience delays in establishing connections and SMS.

Improvement has been seen widely across the network, links seems to be stable, but it will still take some time to get the signal KPIs back to normal. Customer may still experience delays in establishing data and SMS services when trying to establish connectivity.

Syneverse is having major problems throughout North and South America, affecting a large number of MNOs/MVNOs that use their services in the region. We're still monitoring and will update soon.

In some locations of North and South America, we are seeing an issue with a network partner causing some SIM cards on the 8944* ICCID range to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.

This incident has been resolved.

G1 carrier operation outage appears to be resolved, currently processing backlog of requests

G1 carrier requests are delayed again

G1 carrier requests are processing, now processing backlog of requests

There is a currently a backlog of operations delaying SIM operations on sims with the G1 carrier. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

The source of the problem has been identified and resolved. The backlog of usage events has been cleared and all systems are operating without delay.

Usage reporting and billing of usage are delayed. Network connectivity and other services are unaffected. Hologram engineering is investigating the cause of the delay.

This incident has been resolved.

SMS functionality is restored and backlogged MT SMS have been delivered.

We are continuing to investigate this issue.

We are working with carrier partners to diagnose an issue preventing certain SIMs from receiving MT SMS.

On March 14th, Hologram experienced degraded service, eventually resulting in a loss of connectivity, with one of our connectivity partners. This incident impacted some customers who use SIMs/profiles with ICCIDs beginning with the 8944\* prefix. During the incident, devices using these SIMs/profiles experienced a cellular data outage and degraded cellular data connectivity. # Incident Summary and Impact On March 14th around 23:00 UTC, one of our connectivity partners encountered degraded signalling service via one of their upstream interconnect providers. This degraded signalling eventually resulted in a disruption of all signalling via that interconnect provider. The loss of control plane signalling traffic resulted in the inability for affected devices to attach to towers, establish new data connections, and maintain existing data connections. The resulting degraded service and outage affected all SIMs/profiles with ICCIDs beginning with the 8944\* prefix that attempted connections during the incident. Customers using other profiles were unaffected. The upstream interconnect provider restored signalling services by 02:50 UTC on March 15th, at which time affected devices began reconnecting. Hologram monitored progress until all devices were restored. # Root Cause and Technical Description Root cause analysis is underway, and the interconnect provider and our connectivity partner are working to reproduce the issue in a sandboxed environment. This work is to confirm root cause and determine why performance degradation occurred and did not remain more isolated. # Timeline Root cause analysis is ongoing and we will post an update in the coming weeks including the timeline of events. # Remediation and Next Steps This incident had a widespread impact, and we take reliability very seriously. While Hologram and its partners implement systems and safeguards to maximize reliability, and we observed an improved recovery during this incident due to prior implemented changes, we will identify any additional areas of improvement and work to uncover any other containment gaps to prevent a recurrence of this type of root cause. Root cause analysis is ongoing and we will post an update in the coming weeks as to our findings and our next steps for improved containment and prevention based upon what is learned from this incident. # Conclusion Our aim at Hologram is to provide the most reliable cellular connectivity product available, and we clearly fell short of our customer expectations with this incident. Although the root cause was with an upstream partner, we are ultimately accountable and are deeply sorry for the disruption to our customers who were unable to use cellular data on this profile as expected due to the outage.

We've been monitoring the fix from our upstream providers and we can confirm the issue has been resolved. We will provide more information as we receive it and review the root cause analysis.

We are continuing to monitor for any further issues.

Our upstream provider has reported the issue is fixed and we are now monitoring to ensure that connectivity has been completely restored.

We are seeing an issue preventing some SIM cards with the 89445* prefix from being able to attach to the network and pass data.. We have escalated this to upstream carrier partners and they are actively investigating.

This incident has been resolved.

We are continuing to work on a fix for this issue.

There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

Connectivity has been restored in the region

Connectivity has been restored for almost all affected devices in the region and we are continuing to monitor.

There's a major internet outage affecting West and Central Africa, which might impact cellular connectivity services in some of these countries.

Services appear to be operational again and we've seen overage connectivity health in the US recover. We'll still monitor any further news and events and support our customers.

There's a major nationwide incident in the US, this incident is directly on local carriers infrastructure mostly affecting AT&T but also some T-Mobile and Verizon users. We're monitoring the situation closely and keeping open communications with our network partners.

This incident has been resolved.

There is a currently a backlog of operations delaying SIM operations on some carriers (activations, suspensions, data usage reports, etc). Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

This incident has been resolved.

There is a currently a backlog of operations delaying SIM operations on some carriers (activations, suspensions, data usage reports, etc). Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

This incident has been resolved.

We've identified a backlog on our SIM operations. Activations, state changes and plan changes might be delayed while the system processes the backlog.

This incident has been resolved.

We are seeing usage data recovering and are continuing to monitor

The carrier partner has confirmed the issue and is working on a fix

We have detected an issue with an upstream carrier partner causing some usage data to be delayed. There does not seem to be an issue with any cellular connectivity at this time and the issue is only with usage reporting and billing. We have escalated to our partners and are monitoring the situation.

This incident has been resolved.

There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

We've been monitoring the fix and we can confirm the issue has been resolved. We will provide more information when we get and review the root cause analysis.

The issue has been fixed, we're still monitoring that all configurations are properly done and that there's no remaining issues. We will also work with responsible parties to get a root cause analysis .

We've been advised that our partners provider is still working on resolving the issue, they're past the ETA provided. We're all working on resolving this as soon as possible.

The issue was identified on a backbone link of one of our partners providers. We are constantly pushing to obtain more details regarding this incident and will provide updates as we have more information.

We are seeing an issue with a network partner causing some SIM cards on the 89357* ICCID range to be unable to pass data on the network. We immediately reported to our upstream partner and received confirmation about an incident on their side, we're all investigating and will provide more details about impact and path to resolution.

On August 28, 2023, Hologram experienced an outage with one of our connectivity partners. This incident impacted all customers using SIMs/profiles with ICCIDs beginning with the 8944\* prefix. During the incident, devices using these SIMs/profiles experienced a cellular data outage and degraded cellular data connectivity. # Incident Summary and Impact Shortly after 19:00 UTC on August 28, 2023, one of Hologram's connectivity partners began experiencing degraded performance, followed by an outage. This incident impacted all customers using SIMs/profiles with ICCIDs beginning with the 8944\* prefix. During the incident, devices using these SIMs/profiles were unable to establish cellular data sessions at times, and experienced degraded cellular data connectivity at other times. As the outage was isolated to that single connectivity partner, customers using other profiles were unaffected. Unfortunately, many customers rely upon the affected partner and had devices whose cellular data connectivity was disrupted. Multiple actions were attempted to restore connectivity, and address subsequent network congestion. The most impactful action occurred around 0:500 UTC on August 30, 2023 when traffic was rerouted around an identified fault, after which Hologram's affected customers saw a steady recovery in service. Some devices reconnected right away, others reconnected more slowly due to congestion, and others waited to reconnect until device-specific retry timers reset. By 00:23 UTC on August 31, 2023, 95% of impacted devices had reconnected, and 99% by August 4, 2023. A subset of devices experienced shorter-length-than-expected data sessions after the outage. # Root Cause and Technical Description The root cause was identified as an erroneous configuration change introduced by our upstream partner's upstream interconnect provider, which occurred during planned but unannounced work in one to the interconnect  provider's data centers. The interconnect provider is a critical element that allows traffic to flow between multiple operators' networks, and is also a critical element in implementing routing redundancy to and from operators. Unfortunately, this configuration change resulted in abnormal routing across redundant components that undermined system reliability and safeguards that are in place. The root problem did not cause any of the component services to fail. Because of this, the location of the fault was not immediately apparent to the end-to-end tests that both Hologram and our upstream partner were running. Because individual components were functioning and health checks were \(correctly\) reporting nodes across the networks were healthy, it took a substantial time to identify and fix the root cause. After restarting several hot standby nodes in their network, and temporarily rerouting traffic, our upstream partner discovered that traffic routed via a specific set of data centers operated by their interconnect provider worked as expected. This allowed them to identify the data center that was failing silently and promptly remove it from their network as a workaround. This discovery led to locating the specific configuration that the interconnect provider had applied erroneously during planned works at that data center. # Timeline **August 28, 2023** * Around 19:00 UTC: Interconnect provider introduces erroneous configuration change to one of its data centers. This initially results in degradation and eventually an outage of cellular data for SIMs/profiles with ICCIDs beginning with the 8944\* prefix. * 19:39 UTC: Status page posted following Hologram detecting a probable outage as network conditions deteriorate. Hologram also escalates to our upstream connectivity partner. * 19:51 UTC: Upstream connectivity partner's NOC paged on-call engineering support, and additional engineering support was resourced. Multiple component services in partner's network are cleared of fault, and additional engineering resources are mobilized. **August 29, 2023** * 00:25 UTC: Upstream partner commences attempts to adjust/restart various nodes that are reporting abnormal traffic patterns related to signalling \(control plane for connection and session management\). * 05:36 UTC: Following failed attempts to restore service, including replacing signalling hardware links that were alerting and implementing temporary workarounds to increase signalling performance and capacity, partner executes their crisis management plan. * 10:00 UTC: Partner performed various traffic management actions, including blocking specific incoming traffic, shutting down specific links, and increasing capacity to allow network elements time to recover and catch up. Partner's network monitoring now indicates that their mitigating actions are having a positive effect. * 15:00 UTC: With network congestion stabilizing, and after implementing a shunt to further increase signalling performance, partner begins introducing traffic that was previously temporarily blocked. * 20:00 UTC: Partner observes traffic levels approaching normal, and restores all interconnect partner link configurations to their pre-incident state. * 23:59 UTC: Partner's network monitoring alerts that there is once again abnormal signalling traffic. Partner escalates to interconnect provider, who begins an immediate investigation. **August 30, 2023** * 05:00 UTC: After partner implemented a repeat of prior workarounds and traffic mitigations with the help of the interconnect provider, the interconnect provider now reverts the erroneous configuration change in their affected data center. * 07:30 UTC: Hologram continues to observe devices having issues connecting due to congestion during recovery. * 10:00 UTC: Partner observes signalling traffic levels approaching normal. Interconnect provider has commenced an investigation into their network configuration and planned works activities. * 19:38 UTC: Hologram observes 85% of affected devices have reconnected. * 21:11 UTC: Hologram observes 90% of affected devices have reconnected. **August 31, 2023** * 00:23 UTC: Hologram observes 95% of affected devices have reconnected. Hologram observes a small subset of devices with unstable data connections, with sessions dropping more often than expected. Hologram continues to escalate these findings. * 16:59 UTC: Hologram observes 97% of affected devices have reconnected. Hologram observes a small subset of devices with unstable data connections, with sessions dropping more often than expected. Hologram continues to escalate these findings. **September 5, 2023** * 17:14 UTC: Following the weekend, Hologram observed over 99% of affected devices had reconnected. Hologram continues to investigate the small subset of devices with sessions dropping more often than expected. * September 6, 2023 * 16:40 UTC: With the vast majority of affected devices connected and exhibiting normal behavior, Hologram closes the incident on our status page. # Remediation and Next Steps This incident had a widespread impact, and we take reliability very seriously. While Hologram and its partners implement systems and safeguards to maximize reliability, we have identified several areas of improvement and will continue to work on uncovering any other gaps to prevent a recurrence. **Process:** We are stressing that all planned works, no matter how small, should be announced for the entire connectivity supply chain. This allows Hologram to leverage our reporting and alerting to gauge impact \(or non-impact\) by correlating with known changes. Hologram has already stressed the importance of transparency with its direct partners, but it is evident that there's more work to do to ensure that transparency is extended to all critical upstream providers. Additionally, Hologram is proactively performing a fresh assessment of incident management and disaster recovery plans/procedures up the entire escalation path to ensure accountability for quick identification and resolution when issues arise. **Monitoring:** Hologram already heavily relies upon automated monitoring and alerting, and is committed to continuing to invest and innovate in this area. As a part of our fresh assessment of incident management and disaster recovery of critical providers, we will also assess areas of monitoring improvements for faster identification of issues. **Product:** In addition to the above, which we hope will minimize the frequency and duration of incidents, Hologram is investing in product enhancements that will prevent or minimize downtime when an incident does occur. While the incident is resolved on our status page and cellular data services are restored, we are continuing our efforts to maximize reliability and address these areas of improvement. # Conclusion Our aim at Hologram is to provide the most reliable cellular connectivity product available, and we clearly fell short of our customer expectations with this painful incident. Although the root cause was with an upstream partner, we are ultimately accountable and are deeply sorry for the disruption to our customers who were unable to use cellular data as expected due to the outage. We have started working on the changes outlined above and will continue our diligence to ensure this cannot happen again.

This incident has been resolved. Working with our upstream partner we have identified and addressed the root causes. Device connections have returned to normal levels and we are seeing consistent performance across our network. A very small number of devices could still be experiencing unusual behavior. If you have such a device, please contact support for device-specific diagnostics.

Our data shows that 99% of devices are connecting normally now after the weekend. We're still troubleshooting a small subset of devices that have unstable data connections (sessions dropping more often than expected).

Our data shows that 97% of devices are connecting normally. We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes.

Over 95% of affected devices have restored connectivity, with more devices coming back online each hour. We're still working with our upstream partners to ensure the same success on remaining devices. We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes

Based on our monitoring we've identified 90% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.

Based on our monitoring we've identified 85% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.

Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.

Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.

Our upstream partner is continuing to implement changes to bring devices back online, and is currently working to mitigate signaling congestion. With the current congestion, Hologram continues to see devices failing to register with carrier networks. We are continuing to monitor the situation.

Our upstream partner has shifted traffic to a new node and is beginning to see improvements in IoT device connectivity. We are continuing to monitor the situation. We will continue to post updates here as we learn more.

The previously identified fix for the failures in our upstream partner's network interfaces was not ultimately viable and was not implemented. Our partner believes they have now identified a common problem with the interface failures and are working on a fix.

The likely root cause of the regression has been identified on a node in our upstream partner's network interface. They engaged with their vendors and a solution has been identified, which will be executed in the next 30 minutes. The updated node will come back online and traffic will be gradually increased to it to ensure a stable recovery. The expected recovery time depends on the depth of backlog of connection requests, it will be a slow release to avoid overloading the signal, they're estimating recovery by 0700 to 0800 UTC.

Our upstream partner has identified the root cause of the regression. One of the provider's network interface was unable to support the amount of traffic being released and was compromised. The solution is currently being assessed, in which they'll migrate the signaling services to a different node that is operating normally and has bandwidth to support the incremental traffic. Latest estimate for full resolution is 0700 to 0800 UTC. This estimate is based on the assumption that the applied solutions work as intended.

After trending upwards close to a resolution, we noticed a regression on resolved sims. We escalated with our upstream partners and they confirmed there has been a major regression on the solution that was implemented. We're monitoring the impact and working with our partners to identify the root cause of this regression.

We've been monitoring progress and it's been steadily improving. We're already past 65% of resolution , we will keep monitoring and updating.

The latest update received from our upstream partner indicates signalling has been stabilized and congestion is now overcome with continuous monitoring ongoing. The network is processing traffic as per normal standard, though affected devices will take some time to all reattach. From timelines of previous similar incidents, it is expected that full recovery will likely take until well past midnight UTC. All teams are focusing on improving these timelines wherever possible.

There are still issues with the connectivity on our upstream partners network. The cause of this incident has now been rectified and stability has been confirmed. They're now facing a signaling storm due to congestion, they're restricting the traffic and are slowly increasing throughput to resolve this. We're still unable to provide an ETA but we started seeing improvements.

Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.

Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.

Our upstream partners have stabilized the replacement hardware and continue to work on the remaining network instability issues that are adversely affecting subscribe attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Unfortunately, bringing back the interconnect links have not yet had the expected effect on connectivity. Subscriber attachments are still adversely affected. There is no ETA yet on resolution. We will continue to post updates here as we learn more.

Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Subscriber attachments are still adversely affected. We will continue to post updates here as we learn more.

Our upstream partners have replaced faulty hardware and are continuing to work to resolve the issues that remain. We will continue to post updates here as we learn more.

Unfortunately the fix attempted by our upstream partners did not resolve the issue. They are continuing to investigate. We will post updates here as we learn more.

Upstream partners have identified the issue and are implementing a fix.

The issue has been identified. We will post updates here as we learn more.

This issue is affecting SIMs starting with the 8944* prefix. It appears to be preventing any attachment to the cellular network right now. This has been escalated to the highest level

We are seeing an issue with a network partner causing some SIM cards to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.

This incident has been resolved.

We are seeing traffic properly being rerouted away from the broken link and succeeding, and close to normal levels. We will continue monitoring this new route with our partners.

After reviewing different approaches with our upstream partners, they have started routing traffic via a different link. We don't have an ETA yet to full resolution but we are seeing some 4G registrations succeed.

Our upstream partners have traced the incident to a cut MedNautilus fiber cable and are working to reroute traffic

We are seeing an issue preventing some SIM cards with the 8935* prefix from being able to attach to the network and pass data. We have escalated this to upstream carrier partners and they are actively investigating.

We can confirm the incident has been resolved.

The issue was identified and fixed by the upstream partner. We are seeing steady improvements with devices coming back online every hour. We will continue to monitor.

We are seeing an issue with a network partner causing some SIM cards on the 8944* ICCID range to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.

This incident has been resolved.

We've determined the incident has been resolved and connectivity on the sim range 89357* has been restored.

The issue was promptly identified and fixed, we see almost full recovery, we will keep monitoring.

We are seeing an issue with a network partner causing some SIM cards on the 89357* ICCID range to be unable to pass data on the network. We immediately reported to our upstream partner and received confirmation about an incident on their side, we're all investigating and will provide more details about impact and path to resolution.

This incident has been resolved.

There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.

This incident has been resolved.

Our network monitoring systems have detected an issue with an upstream provider that is causing some SIMs that start with the 8935 ICCID prefix to have issues attaching to the network. Not all SIMs in this range are affected, but some subset is experiencing a problem. This has been escalated with the provider and they are actively working on the issue.

This incident has been resolved.

A fix has been identified and is being implemented by our upstream partner now. We will be monitoring to ensure any operations issued during the incident window are completed.

One of our upstream partners is experiencing an issue that is causing some device operations (such as pauses and activations) as well as SMS to experience significant delays. Cellular connectivity for devices remains unaffected. The issue has been escalated with our partner.