Historical record of incidents for Campaign Registry
Report: "6.10.0 Release"
Last updateThe scheduled maintenance has been completed.
Verification is currently underway for the maintenance items.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
We will be undergoing scheduled maintenance during this time.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "DB under heavy load"
Last updateSome actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.
Report: "redis outage"
Last updatewe already restored the service.
currently there is a partial outage impacting SMS OTP
we are escalating to redis about this. this also happened 12 hours ago during release window. please refer to status.redis.com for their updates.
Report: "IRS failures"
Last updateIRS is currently experiencing a high rate of failed lookups. Retries will continue.
Report: "IRS outage"
Last updateIRS outage
Report: "Reported time-outs on Brand registration attempts via the CSP Portal or API (using blocking mode)"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
We are currently observing time-outs during Brands registration attempts submitted via the CSP Portal or through the API using Synchronous (blocking) mode. Our vendor is currently working on the issue and are monitoring the situation very closely. As an alternative, we recommend using Asynchronous (non-blocking) Brand registration method available via our API. This method will allow to register your Brands without experiencing time-outs.
Report: "AEGIS service outage"
Last updateAEGIS services are back to normal
we got notified by AEGIS of such IRS related outages
we start seeing massive AEGIS failures
Report: "spring4shell vulnerability"
Last updateThis incident has been resolved.
the fix is pushed to production and we're monitoring it.
we upgraded spring framework to another newer version 5.3.18 to apply the official RCE fix
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "OSR outage"
Last updatebeen monitoring it for over 2 hours and it's been stable, thus the incident has been resolved.
NetNumber has restored the connectivity in Boston and validated connectivity.
NetNumber is having a network problem with Boston Data Center and they are working to restore service quickly.
We're experiencing NetNumber Boston connectivity issue, awaiting NetNumber's response.
We are expriencing OSR IPSec connection issues. Looking into it
Report: "OSR down"
Last updateIt's been functioning well since NetNumber fixed OSR and the campaign queue was consumed fine. Campaigns are now able to be registered against OSR again.
NetNumber has restored OSR services and we are able to access it again. Will keep monitoring it for next few hours.
The NetNumber BOS Site and Services remain unavailable at this time. NetNumber is actively working with on-site personnel to restore services. Estimated restoration time is ~06:00am EST
NetNumber confirmed that Boston datacenter is having issues with OSR services, which is not available in Chicago datacenter. They are actively working on it to restore access.
Looks like OSR services are down. We're trying to switch over to another datacenter.
Report: "Duplicate Campaign Billed Events"
Last updateAfter hours of monitoring, we observed that those duplicate events cease to exist. This incident is resolved.
The fix is pushed to production and we're monitoring the results.
The fix is deployed to testing environment and verified.
The bug affected only the expired campaigns. The root cause has been identified.
We discovered the issue that incorrectly generated duplicate CAMPAIGN_BILLED events for the change introduced in the 3.5.0 release.
Report: "Intermittent time-outs towards OSR provider"
Last updateWe have updated our system to mitigate the issue when there is a downstream delay in our partner's network. Any time-out events experienced yesterday should now be resolved and we recommend enabling your retry mechanism to complete Campaign updates/registrations. Our team continues to monitor for any recurrence.
We have implemented the fix, tested it and pushed to production. Will actively monitor production results.
We have a fix for this time-out issue and now are testing it to push to production soon.
Customers may experience intermittent time-out errors while trying to register Campaigns via the TCR Web Portal or API. This issue is due to time-outs towards our OSR partner. The incident has been reported to our OSR partner. Additionally, TCR engineers are working to mitigate the issue while our OSR partner works towards a resolution. We recommend our customers to enable their retry functionality if a time-out error is encountered while trying to register a Campaign.
Customers may experience intermittent errors accessing the CSP, DCA and MNO Web Portals. Additionally, customers may experience intermittent time-out errors when trying to access TCR API endpoints. This is currently impacting Brand and Campaign registration. The issue started at approximately 12:40 PM EST. It is currently under investigation.