Campaign Registry

Is Campaign Registry Down Right Now? Check if there is a current outage ongoing.

Campaign Registry is currently Operational

Last checked from Campaign Registry's official status page

Historical record of incidents for Campaign Registry

Report: "6.10.0 Release"

Last update
Completed

The scheduled maintenance has been completed.

Verifying

Verification is currently underway for the maintenance items.

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Scheduled

We will be undergoing scheduled maintenance during this time.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "DB under heavy load"

Last update
resolved

Some actor is performing some heavy duty search depleting DB resources. Other searches or operations might be impacted around this time, too.

Report: "redis outage"

Last update
resolved

we already restored the service.

identified

currently there is a partial outage impacting SMS OTP

identified

we are escalating to redis about this. this also happened 12 hours ago during release window. please refer to status.redis.com for their updates.

Report: "IRS failures"

Last update
resolved

IRS is currently experiencing a high rate of failed lookups. Retries will continue.

Report: "IRS outage"

Last update
resolved

IRS outage

Report: "Reported time-outs on Brand registration attempts via the CSP Portal or API (using blocking mode)"

Last update
resolved

This incident has been resolved.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are continuing to monitor for any further issues.

monitoring

We are currently observing time-outs during Brands registration attempts submitted via the CSP Portal or through the API using Synchronous (blocking) mode. Our vendor is currently working on the issue and are monitoring the situation very closely. As an alternative, we recommend using Asynchronous (non-blocking) Brand registration method available via our API. This method will allow to register your Brands without experiencing time-outs.

Report: "AEGIS service outage"

Last update
resolved

AEGIS services are back to normal

identified

we got notified by AEGIS of such IRS related outages

investigating

we start seeing massive AEGIS failures

Report: "spring4shell vulnerability"

Last update
resolved

This incident has been resolved.

monitoring

the fix is pushed to production and we're monitoring it.

identified

we upgraded spring framework to another newer version 5.3.18 to apply the official RCE fix

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "OSR outage"

Last update
resolved

been monitoring it for over 2 hours and it's been stable, thus the incident has been resolved.

monitoring

NetNumber has restored the connectivity in Boston and validated connectivity.

identified

NetNumber is having a network problem with Boston Data Center and they are working to restore service quickly.

identified

We're experiencing NetNumber Boston connectivity issue, awaiting NetNumber's response.

investigating

We are expriencing OSR IPSec connection issues. Looking into it

Report: "OSR down"

Last update
resolved

It's been functioning well since NetNumber fixed OSR and the campaign queue was consumed fine. Campaigns are now able to be registered against OSR again.

monitoring

NetNumber has restored OSR services and we are able to access it again. Will keep monitoring it for next few hours.

identified

The NetNumber BOS Site and Services remain unavailable at this time. NetNumber is actively working with on-site personnel to restore services. Estimated restoration time is ~06:00am EST

identified

NetNumber confirmed that Boston datacenter is having issues with OSR services, which is not available in Chicago datacenter. They are actively working on it to restore access.

investigating

Looks like OSR services are down. We're trying to switch over to another datacenter.

Report: "Duplicate Campaign Billed Events"

Last update
resolved

After hours of monitoring, we observed that those duplicate events cease to exist. This incident is resolved.

monitoring

The fix is pushed to production and we're monitoring the results.

identified

The fix is deployed to testing environment and verified.

identified

The bug affected only the expired campaigns. The root cause has been identified.

investigating

We discovered the issue that incorrectly generated duplicate CAMPAIGN_BILLED events for the change introduced in the 3.5.0 release.

Report: "Intermittent time-outs towards OSR provider"

Last update
resolved

We have updated our system to mitigate the issue when there is a downstream delay in our partner's network. Any time-out events experienced yesterday should now be resolved and we recommend enabling your retry mechanism to complete Campaign updates/registrations. Our team continues to monitor for any recurrence.

monitoring

We have implemented the fix, tested it and pushed to production. Will actively monitor production results.

identified

We have a fix for this time-out issue and now are testing it to push to production soon.

identified

Customers may experience intermittent time-out errors while trying to register Campaigns via the TCR Web Portal or API. This issue is due to time-outs towards our OSR partner. The incident has been reported to our OSR partner. Additionally, TCR engineers are working to mitigate the issue while our OSR partner works towards a resolution. We recommend our customers to enable their retry functionality if a time-out error is encountered while trying to register a Campaign.

investigating

Customers may experience intermittent errors accessing the CSP, DCA and MNO Web Portals. Additionally, customers may experience intermittent time-out errors when trying to access TCR API endpoints. This is currently impacting Brand and Campaign registration. The issue started at approximately 12:40 PM EST. It is currently under investigation.