Tango Card

Is Tango Card Down Right Now? Check if there is a current outage ongoing.

Tango Card is currently Operational

Last checked from Tango Card's official status page

Historical record of incidents for Tango Card

Report: "Issues with adding credit cards"

Last update
resolved

This incident has been resolved.

monitoring

Tango systems experienced issues and errors over the past 4 hours (9 AM PST - 1 PM PST) where Users were unable to add or remove a credit card in prod and sandbox environments. Engineering has resolved these issues and is continuing to monitor Sandbox and Prod Environments

Report: "Issues with adding credit cards"

Last update
Resolved

This incident has been resolved.

Monitoring

Tango systems experienced issues and errors over the past 4 hours (9 AM PST - 1 PM PST) where Users were unable to add or remove a credit card in prod and sandbox environments.Engineering has resolved these issues and is continuing to monitor Sandbox and Prod Environments

Report: "User access level not displaying correctly"

Last update
resolved

This issue has been resolved.

identified

Tango portal is currently displaying user Access Level incorrectly. In Team Settings, a user’s Access Level is displayed as “Organization Access” even if they have access to only selected groups or accounts. This is a display issue only. Access Level has not been changed or impacted for any users. We are working on a resolution and anticipate releasing it later today.

Report: "Tango portal login is failing for all users who attempt to access portal"

Last update
resolved

Auth0 has resolved the issue. Logging into Tango portal is now operational again.

investigating

Users are unable to log into Tango portal due to an authentication system failure.

Report: "Disruptions on BHN AWS EKS Clusters impacted Multiple BHN Platforms"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

Order performance appears to be stabilizing. If you encountered a failure in the last 48 hours recommendation is to retry the order at this point.

investigating

We are seeing orders failing due to a supplier issue. We are actively investigating this.

Report: "Disruptions on BHN AWS EKS Clusters impacted Multiple BHN Platforms"

Last update
resolved

Resolution: BHN has successfully resolved the issue. Our team collaborated closely with AWS to address the impact caused by AWS EKS Clusters. We are pleased to confirm that the system has fully recovered as of 12:00 PM PT. We will now initiate the Root Cause Analysis (RCA) process to prevent future occurrences. This issue impacted Tango Platforms where Tango Orders were failing (Tango Portal and Tango API). This is resolved now.

investigating

Business Impact: BHN was experiencing EKS issues in the AWS service that impacted the following systems: 1. Japan Region: • Lawson -> LoadIt flow (Google) o Pre-Authorizations (1032) o Shipments (182) o PSLOADIT (571) o GMGTOP (242) 2. US Region: • Giftcards.com - D2C • Tango - B2B • Rakuten - Affiliate orders

Report: "Credit Card registrations and orders failing in production"

Last update
resolved

In Tango portal it is not possible to add a new credit card or place an order with a credit card. It is not possible to fund with a credit card via API.

Report: "Landing Pages Incident"

Last update
resolved

Tango Teams have resolved this issue and confirmed Landing Pages are functional

investigating

Tango Teams have identified issues with Landing Pages starting at 11:21 pst. The team is investigating issues.

Report: "Intermittent latency across products"

Last update
resolved

Errors have resolved and systems are back to normal. We will continue monitoring.

investigating

We are currently investigating some intermittent latency across Tango products.

Report: "Portal -- not able to resend Reward Links"

Last update
resolved

An update to the Tango portal was released at 10:37am PT that inadvertently completely disabled resends for Reward Links. That update was been reverted at 1:38pm PT and proper functionality has returned. If you experienced this behavior, please retry as issue has been resolved.

Report: "[SANDBOX] An "Unable to retrieve catalog" error was preventing catalogs from loading"

Last update
resolved

This incident has been resolved and Sandbox will continue to be monitored closely.

investigating

A memory issue is causing catalogs in Tango portal sandbox to not load properly.

Report: "We are experiencing an intermittent issue with Sandbox Tango Portal Login. We are investigating this actively."

Last update
resolved

This incident has been resolved.

investigating

We have resolved the cause of the intermittent issues.

Report: "ACH funding via Tango Portal"

Last update
resolved

This issue has been resolved.

investigating

Customers cannot add funds to their account via ACH in Tango Portal. We are investigating this issue and do not currently have an ETA for resolution. Until it is resolved, customers who wish to fund their account can use a credit card or contact funding@tangocard.com for assistance. Funding via credit card is working properly. However, customers may need to refresh their browser tab to confirm that the credit card transaction has completed successfully.

investigating

Customers cannot add funds to their account via ACH in Tango Portal. We are investigating this issue and do not currently have an ETA for resolution.

Report: "Bulk Orders stall at 0%"

Last update
resolved

This incident has been resolved.

monitoring

The problem has been resolved in the Prod environment and we are closing this incident.

identified

We are continuing to work on a fix for this issue.

identified

Tango Teams have identified the problem and we have deployed the solution. We will continue to monitor the behavior.

investigating

We are currently investigating an issue with Bulk Orders. Tango teams are actively investigating and working to resolve this.

Report: "Global Choice Link is NOT doing currency conversion"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

We are currently an investigating an issue with Global Choice Link not performing currency conversion.

Report: "Reported issues with saml SSO and Bulk orders"

Last update
postmortem

[https://tangocard.atlassian.net/browse/INTREQ-8414](https://tangocard.atlassian.net/browse/INTREQ-8414) includes findings, conclusions and decisions. Details are logged to the following slack channels: * tango-incidents-outages * temp-july-eng-on-call-sanity-validation * temp-ecs-hardening * team-eng-tx

resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Tango engineering teams are investigating reported issues with Bulk Orders and saml SSO login. A fix has been applied and we are requesting customer retry.

Report: "Tango Landing Pages unable to load"

Last update
resolved

Recipients may have encountered errors when trying to access their gift card credentials.

Report: "Tango Portal Login Issues"

Last update
resolved

This issue has been resolved now !!

investigating

We have multiple customers raising login related issues... SSO Fully enabled platforms: If SSO is fully activated in your Tango Portal platform and after this when a user is created, clicking on the invitation email allows setting up MFA, but after that fails with an error message: "There was a problem logging you in". Deleted and re-added users: Portal users who are deleted and then re-added are incorrectly directed to the signup flow to set up a password, however, their old password is actually still active. Clicking on the invite and trying to sign in throws an error: "Something went wrong, please try again later" Google SSO users: Some Google SSO customers are not able to login and they get the message: "There was a problem logging you in" The team is working on fixing these issues at the earliest, at the moment we do not have an ETA, but as soon as these are addressed or we have an ETA, we will update the status here immediately.

Report: "Some users were experiencing lose of functionality due to MFA update"

Last update
resolved

This issue has been resolved. All users should have access to their normal functionality in the portal.

monitoring

Some users with limited account access lost portal functionality due to the MFA update. This has been addressed and everyone should now have full functionality restored.

Report: "Users without Organization access are unable to fund their accounts. They get the following warning message - “Funding limit of $NaN reached. To request a limit increase reach out to funding@tangocard.com“"

Last update
resolved

UPDATE: We have fixed this issue, there is no longer a need to refresh the page. WORKAROUND: Please refresh your page and then this issue will go away. This is a temporary workaround. In the meanwhile the team is working to implement a solution for this issue.

investigating

We are investigating this.

Report: "Tango SSO Certificate has an issue"

Last update
resolved

This incident has been resolved.

monitoring

The Tango Cert was updated on June 1 and has an issue. A new Cert has been created and should now be working.

Report: "Tango Portal experiencing delayed/missing data in line items history page"

Last update
resolved

This incident is now resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Customers and users attempting to access/view line item history in Tango Portal may see some delayed and/or missing data. The issue started around 6pm PST yesterday.

Report: "Issue with token generation between Twilio and Tango"

Last update
resolved

This incident is resolved! Please report if any issue is experienced nonetheless.

investigating

We are aware of an issue in production where tokens were not getting generated as expected for MFA auth. It may have impacted customers in the past ~20mins. The issue has been resolved since and we're are actively monitoring it.

Report: "Unable to view account transactions"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are continuing to investigate this issue.

investigating

Our Tango Portal datasource is experiencing unexpected behavior regarding the ability to view account transactions. 
We have investigated the problem and we are about to restore and sync data from the beginning of this week. This means that the transaction history for this week will temporarily be unavailable for 3 -4 hours. We expect this issue to be resolved and fully synced within that timeframe, and updates will be posted to this status page.

Report: "Customer reporting they can not see credit card info"

Last update
resolved

This incident has been resolved.

identified

We are continuing to work on a fix for this issue.

identified

The issue has been identified and a fix is being implemented.

investigating

Some customers are reporting they are unable to see their credit card information. TangoCard is investigating and has mitigated the immediate problem. TangoCard is actively resolving this issue for our impacted customers
 We will provide updates as more information is available.

Report: "Primary Order System Suffering Slowdown"

Last update
resolved

The issue is resolved now. Select orders that failed between 9:30-11:30am PST can be retried and should go through successfully.

identified

The issue has been identified and a fix is being implemented.

monitoring

A fix is in, and we are monitoring. Processing appears to be back to normal flows.

identified

As of around 9:30 am Pacific a slowdown in our order systems causing many errors in our processing system. This issue is identified.

Report: "Account Balance not updating correctly"

Last update
resolved

This incident has been resolved.

investigating

We are currently investigating the issue.

Report: "Tango Reward Link experiencing errors"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified the root cause and are working on a solution.

investigating

We are currently investigating this issue.

Report: "Tango Landing Pages experiencing errors"

Last update
resolved

This incident has been resolved.

monitoring

We have implemented the fix and the landing pages are working again. We will continue to monitor to ensure all errors have been resolved.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently experiencing errors for Tango landing pages. We believe we have identified the issue and are working on a fix.

Report: "Some customers are experiencing issues using Tango Sandbox environment."

Last update
resolved

We are seeing successful logins in Sandbox so confirming that the issue is resolved.

monitoring

We are seeing successful logins in Sandbox so confirming that the issue is resolved.

monitoring

A fix has been identified and we're validating with some affected customers whether the resolution has unblocked them. We're continuing to monitor for any additional concerns.

investigating

Some customers are experiencing issues using Tango Sandbox environment. We're investigating the problem.

Report: "Some customers experiencing issue with Tango Portal in Sandbox"

Last update
resolved

We are seeing successful logins in Sandbox Portal so confirming that the issue is resolved.

investigating

Some customers experiencing issue with Tango Portal in Sandbox. We're investigating the problem.

Report: "Log in via MFA email blocked"

Last update
resolved

Issue is resolved and affected customer(s) are able to log in.

investigating

We're investigating the issue.

Report: "Users encountering Registration Failed error at MFA step (in Self-registration flow)"

Last update
resolved

This issue is resolved now.

investigating

We are continuing to investigate this issue.

investigating

We are investigating this issue

Report: "Elevated error rate for limited number of LPE reward activations"

Last update
resolved

The change has been rolled back and reward activations should be successful moving forward.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We've identified the issue and are working on rolling back the change that is causing the elevated error rate. We will monitor and ensure that this fixes the issue.

investigating

We are currently investigating an elevated error rate for a limited number of LPE reward activations.

Report: "Images for some brands and/or items may not appear"

Last update
resolved

All brand images updated yesterday and today are in a good state

identified

The issue has been identified and the fix has been implemented. We are gathering all brand and images affected and will get the imaged uploaded correctly.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Issues Depositing to Bank or Credit Card in Disbursement Link"

Last update
resolved

We have identified and corrected a networking issue preventing communication with our supplier. All disbursement link orders should now be processed correctly.

investigating

We are currently unable to fulfill "Deposit to Bank Account" or "Deposit to Card" Reward Link options. We are continuing to investigate this issue.

investigating

We are currently investigating this issue.

Report: "Duplicate emails being sent to certain recipients."

Last update
resolved

We have verified orders are no longer causing the resend issue and have corrected all of the underlying data for the impacted line items. Issue is resolved.

monitoring

We've identified the cause of resends and fixed the issue. Currently monitoring for any regressions / recurrences.

investigating

We are aware that some recipients are receiving multiple duplicate emails containing their reward links. Our team is actively investigating the issue.

Report: "Incident preventing redemption of some rewards."

Last update
resolved

This incident has been resolved.

monitoring

We have fixed the affected items, and will continue to monitor and carry out corrective actions to prevent later recurrence.

identified

We are working on corrective actions to make the affected Reward Link products redeemable.

investigating

We are actively investigating an incident which prevents recipients from redeeming rewards for a small number of products.

Report: "RaaS currently down in Production"

Last update
resolved

This incident has been resolved.

monitoring

From 7:51am PT to 8:38am PT, RaaS experienced a period of high latency, resulting in high timeout rates, although a low error rate of 2.6%. Even if your request timed out, the order may have succeeded asynchronously. Please retry with the idempotent external reference id, which will ensure you receive the order already completed rather than placing a new order

investigating

We are currently investigating this issue.

Report: "Reward Link elevated errors"

Last update
resolved

We started seeing elevated errors for Reward Link redemptions around 4:30PM PST. We resolved the issue at 5:08PM PST and redemptions should be back to normal. Please retry as necessary.

Report: "[SANDBOX] Sandbox Rewards Genius, Sandbox RaaS and Sandbox Reward Link are currently down"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

We are currently investigating an issue resulting in Rewards Genius, RaaS and Reward Links to be down in the SANDBOX environment ONLY.

Report: "Email Delivery Failures"

Last update
resolved

All emails have been delivered and we are correctly displaying the current statuses.

monitoring

Services have been restored. We are checking on any potentially impacted deliveries during the incident window.

monitoring

A fix has been implemented and we are monitoring the results.

identified

We have identified the issue and are working on the restoration of services.

Report: "Reward Link URLs experiencing issues"

Last update
resolved

This incident has been resolved.

identified

We believe we have identified the issue and are working on rolling back to previous state to fix the issue.

investigating

We are currently investigating an issue with a small number of Reward Link URLs leading to an error message.

Report: "Intermittent connectivity issues with AWS"

Last update
resolved

Rewards Genius, RaaS, and RewardLink are experiencing intermittent connectivity issues while AWS resolves connectivity issues.

Report: "Notice of Maintenance"

Last update
resolved

This incident has been resolved.

investigating

We are continuing to investigate this issue.

investigating

Dear Partner, Tango Card will be performing maintenance on our order search functionality in response to a recently released AWS urgent hotfix. This may intermittently impact order history searches in Rewards Genius and RaaS. Time Window Start: Tuesday 12/14/2021 5:00 PM PDT End: Wednesday 12/15/2021 12:00 AM PDT Maintenance Description: We will be applying an urgent patch released by AWS Downtime Behavior: During the maintenance period you may experience the following system behaviors: - Order history searches returning no data If you experience this error during the Time Window, please try again after the maintenance window has ended. If you have any questions, please do not hesitate to reach out to your Customer Success Manager or our Technical Support Team. Best Regards, The Account Team @ Tango Card

Report: "Rewards Genius is Under Maintenance"

Last update
resolved

Maintenance has been completed

investigating

We have put Rewards Genius into a temporary maintenance mode. We will provide an updates as available.

Report: "Temporary Outage"

Last update
resolved

The issue has been resolved.

monitoring

The change has been reverted and we are monitoring to ensure stability.

identified

We are experiencing an issue with and upgrade affecting Rewards Genius and Reward Link. We have identified the issue and are reverting changes.

Report: "Seeing intermittent issues"

Last update
resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

Rewards Genius and RaaS API have recovered. We are continuing to investigate efforts to bring Reward Link back online as well

investigating

We are continuing to investigate this issue.

investigating

Rewards Genius and the RaaS API are back online. Working to bring Reward Link back online.

investigating

We are currently investigating this issue.

Report: "Some eGift Cards failing to load"

Last update
resolved

EGift Cards that are hosted on rewardcodes.com were failing to load from 1:08-1:25pm PT. The issue has been resolved and these eGift Cards are now loading.

Report: "Rewards Genius Unavailable Due to Outage"

Last update
resolved

We have resolved this issue and Rewards Genius is back online.

investigating

We have identified an issue with Rewards Genius and have taken the application down while we investigate. We will update as soon as we know more.

Report: "Rewards Genius Order History may be delayed in displaying orders"

Last update
resolved

We have resolved the issue, and all orders are now displaying in order history.

identified

We have identified the issue, and are working on a fix.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating an issue where some orders are not displaying in the Rewards Genius order history tab. We advise that orders placed may have successfully completed, but are not displaying in the order history. We will update once we know more.