Historical record of incidents for Tango Card
Report: "Issues with adding credit cards"
Last updateThis incident has been resolved.
Tango systems experienced issues and errors over the past 4 hours (9 AM PST - 1 PM PST) where Users were unable to add or remove a credit card in prod and sandbox environments. Engineering has resolved these issues and is continuing to monitor Sandbox and Prod Environments
Report: "Issues with adding credit cards"
Last updateThis incident has been resolved.
Tango systems experienced issues and errors over the past 4 hours (9 AM PST - 1 PM PST) where Users were unable to add or remove a credit card in prod and sandbox environments.Engineering has resolved these issues and is continuing to monitor Sandbox and Prod Environments
Report: "User access level not displaying correctly"
Last updateThis issue has been resolved.
Tango portal is currently displaying user Access Level incorrectly. In Team Settings, a user’s Access Level is displayed as “Organization Access” even if they have access to only selected groups or accounts. This is a display issue only. Access Level has not been changed or impacted for any users. We are working on a resolution and anticipate releasing it later today.
Report: "Tango portal login is failing for all users who attempt to access portal"
Last updateAuth0 has resolved the issue. Logging into Tango portal is now operational again.
Users are unable to log into Tango portal due to an authentication system failure.
Report: "Disruptions on BHN AWS EKS Clusters impacted Multiple BHN Platforms"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Order performance appears to be stabilizing. If you encountered a failure in the last 48 hours recommendation is to retry the order at this point.
We are seeing orders failing due to a supplier issue. We are actively investigating this.
Report: "Disruptions on BHN AWS EKS Clusters impacted Multiple BHN Platforms"
Last updateResolution: BHN has successfully resolved the issue. Our team collaborated closely with AWS to address the impact caused by AWS EKS Clusters. We are pleased to confirm that the system has fully recovered as of 12:00 PM PT. We will now initiate the Root Cause Analysis (RCA) process to prevent future occurrences. This issue impacted Tango Platforms where Tango Orders were failing (Tango Portal and Tango API). This is resolved now.
Business Impact: BHN was experiencing EKS issues in the AWS service that impacted the following systems: 1. Japan Region: • Lawson -> LoadIt flow (Google) o Pre-Authorizations (1032) o Shipments (182) o PSLOADIT (571) o GMGTOP (242) 2. US Region: • Giftcards.com - D2C • Tango - B2B • Rakuten - Affiliate orders
Report: "Credit Card registrations and orders failing in production"
Last updateIn Tango portal it is not possible to add a new credit card or place an order with a credit card. It is not possible to fund with a credit card via API.
Report: "Landing Pages Incident"
Last updateTango Teams have resolved this issue and confirmed Landing Pages are functional
Tango Teams have identified issues with Landing Pages starting at 11:21 pst. The team is investigating issues.
Report: "Intermittent latency across products"
Last updateErrors have resolved and systems are back to normal. We will continue monitoring.
We are currently investigating some intermittent latency across Tango products.
Report: "Portal -- not able to resend Reward Links"
Last updateAn update to the Tango portal was released at 10:37am PT that inadvertently completely disabled resends for Reward Links. That update was been reverted at 1:38pm PT and proper functionality has returned. If you experienced this behavior, please retry as issue has been resolved.
Report: "[SANDBOX] An "Unable to retrieve catalog" error was preventing catalogs from loading"
Last updateThis incident has been resolved and Sandbox will continue to be monitored closely.
A memory issue is causing catalogs in Tango portal sandbox to not load properly.
Report: "We are experiencing an intermittent issue with Sandbox Tango Portal Login. We are investigating this actively."
Last updateThis incident has been resolved.
We have resolved the cause of the intermittent issues.
Report: "ACH funding via Tango Portal"
Last updateThis issue has been resolved.
Customers cannot add funds to their account via ACH in Tango Portal. We are investigating this issue and do not currently have an ETA for resolution. Until it is resolved, customers who wish to fund their account can use a credit card or contact funding@tangocard.com for assistance. Funding via credit card is working properly. However, customers may need to refresh their browser tab to confirm that the credit card transaction has completed successfully.
Customers cannot add funds to their account via ACH in Tango Portal. We are investigating this issue and do not currently have an ETA for resolution.
Report: "Bulk Orders stall at 0%"
Last updateThis incident has been resolved.
The problem has been resolved in the Prod environment and we are closing this incident.
We are continuing to work on a fix for this issue.
Tango Teams have identified the problem and we have deployed the solution. We will continue to monitor the behavior.
We are currently investigating an issue with Bulk Orders. Tango teams are actively investigating and working to resolve this.
Report: "Global Choice Link is NOT doing currency conversion"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently an investigating an issue with Global Choice Link not performing currency conversion.
Report: "Reported issues with saml SSO and Bulk orders"
Last update[https://tangocard.atlassian.net/browse/INTREQ-8414](https://tangocard.atlassian.net/browse/INTREQ-8414) includes findings, conclusions and decisions. Details are logged to the following slack channels: * tango-incidents-outages * temp-july-eng-on-call-sanity-validation * temp-ecs-hardening * team-eng-tx
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
Tango engineering teams are investigating reported issues with Bulk Orders and saml SSO login. A fix has been applied and we are requesting customer retry.
Report: "Tango Landing Pages unable to load"
Last updateRecipients may have encountered errors when trying to access their gift card credentials.
Report: "Tango Portal Login Issues"
Last updateThis issue has been resolved now !!
We have multiple customers raising login related issues... SSO Fully enabled platforms: If SSO is fully activated in your Tango Portal platform and after this when a user is created, clicking on the invitation email allows setting up MFA, but after that fails with an error message: "There was a problem logging you in". Deleted and re-added users: Portal users who are deleted and then re-added are incorrectly directed to the signup flow to set up a password, however, their old password is actually still active. Clicking on the invite and trying to sign in throws an error: "Something went wrong, please try again later" Google SSO users: Some Google SSO customers are not able to login and they get the message: "There was a problem logging you in" The team is working on fixing these issues at the earliest, at the moment we do not have an ETA, but as soon as these are addressed or we have an ETA, we will update the status here immediately.
Report: "Some users were experiencing lose of functionality due to MFA update"
Last updateThis issue has been resolved. All users should have access to their normal functionality in the portal.
Some users with limited account access lost portal functionality due to the MFA update. This has been addressed and everyone should now have full functionality restored.
Report: "Users without Organization access are unable to fund their accounts. They get the following warning message - “Funding limit of $NaN reached. To request a limit increase reach out to funding@tangocard.com“"
Last updateUPDATE: We have fixed this issue, there is no longer a need to refresh the page. WORKAROUND: Please refresh your page and then this issue will go away. This is a temporary workaround. In the meanwhile the team is working to implement a solution for this issue.
We are investigating this.
Report: "Tango SSO Certificate has an issue"
Last updateThis incident has been resolved.
The Tango Cert was updated on June 1 and has an issue. A new Cert has been created and should now be working.
Report: "Tango Portal experiencing delayed/missing data in line items history page"
Last updateThis incident is now resolved.
A fix has been implemented and we are monitoring the results.
Customers and users attempting to access/view line item history in Tango Portal may see some delayed and/or missing data. The issue started around 6pm PST yesterday.
Report: "Issue with token generation between Twilio and Tango"
Last updateThis incident is resolved! Please report if any issue is experienced nonetheless.
We are aware of an issue in production where tokens were not getting generated as expected for MFA auth. It may have impacted customers in the past ~20mins. The issue has been resolved since and we're are actively monitoring it.
Report: "Unable to view account transactions"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
Our Tango Portal datasource is experiencing unexpected behavior regarding the ability to view account transactions. We have investigated the problem and we are about to restore and sync data from the beginning of this week. This means that the transaction history for this week will temporarily be unavailable for 3 -4 hours. We expect this issue to be resolved and fully synced within that timeframe, and updates will be posted to this status page.
Report: "Customer reporting they can not see credit card info"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
Some customers are reporting they are unable to see their credit card information. TangoCard is investigating and has mitigated the immediate problem. TangoCard is actively resolving this issue for our impacted customers We will provide updates as more information is available.
Report: "Primary Order System Suffering Slowdown"
Last updateThe issue is resolved now. Select orders that failed between 9:30-11:30am PST can be retried and should go through successfully.
The issue has been identified and a fix is being implemented.
A fix is in, and we are monitoring. Processing appears to be back to normal flows.
As of around 9:30 am Pacific a slowdown in our order systems causing many errors in our processing system. This issue is identified.
Report: "Account Balance not updating correctly"
Last updateThis incident has been resolved.
We are currently investigating the issue.
Report: "Tango Reward Link experiencing errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the root cause and are working on a solution.
We are currently investigating this issue.
Report: "Tango Landing Pages experiencing errors"
Last updateThis incident has been resolved.
We have implemented the fix and the landing pages are working again. We will continue to monitor to ensure all errors have been resolved.
The issue has been identified and a fix is being implemented.
We are currently experiencing errors for Tango landing pages. We believe we have identified the issue and are working on a fix.
Report: "Some customers are experiencing issues using Tango Sandbox environment."
Last updateWe are seeing successful logins in Sandbox so confirming that the issue is resolved.
We are seeing successful logins in Sandbox so confirming that the issue is resolved.
A fix has been identified and we're validating with some affected customers whether the resolution has unblocked them. We're continuing to monitor for any additional concerns.
Some customers are experiencing issues using Tango Sandbox environment. We're investigating the problem.
Report: "Some customers experiencing issue with Tango Portal in Sandbox"
Last updateWe are seeing successful logins in Sandbox Portal so confirming that the issue is resolved.
Some customers experiencing issue with Tango Portal in Sandbox. We're investigating the problem.
Report: "Log in via MFA email blocked"
Last updateIssue is resolved and affected customer(s) are able to log in.
We're investigating the issue.
Report: "Users encountering Registration Failed error at MFA step (in Self-registration flow)"
Last updateThis issue is resolved now.
We are continuing to investigate this issue.
We are investigating this issue
Report: "Elevated error rate for limited number of LPE reward activations"
Last updateThe change has been rolled back and reward activations should be successful moving forward.
A fix has been implemented and we are monitoring the results.
We've identified the issue and are working on rolling back the change that is causing the elevated error rate. We will monitor and ensure that this fixes the issue.
We are currently investigating an elevated error rate for a limited number of LPE reward activations.
Report: "Images for some brands and/or items may not appear"
Last updateAll brand images updated yesterday and today are in a good state
The issue has been identified and the fix has been implemented. We are gathering all brand and images affected and will get the imaged uploaded correctly.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Issues Depositing to Bank or Credit Card in Disbursement Link"
Last updateWe have identified and corrected a networking issue preventing communication with our supplier. All disbursement link orders should now be processed correctly.
We are currently unable to fulfill "Deposit to Bank Account" or "Deposit to Card" Reward Link options. We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Duplicate emails being sent to certain recipients."
Last updateWe have verified orders are no longer causing the resend issue and have corrected all of the underlying data for the impacted line items. Issue is resolved.
We've identified the cause of resends and fixed the issue. Currently monitoring for any regressions / recurrences.
We are aware that some recipients are receiving multiple duplicate emails containing their reward links. Our team is actively investigating the issue.
Report: "Incident preventing redemption of some rewards."
Last updateThis incident has been resolved.
We have fixed the affected items, and will continue to monitor and carry out corrective actions to prevent later recurrence.
We are working on corrective actions to make the affected Reward Link products redeemable.
We are actively investigating an incident which prevents recipients from redeeming rewards for a small number of products.
Report: "RaaS currently down in Production"
Last updateThis incident has been resolved.
From 7:51am PT to 8:38am PT, RaaS experienced a period of high latency, resulting in high timeout rates, although a low error rate of 2.6%. Even if your request timed out, the order may have succeeded asynchronously. Please retry with the idempotent external reference id, which will ensure you receive the order already completed rather than placing a new order
We are currently investigating this issue.
Report: "Reward Link elevated errors"
Last updateWe started seeing elevated errors for Reward Link redemptions around 4:30PM PST. We resolved the issue at 5:08PM PST and redemptions should be back to normal. Please retry as necessary.
Report: "[SANDBOX] Sandbox Rewards Genius, Sandbox RaaS and Sandbox Reward Link are currently down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating an issue resulting in Rewards Genius, RaaS and Reward Links to be down in the SANDBOX environment ONLY.
Report: "Email Delivery Failures"
Last updateAll emails have been delivered and we are correctly displaying the current statuses.
Services have been restored. We are checking on any potentially impacted deliveries during the incident window.
A fix has been implemented and we are monitoring the results.
We have identified the issue and are working on the restoration of services.
Report: "Reward Link URLs experiencing issues"
Last updateThis incident has been resolved.
We believe we have identified the issue and are working on rolling back to previous state to fix the issue.
We are currently investigating an issue with a small number of Reward Link URLs leading to an error message.
Report: "Intermittent connectivity issues with AWS"
Last updateRewards Genius, RaaS, and RewardLink are experiencing intermittent connectivity issues while AWS resolves connectivity issues.
Report: "Notice of Maintenance"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
Dear Partner, Tango Card will be performing maintenance on our order search functionality in response to a recently released AWS urgent hotfix. This may intermittently impact order history searches in Rewards Genius and RaaS. Time Window Start: Tuesday 12/14/2021 5:00 PM PDT End: Wednesday 12/15/2021 12:00 AM PDT Maintenance Description: We will be applying an urgent patch released by AWS Downtime Behavior: During the maintenance period you may experience the following system behaviors: - Order history searches returning no data If you experience this error during the Time Window, please try again after the maintenance window has ended. If you have any questions, please do not hesitate to reach out to your Customer Success Manager or our Technical Support Team. Best Regards, The Account Team @ Tango Card
Report: "Rewards Genius is Under Maintenance"
Last updateMaintenance has been completed
We have put Rewards Genius into a temporary maintenance mode. We will provide an updates as available.
Report: "Temporary Outage"
Last updateThe issue has been resolved.
The change has been reverted and we are monitoring to ensure stability.
We are experiencing an issue with and upgrade affecting Rewards Genius and Reward Link. We have identified the issue and are reverting changes.
Report: "Seeing intermittent issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Rewards Genius and RaaS API have recovered. We are continuing to investigate efforts to bring Reward Link back online as well
We are continuing to investigate this issue.
Rewards Genius and the RaaS API are back online. Working to bring Reward Link back online.
We are currently investigating this issue.
Report: "Some eGift Cards failing to load"
Last updateEGift Cards that are hosted on rewardcodes.com were failing to load from 1:08-1:25pm PT. The issue has been resolved and these eGift Cards are now loading.
Report: "Rewards Genius Unavailable Due to Outage"
Last updateWe have resolved this issue and Rewards Genius is back online.
We have identified an issue with Rewards Genius and have taken the application down while we investigate. We will update as soon as we know more.
Report: "Rewards Genius Order History may be delayed in displaying orders"
Last updateWe have resolved the issue, and all orders are now displaying in order history.
We have identified the issue, and are working on a fix.
We are continuing to investigate this issue.
We are currently investigating an issue where some orders are not displaying in the Rewards Genius order history tab. We advise that orders placed may have successfully completed, but are not displaying in the order history. We will update once we know more.