Historical record of incidents for Marqeta
Report: "Widgets related services are impaired"
Last updateMarqeta engineers are currently responding to an incident that is impacting customers who use our Widgets service. Customers may experience challenges updating pins or activating cards. An engineering team is engaged and actively investigating.
Report: "DV Timeout Errors"
Last updateEngineers quickly identified a potential node capacity issue causing high CPU usage. This may have caused service degradation for ~7 minutes for some customers before engineers resolved the issue. This may have affected authorizations at the time. This incident is resolved.
Report: "Brief Degraded Performance from Visa causing STIPS"
Last updateAt about 15:15 PST we detected a burst of STIPs from Visa. The impact of this incident appears to be over. We are engaging Visa for further explanation of the incident.
Report: "3DS challenges are impaired"
Last updateA fix has been implemented and we have validated successful transactions. We will continue to monitor.
Marqeta has identified an issue that is impacting the challenge flow for our 3DS service. The root cause has been identified and a fix is being implemented.
Report: "Real Time Decisioning service is imparied"
Last updateMarqeta Engineers were alerted to failed requests to our RTD service. A team is engaged and actively investigating the root cause of these alerts.
Report: "DV Timeout Errors"
Last updateEngineers quickly identified a potential node capacity issue causing high CPU usage. This may have caused service degradation for ~7 minutes for some customers before engineers resolved the issue. This may have affected authorizations at the time. This incident is resolved.
Report: "DV Timeout Errors"
Last updateEngineers quickly identified a potential node capacity issue causing high CPU usage. This may have caused service degradation for ~30 minutes for some customers before engineers resolved the issue. This may have affected authorizations at the time. This incident is resolved.
Report: "DV Timeout Errors"
Last updateEngineers quickly identified a potential node capacity issue causing high CPU usage. This may have caused service degradation for ~30 minutes for some customers before engineers resolved the issue. This may have affected authorizations at the time. This incident is resolved.
Report: "Drop in Traffic for Apple Pay Token Provisioning"
Last updateApple's status page reports that their issue with Apple Pay, Wallet, Card, and Cash is resolved.
Apple Pay appears be experiencing an outage that is affecting token provisioning, among other Apple Pay functionality. We are monitoring Apple's support pages for confirmation and further details.
We have identified a drop in traffic for Apple Pay token provisioning across Visa and Mastercard. Engineers are working to identify the source of this drop.
Report: "Delayed clearing file from Visa in Europe, delayed clearing webhooks."
Last updateThis file is completely processed, and all clearing webhooks have been sent from our platform.
We have received the file from Visa, and are processing it now, and Visa customers in Europe will be receiving those webhooks imminently.
We have a delayed clearing file from Visa this morning, resulting in some of this morning's domestic clearings not being processed yet. It should be received shortly and processed as soon as we receive it. We will update this incident as soon as we confirm receipt of the file.
Report: "Private Sandbox APIs Returning 500 Status"
Last updateEngineers have reverted and report that the 500 errors in private sandbox API should be resolved shortly.
Private sandbox API may return a 500 status for some users. Engineers are investigating and will update soon.
Report: "Pulse Outage Affecting Transaction Processing"
Last updatePulse has confirmed that the cause of the increase in declines should be resolved.
We are tracking an improvement in declines. We are awaiting full resolution and confirmation from Pulse.
Cardholders may see an increase in declines on the Pulse network. Pulse is actively working on a fix.
Report: "Sandbox API Failures for Cards"
Last updateSandbox should now be fully operational.
Maintenance is underway to restore sandbox functionality. This is expected to take approximately one hour.
Card creation and setPIN and updatePIN operations in Sandbox may experience API failures, and an increase in non-200 http response codes.
Report: "Tokenization Requests on the Mastercard Network are failing."
Last updateA fix has been implemented that appears to have restored the ability to tokenize cards on Mastercard in the US and EU.
A fix has been implemented and we are monitoring the progress of the issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
The difficulties appear to stem from a sudden change from Mastercard in their outbound certificates. We continue to investigate this as an avenue for resolution. Token-based authorizations are unaffected. Visa customers are not affected. We are continuing to collaborate with Mastercard to understand root cause and timelines for recovery.
We are continuing to experience difficulties with Mastercard. Token-based authorizations are unaffected. Visa customers are not affected. We are continuing to collaborate with Mastercard to understand root cause and timelines for recovery.
As of 10:18am UTC, 03:18 PT Marqeta suffered a drop to 0 for Mastercard tokenization requests. This means that cardholders with Marqeta customers who have Mastercard as their network will not be able to tokenize their cards to digital wallets. Token-based authorizations are unaffected. Visa customers are not affected. We are speaking with Mastercard to understand root cause and timelines for recovery.
Report: "Mastercard Programs Experiencing Degraded Performance"
Last updateWe have discovered the root cause and have implemented a fix. We are monitoring successful token requests.
Some Mastercard programs may be unable to terminate DWTs. We are reaching out to Mastercard for more information.
Report: "Intermittent Tokenization Failures on the Visa Network Impacting Marqeta Customers"
Last updateMarqeta customers on the Visa network may have experienced failures when attempting to add cards to their digital wallets. We have confirmed that the root cause has been resolved and Marqeta customers using the Visa network should no longer be experiencing these failures.
Report: "IVR System Impaired"
Last updateTwilio reports that performance has returned to normal, there should be no additional effects of the outage.
Twilio reports that performance has returned to normal. We will continue monitoring for now to ensure there are no lingering effects.
Marqeta's system provider Twilio is experiencing a service interruption and it is impacting our IVR service. We are engaged with Twilio and monitoring their progress.
Report: "Reporting API Experiencing Degraded Performance"
Last updateReporting should now be caught up after Snowflake's outage.
Snowflake reports that the issue has been resolved. Recovery from the delayed reporting is ongoing.
Snowflake has identified the issue and they are working on a solution. Follow their progress on their status page: https://status.snowflake.com/incidents/9mwvw50z4wnm
Some users of Reports and/or the Reporting API may experience delayed performance or inability to pull/update reports. We are working with Snowflake to determine the cause of this difficulty.
Report: "IVR System Impaired"
Last updateThis incident has been resolved.
Marqeta's system provider Twilio is experiencing a service interruption and it is impacting our IVR service. We are engaged with Twilio and monitoring their progress. Marqeta customers can still utilize the IVR service but they will be forwarded to a rep for every call until this is resolved.
Report: "Some Bank Files Are Delayed"
Last updateDelayed bank files and incomplete files have been sent.
The delay in Bank File transmission has been resolved. The files affected by the delay should be caught back up in the next scheduled file transmission.
The delay in Bank File transmission has been resolved. The files affected by the delay should be caught back up in the next file transmission.
We have observed a delay in Bank File transmission. The cause of the delay has been identified and normal performance will be restored soon.
Report: "Marqeta Real Time Decisioning service is impaired"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Marqeta Engineers have been alerted to a challenge with our Real Time Decisioning service. A response team is engaged and working to isolate the source of the challenge.
Report: "Intermittent STIPs on all card networks"
Last updateThe fix has been implemented and declines have been resolved.
The issue has been identified and a fix is being implemented.
We have identified an issue that is causing intermittent declines in error. We have a response team engaged and are isolating the source of the challenge.
Report: "Mastercard network experiencing challenges"
Last updateThis incident has been resolved.
As of 1:58 AM Pacific Time Mastercard has reported the issue is resolved and STIPs have subsided.
A planned maintenance activity in the Mastercard network is unexpectedly causing intermittent STIPs for some Marqeta customers. A response team is engaged and actively working with Mastercard.
Report: "AWS Maintenance on us-east-2"
Last updateMaintenance is complete on us-east-2.
Maintenance is continuing on AWS us-east-2 throughout this week. There should be no impact to performance.
Maintenance is continuing on AWS us-east-2 throughout early this week. There should be no impact to performance.
We are conducting maintenance on AWS us-east-2 through the weekend. There should be no impact to performance.
Report: "Private Sandbox Experiencing Timeouts"
Last updateTimeouts on the private sandbox have been resolved.
Users may be experiencing timeouts while using the private sandbox service. This should not affect the public sandbox. Investigation is underway to resolve this issue.
Report: "3DS VISA authentication issues"
Last updateWe have completed applying a fix at 09:56 Pacific Time, and no further issues were observed since then.
We have identified a possible cause to the issue and are working on implementing a fix.
Our engineers continue to investigate the issue. As of now, we have identified that the impact is restricted to 3DS VISA transactions where 3DS is challenged. Frictionless transactions are not impacted by this issue.
We are aware of an issue impacting 3DS authentication. We are currently investigating the issue.
Report: "Some OTP is failing to send in US"
Last updateAll cardholders should now be receiving OTPs correctly.
Some cardholders may not receive OTPs for step-up verification during digital wallet token provisioning. The cause is being identified and resolved.
Report: "Bankfiles are not sent to the customers"
Last updateAll bank files have been distributed for all customers.
The fix for this issue continues to be distributed. Remaining affected customers should see a resolution within the next few hours.
The issue has been identified and most bank files should be sent to customers within the next 6 hours.
On 2025-01-16, Marqeta identified an issue that may impact the services we provide. We are working to resolve the issue as soon as possible. Identified Service: Bank Files
Report: "EU based customers may experience failures reaching APIs"
Last updateThis incident has been resolved. Customers hosted on marqeta.eu should no longer experience errors reaching APIs.
Response team continues to investigate the quickest solution to API failures on marqeta.eu. Customers hosted on marqeta.com continue to be unaffected.
Marqeta customers hosted on marqeta.eu may be experiencing failures reaching APIs. This may display as a 500 response status. A response team is engaged and has isolated the source of the challenge. Customers hosted on marqeta.com are not affected.
Report: "STIPs on the Mastercard network"
Last updateThe incident in the Mastercard network has been resolved.
We have identified the source of the challenge and are working with Mastercard to ensure there are no further STIPs.
Marqeta engineers have been alerted to a challenge on the Mastercard network which is resulting in STIPs at 10 minute intervals. A response team is engaged and investigating.
Report: "Marqeta Dashboard Data and Insights Reports are failing"
Last updateWe have verified that the fix was successful. Reports are functioning as expected at this time.
We have identified the cause and are enacting a fix.
We have been alerted to failing reports in the Marqeta Dashboard. A response team is working to resolve the issue.
Report: "Elevated error rates in Disputes API"
Last updateWe have resolved the issue impacting Disputes API and the service is now operating as expected.
A fix has been implemented for Disputes API and we are seeing error rates return to normal, we are continuing to monitor performance.
We are seeing elevated error rates on Disputes API operations. Dispute creation is impacted. We have identified the issue and are taking measures to resolve.
Report: "Intermittent STIPs on the Mastercard network"
Last updateMastercard has confirmed that the issue has been resolved. Marqeta has not observed any further STIPs since 09:04 Pacific Time.
We have engaged Mastercard and are investigating the issue with them.
Report: "Pin Debit transactions are failing in error on the Mastercard network"
Last updateThis incident has been resolved.
The root cause of the incident has been identified and a fix has been deployed. We are seeing transactions processing and completing properly at this time.
Marqeta engineers are engaged with Mastercard to investigate the root cause.
Report: "No/Low token status changed requests received from Visa"
Last updateThe situation is stable, and no further disruptions were noticed. We are waiting for feedback from Visa to identify the root cause.
Inbound token change requests have rebounded to normal levels for now. We are waiting for Visa to understand the root cause and will continue to monitor the situation.
We can see a drop in token status change requests from Visa across all programs between 3:15 PT and 3:40 PT. Our Team is investigating the issue with the highest priority with Visa Support being involved to get more information. Cardholders will be unable to activate their cards.
Report: "Repeat clearings from June posted only to some MasterCard programs"
Last updateThis incident is stable at this time. Remediation efforts will continue. For questions, please contact your Marqeta Representative.
We are remediating this issue. The remediation will be completed soon. Please reach out to your Marqeta Account Manager for the incident report.
We identified that a single clearing file was sent to us this morning by MasterCard containing the clearing transactions from June. All transactions have been posted to customer environments and there are no further transactions to be posted. We are investigating the root cause of this file transmission and also assessing the remediation options.
Marqeta is aware of multiple posted repeat clearings from June posted to some of the MasterCard programs. We are investigating and do not yet understand the source
Report: "Intermittent STIPs on all card networks"
Last updateThis incident has been resolved.
Our customers may have experienced STIPs from 23:23 Pacific on November 8th through 00:07 November 9th due to intermittent database connection errors. The root cause has been addressed and we are seeing all programs operating as expected at this time.
Report: "Intermittent STIPs on the Visa network"
Last updateThis incident has been resolved.
Marqeta engineers have been alerted to intermittent STIPs impacting our customers on the Visa network. A response team is engaged and is working with Visa to identify the source.
Report: "Intermittent STIPs on the Pulse Network"
Last updateThe impacted Acquirer has completed a rollback as of 07:15 Pacific Time. As of that time the STIPs from the Pulse network have subsided.
Working with Pulse, Marqeta engineers have established that a country code discrepancy with a single acquirer is the root cause. Pulse is working with that acquirer to resolve. We will continue to monitor for resolution.
Marqeta Engineers have been alerted to persistent STIPs on the Pulse network. A response team is assembled and engaged in investigating the condition.
Report: "Partial disruption of disputes initiation on Marqeta Dashboard"
Last updateDisputes initiation is now operational. If you continue to see any issues, navigate to https://app.marqeta.com/application-reset and log in again.
We are aware of an issue impacting disputes initiation on Marqeta Dashboard. Our teams are engaged and working to resolve the issue.
Report: "Simulation API 401 errors on some Sandboxes"
Last updateWe have deployed a fix and confirmed no further errors. This incident is now stable.
We have identified the cause of the issue and are working towards a resolution. We don't have yet an ETA for the fix to be implemented.
We are aware of an issue with simulation API in some Sandboxes causing 401 errors. The owning team are investigating. Once the root cause is identified, there will be a rapid QA and deploy phase. We will update the incident once the root cause is identified. Since the root cause has not yet been identified, there is currently no ETA for this fix.
Report: "MQD partial outage"
Last updateA recent change has been reverted and Marqeta Dashboard is functioning as expected. For the duration of the impairment the User Management function was unavailable. If you are experiencing issues with this function please reset at the following link: https://app.marqeta.com/application-reset
We have identified the root cause and are working to restore Marqeta Dashboard at this time.
We have been alerted to some portions of the Marqeta Dashboard being unavailable. We have a response team engaged and working to isolate the cause.
Report: "Tokenization Failures on the Visa Network"
Last updateThe issue within the Visa network has been resolved. Due to the nature of the failure, if Digital Wallet Tokens continue to fail they will need to be reprovisioned.
We are continuing to work on a fix for this issue.
The issue has been isolated to the Visa Network and we are working with them to get an ETA on resolution.
Our Engineers have been alerted to challenges in the Visa network. We have a response team gathered and we are working with Visa to isolate the source.
Report: "marqeta.eu is down"
Last updatemarqeta.eu is accessible again, APIs are no longer impacted.
We have identified an issue with our Edge routers for the marqeta.eu domain, and the owning team are online and attempting to remediate. We will update soon with any progress.
marqeta.eu is inaccessible due to a routing issue. This is impacting all APIs, though most transactions are flowing properly. marqeta.com hosted customers are not impacted.
Report: "STIPs on the Visa and Pulse Networks"
Last updateAs of 6:12 Pacific Time the impacted POP has been returned to service and transactions are completing as expected.
As of 9:32 Pacific we have restored payment processing by draining the impacted Point of Presence. The team will continue to monitor this.
As of 9:01 Pacific Time Marqeta engineers were alerted to transaction errors on the Visa and Pulse networks. A response team has been engaged and is assessing the issue.
Report: "STIPs on the Visa and Pulse Networks"
Last updateAll traffic has been restored to the impacted POP and payment processing is operating normally.
Update as of 6:24 PM Pacific: We have identified a network event that coincides with the start of the STIPs and it has recovered. We have left the impacted POP drained of payment processing traffic. We expect no further STIPs at this time and are continuing to monitor before restoring the impacted POP.
As of approximately 5:30 Pacific Time we have identified that all STIPs were generated at one of our three Points of Presence. We have moved all traffic to other regions and are monitoring the situation as we investigate the impacted Point of Presence.
At 5:15 PM Pacific Time Marqeta Engineers were alerted to STIPs on the Visa and Pulse networks. A response team has been engaged and is currently investigating the cause.
Report: "VISA network experiencing challenges"
Last updateWe have confirmed that Visa's network is operating normally and all traffic has been restored.
We are continuing to monitor the issue and are working with VISA on a timeline to shift traffic back to the affected region.
We are continuing to monitor the issue and are working with VISA towards a resolution.
Marqeta has shifted traffic away from the impacted region. We are continuing to work with VISA on the resolution.
Marqeta has detected an issue with our network connection to VISA. VISA has acknowledged the issue is on their end and we are working together to resolve the issue.
Report: "Mastercard network experiencing challenges"
Last updateThe Mastercard network has returned to normal operation.
The Mastercard network is experiencing an outage and it is impacting Marqeta Mastercard programs. We have a response team assembled and are in contact with Mastercard.
We are continuing to investigate this issue.
We have detected a challenge with tokeniziation requests coming from the Mastercard Network. We have a response team engaged and we are working with Mastercard to resolve.
Report: "AWS Service Disruption is impacting Marqeta customers"
Last updateAWS resolved their service disruption and Marqeta has verified that all applications are operational.
AWS is experiencing a service disruption and it is causing intermittent service impacts to Marqeta customers. We have a response team assembled and we are monitoring the health of all applications.
Report: "Transaction failures on the Visa and Pulse Networks"
Last updatePayment processing has remained stable since 12:05 Pacific Time. We have correlated the burst in STIPs to a brief outage with an ISP. We are standing down the response team and will continue to monitor.
We have identified a burst of STIPs on these two networks at approximately 12:00 Pacific Time clearing at 12:05 Pacific Time. A response team is engaged and investigating the incident.
Report: "Multiple services impacted"
Last updateAll Marqeta services are now stable. Queues are draining and there will be some delay until GET transactions end point is reconciled. ETA 3 hours.
The majority of Marqeta services are now stable. Velocity control check capabilities and the GET transactions been restored and we are processing a backlog. Next update will come in 60 minutes.
The majority of Marqeta services are now stable. Velocity control check capabilities and the GET transactions end points are in the process of being restored. Next update will come in 60 minutes.
The majority of Marqeta services are now stable. We have enabled ACH processing and all transactions are caught up. Velocity control check capabilities and the GET transactions end points may be delayed to support service recovery. Next update will come in 60 minutes.
The majority of Marqeta services are now stable. We are progressively enabling ACH processing and expect all transactions to be processed successfully within the next few hours. Velocity control check capabilities are intentionally delayed to support service recovery. Next update will come in 60 minutes.
We continue to see recovery across multiple services. The ACH processing restart is in progress and velocity control check capabilities are intentionally delayed to support service recovery. Next update will come in 15 minutes.
We are seeing recovery across multiple services. ACH processing is paused and velocity control check capabilities are intentionally delayed to support service recovery. We are planning controlled restart of ACH processing starting at 12:00 PT. The status page has been updated to reflect current service performance. Next update will come in 15 minutes.
We continue to see improvement from remediation efforts. We have updated the status page to reflect current service performance. Next update will come in 15 minutes.
We continue to see improvement from remediation efforts. We have updated the status page to reflect current service performance. Next update will come in 15 minutes.
We are beginning to see improvement from remediation efforts. The team is closely monitoring impact and we will update service status in the next update if we continue to see improved service performance. Next update will come in 15 minutes.
The team continues to be engaged on implementing a remediation strategy. We continue to closely monitor impact. Next update will come in 15 minutes.
We continue to see errors after initial remediation rollout. The team is engaged on further resolution paths. We continue to closely monitor impact. Next update will come in 15 minutes.
We continue to see errors after initial remediation rollout. The team is engaged on further resolution paths. We continue to closely monitor impact. Next update will come in 15 minutes.
We continue to see errors after initial remediation rollout. The team is engaged on further resolution paths. We continue to closely monitor impact. Next update will come in 15 minutes.
We continue to rollout changes to support root cause resolution. We continue to closely monitor impact. Next update will come in 15 minutes.
We continue to rollout changes to support root cause resolution. We continue to closely monitor impact. Next update will come in 15 minutes.
The root cause fix is in the initial stages of rolling out. We continue to closely monitor impact. Next update will come in 15 minutes.
Root cause has been identified and we are working on a fix. Next update will come in 15 minutes.
The root cause has been identified and we are working on implementing a fix now. We have updated the status page to reflect wider service impact. Next update will come in 15 minutes.
We are working on determining the root cause, the status page has been updated with Marqeta services that are impacted. We will send an update in 15 minutes.
We are continuing to investigate this issue.
Marqeta detected an issue with authentication services impacting multiple APIs.
Report: "Customer APIs Unavailable"
Last updateThe software fix has been completed and we ahve seen services restored to normal operation
We have been alerted to unavailability of several customer APIs and have identified the root cause. A software fix is underway
Report: "Several API endpoints returning HTTP 500/HTTP 403"
Last updateMarqeta resolved this incident at 16:59 UTC. The rollback was successful, and the team has verified that there are no further errors. If you have any questions or concerns, please reach out to your Customer Success representative. Thank you for your patience and understanding, and we apologize for any inconvenience.
Marqeta has identified an incident that may have an impact on several API endpoints returning HTTP 500/HTTP 403. We are actively working to resolve the situation. It was determined that the problem originated from a deployment that took place on Friday, 2024-06-21. The team chose to revert the deployment as a corrective action. We apologize for any inconvenience this incident may cause, and will periodically provide updates here until the incident is resolved.
Marqeta detected an incident beginning at 2024-06-24 10:40:00 UTC that may cause several API endpoints returning HTTP 500/HTTP 403.
Report: "STIPs on the Visa and Pulse Networks"
Last updateThere have been no further STIPs. We have identified a brief Network outage that coincides with the time of the event and are working with our underlying provider to understand the root cause. This incident is stable and we expect no further impact.
At 12:14 Pacific Time and ending at 12:18 Marqeta Engineers were alerted to a service impacting incident. There is no ongoing impact at this time. We are investigating the root cause.
Report: "Marqeta Dashboard is unreachable"
Last updateWe have identified the root cause and we have implemented a fix. MQD is now functioning as expected. Some users may need to log out of the service and log back in to resume normal function.
MQD- Marqeta Dashboard is currently unreachable. We have a response team engaged and we are investigatig.
Report: "Webhook retries are failing"
Last updateWe have implemented a fix and verified that webhook retires ar enow functioning normally.
A fix has been implemented and we are monitoring the results.
We have identified the root of the challenge and are currently working on an update to resolve the issue.
Webhook retries are failing to initiate. A response team has been assembled and is evaluating the root cause.