Historical record of incidents for BetterNow
Report: "Increased error rate"
Last updateWe are currently investigating this issue.
Report: "Errors in fundraisers API endpoint"
Last updateAn error in a deploy caused an increased error rate for API requests to the fundraisers endpoint. The deploy has been rolled back as of 15:22 CEST
Report: "Increase error rate"
Last updateThis incident has been resolved.
Services are recovered, we continue to monitor traffic levels and error levels
Autoscaling has caught up to the increased traffic and it looks like services are recovering
We are currently investigating this issue.
Report: "Increased error rate"
Last updateThis incident has been resolved.
Our autoscaling has caught up to a spike in requests and the error rate has normalized. We continue to monitor the health of the system.
We are currently investigating this issue.
Report: "Errors with Swish payments"
Last updateSwish has resolved their incident. We continue to monitor the error rate for Swish payments. https://status.swish.nu/incidents/1fwn9ny2lsh3
We see an elevated error rate for Swish payments for many merchants. More information can be found on the Swish status page at https://status.swish.nu/incidents/1fwn9ny2lsh3
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
Report: "Downtime from 19:53-20:42 CET, October 28th"
Last updateOur monitoring shows a period of downtime between 19:53 and 20:42 CET yesterday, October 28th. This was due to an issue with our upstream provider, the details are here: https://status.heroku.com/incidents/2721
Report: "Increased response times"
Last updateThis incident has been resolved.
We are seeing response times improve in all regions.
This looks to be a repeat of yesterday's incident: https://status.heroku.com/incidents/2685
For now it seems our monitoring from North America and South America are seeing the worst response times, while Europe and Asia-Pacific are only slightly elevated.
We're seeing longer response times than normal from some regions.
Report: "Increased response times"
Last updateThis incident has been resolved.
Our upstream provider Heroku is reporting problems. Details are at https://status.heroku.com/incidents/2684
We continue to see excessive response times and we're still trying to determine the cause.
We're seeing much longer response times than normal from some regions.
Report: "Increased error rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Increased error rate"
Last updateThis incident has been resolved.
The error rate has returned to normal levels. We continue to monitor the system.
We are currently investigating this issue.
Report: "Clearhaus payment issues"
Last updateThis incident has been resolved.
An update from Clearhaus: "We have seen a significant increase in approval rates and the connectivity is now back to normal. Due to the severity of this issue and the previous sudden change in approval rate we are still actively monitoring this situation." We will continue to monitor this as well.
From Clearhaus: "Our upstream provider has confirmed and is aware of a connectivity issue. Currently almost all authorizations, voids and Mastercard credits are impacted"
Clearhaus is reporting issues with an upstream provider https://status.clearhaus.com/incidents/hp85pwr0m3yp
Report: "Increased error rate for Swish payments"
Last updateThis incident has been resolved.
We are seeing a reduction in error rates and will continue to monitor the system and Swish's incident until it is resolved.
Swish has an incident open for this at https://status.swish.nu/incidents/6xqpkqcsy8bb
We are currently investigating this issue.
Report: "Increased Error rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Increased Error count"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Increased Error Rate"
Last updateFor a short period after 15:24 we saw an increase rate of errors on some pages. We have rolled back the deploy causing the issue and the error rate has returned to normal.
Report: "Dashboard/administration errors"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Errors for logged-in charity users"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Logged-in charity users are getting errors as they try to use the system. Non-charity users (fundraisers and donors) are not affected. A fix is being deployed.
The issue has been identified and a fix is being implemented.
Report: "Timeout errors"
Last updateThis incident has been resolved.
We saw a brief period of network timeout errors with the BetterNow platform and api. For now it looks like a fix has been implemented by our upstream providers and the services have recovered. We will continue to monitor the services.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Errors loading images"
Last updateThis incident has been resolved.
Our upstream providers have applied a fix. We continue to monitor for errors.
Our upstream providers have confirmed that there are intermittent problems loading the images from some European countries. We are working with them on a resolution to this issue.
We are currently investigating this issue. Our monitoring shows intermittent errors loading images. We are in contact with our upstream providers regarding this issue.
Report: "Donors received an error instead of being sent to pay their donations"
Last updateBetween 16:04:05 and 18:35:39 on March 15th donors received an error instead of being sent to our payment partners to pay their donations. This was due to an error in code we deployed and was compounded by a missing alert when the specific error was triggered. At 18:35 we rolled back the bad deploy and added the missing alert to our monitoring system.
Report: "Connection errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Problems with MobilePay online"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our payment provider is experiencing problems with MobilePay online transactions. Their status is here: https://status.quickpay.net/incidents/414 This does not affect card transactions or Swish or Vipps
Report: "Some transactions processed twice"
Last updateWe have notified the affected customers by email. Any additional questions should be directed to orgsupport@betternow.org
We now have a list of the 49 affected transactions. Clearhaus writes "Corrections will be sent starting Monday, and while you may see the effect already Tuesday, it could take several days also depending on your bank. You should see the correction no later than October 21st."
One of our acquiring partners (Clearhaus) mistakenly processed transactions twice. All transactions (captures, refunds and credits) processed between 2021-10-06 19:00:00 UTC and 2021-10-07 18:59:59 UTC were affected for both Visa and Mastercard, with the exception of Visa credits. Clearhaus is working with Visa and MasterCard to automatically reverse the transactions during the night between Monday and Tuesday. Therefore, you do not need to manually refund any double donations. Clearhaus' incident report is here: https://status.clearhaus.com/incidents/t8z9l54g6tbw
Report: "Errors connecting to payment provider (QuickPay)"
Last updateThis incident has been resolved.
Quickpay's incident is here: https://status.quickpay.net/incidents/409 They are currently reporting that the issue is fixed, we will continue to monitor for errors
We are currently investigating this issue.
Report: "Intermittent Payment Errors"
Last updateThis incident has been resolved.
QuickPay reports that their systems are operational again. We'll continue to monitor for problems.
Some QuickPay payment requests are refused with errors. https://status.quickpay.net/incidents/407 is their status issue for the problem
Report: "Connection errors"
Last updateThis incident has been resolved.
We are continuing to monitor for further issues. You can see more information from our upstream provider at https://status.heroku.com/incidents/2359
A fix has been implemented and we are monitoring the results.
Unfortunately we are continuing to see routing errors.
A fix has been implemented and we are monitoring the results.
Our upstream provider is not currently routing requests to our apps. We are working with them on a fix.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Problems with second-factor authentication"
Last updateThis incident has been resolved.
We are currently investigating this issue. It seems as though some administrator accounts cannot provide 2 factor authentication. This affects usage of the dashboard, fundraisers, donations and payments are not affected.
Report: "SWISH payment failures"
Last updateThis incident has been resolved.
The error rate has returned to normal levels and we're monitoring the situation
We see intermittent connection errors between our payment service provider and Swish for some Swish payments in Sweden. Card payments and payments in other countries are not affected.
We are currently investigating this issue.
Report: "Donation creation errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some donations via OnlineFundraising are refused with a system error. This is only for charities using OnlineFundraising for payments, not for those using QuickPay/Clearhaus.
Report: "connection errors"
Last updateThis incident has been resolved.
The connection errors seem to have subsided and we are monitoring the systems.
We are currently investigating this issue.
Report: "Slow image loading"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Increased donation error rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Datastore Failure & Replacement"
Last updateOur secondary datastore failed and its standby replica was promoted to production. There were circa 6 minutes of downtime starting at 02:58 CEST.
Report: "Increased error rate"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Invalid SSL certificate on api.betternow.org"
Last updateDue to a configuration error our api.betternow.org endpoint had an invalid SSL certificate between 2020-03-17 16:00 CET and 2020-03-18 09:35 CET.
Report: "Partial outage"
Last updateOur provider has recovered and all outgoing emails have been delivered.
We have implemented a change so that other background job processing can complete, and the queues are now clear of other jobs. Sending emails, such as for donation receipts and for notifying fundraisers of new donations, will still be delayed until our service provider recovers.
One of our service providers has an outage that is affecting transactional email sending along with other work processed by our background job queues.
We are currently investigating this issue.
Report: "Retroactive: Platform & API availability"
Last updateA bad deploy resulted in a 4 minute outage for the API and BetterNow Platform. The deploy has been rolled back and the sites are operational again.
Report: "Continuing Connection Errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Connection Errors"
Last updateThis incident has been resolved.
A fix has been put in place and we're monitoring the results.
We are currently investigating this issue.
Report: "Elevated Errors"
Last updateA bad deploy caused a short period of elevated errors. We've rolled back and deployed a fix.
Report: "Connection errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Connection errors"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our monitoring systems have alerted us to connection errors. We're investigating.
Report: "Increased error rate"
Last updateThis incident has been resolved.
We made some configuration changes and the error rate has now returned to normal levels. We'll continue closely monitoring the affected services.
Our monitoring systems have alerted due to an increased number of errors. We are investigating.
Report: "Slow Requests"
Last updateThis incident has been resolved.
We have identified the issue and put a fix in place. We'll continue monitoring the affected services.
Our monitoring systems have alerted us to an increase in the number of slow requests. We are investigating the issue.
Report: "Intermittent Server Errors"
Last updateThis incident has been resolved.
Error rates are returning to normal. We'll continue to monitor the heath of the system.
We're seeing an increased percentage of failed requests and are investigating
Report: "Increased failures due to upstream routing errors"
Last updateWe're resolving this issue as error rates have been normal all day and the upstream incident has been resolved.
Our last failure was at Apr 05 15:05:15 CEST. We will keep monitoring the failure rate as long the upstream Heroku incident is not resolved.
We are seeing the error rate return to normal, and continue monitoring the progress on the Heroku issue.
Unfortunately the error rate is increasing again. More information about this issue can be found on Heroku's incident page: https://status.heroku.com/incidents/1091
We're seeing a much lower error rate, even though this incident is ongoing on Heroku's side. We will continue to monitor the error rate closely.
We are unfortunately affected by the Heroku partial outage. Our monitoring shows up to 30% of inbound requests are failing.
Report: "Cloudbleed Response (Retroactive)"
Last updateOut of an abundance of caution, we have revoked all current user sessions in response to the "CloudBleed" security issue (https://en.wikipedia.org/wiki/Cloudbleed). This means you will need to log in again the next time you edit your Fundraiser or Charity. BetterNow has never used CloudFlare for any services, but as the impact of the issue is not yet fully known, and at least one of our vendors uses CloudFlare, we decided to revoke all current sessions and rotate the credentials we use to communicate with the affected vendors. It is important to note that our investigation shows no evidence that any breach of our service took place, but again we are acting out of an abundance of caution. "Better safe than sorry."
Report: "Errors connecting to payment processor"
Last updateThis incident has been resolved.
Some payment requests are failing. A fix has been deployed and will be live within the next few minutes.
Report: "Database failure"
Last updateOur monitoring systems are showing that all components are stable and online. The failover to the standby database succeeded and a new replica has been brought online.
The platform and API are operational again, and we're keeping a close eye on the system.
We are switching to our standby database replica and expect to bring the affected components online in a few minutes.
We are investigating problems due to a database failure
Report: "Connectivity problems"
Last updateWe've seen normal levels of traffic for the last hour and are ready to enjoy the weekend sun. :)
Connectivity seems to be restored by our service provider. We will continue to monitor the system closely before marking this incident as resolved.
Our provider is continuing to work on resolving the issue. We will update this page with more information as soon as we have it, and latest 16:30 CEST
Our upstream provider has confirmed the issue and is investigating. We will update this page with more information as soon as we know more, and latest 16:00 CEST.
Our monitoring systems have notified us of connection errors and we're investigating.
Report: "Connectivity errors"
Last updateThe site is healthy again.
We've deployed a fix and are continuing to monitor the health of the site.
Our monitoring has detected an increased number of errors. We're investigating.