Historical record of incidents for Pike13
Report: "SmartWaiver outage"
Last updateExternal partner SmartWaiver is having issues. This is preventing Pike13 users from signing waivers. https://status.smartwaiver.com/
Report: "Application slowness and timeouts"
Last updateServer pool failed to scale up with increasing load. Fixed.
Report: "Brief outage caused by increased database load"
Last updateBetween 4:19pm and 4:22pm US Pacific Time, there was an intense spike in load on the database that cause Pike13 applications to stop responding.
Report: "Application Slowness"
Last updateMaintenance complete. Systems operating normally now. We will be monitoring for recurrence of the intermittent slowness.
The hardware replacement is in progress.
Slightly delayed while complete a full database backup first. Estimated time of outage now 11:00pm EDT.
We are going to replace the hardware for the database in 10 minutes from now. There will be a 5-minute outage.
Pike13 had been intermittently slow throughout the day. We are investigating.
Report: "Database connection waits"
Last updateDatabase restart. All system operational. Continuing to monitor performance.
We are currently investigating this issue.
Report: "500 Errors on Today page for some sites (only affecting Pike13 Support staff)"
Last updateAfter further analysis, it has been determined that this issue only affected Pike13 Support staff. No customers or clients were impacted.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Reporting API Server Hiccup (no impact)"
Last updateOne of the servers in the Reporting API cluster had connectivity issues and was removed from rotation. The other servers in the cluster were able to handle all of the traffic, which is the purpose of having the redundancy. Everything working normally.
There does not appear to be any operational impact, but we are checking to be certain.
Something isn't working right now. We're checking into it and will update you with more information ASAP.
Report: "Major Outage - Database Crash"
Last updateWe're back. Pike13 is up and running. Sorry for any trouble.
Systems are back up. Still investigating cause and stability.
We are continuing to investigate this issue.
Something isn't working right now. We're checking into it and will update you with more information ASAP.
Report: "Some pages failing with 500 Error"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Reporting Data Latency"
Last updateThe Reporting system is now fully caught up and operating normally.
The issue has been resolved. They system is catching up. Next update will be when latency is back to normal.
We are continuing to work on this issue.
Data processing into Reporting is very delayed. We are working on the problem.
Report: "Reporting API Degraded Performance"
Last updateThe unhealthy server has been replaced. All systems back to normal. There was no outage associated with this incident. Only a temporary reduction in capacity and redundancy.
The issue has been identified and a fix is being implemented.
Systems are still functioning normally. One of the servers in the cluster failed and must be replaced, but the other servers are carrying the load.
This alarm is regarding the Reporting API.
Something isn't working right now. We're checking into it and will update you with more information ASAP.
Report: "Reporting and API Data Latency"
Last updateReporting data latency has returned to normal.
Data latency is improving slowly, but still has a ways to go. We do not expect it to be completely caught up until sometime tomorrow.
Data updates to the reporting system and the reporting API halted. They have been restarted and are working normally again. Updates are catching up, but are still behind current by approximately 18 hours.
Report: "Minor Outage"
Last updateThis incident has been resolved.
Emergency restart of primary database.
Report: "Minor Outage"
Last updateThis incident has been resolved.
The primary database received a large spike in requests. It has recovered.
Something isn't working right now. We're checking into it and will update you with more information ASAP.
Report: "Minor Outage"
Last updateThis incident has been resolved.
Systems are recovering and returning to normal operating state.
Report: "Minor Outage"
Last updateThis incident has been resolved.
This incident began at 4:20am PST and was resolved at 6:34am PST.
Something isn't working right now. We're checking into it and will update you with more information ASAP.
Report: "Application outage"
Last updateMaintenance release deploy to production caused widespread 500 errors across the platform until the new code propagated to all new hosts.
Report: "Reporting Lag"
Last updateRegular Reporting and the reporting_api have returned to service.
A portion of the reporting infrastructure is under repair. We are using every resource available to expedite this, but expect the service to be down through the end of September. We are utilizing premium reporting and its connection to a different database to offer access to mission-critical data, like transaction reports. Contact help@pike13.com if you need access to this tool. The Core API is functioning, the reporting API does not have a complete data source.
Pike13 sites are operational.
We are continuing to work on a fix for this issue.
Pike13 sites has been paused briefly in order to upscale resources being impacted by the reporting sync.
We are continuing to work on a fix for this issue.
New changes are making their way into reporting as of 5pm Pacific time today (Sept 8). We are still working on populating data from 9/4 - 9/8 5pm. Please rest assured that all data and transactions are in production and will be migrated over to reporting as soon as possible. We apologize for this disruption.
Pike13 and reporting tools are restored. Reporting and the Reporting API are still missing transaction data from Sept 4, 2023 to now. We are still working to make that data available.
Report: "Background Jobs Delayed, Sidekiq down"
Last updateThis incident has been resolved. We will continue monitoring as the backlog of notifications is rolled out.
The issue has been identified and repairs are in progress. Live Payments are not impacted. Auto-billing from last night is scheduled to run later today.
Report: "Email Delivery"
Last updateEmail has been restored. Emails that should have been sent from midnight to noon PST Friday 8/18 may not be delivered as expected.
Pike13 is not currently delivering emails. We are investigating this issue and will update this page as possible.
Report: "Payment Processing"
Last updateThis has been resolved. We apologize for the brief interruption. If you are still having any trouble, please contact help@pike13.com.
Pike13 is experiencing a payment connection error during the roll-out of a new payment gateway. We are working to immediately resolve this and will repost when the resolution is complete.
Report: "Document Signing Disruption"
Last updateThis has been resolved. Please email help@pike13.com if you have any issues.
Our partner for providing document signing service is experiencing a disruption. We are working with them to resolve this issue and will provide an update here when the problem is resolved.
Report: "NMI Gateway is reporting connectivity issues"
Last updateNMI has reported that this issue has been resolved. https://status.transactiongateway.com/
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The NMI gateway is experiencing connectivity issues that may impact some credit card transactions. Please check the status of the NMI Gateway here. https://status.transactiongateway.com/
Report: "Recurring Billing"
Last updateIf you have any trouble with recurring billing please contact help@pike13.com
The recurring billing service was paused and not restarted yesterday during our correction to email service. It is running now and is processing all open invoices that were due today. We verified prior to restart and after running that any invoices that missed payment last night, but received manual intervention today, were skipped in the autorun.
The recurring billing service was paused and not restarted yesterday during our correction to email service. It is running now and is processing all open invoices that were due today. We verified prior to restart and after running that any invoices that missed payment last night, but received manual intervention today, were skipped in the autorun.
Pike13 is collecting information about recurring bills that were not automatically submitted for payment last night. We will rerun those bills for payment today to match the original due date and provide an update when the work is complete. Please note that manually running transactions is working properly.
Report: "Delays with delivery of reports and email notifications"
Last updateThis has been resolved. If you have any issues with notifications or receiving emails from Pike13, please contact help@pike13.com
We are working with a partner to resolve a problem with the delivery of email notifications. The same tool delivers in-application report exports and is therefore also impacted. We expect to resolve this by noon Pacific Time.
Report: "Site slowness due to increased traffic"
Last updateThis has been resolved. If you continue to experience any performance issues, please contact help@pike13.com
We have identified the cause of the abnormal traffic and it was not nefarious. Pike13 is back to normal and the team will continue to watch through the weekend for similar spikes.
Pike13 is investigating an extraordinary traffic occurrence. You may experience slowness or 500 errors until this is resolved. We will update you once we know the cause.
Report: "Client search partial outage"
Last updateThis incident has been resolved.
Client & staff search functionality has been restored
We are continuing to investigate this issue.
We've identified an issue with our client search service and are actively repairing search functionality. The service is currently only returning partial search results and will return complete results shortly.
Report: "SMS Messaging"
Last updateSMS messaging is operating normally now. If your clients or staff are having any issues related to SMS, please email help@pike13.com
Pike13 identified suspicious behavior in our SMS service from unauthorized parties. We paused the service to investigate. After determining that the behavior did not put customer data at risk, but allowed the abuse to slow our SMS transmission, we determined the need to stop the service and make revisions before rereleasing. Please note that notifications are available via email, push and within the web interface. We are working to restore this service as soon as possible, but at this time we expect the outage to last 7-10 days.
Report: "Reporting hardware failure"
Last updateOur hardware for reporting was failing and was replaced by AWS. There was an outage for about 3 minutes. Please let us know if you continue to see any issues with reporting after that time.
Report: "Payment Processing on NMI down for previous 30 minutes"
Last updateNMI allowed security certs to expire at 17:30 Pacific time. Pike13 had no advance warning about this timing, which we will remedy moving forward. NMI provided the update within 30 minutes and payment processing is now back to normal. For full details visit NMI status page, https://status.transactiongateway.com/incidents/spm64fl78g24
Report: "Maintenance Window Extended"
Last updateThis incident has been resolved.
The transition to a larger, speedier ecosystem is still underway.
Report: "Processing Delay"
Last updateThis issue has been resolved. If you are experiencing any other trouble, please email help@pike13.com
Pike13 is experiencing a delay processing background tasks such as sending notifications and schedule/event changes. We have identified the issue and your site will catch up with all background tasks while Pike13 catches up on the backlog of items to process. We will update this issue as soon as everything has caught up. If you are having any other issues, please contact help@pike13.com
Report: "Intermittent errors"
Last updateThis incident has been resolved and all services are fully functional
Pike13 is experiencing intermittent outages related to a recent optimization. The team is monitoring and addressing the issues as they are identified.
Report: "May 2, 2022 Outage"
Last updatePike13 had service degradation affecting some sites. All systems should be back to normal as of 12:30 PST.
We have identified and resolved the issue but will continue monitoring.
We are receiving reports that some Pike13 pages are not loading. We are investigating the issue now.
Report: "Site Slowness, Enrollment data lag"
Last updateAs of 16:00 PDT site performance returned to optimized performance.
We are continuing to monitor for any further issues.
As a continuation of the outage earlier today, there are background processes running to backfill enrollment and other queued data from today's transactions. Currently, the estimate to process the background data is 4 hours, target completion 16:00 PDT. During this period all Pike13 services may perform more slowly than usual.
Report: "Site Outage"
Last updateTwo AWS services became overwhelmed and returned intermittent server 500 errors. Server resources were increased necessitating a database restart which caused a site outage from ~11:20- 11:25 am PDT
Report: "Investigating Client App issue"
Last updateWe have resolved the issues impacting the client app on Android devices. Please uninstall and reinstall the app for the current version.
We have received reports of events not appearing in Upcoming Activities and our team is investigating this issue.
We have released an update that is currently being rolled out to the Google Play store. Once the March 31st update is available, we advise that you uninstall the Pike13 Client App and install the new updated app.
We have identified the issue with activities in the Client App on Android devices. We have submitted a fix and this should be resolved soon.
We have received reports of activities not loading in the Client App and our team is investigating this issue.
Report: "Investigating Intermittent Connectivity Issue"
Last updateThis incident has been resolved.
Our team has identified a potential cause and are working to resolve this issue. Please continue to check our status page for updates.
Report: "Investigating Intermittent Connectivity Issues"
Last updateThis incident has been resolved.
We have identified the issue and anticipate this will be resolved by 9:30 AM. Please continue to check our status page for updates.
Report: "Performance issues"
Last updateThis has been resolved by AWS. Please contact help@pike13.com for any issues you may have.
AWS has resolved their issues and Pike13 should be operating normally for everyone now.
Our infrastructure is on AWS and they have reported issues that affected Pike13. From AWS https://status.aws.amazon.com/: 7:43 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region. 7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region. 8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery. 8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.
We are currently monitoring issues with Pike13 sites not loading. From about 7:40 - 8:00 am Pacific time users were seeing their pages time out or load very slowly.
Report: "Reporting lag"
Last updateDue to last night's maintenance, reporting was lagging behind the usual interval of one hour. This has been fixed as of 10:00 am Pacific Time today. If you are experiencing any issues please contact help@pike13.com.
Report: "Premium Reporting Outage"
Last updateWe have confirmed that Premium Reporting is back to normal after making the changes.
We are currently working on fixing Premium Reporting. We have identified the issue with our vendor and are making necessary changes. Please know that regular reporting is still functional an up-to-date. This is ONLY affecting access Premium Reporting. No data has been affected by the outage.
Report: "Reporting outage"
Last updateWe monitored the status of reports all weekend and it has successfully caught up with the backlog from Oct 1 - Present. Reporting will still have a standard delay of about an hour for the most recent information. Please contact us at help@pike13.com to let us know of any discrepancies.
Our reporting database has successfully caught up with the backlog from Oct 1 - Present. Reporting will still have a standard delay of about an hour for the most recent information. Please contact us at help@pike13.com to let us know of any discrepancies.
Our reporting database is continuing to process transactions from production without failure. It is currently adding data from transactions that occurred after 12pm, Oct 5, 2021. Increased or decreased transactional volume across our businesses impacts the volume to process and subsequently the completion time. The NEW estimate for reporting to catch up is 8pm Pacific Time to reflect the pace we saw in the last 24 hours.
While data is currently migrating from production to reporting, we are estimating that reports should be caught up with data from Oct 1, 2021 - Present by Oct 8, at 5 pm Pacific Time. We appreciate your patience as we worked through this. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.
Yes, data is beginning to populate in reports, BUT... Reports related to transactions occurring between Oct 1, 2021 and present will be changing until the reporting database has fully caught up with the backlog from production. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.
Reports related to transactions occurring between Oct 1, 2021 and present will be changing until the reporting database has fully caught up with the backlog from production. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.
We have identified the issues with reporting and are diligently working to get data from production into the reporting database so that your reports will have information from Oct 1, 2021 - today. We greatly appreciate your patience while we work through this.
While we continue to investigate and resolve the issue with reporting not reflecting changes from Oct 1 through today, we wanted to provide a temporary notification setting change to give you more data for transactions going forward. Owners and managers can click on the Notification icon in the top right of their Pike13 page. Click on the settings link in the right pane. We recommend enabling: 1. A staff member sells something 2. A client buys something
We are continuing to investigate the issues around reports missing data from Oct 1 through today. Please rest assured that transactions and activity are completing and the data still exists in the production database.
We are currently investigating issues around reports missing data. The information is stored in the Pike13 database, but not included in reports. As of this time, it appears to be affecting data from Oct 1, 2021 through today.
Report: "Reporting Update in Progress"
Last updateReporting update complete! During the outage on 8/27, all activity was properly recorded in production, but failed to migrate to our reporting database. We tried several approaches to migrate it and were successful today. If you have run reports or taken action related to activity on 8/27/2021 or early morning 8/28/2021 we strongly recommend you rerun those reports to validate your data. We apologize for the inconvenience, especially at the end of the month when many reports are widely used. We have learned from this and will plan future database maintenance earlier in the month going forward. If you are still having trouble with any reports, please let our team know by emailing help@pike13.com.
Reports related to any transactions on Aug 27 may be inaccurate. We are continuing to work on the reporting database entries that were impacted by the 8/27 outage, and while this has been a priority, the complete resolution continues to evade us.
We apologize for the continued absence of reporting data from 8/27. We are resetting the entire reporting infrastructure to ensure its accuracy and stability. This work will unfortunately continue through the weekend. We will change our status as soon as things are resolved, at the latest 9/6.
We are still working with a third party to setup a secondary reporting connection which should allow for many of the reports to become functional. We will provide another status update by 9/3/21 at 12pm PDT.
We are still working with a third party to setup a secondary reporting connection which should allow for many of the reports to become functional. We will provide another status update by 9/2/21 at 12pm PDT.
Our data migration has taken much longer than we anticipated. We have engaged a third party to setup a secondary reporting connection tonight which should allow for many of the reports to become functional overnight. We will provide another status update 9/2/21 9am PDT.
We are still working to migrate data to the reporting database. The new estimated time of completion is 09/01/2021 at 6 pm PDT.
We are continuing to work on a fix for this issue.
We are still working to migrate data to the reporting database. The new estimated time of completion is 09/01/2021 at 9am PDT.
We are still working to migrate data to the reporting database. The new estimated time of completion is 08/31/2021 at 9pm PDT.
We are currently working to migrate data from production into the reporting database that was missed on Friday 8/27/2021. The data is stored and will be made available in reports as soon as possible. The current estimated time of completion is 08/31/2021 at 6 PM PDT.
Report: "Temporarily Suspending Email"
Last updateAll email suspensions have been cleared.
We are temporarily suspending emails for certain customers. We will reach out prior to suspending your emails.
Report: "Degraded Site Performance"
Last updateWe've resolved the underlying issue. Please reach out if you notice any other issues.
We've identified and implemented a fix for the degraded performance. We will continue to monitor the application.
We are still working on resolving the issue. Stay tuned for more updates.
We've identified the source of the degradation and are working on resolving it.
We're investigating reports of degraded performance.
Report: "Schedule page unavailable."
Last updateThis incident has been resolved. Thank you for your patience.
The Schedule page is down. We are currently investigating.
Report: "Intermittent payment processing problems"
Last updateResolved - Our payment gateway partner NMI (Network Merchants) resolved all issues with credit card processing last night and has been stable since then.
Our payment gateway partner, NMI, has stabilized their platform. Credit card payments are working. We will continue to monitor. You can monitor NMI's status here: http://status.transactiongateway.com
Our payment gateway partner NMI is having intermittent problems. Credit card payments may not work. You can monitor NMI's status here: http://status.transactiongateway.com
Report: "Payment processing outage"
Last updateOur payment gateway partner NMI (Network Merchants) has resolved issues with our credit card processing. Their remaining issues are not affecting Pike13.
Our payment gateway partner NMI (Network Merchants) has restored credit card processing. We will continue to monitor their status: http://status.transactiongateway.com/
Our payment gateway partner NMI (Network Merchants) is experiencing a another major outage. Credit card payments are not working. You can monitor NMI's status here: http://status.transactiongateway.com
Report: "Payment processing outage"
Last updateOur payment gateway partner NMI (Network Merchants) resolved all issues with credit card processing last night and has been stable since then.
Our payment gateway partner NMI (Network Merchants) has restored credit card processing. We will continue to monitor their status: http://status.transactiongateway.com/
Our payment gateway partner NMI (Network Merchants) is still experiencing a major outage. Credit card payments are not working. You can monitor NMI's status here: http://status.transactiongateway.com
Our payment gateway partner NMI (Network Merchants) is experiencing a Major Outage. Payments are not able to run at this time.
Report: "Reporting and Reporting API not working"
Last updateReporting is back! Sorry for the interruption!
Reporting and the Reporting API is not working, we are investigating.