Pike13

Is Pike13 Down Right Now? Check if there is a current outage ongoing.

Pike13 is currently Operational

Last checked from Pike13's official status page

Historical record of incidents for Pike13

Report: "SmartWaiver outage"

Last update
investigating

External partner SmartWaiver is having issues. This is preventing Pike13 users from signing waivers. https://status.smartwaiver.com/

Report: "Application slowness and timeouts"

Last update
resolved

Server pool failed to scale up with increasing load. Fixed.

Report: "Brief outage caused by increased database load"

Last update
resolved

Between 4:19pm and 4:22pm US Pacific Time, there was an intense spike in load on the database that cause Pike13 applications to stop responding.

Report: "Application Slowness"

Last update
resolved

Maintenance complete. Systems operating normally now. We will be monitoring for recurrence of the intermittent slowness.

investigating

The hardware replacement is in progress.

investigating

Slightly delayed while complete a full database backup first. Estimated time of outage now 11:00pm EDT.

investigating

We are going to replace the hardware for the database in 10 minutes from now. There will be a 5-minute outage.

investigating

Pike13 had been intermittently slow throughout the day. We are investigating.

Report: "Database connection waits"

Last update
resolved

Database restart. All system operational. Continuing to monitor performance.

investigating

We are currently investigating this issue.

Report: "500 Errors on Today page for some sites (only affecting Pike13 Support staff)"

Last update
postmortem

After further analysis, it has been determined that this issue only affected Pike13 Support staff. No customers or clients were impacted.

resolved

This incident has been resolved.

monitoring

A fix has been implemented and we are monitoring the results.

identified

The issue has been identified and a fix is being implemented.

investigating

We are currently investigating this issue.

Report: "Reporting API Server Hiccup (no impact)"

Last update
resolved

One of the servers in the Reporting API cluster had connectivity issues and was removed from rotation. The other servers in the cluster were able to handle all of the traffic, which is the purpose of having the redundancy. Everything working normally.

investigating

There does not appear to be any operational impact, but we are checking to be certain.

investigating

Something isn't working right now. We're checking into it and will update you with more information ASAP.

Report: "Major Outage - Database Crash"

Last update
resolved

We're back. Pike13 is up and running. Sorry for any trouble.

monitoring

Systems are back up. Still investigating cause and stability.

investigating

We are continuing to investigate this issue.

investigating

Something isn't working right now. We're checking into it and will update you with more information ASAP.

Report: "Some pages failing with 500 Error"

Last update
resolved

This incident has been resolved.

identified

The issue has been identified and a fix is being implemented.

Report: "Reporting Data Latency"

Last update
resolved

The Reporting system is now fully caught up and operating normally.

monitoring

The issue has been resolved. They system is catching up. Next update will be when latency is back to normal.

investigating

We are continuing to work on this issue.

investigating

Data processing into Reporting is very delayed. We are working on the problem.

Report: "Reporting API Degraded Performance"

Last update
resolved

The unhealthy server has been replaced. All systems back to normal. There was no outage associated with this incident. Only a temporary reduction in capacity and redundancy.

identified

The issue has been identified and a fix is being implemented.

investigating

Systems are still functioning normally. One of the servers in the cluster failed and must be replaced, but the other servers are carrying the load.

investigating

This alarm is regarding the Reporting API.

investigating

Something isn't working right now. We're checking into it and will update you with more information ASAP.

Report: "Reporting and API Data Latency"

Last update
resolved

Reporting data latency has returned to normal.

identified

Data latency is improving slowly, but still has a ways to go. We do not expect it to be completely caught up until sometime tomorrow.

identified

Data updates to the reporting system and the reporting API halted. They have been restarted and are working normally again. Updates are catching up, but are still behind current by approximately 18 hours.

Report: "Minor Outage"

Last update
resolved

This incident has been resolved.

investigating

Emergency restart of primary database.

Report: "Minor Outage"

Last update
resolved

This incident has been resolved.

investigating

The primary database received a large spike in requests. It has recovered.

investigating

Something isn't working right now. We're checking into it and will update you with more information ASAP.

Report: "Minor Outage"

Last update
resolved

This incident has been resolved.

investigating

Systems are recovering and returning to normal operating state.

Report: "Minor Outage"

Last update
resolved

This incident has been resolved.

investigating

This incident began at 4:20am PST and was resolved at 6:34am PST.

investigating

Something isn't working right now. We're checking into it and will update you with more information ASAP.

Report: "Application outage"

Last update
resolved

Maintenance release deploy to production caused widespread 500 errors across the platform until the new code propagated to all new hosts.

Report: "Reporting Lag"

Last update
resolved

Regular Reporting and the reporting_api have returned to service.

identified

A portion of the reporting infrastructure is under repair. We are using every resource available to expedite this, but expect the service to be down through the end of September. We are utilizing premium reporting and its connection to a different database to offer access to mission-critical data, like transaction reports. Contact help@pike13.com if you need access to this tool. The Core API is functioning, the reporting API does not have a complete data source.

identified

Pike13 sites are operational.

identified

We are continuing to work on a fix for this issue.

identified

Pike13 sites has been paused briefly in order to upscale resources being impacted by the reporting sync.

identified

We are continuing to work on a fix for this issue.

identified

New changes are making their way into reporting as of 5pm Pacific time today (Sept 8). We are still working on populating data from 9/4 - 9/8 5pm. Please rest assured that all data and transactions are in production and will be migrated over to reporting as soon as possible. We apologize for this disruption.

investigating

Pike13 and reporting tools are restored. Reporting and the Reporting API are still missing transaction data from Sept 4, 2023 to now. We are still working to make that data available.

Report: "Background Jobs Delayed, Sidekiq down"

Last update
resolved

This incident has been resolved. We will continue monitoring as the backlog of notifications is rolled out.

identified

The issue has been identified and repairs are in progress. Live Payments are not impacted. Auto-billing from last night is scheduled to run later today.

Report: "Email Delivery"

Last update
resolved

Email has been restored. Emails that should have been sent from midnight to noon PST Friday 8/18 may not be delivered as expected.

investigating

Pike13 is not currently delivering emails. We are investigating this issue and will update this page as possible.

Report: "Payment Processing"

Last update
resolved

This has been resolved. We apologize for the brief interruption. If you are still having any trouble, please contact help@pike13.com.

identified

Pike13 is experiencing a payment connection error during the roll-out of a new payment gateway. We are working to immediately resolve this and will repost when the resolution is complete.

Report: "Document Signing Disruption"

Last update
resolved

This has been resolved. Please email help@pike13.com if you have any issues.

investigating

Our partner for providing document signing service is experiencing a disruption. We are working with them to resolve this issue and will provide an update here when the problem is resolved.

Report: "NMI Gateway is reporting connectivity issues"

Last update
resolved

NMI has reported that this issue has been resolved. https://status.transactiongateway.com/

monitoring

We are continuing to monitor for any further issues.

monitoring

A fix has been implemented and we are monitoring the results.

investigating

The NMI gateway is experiencing connectivity issues that may impact some credit card transactions. Please check the status of the NMI Gateway here. https://status.transactiongateway.com/

Report: "Recurring Billing"

Last update
resolved

If you have any trouble with recurring billing please contact help@pike13.com

monitoring

The recurring billing service was paused and not restarted yesterday during our correction to email service. It is running now and is processing all open invoices that were due today. We verified prior to restart and after running that any invoices that missed payment last night, but received manual intervention today, were skipped in the autorun.

identified

The recurring billing service was paused and not restarted yesterday during our correction to email service. It is running now and is processing all open invoices that were due today. We verified prior to restart and after running that any invoices that missed payment last night, but received manual intervention today, were skipped in the autorun.

identified

Pike13 is collecting information about recurring bills that were not automatically submitted for payment last night. We will rerun those bills for payment today to match the original due date and provide an update when the work is complete. Please note that manually running transactions is working properly.

Report: "Delays with delivery of reports and email notifications"

Last update
resolved

This has been resolved. If you have any issues with notifications or receiving emails from Pike13, please contact help@pike13.com

identified

We are working with a partner to resolve a problem with the delivery of email notifications. The same tool delivers in-application report exports and is therefore also impacted. We expect to resolve this by noon Pacific Time.

Report: "Site slowness due to increased traffic"

Last update
resolved

This has been resolved. If you continue to experience any performance issues, please contact help@pike13.com

monitoring

We have identified the cause of the abnormal traffic and it was not nefarious. Pike13 is back to normal and the team will continue to watch through the weekend for similar spikes.

investigating

Pike13 is investigating an extraordinary traffic occurrence. You may experience slowness or 500 errors until this is resolved. We will update you once we know the cause.

Report: "Client search partial outage"

Last update
resolved

This incident has been resolved.

investigating

Client & staff search functionality has been restored

investigating

We are continuing to investigate this issue.

investigating

We've identified an issue with our client search service and are actively repairing search functionality. The service is currently only returning partial search results and will return complete results shortly.

Report: "SMS Messaging"

Last update
resolved

SMS messaging is operating normally now. If your clients or staff are having any issues related to SMS, please email help@pike13.com

identified

Pike13 identified suspicious behavior in our SMS service from unauthorized parties. We paused the service to investigate. After determining that the behavior did not put customer data at risk, but allowed the abuse to slow our SMS transmission, we determined the need to stop the service and make revisions before rereleasing. Please note that notifications are available via email, push and within the web interface. We are working to restore this service as soon as possible, but at this time we expect the outage to last 7-10 days.

Report: "Reporting hardware failure"

Last update
resolved

Our hardware for reporting was failing and was replaced by AWS. There was an outage for about 3 minutes. Please let us know if you continue to see any issues with reporting after that time.

Report: "Payment Processing on NMI down for previous 30 minutes"

Last update
resolved

NMI allowed security certs to expire at 17:30 Pacific time. Pike13 had no advance warning about this timing, which we will remedy moving forward. NMI provided the update within 30 minutes and payment processing is now back to normal. For full details visit NMI status page, https://status.transactiongateway.com/incidents/spm64fl78g24

Report: "Maintenance Window Extended"

Last update
resolved

This incident has been resolved.

monitoring

The transition to a larger, speedier ecosystem is still underway.

Report: "Processing Delay"

Last update
resolved

This issue has been resolved. If you are experiencing any other trouble, please email help@pike13.com

monitoring

Pike13 is experiencing a delay processing background tasks such as sending notifications and schedule/event changes. We have identified the issue and your site will catch up with all background tasks while Pike13 catches up on the backlog of items to process. We will update this issue as soon as everything has caught up. If you are having any other issues, please contact help@pike13.com

Report: "Intermittent errors"

Last update
resolved

This incident has been resolved and all services are fully functional

investigating

Pike13 is experiencing intermittent outages related to a recent optimization. The team is monitoring and addressing the issues as they are identified.

Report: "May 2, 2022 Outage"

Last update
resolved

Pike13 had service degradation affecting some sites. All systems should be back to normal as of 12:30 PST.

monitoring

We have identified and resolved the issue but will continue monitoring.

investigating

We are receiving reports that some Pike13 pages are not loading. We are investigating the issue now.

Report: "Site Slowness, Enrollment data lag"

Last update
resolved

As of 16:00 PDT site performance returned to optimized performance.

monitoring

We are continuing to monitor for any further issues.

monitoring

As a continuation of the outage earlier today, there are background processes running to backfill enrollment and other queued data from today's transactions. Currently, the estimate to process the background data is 4 hours, target completion 16:00 PDT. During this period all Pike13 services may perform more slowly than usual.

Report: "Site Outage"

Last update
resolved

Two AWS services became overwhelmed and returned intermittent server 500 errors. Server resources were increased necessitating a database restart which caused a site outage from ~11:20- 11:25 am PDT

Report: "Investigating Client App issue"

Last update
resolved

We have resolved the issues impacting the client app on Android devices. Please uninstall and reinstall the app for the current version.

monitoring

We have received reports of events not appearing in Upcoming Activities and our team is investigating this issue.

monitoring

We have released an update that is currently being rolled out to the Google Play store. Once the March 31st update is available, we advise that you uninstall the Pike13 Client App and install the new updated app.

investigating

We have identified the issue with activities in the Client App on Android devices. We have submitted a fix and this should be resolved soon.

investigating

We have received reports of activities not loading in the Client App and our team is investigating this issue.

Report: "Investigating Intermittent Connectivity Issue"

Last update
resolved

This incident has been resolved.

investigating

Our team has identified a potential cause and are working to resolve this issue. Please continue to check our status page for updates.

Report: "Investigating Intermittent Connectivity Issues"

Last update
resolved

This incident has been resolved.

identified

We have identified the issue and anticipate this will be resolved by 9:30 AM. Please continue to check our status page for updates.

Report: "Performance issues"

Last update
resolved

This has been resolved by AWS. Please contact help@pike13.com for any issues you may have.

monitoring

AWS has resolved their issues and Pike13 should be operating normally for everyone now.

monitoring

Our infrastructure is on AWS and they have reported issues that affected Pike13. From AWS https://status.aws.amazon.com/: 7:43 AM PST We are investigating Internet connectivity issues to the US-WEST-2 Region. 7:52 AM PST We are investigating Internet connectivity issues to the US-WEST-1 Region. 8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-2 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery. 8:01 AM PST We have identified the root cause of the Internet connectivity to the US-WEST-1 Region and have taken steps to restore connectivity. We have seen some improvement to Internet connectivity in the last few minutes but continue to work towards full recovery.

monitoring

We are currently monitoring issues with Pike13 sites not loading. From about 7:40 - 8:00 am Pacific time users were seeing their pages time out or load very slowly.

Report: "Reporting lag"

Last update
resolved

Due to last night's maintenance, reporting was lagging behind the usual interval of one hour. This has been fixed as of 10:00 am Pacific Time today. If you are experiencing any issues please contact help@pike13.com.

Report: "Premium Reporting Outage"

Last update
resolved

We have confirmed that Premium Reporting is back to normal after making the changes.

identified

We are currently working on fixing Premium Reporting. We have identified the issue with our vendor and are making necessary changes. Please know that regular reporting is still functional an up-to-date. This is ONLY affecting access Premium Reporting. No data has been affected by the outage.

Report: "Reporting outage"

Last update
resolved

We monitored the status of reports all weekend and it has successfully caught up with the backlog from Oct 1 - Present. Reporting will still have a standard delay of about an hour for the most recent information. Please contact us at help@pike13.com to let us know of any discrepancies.

monitoring

Our reporting database has successfully caught up with the backlog from Oct 1 - Present. Reporting will still have a standard delay of about an hour for the most recent information. Please contact us at help@pike13.com to let us know of any discrepancies.

monitoring

Our reporting database is continuing to process transactions from production without failure. It is currently adding data from transactions that occurred after 12pm, Oct 5, 2021. Increased or decreased transactional volume across our businesses impacts the volume to process and subsequently the completion time. The NEW estimate for reporting to catch up is 8pm Pacific Time to reflect the pace we saw in the last 24 hours.

monitoring

While data is currently migrating from production to reporting, we are estimating that reports should be caught up with data from Oct 1, 2021 - Present by Oct 8, at 5 pm Pacific Time. We appreciate your patience as we worked through this. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.

identified

Yes, data is beginning to populate in reports, BUT... Reports related to transactions occurring between Oct 1, 2021 and present will be changing until the reporting database has fully caught up with the backlog from production. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.

identified

Reports related to transactions occurring between Oct 1, 2021 and present will be changing until the reporting database has fully caught up with the backlog from production. We will post when everything is current, and caution against taking action regarding the information in those reports until all data is processed.

identified

We have identified the issues with reporting and are diligently working to get data from production into the reporting database so that your reports will have information from Oct 1, 2021 - today. We greatly appreciate your patience while we work through this.

investigating

While we continue to investigate and resolve the issue with reporting not reflecting changes from Oct 1 through today, we wanted to provide a temporary notification setting change to give you more data for transactions going forward. Owners and managers can click on the Notification icon in the top right of their Pike13 page. Click on the settings link in the right pane. We recommend enabling: 1. A staff member sells something 2. A client buys something

investigating

We are continuing to investigate the issues around reports missing data from Oct 1 through today. Please rest assured that transactions and activity are completing and the data still exists in the production database.

investigating

We are currently investigating issues around reports missing data. The information is stored in the Pike13 database, but not included in reports. As of this time, it appears to be affecting data from Oct 1, 2021 through today.

Report: "Reporting Update in Progress"

Last update
resolved

Reporting update complete! During the outage on 8/27, all activity was properly recorded in production, but failed to migrate to our reporting database. We tried several approaches to migrate it and were successful today. If you have run reports or taken action related to activity on 8/27/2021 or early morning 8/28/2021 we strongly recommend you rerun those reports to validate your data. We apologize for the inconvenience, especially at the end of the month when many reports are widely used. We have learned from this and will plan future database maintenance earlier in the month going forward. If you are still having trouble with any reports, please let our team know by emailing help@pike13.com.

identified

Reports related to any transactions on Aug 27 may be inaccurate. We are continuing to work on the reporting database entries that were impacted by the 8/27 outage, and while this has been a priority, the complete resolution continues to evade us.

identified

We apologize for the continued absence of reporting data from 8/27. We are resetting the entire reporting infrastructure to ensure its accuracy and stability. This work will unfortunately continue through the weekend. We will change our status as soon as things are resolved, at the latest 9/6.

identified

We are still working with a third party to setup a secondary reporting connection which should allow for many of the reports to become functional. We will provide another status update by 9/3/21 at 12pm PDT.

identified

We are still working with a third party to setup a secondary reporting connection which should allow for many of the reports to become functional. We will provide another status update by 9/2/21 at 12pm PDT.

identified

Our data migration has taken much longer than we anticipated. We have engaged a third party to setup a secondary reporting connection tonight which should allow for many of the reports to become functional overnight. We will provide another status update 9/2/21 9am PDT.

identified

We are still working to migrate data to the reporting database. The new estimated time of completion is 09/01/2021 at 6 pm PDT.

identified

We are continuing to work on a fix for this issue.

identified

We are still working to migrate data to the reporting database. The new estimated time of completion is 09/01/2021 at 9am PDT.

identified

We are still working to migrate data to the reporting database. The new estimated time of completion is 08/31/2021 at 9pm PDT.

identified

We are currently working to migrate data from production into the reporting database that was missed on Friday 8/27/2021. The data is stored and will be made available in reports as soon as possible. The current estimated time of completion is 08/31/2021 at 6 PM PDT.

Report: "Temporarily Suspending Email"

Last update
resolved

All email suspensions have been cleared.

investigating

We are temporarily suspending emails for certain customers. We will reach out prior to suspending your emails.

Report: "Degraded Site Performance"

Last update
resolved

We've resolved the underlying issue. Please reach out if you notice any other issues.

monitoring

We've identified and implemented a fix for the degraded performance. We will continue to monitor the application.

identified

We are still working on resolving the issue. Stay tuned for more updates.

identified

We've identified the source of the degradation and are working on resolving it.

investigating

We're investigating reports of degraded performance.

Report: "Schedule page unavailable."

Last update
resolved

This incident has been resolved. Thank you for your patience.

investigating

The Schedule page is down. We are currently investigating.

Report: "Intermittent payment processing problems"

Last update
resolved

Resolved - Our payment gateway partner NMI (Network Merchants) resolved all issues with credit card processing last night and has been stable since then.

monitoring

Our payment gateway partner, NMI, has stabilized their platform. Credit card payments are working. We will continue to monitor. You can monitor NMI's status here: http://status.transactiongateway.com

identified

Our payment gateway partner NMI is having intermittent problems. Credit card payments may not work. You can monitor NMI's status here: http://status.transactiongateway.com

Report: "Payment processing outage"

Last update
resolved

Our payment gateway partner NMI (Network Merchants) has resolved issues with our credit card processing. Their remaining issues are not affecting Pike13.

monitoring

Our payment gateway partner NMI (Network Merchants) has restored credit card processing. We will continue to monitor their status: http://status.transactiongateway.com/

identified

Our payment gateway partner NMI (Network Merchants) is experiencing a another major outage. Credit card payments are not working. You can monitor NMI's status here: http://status.transactiongateway.com

Report: "Payment processing outage"

Last update
resolved

Our payment gateway partner NMI (Network Merchants) resolved all issues with credit card processing last night and has been stable since then.

monitoring

Our payment gateway partner NMI (Network Merchants) has restored credit card processing. We will continue to monitor their status: http://status.transactiongateway.com/

identified

Our payment gateway partner NMI (Network Merchants) is still experiencing a major outage. Credit card payments are not working. You can monitor NMI's status here: http://status.transactiongateway.com

identified

Our payment gateway partner NMI (Network Merchants) is experiencing a Major Outage. Payments are not able to run at this time.

Report: "Reporting and Reporting API not working"

Last update
resolved

Reporting is back! Sorry for the interruption!

investigating

Reporting and the Reporting API is not working, we are investigating.