Historical record of incidents for BigPanda
Report: "Rollout Cloudflare WAF Prod-US"
Last updateThe scheduled maintenance has been completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
In support of our commitment towards improving our services and infrastructure, we will be performing the following maintenance to the BigPanda platform.Start Time: June 04, 2025 07:00 UTCEnd Time: June 04 2025 11:00 UTCDuration: 4 hourWhat's Happening?During this maintenance window, we will be deploying an update to our network security infrastructure by enabling Cloudflare WAF in the US Production region. We will be modifying the primary DNS configuration as part of this rollout. During this time, users may experience intermittent network connectivity issues, which could impact access to the UI and API.Component(s) undergoing maintenance:• API• BigPanda Console• Incident SharingWhat's the Impact?We're taking precautionary steps to reduce sustained impact during this maintenance; however, users may experience intermittent connectivity to the UI.What isn't Impacted?All other areas of the BigPanda platform - including Inbound Integrations, Analytics - will be unaffected by this maintenance.Is User Action Required?Users are required to check their legacy network devices for compatibility with Cloudflare WAF. You can find the steps to do this [here]The planned maintenance has successfully completed our QA validation and regression process. We will be closely monitoring the maintenance work, and BigPanda's status page will be updated throughout the maintenance period with any changes to service operations.We apologize for any inconvenience that this may cause. If you have any questions, please reach out to BigPanda Support via our live in-app chat or via support@bigpanda.io.
Report: "BigPanda Support Portal Ticket Submission Down"
Last updateThe incident is resolve and you can now resume normal ticket submission to the BigPanda Support Portal
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing issues with our ticketing vendor that we’re working to resolve soon. We are keeping an eye out for urgent matters, and for now it is advisable to reach out to us via chat.
Report: "BigPanda Support Portal Ticket Submission Down"
Last updateThe incident is resolve and you can now resume normal ticket submission to the BigPanda Support Portal
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently experiencing issues with our ticketing vendor that we’re working to resolve soon. We are keeping an eye out for urgent matters, and for now it is advisable to reach out to us via chat.
Report: "Console Loading Delay"
Last updateThe Issue has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented by restarting one of the affected services.
We are currently experiencing latency issues with the BigPanda console. Our team is actively investigating the issue and working to resolve it as quickly as possible.
Report: "Console Loading Delay"
Last updateThe Issue has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented by restarting one of the affected services.
We are currently experiencing latency issues with the BigPanda console. Our team is actively investigating the issue and working to resolve it as quickly as possible.
Report: "Production Environment Latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are currently experiencing latencies in our pipeline related to the Correaltion and Alert filtering processes. Our R&D team is actively working on remediating this issue.
Report: "Console"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Our R&D team is actively working to implement a fix for issues affecting the BigPanda console.
We are currently experiencing issues with the BigPanda console and are investigating this matter to identify the affected components. Additional updates will be shared as more details become available. We apologize for any inconvenience that this may be causing.
Report: "General Pipeline Latency"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are seeing latencies in the BigPanda pipeline an we are actively investigating this issue.
Report: "BigPanda - Unified Analytics (Sisense) Widgets Not Loading"
Last updateOur Unified Analytics services have returned to their normal operation.
A fix has been implemented and we are currently monitoring for stability within our Unified Analytics service. Any additional updates will be shared as more details become available.
We have identified the issue and we continue to work with our third-party analytics provider (Sisense) to restore full functionality to our Unified Analytics service. Additional updates will be shared as more details become available.
Sisense, a Third Party Service, is currently experiencing issues and some Unified Analytics Dashboard widgets may not load data results at this time. Our teams are currently engaged and investigating. <b>What's impacted?</b> Unified Analytics Analytics <b>What’s not impacted?</b> Pipeline APIs Incident Sharing BigPanda Console We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Bigpanda- Outbound RDS Locked Shares"
Last updateThis incident has been resolved. All services are operational and functioning at nominal levels. We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
After investigation, the team identified the problematic area to be within the caching database. The team took corrective actions, and the issue has been mitigated. Sharing activity to ServiceNow and Jira ticketing systems should be operating nominally. We will continue to monitor the situation and provide further updates as necessary.
Our team is investigating this issue. We have identified that out-of-the-box ServiceNow and Jira ticketing integrations are affected. There is no impact on Initial shares sent to the target systems. However, there is a delay in subsequent updates to the shared tickets.
Report: "Unstable Unified Analytics"
Last updateOur Unified Analytics services have returned to their normal operation. <b>Start time:</b> 09/10/2024 15:55 UTC <b>End time:</b> 09/10/2024 17:00 UTC <b>What was impacted?</b> Unified Analytics We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team is actively investigating the issue to identify the root cause. We have mobilized responders and are working to escalate the issue with our third-party provider, Sisense.
Report: "Inbound services are experiencing latencies"
Last updateThis incident has been resolved.
We are seeing reduced latencies for most affected clients. We will continue to monitor the situation.
Some customers might still experience latencies. We have identified the issue and are working to mitigate it.
We are currently investigating this issue.
Report: "BigPanda - Outbound sharing latencies"
Last updateThis incident has been resolved.
The issue has been mitigated and we continue to monitor the situation.
A fix has been implemented, and latency is declining. We continue to monitor the situation. (correction to current status, as latency is still declining, but not fully reduced)
A fix has been implemented, and latency is declining. We continue to monitor the situation.
The issue has been identified and a fix is being implemented.
We are currently encountering delays in sharing incidents from BigPanda to outbound integrations. We are investigating the issue.
Report: "BigPanda - Outbound sharing latencies"
Last updateThis incident has been resolved.
The issue has been mitigated and we continue to monitor the situation.
A fix has been implemented, and latency is declining. We continue to monitor the situation as delays are still possible.
The issue has been identified and a fix is being implemented.
We are currently encountering delays in sharing incidents from BigPanda to outbound integrations. We are currently investigating the issue.
Report: "OIM Latencies"
Last updateThis incident has been resolved.
The latency is now resolved and our OIM service is fully operational. We are continuing to monitor.
A fix has been implemented and we are monitoring the results.
The queue is now starting to clear and latencies dropping
We are continuing to investigate. We have determined that this is not a complete outage in OIM as many events are still being processed. Only some events are being held in a queue.
Some OIM events are queued and not making it to the BigPanda console. The team are engaged and we are currently investigating this issue.
Report: "Update on Sisense Customer Data Compromise"
Last updateBigPanda Security Team has completed their investigation and the risk is contained, and we are considering this incident as resolved. As planned on May 28th 2024, we successfully reconnected Unified Analytics for the majority of our customers, and we continue to work with customers that have specific requirements. We are committed to full transparency with our clients and encourage you to contact us (trust@bigpanda.io) if you have any questions or concerns.
BigPanda was impacted by a security incident within a technology partner on Apr 11, 2024. We have completed an investigation and the risk is contained. With our commitment to transparency, we want to provide guidance and clarity about our plans moving forward. Background The breach started within a 3rd party vendor (Sisense) that BigPanda utilizes for the Unified Analytics dashboards. Full details can be found at https://status.bigpanda.io/ Both BigPanda and Sisense independently engaged external security specialists to conduct a forensic investigation of the incident. BigPanda is now confident that the risk is over. BigPanda will share an independent forensics statement with customers and provide an attestation that BigPanda’s platform is secure. We have also validated, with Sisense, that all necessary security measures were taken on their side. We can provide an attestation letter from Sisense as well. If you have any questions or concerns, we are available (trust@bigpanda.io). Looking Forward With the incident behind us, we want to get our Unified Analytics service fully restored for customers ASAP. Here is our plan: Reactivation BigPanda will reconnect to the Sisense SaaS platform and re-enable Unified Analytics on May 28, 2024. This will be the default action for all customers. Early Access If you would like earlier access to Unified Analytics, BigPanda can reconnect individual accounts starting on May 23rd. To discuss this possibility, please contact trust@bigpanda.io Opt-out If you do not wish to reconnect your organization's data to the Sisense SaaS platform, BigPanda will support that decision. To opt-out, please contact trust@bigpanda.io. We will set up a call to discuss your concerns, build a unique timeline and discuss potential alternatives for your organization. Summary BigPanda takes operational security very seriously. The lessons from this incident will guide us to further harden both our technology and internal processes. We are committed to full transparency with our clients and encourage you to contact us (trust@bigpanda.io) if you have any questions or concerns.
We are continuing to monitor for any further issues.
Following our latest notice on Saturday April 13th, 2024, we are writing to provide our Customers with an update on the latest available information regarding the Sisense Data Breach. On Tuesday April 16, 2024, BigPanda was informed by Sisense that the Breach involved Sisense data cubes. With this new information, we are finalizing a process to export the data that was compromised in the Sisense cubes and provide it to our Customers. We expect this process to be complete by the end of the week, at which time we will be able to schedule individual times upon request. Our investigation is still ongoing. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
BigPanda became aware of a Data Breach through its subprocessor Sisense on Thursday April 11th, 2024. We informed Customers that same day of the Breach via the BigPanda Status Page (status.bigpanda.io). We are writing now to notify our Customers and provide an update on the latest available information. Incident Summary A Threat Actor compromised the Sisense platform and gained access to BigPanda Customer Data through Snowflake. To our knowledge, the situation has been contained. Our investigation indicates that at no time did the Threat Actor gain access to BigPanda’s Core Platform. BigPanda’s Use of Sisense and Snowflake The BigPanda Core Platform stores metrics data in Snowflake, which is then loaded into Sisense to deliver metrics reporting via Unified Analytics. Incident and Data Scope Sisense has confirmed the Breach but has yet to provide detailed information about the incident. After BigPanda’s internal investigation, it was confirmed that the Threat Actor accessed BigPanda Customer Data stored in Snowflake via Sisense. It was determined that a read-only service account credential for Snowflake was compromised due to the Breach, thus enabling the threat actor to make SQL queries on the Snowflake database. The BigPanda Customer Data accessed included usernames (which contain BigPanda customer email addresses) and alert and incident tag information. This data in Snowflake enables customers to identify trends within their environments, examples are included below: Username (email address), app_id, source_host, priority, start_time, end_time, date, etc Incident Containment & Remediation Upon becoming aware of the incident, the BigPanda Security Incident Response Team immediately launched an investigation, which included execution of the following steps: - Requesting information and contacting Sisense; - Reviewing all internal systems audit logs for abnormal activity; - Performing programmatic updates to rotate user access keys and BigPanda’s SSO token via prepared scripts; - Rotating service account keys for systems that feed data to Sisense cubes. - Stopping any new Customer Data from being sent to Sisense. We will re-evaluate this decision in the days to come based on information that will be provided to us from Sisense. A more detailed list of the actions taken by the BigPanda Security Incident Response team to ensure the containment of the Breach are as follows: - Changed all Sisense-related passwords on my.sisense.com - For non-Single Sign-On (SSO) access: -- Replaced the Secret in the Base Configuration Security section with your GUID/UUID. -- Reset passwords for all users in the Sisense application. -- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user. - For SSO access: -- Updated sso.shared_secret in Sisense and updated the newly generated value in the SSO handler. -- Rotated the x.509 certificate in our SSO SAML identity provider. -- Rotated the OpenID client secret. -- Updated SSO settings in Sisense with the revised values. -- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user. -- Reset credentials in the database used by the Sisense application. Ongoing Investigation and Mitigation BigPanda is continuing to investigate the issue and will conduct a full external forensic investigation of its Data Platform. We will continue to provide updates as they become available. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
We want to inform you that we have identified and stopped some suspicious activities from an unidentified threat actor. Pursuing our audit of this security incident, we were able to identify there was read-only access to some user data. This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering. As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed. The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event. All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures. For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.
Report: "User Login Issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some users may be experiencing issues with user login. Our teams are engaged and investigating. <b>What's impacted?</b> User Login We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Search Latency"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Pipeline Latencies"
Last updateThis incident has been fully resolved.
The team have performed a rollback and a manual restart and events are being processed. We are monitoring the results.
Some users may be experiencing data pipeline latencies. Our teams are engaged and investigating. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> All other areas of the platform should not be impacted. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda - Outbound sharing latencies"
Last updateThis incident has been resolved.
As of 10:14 PM PST we have implemented a mitigation and are actively monitoring. Only customers using the auto-share delay feature are impacted.
We are currently encountering delays in sharing incidents from BigPanda to outbound integrations. We are currently investigating the issue.
Report: "BigPanda - Incidents API 503 errors"
Last updateThis incident has been resolved
A fix has been implemented and we will continue to monitor for results.
The issue has been identified and a fix has been implemented.
Some users may be experiencing issues with Incidents API. Our teams are engaged and investigating. <b>What's impacted?</b> • Tier 1 APIs: Automation & Event Processing • Tier 2 APIs: Supplemental • Tier 3 APIs: Tertiary We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Pipeline Latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are making progress on investigation and we are continuing to work on this issue to mitigate impacts to our customers as quickly as possible.
The team have performed a deployment rollback and a manual restart. We are currently seeing throttler crashes that we are currently investigating.
We are continuing to investigate this issue.
Some users may be experiencing data pipeline latencies. Our teams are engaged and investigating. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda - Outbound sharing latencies"
Last updateThis incident has been resolved.
A fix has been implemented and we are currently processing the backlog in our queue. We are continuing to monitor our Pipeline services for any change in operational capacity.
The issue has been identified, regarding delays in sharing incidents from BigPanda to outbound integrations. This is also affecting related sharing services that are usually highlighted within the Activity Feed. Our integrations team is still engaged.
We are currently encountering delays in sharing incidents from BigPanda to outbound integrations. We are currently investigating the issue.
Report: "Incidents not resolving via External System"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue
BigPanda users may be experiencing issues with resolving incidents via external systems. Our teams are engaged and investigating. <b>What's impacted?</b> Incident Sharing We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Event Processing Latencies"
Last updateThe incident has been resolved and pipeline services have returned to normal operational service levels.
A fix has been implemented and we are currently monitoring the rest of our pipeline services.
The issue has been identified and our teams are currently engaged with remediating the issue for our Alert Enrichment services. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
BigPanda users may be experiencing latencies within their BigPanda instance, mainly in Event Processing services. Our teams are engaged and investigating. <b>What's impacted?</b> Event Processing - Alert Enrichment We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda - Login Services Unavailable"
Last updateThis incident has been resolved, and BP users are now able to login to the console again. We apologize for any inconvenience that this may have caused. Should you continue to experience any accessibility issues, please reach out to us via our in-app chat or at support@bigpanda.io.
A fix has been implemented, and the issue should be mitigated. We will be monitoring within the next 30 minutes for any further developments.
We are continuing to investigate this issue.
Users may be experiencing issues with logging in BigPanda. Our teams are currently engaged and investigating. <b>What's impacted?</b> • Login • SSO We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Third Party AWS Issues"
Last updateAWS has marked their incident as resolved and our services have fully recovered.
AWS has recovered many services and are working towards full recovery. We are continuing to monitor impact and work towards full resolution.
AWS is reporting improvement in their services. We’re not fully recovered yet, but are seeing fewer errors and more expected traffic.
Our team has identified that the impact is to most inbound and outbound integrations. Our team is currently in the process of switching routing for inbound and outbound integrations to direct to the US-WEST region. We will update you when this is complete.
Our 3rd Party Service, AWS, has identified the root cause of their service issues, and are actively working to resolve this issue. We have also added on BigPanda Services affected here, and we are working to mitigate pipeline impacts as quickly as possible.
Our third party service AWS is currently experiencing issues. Our teams are engaged and investigating. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda - Unified Analytics"
Last updateOur Unified Analytics services have returned to their normal operation. <b>Start time:</b> 06/12/2023 15:25 UTC <b>End time:</b> 06/12/2023 17:49 UTC <b>What was impacted?</b> Unified Analytics We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
A fix has been implemented and we are currently monitoring for stability within our Unified Analytics service. Any additional updates will be shared as more details become available.
We are working with our third-party Analytics provider (Sisense) to restore full functionality to our Unified Analytics service. Additional updates will be shared as more details become available.
We are currently experiencing issues where dashboards & reports within Unified Analytics are not loading due to widget errors. Our teams are currently engaged and investigating on the issue. <b>What's impacted?</b> Unified Analytics Analytics <b>What’s not impacted?</b> Pipeline APIs Incident Sharing BigPanda Console We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Event Processing Latencies"
Last updateThe incident has been resolved and pipeline services have returned to normal operational service levels.
We are continuing to monitor our Pipeline services for any change in operational capacity.
The issue has been identified and our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity.
BigPanda Users may be experiencing Pipeline latencies, mainly in Event Processing services. Our teams are engaged and investigating. <b>What's impacted?</b> Pipeline: Throttler Service We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "UI not rendering"
Last updateThis incident has been resolved, and the BigPanda UI is now operating within normal service levels.
We have implemented the fix and are monitoring the results.
The issue was identified and a fix was implemented for our UI issue.
We are still working on our UI issue and you may still experience slowness.
We are currently experiencing issues in our UI and are investigating.
Report: "BigPanda - ServiceNow Autoshares"
Last updateThis incident has been resolved. All services are operational and functioning at nominal levels.
The backlog continues to process, and at this time, over 75% of backlogged data has been successfully processed.
The backlog continues to process, and at this time, over 50% of backlogged data has been successfully processed. The root cause was due to a sudden and abnormal influx of data into the processing queue that very quickly overwhelmed the service. We have identified and mitigated the source of this increased volume. We have scaled out the service to maximize processing power.
We are still processing the backlog and monitoring this.
We have configured an additional trigger for the queue. We are still processing the backlog.
We are continuing to see a decrease in the backlog and this is continuing to be processed.
We have reached a peak in our queue. The number left to process is decreasing, but this will still take some time.
The issue has been identified and we are currently processing the backlog.
We are currently investigating an issue with shares to ServiceNow.
Report: "Pipeline Latencies"
Last updateThis incident has been resolved. Pipeline should be operating as normal.
A fix has been implemented and we are noticing the lags decreasing.
Correlation specifically has not had any lags in the past 20 minutes. We still have lags in alert filtering. We are continuing to work on this issue.
The issue has been identified and we are seeing that the lag in correlation is over, but the lag in alert filtering seems to still be going on. We are working on this.
Some users may be experiencing data pipeline latencies. Our team are looking into the issue.
Report: "Incident tags missing from the UI"
Last updateIncident has now been resolved. Incident tags should now be visible as normal within the UI.
The fix has been implemented and we are monitoring this.
Issue has been identified and a fix is being implemented.
We are aware of an issue where some users have reported that they're not able to see Incident Tags in the incident view. We are investigating this issue.
Report: "UI is down."
Last updateIssue has been resolved.UI should be accessible as normal.
Services seem to be running as normal again. UI should be accessible and we are monitoring this.
The issue has been identified. UI is starting to load. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Some users may be unable to access their UI at this time. Our teams are engaged and investigating.
Report: "BigPanda - SSO Login Unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring as systems stabilize.
The issue has been identified and a fix is currently being implemented. The next update will be provided in the next 30 minutes.
Users may be experiencing issues with logging in BigPanda through SSO. Our teams are engaged and investigating. <b>What's impacted?</b> • Login • SSO We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Bigpanda Console Issue"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring as systems stabilize.
Some users may be experiencing latencies in their pipeline. At this time our teams are engaged and investigating. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda Console issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the issue and are working to remediate the issue. Note that users may still experience delays in new incidents arriving into the console, incident updates, and autoshares.
The issues within the pipeline are stabalising however some orgs may still see a lag. We are continuing to investigate the issue.
We are aware there maybe potential issues regarding the BigPanda console and are currently investigating.
Report: "BigPanda - Unified Analytics (Sisense)"
Last updateOur Unified Analytics services have returned to their normal operation. <b>Start time:</b> 12/20/2022 18:26 UTC <b>End time:</b> 12/20/2022 20:33 UTC <b>What was impacted?</b> Unified Analytics We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
A fix has been implemented by Sisense and we are currently monitoring for stability within our Unified Analytics service. Any additional updates will be shared as more details become available.
Sisense has identified the root cause of the issue, and are in the process of applying mitigations to address the issue. Any additional updates will be shared as more details become available.
A fix has been implemented and we are currently monitoring for stability within our Unified Analytics service. Any additional updates will be shared as more details become available.
We continue to work with our third-party analytics provider (Sisense) to restore full functionality to our Unified Analytics service. Additional updates will be shared as more details become available.
Sisense, a Third Party Service, is currently experiencing issues. Unified Analytics dashboards may not load for end users. Our teams are currently engaged and investigating. <b>What's impacted?</b> Unified Analytics Analytics <b>What’s not impacted?</b> Pipeline APIs Incident Sharing BigPanda Console We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Login Unavailable"
Last updateThis incident has been resolved, and users should be able to login to the console again. We apologize for any inconvenience that this may have caused. Should you continue to experience any accessibility issues, please reach out to us via our in-app chat or at support@bigpanda.io.
A fix has been implemented and users should be able to login to the console again. Please reach out to support@bigpanda.io if you continue to experience any accessibility issues. We will continue to monitor the service to ensure the service is working correctly.
Our team has identified the cause of the users database outage and are currently working on implementing a fix. We will provide periodic updates as we have more information.
The users database is currently unavailable. We are investigating the issue and will provide periodic updates as we work to bring the database back to service.
Report: "Login & lag issues"
Last updateThe incident has been resolved and all services are back to normal. However we will still be monitoring for a while to ensure it remains as such.
A fix has been implemented, majority people should start to see improvements in the service.
The issue has been identified and we are working on a resolution.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
Our R&D team is currently looking into the root cause and are working on a resolution. We have taken some steps to mitigate the issue. Having restarted a service, the impact may have caused login issues or extreme slowness when BigPanda. We are now just waiting for the restart to finish. We will keep you updated on the progress. <b>What's impacted?</b> • Login • UI responsiveness • Pipeline
Report: "BigPanda - Unified Analytics (Sisense)"
Last updateOur Unified Analytics services have returned to their normal operation. <b>Start time:</b> 11/29/2022 16:07 UTC <b>End time:</b> 11/29/2022 17:59 UTC <b>What was impacted?</b> Analytics (v2 & v3) We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
A fix has been implemented and we are currently monitoring for stability within our Analytics service. Any additional updates will be shared as more details become available.
Sisense has identified the root cause of the issue, and are in the process of applying mitigations to address the issue. Any additional updates will be shared as more details become available.
We continue to work with our third-party analytics provider (Sisense) to restore full functionality to our Unified Analytics service. Additional updates will be shared as more details become available.
Sisense, a Third Party Service, is currently experiencing issues. Unified Analytics dashboards may not load for end users. Our teams are currently engaged and investigating. <b>What's impacted?</b> Unified Analytics Analytics <b>What’s not impacted?</b> Pipeline APIs Incident Sharing BigPanda Console We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Activity Feed Issue"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>Start time:</b> September 29, 2022 14:16 UTC <b>End time:</b> September 29, 2022 14:38 UTC We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
The team has mitigated the issue. The activity feed should be functioning again. We will continue to monitor for any change in operational capacity.
Our team has identified the issue and have implemented a fix for the issue and are monitoring all connected services.
Some users may be experiencing issues with the activity feed spinning. Our teams are engaged and investigating. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "BigPanda - Analytics (v2 and v3)"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>Start time:</b> 09/28/2022 18:10 UTC <b>End time:</b> 09/29/2022 00:07 UTC <b>What was impacted?</b> Analytics (v2 & v3) We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Analytics v2 and v3 issues have been mitigated and is back to normal functionality. We will continue to monitor for any change in operational capacity.
AWS has identified the root cause of the issue, and are in the process of applying mitigations to address the issue. In the meantime, we are also working internally on mitigating the Analytics issue. Any additional updates will be shared as more details become available.
We continue to work with our third-party analytics provider to restore full functionality to our Analytics service. Additional updates will be shared as more details become available.
AWS service is currently experiencing issues. Our teams are engaged and investigating. <b>What's impacted?</b> Some users may be experiencing issues with Analytics (v2 and v3). No new Analytics data is being processed, though it is being queued in SQS. <b>What's not impacted?</b> BigPanda Console and event pipeline services are not impacted. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Issues accessing the UI."
Last updateOur team has resolved the incident and services have returned to their normal operation. Service was restarted and it's resources were increased. <b>Start time:</b> 13:13 UTC <b>End time:</b> 13:46 UTC <b>What was impacted?</b> Big Panda Console We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue and are monitoring all connected services. The issues affecting the BigPanda UI have been mitigated, and all BigPanda services are now operating within normal operational levels for all customers. We apologize for any inconvenience that this may have caused.
The issue has been identified and mitigation actions are underway. Users may still be experiencing issues in accessing their orgs. Additional updates will be shared as more details become available. We apologize for any inconvenience that this may be causing.
Some users may be experiencing issues with logging into the UI. Our teams are engaged and investigating. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Alert Enrichment Issue"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>Start time:</b> August 6, 2022 23:12 UTC <b>End time:</b> August 7, 2022 00:40 UTC We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity.
Our team has identified the root cause and are working on a resolution.
Some users may be experiencing issues with Alert Enrichment. Our teams are engaged and investigating. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Pipeline Latencies"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>Start time:</b> 04:25 UTC <b>End time:</b> 06:32 UTC <b>What was impacted?</b> Pipeline We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> Pipeline
Our team has identified the root cause and are working on a resolution. <b>What's impacted?</b> Pipeline
Some users may be experiencing Pipeline latencies. Our teams are engaged and investigating. <b>What's impacted?</b> Pipeline We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Pipeline Latencies"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>What was impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What was not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted. We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted.
Some users may be experiencing data pipeline latencies. Our team has identified the root cause and are working on a resolution. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Lags in inbound alert pipeline"
Last updateOur team has resolved the incident and services have returned to their normal operation. A change to our throttler service resulted in a temporary backup of alerts that has since been remediated. <b>Start time:</b> 4:40AM PDT <b>End time:</b> 6:10PDT <b>What was impacted?</b> Inbound pipeline delays which caused lags in the incident feed We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Inbound pipeline & alert processing
Our team has identified the root cause and are working on a resolution. <b>What's impacted?</b> Inbound pipeline
Some users may be experiencing lags in alert processing. Our teams are engaged and investigating. <b>What's impacted?</b> Inbound pipeline We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Out-of-the-Box ServiceNow and Jira Integrations"
Last updateOur integrations team has resolved the incident and services have returned to their normal operation. Backlog of updates have successfully been processed, and live share notification processing is back to normal operating levels. <b>What's impacted?</b> • Backlog of Incident Updates (Comments, Resolves, etc) • This should not delay any current/live share notification processing <b>What’s not impacted?</b> • Incident Sharing We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our integrations team is continuing to process backlog of updates. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Backlog of Incident Updates (Comments, Resolves, etc) • This should not delay any current/live share notification processing <b>What’s not impacted?</b> • Incident Sharing We apologize for any inconvenience that this may be causing.
Our integrations team is continuing to process backlog of updates. <b>What's impacted?</b> • Backlog of Incident Updates (Comments, Resolves, etc) • This should not delay any current/live share notification processing <b>What’s not impacted?</b> • Incident Sharing We apologize for any inconvenience that this may be causing.
Our team has implemented a fix for the issue. Backlog of updates are still being processed gradually. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Backlog of Incident Updates (Comments, Resolves, etc) • New shares might experience some delays processing updates <b>What’s not impacted?</b> • Incident Sharing Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Our integrations team continues to work on processing backlog of updates that failed to process. <b>What's impacted?</b> • Backlog of Incident Updates (Comments, Resolves, etc) • New shares might experience some delays processing updates <b>What’s not impacted?</b> • Incident Sharing We apologize for any inconvenience that this may be causing.
Our team has identified the root cause and deployed a fix. All updates (comments, resolves) that were not processed are being reprocessed. <b>What's impacted?</b> • Incident Updates (Comments, Resolves, etc) <b>What’s not impacted?</b> • Incident Sharing Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Some users may be experiencing issues with our OOTB ServiceNow and Jira integrations. Our teams are engaged and investigating. <b>What's impacted?</b> • Incident Updates (Comments, Resolves, etc) <b>What’s not impacted?</b> • Incident Sharing We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Incident Tags"
Last updateOur team has resolved the incident and services have returned to their normal operation. Incident Tags, including calculations against Incidents are be functioning again. <b>What was impacted?</b> Incident Tags - Users may have been unable to load Incident Tag configurations, and Incident Tags may have been calculated against Incidents <b>What was not impacted?</b> All other areas of the pipeline, including alert ingest, processing, console updates, and sharing were operational throughout this incident. We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
The team has reverted a recent change done to the service. Incident Tags, including calculations against Incidents should be functioning again. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> Incident Tags - Users may not be able to load Incident Tag configurations, and Incident Tags may not be calculated against Incidents <b>What’s not impacted?</b> All other areas of the pipeline, including alert ingest, processing, console updates, and sharing remain operational.
Some users may be experiencing issues with Incident Tags. Our teams are engaged and investigating. <b>What's impacted?</b> Incident Tags - Users may not be able to load Incident Tag configurations, and Incident Tags may not be calculated against Incidents <b>What’s not impacted?</b> All other areas of the pipeline, including alert ingest, processing, console updates, and sharing remain operational. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "[Console Issues] Incidents under maintenance being AutoShared"
Last updateOur team has resolved the incident and services have returned to their normal operation. We identified the rootcause to be related to the maintenance performed to our Outbound service and rolled back. Another maintenance window will be scheduled in the near future to complete the deployment. <b>Start time:</b> 12:30 UTC <b>End time:</b> 14:28 UTC <b>What's impacted?</b> • Incidents under maintenance <b>What’s not impacted?</b> • Login • Incident Feed • Incident Feed Search • Automatic Incident Triage • Root Cause Changes • Administrator Screens • Incident Actions • Incident Activity Feed • Unified Search • Analytics • Integration Diagnostics / Troubleshooting We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Incidents under maintenance <b>What’s not impacted?</b> • Login • Incident Feed • Incident Feed Search • Automatic Incident Triage • Root Cause Changes • Administrator Screens • Incident Actions • Incident Activity Feed • Unified Search • Analytics • Integration Diagnostics / Troubleshooting
Some users may be experiencing issues with incidents under maintenance being Autoshared. Our teams are engaged and working to resolve the issue. <b>What's impacted?</b> • Incidents under maintenance <b>What’s not impacted?</b> • Login • Incident Feed • Incident Feed Search • Automatic Incident Triage • Root Cause Changes • Administrator Screens • Incident Actions • Incident Activity Feed • Unified Search • Analytics • Integration Diagnostics / Troubleshooting We are working to mitigate impact to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing
Report: "ServiceNow Sharing Issues"
Last updateOur team has resolved the issue and shares to ServiceNow have returned to their normal operation. <b>Start time:</b> 21:15 UTC <b>End time:</b> 22:15 UTC <b>What was impacted?</b> • Incident Sharing We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has mitigated the issues and shares to ServiceNow should be operating normally again.
Some users may be experiencing issues with sharing to ServiceNow. Our teams are engaged and monitoring. <b>What's impacted?</b> • Incident Sharing We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.
Report: "Pipeline Latencies"
Last updateOur team has resolved the incident and services have returned to their normal operation. <b>Start time:</b> 17:55 UTC <b>End time:</b> 19:09 UTC <b>What was impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What was not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted. We apologize for any inconvenience that this may have caused. Should you continue to experience any issues, please reach out to us via our in-app chat or at support@bigpanda.io.
Our team has implemented a fix for the issue. We will continue to monitor for any change in operational capacity. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted.
Latencies have been mitigated for the majority of impacted customers, and the team is working on mitigation efforts for the few still experiencing latencies. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted.
Our team has identified the root cause and are working on a resolution. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted.
Some users may be experiencing data pipeline latencies. Our teams are engaged and investigating. <b>What's impacted?</b> • Data Pipeline - All data is being accepted into the BigPanda platform, but we are experiencing delays within pipeline services. Due to this, updates to users' Incident Feeds may be delayed. <b>What’s not impacted?</b> Other areas of the platform - including data ingest, analytics, incident sharing, user actions - should not be impacted. We are working to mitigate impacts to our customers as quickly as possible. Additional updates will be shared every 30 minutes or less as more details become available. We apologize for any inconvenience that this may be causing.