Historical record of incidents for Demandbase
Report: "slack notification delivery failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and fix will be pushed tomorrow morning.
We are aware of issues. It was due to a bad workspaceId lookup that was pushed to prod as part of last week's release.
We are currently investigating an issue with slack notification delivery.
Report: "slack notification delivery failures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and fix will be pushed tomorrow morning.
We are aware of issues. It was due to a bad workspaceId lookup that was pushed to prod as part of last week's release.
We are currently investigating an issue with slack notification delivery.
Report: "Login errors"
Last updateThis issue is resolved. A handful of users may be asked to reset their passwords before logging in.
The roll back was successful. We are following up with a handful of users who may still be experiencing issues.
For the remaining users who are still experiencing issues our authentication vendor will roll back the upgrade that is causing the issue tomorrow morning. We will post an update at that time.
Due to an issue with our authentication provider some users are unable to log in. We are investigating
Report: "Authentication issues"
Last updateThe fix has been applied and the issue has been resolved.
We are aware of authentication issues for some users. We have identified the issue and preparing a fix.
Report: "High Salesforce API usage"
Last updateA configuration change has been applied and has resolved the issue.
The issue has been found and a hotfix is being applied.
DB1 Marketing customers integrated with Salesforce may be experiencing a higher than normal volume of API calls. We are currently investigating
Report: "LinkedIn integration generating errors"
Last updateThe fix has been deployed and we will continue to watch to make sure the issue doesn't return.
The issue has been identified and we are working on a fix.
We are investigating an issue with our LinkedIn integration
Report: "We are currently experiencing DB1 application load time issues and are working to resolve the issue."
Last updateWe have identified the issue and applied the fix.
We are currently investigating this issue.
Report: "Demandbase one slow load times."
Last updateThis incident has been resolved.
We have implemented a fix and are monitoring the results.
We continue to experience performance issues but we have identified the issue and are working on a fix.
The application load times have returned to normal. We are still investigating the root cause.
We are currently experiencing DB1M application load time issues and are working to resolve the issue.
Report: "Investigating increased latency in DB1"
Last updateWe have solved the issue and the performance should be returning to normal.
We are aware of increased load times in DB1 and the team is investigating.
Report: "Blank snapshot emails"
Last updateThis incident has been resolved.
We are aware of an issue where some customers received blank snapshot emails today. We will resend these once the issue is resolved.
Report: "Overnight Data Processing Delay"
Last updateThis incident is resolved and the overnight data processing is complete for the majority of tenants. However, some tenants were not refreshed today and will begin seeing up to date data tomorrow morning.
Some DB1 tenants may be be experiencing a delay in overnight data proceessing. Our engineering team has resolved the issue and process is running.
Report: "Overnight Data Processing Delay"
Last updateThis incident has been resolved.
The majority of tenants have been refreshed. We will continue to monitor the remainder that are still finishing up.
We expect the remaining tenants to be up to date in the next 4 to 6 hours
Some DB1 tenants may be be experiencing a delay in overnight data proceessing. Our engineering team has identified the issue and is working on a resolution..
Report: "web.demandbase.com is experiencing errors"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "DB1 errors"
Last updateThis incident has been resolved.
We are aware of an issue with DB1 and believe it has been resolved.
Report: "Subscription email delay"
Last updateOur email vendor fixed the problem and we've resubmitted the subscription emails. They should go out in the next couple of hours.
Our email provider is experiencing delays in delivering subscription emails.
Report: "tenant configuration issues"
Last updateThis incident has been resolved.
We have identified the issue and are working on a fix for the affected tenants.
We are investigating reports of configuration issues in some of our customer tenants.
Report: "web.demandbase.com is experiencing errors"
Last updateFix has been applied.
We are continuing to work on a fix for this issue.
We have identified the issue and we are working on the fix.
We are currently investigating this issue.
Report: "CSV Data Import Issues"
Last updateThe fix has been applied and we have confirmed the imports are working again.
We have identified an issue affecting CSV Data Imports, which may result in a server error for some users. A fix for this issue is underway. We apologize for any inconvenience that may cause and appreciate your patience. We expect to have a resolution in the next 2 hours. We will provide updates as needed.
Report: "Blank dashboards for some customers"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Several customers have reported blank dashboards in their tenants. We have identified the issue and working on the fix.
Report: "Authentication errors in Salesforce integration"
Last updateWe have identified a work around. This issue was introduced in a recent Chrome update. To revert the policy change in Chrome disable 3rd party story partitioning. The setting can be found by putting chrome://flags/#third-party-storage-partitioning into the Chrome address bar.
We are investigating authentication errors with customers who have the Demandbase ABX package installed in Salesforce.
Report: "Delay in refreshing data"
Last updateWe confirmed the issue is resolved. Today's refresh was successful.
Todays refresh completed except for a small handful of tenants. We are expecting the next refresh to catch those remaining tenants and be up to date by 6am EST.
We have identified the issue and we are running the job to refresh the data now.
Description: We are currently addressing a performance issue that has resulted in a delay in refreshing data for approximately 50% of our customer tenants. The data regeneration process, which typically occurs every 24 hours, is experiencing delays, preventing the update of information from yesterday. Impact: You may notice that the data in your tenant has not been updated with the latest information from the previous day. We understand the importance of timely and accurate data for your operations and apologize for any inconvenience this may cause. Actions Taken: Our engineering team is actively investigating the root cause of the performance problem and working on a resolution. We are prioritizing this issue to ensure a swift resolution and minimize any further impact on your daily operations. What You Can Do: We appreciate your patience and understanding during this time. If you have any urgent queries or specific concerns related to your data, please reach out to our support team via the support portal https://supportportal.demandbase.com. We understand the importance of the data in our platform to your business, and we are committed to resolving this incident as quickly as possible. We sincerely apologize for any inconvenience this may cause and appreciate your continued trust in Demandbase.
Report: "Activity Duplication"
Last updateOn Wednesday, July 19th, we identified an issue resulting in some activity duplication. Our team promptly deployed a fix on July 20th, and cleanup is underway, set to finish by Sunday, July 23rd. No need to worry; you likely didn't notice anything. If historical journey stages are of concern, a partial rebuild is available as an option by contacting Support. Thank you for your understanding. Reach out to support@demandbase.com if you have any concerns.
Report: "Demandbase Sales Cloud login issues"
Last updateService has been restored to all impacted tenants. Thank you for your patience.
We have mitigated the issue and are currently monitoring the status of the impacted service. We will post an update once service has been restored to all impacted tenants. Thank you for your patience.
We are currently investigating the issue.
Report: "DB1 login issues"
Last updateWe have resolved the issue.
We have identified the issue and implementing the fix.
We are currently investigating this issue
Report: "[Resolved] DB One Platform UI is displaying "Oops error" page due to increased errors on backend. We are currently investigating the issue and post updates shortly."
Last updateWe have resolved the issue through a rollback process. We will be conducting detailed post-mortem on prevent the issue from recurring.
We have isolated the issue to a repeated calls made by an UI component to the backend service and we are in the process of issuing a rollback.
What: DB One Platform UI is displaying "Oops error" page due to increased errors on backend. We are currently investigating the issue and post updates shortly. Impact: All Tenants using the DB1 platform ETA: Team is investigating. We will be posting periodic updates
Report: "ABX Cloud & Advertising Cloud are undergoing emergency maintenance"
Last updateWe have resolved the issue with backend system and the maintenance page has been removed. The platform should be fully functional now. We will be conducting a post-mortem to prevent future incidents with similar failure mode.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
What: ABX Cloud & Advertising Cloud are undergoing emergency maintenance since the website is down. Users will see a maintenance page until the team isolate the issue. Impact: All Tenants ETA: We are still investigating the root cause.
Report: "[Resolved] Lead to Account Matching is not functioning for AS 2.0 customers due to code change in this week's release."
Last updateWe have applied the fix to production and lead to account matching should be fully functional again. We will be conducting a post-mortem to prevent such issues from recurring.
The issue has been isolated to handling of null values for the migrated tenants. We are currently testing a fix and will be rolled into production shortly.
What: We are investigating an issue in lead to account matching module where there was code change to metadata that is resulting in parsing errors. Impact: This issue impacts functioning of lead to account matching service for customers that been migrated to analytics service 2.0 ETA for Resolution: Team is actively investigating and we will providing periodic updates
Report: "[Resolved] Some of our AS 2.0 customers are experiencing inconsistent account targeting from their account list in their advertising campaigns."
Last updateWe applied a fix for this issue and confirmed that all account lists used by advertising. We will be conducting a post-mortem to prevent such issues in future.
We are continuing to work on a fix for this issue.
We have isolated the inconsistency in advertising campaign reporting and currently working on a hotfix to address the issue.
What: We are investigating an issue in AS 2.0 where some of advertising campaign targeting have cases where we are targeting less accounts than what is in an account list. In some cases we are not targeting any accounts. Impact: This issue impacts about third of our AS 2.0 customers. ETA for Resolution: Team is actively investigating and we will providing updates.
Report: "Sales Intelligence Cloud users are reporting errors with exports"
Last updateThe issue has been resolved.
We are able to reproduce the issue and we are working on a fix.
We are continuing to investigate this issue.
We are investigating reports that there are issues with exporting in Sales Intelligence Cloud.
Report: "Authentication issue for legacy InsideView customers"
Last updateThis incident has been resolved.
The fix has been confirmed in production.
We are testing a fix for the remaining issue and plan to release it shortly.
We have solved the original issue. We have identified an additional issue we are working on solving now.
We have identified the issue and it is currently limited to a handful of users in a single account.
Legacy InsideView customers who are also Demandbase customers are experiencing issues logging in. We are investigating the issue.
Report: "[Identified] AS 2.0 customers are currently experiencing "Site Analytics -> Accounts table" is not populating data."
Last updateWe have mitigated this issue. We will be conducting a post-mortem to prevent future occurrences.
We have isolated the issue of blank accounts table in site analytics page in AS 2.0 to calls to a new field for person.id getting added. We are currently executing the mitigation path.
What: We are investigating an issue in AS 2.0 where some of the data is not populating due to a backend update. Impact: This issue impacts all tenants that have been migrated to analytics service 2.0. ETA for Resolution: Team is actively investigating and we will providing updates.
Report: "[Resolved] AS 2.0 customers will not see person name being displayed in UI."
Last updateThis incident has been resolved successfully. We will be conducting a post-mortem on this issue to prevent it from happening in future.
The team has isolated the issue to a backend change. We have triggered the mitigation plan.
What: We are investigating an issue where a backend data is missing a set of tables which is causing the person names for AS 2.0 tenants will not be displayed. Impact: This issue impacts all tenants that have been migrated to analytics service 2.0. ETA for Resolution: Team is actively investigating and we will providing further updates.
Report: "Advertising is incorrectly reporting clicks"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue that is causing clicks and click-based metrics to be over-reporting.
Report: "Sales Cloud is not loading"
Last updateThe root cause was found and the issue is fixed.
We have put in a temporary fix that has brought the applications back up. We are still working on fixing the root cause.
We are currently investigating the issue.
Report: "Updates to Journey edits on DB1 are locked for several customers"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
The issue has been resolved by a hotfix release.
The fix has been deployed and we are monitoring the results.
We have identified the issue and testing a fix.
Some customers are experiencing issues making Journey updates. We are investigating.
Report: "Data Integrity Analyitcs Dashbaord"
Last updateThe issue has been fixed. The team will continue working on backfilling the missing data.
Patch to fix the issue has been applied.
Customer dashboards are missing data.
Report: "Syncing errors for some customers"
Last updateThis incident has been resolved.
For a handful of customers marketing automation and Salesforce syncing is experiencing issues.
Report: "[Resolved] DB One Platform UI is displaying "Oops error" page due to increased errors on backend"
Last updateThe backend issue has been resolved and the platform is back to being functional. We will be conducting a post-mortem on this incident.
DB One Platform UI is displaying "Oops error" page due to increased errors on backend. We are currently investigating the issue and post updates shortly.
Report: "[Resolved] Sales Intelligence Cloud: Small subset of customers are unable to update existing contacts in CRM or export /create new contacts"
Last updateThis issue is now resolved by applying an emergency hotfix. Customer should be able to update the contact information. We will be conducting a post-mortem on how to prevent such issues in future.
We are continuing to work on a fix for this issue.
We have identified the cause of not being able to update the contact information in Sales Cloud. We are actively working on a mitigation fix in the next hour or so.
We are currently investigating an issue with updating existing contact information in CRM and/or create new contacts. A small subset of customers are facing this issue.
Report: "[Resolved] DB1 platform is missing advertising data for Demandbase tenant"
Last updateThis issue is now resolved. Campaign data is visible again. We will be conducting a detailed root cause analysis.
We have isolated the missing campaign data in Demandbase tenant and are currently taking steps to remediate the issue.
We are currently investigating an issue related to campaign data missing in DB One platform for Demandbase tenant.
Report: "[Issue Resolved] DB One Platform UI is displaying blank page after login for some tenants"
Last updateWe have isolated this issue to be a front end component throwing error. We have mitigated the problem by an UI rollback. We will be conducting a post-mortem on this issue.
In DB One platform some tenants are seeing a blank page after login. We are currently investigating
Report: "[Issue Resolved] DB One Platform experiencing increased error rate and we are currently investigating"
Last updateWe have resolved the issue related to backend errors. The system is fully functional now. We will be conducting a post-mortem on this incident.
We have identified and isolated the underlying issue and currently working on mitigation
We are currently experiencing increased error rate in the backend systems that we are investigating. We will be posting updates as we make progress
Report: "DB Classic Platform is experiencing connectivity issues which is impacting targeting functionality of campaigns"
Last updateThis incident has been resolved and we have identified the connectivity issue to our backend database. We will be doing a root cause analysis on this issue.
We are currently investigating connectivity issues in DB Classic platform which is impacting creating/editing functionality of campaigns
Report: "Demandbase One platform is experiencing login issue and team is investigating"
Last updateThe issue is now resolved. We will be conducting a post mortem on this issue.
Demandbase One platform is experiencing login issue
Report: "Demandbase One platform is currently undergoing maintenance"
Last updateThis incident has been resolved.
Maintenance is now complete. The site is operational
Demandbase One platform is currently undergoing maintenance
Report: "Demandbase One Platform experiencing slow web logins on Feb 1st"
Last updateThis incident has been resolved. We will be conducting a detailed post-mortem on this incident.
We are experiencing a recurrence of slowness in web logins and increased stale connection caches. We are currently investigating.
Report: "Emergency Maintenance Downtime: You will see a maintenance page when instead of login"
Last updateThis incident has been resolved. We will be conducting a detailed post-mortem on this incident.
We are taking an emergency maintenance downtime
Report: "Incident Summary: Demandbase One Platform experiencing slow web logins on Jan 31 7:30am PT"
Last updateWe have isolated and resolved the root cause of the issue causing the platform to respond slowly. We are currently working on post-mortem and will be able to shared additional details shortly.
We are taking an emergency downtime to mitigate the underlying issue causing the slowness.
We are actively looking into a login issue in our Demandbase one platform
Report: "We are experiencing increased network latency in some regions of our infrastructure"
Last updateWe have isolated the problem to cached connections in our caching layer and the issue has been mitigated.
We are currently experiencing increase network errors in our infrastructure region. We are working with AWS team to isolate the issue. The impact might be that you are experiencing slower login times into the Demandbase platform.
Report: "AWS us-east-1 availability zone 4 experienced a power failure that impacted Demandbase One, Demandbase Classic, Insideview and Ads platform"
Last updateAll Demandbase platforms are fully functional now. We will continue working on post-mortem and learning from this incident.
A fix has been implemented and we are monitoring the results.
We have fully recovered from this incident in our Sales Intelligence Cloud and now running sanity checks
We are continuing to investigate this issue.
We are closely working with our AWS partner team. Below are the updates - We are investigating increased EC2 launch failures and networking connectivity issues for some instances in a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Other Availability Zones within the US-EAST-1 Region are not affected by this issue. 05:18 AM PST: [NDA update] We continue to make progress in restoring power to the affected data center within the affected Availability Zone (USE1-AZ4) in the US-EAST-1 Region. We have now restored power to the majority of instances and networking devices within the affected data center and are starting to see some early signs of recovery. Customers experiencing connectivity or instance availability issues within the affected Availability Zone, should start to see some recovery as power is restored to the affected data center. RunInstances API error rates are returning to normal levels and we are working to recover affected EC2 instances and EBS volumes. While we would expect continued improvement over the coming hour, we would still recommend failing away from the Availability Zone if you are able to do so to mitigate this issue. 05:39 AM PST [NDA update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. Network connectivity within the affected Availability Zone has also returned to normal levels. While all services are starting to see meaningful recovery, services which were hosting endpoints within the affected data center - such as single-AZ RDS databases, ElastiCache, etc. - would have seen impact during the event, but are starting to see recovery now. Given the level of recovery, if you have not yet failed away from the affected Availability Zone, you should be starting to see recovery at this stage. 5:55 AM PST [NDA Update] We have now restored power to all instances and network devices within the affected data center and are seeing recovery for the majority of EC2 instances and EBS volumes within the affected Availability Zone. We continue to make progress in recovering the remaining EC2 instances and EBS volumes within the affected Availability Zone. If you are able to relaunch affected EC2 instances within the affected Availability Zone, that may help to speed up recovery. We have a small number of affected EBS volumes that are still experiencing degraded IO performance that we are working to recover. Given that some hardware lost power, there is some risk of impaired hardware, which may delay recovery for a very small number of EC2 instances, which we will message to affected customers if that happens.
Report: "DB One Platform experiencing outage across Demandbase One, DB Classic and Sales Cloud platform as there is backend infrastructure degradation"
Last updateThis incident has been resolved. We will be working with AWS on understanding the root cause of yesterday's network issues. All platforms are functional.
Update: AWS have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. They are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. AWS will continue to work toward full recovery for all impacted AWS Services and API operations.
Update: AWS have executed multiple shifts of internal service traffic to a separate set of network devices. They are now beginning to see significant recovery for AWS service availability in the US-EAST-1 Region. We are continuing to closely monitor the health and to continue identifying the specific traffic flows that were leading to congestion within these devices. This is complex, as the issue is not simply the volume of traffic, but rather the rate of new connections that is causing unexpectedly large load on the impacted devices. We expect to see continued improvement in service availability and are monitoring for full recovery.
Update: AWS have identified that an increase in traffic between specific internal services is causing congestion within several network devices in the US-EAST-1 Region. While we continue to work across every service to investigate and mitigate impact, we are focusing our efforts on a specific mitigation for this traffic increase. We are taking action over the next 30-60 minutes that we expect will put us on the path to recovery, however we do not have an ETA for full recovery yet.
Update: We are working closely with our infrastructure vendor AWS on resolving the widespread network outage in us-east-1. While we are still seeing incremental recovery, it is not consistent and we do not have an ETA for full recovery. All teams are actively engaged and working on resolving this impact as quickly as possible.
We are currently experiencing service degradation as well as partial outage in our infrastructure (us-east-1) and this impacts users of Demandbase One, DB Classic and Sales Cloud platform. Multiple infrastructure services are impacted and we are currently working with our infrastructure partner to get resolution to this outage. We are attempting multiple paths to mitigating this issue, and have all teams engaged. Unfortunately at this time we do not have an ETA to when this issue will be mitigated