Historical record of incidents for NextRoll
Report: "Unable to edit Facebook campaign"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Issue: - Facebook campaigns are currently experiencing an issue where they cannot be edited. Functional Impact: - Unable to edit Facebook campaigns Next Steps: - Engineering is currently addressing the issue. Comms will be sent once this is complete.
Report: "[Sev2 - RollWorks Only] LinkedIn audiences sync issues"
Last updateThis incident has been resolved.
ISSUE: The RollWorks/LinkedIn integration is experiencing issues when syncing audiences. IMPACT: Under the LinkedIn Settings, customers may see “LinkedIn integration is experiencing some technical issues“ with its connected ad accounts showing a "Disconnected" status. NEXT STEPS: Engineering is currently addressing the issue. Comms will be sent once this is complete.
Report: "Unable to choose ads for new and existing RollWorks web campaigns"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "[Sev2 - RollWorks Only] Ingestion issue with Salesforce Contact and Lead objects"
Last updateThis incident has been resolved.
Issue: There is currently an ingestion issue with the Salesforce Contact and Lead objects, which affects their use as data sources for filtering in Contact Lists and Contact Discovery Workflows. Functional Impact: - Sync errors with existing Contact Lists and Contact Discovery Workflows that use the Salesforce Contact and Lead objects as data sources. - Unable to create new Contact Lists and Contact Discovery Workflows using the Salesforce Contact and Lead objects as data sources. Next Steps: The engineering team is currently investigating the issue.
Report: ""Create Dynamic Ads" button is currently down"
Last updateThis incident has been resolved.
The 'Create Dynamics Ads' button via the 'Create' dropdown in the Ads Library is currently down. The button is leading to a form to report the error. The behaviour is currently visible to users with the Create Dynamic Ads functionality enabled. Our engineering team is actively working on a fix.
Report: "[Sev 2] Unable to sync Account Lists to LinkedIn"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
Issue: There is currently an issue with Account Lists, where audiences cannot sync with LinkedIn. Impact: - Unable to push Account Lists' audiences to LinkedIn. - In some instances, the LinkedIn Ad Accounts settings might indicate that the integration has been disconnected. Next Steps: Engineering teams have identified the cause of the issue, and are working toward resolution.
Report: "Scheduled reports for AdRoll and RollWorks are not being sent."
Last updateThis incident has been resolved.
Impact: - Customers that have scheduled reports for recurring delivery (daily, weekly, monthly) are not receiving their reports. - On-demand reports are continuing to function correctly. Next Steps: - Engineering teams have identified the cause of the issue, and are working toward resolution. - When resolved, all queued weekly and monthly reports will be sent. Queued daily reports will not be sent, as this would result in delivering several reports at once to customer inboxes. Instead the next daily report will resume as normal on February 4th, 2025.
Report: "[Sev1 - RollWorks Only] Command Center, Journey Stage, and Salesforce data source unavailable"
Last updateUpdate: - The Salesforce data source is now fixed and available for Journey Stages, Account Lists, and Enhanced Contact Lists.
Updates: - There are no longer any missing sections of the dashboard - Issues related to the Salesforce data source filters, specifically with Account List (created from Command Center), Enhanced Contact Lists, or Journey Stage using Salesforce Accounts, Opportunity, and Campaign objects, are still under investigation
Issue: RollWorks is currently experiencing two issues: - The Command Center and Journey Stage are missing from the Dashboard - Salesforce data source is unavailable for Account List Enhanced Contact List, and Journey Stage. Impact: - Unable to access Command Center and Journey Stage - Unable to create/modify Account Lists, Enhanced Contacts Lists, and Journey Stage using Salesforce data source filters Next Steps: Engineering has resolved the issue with access to the Command Center and Journey Stage but is currently looking into the issue with Salesforce data source, which is affecting certain areas of the Command Center and Journey Stage.
Report: "RollWorks app not loading"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Most areas in the RollWorks app are currently not loading.
Report: "The playbook edit page is currently unavailable in the dashboard."
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Scheduled app.adroll.com Maintenance"
Last updateThe maintenance has been completed. All system services have been restored.
We will be undergoing planned maintenance on Thursday 2024-05-23 from 6:00 PM until 7:30PM US PDT (UTC-7). During this time, the dashboard and all functionality on app.adroll.com as well as the API (services.adroll.com) will be unavailable. All other systems including ad serving and pixel serving will be functioning as normal during this time. If you have any questions about this outage, please reach out to your NextRoll representative or contact us at support@nextroll.com.
Report: "Incorrect "Pixel Connection Lost" warning message"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Audience sync issues with RollWorks/LinkedIn integration"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Ads API hard deleting data denoting relationships between AdGroups and Ads"
Last updateThe recovery of data denoting the relationship between AdGroups and their Ads is completed.
A fix for the issue has been deployed to production. In addition, a recovery procedure has also been defined, and we estimate to have a majority of the data denoting the relationship between AdGroups and their Ads restored around Tuesday/Wednesday next week.
Issue: Our Ad APIs are having issues where data denoting the relationship between AdGroups and their Ads are being hard deleted from Web Retargeting campaigns' AdGroups instead of soft-deleting them. Impact: Affects Web Retargeting campaign details and reports page where deleted Ads under their parent AdGroups are no longer shown. Scheduled reports are also missing deleted Ad metrics. Next Steps: A fix for this will be deployed in the next couple of hours. Engineering is currently evaluating possible strategies to recover Ad data that has been hard-deleted.
Report: "Degraded experience when creating or editing campaigns"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "AdRoll Email campaign delivery and some Web Retargeting campaigns in Automation are currently experiencing an outage."
Last updateThis incident has been resolved.
Our engineering team is currently investigating an outage that affecting AdRoll Email campaign delivery and some Web Retargeting campaigns in Automation.
Report: "RollWorks data loading issues"
Last updateThis incident has been resolved.
Various areas in the RollWorks app may have trouble fully displaying data. The issue has been identified and a fix is being implemented.
Report: "Granular Conversion Report (GCR) not being generated and sent"
Last updateThis incident has been resolved.
The issue has been identified.
Issue: Granular Conversion Report (GCR) is not being generated and sent for both scheduled & on-demand due to an email decryption issue. Functional Impact: Unable to generate and send GCRs. Next Steps: Our engineering teams are working to resolve the issue.
Report: "LinkedIn Campaigns integration outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
LinkedIn Campaigns integration is currently experiencing an outage. With this, all of our LinkedIn campaigns have stopped spending. Our engineering teams are working directly with LinkedIn to resolve the issue from their side.
Report: "Ads service loading issues"
Last updateThis incident has been resolved.
AdRoll and RollWorks are currently experiencing load issues for areas that rely on the Ads backend services.
Report: "Delayed segmentation of untrackable converting cookies"
Last updateThis incident has been resolved.
Note: Conversions from untrackable cookies are not surfaced in attributed conversions nor website audiences. The only impact is to general site metrics conversion figures – and the AdRoll Cross-Channel dashboard. No data has been lost. Untrackable conversions should resume regular processing by this time tomorrow, and a backfill will be conducted next week for unprocessed timestamps.
Report: "d.adroll is experiencing sporadic connectivity outages."
Last updateA fix was made to resolve the issue, and budgets should spend as normal for the rest of the day UTC.
We are investigating the issue.
Report: "Parts of the dashboard are experiencing a downgrade in service for some users."
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
The app.adroll.com dashboard page is currently experiencing a downgrade in service where some pages are inaccessible. The engineering team is currently investigating the issue.
Report: "Temporary Universal Campaign (Voltron) outage affected a few customers"
Last updateEarlier today, the system used to set up Universal/Automated/Proxy/Cross-channel campaigns & Playbooks (Voltron) – had a temporary outage. This system recovered after that. Universal campaign creation and updates to related strategies during that short window would have been absent from the UI. Underlying inventory campaigns and other non-Voltron objects were not impacted, and the related database is being backfilled.
Report: "Theres a delay in processing spend data for Tuesday May 9th."
Last updateThe issue has been resolved and all backfill data is now complete and available in the dashboard.
The issue has been identified and a fix is being implemented.
Our engineering team has identified the issue and is currently working on a solution to resolve the matter and restore data.
Report: "adroll.com marketing site is currently experiencing an outage."
Last updateThis incident has been resolved.
Our engineering team has resolved the issue and is currently monitoring the site for any residual issues.
We are continuing to investigate this issue.
The adroll.com marketing domain is currently down. All other systems like the front dash (app.adroll.com), the developer portal (developers.nextroll.com), nextroll.com and rollworks.com are unaffected and working as normal. Our engineering team is aware of the outage and is actively working on resolving it.
Report: "SMS service issue for Two Factor Authentication (TFA)"
Last updateThis incident has been resolved.
Issue: Our SMS service for Two Factor Authentication (TFA), is currently experiencing an incident which impacts their SMS service. Impact: SMS could arrive with some delays or might not arrive at all. Next Steps: Engineering will monitor the status of the incident.
Issue: Our SMS service for Two Factor Authentication (TFA), is currently experiencing an incident which impacts their SMS service. Impact: SMS could arrive with some delays or might not arrive at all. Next Steps: Engineering will monitor the status of the incident.
Report: "LinkedIn audiences not creating or updating in RollWorks"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue where LinkedIn audiences are not creating or updating in RollWorks.
Report: "Degraded service with our third party subscription management system"
Last updateHey all, this incident has been resolved.
Our third party subscription management tool is experiencing a degradation in their service which is causing intermittent issues with account sign ups and our billing and subscription systems. A fix has been pushed and the third party is monitoring the progress of the fix.
Report: "[Sev2 RollWorks] Interruption with receiving daily update of G2 data"
Last updateThis incident has been resolved.
As of July 5th, all RollWorks customers who are integrated with G2 are currently not receiving the normal daily update of data as G2 has experienced an issue over the past weekend. G2 will be backfilling the missing data and is treating this issue with priority.
Report: "Transactional emails are not sending due to system outage"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Mailgun is having a major outage and is affecting the ability to send emails. We will be monitoring the Mailgun outage, and we will follow up as soon as we have additional info.
Report: "Users are unable to access any web campaign edit buyflow"
Last updateThis issue has been resolved. Users should now be able to edit web campaigns without receiving an error.
Upon attempting to edit any web campaign users will see the "Oops, page not found" screen. Social campaigns are unaffected and still able to be edited.
Report: "[AdRoll Only] Cross-Channel Dashboard is currently unavailable"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Dashboard is not loading on certain tabs"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "The dashboard was unable to load due to a third party's API failures"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Report: "A small subset of customers are unable to load the homepage dashboard."
Last updateThis incident has been resolved.
The issue has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
A small subset of customers are experiencing an issue where they're unable to access the app.adroll.com dashboard. Our engineering team is working on a fix for the affected users. Please note that no data has been lost and any new users will not face this issue.
Report: "There are currently issues impacting app.adroll.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
Engineering is aware of the issue and are currently investigating. We will follow-up once we know more.
Report: "AdRoll BU reports that span more than a month are experiencing delays."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Inability to display some RollWorks Reporting data (RollWorks Only)"
Last updateWe’re happy to report that Journey Stages has been taken out of maintenance. Many thanks to the RollWorks Engineers who worked tirelessly to get this product back up and working for everybody! This will be the last communication about this escalation. Thank you.
As a follow-up to Friday’s email, we wanted to share an update on the Journey Stages outage, which is related to some underlying ClickHouse issues. We have found and fixed the root cause for one issue which was due to a bug in ClickHouse itself. That was patched over the weekend, and investigation is continuing on the root cause of one other issue. We have made some infrastructure changes that we believe will better position us against future issues, and we are testing these changes in production now. As a reminder, no data has been lost. Customer access in the production UI has been limited out of an abundance of caution while engineering finishes testing the back-end changes we have made.
Engineering is continuing to work towards a resolution on Journey Stages. While the issue continues to be ongoing the page will remain in maintenance mode. The expected resolution for this issue is likely to be on or around this Wednesday, August 18, 2021. We will continue to update you as new information becomes available
The issues we’ve been experiencing with Journey Stages data specifically have unfortunately returned. As a result the Journey Stages page has been put into maintenance mode so any user navigating to that page will see a "We are under maintenance" screen. We will be following-up later today with more info regarding this maintenance.
This is just a quick update that June & May Journey Stages data is now available in the dashboard. April's Journey Stages data backfill is in progress and we will provide an update once this has been complete.
In efforts to immediately allow users to load Journey Stages in the most recent days, data is being backfilled starting from the most recent months. With that note, data for August & July is now available for Journey Stages. June will be shortly available, followed by May and so forth. For any date that is unavailable you will simply see a "?" and a warning "unable to get accounts".
RollWorks reporting pages related to Playbooks, New Influenced Opportunities and Account Card features will now fully load. The Journey Stages page is still affected and we are currently working on its resolution.
RollWorks reporting related to Playbooks, Journey Stages, New Influenced Opportunities and Account Card features are all affected. Pages that display this data will now only partially load (omitting the related reporting data). We're working on a fix.
Report: "Subscription management vendor outage impacting new AdRoll customer signs-ups and subscriptions"
Last updateThis incident has been resolved.
Paid subscription actions are unavailable on app.adroll.com.
Report: "Facebook ads service interruption due to the global Facebook API outage."
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Unable to show Journey Stages beyond Oct 11"
Last updateThis issue is now resolved. This impacted a subset of customers with Salesforce stages. Engineering has fixed the issue for Oct 12 & 13. However, customers won’t be able to select it for a couple of hours until the system refreshes.
The Journey Stages page is currently unable to show stages data for October 12th & 13th. Engineering is actively looking into the issue.
Report: "Updating the accounts in the TAL is failing due to database issues"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Engineering will need to modify the TAL database. A 15 minute downtime will be necessary at 10:00am PST while we address this issue. During this downtime, users will be unable to view or edit TAL information.
Report: "Delay in reporting metrics for October 18th"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
There are delays with our Unified Attribution Pipeline, and as a result reporting metrics for October 18th are absent for products across both AdRoll & RollWorks. The responsible engineering team is aware and working on a fix.
Report: "Exporting of data to Marketo, Hubspot, and Salesforce is delayed."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Some users may experience 403 errors logging into app.adroll.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Some users with TFA (two-factor authentication) enabled may be unable to login to their account. This is related to cookies set for a now deprecated analytics service (Hotjar). This results in a 403 error page. Please, clear browser cache and cookies and load a new window. The issue should no longer replicate after this step.
Report: "Cross-channel upper funnel campaign issues (web only)"
Last updateWe have released the fix this morning and this issue has been resolved.
The expected fix will go out sometime next week. We'll provide more updates early next week. Thank you!
We are continuing to work on a fix for this issue.
We are continuing to work on fixing this issue. We'll have an update on Monday after the holiday weekend.
We are continuing to work on a fix for this issue. Thank you!
We are continuing to work on a fix for this issue. We'll have another update on Monday morning. Thank you!
Cross-channel upper funnel campaign is having issues. Engineering team have identified a fix and are working on it.
Report: "Negatively targeted geotargets are not being correctly saved."
Last updateWe have released the fix this morning and this issue has been resolved.
Negatively targeted geotargets are not being correctly saved. Engineering is looking into the issue.
Report: "Campaign dashboards are not showing data"
Last updateAdRoll campaign dashboards were not showing data. However, the issue has been identified and resolved.
Report: "A delay in conversions reporting data."
Last updateWe're currently experiencing a delay in conversion reporting data in the dash for AdRoll and Rollworks BU campaigns. The delay started around February 2nd. The Engineering team has fixed the issue and is working on a process to backfill the conversions data in the dash.