Is Vero Down Right Now? Check if there is a current outage ongoing.

Newsletter sending has resumed normal operations. However, we will continue to monitor through the weekend. Please reach out to support@getvero.com if you experience additional delays or have any questions.

We have identified the root cause of the issue and have applied a fix. We will continue monitoring throughout the day until sending returns to normal operations.

We are experiencing significant delays with Newsletter sending. The issue began overnight, and we are taking steps to mitigate but service will take some time to return to normal. Once we are positive we have addressed the root cause of the issue, we will provide an update and monitor the issue.

On Friday, May 30, customers experienced significant delays in the delivery of one-to-many newsletter campaigns. We know how critical timely delivery is and are providing a post mortem. **What happened** Between 0700 30 May - 0130 31 May UTC, scheduled newsletter campaigns were delayed due to a combination of factors: 1. **Downstream service disruption**. A configuration issue at one of our external partners impacted message throughput. 2. **Authentication issues.** A component of our infrastructure experienced a disruption related to service authentication with another internal service, which contributed to the backlog. These issues together resulted in prolonged delivery times for a subset of campaign traffic. **Resolution** We worked with our external partner to address the configuration issue and restored full functionality to the affected internal service. Message queues were cleared and delivery returned to normal by 0130 31 May. **Preventative and detective measures** To improve reliability and reduce risk going forward, we are: * Enhancing proactive monitoring and alerting for system-level authentication and connectivity health. * with our third-party partner to strengthen deployment safeguards. -- We’re committed to ensuring the dependability of your campaigns. While this incident was complex and multi-faceted, we take full responsibility and apologise for the inconvenience. Thank you for your trust and patience. Please email us at [support@getvero.com](mailto:support@getvero.com) if you have any questions.

Newsletter sending has resumed normal operations. However, we will continue to monitor through the weekend. Please reach out to support@getvero.com if you experience additional delays or have any questions.

We have identified the root cause of the issue and have applied a fix. We will continue monitoring throughout the day until sending returns to normal operations.

We are experiencing significant delays with Newsletter sending. The issue began overnight, and we are taking steps to mitigate but service will take some time to return to normal. Once we are positive we have addressed the root cause of the issue, we will provide an update and monitor the issue.

All stats should now be caught up. We have monitored overnight and observed no additional issues.

This issue has been identified and patched. We are currently monitoring to ensure the issue has been fully addressed. In the meanwhile, delivery stats are being backfilled but will take some hours before they are fully accurate.

We are receiving multiple reports that the "Delivery" metric in the Vero UI and integrations via webhooks are currently under-reporting.We have identified the cause of this issue. At this time it appears it relates to reporting only: messages are still being successfully delivered.We will provide an update as soon as we have deployed a fix and are monitoring the solution.Please email support@getvero.com with any questions.

All stats should now be caught up. We have monitored overnight and observed no additional issues.

This issue has been identified and patched. We are currently monitoring to ensure the issue has been fully addressed. In the meanwhile, delivery stats are being backfilled but will take some hours before they are fully accurate.

We are receiving multiple reports that the "Delivery" metric in the Vero UI and integrations via webhooks are currently under-reporting. We have identified the cause of this issue. At this time it appears it relates to reporting only: messages are still being successfully delivered. We will provide an update as soon as we have deployed a fix and are monitoring the solution. Please email support@getvero.com with any questions.

This incident has been resolved.

We have identified an issue introduced during a recent upgrade that affected data processing for newsletters sent through Vero 2.0. On 16 March 06:26 UTC we made a change to increase the send rate of newsletters sent through Vero 2.0. Due to an issue in this deploy, some email profiles have since been incorrectly identified, leading to: 1. Missing report data for some campaigns. 2. Emails sent to previously-unsubscribed users. We’ve deployed a fix and are currently back-filling the data. From this time, unsubscribed users *are* being correctly respected. Additionally, we’re working to recover the missing reporting stats and correct any previously-impacted profiles. This process is ongoing, with full restoration expected within the next 24 hours. We’ll provide further updates as the process progresses. We understand the impact of this bug and apologise for the inconvenience caused. Please email via support@getvero.com with any questions.

This incident has been resolved.

We are currently investigating an issue in which the main Customers page is not displaying all customer data.

The root cause of the issue has been identified, and an appropriate configuration change has been made. Service appears to be restored, and we are marking this issue as resolved. We will continue to monitor but do not anticipate additional issues. Please reach out to support@getvero.com with any questions or concerns.

We are currently experiencing an API outage. Our platform team is investigating but has yet to identify a root cause.

This incident has been resolved.

We have now processed 98% of the backlog and are working through the final 2%. We will leave this issue as "Monitoring" until we've fully cleared the backlog.

We have been monitoring the situation since identifying and addressing the issue earlier today. We are still working through the backlog of webhooks and will continue to monitor until we are back in real time.

We have identified the cause of the issue.

We are currently investigating this issue.

The incident related to delays in webhook processing has resulted in delays to campaign reporting in Vero’s UI. No data has been lost and all reports will be backfilled.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are still investigating. We apologise for the delay. We will provide further updates as soon as we can.

We're investigating reports of delays processing API events and triggering automated campaigns.

This incident has been resolved.

We are currently investigating this issue.

We've now resolved this issue. We will reach out to the affected customers to discuss any follow-up questions. If you have any questions, please email us at support@getvero.com.

One of our email delivery processing clusters is experiencing delays, affecting a portion of our customers. This is impacting automated (transactional, behavioural, workflow) campaigns, resulting in significant delays. We have found the root cause and are working on a fix now.

This incident has been resolved.

We are currently monitoring delays in newsletter sending.

At 11:17 UTC (1 April), our team was alerted to delays in automated campaign processing. At this time we marked behavioural, transactional and workflow processing as degraded. This affected all customers. By 11:54 UTC, our team had isolated the root cause: delayed filter evaluations on one of our data sources. At 12:30 UTC, processing performance had returned to normal but there was a large backlog of automated campaign jobs to work through. We continued to monitor as the backlog reduced and began work on a fix to the root cause. At 18:49 UTC, we noticed a spike in processing again. This was quickly resolved (as of 19:30). We continued to monitor the backlog. As of 21:30 UTC, the backlog has cleared and workflow processing is 100% realtime for all but handful of customers (with larger backlogs). These remaining backlogs should be fully cleared by 00:00 UTC. At 21:56 we deployed a patch for the root cause. We believe we have fixed the underlying cause and this will not occur again. We will monitor extra closely over the coming 48 hours to ensure this is the case. If you have any questions, please email support@getvero.com. Thank you. We also want to note that this incident was unrelated to the database upgrades/maintenance conducted over the weekend. These changes have gone smoothly. -- Processing has returned to normal and we can report the following metrics for the key time windows. These are the P50 to P90 reports for workflow node processing. This represents how long it takes to process a node vs. when the node was queued for evaluation. We expect a p90 of 5 minutes across all nodes. Note that this include **all node types** across all workflow types (transactional, non-transactional, etc.). For example, A node with 50 conditions looking back across a large date range will take longer to evaluate than a simple email node. 10:00 - 21:30 UTC 1 April (catching up on backlog) • p50 = 1 hour • p75 = 2.45 hours • p90 = 4.6 hours 21:30 - 00:00 UTC 1 April (tail of backlog, with majority of customers in realtime) • p50 = 48 minutes • p75 = 1.8 hours • p90 = 3.1 hours 00:00 - 01:00 UTC 2 April (backlog fully processed and processing in realtime) • p50 = 31 seconds • p75 = 3 minutes • p90 = 5.4 minutes

We believe we have addressed the underlying configuration issues when led to last week's major service degradation. Over the past few months we've been working on some important optimizations and security patches to several of our key data stores and infrastructure. These changes were meant to be rolling changes with no production impact and many of the changes have been. Unfortunately, some of the changes we made led to unanticipated issues, causing degraded performance across the majority of Vero's services. One of the database shards on which we store and read user properties has been falling out of sync with the primary data store. Unfortunately, the services that rely on this particular data source include those that personalize email content and evaluate automated campaigns and workflows. These services will wait for eventual consistency but have a low tolerance for delays in order to deliver the speed our customers expect. This has caused periodic, significant delays. We've been able to uncover the source of these issues and have made configuration changes that have resolved the synchronization lag. We are monitoring to ensure there are no further issues. Separately, we recently made some important optimizations to our event storage to improve performance. Despite significant planning, these changes led to unexpected degredation in some of the queries related to automated email delays. Our platform team has been hard at work optimizing these queries and we've seen major improvements. Processing is back to BAU at this time. We are monitoring and are continuing to make further optimizations until we've seen the net performance improvements we're targeting. It has taken us longer than we would like to roll forward and adjust the configuration to return performance to the needed levels. We have been monitoring and have seen all services operating as expected. We will continue monitoring until we are certain we have restored the systems to their full capacity.

We are continuing to monitor the situation and make adjustments to improve performance. We will share a post mortem when we are satisfied that the situation is resolved

We apologize for the service degradation experienced intermittently throughout this last week. These have been caused by some large, necessary infrastructure improvements we are making to our system. The primary changes have now been made and we expect to see improvements to any remaining performance issues over the next several hours. We will continue to monitor the situation over the weekend, and provide a more detailed post-mortem afterwards. Thanks for your patience. If you have any questions, please emails us via support@getvero.com.

This incident has been resolved.

Between ~7am UTC Monday and ~10pm UTC Monday the Vero system experienced periodic outages/delays affecting: • Reports in the UI • Automated/workflow campaign evaluation speed. • API ingestion. • The UI itself. We know these outages have a major impact on your end-user experience and we apologise for the inconvenience caused. -- For those interested, at ~10pm UTC Sunday, we conducted some unplanned maintenance to one of our core data indexes. Whilst unplanned, we made the required changes after planning the alternatives and thinking through the risks. The new configuration is designed to be more performant, running on more modern architecture. This should have been a very run-of-the-mill, invisible upgrade. It took some time to become apparent but the new configuration was not performing to specification. By 7am UTC Monday this was leading to delays in several services. Throughout Monday (UTC) our team actively managed the situation to prioritise API and automated email processing was prioritised (as these are the most critical systems). As of ~10pm UTC Monday, all services expect reports were returned to normal processing speeds. As of ~1am UTC Tuesday, reports have also returned to normal processing. We will be conducting a post-mortem internally to learn and better plan for future changes to this specific service. -- At this time we are continuing to monitor the situation. We will mark this issue as resolved once we are comfortable things have been operating as normal for ~24 hours. -- We work hard to ensure 99.99%+ uptime on all core campaign processing. All changes we have been making to our infrastructure recently are in the service of: • Faster automated/workflow campaign processing. • Delivery of new channels such as SMS (and beyond). Thank you for your support. If you have any questions please email us at support@getvero.com.

We've encountered performance issues resulting in degraded performance across the product. API processing, segment calculation, email sending, and the UI have all been impacted. No data has been lost. At 7:15AM Monday UTC, we were alerted to an unusually high number of unprocessed API jobs. These jobs eventually came back down after intervention by our platform team. Throughout the morning, the API queue has seen multiple additional spikes, and we've seen degraded performance across the entirety of the application. We are continuing to investigate the cause and will update when we have determined a course of action.

After monitoring for 24 hours, we are confident this issue has been resolved. Segment calculations have returned to normal processing, and newsletters are going out without further issue.

We have been monitoring and the system has been stable for the last six+ hours. We'll continue to monitor for a full 12 hours before marking this incident as resolved. Thanks again for your patience and support.

As of 2:50am Tuesday UTC all newsletters and segments have now caught up and are processing in real time. We are monitoring the changes we've made to ensure there are no regressions. We will continue to monitor over the next several hours and will post further updates as required. Once we are comfortable everything is stable we will mark this incident as resolved.

**At the present time all newsletters and UI-triggered segment calculations are processing at a significantly reduced rate.** Many newsletters are delayed by several hours. We've found the root cause. We acknowledge the impact this has and we are working as hard as we can to resolve this issue. Further background: • At 10pm Saturday UTC we were alerted to processing delays in segments and newsletters for a small number of customers. These were reviewed and worked through individually. • At 7am Sunday UTC we were alerted to delays for a larger portion of customers. In light of this development, our platform team began a deeper review. Affected segments and newsletters were worked through by ~Sunday 2pm UTC and we were not of the view there was a systemic issue. Changes we made appeared to have resolved the issue. • At 4pm Monday UTC we began noticing a new increase in delays. • By 7pm Monday UTC we had uncovered the root cause of this issue. • By 10pm Monday UTC we had taken action to reduce processing speeds and posted an official incident. As of 2am Tuesday UTC (the present time) we are still processing newsletters at a reduced rate. We will post an update once we have a clear estimate of when newsletter sending will be back to normal rates. If you have any questions, please email us via support@getvero.com.

We've identified an issue resulting in poor performance for segment calculations. It was occurring sporadically over the last 48 hours, however we now understand it to be systemic. Our platform team is working to identify the root cause, and currently investigating several possibilities. In the interim, manual segment refreshes (via the UI) have been disabled to prioritize queries from newsletters. Workflows and automations are unaffected.

This incident has been resolved.

Currently working to fix an outage causing 503 errors loading the Vero Connect UI

Our upstream provider has reported that they are continuing to monitor their mitigation efforts. Over the last 24 hours, we've seen a return to normal sending and no substantial newsletter delays. We'll continue to monitor the situation, and provide future updates should we begin to experience delays again.

While we continue to experience some intermittent issues with our upstream provider, we've seen a significant improvement in processing speed since yesterday morning. Our upstream provider has reported that they are continuing to investigate and monitor the situation, and we will be monitoring as well.

At approximately 15:00 UTC, we noticed an increasing number of connection errors with an upstream provider resulting in serious email delays for newsletters. This provider has since provided a status update that they have "Implemented DDoS protection and monitoring results", but we are continuing to see intermittent connection errors and have reached out to the upstream provider to gather more details. In the interim, expect long delays on newsletter sending in both our Cloud and Connect products.

Last week, beginning ~10:00 25 October UTC and ending around 18:00 26 October UTC \(a period of around 48 hours\), we experienced several periods of significant API and email processing delays. This instability was caused by an inefficient query pattern behind one of the conditions used in workflows. This inefficiency was revealed based on a usage pattern we had not seen before. Once identified as the root cause, we took steps to mitigate this issue and introduce guardrails to prevent it arising in the future. We monitored for several days to ensure that all was fully resolved. We have not seen the issue since and consider the issue resolved. We are sorry for any inconvenience caused. If you have any follow up questions please contact us at [support@getvero.com](mailto:support@getvero.com).

This incident has been resolved. We will follow up with additional details soon but confirming all API requests and emails have been processing in real-time for the past 24 hours. In the meantime, please don't hesitate to email support@getvero.com if you have any urgent questions about your account.

Over the last 48 hours (since ~10:00 25 October UTC) we've experienced several periods of significant email processing delays. Yesterday and today (26 October UTC) our operations and engineering team have taken multiple steps to mitigate these issues. As a result of these measures we have seen improvements to processing transactional emails (as evident in the data below). We are still investigating the root cause of these issues and will provide a post mortem once we are confident all problems have been resolved. Until this time it is possible that further delays may occur. We will continue to investigate. To help users troubleshoot we are sharing average delay times during the affected periods. All times below are in UTC. For transactional automated messages, **average** delays were as follows: Tuesday 25 October • 10:00 - 16:00 = 90 mins Wed 26 October • 09:00 - 18:00 = 22 minutes For behavioural automated messages and workflows, **average** delays were as follows: Tuesday 25 October • 10:00 - 16:00 = 160 minutes • 19:00 - 11:59 = 110 minutes Wed 26 October • 00:00 - 2:00 = 110 minutes • 9:00 - 18:00 = 4 hours If you have any questions please contact us at support@getvero.com.

Over the last 30 hours we've been experiencing intermittent delays affecting API processing and resulting in transactional, behavioural, workflow and newsletter delays. We've been updating each of the components that have been affected as they occur. As of right now API processing and emails including behaviourals, workflows and newsletters are real time. Unfortunately transactional emails are currently seeing processing delays and we're working through a backlog of statistics, logs and webhooks. Throughout this time we've been focused on identifying the root cause and are working hard to ensure all processing remains real time. We'll be posting updates as we receive them and will follow up with a detailed post-mortem once the issue is resolved. If you have any questions about your account please email support@getvero.com

We experienced an elevated level of API errors between 22:50-23:00 UTC. During this time API requests were not accepted. This is now resolved and all API requests are being accepted and processed. Please reach out to support@getvero.com if you have questions.

This incident has been resolved.

Mailgun has updated that a fix has been implemented, and email sending is returning to normal. We are monitoring results.

At 18:25 UTC, we received notification from our Email Service Provider, Mailgun, that they are experiencing an issue impacting their API as well as Outbound Delivery. If you send via Mailgun, a custom Vero domain, or a Vero-controlled domain, you may experience delays with email delivery. We'll continue to monitor their status page for details and will update here. You can follow along at https://status.mailgun.com/incidents/9djjsl006cf6 or follow here for updates.

Overnight, shortly before 8:00AM UTC, our internal monitoring system alerted us to issues with our API processing. Upon investigation we realized that API processing, email processing, and several other services \(including access to the UI\) were being impacted. Investigation revealed that an unusual pattern of heavy API traffic was the cause. Actions were taken to mitigate the issue, and we were able to restore functionality to our services, and begin processing the backlog. At approximately 1:57PM UTC, the backlog had been fully processed and all API requests and emails began to be processed in real time. We'll be looking into how to optimize and guard against this particular pattern of API traffic in the future.

This incident has been resolved. All API requests and emails are being processed in real time. If you experience any further issues please email support@getvero.com and we will investigate ASAP.

API requests are now processing 🎉. All services are now operational and we are working to process a backlog of requests. Once all data is processing real time we will resolve the issue and follow up with additional details. As always, please don't hesitate to email support@getvero.com if you have any questions

Emails are now processing. We expect there will still be some delays in sending emails whilst we process the current backlog. If you have any questions regarding your campaigns please don't hesitate to email support@getvero.com.

The UI is now accessible! We've also identified the root cause and should be able to process API requests and emails shortly.

We're currently seeing delays in processing API requests and emails as well as loading the UI. Our team has been working to mitigate these issues and we will provide further updates as soon as possible. Please don't hesitate to contact support@getvero.com if you have any questions.

This incident has been resolved.

We are currently seeing delays in event and email processing. Although we have scaled up additional capacity we're still investigating the root cause. No events or emails have been lost.

On Monday 16th of May at 9am UTC we experienced performance degradation in our event stream processing service. As we process events at a significant volume this most notably led to delays in processing emails and statistics whilst we restored performance. Once the issue was solved it took some time to process the backlog which completed on Tuesday 17th of May at 8:12am UTC.

All has been processing as expected for several hours and we're now confident the issue is resolved. We'll follow up shortly with a full post-mortem.

All services are now operational and we are working to process a backlog of requests. It's possible we will still experience some intermittent delays over the next 24 hours, as we continue to solve the root cause. Until then we'll keep components marked as degraded. Our support team will be reaching out to customers whose newsletters have been affected. Please email support@getvero.com if you have any questions regarding your account.

We are currently seeing a degradation in performance which has predominantly affected automated email processing. Further impacted systems that have seen intermittent delays are calculating segments, processing newsletters as well as the campaign statistics, logs and webhooks that result from sending emails. Our team has been working to mitigate these issues and we will provide further updates shortly - we will ofcourse write a full post-mortem once all is resolved. Please don't hesitate to contact support@getvero.com if you have any questions.

We're consistently working to evolve Vero's underlying infrastructure to make the platform more efficient and secure. At the end of March we made some changes to the core database powering "event property" conditions. As a result of this change a series of maintenance tasks failed between March 26th and March 31st. As a result, at March 31st at 4:27am UTC we began observing delays with newsletter and segment calculations that relied on the event data property condition. We subsequently diagnosed the root cause of the database performance degradation and fixed and reinstated the maintenance operations. As a significant amount of data had accumulated during the intervening period we had to balance running new processing for segment calculations and the maintenance operations which needed to catch up on the data that had not been processed since the incident first occurred. We saw some gradual improvements as the maintenance progressed and, since this process fully completed \(April 11th, 2:35pm UTC\), we have seen segment calculations return to nominal performance. As part of the \(unrelated\) scheduled maintenance operations that also took place over the weekend of April 10th and 11th we were able to make changes to the processing system that will prevent this occurring again. We are adding further monitoring to ensure any future issues would be detected more promptly. If you have any questions, please contact us at [support@getvero.com](mailto:support@getvero.com).

Segments with the condition "has triggered event with property" are now calculating in the expected time. This means any related newsletters will also be sending as expected. We'll be following up with a detailed post mortem shortly. In the meantime please contact support@getvero.com if you have any questions related to your account.

Maintenance operations to optimise segment calculation speed have completed over the weekend and we are monitoring the results.

Maintenance operations to optimise segment calculation speed are continuing to run. Additionally the system maintenance we will complete this weekend will contribute to improvements. Only the condition "has triggered event with property" is affected with all other segment conditions running efficiently. If you utilise this condition for a newsletter you may see some delay, we'll reach out if this is the case and can provide more information regarding your account. We'll follow up on Monday with an update on the status of this issue and will ofcourse follow up with a detailed postmortem once this is resolved. If you have any questions about your segments or newsletters please email support@getvero.com

We've identified the root cause of the performance degradation and we are running maintenance operations to resolve the issue. This maintenance operation will run for the remainder of the week. We will provide another update on progress within 24 hours. In the meantime we've been taking mitigating actions to ensure that affected segments and newsletters continue to process, though you may still experience delays. Please email support@getvero.com if you have any questions regarding your account, segments or newsletters.

On Thursday 31st March at 4:27am UTC we began to experience delays in calculating segments that reference the condition "has triggered event with property". This is a result of a performance degradation on the database responsible for segmentation of events. If your newsletter references a segment that queries for the condition "has triggered event with property" it may be delayed whilst we perform maintenance on this database. We will provide further updates as soon as they become available. Note that automated messages (Workflows, Behavioural and Transactional campaigns) remain **unaffected** and are **operating as normal**. Please email support@getvero.com if you have any questions regarding your account, segments or newsletters.

All user property data is up to date for segmentation! Please email support@getvero.com if you have any questions related to your account.

On February 15th at 08:57:01 (GMT+11) we deployed an update to the data platform used to calculate large segment queries as part of routine upgrades we have been making to support our security posture. Unfortunately these upgrades introduced an error that resulted in new and updated user property values not being updated in this data platform. This was ongoing from Feb 15 at 08:57am (GMT+11) to Feb 17 at 04:58am (GMT+11). During this window, segments continued to refresh without issues but customers whose segments utilise this data platform queried potentially stale data. We have identified and fixed the issue and are in the process of fully backfilling all user property updates, this should be complete within 10-24 hours. No data has been lost but refreshed segments may still be inaccurate until the backfill process has been completed. If you have any questions about your account please email support@getvero.com

We can confirm this incident is now resolved. If you have any questions about the impact to your account please email support@getvero.com.

The previously mentioned fix has resolved the issue. The team is continuing to work to recover missed email event logs.

The team has identified the root cause and implemented a fix. Newly generated email logs will appear in the UI. We are currently working on recovering logs that were missed since 2022-02-02 21:30 UTC.

We're currently unable to record delivered, opened and clicked email results. This means that in both your logs and reports, emails will look sent but not delivered. We can confirm emails are currently delivering and we are working to ensure all email results are up to date in the UI. This will affect AB tests, segments and filters referencing email results. If you have any questions please email support@getvero.com.

We have completed a thorough review of all campaigns and can confirm this incident is now resolved. If you have any questions about the impact to your account please email support@getvero.com.

Yesterday at 04:50 2022-01-20 UTC we deployed an update to our content editor. This update included improvements to the "Snippets" feature of the Drag and Drop editor. As a result of this change an error was introduced resulting in two potential outcomes: 1. Some content using Snippets experienced formatting issues. This meant emails were sent incorrectly rendered. 2. Some content built with our legacy editor could not render. This mean that these emails did not send (and will not retry). We identified and solved this issue at 12:38am 2022-01-21 UTC. The error was live for up to 20 hours. We are currently conducting a post-mortem analysis. Based on the work we have done so far it appears that, fortunately, this has affected a minor number of campaigns, with 99% of customers unaffected. We are continuing this analysis and, once we are certain of the results, will close this issue and email customers with the details of the affected campaigns. All campaign content and emails are now sending as expected and we're confident this issue is fully resolved going forward without any lingering affects. If you have any questions about the impact to your account please email support@getvero.com.

A fix has been implemented and content is rendering as expected. We'll be following up with further details of the issue and impact.

We are currently investigating an issue with rendering the content of emails. If you have any questions about your account please email support@getvero.com and we will follow up.

This incident has been resolved.

API performance and error rates have continued to be at normal levels over the past few hours. We are still awaiting confirmation of final resolution from our upstream service provider but in the meantime we will continue to monitor for further issues.

We're still waiting on a resolution from our upstream service provider, but thanks to their mitigation efforts we've seen a return to typical API performance and error rates in the past hour. We will continue to monitor for further issues.

We've seen some improvements in API performance and error rates however we are still impacted at this time, our upstream service provider is working on a fix

We've identified an issue with an upstream service provider and will update again when we have further information on a fix

We're experiencing an elevated level of API errors and are currently looking into the issue.

This incident has been resolved, we are continuing to monitor for further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented

We are currently seeing delays in event and email processing and we're investigating. No events or emails have been lost.

This incident has been resolved, all services are real time and operating as normal. If you have questions or want to discuss potential or observed impact on your account please email us at support@getvero.com.

At 02:00 UTC today we identified processing delays with some of our workers. After several hours of investigation and intermittent reliability issues we made the decision to pause our workers temporarily in order to determine the root cause. As a result we were able to pinpoint the cause: a regression in a code deploy made earlier today causing worker configuration issues and resulting in workers consistently failing before completing their jobs. Once fixed we were able to redeploy our worker fleet with services resuming at 10:12 UTC. Since this time we have been working through the backlog of accumulated API calls and automations for the Vero Workflows product. Please see a summary of the affected service timeline below. No data was lost during this incident. **Vero Newsletters** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters. 07:50 UTC - 10:00 UTC: not able to send newsletters. **Vero Workflows** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters (within Vero Workflows). 07:50 UTC - 10:00 UTC: - Not able to send newsletters. - No automated processing. - No API call processing (though all data was safely collected). 10:00+ UTC: working through API and automation backlog (API real time as of 9.15pm). -- We will continue to monitor and will resolve this incident once all services are real time and operating as normal. If you have any questions, please email us at support@getvero.com. Thank you for your patience!

We are continuing to monitor for further issues but we anticipate that our system backlog should be processed in approximately another hour, we will update our status components back to Operational as those workloads are real-time again.

A fix has been implemented and we are currently monitoring. We expect there will be delays in processing emails whilst we process a backlog of requests over the next few hours.

The issue has been identified and a fix is being implemented

We are currently experiencing delays in processing emails which our team is actively working to resolve. This includes both newsletters and now automated campaigns. If you have questions or want to discuss potential or observed impact on your account please email us at support@getvero.com.

We are continuing to investigate this issue - newsletters are sending but noting we are seeing queuing delays of 30+ minutes

We are currently investigating delays in queuing of newsletter campaigns

This incident has been resolved.

We are continuing to monitor for further issues and are also performing related maintenance activities.

The backlog of API and campaign requests have been cleared and processing times have now returned to normal. We are continuing to monitor for further issues and are also performing related maintenance activities.

We have recovered the affected services and are processing through the backlog. We estimate this will take around three hours, at which time all API processing should return to normal. We will post an update once we confirm these timelines and/or if there are further updates. Whilst we are processing the backlog there will be delays to processing Workflows and other automated campaigns.

We have identified an issue with a database that has failed and we are currently working our service provider to have our services resumed as soon as possible.

We're experiencing an elevated level of API errors and are currently looking into the issue.

Changes to content built with the drag and drop editor are successfully saving again 🎉. We will follow up with details on what occurred. If you have any questions or concerns related to campaign content edited between 7am-11:56am UTC on August 11th please email support@getvero.com.

We have identified an issue where changes to content built in the drag and drop editor can not successfully save. This would only affect changes made since 7am UTC (August 11th). We believe this to be related to a recent update released to the editor and are taking steps to resolve it.

This incident has been resolved.

We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

Over the past 12 hours we have monitored our systems. As a result we know that there is data missing for user properties updated on Sunday 27 June UTC and are currently restoring the user property data from that period. This only affects segmentation as it relates to sending newsletters. It does NOT affect content. User property updates made **before** ~00:00 Sunday 27 June are complete and accurate. User property updates made **since** ~14:00 UTC Sunday 27 June are also complete and accurate. If you update user properties regularly (e.g. daily) then this is likely to have minimal or no impact on your newsletters. If you imported specific data on Sunday 27 June UTC and are using it for segmentation in a newsletter then this data will not yet be available. If you have any concerns we recommend you hold off scheduling newsletters and/or reach out to us. We're happy to discuss options via support@getvero.com. This update may take 24 hours or more. We will leave this incident up until the import is complete. **Important:** Workflows use a different system for loading user properties and all user property filters are complete and accurate in Workflow processing. This only impacts user properties updated on Sunday 27 June for use in segmentation for newsletters. Thanks again for your patience.

We have now fully restored the affected data store. Newsletter processing has been reenabled. Newsletters that were affected over the last several hours will now be processed as we push through the backlog. We anticipate there will be some delays whilst this occurs. There will be some ongoing work as we do further background syncing and monitor to ensure everything is returned to a stable state. We'll post a further update when we are comfortable this is the case, at which point we'll resolve this incident. Thank you for your patience. If you have any questions, please email us via support@getvero.com.

We have made progress with our service provider and are currently restoring the affected data store. We estimate this restore will take another ~3 hours. We will provide an update when the restore is complete and newsletters are processing as normal. We have emailed affected customers. We also want to confirm that API tracking and Workflow processing have remained unaffected and are processing normally.

We have been alerted that one of the databases powering segment calculation has failed. This has affected a small portion of customers over the last six hours. This is an issue with our service provider. We are working with our service provider to restore service and will post an update when that occurs. We will also investigate why we were not alerted earlier. If you have any questions, please email us at support@getvero.com.

At 10:45 UTC 12 June we received alerts that some customers' newsletters were delayed. Our team responded, made adjustments and reviewed our systems. Reporting confirmed that newsletters were sending as normal. At 16:00 UTC we were again alerted to delays in newsletter processing. Our team re-investigated and found that one of our database clusters was no longer performant. This issue affected our ability to calculate segments and schedule newsletters, leading to delays for the majority of newsletters scheduled between 10:45 UTC and 17:30 UTC. Our database was affected as a result of regular maintenance processes we run. Unfortunately this issue both affected scheduling and the regular health checks that report on newsletter delays. Since 18:00 UTC, newsletters have been sending as normal. Data collectiong, workflows and automated messages were **not** affected by this incident. Based on what we have observed from this incident we will make changes to our maintenance processes and health checks. If you have any questions, please conact us at [support@getvero.com](mailto:support@getvero.com). Thank you.

Newsletters have been sending as normal since approximately 18:00 UTC 12 June.

We are now processing newsletters again as normal but have a large backlog to work through. This will take us some time. We will continue to work through this backlog and monitor this issue until we're comfortable everything has caught up. We will update this incident to "Resolved" at that time (or post further updates if there are any other changes in state).

We believe we have identified the cause of the newsletter sending delays. We are currently working through a backlog of delayed sends.

We have observed delays in sending newsletters and are investigating the root cause of this issue.

Our email delivery partner has reported the issue has been resolved.

We are continuing to monitor for any further issues.

We're currently seeing some delays due to a downstream issue our email partner is experiencing. This will affect both your email previews as well as email delays. Please email support@getvero.com if you have questions about your campaigns.

At 12:31 UTC 17 April we were alerted to intermittent issues with Vero's UI. By 01:57 UTC we had identified the root cause: a recent deploy to the search cluster powering elements of our UI had led to unforeseen load on that cluster, causing it to fail. We rolled the changes back and, unfortunately, had to initiate a full recovery on the cluster involved. UI access was restored at approximately 05:30 UTC 17 April, with significant delays in the Logs and Customers functionality whilst the restore continued. The recovery process completed at approximately 09:45 UTC 17 April. During the period 12:31-09:45 UTC, **only** UI access was interrupted. Backend processing **continued as normal**, with no effect on newsletters, workflows or other functions. We are continuing to monitor to ensure the new cluster is performing as expected and there are no further interruptions. If you have any further questions, please email us at [support@getvero.com](mailto:support@getvero.com).

This incident has now been resolved. We are continuing to monitor for any further issues.

We are continuing to monitor for any further issues.

All parts of the Vero UI have been reliably available again over the past hour and we will continue to monitor. There will be some delays to accessing recent campaign statistics and logs data in certain sections of the Vero UI but we expect any backlog to be processed over the next few hours and will update our status components accordingly.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue

We are have been experiencing intermittent issues causing parts of Vero's UI to be slow or unresponsive, we are currently working to resolve the issue.

Conversion results have been updating in realtime for the last 48 hours and we can confirm this incident is now resolved! 🎉

Messages, logs, reports and webhooks continue to be processed in realtime. We are still processing the backlog of conversion statistics which we anticipate will be complete within 6-12 hours. If you have any questions about your account please email us at support@getvero.com.

All messages, logs, reports and webhooks are processing realtime. Conversion statistics will be updating in realtime within the next 12 hours. We'll continue to monitor and provide updates as they occur.

We are still processing at a reduced rate. Reports and Logs are particularly affected, as are webhooks. We've updated the components below to reflect this. Message throughput has been affected periodically since 1:30pm UTC (Wednesday) and we have been updating the affected components to reflect this as it occurs. We are making progress on this issue and apologise for the extended impact.

On Tuesday 23rd March at approximately 6:30pm UTC we observed delays in processing automated messages (behavioural, transactional and workflows). We identified an issue with our round-robin system. We debugged and fixed the root cause by approximately 9.30pm UTC. During the period 6:30pm-9.30PM UTC automated messages were operating at a severely reduced throughput, resulting in a backlog of automated messages. The backlog of automated messages was fully processed and running in realtime by 11pm UTC. Newsletters sent between 9.30pm-12:30am UTC (Wednesday) were sent at a reduced rate whilst this backlog was processed, resulting in some delays. We've been posting updates to the various system components on this status page over the last 12 hours to reflect the above delays as they occurred. -- Since 12:30am UTC we have identified a second issue with some of the raw hardware provided by our cloud provider. **It appears this is a hardware issue and is not related to the earlier incident**. Messages **are** currently sending in realtime but we are operating at an overall reduced capacity with some delays in aggregating conversion metrics and sending webhooks. We're working with our cloud provider to resolve this issue, at which time we'll be able to scale to full capacity. In the interim we're monitoring and will post updates if the situation changes and message send speeds are affected. If you have any questions, please email us at support@getvero.com.

We're experiencing an elevated level of API errors and are currently looking into the issue.

We're experiencing an elevated level of API errors and are currently looking into the issue.

This incident has been resolved.

Our API error rates have decreased and we are continuing to monitor.

We're experiencing an elevated level of API errors and are currently looking into the issue.

We can confirm that all statistics are now indexed in the UI and this issue is resolved.

Confirming that newsletters did in fact send yesterday, however we're seeing instances where campaign results were not able to display in the UI. We're working to ensure these are updated ASAP. If you have any questions please email support@getvero.com

Segments and newsletters are now processing real time. We'll continue to monitor. Please email support@getvero.com if you have any questions about your account.

We believe the issue has been resolved now and we are processing the backlog of segment queries and newsletters.

We are currently investigating an issue affecting the calculation of segments which will mean your newsletter may experience delays if attempting to send.

At 03:30 UTC today (Thursday) we deployed the full release of our new drag-and-drop editor and campaign editing workflow. This is a huge change at all layers of our application. Despite weeks of rigorous QA, we ran into two minor edge cases today: 1. *Inability to send automated emails that used a template*. Automated messages that used a template (any template) could not find their email content and were failing to send. We manually fixed campaigns as we identified them and fully resolved this issue at approximately 08:00 UTC today. Any affected sends were retried and no emails were sent with incorrect content. 2. *Issues rendering old, `vero-editable` templates*. Our old editor didn't support `vero-editable` tags _within_ `vero-editable` tags. Our new release is more stringent in enforcing this, causing templates with nested `vero-editable` tags to render strangely. This issue was resolved around 10:30 UTC today. No campaign content was lost or altered *unless* a campaign was opened, viewed in its "garbled" form and saved during the 03:30-10:30 UTC window today. If this occurred for you, please email us and let us know. We will continue to resolve any bugs related to the new release if and when they are reported. We are resolving this issue as there are no longer any issues with email rendering or sending. Thanks and we hope you enjoy the new editor and workflow! 🎨

Your campaign content is now rendering as expected in the UI 🥳. If you saved your campaign between 3:45am-11:20am UTC the content may be cached in the UI. Please contact support@getvero.com and we can manually refresh them. We'll be following up with further details on what occurred.

Emails that were previously being discarded are now sending as expected. We should have further updates on editing content shortly.

We're investigating reports that some email templates are rendering differently than expected. If this is the case you should see these emails being discarded instead of sent. This ensure's that users will not receive incorrect content. Please email support@getvero.com if you have any questions regarding your account or emails.

This incident is ultimately more of an advisory. Our SSL certificate was cross-signed with the AddTrust (legacy) root certificate. This certificate expired on 30 May 2020 in line with Sentigo's notice: https://sectigo.com/resource-library/sectigos-addtrust-root-is-soon-to-expire-what-you-need-to-know We did not anticipate this would impact any customer systems, in line with Sectigo's advice: "After this date, clients and browsers will chain back to the modern roots that the older AddTrust was used to cross sign. No errors will be displayed on any updated, newer device or platform which has had updates." In response to some customer reports, we have now updated our SSL certificate: "For business processes that depend on very old systems, Sectigo has made available a new legacy root for cross-signing, the 'AAA Certificate Services' root." There should be no further issues across any of our endpoints or platforms. If you have any questions, please email us on support@getvero.com

We have been alerted that there are reports of connection issues to our API and we are investigating.

On Wednesday 27th May at approximately 10am UTC we observed intermittent delays in API processing and, from approximately 12pm, delays in workflow and behavioural email processing. These delays were not fully resoled until 4pm UTC. From that time, the queues were processing normally. We have made several changes in response to the delays seen on the 26th and 27th May and are confident that delays in email or API processing should not occur again. We monitored these queues through Thursday 27 May and will continue to do so. Both API and email status components were updated to reflect the delays as they occurred. If you have any questions, please email us at support@getvero.com.

We are sharing this post mortem to provide an easy-to-navigate record of our prior component updates this week. On Tuesday 26th May at around 12pm UTC we began observing intermittent delays in email processing and statistics collection. Upon investigation, this appeared to be related to some infrastructure changes we had recently deployed. Our Operations Team was able to manage many of these delays as they occurred, but message queues for **some** customers were delayed on and off between 12pm UTC and 6pm UTC. From 6pm processing had returned to real time. In response to these delays, we reviewed and adjusted the previously deployed infrastructure changes. We have updated the status of all relevant components in real time throughout the week. If you have any questions, please email us at support@getvero.com.

At 17:18 UTC 18 May 2020, one of the databases backing our API began experiencing issues. Between 17:18 and 17:21 a small number of API requests were dropped (approximately 800 total). We responded to the issue and between 17:21 and 17:40 we were able to rectify the core problem. By 17:47 we were processing at normal capacity. It took approximately one hour to work through the backlog of API requests that had accumulated. Triggered messages were delayed during this period. No data was lost. Aside from the ~800 dropped requests, no data or emails were lost. This issue was caused by a regression introduced with a recent infrastructure change. We are reviewing our tests and QA moving forward. Thanks. If you have any questions, please write to us at support@getvero.com — we're happy to answer them.

At 17:32 UTC yesterday (29th April) we observed degrading performance in processing emails and updated the email status component whilst investigating. We identified an increasing number of errors related to an associated database, which we modified and rebooted. We were able to quickly regain access to the affected database and begin processing the backlog of emails by 18:20 UTC. Email processing and statistics were delayed for a total 48 minutes and are now processing real time. All API processing and segment queries remained realtime throughout and were not affected. If you have any questions about your account please contact support@getvero.com.

This incident has been resolved.

Newsletters are sending as normal. We are continuing to monitor segment calculations and will resolve this incident when segments are operating as normal.

Our provider has identified the root cause and the underlying system is now operating as normal again. We are restoring services to normal and monitoring.

We are currently experiencing issues calculating segments and sending newsletters. This is a result of issues with our primary segmentation data store. We are working with the service provider responsible for this service to understand the root cause. We will provide further updates as soon as they become available. Note that automated messages (Workflows, Behavioural and Transactional campaigns) and data collection (updating user properties and tracking events) remain **unaffected** and are **operating as normal**.

At 12:30 UTC yesterday, several of the nodes in the search cluster powering the Logs and Customer Profile pages in Vero went offline. We were alerted to this issue and, by 13:20 UTC, had begun to restore the affected nodes. Due to recovery time the UI was returned to normal at 16:30 UTC. During the intervening period the UI was unavailable. Once restored, there were delays in processing logs into the UI as the accumulated backlog caught up. No other processes were affected. We have investigated and understood the root cause of this issue. Fortunately, we have been working toward replacing this component of our architecture over the last few months as part of planned infrastructure changes. We will be finalising this work within the new two weeks and are confident that this issue should not arise again. If you have any questions please email us at support@getvero.com.

Sendgrid have been able to quickly resolve the issue 🎉. All content (including non-roman characters) should now be rendering as expected.

Based on testing we have conducted, Sendgrid appears to be incorrectly encoding non-roman characters in outgoing emails. This is resulting in end users receiving content that replaces the non-roman characters with random encoded characters. This will only affect Vero customers who use Sendgrid accounts to deliver their emails. We have raised these examples with Sendgrid and will keep you updated. Please refer to this post for more publicly available information: https://news.ycombinator.com/item?id=21453178 We wanted to make you aware of this occurrence. If you want to discuss options such as switching to an internal Vero email provider, please email us at support@getvero.com.