Is sendwithus Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Some customers are experiencing intermittent connectivity issues with Sendwithus. The team is investigating.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Our upstream image upload management provider is currently experiencing an outage and our engineering team is looking into it.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some customers are experiencing higher than normal error rates in the dashboard and API. The Sendwithus team is investigating.

# Summary Between 18:33 July 13th and 11:20 July 14th 2023 \(PDT\), inline attachments from a small number of emails were incorrectly attached to an unknown number of emails sent by other accounts. # Impact & Metrics | **Question** | **Answer** | | --- | --- | | What was the impact | Incorrect inline attachments were incorrectly attached to a maximum of 17% of overall emails sent by Sendwithus | | Who was impacted | A subset of Sendwithus users sending emails between July 13th 6:33 pm PDT and July 14th 11:20 am PDT may have been impacted | # Detection & Response‌ | **Question** | **Answer** | | --- | --- | | When was the incident detected | July 14th 2023, 10:21 am PDT | | How was the incident detected | An impacted user reached out to the Sendwithus Customer Support team and the incident was quickly escalated | # Timeline _Times are in Pacific Time_ **Jul 13, 2023** **09:44** A new feature related to sending inline attachments was released **18:33** The first detection of a send using the feature since the change **Jul 14, 2023** **10:21** A user reported some of their recipients were receiving unexpected inline attachments on their emails **10:45** Customer Support notifies on-call engineers and the incident is escalated **11:00**   Status page updated **11:10** Resolved the issue by reverting to the previous version of Sendwithus **12:40** Underlying issue identified **13:00** Engineering team began an investigation to determine total user impact **17:00** First emails to personally reach out to impacted users sent # Root Cause Analysis **Problem:** A user has reported that some of their emails contain an unexpected attachment. Why? **Reason:** Recent changes were made to our systems to support sending an arbitrary number of inline attachments. This change introduced a bug which resulted in a variable being reused for subsequent emails sent by the same process. Why? **Reason:** An optional parameter in a function definition had a default reference-type value. The issue was not identified before deploying to production. Why? **Reason:** Our current testing, linting, and code review processes were not sufficient to identify the problem. # What can be improved * Changes are being made to our testing and linting processes to catch this category of bug in our CI pipeline * The transient nature of inline attachments and a lack of thread/process level identification in our archived transaction logs has made it difficult to identify exactly which customers and emails were affected. Due to this only the maximum number of _potential_ impacted emails can be identified * Increased controls to improve cross-tenant isolation of information # Corrective actions * The problematic change was quickly identified and reverted after we became aware of the issue * Changes have since been made to the codebase to reduce the probability of similar issues manifesting elsewhere

A subset of emails sent between 2 am and 11 am PDT July 14th had incorrect inline attachments added to them. Our team has identified and resolved the issue. Sendwithus support will be reaching out to impacted customers.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Our engineering team is currently investigating an issue with the Email logs page. Email sending may be delayed.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Some customers are running into issues connecting to the Sendwithus API and dashboard. Our engineers are currently looking into the issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Some customers may be experiencing elevated error rates on the Sendwithus API. We are investigating.

The root problem has been resolved. Sendwithus support will be reaching out to impacted customers.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We have temporarily disabled the feature while we investigate.

We are investigating the issue where users are seeing incorrect data in the customer's panel

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We're experiencing issues with forwarding Mailgun webhooks to accounts. The team has identified the problem and are working on a resolution.

The underlying problem has been resolved and all failed emails have been resent.

A fix has been implemented and we are monitoring the results.

Some customers emails are failing to send correctly due to a rendering change. We've identified the problem and are working on a fix. We will resend any failures once the solution is live.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are continuing to investigate this issue.

Some customers may be experiencing slower dashboard loading times. The Sendwithus team is investigating. Email is continuing to send normally.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

Some customers may be experiencing a slower than normal dashboard. The Sendwithus team is investigating. Email is continuing to send normally.

This incident has been resolved.

Some customers have reported connection issues with the Sendwithus dashboard. Our engineering team is currently investigating.

This incident has been resolved.

Sendwithus performance has returned to normal. We are continuing to monitor the results.

A fix has been implemented and we are monitoring the results.

The issue has been identified and we are implementing a fix.

Some customers may be seeing Sendwithus API and application errors. We are investigating the issue and will have an update soon.

This incident has been resolved.

A fix has been implemented and we are monitoring the results. Emails are sending normally again.

We've identified an issue with email sending and some messages may be delayed. A fix has been implemented.

This incident has been resolved.

We have not seen any errors in the last several hours. Our team is continuing to monitor.

We have seen a significant decrease in delayed emails from SendGrid accounts and our team will continue to monitor.

Our team has identified an issue where emails may be delayed for customers using SendGrid. We have released some changes to help mitigate this issue and we are actively working with the SendGrid team to get this resolved.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue that is causing intermittent errors on the Sendwithus API and are working to resolve the issue. HTTP 5xx errors are safe to retry.

The dashboard has been running smoothly for the last several hours.

A fix has been implemented, dashboard access has been restored, and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified a problem with the reliability of the Sendwithus API and were working on a resolution. HTTP 5xx errors are safe to retry.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some customers have reported connection issues with the Sendwithus dashboard. Our engineering team is currently investigating.

This incident has been resolved.

A fix has been implemented and we are monitoring the results. Logs should now be accessible.

We are currently investigating an issue with log search in the Sendwithus Web Interface. Emails are sending normally.

Thank you for your continued patience. As a result of our fix 5xx errors have been reduced dramatically and the Sendwithus API is handling requests normally again.

A fix has been implemented and we are monitoring the results.

API errors have been reduced and dashboard access has been restored. We are continuing to work on a solution.

We have identified an issue that is causing intermittent errors on the Sendwithus API. We are working to resolve the issue. API calls are safe to retry.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue that is causing a higher than normal error rate on the Sendwithus API. We are working to resolve the issue. API calls are safe to retry.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have a identified an issue that is causing a higher than normal error rate on the Sendwithus API. We are working to resolve the issue.

This incident has been resolved.

We're currently seeing services come back online and email is sending again. We will continue to monitor and post updates as necessary.

We've identified the issue with an upstream service provider and are working with them on a fix.

We are currently investigating an issue with our API

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We've identified an issue with an upstream service provider which is causing an increase in API timeouts. We're working with them to resolve the issue.

This incident has been resolved.

Our upstream service provider is still experiencing issues, but our metrics shows Sendwithus is operating normally. We will continue to monitor and post relevant updates here.

Our upstream service provider is still working on mitigating the issues that are affecting Sendwithus. We will continue to monitor and post updates here.

Users may see issues loading the Sendwithus dashboard and delays in analytics. Emails are continuing to send.

We've a identified an issue with an upstream service provider and are assessing the impact to Sendwithus.

This incident has been resolved. Our engineering team is currently working on recovering backlogged messages.

A fix has been implemented and we are monitoring the results.

We are continuing to work on a fix for this issue. Users may not be able to access the templates page.

The issue has been localized to an upstream service provider and we are working to resolve it.

Sendwithus engineers have been notified of reliability issues on the email API and are investigating.

This incident has been resolved.

Thank you for your patience. The API has successfully been restored and our team is continually monitoring the situation.

Our engineering team have deployed a change which required DNS updates. Some DNS values may be cached. If you are still experiencing 503 response codes, we recommend flushing your DNS cache to receive the updated values.

The issue has been identified and a fix is being implemented.

We have received reports of the Sendwithus API returning a high volume of timeouts. Our engineering team is currently investigating.

This incident has been resolved.

Our engineering team has implemented a fix and sending speeds are beginning to return to normal.

We are currently investigating reports of delayed emails.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

An upstream provider issue is impacting the ability to upload files to Campaigns and the image library for some customers. We're working with them to resolve the issue.

We are continuing to monitor for any further issues.

An upstream provider issue is impacting the ability to upload files to Campaigns and the image library for some customers. We're working with them to resolve the issue.

This incident has been resolved.

The issue has been identified as a problem with an upstream service provider. The system quickly recovered and we are monitoring. API error rates have returned to normal.

Our systems are reporting higher than normal errors on the Sendwithus API. Our team is investigating.

The logs pages have returned to normal.

We have identified an issue with our logs system that is causing intermittent issues with some users. Our engineers are working on a fix. Emails are still sending normally.

SendGrid has returned to normal and we're seeing that emails are sending normally. Please reach out to support@sendwithus.com if you would like us to resend any failed emails for you.

We have been closely monitoring the SendGrid outage. Starting at 9:00AM PDT we have seen a significant percentage of emails fail when sending through SendGrid. If you would like your emails to be resent please reach out to support@sendwithus.com and our support team can help out. We can resend once SendGrid has returned to normal. More information can be found at their status page: https://status.sendgrid.com/ Customers using a different email service provider are not affected.

This incident has been resolved.

We're experiencing an elevated level of API errors due to an issue with an upstream hosting provider. Our engineers are monitoring the issue.

This incident has been resolved.

Due to an issue with an upstream hosting provider users will have likely seen a higher than normal rate of HTTP 500 error codes from the Sendwithus API between 16:00 UTC - 17:00 UTC. Our engineers are monitoring the issue.

This incident has been resolved.

Sendwithus is experiencing an issue with an upstream provider which has resulted in an increase in API errors. API calls that result in a 503 are safe to retry. Email is sending normally.

Due to an issue with our platform provider, some customers experienced higher failure rates on the API. Customers would have seen 404's resulting from our API endpoints between 18:04 UTC — 22:35 UTC. All 404s can be safely retried. We're working on better monitoring to alert our users to unexpected availability issues sooner.

This incident has been resolved.

The root issue has been resolved and a solution has been put in place, we're monitoring the results.

There is an abnormal increase of server errors on our email related APIs. Our engineers are investigating. Failed requests are safe to retry.

This incident has been resolved.

System stability has normalized. We're continuing to monitor.

System stability is returning to normal and our engineers are monitoring.

An issue with an upstream service is impacting Sendwithus Connect and Campaigns. Emails will still be sent, but may be delayed until the issue is resolved. Updates will be posted here.

The incident has been resolved and the dashboard — specifically log and customer search — is working again for all users. Emails sent during this period were sent, but there may have been some minor data loss. Please reach out to our support team if you have any questions.

We have released a fix for this issue and currently monitoring the results. The dashboard should be coming back online for all users.

We are currently experiencing issues with logs and log search. The Sendwithus engineering team is investigating.

# Summary Due to an issue during a database migration on 2019-02-14, Drip Campaign activations failed to activate during a 27 hour period from 2019-02-14 12:50 PST to 2019-02-15 15:36 PST. Any Drip Campaign emails scheduled to send before the incident and any deactivations made with the API or automation triggers continued to work as normal. We apologize for any disruption this may have caused for our customers and their business. We have already notified all account administrators for effected accounts. ## What happened Our infrastructure team executed a planned data migration of the database responsible for Drip Campaigns. During this time all new Drip Campaign activations were to be made redundant across both the old and new datasets. An error in our code to configure this redundancy caused loss of data during this migration period. This ultimately prevented any new Drip Campaigns from properly activating. ## What was affected During the incident window: - Drip Campaign stopped activating via the API - Drip Campaign stopped activating via automation triggers - No new Drip Campaigns could be created ## Who was affected Any accounts that activated a Drip Campaign during the period above would have been affected. This includes Drip Campaign activations through the API and through Automation Triggers. If this was the case, your account administrator An email has been sent to your Sendwithus administrators if your account was affected. ## What Sendwithus is doing to fix it While do mistakes happen, it's most important to us that they're caught early so they can be swiftly corrected. To mitigate the likelihood of this happening in the future we will be: - Adjusting our alert thresholds for this particular issue. - Adding more thorough migration testing processes

This incident has been resolved.

We've identified the issue, deployed a solution, and are now monitoring the results.

Some customers are experiencing issues with activating drip campaigns and running legacy segmentation. The Sendwithus engineering team is currently investigating. API triggered emails are still sending normally.

SendGrid has resolved the issue and our engineers have confirmed the results.

SendGrid's status page has been updated and can be followed along with here: http://status.sendgrid.com/incidents/v80113h5txql

SendGrid is experiencing delivery issues for a fraction of users and some emails may not be going through. We are working with them to resolve the issue. You are unaffected unless you use SendGrid.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been pinpointed to a back end service and a fix is being implemented to resolve the issue.

We are currently investigating issues with the logs page in the Sendwithus dashboard.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating issues with the Sendwithus logs page not loading. Email sending is unaffected.

This incident has been resolved.

The root issue has been resolved and a solution has been put in place, we're monitoring the results.

Some customers may be experiencing intermittent connection timeouts. We have identified an issue with an upstream provider and are working towards a solution.

The issue stemmed from an issue with an upstream service provider that we were able to quickly recover from. Any successful API requests during the outage were resent.

The root issue has been resolved and we're monitoring the results and assessing impact.

We've identified the issue and we're working on bringing Sendwithus back to full capacity.

We are continuing to investigate this issue.

We are investigating elevated errors on Sendwithus. The Sendwithus web portal is currently unavailable.

One of our downstream logging providers had a data breach incident on 2018-10-02 at 9:46 PDT. For a period of 6 hours, there was the potential for Sendwithus logs to appear in other accounts that also used that logging provider. We are still investigating if our logs were compromised, but at this time we recommend rotating API keys for any users of our Segment.IO or the Automation integration. If your account was potentially impacted, we will be reaching out directly. For more information please contact support@sendwithus.com

This incident has been resolved.

The issue has been identified as an issue with an upstream provider. Error rates have dropped considerably and we're continuing to monitor the situation.

We're currently elevated server errors being returned from Sendwithus' API.

This incident has been resolved.

We've deployed a solution and are monitoring. The Sendwithus Web Interface should be working for everyone now.

We've identified an issue with our platform provider and our engineers are working on a solution.

We are currently investigating reports of errors on when using the Sendwithus Web Interface. The API is unaffected.