Is Mailprotector Down Right Now? Check if there is a current outage ongoing.

Bounces for "Banned Sending IP" and "Blocked using Spamhaus" have been all but absent for the last seven days. Confirmation of resolution has not been received from Microsoft, but the current status of deliverability and information from Microsoft gives us reasonable confidence that this issue is on a path to resolution. We will continue monitoring email deliverability for this incident. As a reminder, if there is a recurrence, Microsoft recommends that the sender follow the delisting instructions in the NDR (bounce notification) and resend the email.

Mailprotector has noticed an increase in the amount of emails with a “Banned Sending IP” bounce from M365 within the last 12 hours. This previously was resolved on May 1st, but the issue has returned. This is still occurring in very small volumes, but we understand any bounce from the system is a disruption in business function. Due to this, we have reopened our case with Microsoft and are attempting to re-engage them. We would recommend having the sender send the email again since it will go through the second time. It is important to note that this is not an outbound reputation issue. Microsoft recommends that the sender follow the delisting instructions in the NDR (bounce notification) and resend the email. Mailprotector is attempting to work the problem with Microsoft to resolve the issue permanently.

Bounces for "Banned Sending IP" and "Blocked using Spamhaus" have been all but absent for the last seven days.Confirmation of resolution has not been received from Microsoft, but the current status of deliverability and information from Microsoft gives us reasonable confidence that this issue is on a path to resolution. We will continue monitoring email deliverability for this incident.As a reminder, if there is a recurrence, Microsoft recommends that the sender follow the delisting instructions in the NDR (bounce notification) and resend the email.

Mailprotector has noticed an increase in the amount of emails with a “Banned Sending IP” bounce from M365 within the last 12 hours. This previously was resolved on May 1st, but the issue has returned.This is still occurring in very small volumes, but we understand any bounce from the system is a disruption in business function. Due to this, we have reopened our case with Microsoft and are attempting to re-engage them.We would recommend having the sender send the email again since it will go through the second time. It is important to note that this is not an outbound reputation issue. Microsoft recommends that the sender follow the delisting instructions in the NDR (bounce notification) and resend the email. Mailprotector is attempting to work the problem with Microsoft to resolve the issue permanently.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Database maintenance for the Bracket encryption product will be done on May 28, 2025, starting at 10:00 PM ET and concluding by midnight ET.The Bracket portal will be unavailable during the maintenance window. Encrypted emails will be queued, and users should expect delays in receiving notifications and sign-in links. Email flow will be unaffected otherwise.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Required database maintenance is planned for Wednesday, May 21, 2025, beginning at 10:00 PM ET.The Mailprotector Console will not be available periodically during the maintenance. Mail flow will NOT be affected.Partners or users attempting to access the Console (via review notification, bookmark, etc.) will be presented with a maintenance notification page.Most importantly, mail flow will not be impacted by the maintenance window.

Since April 29, there have been unnoticeable bounces. While we cannot be certain that Microsoft will not repeat the issue, the status is considered resolved at this time. If you receive a bounce for "banned sending IP" or "blocked using Spamhaus", please open a support request.

Since April 30, little to no bounces have been observed. We are continuing to monitor the situation and waiting for Microsoft's response to explain the reason and a permanent resolution.

Bounces over the last 7 days have subsided, but an increase has been observed today, April 29. The number of bounces is tiny; however, we know a single bounce can frustrate a customer. Communication with Microsoft was escalated last week, and we await more information.

Mailprotector is aware of intermittent bounces of inbound emails to Microsoft 365 domains. The bounces reference one of two reasons: "banned sending IP" or "blocked using Spamhaus." In both cases, Microsoft and Spamhaus do not indicate that the IP addresses are blocked when a delist request is made. Mailprotector has two open and active support requests with Microsoft to explain and resolve the situation. Mailprotector has been attempting to resolve the issue with Microsoft for over a week. Unfortunately, we have seen increased bounces in the last 24 hours. Partners and users may click on the delist links in the bounce notification. However, the delist will reply that the IP addresses are not listed. It is important to note that this is not an outbound reputation issue. Microsoft is randomly rejecting emails from IP addresses that only relay inbound emails after being filtered. The IP addresses are added to the recommended inbound connector for M365 deployments.

Over the weekend, a small pool of inbound gateway servers bounced incoming emails to CloudFilter because of a database connection issue. The NDR message indicates: The response from your-domain-com.inbound.emailservice.io was: 5.7.1 username@your-domain.com: Recipient address rejected: Access denied The team responded and resolved the issue quickly. Fortunately, weekends have a comparatively low email volume. However, if a user was expecting an email and did not receive it, they may need to request the sender to retry sending it. A thorough investigation of the incident is being performed, and the issue will be adjusted, monitored, and mitigated for the future.

Monitoring systems alerted the team to an issue with approximately 20% of inbound email processing. The issue has been identified and resolved. The issue will have impacted emails that show a delay in the logs this morning, but no further impact is expected.

Earlier this morning, Bracket encryption scaling added resources that did not respond to connection requests correctly. Users may have received a bounce notification if their encrypted email attempt connected to this subset of resources. The bounce notification error was: The bounce is: <enduser@companydomain.com.encrypt.bracket.email>: host encrypt.bracket.email[18.223.92.55] said: 550 A secure TLS connection is required. The problematic resources were identified and removed, and the issue has been resolved. Users can send encrypted Bracket emails as expected.

Inbound mail flow from CloudFilter is operating as expected.

The database issue has been resolved. Inbound mail delivery should soon return to expected performance. We are continuing to monitor the performance.

The team has identified an issue with a database and is working to resolve it.

We have received reports of delayed email delivery from CloudFilter and are investigating the situation.

Mail flow is functioning at expected performance levels.

Mail flow response times are reaching 30 seconds or less. We will continue to monitor the situation until the permanent fix is deployed.

Mail flow response times are continuing to decrease. A fix is planned for deployment to prevent this incident from occurring again. Another update will be provided to confirm improvement or resolution.

The issue has been identified and mitigated. The team is working through queued jobs, but mail flow could be delayed by 10 minutes. We are working quickly to return mail flow to expected performance.

We are investigating an impact on mail flow.

Inbound mail flow from CloudFilter is operating as expected. The issue has been resolved.

The issue has been addressed, and the queues are emptying. We are monitoring the progress closely and will confirm once the issue is resolved.

We are aware of inbound email delays from CloudFilter. The team has identified the issue and is working on a fix.

Deferrals of emails have slowed substantially but have not ceased altogether. Deferrals to certain email service providers will always occur due to several factors, including domain and sender reputations. Mailprotector deems the IP reputation problem resolved at this time. Mailprotector does not control domain and sender reputations. IP reputation management is influenced by Mailprotector's sending infrastructure management, but it is not solely in our control. Deferrals may happen on occasion. Users will see a deferral notice under the right circumstances as we continue to manage the sending infrastructure to build additional resilience and strictly enforce the sending policy. If a deferral or bounce is questionable, please open a support request, and Partner Success will gladly explain the situation as best as possible. In a pinch, the option is to send emails directly from the email service provider rather than CloudFilter's smarthost relay.

Mailprotector has observed improved deliverability to Rackspace and AT&T domains. However, we are cautious, given that email volume is typically reduced between the Christmas and New Year's Day holidays. Over the coming weeks, we will move affected servers back into full production and wait for stabilized delivery responses. We expect some deferrals because these email service providers do not offer a means to set an expected volume of emails. We cannot eliminate deferrals entirely, especially if an email is considered questionable for spam or malicious intent. If the process moves forward smoothly and as expected, we will send one more update in a couple of weeks. We hope to consider this incident complete.

Mailprotector resolved a significant reputation issue that affected AT&T and Yahoo domains. As part of that remediation, the volume of emails from affected servers significantly dropped. As the affected servers have been put back into service, some domains using Prodigy (part of AT&T and Yahoo) and Rackspace are deferring email delivery due to the rapid volume change. Mailprotector is mitigating the occurrence of deferrals by metering the volume of affected servers. This process will take several more weeks to reach full resolution.

The reputation issue with AT&T and Yahoo domains has been resolved. We have received confirmation from an AT&T representative that none of Mailprotector's sending servers are blocked.

Evaluating the logs bounces for reputation issues have been resolved. However, Mailprotector continues to see occasional deferrals from AT&T domains as servers are moved back into full production. The deferrals will subside as the server volume is slowly increased. We continue monitoring the logs to minimize the deferrals.

Deliverability has continued to improve over the past few days. The slow increase in the volume of emails from formerly affected servers is working, although it still creates some deferrals because of the protective methods used by AT&T property domains. The positive result of this "warming" strategy will continue. Another update will be sent when we have significant news or, hopefully, a resolution announcement.

Mailprotector has slowly increased the volume of emails sent from affected servers. This step is necessary to improve the server's reputation. However, due to their greylisting methods, AT&T and Yahoo email services may still defer some sending emails for 24 hours. The first increase in volume began earlier today, and we anticipate another increase later this week. Updates will be added as we have more information.

In the last 24 hours, Mailprotector has improved with Yahoo domains. However, AT&T domains are still deferring or bouncing with some servers. Unfortunately, the deferrals are held for 24 hours, which delays the user and Mailprotector learning that a server is still affected. Servers found to be still affected have been rotated out of full production to minimize the impact on users. Deferrals and bounces will still occur since the servers will continue to be used at a significantly reduced rate.

We are continuing to work on a fix for this issue.

More progress has been made towards improving deliverability to AT&T/Yahoo-owned email domains. Mailprotector has received an automated response from AT&T, which indicates that the request for reputation delisting is being escalated. We do not know how long that will take, but seeing a meaningful behavior change is encouraging.

Mailprotector has observed a decrease in bounced emails; however, there has been an increase in deferrals. We have moved resources in response to this behavior and have reached out to AT&T/Yahoo again. We continue to work on resolving the issue to resume expected deliverability. Please see https://support.mailprotector.com/hc/en-us/articles/31650407848852-Deferrals-or-bounces-sending-email-to-AT-T-or-Yahoo-email-addresses if you need to consider a workaround.

Progress has been made with AT&T services, but it is not completely resolved. Some deferrals are still being reported. Yahoo has also improved, and we continue reporting evidence to resolve the deliverability issues. Reports of deferrals and bounces have decreased significantly. However, we will still monitor and manage the situation until the outbound servers are 100% healthy.

The team has observed progress on some IP addresses but is still waiting for others to present better deliverability statistics. We continue following the email service providers' processes and rotating resources as needed. Users will continue to receive some deferral and bounce notifications.

For more information about the deferral and bounced emails to AT&T and Yahoo email services, please see https://support.mailprotector.com/hc/en-us/articles/31650407848852-Deferrals-or-bounces-sending-email-to-AT-T-or-Yahoo-email-addresses

Mailprotector is aware of IP reputation issues sent to email addresses associated with AT&T and Yahoo. The team has been working to mitigate the chances that emails to those email services are deferred or bounced, but AT&T and Yahoo control this. More information is being prepared to provide options for temporary workarounds. A link to that information will be shared soon.

We have confirmed the TLS connections between the affected CloudFilter outbound gateway servers and Bracket have been working as expected.

The TLS connection issue from CloudFilter's outbound gateway to Bracket is fixed, and we are monitoring the logs to ensure resolution. Bracket encryption from CloudFilter-protected accounts is operating as expected.

The issue has been identified, and the team is working on implementing the fix.

Some attempts to send Bracket encrypted emails from CloudFilter-protected accounts are failing due to a TLS error. We are investigating the issue. Sending Bracket emails directly from https://bracket.email or Shield is working as expected.

An issue that started at 12:21 PM ET resulted in lost logs from 12:20 PM to 2 PM and likely duplicated some logs in the early morning. AWS performed a maintenance task that restarted a transport server in an ungraceful manner. This caused the logging service to lose its place in the log that it was duplicating. The logging service started at the beginning of the log file and writing logs. The problem was discovered, and it forced a rotation of the mail log (to start over fresh) and a re-addition of the affected server to the load balancer. The event may have duplicated some log messages, dropped others, and created odd-looking data in the mail flow logs in the Mailprotector Console during the event. However, mail flow was not affected by this event.

AWS has confirmed the issue with S3 storage has been resolved.

Per AWS, "Oct 07 1:06 PM PDT Between 12:27 PM and 12:52 PM PDT we experienced increased API error rates for PUT and GET requests to S3 in the US-EAST-2 Region. Other AWS Services that use PUT and GET S3 APIs were also affected. The error rates have recovered, and we continue to investigate root cause." The service is restored, and Shield mail delivery performance resumes the expected performance. Depending on when the email was sent, some mail delivery delays may occur.

Email delivery to Shield is impacted by an outage of Amazon Web Services' S3 storage service. The team will work with AWS to restore expected services as quickly as possible.

The AWS outage has been resolved, and CloudFilter's impact has been minimal. Quarantine access and performance are operational.

Emails destined for quarantine by CloudFilter may not be accessed or released due to a problem with Amazon Web Services' S3 storage service. The team is working with AWS to restore expected performance as quickly as possible.

An organization sent a poorly constructed email just before 8:00 AM ET on Tue, Oct 2, with over 2,000 email addresses in the CC field. The header of the email was too long to process as expected, creating a unique situation that contributed to a large amount of resource utilization. Once the team identified the email message, custom handling for this poorly formed email was built. This allowed the message to be processed as expected without increasing resources. Future incidents with poorly constructed emails that exceed typical header data will be handled gracefully.

Mail flow has remained healthy, and email delivery performance is resolved.

Mail flow has been restored to regular operation. The team is continuing to monitor the situation and ensure a complete resolution.

The team has identified the issue, and mail flow performance has improved. Email delivery rate is just under one minute and is continuing to improve.

Some emails may be delayed over 30 minutes. The team is investigating the issue.

Email delivery delays have been observed on Shield, and the team is investigating the issue. At this time, the delays are over 15 minutes.

From approximately 3:35 PM ET outgoing emails were being held for review at an aggressive rate. The team was alerted and resolved the verification rule causing the aggressive filtering about an hour later. Emails sent during this time will need to be released by the sender. Senders will have received a notification stating their email(s) been held for review and clicking the release button will deliver the message to the intended recipient(s).

Link analysis has been modified to improve performance and X-ray should be working as expected.

Link analysis in X-ray is running slow and may not show results. The team is investigating the issue. Mail flow is operating normally and has not been impacted by the issue.

This incident has been resolved.

We believe we have identified the source of the spike in traffic. Email delivery is returning to normal, and we will continue to monitor its performance.

At approximately 8:15 AM EDT, alerts indicated delayed email delivery due to a spike in traffic. The team has improved email delivery by scaling systems and continues investigating the issue.

This incident has been resolved.

A fix has been implemented, and Shield is performing as expected. Link analysis in X-ray is processing a backlog, but this has no impact on Shield's performance at this time. The team will continue to monitor until the backlog is processed.

The cause has been identified and isolated. Performance has returned to normal while the team addresses the root cause.

The team is aware that Shield components are responding slowly and is investigating the issue. Access to the Shield UI and loading the HUD in emails may time out.

On July 25, 2024, between approximately 5:30 PM ET and 7:00 PM ET, DMARC results with a reject policy were sent to quarantine rather than bounced as per RFC. This temporary change was made after reports of false DMARC failures. The team evaluated the logs and spotted intermittent SPF failures in the DMARC evaluation process. An adjustment was applied to eliminate the SPF check failure. As of 7:00 PM ET, a failed DMARC with a reject policy has resumed bouncing those emails. Evidence of emails bounced due to a sending domain's DMARC policy will be found in the logs. NOTE: A DMARC policy is set by the owners of the sending domain, not Mailprotector or a recipient. If the policy is set to reject messages that fail DMARC alignment, the bounce is made regardless of Allow rules.

From approximately 3:40 PM ET, manager accounts could not log into emailservice.io or partner-branded URLs. The team resolved the issue, and manager logins are confirmed to be working as of 5:15 PM ET. Thank you.

This incident has been resolved.

Mailprotector modified SpamCop results' scoring last week to reduce false positives on emails from Microsoft's Exchange Online (M365). Unfortunately, the modification did not have as much impact as we had hoped. Microsoft continues to have a range of IP addresses listed on the SpamCop database. After careful consideration, Mailprotector will continue to perform the SpamCop check but exclude it from the scoring process until further notice. We will continue to monitor the situation.

A sensitivity adjustment has been implemented to reduce the false positive rate on SpamCop results from Microsoft's IP addresses. Mailprotector will continue to monitor the situation. Please be aware there is a very low probability spam leakage may occur on an email with only a SpamCop signal, but the reduction in false positives should be noticeable.

In the last few weeks, Microsoft has experienced email reputation problems that resulted in their IP addresses getting listed with SpamCop. Mailprotector has been monitoring the situation, and we've attempted to mitigate the impact on CloudFilter and SafeSend, but ignoring the signals from SpamCop would introduce a significant risk. Unfortunately, Microsoft is responsible for correcting the behavior that is causing their IP reputation problems. The situation is complex when the recent spam reports from onmicrosoft.com domains are added to the variables at play. False positives can be frustrating, and Mailprotector attempts to minimize false positive rates without sacrificing sound security practices. Microsoft's infrastructure is considered a high-risk source, given the frequency of compromised mailboxes and impersonation using the onmicrosoft.com domains. Therefore, we expect a higher-than-normal false positive rate from M365-hosted domains. Mailprotector's team is monitoring the situation, addressing false positives strategically, and assisting partners as best we can. M365-hosted domains using a smart host, such as Mailprotector, are better positioned for successful email deliverability. Domains without a smart host and relying on Microsoft's email infrastructure are most affected. Mailprotector will provide more information if we receive it. Thank you.

One of the inbound transport servers was automatically rebooted for maintenance and could not handle the imposed load as it scaled back up. A server reboot is typically infrequent and mundane, but this particular transport server did not respond as typically expected. While the other servers could handle their current load, it was decided to resize in order to handle unexpected periods of higher demand in the future. All inbound transport servers have been resized to prevent similar delays, and more monitoring/alerting metrics have been added.

Maintenance was performed on the impacted instance. It has been added back into production and we are no longer seeing delivery delays. We will continue to monitor performance.

The team is currently investigating inbound mail delays. Delayed mail is primarily associated with a singular instance, and the instance has been temporarily removed from production. This should improve delivery times while the team continues to track down the root of the issue.

The scanning is operating as expected. By design and for security best practices, an incomplete scan of a message "fails to closed" and quarantines. Users must release messages from the quarantine if an email was impacted by an incomplete scan.

The false positive rate includes emails without attachments as well. The team is continuing to work on the issue. Users will need to release false positive messages from the quarantine.

Emails with a file attachment may not complete the file scan resulting in a quarantine result. The team is investigating the issue.

Inbound mail flow performance is operating as expected. A post-mortem follow-up will be added to this incident in the coming day or two.

We are continuing to investigate the issue. Reports have been received of some emails being delayed by 45 minutes.

We are currently investigating inbound mail flow delays. Observed delays are at 5 to 10 minutes. Some messages may experience longer delays.

The quarantine release queue has cleared.

The inbound email flow is operating as expected. Releasing multiple quarantined emails from the Console is delayed by as much as 40 minutes. A mass quarantine release has been requested and is being processed. To reiterate, normal email flow is not impacted. Only multiple or mass-released emails from the quarantine will experience a delay. The team is monitoring the quarantine release queue.

On May 30, 2023, mail flow monitoring systems recorded pressure during peak loads at the top of each hour. The volume of messages was higher than usual, not unexpectedly, returning from a holiday weekend. However, the mail processing speed was not performing as expected. Monitoring metrics indicated messages took over a minute to a few minutes to pass through the gateway and transports. The team increased resources and identified the bottleneck at the logging layer. Additional metrics have been added to analyze performance. However, some metrics caused more load on the system and slowed mail processing. The logging changes were adjusted and improved mail processing performance quickly. The mail processing slowdown was not anticipated but has allowed the collection of valuable information to continue improving the performance and infrastructure of mail processing in the future. Reports from partners indicate that CloudMail experienced the most effect from the performance degradation. However, a few reports have correlated to delayed emails sent and received that were likely affected by this issue. Monitoring during peak hours today, May 31, 2023, has confirmed yesterday's changes resolved the issue.

Mail delays of several minutes were observed during peak hours yesterday, May 30, 2023. The slowdown in processing messages was determined and resolved by the close of business. Changes to the logging system required adjustment, and the resolution has been confirmed as of the early afternoon on May 31, 2023.

The error displayed after sending emails from the CloudMail web UI is resolved. Messages sent from CloudMail were being delayed just enough to the outbound gateway to cause the error in the webmail UI. Mail flow was handled more gracefully on the backend. The team will continue to monitor the handoff between CloudMail and the outbound gateway through May 31.

We are investigating an issue with the CloudMail webmail UI displaying "an error occurred during sending the message" despite the email sending successfully. The UI may save the sent email to the Drafts folder. An optional workaround is to use webmail2.cloudmail.email

Log processing is operating normally.

Log processing is steadily improving. Current log delays are under 18 minutes. Log processing should return to normal within 90 minutes at the current trend.

Logging has begun recovering, but there is a large amount of data to process. Log data will continue to appear delayed while the log processing catches up. An estimated time for full recovery is not yet known. Email flow is delivering as expected. Log processing does not affect mail flow.

The team has identified an issue with mail log processing. Mail logs are behind by as much as 45 minutes in the Console. Mail flow is not affected by the log processing delay. Email is delivering as expected.

At approximately 4:00 AM ET, mail flow experienced delays of up to 15 minutes following an update of anti-virus definitions. The definition update was rolled back and replaced with a new definition update around 7:00 AM ET. The team continues monitoring mail flow and assessing the anti-virus definition updates. Mail flow is currently operating as expected.

User sync errors to Microsoft Graph have been resolved.

Mailprotector is aware of an issue with Microsoft 365 causing user sync errors. The error "Invalid response from Microsoft Graph API: 401" concerns an advisory posted in the Microsoft 365 Admin Center of affected domains. The error will continue every hour when the user sync runs against Microsoft Graph API. Partners can disable the automatic user sync until Microsoft resolves the issue to prevent the continued error alert.

The inbound gateway servers experienced degraded performance on the morning of Friday, March 31, 2023. The volume of emails was not unusually high, and further investigation into the performance degradation revealed a poorly constructed custom rule. The associated message rule was removed, and additional system resources were allocated to expedite email processing. For almost three years, Mailprotector has allowed partners to add regular expressions to Message Rules. Since introducing the feature, poorly written regular expressions have caused several performance issues. In the interest of consistent and predictable email delivery performance, Mailprotector has removed the option to add regular expressions to message body searches. In addition to the Message Rules modification, the team is working on enhancing the infrastructure to eliminate disruptions or delays in mail flow during peak hours. These enhancements include improving monitoring and scaling capabilities to the infrastructure, allowing for increased elasticity to adapt to the volatile and unpredictable nature of SMTP traffic. Mailprotector understands the critical nature of healthy email flow to businesses. The difficult decision to reduce a partner feature and continued infrastructure optimization are small pieces of the commitment to predictable email security performance. If you have questions about the Message Rule change, please get in touch with the Partner Success team. Thank you.

This incident has been resolved.

Surge queues have improved, and we will continue monitoring the traffic.

Surge queues are high again, and the team is working on managing the mail flow.

The surge queue has been recovered, and inbound email delivery is operating at expected levels. We will continue closely monitoring the traffic for a couple of more hours.

We are investigating inbound email delays associated with a high volume of traffic.

Log data comparisons indicate the certificate error was the root cause of yesterday's performance. The team is closely monitoring the outbound mail flow. Further investigation into the certificate error and the addition of new monitors for this type of situation are ongoing. As of this update, outbound mail flow is operating as expected, and Mailprotector expects it to remain healthy going forward.

Log data and system observation show outbound mail flow is operating as expected. We are continuing to monitor things closely and will send another update to confirm the resolution later this morning.

The outbound surge queue is normal; however, we will continue to monitor the services closely.

We are continuing to work on the outbound delivery issue. There is no evidence of a denial of service attack, and we are working on other possible root causes.

The team continues to manage the queues and seek the issue's root cause. The behavior is outside the bounds of regular activity, and the team is investigating a potential denial of service attack pattern.

The team is investigating delays with outbound email delivery. A surge of activity has resurfaced that is being actively managed to push the queues forward.

The outbound queues are healthy and delivering as expected.

Outbound queues are beginning to clear, and we are continuing to monitor the progress.

We have identified a surge in outbound email delivery. The team is investigating the issue.

Additional resources to the filtering infrastructure were added at approximately 4:30 pm ET on Thursday, December 1, 2022. Roughly 11% of emails received through CloudFilter either bounced or deferred during: * 12/1/22 4:30 pm to 6:00 pm * 12/2/22 5:30 am to 2:00 pm The additional resources had misconfigurations which caused a failure to deliver email to the next hop in the CloudFilter mail flow. The resources were added to resolve a resource constraint pattern observed at the top of the hour during early business hours. Email messages that entered the queue on the new resources were locked into the Postfix process of performing SMTP retries until reaching the bounce threshold or delivery. The bounce threshold was 12 hours, creating a delay in observing the problem. We estimate that 54,000 messages across the above timeframe were bounced. **Detection** The incident was detected when Partner Success researched tickets that seemed to correlate to the mail delivery issues through the newly-deployed resources. Partner Success escalated the issues to the operations engineer, who gathered information and escalated it to the on-call engineer. **Recovery** The issue was resolved by removing the new resources from production service. However, several attempts were made to resolve the issue at different points throughout the outage window. Ultimately, the issue was resolved by adding new external IP addresses to Postfix configurations and the transport servers' security group. **Next Steps** The incident exposed gaps in the documentation of legacy mail infrastructure and processes for rolling out changes to the infrastructure. Several changes in the process will be implemented, including but not limited to: * Comprehensive run book for adding new resources to the CloudFilter cluster * Account for identified "blind spots" * New staging environments for validating changes * Additional key metrics for post-deployment observation * A review of the data collected during this incident is ongoing * Determine metrics that were missing * Different alerting or notifications to get ahead of partner reports from tickets The incident and affected resources are resolved. However, the team will continue implementing processes to prevent a repeat of mail flow performance problems. The effort does not end with resolving this incident, but rather a refocusing on the iterative improvement of stable infrastructure management. **Anticipated FAQs** * Can the bounced emails be resent? * No. SMTP \(Simple Mail Transport Protocol\) does not keep an email after it is bounced. It is removed from the queue. * Can I receive a list of emails that were bounced? * Unfortunately, no. The logs are not organized in a way to pull that information together. Individual log details show the SMTP response in the timeline, a manual process in the Console.

Additional resources to the filtering infrastructure were added at approximately 4:30 pm ET on Thursday, December 1, 2022. Roughly 11% of emails received through CloudFilter either bounced or deferred during: 12/1/22 4:30 pm to 6:00 pm 12/2/22 5:30 am to 2:00 pm The additional resources had misconfigurations which caused a failure to deliver email to the next hop in the CloudFilter mail flow. The resources were added to resolve a resource constraint pattern observed at the top of the hour during early business hours. Email messages that entered the queue on the new resources were locked into the Postfix process of performing SMTP retries until reaching the bounce threshold or delivery. The bounce threshold was 12 hours, creating a delay in observing the problem. We estimate that 54,000 messages across the above timeframe were bounced.

The Bracket encryption queue has drained, and notification messages from emails originating in an email client should perform as expected.

The team has found Bracket encrypted emails sent from an email client (Outlook, smartphone or tablet email app, etc.) to be slow converting to a secure thread. The team is working on clearing the queue as quickly as possible. Observed delays have been up to 30 minutes. Encrypted messages created inside the Bracket portal are working as expected.

Outbound messages from Mailprotector's relay/smarthost were delayed approximately 30 minutes between 11:43 am and 12:17 pm ET. The problem has been identified and addressed, resolving the delays. Outbound email flow is as expected at this time.

On Tuesday, October 11, inbound email delivery to some Microsoft Office 365 domains was delayed between several minutes and 24 hours. Investigating the issue, Mailprotector determined that Microsoft was dealing with a high-risk email threat and graylisted two of Mailprotector’s transport IP addresses. Microsoft has confirmed they made changes to the email perimeter on Tuesday due to a spike in connection volume they observed across all sending environments. The spike in volume was not specific to Mailprotector’s IP addresses but rather a more global change. Microsoft would not provide specifics, but the description aligns with some denial-of-service attacks. Unfortunately, Mailprotector’s two IP addresses were caught in the changes made by Microsoft. Microsoft expects no anticipated repeat of this situation, and no guarantee of trusting Mailprotector IP addresses could be made. However, Microsoft also confirmed that Inbound Connectors would help bypass the type of changes made on Tuesday and should prevent the excessive delays experienced by some domains. Therefore, Mailprotector recommends adding the Inbound Connector to all Exchange Online deployments. Documentation will be updated to reflect the new best practice considerations.

The graylisting by Microsoft appears to have stopped last evening, and this morning's logs show the expected mail flow. The issue is considered resolved. However, Microsoft may change the perimeter behavior in the future. Mailprotector recommends implementing the Inbound Connector for domains in Microsoft Office 365. If you have further questions, please open a support request at https://support.mailprotector.com/hc/en-us/requests/new

Microsoft appears to have changed the Exchange Online perimeter, which may have introduced graylisting behavior. Microsoft's Exchange Online Advisory EX444758 is believed to be the reason for the changes. The start time of the advisory is 9:40 AM which coincides with email delivery delays appearing in Mailprotector's logs. Mailprotector has not changed or altered any email delivery operations. The email delays are the result of Microsoft changing the behavior of the Exchange Online perimeter. Any domain still impacted by Microsoft's changes today should implement the Inbound Connector for Mailprotector. The KB article that provides the information to implement the connector is found at https://support.mailprotector.com/hc/en-us/articles/115005484203-Office-365-Inbound-Connector. Please configure the Inbound Connector and ensure it is enabled. Mailprotector will update recommendations and documentation to emphasize the importance of using the Inbound Connector. The connector will provide additional trust information to Microsoft and reduce the chances of delaying email.

Microsoft has "grey listed" some of Mailprotector's inbound transport IP addresses, causing mail delivery delays. The team is working diligently with Microsoft to understand why the addresses have been grey-listed and request status removal. Partners trying to improve mail flow can implement the inbound connector to the tenant. The inbound connector is not a resolution but can enhance the trust between Mailprotector and Microsoft's Exchange perimeter. Please use the following link for the connector KB article. https://support.mailprotector.com/hc/en-us/articles/115005484203-Office-365-Inbound-Connector

We are receiving reports that a Microsoft Office 365 issue is delaying email delivery. Emails are being queued at Mailprotector. Emails are automatically retried using an exponential backoff method. Once Microsoft's service issue has been resolved, emails will automatically be delivered. Mailprotector is continuing to monitor updates from Microsoft for more information.

Around 9:00 am ET, IP addresses on the Amazon Web Services (AWS) back end began to point to expired destinations. The cause of the issue is undetermined because it occurred in the network layer of the AWS load balancers, not Mailprotector's infrastructure. Once the behavior was identified, the team manually pointed servers to the updated destinations until the load balancers were automatically updated. The load balancer issue caused a tiny percentage of emails to stall on the way out to the internet. The load balancers are updated, and the team has not seen a destination change that would cause further issues. Today's behavior is unrelated to last week's inbound mail delivery delays. However, the team is examining the situation and working with AWS to add monitors that would provide a more timely response to changes with the load balancers.

Inbound email delivery has been stable since Monday afternoon, August 8. Last week's delivery delay appeared to be a disk constraint. The team scaled the system and moved processes to faster disk resources. Unfortunately, increasing the disk performance moved the resource constraint to another layer of the system, and the problem was repeated on Monday morning. Mailprotector understands the importance of consistent email flow, and delays can create more business disruption than many other technical issues. The team has over-provisioned the systems to ensure the delivery delays do not repeat. At the same time, new monitoring and warning alerts are being developed to scale resources better ahead of noticeable performance changes. Historically, the transport systems have run smoothly since they do nothing more than complete the delivery of a message to a destination email server. The monitors and alerting added to the transport servers will provide the team more visibility to react in a timelier manner and prevent delays as we continue to grow with our partners. The inbound email delays are resolved, and we do not anticipate further performance problems.

Inbound email delivery is stable and performing as expected. The team is still working on further investigation. A post-mortem will be posted as soon as possible.

All transport queues have caught up, and new emails are being delivered in a few seconds. The team will continue to monitor the transport system.

We are continuing to investigate the issue and work to reduce delays as quickly as possible.

We are investigating inbound email delivery delays. The issue is intermittent. Emails are being delivered within 30 seconds upwards to 90 minutes. The average delay is 10 to 20 minutes. Last week's system upgrades were successful, but the issue is still under investigation.

Inbound email delays have returned to normal again. Increases in resources have been staged, and we will monitor the system in the morning.

The team has observed the inbound email delay creep back up. We are working on the issue.

The inbound mail flow has returned to expected performance.

A large volume of mail caused inbound email delays during the early hours of the business day. The mail flow is settling back down to expected levels. The team will add additional resources to the system overnight for added capacity during high-volume periods. We continue monitoring the transport servers until performance returns to expected levels.

We are investigating inbound email delays between 2 - 10 minutes. The delays may present bad mail flow statuses in the Console for affected domains.

A change to a database table had unintended effects on the Console web app. The change has been rolled back.

A fix has been implemented and we are monitoring the results.

We are investigating reports of intermittent 500 errors when accessing the Console.

This incident has been resolved.

Log latency has normalized. We are continuing to monitor the situation.

We are investigating unusually high log latency.

This incident has been resolved.

Inbound mail delivery performance has improved. This morning, delays occurred at the top and half-hour marks, with the most significant observed performance difference around 11:00 AM ET. The team continues to monitor the situation, but regular performance has resumed.

We have observed a slower than expected processing of inbound emails. The team is investigating the issue.

This incident has been resolved.

Spamhaus has removed the server from the blocklist. We will continue monitoring the IP. Normal inbound mail flow has resumed for messages delivered from 50.0.70.91.

One of the inbound transport servers has been listed on a Spamhaus blocklist. Some email providers are bouncing messages being delivered by this server. The team is working on delisting the server. Bounced messages will notify the sender, and their emails will need to be resent. Default Mail Flow thresholds in the Mailprotector Console may show a mail flow problem due to the bounces.

The rendering of the Bracket portal has been resolved. An invalid certificate made stylesheets and visual assets inaccessible to the portal. The certificate has been replaced.

The team has identified the cause of the rendering issue and is working on the resolution.

The login page for Bracket is not rendering the style correctly. The team is investigating the issue. Bracket functionality is unaffected.

Outbound queue lengths have returned to normal. The remaining messages still in the queue should be processed and delivered shortly. The team will continue to monitor the situation, but at this time, outbound mail flow is operating as expected.

Outbound queue lengths are improving, and the queues are draining. The team is continuing to monitor the progress.

The team is continuing to work on the issue. Outbound emails are still delayed but delivered once through the queue.

The team has identified a potential cause for the outbound delay of emails. They are adjusting settings to confirm the possible resolution and processing the outbound queues as quickly as possible.

Outbound emails are delayed thirty minutes to an hour. The team is investigating the issue and attempting to process the queues as quickly as possible.

CloudMail mail flow is restored to expected performance. Deferred emails have already been delivered as well.

Storage alerts on some CloudMail resources were not triggered as expected. The team is addressing the issue to restore expected mail flow to CloudMail mailboxes.

We are investigating reports of inbound emails to CloudMail accounts being deferred. Outbound sending appears to be working, but webmail intermittently reports a quota error. Outbound email does seem to be sending successfully despite the UI error.

Outbound queues were not processing fast enough. The team increased processing performance, and the delays have been resolved. Outbound queues will be monitored to ensure the improved performance is maintained.

The team is continuing to investigate the issue. Observed delays can be up to one hour. Inbound mail flow is not affected.

We are investigating deferred messages and failed SMTP Authenticated connections of outbound messages. Inbound mail flow is not affected.