Historical record of incidents for Lob
Report: "Issues exporting mail data"
Last updateWe are currently experiencing issues Exporting mail data. We are investigating the issue and apologize for the inconvenience
Report: "Rendering delays"
Last updateWe are currently experiencing delays with rendering. As a result, proofs and thumbnails may take longer than usual to appear in the dashboard. We've identified the root cause and are actively working on a fix.
Report: "Elevated API Errors"
Last updateThis incident has been resolved. All endpoints and services are operational.
We're experiencing an elevated level of API errors on our US verifications and reverse geocode lookup endpoints. We are currently looking into the issue.
Report: "Elevated API Errors"
Last updateThis incident has been resolved. All endpoints and services are operational.
We're experiencing an elevated level of API errors on our US verifications and reverse geocode lookup endpoints. We are currently looking into the issue.
Report: "International verifications returning 500s"
Last updateThis incident has been resolved. All services are operational.
We are currently experiencing a service disruption affecting our international address verification service for countries outside the US, UK, Canada, and Mexico. Requests for address verification in all other countries are currently returning errors (HTTP 500). Verification services for the US, UK, Canada, and Mexico remain fully operational.
Report: "International verifications returning 500s"
Last updateThis incident has been resolved. All services are operational.
We are currently experiencing a service disruption affecting our international address verification service for countries outside the US, UK, Canada, and Mexico. Requests for address verification in all other countries are currently returning errors (HTTP 500).Verification services for the US, UK, Canada, and Mexico remain fully operational.
Report: "Email Delivery Issue (Resolved)"
Last updateThis incident has been resolved.
We experienced an issue sending emails between Apr 29 at 10:45 AM PT and Apr 30 at 10:45 AM PT. Our team is currently in the process of resending the affected emails. If you were expecting an email during this time and have not received it, please contact us at support@lob.com. We apologize for the inconvenience and appreciate your patience.
We experienced an issue sending emails between Apr 29 at 10:45 AM PT and Apr 30 at 10:45 AM PT. Our team is currently in the process of resending the affected emails. If you were expecting an email during this time and have not received it, please contact us at support@lob.com. We apologize for the inconvenience and appreciate your patience.
Report: "Elevated Errors"
Last updateAll services are now operational.
On April 30, 2025 from 10:40 - 11:00 AM PST api.lob.com returned an elevated number of "Service Unavailable" 503 status code failures impacting most requests across all endpoints. This regression was caused by an update which has since been rolled back. All services are now operational.
Report: "Mail Analytics charts include no data"
Last updateWe identified the root cause of this issue and put a fix in place. Please reach out to Lob support if you continue to see issues with charts and graphs in the Lob Dashboard.
Some charts in the UI, including under the Mail Analytics tab, are including zero records. We're currently investigating the issue, and will provide an update soon.
Report: "Elevated API Errors"
Last updateBetween 7:00 AM and 8:00 AM Pacific Time on Wednesday, February 19, Lob's APIs returned 500 errors for a portion of requests. This was resolved by an automated fail-over process, and Lob engineers are currently investigating the root cause to prevent similar issues in the future.
Report: "Elevated API Errors"
Last updateThis incident has been resolved.
Lob was down from approximately 4:23 PM and 4:49 PM PT . The Lob dashboard was down and API requests may have returned a 500 during this time. We apologize for the inconvenience. All systems are operational now.
Report: "Elevated API Errors"
Last updateWe think the impact from the issue is over.
A fix has been put in place and we are monitoring the situation.
We're experiencing an elevated level of API errors and are currently looking into the issue.
A fix has been implemented for elevated API errors. We will keep monitoring to ensure the issue is fully resolved.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Elevated API Errors"
Last updateThis incident has been resolved.
A fix has been implemented for the occasional elevated API errors. We will keep monitoring to ensure the issue is fully resolved.
We've identified the cause of the elevated level of API errors and are working on a fix.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Webhook events delayed"
Last updateAll affected events have been emitted and is back to normal.
The majority of the delayed events have now emitted as expected and are working on emitting the remainder.
We encountered and identified an issue with our webhook pipeline. We are working on a backfill of the backlogged events, but are experiencing a delay.
Report: "Expected downtime in QR Codes API due to maintenance"
Last updateMaintenance is complete.
QR Codes API might experience degraded performance during maintenance window.
Report: "One-Time Campaign Delays"
Last updateWe have identified and resolved the issue with campaign delays.
Campaign processing is currently experiencing delays.
Report: "Issue uploading CSVs for Campaigns"
Last updateThis incident has been resolved.
We are currently experiencing an issue uploading CSVs. We are investigating the issue. We apologize for the inconvenience
Report: "Delays viewing Previews, Proofs and Thumbnails from the Lob dashboard"
Last updateThis should be fixed now. We apologize for the inconvenience
You might experience delays viewing Previews, Proofs and Thumbnails from the Lob dashboard. We apologize for the inconvenience.
Report: "Delays viewing Previews, Proofs and Thumbnails from the Lob dashboard"
Last updateThis should be fixed now. We apologize for the inconvenience
We are experiencing delays viewing Previews, Proofs and Thumbnails from the Lob dashboard. We are fixing the problem at this time. We apologize for the inconvenience.
Report: "Support Ticketing System is Offline"
Last updateAccess to Lobs ticketing system has been restored.
Lob's support ticket system is currently offline. We are working to resolve the issue in a timely manner and will update soon.
Report: "Increased Errors impacting mail creation"
Last updateThe issues with our 3rd party provider have been resolved, and all systems are operational.
Our 3rd party provider has recovered connectivity, errors in mail creation have recovered, and we are closely monitoring Lob systems.
We have identified an issue with a 3rd party service provider, causing issues with mail creation.
Report: "Some In Transit Event Webhooks Missing"
Last updateWe have discovered that some In Transit event webhooks that should have been triggered between April 1st and April 22nd are missing due to an ingestion issue with some In Transit tracking scan events. We have identified the root cause and have put a solution in place to alert us in the unlikely event that this condition is reached again.
Report: "Lob Print and Mail APIs and Dashboard returning 500s"
Last updateThis incident has been resolved.
Lob Print and Mail API and Dashboard were impacted between 12:23 PT to 12:54 PT. 500s may have been returned during this time
Report: "Elevated International Verification API Errors"
Last updateWe've resolved the issue and the endpoint is operational.
We've deployed a fix and are monitoring the solution.
We're experiencing an elevated level of 500 errors on our International Verifications endpoint. We are currently investigating the issue.
Report: "Degraded Performance"
Last updateFunctionality has been restored across all API endpoints. Tracking events and spending summary accuracy are up to date.
We have resolved degraded performance and continue to monitor. Please note that events or spending summary accuracy may be delayed.
We are currently experiencing degraded performance. As a result, the number of API requests per second may be impacted, and tracking event updates may be subject to delays.
Report: "Elevated API Errors"
Last updateWe're experiencing an elevated level of API errors and are currently looking into the issue.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
We're experiencing an elevated level of API errors and are currently looking into the issue. Some dashboard metrics (e.g. volume and spend) will be somewhat delayed until we can resolve the concern.
Report: "Elevated API errors"
Last updateThis incident has been resolved.
We identified an issue impacting search and list functionality. The issue is resolved and functionality is restored.
Report: "Issue Creating Resources"
Last updateThis incident has been resolved.
We have fixed the issue and are monitoring the situation
We are investigating issues creating Postcards, Letters, Checks and Self-mailers at this time
Report: "Intermittent Issues with International AV"
Last updateThe issue has been resolved.
We think the impact from this issue is over. We will continue to monitor until we are confident all customer issues are resolved.
We are still working on resolving the intermittent issues our customers have been seeing with International AV.
Customers may see some intermittent issues with our International AV services. We are actively working to resolve the issue.
Report: "Data Delays - Mail Analytics Dashboard"
Last updateAll operations have continued to show as normal.
We were able to implement a fix that brings the data within our Mail Analytics Dashboard up to date. We will continue to monitor this over the next few days.
We are still actively working with a dependent cloud provider to resolve the data delays within our Mail Analytics Dashboard. A fix is actively being worked on.
We've identified an issue with a dependent cloud provider that is currently causing data delays within our Mail Analytics Dashboard. A fix is actively being worked on.
Report: "Brief delay in rendering postcards"
Last updateAll operations have returned to normal.
We noticed that postcards were taking longer than normal to render. The times have already returned to normal. We are continuing to monitor.
Report: "Delays in processing campaigns"
Last updateThe issue has been fixed
We are investigating delays in the processing campaigns at this time
We are investigating delays in the processing of some Campaigns at this time
Report: "Data Delays - Tracking Event Webhooks"
Last updateAll tracking events have been successfully processed and are now up to date.
The issue with our dependent cloud provider has been resolved and we are now working to ensure all tracking events are up to date.
We've identified an issue with a dependent cloud provider that is currently causing data delays for our tracking events. A fix is actively being worked on.
Report: "Elevated error rates in Lob API"
Last updateThe impact has been resolved.
A fix has been implemented and we are monitoring the results.
We have identified the problem and have put a remediation in place.
We are currently seeing elevated errors in our API and investigating
Report: "Unexpected format in rendering related webhook events"
Last updateThis incident has been resolved.
We identified and fixed an issue related to webhook events that were being sent in an unexpected format. This only impacted rendering events (.rendered_pdf and .rendered_thumbnails). The events mistakenly contained tracking objects instead of tracking_events between August 10th, 3:30 PM PT to August 15th, 5PM PT. Please reach out to us via lob.com/contact if you need assistance with this issue
Report: "Tracking Event Webhooks - Naming Convention Change"
Last updateHistorical webhook events have been replayed successfully with the correct naming convention.
From 3pm PDT on July 27th to 2pm PDT on July 28th, tracking event webhook names were unintentionally shifted from 'Title Case' to 'snake_case'. A fix has been implemented for tracking events going forward, and impacted historical webhook events will be replayed.
Report: "Data Processing Delays - Reporting Tools Affected"
Last updateOur data processing infrastructure is back up-to-date.
Our data processing infrastructure is running behind which is causing inaccuracies in our volume and spend statistics on the Dashboard. No data has been lost and the system should be caught up shortly.
Our data processing infrastructure is running behind which is causing inaccuracies for usage volume and spend statistics on the Dashboard. No data has been lost and the system should be caught up shortly.
Report: "Campaign Processing Delays"
Last updateThe issue has been fixed.
We are investigating delays in processing of some campaigns uploaded to our dashboard
Report: "Issues with Address Verification endpoints"
Last updateAll Systems Operational
We were able to resolve the 500 errors at 11:40 AM PST. We will continue to monitor the situation.
We've identified an issue with a dependent system that is currently causing errors on our /v1/us_verifications , /v1/us_reverse_geocode_lookups and /v1/us_autocompletions endpoints. we are working on fixing the issue.
Report: "Rendering Delays"
Last updateSystem performance has recovered.
We've spotted that something briefly went wrong. We've already investigated the issue and rendering times have already decreased. We are monitoring to ensure it doesn't recur.
Report: "Rendering Delays"
Last updateSystem performance has recovered.
Rendering times have recovered and the system is once again performing as expected.
We are experiencing a delay in rendering times due to a delay in infrastructure scaling. We are expecting rendering times to recover shortly.
Report: "Data Processing Delays - Reporting Tools Affected"
Last updateThis incident has been resolved.
We have fixed the issue and are monitoring the sitution
We are investigating delays in processing campaigns uploaded to our dashboard.
Report: "Elevated API Errors"
Last updateAll Systems Operational
API errors and we are currently monitoring the API.
We're experiencing an elevated level of API errors and are currently looking into the issue.
Report: "Data Processing Delays - Reporting Tools Affected"
Last updateData processing for our Volume and Spend metrics should now be up to date. Thank your for patience as we resolved this concern.
Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly.
Our Lob Dashboard Volume and Spend count is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly.
Report: "International Autocomplete is experiencing issues"
Last updateWe have resolved the dependencies and service on our International Verification Product has now been restored.
We are still working on our dependency concern with our International Autocomplete API. Please advise that service to the International Autocomplete API is suspended at this time and we will provide an update here with any change to this status.
We have identified an issue with a dependency for our International Autocomplete API and are currently working on a fix. Please advise that service to the International Autocomplete API is suspended at this time. We will provide an update shortly.
We've identified an issue in our International Autocomplete endpoint. We are working on a fix.
Report: "Rendering Delays"
Last updateThe fix is in production and the render queue has been processed.
Order Renders are currently delayed to a provisioning issue. We have identified a fix and are applying it to production.
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Report: "International Address Verification is experiencing issues"
Last updateWe've fixed and verified the solution. The endpoint should now be operational.
We've identified an issue in our International Address Verification endpoint and are working on a fix.
Report: "International AV - Issue with secondary addresses"
Last updateWe think the impact from the issue is over.
We've deployed a fix and are monitoring the issue.
We've noticed that some addresses with secondary information will return "deliverable_missing_info" instead of "deliverable". We are investigating.
Report: "Issues with International AV"
Last updateWe think the impact from the issue is over.
We've deployed a fix and are now monitoring the issue.
We received reports of issues with our international address verification product. We believe we have identified the root cause and are working on releasing a fix.
Report: "International AV Requests Failing"
Last updateThis issue is now resolved.
We have identified the issue and have deployed a fix.
We are investigating an issue with our International Address Verification endpoints.
Report: "Elevated API Errors"
Last updateThis issue is now resolved.
We have identified the issue and have deployed a fix.
We're experiencing an elevated level of API errors in our International Verification API and are currently looking into the issue. We do not believe this to be a concern for other APIs, including US verifications.
Report: "Rendering delays"
Last updateThe impact from this issue has been resolved. All mail is flowing as expected.
Systems have scaled up to meet demand and orders are once again being rendered promptly.
Our systems are failing to scale up in demand to increased order volumes, and order rendering is delayed. Proof PDFs and thumbnails will take longer than usual to become visible.
Report: "Rendering delays"
Last updateThis issue has been resolved.
Order rendering times have returned to normal. We will continue to monitor system performance.
We are experiencing delays in one of our rendering components. Proof PDFs and thumbnails will take longer to become available than usual. Mail delivery will not be affected by this issue.
We are experiencing delays in one of rendering components. Orders will take longer to render than usual. Mail delivery will not be delayed by this issue.
Report: "Rendering Delays"
Last updateOur systems are continuing to render orders on time.
We have quarantined the problematic orders, and rendering times have returned to normal. We will continue to monitor system performance.
A number of orders were submitted with badly constructed PDFs that are taking excessively long times to process. We are taking remedial action to quarantine those orders and unblock processing.
Our rendering infrastructure is running behind which is causing delays in order rendering.