Is Sonar Down Right Now? Check if there is a current outage ongoing.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and currently working on implementing a fix at this time.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and currently working on implementing a fix at this time.

We are currently investigating this issue.

The scheduled maintenance has been completed.

Scheduled maintenance is currently in progress. We will provide updates as necessary.

We will be performing infrastructure maintenance in the US region. During this window, the application will experience multiple periods of unavailability, with each interruption expected to last between 10 and 15 minutes. Additionally, asynchronous workloads such as backend job processing will experience delays.Our team will work to minimize service impact and will restore full functionality as quickly as possible. Updates will be shared if there are any deviations from the expected timeline.

This incident has been resolved.

The Sonar application is currently stable and monitoring at this time.

We are currently investigating this issue at this time. Sonar is currently experiencing high load times.

This incident has been resolved.

The Sonar application is currently stable and monitoring at this time.

We are currently investigating this issue at this time. Sonar is currently experiencing high load times.

This incident has been resolved.

We are currently monitoring to reviewing root cause.

We encountered a brief outage affecting the Sonar application. Our team is actively working to isolate the issue and identify the root cause.

This incident has been resolved and all systems should be functioning normally.

The issue has been identified and the deployment has been reverted.

We are currently investigating this issue.

This incident has been resolved.

The Webhook queue is back to normal. Currently monitoring at this time.

We are currently investigating this issue. The Webhook queue is currently under high load at this time.

This incident has been resolved.

The Webhook queue is now stabilized, and we are currently monitoring.

Webhook jobs are continuing to process at an accelerated rate. Wait times are beginning to come down to under 20 minutes.

We've identified the root cause and implemented some adjustments. Webhooks are still currently experiencing delays but are now processing at an accelerated rate.

We are currently investigating a high delay with webhooks at this time.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and currently working on a permeant fix. Service is currently restored at this time.

We are continuing to investigate this issue.

We’re currently investigating elevated error rates when connecting to our V2 Web and GraphQL endpoints.

This incident has been resolved.

We experienced a temporary increase in web request latency due to multiple web pods cycling unexpectedly. This led to a brief spike in response times as our system automatically worked to restore capacity. All services are currently stable, and we are actively investigating the root cause for the web pod failures.

We have implemented a permeant fix and this incident is resolved.

We are currently seeing the FCC API endpoint have intermittent failures at this time. We have implemented a bypass. We are continuing to monitor the FCC Gov Census Check API and will remove the bypass once the FCC Gov Census Check API is stable.

This incident has been resolved.

Currently, stabilized. Multiple pods and nodes cycled due to a secret rotation.

We are currently investigating an issue with increased 503 gateway errors.

This incident has been resolved.

We are now in monitoring status. The queue has stabilized and is processing at normal wait times.

We have identified the issue and implemented a fix. We are currently monitoring the situation, and the queue size is trending downward at this time.

We are currently experiencing a high volume of webhook jobs and are actively investigating the issue.

This incident has been resolved.

The emergency failover has been successfully completed. We are currently monitoring the systems

We are currently performing an emergency failover to ensure system integrity. During this process, the application may become temporarily unresponsive.

This incident has been resolved and services are back to normal.

We believe we have resolved the issue and are continuing to monitor. Performance appears to be improving and we've confirm usage results are looking better.

We believe we have identified the issue and are attempting to resolve.

Degraded performance is still an issue so we are continuing to investigate.

A fix has been implemented and we are continuing to monitor to confirm if the issue has been resolved.

This incident has been resolved.

We are currently seeing intermittent failures on the FCC API endpoint. We have implemented a bypass. We are continuing to monitor the FCC Gov Census Check API and will remove the bypass once the FCC Gov Census Check API is stable.

We are currently seeing intermittent failures on the FCC API endpoint. We are working on implementing in a bypass for the FCC Gov Census. US address creations may fail until the bypass is enabled.

The provisioning queue issues have been resolved.

The provisioning queue has stabilized and we are continuing to monitor.

A fix has been implemented and we are monitoring the results to determine if further action is required.

We are continuing to investigate the issue with the provisioning queue.

We are currently investigating an issue with the provisioning queue, being at high volume. This is causing delays on provisioning jobs at this time.

This incident has been resolved.

We are currently seeing the FCC API endpoint have intermittent failures at this time. We have implemented a bypass. We are continuing to monitor the FCC Gov Census Check API and will remove the bypass once the FCC Gov Census Check API is stable.

This incident has been resolved. FCC Census API has been restored.

We have implemented in the bypass. We are continuing to monitor the FCC Gov Census Check API and will remove the bypass once the FCC Gov Census Check API is stable.

We are currently seeing the FCC API endpoint have intermittent failures at this time. We are working on implementing in a bypass for the FCC Census. US address creations may fail intermittently until the bypass is enabled.

This incident has been resolved.

V2 is currently up and stable at this time. We are monitoring and investigating root cause at this time.

We received multiple alerts on V2 reachability. V2 US is back up, and we are investigating the root cause at this time.

This incident has been resolved.

We've isolated the issue and the application is stable at this time.

We are currently investigating an increase in ingress errors

This incident has been resolved.

We have identified the issue and currently monitoring. The application has stabilized.

Application is currently stabilized at this time. We are continuing to investigate the issue and monitor the application.

We've isolated the issue to web pods at this time. Azure maintenance has completed, and we are still seeing failures at this time.

Azure is performing an emergency maintenance on multiple node pools. This is causing pods to hang for a few seconds while the maintenance on the node pool is being completed.

We are currently investigating an issue with application slowness.

This incident has been resolved.

We have deployed additional workers and the queue is now cleared out. We are currently investigating for root cause and monitoring the queues closely.

We are currently investigating an issue with the provisioning queue, being at high volume. This is causing delays on provisioning jobs at this time.

This incident has been resolved.

AT&T Connectivity is restoring at this time. We will continue to monitor.

AT&T is currently having ongoing issues connecting to Azure IP addresses. This is impacting customers who attempt to access V2 while on AT&T's network. We have engaged Azure support at this time.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

The webhook queue is processing at normal rate. Will continue to monitor

We've implemented a fix and currently monitoring.

The Webhook queue is slowly processing at this time. We are currently investigating this issue.

This incident has resolved.

The FCC Census API is back online. We are currently monitoring for stability before re-enabling it within the application.

No ETA from FCC. Bypass is still enabled at this time.

We are currently seeing the FCC API endpoint down at this time. We have implemented in a bypass at this time to allow serviceable addresses to be created.

This incident is resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue and working on a hotfix to resolve this issue.

We are currently investigating an issue with slow response times within the application.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating an issue with slow response times within the application.

This incident has been resolved.

Currently, the provisioning queue is stable. We are closely monitoring the queue.

The Provisioning queue has caught up. We are still investigating the root cause of this issue.

We're still processing at a steady rate and seeing wait times drop as well. Continuing to investigate the root cause.

We're still processing at a steady rate and seeing wait times drop as well. Still investigating the root cause.

We're still processing at a steady rate and seeing wait times drop as well.

We're still processing at a steady rate and seeing wait times drop.

We're still processing at a steady rate and seeing wait times drop.

The provisioning queue is continuously dropping at a steady rate, with wait timers dropping as well.

We are continuing to investigate the root cause. At this time, provisioning queue wait times have dropped. We'd advise not to run a re-sync as this will continue to increase the queue.

We are continuing to see the provisioning queue process at a high rate.

The queue is still processing and going down. We are still investigating the root cause of this issue at this time.

We have increased the provisioning queue workers while investigating the root cause of this issue.

We are currently seeing an increase in job count for provisioning queue. This is impacting wait times for jobs within this queue to complete.

This incident has been resolved.

Webhooks queues have cleared out and processing at a normal rate. We will continue to monitor

The Webhook queue is now under 15 minute waits for processing. We are working to identify the issue and implement a permanent solution for this issue.

We are currently seeing a delay in webhook processing at this times.

Snowflake has restored. We will continue to monitor.

Snowflake has begun to implement a fix, and we are starting to see services restore for reporting. We will continue to monitor and work with snowflake until this issue is fully resolved.

The Snowflake service provider has identified an issue which is causing data not to load in reports. Snowflake is working to resolve this issue and is currently working on implementing in a fix.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue causing the reporting module not to load. We are working on implementing a fix at this time.

We are continuing to investigate this issue and working with looker to resolve this issue.

We are currently investigating an issue with the reporting module not loading for users.

This incident has been resolved.

All services are back up and we are continueing to monitor at this time.

Application is now back online. We are bringing up back job processing and continuing to monitor closely.

We are still actively working on bringing up services. Backend Services have been brought back online and working, turning the application on.

We have received additional updates from Azure and seeing services restored at this time. We are working on restoring services at this time and validating backend services.

We are starting to see some backend services from Azure restored at this time. Once all Azure services restored, we will begin to bring the app back online.

This is still an ongoing issue. Azure is experiencing a complete region outage at this time, impacting multiple services. No ETA on restoration from Microsoft at this time.

We are still experiencing an outage due to Azure issues. We are working closely with Microsoft for updates

Azure CentralUS is now majorly impacted at this time, causing large-scale failures.

We have begun to experience additional failovers caused by Microsoft services being impacted.

We have received monitoring alerts for Azure failures. This caused some failover with a brief hit causing pages to stop loading briefly.

FCC bypass removed and data fix completed. We will continue to monitor the FCC Census Server endpoints.

Bypass has been implemented. We will continue to monitor the FCC Census servers.

We are currently seeing the FCC API endpoint down at this time. This is impacting address creation for US addresses at this time.

This incident has been resolved.

The Autopay queue has cleared out at this time. We are monitoring and reviewing what caused the high latency.

We are currently experiencing slower than normal processing times for autopay.

This incident has been resolved.

FCC API endpoint is back online. We are removing the bypass and will continue to monitor. We will begin to update the addresses in the backend that were created during this incident.

Verified fix has been implemented in, and US address creation is working. We will continue to monitor the FCC Census servers.

We have identified the issue and currently implementing in a fix.

We are currently seeing the FCC API endpoint down at this time. This is impacting address creation for US addresses at this time.

This incident has been resolved.

We are seeing the errors clearing out and currently monitoring.

We are currently seeing an increase in errors/timeouts accessing V2 US application. We have begun to investigate the issue at this time.

V2 in the US region experienced slowness that caused the application to become unresponsive. The application is currently stable and we are investigating the root cause.

This incident has been resolved.

We currently see autopay queues processing at normal rate. We are currently monitoring and reviewing logs to isolate the root cause.

The Autopay queue is almost cleared up. We still see higher than normal processing times with autopay, and currently working on isolating this issue.

We are continuing to investigate this issue. Autopay jobs are still running and autopay run latency spikes have gone away.

We're currently experiencing longer processing times for billing at this time. This is causing high wait times for autopay and invoice generation.

This incident has been resolved.

A fix has been implemented, and we are currently monitoring.

The issue has been identified, and a fix is being implemented.

Our team is working diligently with the Email Transactional Provider to identify the root cause and implement a resolution as quickly as possible.

IPPay has resolved this incident.

Please be advised that the IPPay gateway is experiencing technical difficulties. This may result in delays or errors when attempting to tokenize or create payments. IPPay is investigating the issue and working towards a fix. We will share updates as we receive them.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified an issue and currently working on a fix. V2 UI is currently down.

This incident is resolved. Sonar is currently beginning the cleanup process due to this incident. All ACH Bank processing are working correctly now.

Auto-Pay scheduled for today through Sonar instances utilizing the following bank account processors (BMO Harris 80 byte Canadian ACH format, 1464 byte Canada ACH, Canada ACH for Central 1, NACHA ACH, Canada ACH for Royal Bank of Canada, Canada ACH FTP Upload for Royal Bank of Canada) are experiencing instances of duplicated charges in Sonar. Sonar has temporarily disabled the auto-pay function for the affected process and is actively addressing the issue. It's important to highlight that eCheck and Credit Card payments remain unaffected by this incident.

This incident has been resolved at this time.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

We identified an issue with billing/autopay not processing correctly. Sonar performed a hotfix for a bug discovered in the last deployment. Sonar is continuing to monitor closely.

Due to the incident at night job processing is currently delayed causing billing and auto-pay to be delayed in processing. Sonar is working to get these jobs processed as quickly as possible.

Backend Redis Clusters experienced a failover and out of memory errors. This caused both cache and job processing to halt, impacting US V2 accessibility and job processing intermittently.

This incident has been resolved.

We are monitoring the results. We increased the pod count and still investigating the root cause for this spike.

We are currently investigating the issue reported with general slowness and GraphQL errors

This incident has been resolved.

All services have restored, and we are continuing to monitor. Azure experienced a large-scale outage in the region, causing multiple nodes to fail.

The service provider has begun restoring service.

We are currently investigating issues with our cloud service provider.

This incident has been resolved.

We received alerts of V2 US region accessibly briefly while the cache cluster failed over. Azure performed an unplanned restart due to managed disk connectivity within the availability region on the cache cluster. No data was lost during this failover, as only sessions are stored.

This incident has been resolved. Sonar performed a fix on any US addresses missing a FIP and is continuing to monitor.

FCC Gov Census Block Server is back online. We have turned off the bypass and validate U.S. addresses are validating against the FCC Gov Census Block Server. Sonar is working on updating the FIPS data for U.S. serviceable addresses created during the bypass.

We have implemented a workaround that restores U.S. serviceable address creation until the FCC Gov Census Block Server is back online. U.S. serviceable addresses created during this period will be missing FIPS data, which is used to calculate county taxes. Once the FCC service has been restored, Sonar will perform a correction on those addresses missing FIPS data.

We are currently working on a hotfix to bypass the FCC validation. A workaround, you can add an account with an inactive/lead status without a serviceable address and place this address in the notes.

FCC has been notified and has created an internal ticket and escalated the ticket within the FCC for the server being down. No ETR at this time.

We have received an alert that the FCC Gov Census Block Server is offline. Sonar is reaching out to the FCC at this time. This is currently impacting US addresses from being created in the system.

This incident has been resolved.

A fix has been implemented by IPPay and currently monitoring.

Per IPPay they are actively moving customers off this bank database. Per IPPay a small number of merchants use this bank.

IPPay was notified by their bank that their database is down, preventing transactions from processing. They are actively investigating the issue. We have not been provided with an ETA on when services will be restored. We will provide you with updates as they become available to us.