Is Restaurant365 Down Right Now? Check if there is a current outage ongoing.

We are currently investigating intermittent connection troubles to FTP services, affecting EDI transmissions.

We are currently investigating an issue where a certain navigation menu is missing from Custom Financial Reports for some customers.

We still continue to see recovery and will now consider this incident resolved. 8:30am PST: Slowness reported 9:45am PST: Recovery and return to service 10:48am PST: Incident resolved

We are now seeing recovery as of 9:45 PST. We will continue to monitor this incident before fully resolving.

We are currently investigating reports of slower than normal load times.

We still continue to see recovery and will now consider this incident resolved. 8:30am PST: Slowness reported9:45am PST: Recovery and return to service10:48am PST: Incident resolved

We are now seeing recovery as of 9:45 PST. We will continue to monitor this incident before fully resolving.

We are currently investigating reports of slower than normal load times.

The incident is now resolved as we continue to see recovery.

A fix was implemented and we continue to see recovery. We will continue to monitor the fix implemented to ensure the issue doesn't persist before resolving.

We are currently investigating reports of issue with running My Reports

The incident is now resolved as we continue to see recovery.

A fix was implemented and we continue to see recovery. We will continue to monitor the fix implemented to ensure the issue doesn't persist before resolving.

We are currently investigating reports of issue with running My Reports

This incident has been resolved.

This incident is resolved.

A fix has been implemented and we are seeing recovery. We will continue to monitor the fix implemented before resolving this incident.

We are currently investigating the issue preventing files to import into R365.

This incident has been resolved.

This incident is resolved.

A fix has been implemented and we are seeing recovery. We will continue to monitor the fix implemented before resolving this incident.

We are currently investigating the issue preventing files to import into R365.

The fix implemented resolved the issue for the connection failure. We will mark this as resolved.

A fix was implemented and we are now seeing recovery as files are now able to import. We will continue to monitor the fix implemented before fully resolving this incident.

FTP is down. Reports of files being unable to import into R365. We are currently investigating the issue.

This incident has been resolved.

A fix has been implemented, and we are seeing recovery. We will continue to monitor.

We are investigating reports of some DSS's not importing.

We continue to see recovery and will now mark this issue as resolved.

The fix was deployed and we are now seeing recovery.

We have identified the root cause of the Expandshare outage and we deploying the fix.

The deployed fixed has resolved the missing Brink DSSs, and we're seeing recovery.

We are continuing to monitor for any further issues.

A fix has been implemented, and missing DSSs are importing as expected.

The root cause of missing DSSs has been identified, and a fix is being prepared.

We are currently investigating reports of DSS's not importing for some Brink POS customers.

`Root Cause Analysis (RCA) - Incident Title: ` `Missing/Delayed DSS ` `Date: 3/14/2025 - 3/17/2025 ` ` ` 1. Summary `› Incident Overview: Between March 14 and March 17, 2025, multiple related incidents disrupted the DSS ` `(Daily Sales Summary) used by customers to view sales and labor data. On March 13, a new pipeline for the ` `Comida Director was deployed and 3700 locations were migrated, which triggered a series of issues. A ` `combination of a code change, a later Customer Setting Service (CSS) outage and an increase of locations ` `using this application, caused the Director to become overwhelmed by the increased load. This resulted in a ` `backlog of missing DSS records and inaccurate reporting. ` `› Timeline ` `o February 2, 2025: ` ` Comida Director upgrade began rolling out to various locations in a structured approach. ` `o March 13, 2025: ` ` 4:45 pm: Pipeline for the Comida Director deployed to production. ` ` 7:00 pm: 3700 new locations were migrated to the Comida Director. ` `o March 14, 2025: ` ` 4:05 am: Alerts triggered for missing DSS. ` ` 4:30 am: Investigation commenced. ` ` 6:30 am: Peak missing DSS count reached 1,930; the problematic deployment from 3/13 ` `was reverted. ` ` 6:45 am: Recovery steps initiated. ` ` 9:15 am: A 911 call was started to identify a solution for improved recovery time. ` ` 9:31 am: Manual intervention further accelerated recovery. ` ` 10:05 am: Recovery reached acceptable levels; the incident call ended. ` ` 11:30 am: Full recovery achieved. ` `o March 16, 2025: ` ` 2:17 am: Customer Setting Service (CSS) went down. ` ` 2:27 am: CSS came back online. ` ` 2:30 am: 1,097 missing DSS records were noted. ` ` 2:42 am: CSS experienced another outage. ` ` 3:10 am: A post in the 911 channel indicated CSS downtime and failing imports.` `2 ` ` 3:17 am: CSS was restored. ` ` 7:00 am: Peak missing DSS count hit 6,286; manual steps using a local version of poolx ` `were initiated. ` ` 7:15 am: Early recovery signs observed with numbers slowly dropping. ` ` 8:00 am: 911 channel updated that support cases were still coming in and imports were ` `lagging. ` ` 8:02 am: It was confirmed that DSS records were loading—albeit very slowly. ` ` 11:24 am: Missing DSS count dropped to approximately 2,500. ` ` 12:50 pm: Full recovery was reached. ` `o March 17, 2025: ` ` 5:15 am: Degradation began with an alert for 744 missing DSS records. ` ` 7:30 am: Missing DSS count peaked. ` ` 7:57 am: A 911 call was initiated regarding the issue. ` ` 8:00 am: Manual intervention using poolx locally commenced. ` ` 8:30 am: DevOps scaled up the minimum replicas, allowing more than one Comida ` `Director instance to run (without effect). ` ` 8:40 am: An index was applied to the PosSystemTenantLocationDss collection (without ` `effect). ` ` 9:15 am: DSS team continued investigation. ` ` 9:36 am: Missing DSS count decreased to 894. ` ` 10:05 am: Missing DSS count further dropped to 122. ` ` 10:30 am: Full recovery was confirmed; concurrently, with Arch’s assistance, the team ` `identified the true root cause for all incidents over the last 4 days, which was related to ` `the Director not being able to scale as expected. The Director was processing only 10 ` `locations at a time. ` ` 11:00 am: A three-point action plan was formulated: ` ` Fix the code to increase parallelism from 10 to 100. ` ` Revert the additional 3700 locations migrated on March 13. ` ` Develop an app/process to relieve pressure on the Director (using poolx with ` `manifest). ` ` 4:40 pm: The merge request (MR) to increase parallelism to 100 was merged. ` ` 5:30 pm: The pipeline for the parallelism change was deployed to production. ` ` 5:40 pm: Logs confirmed the positive effect of the change. ` ` 6:15 pm: The Director had already created 1,000 jobs since the deployment. ` ` 10:00 pm: The extra locations added on March 13 were rolled back. ` `› Immediate Impact: Customers experienced missing or delayed DSS imports, leading to inaccurate sales ` `and labor data in reports. This disruption resulted in increased support cases and a degraded customer ` `experience over multiple days.` `3 ` Scope `› Affected Environments: All environments (CAN1, CAN2, Production, Post Production) ` `› Geographical Impact: N/A ` `› Service Impact: ` `o Primary: Daily Sales Summary (DSS) – missing or delayed data imports. ` `o Secondary: Comida Director – inability to process increased load leading to cascading failures. ` `o Tertiary: Customer Setting Service (CSS) – intermittent outages further compounded the issue. ` ` ` 1. Root Cause `› Cause Identification: The incident was triggered by a combination of a code change and the deployment ` `of a new pipeline for the Comida Director on March 13, which migrated 3700 locations. This, along with a ` `subsequent CSS outage and an increased number of locations using the application, caused the Director to ` `become overwhelmed. ` `› Technical Explanation: The root cause was determined to be the Comida Director’s limited parallelism—it ` `was processing only 10 locations at a time, which was insufficient to handle the increased load. This ` `limitation, compounded by CSS outages and the additional load from the new locations, led to a backlog of ` `missing DSS records and inaccurate reporting and Comida Director could not keep up. ` `› Contributing Factors: ` `o Increased load from the migration of 3700 locations. ` `o CSS instability with multiple outages on March 16. ` `o The initial code change and its subsequent reversion did not resolve the underlying scalability ` `issue. ` ` ` 1. Impact Analysis `› Number of Cases: ` `o March 14: 120 reported cases ` `o March 16: 92 reported cases ` `o March 17: 91 reported cases ` `› Severity of Impact: Severity 1 ` ` ` 1. Corrective Actions `› Immediate Fixes: ` `o March 14:` ‌ ` ` ` ` `4 ` ` Reverted the code change deployed on March 13. ` ` Executed manual interventions to process missing DSS records. ` `o March 16: ` ` Conducted manual recovery steps using a local poolx instance during CSS instability. ` `o March 17: ` ` Initiated early manual interventions and attempted scaling up Director replicas (which did ` `not resolve the issue). ` ` Applied an index to the PosSystemTenantLocationDss collection (without effect). ` ` Increased parallelism from 10 to 100 and rolled back the extra locations migrated on ` `March 13. ` `› Root Cause Resolution: A change was pushed across all environments to increase parallelism from 10 to ` `100, with a subsequent change increasing parallelism from 100 to 200 deployed on March 18 and an ` `increase from 200 to 400 on March 20. Additionally, the extra locations added on March 13 were rolled ` `back. As of March 26, parallelism has been increased to 800, with success. ` `› Communication: Real-time updates were communicated via the 911 channel and the status page was ` `used to update customers on March 14 and March 17. ` ` ` 1. Preventative Measures `› Process Improvements: ` `o Implement rigorous load testing in a dedicated dev environment to simulate production load. ` `o Establish a canary environment for the Comida Director to test code changes with a subset of ` `customers before full production rollout. ` `› System Enhancements: ` `o Develop an application/process to relieve load pressure on the Director using poolx with manifest. ` `o Continuously adjust and optimize the parallelism settings, as needed. ` `› Monitoring and Alerts: ` `o Continue to monitor ongoing throughput to proactively identify and adjust additional parallelism ` `configuration, as needed. ` ` ` `Document Approval: ` `› Prepared by: [Jeremy Eubanks, Associate Manager Information Security, 3/26/2025] ` `› Reviewed by: [Lauren Harden, Sr. Director Database, DevOps, Information Security, 3/26/2025] ` `› Approved by: [Jack Mossman, Senior Director, Integrations Technology, 3/28/2025] ` ` ` `Distribution: ` `› Intended Audience: Engineering teams, DevOps teams, and affected customers.`

As we continue to see recovery, we will now considered this to be resolved.

We are now seeing recovery after implementing a fix. We will continue to monitor and provide additional updates.

We are currently seeing degraded performance which is causing delays with DSS imports. While we continue to investigate the root cause of the issue, we will continue to provide updates.

A fix was deployed and we continue to see recoveries. Issue is resolved.

We were able to identify the root cause and we are now implementing the fix.

We are currently investigating reports of the Location field not showing when creating AP Payments from Bank Activity along with a Guid Error. We will continue to provide updates as we continue our research.

As we continue to see recovery, we will now mark this as resolved.

The fix has been deployed and we continue to see recovering.

We have identified the root cause and are working on the resolution.

We are currently investigating reports of Missing DSS. We will continue to provide updates as we investigate further.

The R365 App is now available within the Google Play store.

Our team has been notified that the R365 Red App is currently unavailable in the Google App Store. We are actively working with Google to investigate the reason for the app's removal and to have it restored. The current version of the app, 10.18.0(7320), is functioning properly at this time. Please do not delete the app in an attempt to resolve any issues you may be experiencing

We continue to see recovery with the fix implemented. The issue is now resolved.

A fix has been implemented and we are now seeing recovery. We will continue to monitor the results and will provide further updates.

We have identified the issue and are implementing the fix.

We're investigating reports that some Daily Sales Summaries are missing Labor Details. We will continue to provide updates as we investigate further.

Full recovery across all instances. Issue resolved. Postmortem Summary: 2:13 pm PST - Issue reported - 504 Gateway Time-Out Error 2:25 pm PST - Fix implemented 2:33 pm PST - Full recovery across all instances

A fix has been implemented and we are now seeing recoveries across all customer instances.

We are continuing to work on a fix for this issue.

We have identified the issue that is causing the 504 Gateway Time Out Error and are currently implementing a fix.

9/17/2024: At 1:30am PST, an issue was identified that prevented the import of the Daily Sales Summary. At 5:15am PST, the issue was resolved we then began to see recovery along with a downward trend in missing DSS's. At 8:30am PST, our system was fully stabilized.

The connectivity issues have been resolved. Microsoft Azure cleared the issue on their end and customers have confirmed service is restored.

At 5am PST we became aware of connectivity issues being reported . The issue is affecting customers using AT&T for network connection. Microsoft Azure is aware and their status page can be viewed here: https://azure.status.microsoft/en-us/status . We recommend using an alternate ISP for internet connection in the interim. We will continue to provide updates as we receive them from Microsoft Azure.

Customers unable to login to the R365 Application via either the desktop or mobile devices Start Time: 8/26/24 8:52 PM PST End Time: 8/26/24 9:34 PM PST

We continue to see full recovery after the fix was implemented and consider the issue to be fully resolved.

Just a short time ago, we began to see overall system slowness which also began to trigger some error messaging. We swiftly identified the root cause and were able to implemented a fix. While we are seeing recovering, we will continue to fully monitor the fix implemented to ensure system performance is no longer degraded.

This incident has been resolved.

We are continuing to monitor for any further issues.

Microsoft Azure updated their status post (https://azure.status.microsoft/en-us/status/history/ then expand July 18) to give information on the cause of the incident and communicated that a majority of their services are now recovered. R365 application may still experience some latency as our systems work to catch up and there may be a delay in DSS and some report data. We are continuing to monitor our system as we return to full operational capacity. R365 application is also recovered from the Crowdstrike outage, and we are continuing to monitor and verify all infrastructure has healed completely and there is no residual impact.

We are continuing to see recoveries from Azure's outage across our application. Azure has stated that the majority of services are fully recovered, however some services are still impacted and we may still receive intermittent HTTP 500-level response codes, experience timeouts or high latency. Microsoft Azure's status post indicates their incident is still active. We are continuing to monitor Azure's status updates and are actively monitoring the state of our platform. We were additionally impacted by the Crowdstrike outage which caused unexpected reboots, machines crashing and being unable to stay online. Our team is also working to mitigate these issues

We are beginning to see some recoveries. We are continuing to monitor and verify all resources are fully operational. There may still be intermittent issues and/or degraded performance at this time

Update from our Cloud Service Provider: We’ve determined the underlying cause and are currently applying mitigation. We will start to see incremental recovery in next 90 minutes. We will continue to provide updates as they are available.

Our Cloud service provider is currently experiencing an outage that has impacted all R365 customers ability to access their database. Our engineers are monitoring the status of this issue with our Cloud Service provider, and we will provide updates as they are available.

Our Cloud service provider is currently experiencing an outage that has impacted all R365 customers ability to access their database. Our engineers are monitoring the status of this issue with our Cloud Service provider, and we will provide updates as they are available.

Our Cloud service provider is currently experiencing an outage that has impacted all R365 customers ability to access their database. Our engineers are monitoring the status of this issue with our Cloud Service provider, and we will provide updates as they are available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating the issue of getting an error when attempting to log into a database.

R365 APP force update has been deployed. Issue Resolved.

An APP update was expediated to the Apple Store and deployed. Once updated to the newest version, the APP is showing signs of stability. We will continue to monitor the results.

The R365 APP issue has been identified and our fix has been expedited to the Apple store for review. The Apple store review process could take a matter of days for deployment. Once the fix has been cleared by Apple, we will update this messaging.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We have identified the issue and are implementing a fix.

This incident has been resolved and customer should no longer experience any issues with the Payroll Overview Screen loading.

We have had reports of the Payroll Overview screen not loading properly and appearing blank. Our Development team is aware of this issue and actively working to resolve. We recommend refreshing the page every few minutes to capture any fixes that are immediately made, and will update this status once the issue has been fully resolved.

This incident is now resolved.

A fix was implemented and we are monitoring the results.

We are currently investigating this issue.

This incident is resolved

As we continue to monitor the fix that was implemented, daily sales imports could be slightly delayed.

A fix has been implemented and we are monitoring the results.

We are currently investigating the issue.

This incident is resolved.

A fix has been implemented and we are monitoring the results.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating an issue with users being unable upload files to DTP.

At this time, the matter has been fully resolved.

Our Support Center functionality has been restored. We will continue monitoring the situation closely. We appreciate your patience during this incident.

Our Support Center is currently unavailable due to this issue. We are actively addressing the matter with our provider and will continue to update this page.

Due to an issue with our provider, we are currently unable to respond to support tickets. If you need urgent help, please contact us via live chat. No tickets have been or will be lost due to this issue and we will address them as soon as we're able. We apologize for the inconvenience.

Our provider has confirmed the outage has been resolved. All R365 services and sites should be available and functioning properly at this time.

Our provider has confirmed the outage and an active investigation. We will continue to deliver updates here as they become available.

We are investigating a major outage with our cloud provider. Several R365 services and sites are currently unavailable.

This incident has been resolved.

We are continuing to investigate this issue.

Our direct bank connection partner Yodlee is currently down. This is preventing the majority of R365 customers to extract bank data. We are working with them in hopes to have the issue resolved soon.

This incident has been resolved.

We will continue to monitor the intermittent connections.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue

At this time, we consider the matter resolved. If you are unable to access the system, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

A fix has been implemented and R365 Intelligence is now available.

R365 Intelligence is currently unavailable. Our Engineering team is diligently working to restore access. We will post updates here as we make progress.

This incident is resolved.

A fix has been implemented and repolls are being preformed to update the DSS data being imported.

This issue has been identified and we are implementing a fix.

We are currently investigating an issue which is preventing Toast POS polling data from importing correctly.

At this time, we consider the matter resolved. If you are unable to access the system, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

At this time, access has been restored. Our Engineering team is monitoring performance closely. If you are unable to access the system, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

R365 Workforce is currently unavailable. Our Engineering team is diligently working to restore access. We will post updates here as they we make progress.

At this time, we consider this issue to be resolved. If you experience issues with Daily Sales Summary polling at one or more of your locations, please reach out to our Support team and we will be happy to investigate: https://help.restaurant365.net/en/support/home#contact

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

Our development team is running further test as we work towards a resolution.

We are continuing to work on a fix for this issue.

The issue has been identified and we are working on a resolution. Updates to come.

We have verified that our fix from earlier this morning has resolved the issue for all affected customers. You may still need to repoll your POS data and/or re-trigger daily report subscription emails. For repolling instruction, see https://shorturl.at/GITY5 For instructions on resending a report subscription, see https://shorturl.at/sAEOX If you need assistance with either of these procedures, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

Overnight we became aware of an issue affecting the DSS file imports for some customers that prevented sales and labor data from importing correctly. Our engineering team quickly identified the problem and implemented a fix. At this time, we believe the issue to be resolved, but you may still need to repoll your POS data and/or re-trigger daily report subscription emails. For repolling instruction, see https://shorturl.at/GITY5 For instructions on resending a report subscription, see https://shorturl.at/sAEOX If you need assistance with either of these procedures, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

We have verified resolution for nearly all impacted users. If any members of your team are still experiencing missing functionality that should be present in accordance with their user permissions, please contact our Support team for further database-specific troubleshooting: https://help.restaurant365.net/en/support/home#contact

At this time, we believe all affected database permissions have been restored and we are monitoring system performance closely. If you experience any further related issues, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

Our Engineering team has identified the source of the problem and is working on a solution. We expect the recovery process to begin shortly, but it may take some time to impact all affected databases. We will continue to deliver updates here as they become available.

Some customers are reporting inability to access parts of the system that should be accessible to them based on their user permissions. Our Engineering team is currently investigating. We will post updates here as they become available.

This incident has been resolved.

The fix was implemented and we monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating the issue.

We have received no further reports of login errors since our last update. If you experience an error, please clear your browser cache or access the login page via a private browser window. If you are still unable to log in after doing so, please contact our Support team: https://help.restaurant365.net/support/home/#contact

At this time, we believe the matter is now resolved for all affected customers. Some users may have to clear their browser cache or access the login page via a private browser window. If you are still unable to log in after doing so, please contact our Support team: https://help.restaurant365.net/support/home/#contact

Our Engineering team has taken corrective action and we are currently seeing service recoveries for some customers. We are continuing to address the issue as a top priority.

Some customers are receiving error messages when attempting to log in to their databases. Our Engineering team is aware of this issue and is working on it as a top priority. We will post updates here as they become available.

Our engineering team has resolved the issue.

We are experiencing intermittent system issues that are impacting some customers' ability to access R365. Our engineering team is addressing this matter as a top priority. We will post updates here as they become available.

This incident has been resolved.

Our 3rd party aggregator, Yodlee, is experiencing network connection issues preventing the retrieval of bank activity for R365 customers. Yodlee is working on correcting the issue at their highest priority to hopefully have this issue resolved soon.

At this time, we consider the matter resolved. If you are having trouble accessing your database, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

At this time, all reported outages have been restored. If you are having trouble accessing your database, please contact our Support team: https://help.restaurant365.net/en/support/home#contact Our Engineering team will continue to monitor the network to ensure continued performance integrity.

Our Engineering team has verified a network disruption that has now stabilized. We are beginning to see service recovery for affected customers, but we are continuing to diagnose and monitor.

We have received reports of outages affecting some of our customers. Our Engineering team is currently investigating the issue as a top priority. We will post updates here as they become available.

This incident has been resolved.

We have received reports of some Android users being unable to log into the Red App. We are currently investigating and will update this page as more details become available.

This incident has been resolved.

A fix has been implemented and Toast POS data is now importing. We will continue to monitor until we are confident this is fully resolved.

We are continuing to investigate this issue.

We are currently investigating an issue preventing Toast POS data from auto importing

This incident has ben resolved. Changes were implemented to ensure successful imports daily from Toast POS.

We are continuing to work on a fix for this issue.

Due to a Toast update last night, DSS files for Toast users this morning are missing sales data. We are aware of this issue and our Engineering team is working both internally and with Toast representatives on a solution. If any further action is required on the part of Toast users, we will post details to this page.

This incident has been resolved.

The development team has identified the issue here and deployed a fix. You should now be able to successfully export new hires to Payroll. If you are still seeing any issues with this process please reach out to our Support team at https://help.restaurant365.net.

Our team has been made aware of an issue when attempting to export completed new hires to Payroll. Customers are currently receiving the following error: "R365 Export Error: Connection Reset By Peer. Please contact Payroll for more information." The issue is currently being investigated by our Development team and we will update this page as we have more information.

This incident has been resolved.

At this time, a fix for the issue with pages not loading has been implemented and we are monitoring performance. If you are still experiencing access issues, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

We have received reports of certain pages within R365 failing to load (Smart Ops, AP Center, Documents to Process). We are currently investigating and will update this page as more details become available.

We have seen no further issues with Multi-Instance Switching. If you experience an issue with the feature, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

At this time, Multi-Instance Switching access has been restored and we are monitoring performance. If you are still experiencing access issues, please contact our Support team: https://help.restaurant365.net/en/support/home#contact

Our Engineering team has identified the cause of the issue and is currently working on a fix.

Customers that use Multi-Instance Switching are reporting an inability to access the feature. We are currently investigating and will update this page as more details become available.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented by GoDaddy and we are monitoring the results. All sites should now be accessible.

Our DNS provider, Go Daddy, is currently experiencing an outage that is affecting our DNS servers and customer access to databases. Our engineers are in contact with Go Daddy and working on the issue. We apologize for the inconvenience.