Is Aryeo Down Right Now? Check if there is a current outage ongoing.

We have identified a workaround the current GCP outage, and the service capabilities have fully recovered. We are monitoring for continued impact.

We are still investigating impact numbers and severity of the issue. We are observing that download capabilities are failing. Part of this infrastructure is powered by Google Cloud, which is experiencing a major outage at this time, and is the root cause of these issues.

This incident has been resolved.

Marketing material exports are experiencing an issue and the engineering team has engaged and is investigating.

A recent update to Aryeo's infrastructure was deployed to improve application latency. This led to errors in the application for approximately 3 minutes, as that configuration applied. These errors were resolved immediately by the engineering team, restoring application accessibility with the intended improved latency.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

Microsoft's public status page still identifies an issue: https://portal.office.com/servicestatus.

Microsoft has updated their public status page with the issue: https://portal.office.com/servicestatus. We are monitoring closely for a resolution.

On Monday morning, there were widespread reports that Microsoft's calendar services have been down or degraded. See report here in USA Today: https://www.usatoday.com/story/tech/news/2024/11/25/microsoft-365-outlook-down-outage/76567014007/. This will affect any companies that integrate with outlook for their calendar integrations. Customers may see slower loading times when booking new orders as a result, and access to your teams external availability may be slow on calendar pages. The Aryeo team is monitoring these reports, and will provide updates here following a resolution from Microsoft.

The issue with our 3rd party vendor has now been resolved. All media downloads and gallery views should be performing as expected.

A 3rd party Aryeo uses for downloading media assets is experiencing degraded performance. This is affecting the ability to download image assets uploaded in the last 12 hours. Media assets uploaded before then can be viewed and download as expected. Importantly, uploaded media assets are stored and saved successfully. Files are not a risk of being "lost". Users can continue to upload media assets as needed for business operations. It's the download of those assets that's currently affected.

The issue has been resolved by our 3rd party vendor.

An issue has been identified with our 3rd party vendor supporting listing media upload from Google Drive and OneDrive. We have engaged their team and are working to resolve the issue. Standard media upload and "Add from Dropbox" is unaffected.

On thursday morning, 10/17, Aryeo completed the full roll out of our new customer team UIs. The full release notes are captured here: [https://aryeo.canny.io/changelog/feature-release-customer-teams](https://aryeo.canny.io/changelog/feature-release-customer-teams). # Summary Of Customer Teams Functionality In summary, [since the launch of Aryeo 2.0](https://www.youtube.com/watch?v=9QGJxxep600) in the summer of 2020, Aryeo has supported teams for agents. However, these teams were restrictive, allowing you to only grant access to one agent at a time. If # Migration Requirements In order to roll out customers teams and unify Aryeo’s existing teams functionality with that of customer groups, changes to Aryeo’s UIs were necessary. ‌ ‌ ‌ Displaying a team that was not historically visible in the UI would be confusing.

As part of the roll out of the new customer team UIs, many agents reported difficulty access this customer portals and logging in. The team is investigating.

This issue has been resolved

We are continuing to investigate this issue.

We are currently investigating this issue.

The issue with the recent deployment has been fixed.

The issue has been identified and a fix is being implemented.

We've identified the cause of the issue as a recent deployment. Correcting the issue.

We are continuing to investigate this issue.

We are currently investigating this issue.

Aryeo engineering was alerted to a significant increase in request volume, caused by a malicious actor. This increase in request volume led to overall slowness of the platform's performance and intermittent Cloudflare errors as users attempted to access the site. Aryeo's engineering team blocked this actor, returning the platform to nominal performance.

On Saturday morning, an automatic security update to some of Aryeo's infrastructure introduced some unintentionally upgrades to our underlying services. As soon as the updates were completed, Aryeo's engineering team was alerted to the issues and jumped in to investigate the cause. Photography companies began experiencing 500 errors in their dashboard as well as 500 errors in their customer portal. Aryeo engineering identified the root cause and was able to roll back the changes, returning the system to fully operational. To prevent this in the future, we completed a full incident analysis and are auditing our infrastructure this week to prevent unintentional upgrades, with full system reliability as the sole goal.

Dropbox resolved this issue yesterday and after monitoring full functionality has returned. Please reach out to support if you notice any other issues.

Dropbox is currently having errors with their API. This is resulting in issues for users to upload from Dropbox into Aryeo. We'll keep our customers notified here as we know more information, and we'd recommend to upload files via our standard uploader while their issues are resolved.

Earlier this afternoon we were notified of a system outage by both customers and our internal monitoring tools. Upon investigation, our server was being bombarded with web scraping requests from a number of other countries. This resulted in either slow or non-loading pages for our users. Once identified we blocked the IP addresses in questions and rebooted our servers, returning operations. Going forward we are adding in additional automated tools to help stop future scraping attempts like this before they occur. Thank you for your patience as our team resolved this today.

The incident has been resolved.

We are currently investigating an issue with the Aryeo web application and API.

Issue has been resolved. Aryeo team is working on migrating off of Nylas infrastructure as soon as possible to prevent future issues like this.

Nylas has updated their status page that their issues begun to resolve and they are monitoring closely. We are seeing API times back to normal from our side. Aryeo is automatically handling re-syncing appointments, and we are working on ways to help notify accounts that may be impacted by double bookings.

We are continuing to monitor and will re-enable as Nylas has resolved their API issues.

Our calendar provider (Nylas), is currently experiencing delays with their calendar API. As a preventative measure we've paused functionality with external calendars to ensure clients can continue to book and view events. Note, as of right now you will not see external events on your calendars, and new events won't be sent to external calendars. As soon as Nylas comes back online we will automatically sync all of these events back to everyone's calendars.

We've upgraded our download servers and are monitoring download speeds. Downloads should be begin significantly faster than before.

We have confirmed that some downloads are taking longer than normal to begin the download/zipping process. We are rebooting the download servers and investigating the root cause.

There have been some initial reports of slow download speeds. Our team is investigating.

This incident has been resolved.

Calendar syncing services are back online and appointments scheduled this evening are being successfully resync'd to Google and Outlook calendars. This process should be complete within the next 30 minutes. We are continuing to monitor for any further issues.

Appointment backfill sync starting.

New calendar connections are coming back online and we are monitoring closely. We are about to begin re-sync of any appointments scheduled over the past hour to ensure they are all up to date on your calendars.

Our calendar provider has confirmed a syncing issue with Google and Outlook calendars. We are awaiting confirmation that these connections are back online, and then we will re-sync all appointments created during the outage.

As part of this issue, syncing of Aryeo appointments to Google and Outlook calendars is delayed. After resolution of the issue, all appointments will be re-synced to ensure accuracy of your calendar syncs.

We are currently investigating an issue preventing setting up new calendar integrations with Google or Outlook calendar. Note, all current integrations previously set up are unaffected.

Calendar Sync Update: calendar syncs are returning to normal right now. We are monitoring our calendar provider and ensuring all order forms and appointments are properly syncing. We will resolve this incident after another hour of monitoring to ensure there are no further issues with appointments being sent to external calendars. Thank you for your patience!

Our calendar provider is experiencing an API outage right now. This is impacting Aryeo's ability to read or send external calendar events. Our team is doing everything in our power to resolve this and ensure calendar functionality can be restored asap. We made an immediate update that will allow your agents to continue placing orders, but you will currently not see these appointments in your external Google or Outlook calendars. All appointments will remain visible in your Aryeo dashboard. We will keep you appraised of updates as soon as we have further information.

Marketing this incident as resolved.

Issues from this morning should be resolved - please reach out to support if you are still having problems. We are monitoring closely to ensure everything is fixed before closing this ticket.

Update: a fix was released shortly after reports of issues this morning and we are continuing to monitor the platform to ensure it's fully resolved. We have no active problems from this morning, but will keep the status in "monitoring" for the next couple of hours to ensure it's fully resolved. Background: We had an outage this morning that brought down most of the site, impacting many of our customers (and their customers). We had shipped a database change that caused significant errors for those trying to access the site. These outages aren’t acceptable and I promise we don’t take them lightly. We’re making some major changes going forward to ensure Aryeo provides stability and innovation: - New features/enhancements: going forward, all new features will be shipped in non-business hours. This is already true for most features, but we’re making this a formal policy for all features from now on - Aryeo System Status page: we’ll be adding a real-time status page that will show the system status of our platform, allow you to receive updates on issues, and see a public history of problems and resolutions - Tripling our support team: hiring an additional 14 team members as soon as possible, allocating additional engineering resources to support, and bringing on specialized product experts for power users of our platform The cause (ironically) of many of these issues has been related to a database migration we have been undergoing to get off of our legacy infrastructure. Our new infrastructure will make it easier for us to ensure the highest uptime, and also launch features like full white-labeling. I appreciate all of the support and understanding over these past couple of weeks, and I promise we’ll get these resolved and do so with the most transparency possible.