Is Chili Piper Down Right Now? Check if there is a current outage ongoing.

Google IAM service is currently experiencing issues resulting in pages that display Google calendars to either load slowly, or not at all. We are awaiting a fix from Google. All customers across all platforms are impacted.

### **Incident Overview** Some users experienced intermittent failures when accessing ChiliCal scheduling links while others began experiencing no access to any booking links within their organization. Affected users were presented with a **“Something Went Wrong”** error screen. The issue did not affect all links, but occurred unpredictably across different users, affecting scheduling and handoff workflows. After resolution, we found that some links were impacted further by caching of services which delayed the rollout of the fix to all users and made the impact’s duration longer than expected. ### **Timeline** * **09:00 - 09:30 UTC** – Incident reported. Users began experiencing intermittent issues with ChiliCal scheduling links. * **09:45 UTC** – Engineering team began investigation and confirmed issue reproducibility. * **10:00 UTC** – Initial fixes for the root cause were implemented as a hotfix * **10:30 UTC** – Suspected caching-related issue identified as extending the issue’s behavior * **12:45 UTC** – Team forced cache clear across relevant instances to expedite recovery. * **13:08 UTC** – Systems showed recovery. Monitoring phase initiated. * **Post-Incident** – Recommended browser cache clear for users still seeing the issue. Ongoing monitoring confirmed full resolution. ### **Root Cause** On April 28th we found that our JS framework \(NextJS\) [did not provide deterministic application builds](https://github.com/vercel/next.js/issues/63201). We deployed [a fix ](https://github.com/vercel/next.js/discussions/59493#discussioncomment-7824870)for this as an unrelated incident. While fixes were implemented to resolve the library’s update, this incident was also impacted by stale or inconsistent **application-level caching** that impacted the rendering of certain ChiliCal scheduling links. The cache did not invalidate properly after recent backend changes, resulting in outdated or broken state served to some users. ### **Resolution** The team manually forced a cache clear on affected services. This action immediately restored functionality for most users. Those still affected were likely experiencing residual issues from **local browser cache**, which was resolved with manual clearing. ### **Preventative Actions** #### Completed: * Manual cache purge on affected infrastructure. * Different technique used to handle the stop-gap measure we had in place previously which is not reliant upon framework-specific functions. * Communication to Customer Love and affected stakeholders with recommended local steps. #### Planned: * Implement automated cache invalidation tied to relevant backend deployments. * Improve detection and alerting for cache-related anomalies \(e.g., error rate spikes\). * Migration from our current JS library / framework to an alternative with broader support and flexibility. ### **Outcome / Next Steps** * Caching mechanisms need tighter integration with deployment workflows to avoid serving stale content. * Partial or intermittent failures are harder to detect and require better instrumentation. * Proactive cache management and user-friendly fallback experiences are key to resilience. * A new and more robust JS framework is needed to avoid this scenario in the first place. The ChiliCal incident on April 29, 2025, was resolved with no lasting impact, but highlighted important areas for improvement in cache handling and monitoring. Work is underway to strengthen these areas and reduce the likelihood of recurrence.

This incident has been resolved.

The underlying issue appears to have been caused by caching which naturally took time to clear. Our team has forced the instances to clear and we should see service recovered now. Local cache may still be a factor in displaying these pages, so please try clearing browser cache if you see the issue remain. Our team is monitoring from this point forward to ensure it's fully resolved.

Some users are experiencing issues when accessing Chili Cal scheduling links. In certain cases, the links fail to load and instead display a "Something Went Wrong" error screen. This behavior is not affecting all links, but the issue appears to occur intermittently across multiple users. Our team is actively investigating the root cause of the issue and working to restore normal functionality as soon as possible. Updates will be provided as more information becomes available.

Our engineers identified an issue that caused our system to stop tracking changes made to calendar invites by customers in Outlook. Fixing the bug, and triggering a re-sync of the services caused all of these previous updates to trigger finally, even if they were severely outdated or days old. Although this would have caused a flood of notifications, this would be a one-time side effect of fixing the underlying bug, and is actually indicating that those users who received the notifications are now in a good state. This could have impacted all forms of notifications including reschedules, cancelations, invitations, reminders, and so forth. It only impacted customers connected with Outlook. The issue is already resolved to prevent it from continuing, but old notifications may still be making their way through the system. Please expect some notifications may show up as a result of the above.

Zoom has successfully restored its services, and they are now fully operational once again. During this outage, users' Zoom integrations may have been affected, resulting in the status indicators "In Trouble" or "Disconnected. " If this situation pertains to a user, they must reconnect their Zoom account at https://fire.chilipiper.com/app/integrations. Super Admins can access the list of affected users through the following links and are advised to notify them to reconnect: - https://fire.chilipiper.com/fire/admin/users/active?zoom=InTrouble - https://fire.chilipiper.com/fire/admin/users/active?zoom=Disconnected For meetings scheduled during the Zoom outage, users may need to manually add a Zoom meeting to their calendar invitations. Additionally, the meeting reminders will not include the Zoom links manually added to the calendar event.

Zoom is experiencing an outage across multiple services on the zoom.us domain, which will affect new meetings booked via Chili Piper and prevent the generation of Zoom links. We are actively monitoring Zoom's status page https://status.zoom.us/incidents/pw9r9vnq5rvk to learn when their services will return.

Our tracking of the delays indicates a near zero-latency after our fix was deployed, so we are considering this incident resolved. If you continue to experience any delays in retries or issues where ownership / lead updates are not performed, please reach out to support@chilipiper.com for further assistance.

We've deployed a fix that will help improve the CRM action delays / failures. However, we will need to monitor traffic for at least today in order to determine if the fix ultimately did the trick. We are actively monitoring the situation, and the impact of this fix, and will act quickly if further action needs to be taken.

We have identified a larger scope for the incident, in that all CRM Actions can be impacted by delays or failures. This can include Lead / Contact creations prior to any Event creation. As a side-effect from some of the issues, Event records have been duplicated in some instances as well. This is following an "at least once" method where it will attempt to create the event in the case that there may have been an issue, to ensure at least one event is created. This is not an intended side-effect, and we are investigating how to mitigate this aspect of the issue as well.

In our Fire platform we have a mechanism that will retry an CRM action if something fails, or if a Lead doesn't exist yet, and so forth. We have heard that there are long delays in executing these updates which include updates to ownership as well as other fields and creation logic. This is a top-priority issue, and our engineers are investigating the cause as we speak. As reported, this only impacts actions that required a "retry", and is only within our Demand Conversion Platform products (Fire).

We identified a third-party script was causing the admin center page to hang. We have blocked this script, and pages now load as expected. No further impact is expected as a result of blocking this script.

We have discovered that as of the latest update in Google Chrome version 135, the "Admin Center" in our legacy app (apps.chilipiper.com) becomes unresponsive due to memory usage. You are still able to navigate around workspaces, and from workspace-to-workspace, but the browser will become unresponsive if you attempt to navigate back to the Admin Center. Our engineers are investigating the cause right now as it relates to v135. Older versions of Chrome are confirmed not to be impacted. Other browsers or chrome-based browsers appear to be unaffected as well. This does not impact our Demand Conversion Platform (Fire) app.

Our team was made aware of a wide-spread issue involving the ChiliCal scheduler for Handoff not loading routers, 404 pages being experienced, errors indicating a connection cannot be established, and perpetual loading of Concierge routes. This issue was a result of unplanned database maintenance which triggered the temporary outage. This does not appear to have impacted our legacy / classic services at this time. We are seeing all services up and running after a brief approximately 20 minute outage.

We are currently investigating an issue where Chili Piper Concierge or Scheduling links display a "Something Went Wrong" error page for users with multiple languages set in their Google Chrome settings. Our team is treating this with the utmost urgency, and we are working diligently to identify and resolve the cause. We will provide updates as soon as we have more information.

Issue is now resolved and users will no longer see the "Something Went Wrong" Page if they have multiple languages in their browser. Thank you for your continued patience and understanding.

We've confirmed the fix is in place, and routing is working as expected.

Customers using our "marketing.js" Concierge snippet on their landing pages will have experienced an issue where no submissions were coming through. Symptom is that the snippet sends an error to the Chrome developer console noting an "undefined" "jwt bearer token". We were alerted to this incident around 12:45 PM ET, and the code causing the issue was rolled back at approximately 2:00 PM ET, meaning there was a window of just over an hour where no submissions would come through. As we've rolled back the code, we are now monitoring to determine if the incident is now resolved and will act quickly if we still see any occurrences.

Some users may experience a "Something went wrong" message when accessing admin-center dashboards or pages in our Fire UI. This was due to a caching issue with a previous deployment. To fix this issue, simply clear your browser cache, or clear all browsing data. In your browser you can press COMMAND + Shift + R (or CTRL + Shift + R) to hard-refresh the page. Or, you can use the browser's settings menu to select "Delete Browsing Data", making sure to select the cached images and files for deletion. Otherwise, this issue will resolve itself with time as the cache expires. Please don't hesitate to reach out to support@chilipiper.com if you have attempted to clear your cache and still experience issues loading pages.

Our Engineering Team identified the root cause of this issue and deployed the required fix. HubSpot actions like "Update Property", "Create/Update Contact, and "Create Engagement" nodes should work as they should from now on. If you still notice any issues, please reach out to support@chilipiper.com with examples so they can be further investigated.

We are currently experiencing an issue where HS custom properties are not being updated as expected. Additionally, the router flow is encountering a blockage at the "Create/Update Contact" node, preventing the execution of subsequent tasks. Our team is actively investigating the root cause and working to resolve this issue as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience as we work toward a solution.

The Legacy Instant Booker Chrome Extension has been published to the Chrome Webstore and has resolves this issue. The version of the latest extension should be 3.0.5566. If you continue to have issues ensure that: 1) You are logged into Chili Piper 2) You have reinstalled the Chrome Extension 3) Refreshed the app you are using, ie: Gmail. For further issues, please reach out directly to support@chilipiper.com for additional support.

For customers on our Demand Conversion Platform, the fix has been deployed for our ChiliCal Co-Pilot app and will be working as expected across all apps. Please be sure to update to version 1.308 to get this fix. Our legacy Instant Booker app is still waiting to be published, but we expect this to be fully resolved by tomorrow. In the meantime, please utilize the workarounds we have provided in order to open Instant Booker. We will provide an update once this is deployed to customers as well.

As a quick update, we are still waiting on our app getting published to the webstore, but the fix was confirmed to work in our QA testing. Thank you for your continued patience! In the meantime, if you are a Salesforce admin you may want to consider adding a custom button to launch Chili Piper on various records within Salesforce which has benefits even after the issue is resolved. You can find instructions on setting this up here: https://help.chilipiper.com/hc/en-us/articles/360053021734-Instructions-for-adding-Custom-Button-in-Salesforce-to-Book-Meetings

Our team has identified the root cause of the issue and a new version of the extension was submitted to the Google webstore where it should publish to all customers soon. As mentioned previously as a workaround you can continue to use Instant Booker/Handoff by pasting the prospect’s email address directly into https://apps.chilipiper.com/meetings/instant-booker.

We have identified an issue affecting our Google Chrome extension users, specifically those using version 13.0 and above. Our engineering team is actively investigating the root cause and working diligently to implement a resolution. We understand the impact this may have on your experience and are committed to resolving the issue as quickly as possible. We will provide updates as soon as more information becomes available. As a temporary workaround, you can continue to use Instant Booker/Handoff by pasting the prospect’s email address directly into https://apps.chilipiper.com/meetings/instant-booker. Thank you for your patience and understanding.

Chili Piper experienced an incident as a result of a service migration that impacted the ability to load Chili Piper's Concierge snippet (marketing.js) in our classic UI / legacy product. Only a subset of around 200 customers were impacted, with no "Demand Conversion Platform" Concierge customers included. This did target both production and sandbox accounts. The impact was that when customers filled out an online form connected to Concierge, Chili Piper would not have presented an option to book a time. Effectively, these prospects would act as though they were disqualified but may not show up in your Concierge reporting. In the developer console, a "500 (Internal Server Error)" error would have been shown. The incident was resolved at approximately 9:00am EDT for all customers when the migration service was rolled back. It cannot happen again as it would require a manual intervention in order to initiate the migration service from this point forward.

This incident has been resolved.

Our engineers are investigating reports of the Distro routers screen in the admin center showing an error message: "Could not load Routers" followed by a blank page. We do not expect routing itself to be impacted by this issue. Additionally, not all customers are impacted by the issue - if you are able to view your list of Distro routers, you are not currently impacted and can use Distro services as usual. No other products are impacted.

We have reverted the change that initiated the Outlook add-in issues and will ensure future deployments are stable. All impacted customers should be back up and running.

We are investigating issues with our production Outlook Add-in within our classic product suite for Microsoft customers. This may impact your ability to open the Instant Booker app within Outlook Desktop or Web. Using Instant Booker from within Chili Piper directly will still function as expected. The standard Chrome Extension can still be used to book meetings directly form the extension itself, just not as buttons within Outlook. Existing meetings will not be impacted. This does not impact Concierge form bookings, or meetings booked via booking links. Google customers are not impacted at this time. Our Site reliability team is looking into this, and we expect a fairly rapid solution. Thank you for your patience in the meantime!

We are considering this issue fully resolved. We have monitored our incoming logs and confirmed the disconnects have stopped occurring since we rolled out our fix to all customers. The root cause was confirmed to be the update in the "Summer 24" release from Salesforce that blocked simultaneous refresh tokens. We resolved the issue by adding a more robust retry strategy in addition to better logic in how we handle refresh tokens. As these changes were not trivial and required significant engineering resources, the fix took longer than usual to develop. As an outcome, the retry mechanism will remain in place and our notification emails related to dropped connections were adjusted to better guide admins should any unrelated disconnect issues occur in the future.

Our fix is now deployed to our production environments, and we are now monitoring in realtime to validate that the fix is working as expected. Please make our team aware of any additional disconnects you experience via support@chilipiper.com so that we can act quickly to investigate it further.

We have successfully replicated and validated the issue's root cause through testing. Our solution is confirmed to resolve the disconnects and will be deployed to our production environment shortly. Once it has deployed to all customers, we will provide another update to monitor the fix before confirming it is fully resolved.

We have begun deploying some solutions to our testing environments to start validating them internally. Once we have completed tests we will start rolling it out to customers to monitor the fix. We have also updated our automated email sent to admins in order to make it clearer where to go in order to reconnect. Previously it had indicated the personal-level connection, but please be sure to reconnect the global admin-level connection as the new notification indicates.

We understand the urgency and concern this incident has caused. The resolution is taking longer than anticipated due to the need for significant changes in our authentication infrastructure rather than if it was a more trivial bug. This issue has been escalated internally and is our top priority. We've allocated additional engineering resources to expedite the fix so we can get it rolled out as soon as possible. We aim to have a solution deployed to our testing environment before the end of today. Once verified through quality assurance, it will be rolled out to all customers. In the meantime, please continue to monitor your integration user email for any Salesforce disconnects. Thanks again for your patience and understanding.

We are currently investigating an issue affecting the global Salesforce connection (integration user) for a small subset of customers using our legacy product line. Please note that our Demand Conversion Platform products and HubSpot CRM customers are not impacted. While disconnected, there may be issues with creating or updating Lead records or Events. Additionally, routing will be limited to form data and will not function properly based on existing Lead/Contact data. Our preliminary findings suggest that recent updates from Salesforce, which block specific refresh token strategies, may be the underlying cause. We are working on a fix to implement a more robust refresh token logic. We have confirmed that a notification email is sent to the integration user email account when a disconnection occurs, prompting them to reconnect as soon as possible. We are actively working on a solution. In the meantime, please be aware that the service may disconnect again even after reconnecting. To mitigate this, we have implemented active monitoring to detect disconnections promptly and may reach out directly if needed. We urge admins to monitor their integration user inbox closely to ensure timely action in the event of a disconnection. Thank you for your understanding and patience as we resolve this issue.

Our team identified an issue that's preventing Chili Piper from successfully trigger upon form submission. We have quickly resolved the issue by applying a hotfix. Chili Piper show now be triggering properly and the calendar would appear when a prospect successfully qualifies for a booking.

Our team identified an issue with a new security policy that was preventing Chili Piper assets from loading. Our engineers have already deployed a fix for this and we should see it resolved. While this was impacting Chili Piper, routing and booking links were impacted as well as internal pages and they were instead presenting a blank page or could not be loaded. We have resolved the issue by reverting the change and have corrected the underlying issue moving forward.

We are marking this issue as resolved since incoming data shows services are recovered, error rates have dropped, and loading times are stabilized. Our team will continue to monitor services closely to ensure they remain stable moving forward, and actions have already been taken to ensure this stability is maintained.

Services have been restored, and several sub-services were restarted in order to further support the recovery of all instances. We are seeing load times gradually return to normal with only a minor performance impact reported. We will continue to monitor the recovery closely to ensure our services remain stable.

We have begun to restore services and have reduced the load on the impacted databases. Databases instances are being restarted to help mitigate the issue. This is showing improvement on load times, however users may still experience long delays or see an "Error: Server Error" 502 message with degraded performance of all Chili Piper pages, links, Instant Booker, and Concierge. We should start seeing more services gradually become healthy in time, and the loading speed will continue to increase, however due to the increased load times we are still considering this a partial outage.

We are currently investigating an issue impacting our database which is preventing Chili Piper assets from loading. Instant Booker as well as Chili Piper's dashboard are both impacted. Some booking links appear to load, but performance is extremely degraded. Pages may load after a long delay. Our Site Reliability infrastructure team is actively investigating this with priority, and we will provide an update once this issue is resolved.

This incident is now considered resolved. We have not received any additional reports of long loading times or performance issues since our last update. We have documented the issue and taken the necessary measures to prevent this in the future. Thank you for your patience while we worked on this!

All systems have returned to a stable status. Our engineers are monitoring now to ensure that this is maintained. Please continue to contact our Customer Love team at support@chilipiper.com if you are still experiencing any slowness or performance issues with Chili Piper.

Having identified the issue, our engineers are making significant progress working on a fix. We apologize for the extended duration of this incident and sincerely appreciate your patience. We understand its impact on your experience and are fully committed to resolving it swiftly.

We are seeing instances of concierge and booking link pages loading endlessly and eventually timing out. Our engineering team is currently investigating. The impact appears to be in a smaller subset of databases. We will provide additional updates here as we have them.

We have successfully resolved the issue with Distro Logs that was affecting a small group of customers. Thank you for your patience while we worked to fix the problem.

Our Site Reliability team has stabilized all affected functionalities except for Distro, which is still experiencing slowness. The engineering team is currently working on a permanent fix. We appreciate your understanding.

Our Site Reliability team identified an issue in one of our databases. They are currently working on the fix.

Some users have reported longer loading times and issues saving Chili Piper assets. They are mainly affecting Distro. Our Site Reliability team is aware of the problem and is working on a solution. We anticipate a fix will be implemented soon.

The issue with Google Calendar has been resolved, and the user's availability is now functioning correctly in Chili Piper. For further details regarding the impact, kindly refer to the Google Status page at https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ

Some customers who use Google Calendar and attempt to book appointments via Instant Booker are experiencing problems. The problem is linked to a recent issue that occurred on Google Calendar, which is explained on this page: https://www.google.com/appsstatus/dashboard/incidents/7uJZ5F1Uy4n1n74iMacQ. Due to this issue, some users may have problems with their availability on online booking links. We are closely monitoring the situation and expect that Instant Booker and user availability will return to normal once Google resolves the problem.

We are aware of an issue with Instant Booker not loading and prompting the following error message: "Error: Server Error The server encountered a temporary error and could not complete your request. Please try again in 30 seconds" Our engineers have identified the cause and are working on a fix.

The issue with Instant Booker has been resolved for all customers. We're now taking the necessary measures to prevent this from happening in the future. Thank you for your continued patience and understanding.

A fix has been implemented and we are monitoring the results.

The issue with Instant Booker has been resolved for all customers. We have documented the issue and taken the necessary measures to prevent this in the future. Thank you for your patience while we worked on this!

Our engineers have made significant progress working on a new fix. They have also identified the subsection of users impacted by this issue. We apologize for the extended duration of this incident and sincerely appreciate your patience. We understand the impact it has on your experience and are fully committed to resolving it as swiftly as possible.

We are aware of an issue with Instant Booker not loading and prompting Server Error. Our engineers have identified the cause and are working on a fix.

This incident is now considered resolved. We have not received any additional reports of long loading times or performance issues since our last update. Our engineers successfully mitigated the incident by restarting services and providing those services with additional resources. Moving forward we will continue to explore options to improve this to both prevent it from happening again and ensure it can scale in an automated way. If you have any additional questions or concerns, please don't hesitate to reach out to Chili Piper at support@chilipiper.com

All systems have returned to a stable status. Our engineers are monitoring now to ensure that this is maintained. We are actively developing a strategy to ensure this doesn't happen again in the future and will have more details on a resolution shortly. Please continue to reach out to our Customer Love team at support@chilipiper.com if you are still experiencing any slowness or performance issues with Chili Piper currently.

Some users have reported that there are longer than expected loading times across all Chili Piper assets. This may impact internal admin pages as well as booking links and form widgets through Concierge. Our Site Reliability team is already working on a resolution as they have identified the issue and should have a fix deployed shortly.

Our engineers have resolved the majority of issues with the Chrome Extension, and we believe that any remaining problems are likely the result of edge cases unrelated to the initial incident. If you are experiencing issues suggesting times in email, please be sure to uninstall and reinstall our Chrome Extension at https://chrome.google.com/webstore/detail/chili-piper-automate-sche/digckgagekofoajjlaeceojmdokllfmf If after doing this you continue to experience problems with the Chrome Extension, please reach out to support@chilipiper.com

Our team has been working diligently to address the issue, but we've encountered some challenges in consistently replicating it across most user environments. Our engineers have made significant progress in understanding the root cause and are currently testing a new fix. Once we get confirmation from internal users that this is working, we will deploy it to all customers and keep you informed of its progress. We apologize for the extended duration of this incident and sincerely appreciate your patience. We understand the impact it has on your experience and are fully committed to resolving it as swiftly as possible.

We have received reports that our Chrome Extension, which provides quick access to Chili Piper features and enables icons within apps, is not working for a subset of users. The reported issues are varied and may include: * Inability to add suggested times to Emails * Booking links not loading in the extension * Buttons not appearing in apps such as Gmail * No show toggle missing from calendar invites * Reassign button missing from calendar invites The primary Chili Piper app does continue to function properly, and many of the features can be accessed either directly from your Chili Piper dashboard. Many users have reported that clearing their cache, re-downloading the chrome extension, and making sure all Chrome tabs are closed completely before trying again has helped. Our engineers are actively deploying fixes for this and have had positive results from some users. As such, please continue check the problem you are experiencing periodically in case this is now resolved for you. Our engineering team is dedicated to working on a fix for all users and will not stop until we get confirmation that this has been fully resolved.

We are considering this incident resolved. The full scope included issues with the reports dashboard, profiles, and some login issues. We were able to resolve this issue by reverting a recent deployment. This did not impact a customer's ability to book meetings, or the routing of events / meetings. Overall the impact was low. We are taking actions to ensure that future deployments do not cause the same impact. Thank you for your patience and understanding while we worked to resolve this incident.

We have mitigated the issue via the previously mentioned rollback and all systems are back up and operational. We are monitoring this to ensure stability is maintained.

We are continuing to work on a fix for this issue.

Our team is currently investigating a variety of issues that may have affected your user experience, such as reduced performance when loading internal pages and some internal features, including Instant Booker and Reports Dashboard for admins, being impacted. We have been made aware that some users are encountering multiple login prompts that are difficult to bypass. We understand the frustration this can cause and our team is working to rectify this issue as quickly as possible. To mitigate these problems, we are actively rolling back some recent changes and implementing additional measures to ensure that our platform runs smoothly. Thank you for your patience and understanding. We will keep you updated on our progress and apologize again for any inconvenience this may have caused.

All email notification systems are back up and operational for all customers. Emails that were scheduled prior to the fix may have been lost as part of the service restart, however all emails and notifications moving forward appear to be functioning properly. We will continue to monitor internally but are considering this incident resolved. If you continue to experience any issues with emails, please be sure to contact support@chilipiper.com for further assistance.

A fix has been implemented for the majority of customers and we are already seeing improvement in services. We will continue to monitor to ensure this is resolved and are working to push the fix out to all customers now. Until the fix is fully deployed, some customers may still experience issues with email notifications. We will provide an update here when all systems are operational.

Our team has identified the probable cause for the email notification issue and are working to deploy a hotfix. Once we have rolled out a fix, we will monitor to ensure proper resolution of the issue.

We are currently investigating an influx of reports that some emails are not being sent or received. Not all customers are impacted, and even impacted customers appear to have most emails sent correctly. The scope of the issue impacts reminder emails, booking confirmation emails, and other alerting emails for admins. These are both internal and public-facing. While we work to resolve this issue you may notice some emails are not received by yourself, your team, or your customers.

This incident has been resolved. Thank you for your understanding and patience on this and we are sorry again for any inconvenience this issue might have caused you.

Between 9:00 PM GMT and 9:10 PM GMT, we received reports of 502 errors (timeout) being displayed while navigating in Chili Piper's UI, extension, and also while triggering routers. The issue was fixed by our Infrastructure Team after identifying an overload in our database. We're currently monitoring the performance, which has been stable, before moving this to resolved. Thank you for your understanding and patience on this and we are sorry for any inconvenience this issue might have caused you.

We monitored all services and all Chili Piper components are now 100% operational. Thank you for your patience while we pushed these critical updates!

Maintenance has been completed and all services are slowly recovering. We are monitoring and will post again once everything is 100% operational, across all clusters.

As part of the maintenance, our team is pushing out some critical updates that are taking longer than expected - this impacts customers on the Prod cluster. We will continue to keep you updated, please stay tuned as we should have an update very soon.

We are in the process of finalizing our scheduled maintenance for our production clusters. As such, you may experience some timeouts with loading the Dashboard or booking links. We are speeding up the process with our Engineering Team and working to restore all services as soon as possible, we will keep you updated.

This incident has been resolved. Thank you for your understanding and patience on this and we are sorry for any inconvenience this issue might have caused you.

Our engineering team identified the cause of the issue and a fix was created. If you're still not able to see the Chili browser icon on Gmail, Outreach, etc, please clear the browser cookies and login through https://apps.chilipiper.com/login.html?success=true

The issue has been identified and we are working to hotfix the issue now. Customers should start seeing this issue resolved as our internal caches clear.

We have received reports of login issues within our Chrome Extension and Instant Booker. This may cause the usual icons not to appear properly within your apps such as Gmail or Salesforce, and may not allow you to login to the extension properly. Our engineering team is investigating the cause, so we will have more information on the fix shortly.

This issue was reported August 4th and was since resolved. Reports came in that invites were not being sent to prospects and that other email services, including reminders and notifications were not functioning. Our team determined that invites (meetings on Calendar) and notification emails were not sent to a subset of our customers during this time, and these meetings / emails are not recoverable. Meetings booked during this time may not have appeared on the prospect / assignee calendar. The cause was that one of our new web services connected to notifications failed silently. Our Site Reliability Team was able to resolve this issue as of 2:30pm EDT August 4. We have implemented further monitoring for this and other services so that if any impacting issues occur in the future, we will be able to mitigate the problem quickly and without user impact.

The newest version of the Chrome Extension was approved by the Chrome store and this is available to everyone (version 3.0.486): https://chrome.google.com/webstore/detail/chili-piper-automate-sche/digckgagekofoajjlaeceojmdokllfmf If you have uninstalled the extension, please just install it back and it will be with the proper version. If you have only disabled it, please follow these steps: 1. Go to the extension settings at chrome://extensions/?id=digckgagekofoajjlaeceojmdokllfmf 2. At the top-right of the page, click on the toggle to enable the extension again; 3. At the top-left, click on the "Update" button. As soon as you confirm that the extension is now active and have the v3.0.486 installed, you can remove the local copy of our extension previously installed: 1. Go to this page: chrome://extensions/?id=kdmalpdadhpjffdocncnecbpkcgkngge 2. At the end of the page, you can click on "Remove extension". Thank you for your understanding and patience on this and we are sorry for any inconvenience this issue might have caused you.

We continue to wait on the Chrome store approval process to publish the changes we have made to the Chrome Extension. Though we have no ETA on this, we expect it should be any moment now. For some additional information into next steps - We have since fixed the pipeline to ensure this is built correctly in the future and have additionally assigned resources to manually test these builds across all environments before release. Any users who have installed the local copy of our extension will eventually need to remove it and install the public copy again. We will provide steps for this once the extension is fixed in the store, and this will not urgently need to happen immediately. In the meantime, please continue to follow the steps listed in our knowledge base article for a temporary solution: https://help.chilipiper.com/hc/en-us/articles/7406298455059

Per our previous update we are still waiting on the Chrome app review process. Unfortunately, this can take some time for Google to complete and although we don't have an exact ETA, it can take up to 24 hours to approve. In the meantime, we have released a local download copy of our fix that can be deployed immediately. Steps are provided in our helpdesk article as well as a video highlighting the step-by-step process. We have confirmed that this local copy resolves the issue once installed. To view these steps and download a local copy, review this page here: https://help.chilipiper.com/hc/en-us/articles/7406298455059 If you opt not to install the local copy the Chrome extension will eventually update automatically once approved on Google's end with no action needed on your end. We will continue to monitor the fix and ensure we provide an update here once it has rolled out to the majority of customers via the Chrome web store.

Our engineering team identified the cause of the issue and a fix was created. The new version was sent to Chrome Review, which can one day or more. In the meantime, please follow the steps described in the following article to make the extension operational again: https://help.chilipiper.com/hc/en-us/articles/7406298455059

The Chrome Extension is currently not working for customers with instances on the production cluster. The issue is not affecting instances running on the canary cluster. Our engineering team is already analyzing the issue to provide a fix.

We are considering this incident resolved as the overall remaining database refresh is no longer impacting major services. Some sub-services may still experience temporary issues including the sending of reminders, and email notifications. If these issues occur, please wait 24 hours and they should be resolved.

The impact of this incident has been minimized as we continue to make progress on the database recovery. We are monitoring this closely and will provide another update once services have returned to a healthy status for the subset of customers impacted.

Our engineers identified the root cause quickly and have restored most services, including booking flows for the subset of impacted customers. Some features will remain offline such as reminders, reports, and automated services until the databases are fully operational.

We are seeing new instances of booking links, and Chili Piper admin pages timing out with "502 Errors'. Out engineering team is investigating currently. The impact appears to be in a smaller subset of databases than yesterday, so the overall resolution will be faster. We will provide additional updates here as we have them.

We have restored all services and databases to fully operational status. Our team will continue to monitor the results to make certain these services remain healthy and have provided additional resources to ensure stability moving forward.

Our team is making good progress in restoring the database services that are still impacted and this may take take an extended amount of time to complete. While this is in progress you may experience issues with reporting, reminders, and some automated services, however booking flows and most admin functions should work normally. There will be no loss of data. Stability of these services is extremely important to us and we are working to ensure this is more robust moving forward. We will provide another update once all services have been restored.

Impact has been minimized and we have a clear plan in place to get back to fully operational mode without any loss of data - we are executing on it right now. Once we're fully healthy we'll re-enable all less essential services such as reminders and we'll investigate what happened in detail.

Our engineers restored all database instances to a 'partial mode' - Chili Piper is now loading as expected and booking flows are back to normal, but reminders, reporting and some automated processes still need to be restored. We are working on restoring all functionality, we will update you once the process is finalized.

Our Team has identified the issue as stemming from the primary database. We are working on a plan to restore functionality without losing any data. We will keep you update!

We are continuing to investigate the root cause with our Reliability Team. The issue manifests in not being able to log into the Dashboard or book meetings. We will keep you updated as new details emerge.

We are seeing instances of the Dashboard loading slowly and timing out with 502 errors. Our Engineering Team is investigating

This incident has been resolved. We are continuing to monitor, but have seen a significantly steep decline in error rates that are now within normal limits. For most customers, no further action is required. If you notice issues with routing, 500 errors, or issues connecting to the CRM, please login as admin and reconnect the sync user at https://apps.na.chilipiper.com/admin-center/integrations. Doing so should put you back in business.

We have deployed a few hotfixes and are monitoring the results now. Customer accounts appear to be recovering after a short wait. In the meantime we will continue to keep an eye on error rates to ensure they are dropping. Once we have confirmed the resolution, we will provide another update.

Please be aware that HubSpot CRM-connected customers may experience intermittent issues related to their connection. The scope of this issue can cause problems routing via web form, logging in with HubSpot, setting queue rules, refreshing fields, or doing Contact update/search. One primary symptom is hitting a "500 error" when attempting to submit a form or do admin tasks. Our engineers have identified several root causes and are working to mitigate this. Additionally, we are refactoring the way these requests happen to ensure this does not impact connections in the future. We will have more information on a resolution soon.

After continuous monitoring, we are confirming this incident is resolved. Our team will continue to monitor the status of Google connections across our infrastructure to ensure stability is maintained. Users who were impacted by the initial incident may find that their calendars are still disconnected. The resolution of this incident does not retroactively restore connections. If you have a user that is disconnected, please ensure they reconnect by logging into Chili Piper with their Google credentials.

We are working to implement fixes for the identified issues and are monitoring the results. Most users should have at least attempted to reconnect their Calendar and we have noticed a significant drop in error rates. Please continue to reconnect your calendars and the calendars of assignees / bookers if trailing reports come in. Once reconnected, the issues should be fully mitigated for those users. If you are experiencing critical issues as a result of the incident, or experiencing difficulty reconnecting, please reach out to our Department of Customer Love support team at support@chilipiper.com or your dedicated Customer Success Manager so that we can investigate.

Our engineering team is working to get the identified issues resolved as quickly as possible, and progress is being made. We urge customers who are impacted to continue to attempt to reconnect their Calendar if they are experiencing unexpected behaviors while using Chili Piper, even if the UI indicates you are fully connected. Login with Google credentials: https://apps.chilipiper.com/login.html?success=true to attempt to reconnect. We have learned that multiple attempts may be required in order for the new credentials to take hold. The impacting issues include, but are not limited to, calendars showing no availability or not loading at all, Instant Booker showing issues sharing or viewing calendars, and issues where the Booker of the meeting receives the event (as organizer) rather than the Assignee.

We have identified some of the underlying root causes of Google calendars disconnects and are continuing to investigate a resolution. It is still recommended that impacted users reconnect their calendars as this resolves the problem in most cases. This can be done by simply logging into Chili Piper and using your Google calendar as the login option here: https://apps.chilipiper.com/login.html?success=true We will continue to provide the latest updates here as they become available.

We are continuing to investigate the issue and are narrowing down the root cause. Impact: - You may see issues displaying the proper availability on booking links or within Instant Booker. - If you are the "Default Booker" within a Concierge queue, you may get more meetings booked to your calendar. - Updates to Google events such as reschedules or cancellations may not happen automatically. As noted previously, reconnecting Google on a user-level renews disconnection and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 Please stay tuned here for updates.

Our engineering team is still investigating the Gcal token disconnects - the issue seems to be related to Google API's permissions scopes. Reconnecting Google on an individual level renews the token and permanently fixes the issue. If you are impacted, check out our help article for the steps to reconnect: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll update this incident with more details.

We are currently investigating an issue with some Google Calendar users not able to book meetings due to Gcal being disconnected. Only a minority of Gcal customers are experiencing this issue, O365 customers are not affected. If booking via Instant Booker, the booker will see the message "You can't book meetings in this calendar'. If booking via a personal or queue link, the prospect will see the message "Select a different slot" To fix this immediately, please change the default booker to a different user with a valid calendar connection, or have the default booker reconnect their calendar under https://apps.chilipiper.com/meetings/personal-settings/integrations Check out this help article for detailed steps: https://help.chilipiper.com/hc/en-us/articles/360053800193 We'll keep you updated via this thread.

Issue is resolved and Chili Piper is loading as expected. Our engineering has deployes further infrastructures changes to make sure this doesn't happen in the future. Thank you for your patience!

We have identified the issue and we are now seeing a decrease in loading issues - this is now resolved. We are monitoring and working on a Root Cause Analysis.

Some customers are experiencing slowness in loading Chili Piper, our Reliability Engineering Team are looking into it. Routing functionality is not impacted. We'll update this post with updates

After an hour of monitoring, we confirm the database issue has been resolved. The booking links and queues links are properly working on the Canary cluster.

A fix has been implemented on the database. Booking links and queue links are operational again for the Canary Cluster. Our engineering team is currently monitoring it.

Our engineering team has identified the issue as stemming from the database of the Canary cluster. We are currently working on restoring the functionality.

Our engineering team is still investigating the issue. Booking links and queue links are affected for the Canary cluster, a small subset of users - production customers are not affected. We'll continue to keep you updated, thank you for your patience!

We are investigating an issue with booking links not working - only our Canary cluster is affected. Dashboard is loading as expected, but individual booking link or links inserted via Instant Booker are affected. We'll keep you updated.

Google has resolved their incident and Chili Piper is back up. Please reload the page.

We identified the issue stemming from Google Cloud, they are having uptime issues - https://status.cloud.google.com/. We are looking into a workaround and monitoring, stay tuned!

We are investigating an issue with Google Cloud impacting logging into Chili Piper, our engineers are on it!

We confirm Salesforce has resolved this - service has recovered and requests to Salesforce are sent as expected. Only a subset of Salesforce users were affected. Incident is resolved.

We are seeing SF service recovering and loading speed is back to normal. We are monitoring and will update this incident shortly for confirmation.

We are seeing reports of Instant Booker loading slowly, with Looking for Lead or Contact message, due to Salesforce experiencing traffic spikes - https://downdetector.com/status/salesforcecom/. Only a subset of Salesforce users are affected. We are monitoring.

We have pushed a fix and performance has been stable, all systems are working as expected. We are continuing to monitor.

We are still in the process of recovering the database cluster which accounts for the length of time this incident has remained open. Our team has been working on alternatives to recover the process faster and will continue to explore options to expedite this. We will continue to provide updates as relevant information emerges.

Our team is still in the process of resolving the incident which is impacting one of our database clusters. Mitigation has taken longer than expected, however our team is working to ensure this is resolved as quickly as possible. Please subscribe to updates on this status page in order to be notified once a resolution has rolled out to all impacted customers.

The issue has been identified and our team is working on a fix. Expect services to resume shortly. Current impact is on booking links, chrome extension and add-in, as well as dashboard. Users impacted may see slow performance or may not be able to access these features.

We have identified that this also impacts our Chrome Extension (Instant Booker) for the same subset of customers. We are actively working to address these issues.

We are continuing to investigate this issue.

We are currently investigating reports of site performance issues, including inability to login affecting a subset of our customers.

Incident was resolved and we took precautions to make sure it doesn't happen again. Leads who did not book were disqualified for this period: March 17, 10 am PST - March 18, 9 am PST. Please qualify these Leads manually - you can check the submissions and timeframe in Prospect History - https://help.chilipiper.com/hc/en-us/articles/360053800293#h_01EZ8GF0022JES6HY0HBNYCCZ6

We found the issue and pushed a fix, routing is working as expected. We are monitoring.

We identified an issue with routing where, in some cases, prospects are disqualified. Our engineering team is looking into it.

The incident has been resolved.

We identified the issue, loading speeds are back to normal. We are monitoring.

We are currently investigating Chili Piper loading slowly. Loading speeds in the Dashboard and Instant Booker are affected and some users may not be able to log in. If you are subscribed to this incident, you will receive automatic updates with the progress. Thank you for your patience while we're looking into this!

This incident has been resolved, loading speeds are back to normal.

We have identified the issue and a fix is in place. We are monitoring.

We are currently investigating Chili Piper loading slowly. Loading speeds in the Dashboard and Instant Booker are affected. If you are subscribed to this incident, you will receive automatic updates with the progress. Thank you for your patience while we're looking into this!

This incident has been resolved and loading speeds are back to normal. Thank your for your patience while we looked into this!

A fix has been implemented and loading speeds are back to normal. We are monitoring.

We are currently investigating Chili Piper loading slowly. Loading speeds in the Dashboard and Instant Booker are affected. If you are subscribed to this incident, you will receive automatic updates with the progress. Thank you for your patience while we're looking into this!

Hi, Around 06.20 PST some reminders were sent in duplicates. This was caused by an outage on Google's end which caused these emails to send in bulk. At 06.30 PST we stopped all duplicate notifications and at 06.40 we pushed a fix to prevent this from happening in the future should Google experience downtime again.

https://status.cloud.google.com/incident/zall/20013 affected our last minute check for both Office and Google's calendar events. The root cause was Google's calendar api's were taking long to reply, this caused a backlog for certain reminders that got queued multiple times. We have improved the logic that avoids repeat reminders to better account for any third party outages / errors.

The latest updates were pushed to the official extension as well and will be propagated automatically by Google. If you don't want to wait for the update, you can install the updated version by uninstalling, then reinstalling your current extension. Correct updated version is 2.27571

We identified the issue - the latest changes were not pushed to our regular extension due to a delay in Google's approval process. Please uninstall the current extension and use this one instead, which has the latest changes: https://chrome.google.com/webstore/detail/chili-piper-chrome-extens/jcdjdendjmkfndafkkomfelogbpoicfd?hl=en-US&authuser=2

We are currently investigating an issue with the Chili Piper extension not inserting Suggested Times or booking links after clicking on Insert in Instant Booker. This is top priority, our developers are on it.

Salesforce confirmed that the intermittent issue was caused by a third-party DNS provider. We are not seeing any other issues and Chili Piper loads as expected, so this incident is considered closed. Should you still see issues logging into SF, please try the workaround they advised here: https://trailblazer.salesforce.com/issues_view?id=a1p4V000001pvYaQAI

The Salesforce login & integrations issues have been resolved by Salesforce and Chili Piper is working as expected, we are monitoring.

Salesforce identified the issue as occurring intermittently for a subset of customers in the APAC and NA region. Their team is investigating - https://status.salesforce.com/generalmessages/510. We will keep updating you with the latest info.

Salesforce is experiencing issues and logging in via Chili Piper & creating leads and events are affected. We are investigating.

The incident has been resolved and the CP icon now displays Gmail & Salesforce and Instant Booker now loads as expected. Make sure to reload the page before launching Instant Booker.

We have identified the root cause and we have a fix pending very soon. We will keep you updated once the fix is deployed.

We are seeing issues with the Chrome extension not populating in Gmail and Salesforce and the Instant Booker not loading. Our engineers are investigating.