Chameleon

Is Chameleon Down Right Now? Check if there is a current outage ongoing.

Chameleon is currently Operational

Last checked from Chameleon's official status page

Historical record of incidents for Chameleon

Report: "Chameleon availability"

Last update
identified

We've identified an issue stemming from this Heroku incident https://status.heroku.com/incidents/2822 (upstream of Chameleon) which is causing us to have limited access to the heroku platform. We use Heroku for auto-scaling, console access, and deployment. This incident is currently affecting the availability of the Chameleon Dashboard, Builder, and our ability to run End-user Experiences (Tours, Launchers, Microsurveys).

Report: "Launcher Segmentation rules bypass"

Last update
resolved

At Apr 16, 2025 12:01pm UTC, Chameleon deployed a code update that caused some Launchers to be shown to users not Segmented to them. Chameleon added an event listener to check for changes to the URL rules, but in case of a new match, this listener accounted only for URL rules, and didn't check Segmentation. The change was rolled back when discovered, at Apr 16, 2025 7:07pm UTC.

Report: "Chameleon application errors / performance"

Last update
resolved

This incident has been resolved, and all systems are operating as expected.

monitoring

We are continuing to monitor system performance as response times and error rates have improved over the past hour. No new end-user-facing timeouts have been observed recently, but we are still actively investigating to ensure stability.

investigating

We are currently investigating slow downs and 503 errors on profile requests.

monitoring

Our new capacity has been online for several hours now and is performing as expected. We will keep this incident open for another 24 hours in case of any new developments or issues. Thanks for out patience

identified

Chameleon continues to experience extremely high load and is taking steps to increase capacity above normal limits to address this. The current impact is that some requests to Chameleon are queued for a long time and/or timing out. In the UI this looks like page not loading, auto-saving not working, and for end-users experiences may not display. This currently appears to be happening for a < 10% of traffic but we'll continue to monitor as we work to resolve the issue

Report: "Profile Write Latency and Errors"

Last update
resolved

This incident has been resolved and systems are back to normal.

monitoring

We have identified the following issues: 1) Experiences sometimes not being delivered to end-users. 2) Experiences sometimes not being able to be opened/edited in Chameleon Builder. We're actively monitoring and investigating the root cause.

investigating

We are continuing to investigate this issue.

investigating

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

Report: "Profile Write Latency and Errors"

Last update
postmortem

Available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

resolved

This incident is resolved

investigating

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

Report: "Profile Write Latency and Errors"

Last update
postmortem

Available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

resolved

This incident is over but will remain open while the final data is processed

monitoring

This incident is has now been partially resolved, some data ingestion delay will be processed over the next few hours as more capacity is brought online to assist in working through a backlog of events and updates from integrations

monitoring

We're still actively monitoring this incident and will report any new information we have

monitoring

This incident is still ongoing, we will have more updates within the hour. The main impact is that requests that typically take 30-50ms are varying from 30ms to 3s, with some timing out and never completing, this is causing some Chameleon experiences to not start when they otherwise would

investigating

Intermittent slowness and timeouts resulting in occasional Application errors

Report: "Profile Write Latency and Errors"

Last update
postmortem

Post mortem available [here](https://status.chameleon.io/incidents/04c5y9b9d4xd)

resolved

This incident has been resolved.

monitoring

Response times and error rates are returning to normal and will continue to monitor. At this time, we know that a large increase in latency when writing to the database caused the latency of essentially all other operations to increase. We will leave this incident open until we're sure that the changes have a chance to full stabilize, the background jobs have processed, and the workload returns to normal levels/latencies.

identified

We are seeing elevated error rates, and various systems being slow or not loading. At the moment this includes the Chameleon dashboard, identifying users, and sending data to Chameleon. We know this is related to the performance of our primary database and will have an update again shortly.

Report: "Profile Write Latency and Errors"

Last update
postmortem

# **Chameleon Incident Report - Feb 10, 2025** ## **Summary** ‌Dashboard pages did not load or were slow, and end-user experiences did not show in a timely manner. This was caused by elevated traffic that resulted in high latency and errors primarily affecting database writes to the `profiles` collection in MongoDB. The issue cascaded into other collections and impacted the overall system health / performance. ## **Main Impacts** * Dashboard availability was reduced leading to errors on pages loading or slowness in loading data * End-user Experiences failed to show, or were delayed, in cases where these depended on updated user attributes \(because Chameleon was slow or not accepting updates to user profiles via the client side\) ## **Root Causes** * Unexpectedly high database “write” traffic exceeded the capacity of our MongoDB instance, leading to operations being queued beyond expected thresholds  * Our utilization of database capacity was higher than best practices in certain metrics \(which meant a smaller buffer before we exceeded capacity\) * Write locks were held longer than expected, causing significant queuing. * The disk was overutilized due to concurrent read and write operations, further increasing response times. * Insufficient RAM allocation to MongoDB exacerbated performance bottlenecks. ## **Resolution** This is what we did to resolve the incident as fast as possible: * Upgraded MongoDB instance RAM and disk capacity. * Monitored database performance post-upgrade to ensure stability. * Lowered concurrency of background processing across certain operations that, if left to run in parallel, would cause a flood or contending write operations. * Identified operations that may not need to be run live \(this will be incorporated into “Future Prevention” below\) ## **Future prevention** This is what we are doing and intend to do, to prevent this type \(similar or related\) incident in future: **Improving our capacity to handle growing volume of database traffic** * Evaluate further MongoDB scaling options to handle peak loads \(short-term\) * Improve our auto-scaling rules and capacity management, e.g. conduct additional load testing to anticipate future demands \(short-term\) * Improve database monitoring to detect and mitigate similar issues \(e.g. we’ve already added “total write time” as a new metric\) before they escalate; we found that our monitoring did not indicate exactly where the problem lay which delayed finding a resolution * Optimize “write” operations to reduce contention and queue buildup, especially those that are not needed to be live and can run in the background with batching, for example when updating company attributes data * Explore adding rate limiting to our Segment integration to prevent batch flooding of data * Accelerate our plans to upgrade our infrastructure to a more modern and state-of-the-art database system \(ClickHouse\) to supplement MongoDB in certain cases **Improving our processes and communication around issues** Partly due to the infrequent nature of incidents like this, we uncovered some material gaps in our approach in communicating these effectively, and accordingly will undertake significant improvements to this, including: * Clearer notices in-product, in our help center, and via email when an incident is occurring * Proactively suggesting to new customers \(and all existing customers\) to join the subscription list for incidents and security updates \(we already have this available via our dashboard and status page but want to increase visibility of it\) * Adding some expected timelines for regular updates to our Status page and completion of the post-mortem report _\(this document\)_ * A proactive email when an incident is resolved \(seems like “resolution” of an incident does not trigger an email update to subscribers from our Status tool, unlike for other incident categories\) and also to send out the post-mortem report * Formalizing support communication \(as we did in this incident\) including a “status ticket” in Intercom where we attach all messages about the incident, and a note in our support bot * Clearer ownership of tasks \(and accountability for these\) during an incident response

resolved

This incident was kept open for monitoring and has been resolved.

investigating

We are currently investigating the cause of intermittent slowness and timeouts resulting in occasional Application errors

Report: "Data collection and log in issues"

Last update
resolved

This incident is resolved (as of 4:30pm pacific). The main impact was that certain API actions were resulting in errors. These errors were most obvious for calls to track events where the result of the API call was a 500 error. In addition, some data was delayed in reaching the intended user because of delays in processing the incoming data. An incident like this will have an investigation followed by implementing the recommended updates. As always, you may reach out to us for further information, if needed.

monitoring

We've addressed the data collection and log in issues customers were experiencing. We are still monitoring this issue while we understand the root cause.

investigating

We are currently investigating issues with data collection and log in.

Report: "Data ingestion delays"

Last update
resolved

This incident has been resolved.

identified

Some data ingestion is experiencing higher than normal latency in being added to Chameleon's analytics charts and aggregate totals. The queues are on track to catch back up soon

Report: "Chameleon support for iframes using a different subdomain than parent page"

Last update
resolved

Chameleon's iFrame-support Beta has now completed its testing and release onto Chameleon's production version. This version now provides support for Chameleon Tours, Surveys, Tooltips, HelpBar experiences. If you have any questions about Chameleon's iframe support, please reach out to our Support team.

monitoring

There is now a beta version of Chameleon iFrame support available. This beta allows for Chameleon Tours and experiences to appear across iframes with different domains (both subdomains and primary domains). Please message Chameleon Support to request to be included in this beta.

identified

Impact: If your use of Chameleon depends on selecting page elements that reside within iframes, those specific Chameleon Experiences may not not be running correctly. Working: Main page is app.acme.com and the iframe is app.acme.com Not working: Main page is app.acme.com and the iframe is content.acme.com Further context is that prior to Chrome 115, iframes on the same origin (e.g. app.acme.com and content.acme.com) allowed their content is be accessible to the parent page. Chameleon uses this same-origin access pattern to find and attach Chameleon Experiences to the element found within an iframe. However, starting in Chrome 115, Google removed support for modifying a property called `document.domain` which permits this behavior. Now, because Google has disabled this feature, Chameleon cannot currently directly access content of iframes with subdomains that differ from the parent page. The Chameleon team is working quickly to restore full support for same-origin iframes. We plan to ship an early version of this to impacted customers in the coming days. If you are using iframes in the way described above, please reach out to your account manager or hello@chameleon.io let us know. Also indicate if you want to be added to the iframe feature preview when it's ready. Workaround: In the meantime, the workaround is to match the domain of the content inside an iframe with the content outside of the iframe (referenced as "Working:" above). Some teams we're in contact with have had success in simply proxying their iframed content through the same domain as their main page (let us know if you want more information on how to do this).

Report: "Chameleon dashboard slowness"

Last update
resolved

We're back to full speed and consider this incident closed. We will post more updates if we see any further dip in our metrics

identified

We are still working on this issue. At this time we're seeing some upstream slowness that is impacting our latency to the database and are working on a resolution

investigating

We're investigating a slowdown of the Chameleon dashboard @ https://app.chameleon.io

Report: "Microsurvey summaries delayed"

Last update
resolved

Some Microsurvey summaries from the previous 7 days were delayed and but will be re-run and delivered over the next few hours

Report: "Snooze settings incident"

Last update
resolved

On Feb 9th, Chameleon determined that a small percentage of some customers' end-user were not able to dismiss a specific type of Chameleon Experience (a Tour or a Microsurvey) due to an issue the application of the Experiences' Snooze settings from a Snooze that first happened prior to Jan 30th. We quickly fixed the issue and migrated the relevant data to bring your users back to the expected state. If your users were impacted by this, you will receive an email from our customer success team with more details.

Report: "Redis instance connection timeouts (2)"

Last update
resolved

We're going to close this incident as Chameleon has now been operating at full capacity for the last 12 hours. If you have any questions about this or anything else Chameleon-related please reach out to Chameleon support.

monitoring

We have increased the network capacity of our Redis cluster and we're in contact with the Heroku Redis team about the latency issues we're still intermittently experiencing. At this time, we're going to consider this incident closed but will continue to monitor until more time has passed without latency issues

investigating

This is a repeat of the previous incident (https://status.chameleon.io/incidents/x12r73srmcg2). Please check back for updates as we will update when we have more information on when all of Chameleon's services are back up and running. Customers may experience issues opening Chameleon's Builder in their applications, exporting reports, or loading certain Chameleon Dashboard pages.

Report: "Redis instance connection timeouts"

Last update
resolved

This incident has been resolved. The analytics data received for the time between Feb 6th, 15:31 UTC and Feb 6th, 16:28 UTC were not processed in real time. Our team in the process of re-running any data received during this time frame including experience analytics. Additionally many background jobs during this will also be re-run including data exports. If you do not receive your data export by the end of the day today please re-export the data or reach out to support

monitoring

Our main redis instance had a failover to its secondary and is catching back up with traffic. This redis instance handles background processing and will take some time to catch back up with realtime. Customers may experience issues opening Chameleon's Builder in their applications, exporting reports, or loading certain Chameleon Dashboard pages.

investigating

We are currently investigating this issue.

Report: "Missing redirect from legacy domain (trychameleon.com, now chameleon.io)"

Last update
resolved

This issue caused in users who clicked the "Dashboard" button from the Chameleon website (https://www.chameleon.io) to be redirected to https://app.trychameleon.com (which then redirected to the Login page at https://www.chameleon.io/login) instead of directly to the Dashboard at https://app.chameleon.io and caused it to appear as through you were logged out. The incident lasted a ~6 hours and was fixed as of Jan 11 08:30 pacific time.

Report: "Chameleon sub-processors notification"

Last update
resolved

We are contacting you as per the terms of our Data Processing Agreement (DPA) to inform you of new sub-processors being implemented by Chameleon. At Chameleon, data privacy and security are a priority and we understand the importance of keeping your company's data secure. We have recently completed our third SOC 2 Type II audit and leverage a continuous security monitoring solution to maintain ongoing compliance. Name Data Location Description OpenAI United States AI API platform Pipedream United States API automation platform These sub-processors have been evaluated according to Chameleon’s third-party risk management process. If you'd like to know more about our privacy and security practices, please visit our security page (https://chameleon.io/security) or contact us at security@chameleon.io. For more information pertaining to GDPR compliance, check out: How Chameleon complies with GDPR (https://help.chameleon.io/en/articles/1632723-how-chameleon-complies-with-gdpr). Thanks, Chameleon Security Team

Report: "HelpBar in GA"

Last update
resolved

HelpBar.ai is now officially live and publicly available. We will be monitoring usage of the API as usage climbs. If you're interested in the launch and supporting it please visit https://www.producthunt.com/posts/helpbar-by-chameleon or www.helpbar.ai

Report: "Chameleon Experience published status"

Last update
resolved

Your live Experiences should now be correctly displayed with the Live label and should be running normally. As such we're marking this incident as resolved. The key impact, in addition to a currently live Experience displaying with the [Inactive] label, is that any Tours / Microsurveys that would have started for the first time during this time period ending 5:45am PDT were kept from starting out of an abundance of caution by our backend -- thanks for being patient while we worked on and deployed this fix.

identified

We've identified an issue that is causing the display of Experiences as [Inactive] in the dashboard and are testing a fix at the moment. Will update when the fix is live.

investigating

We're investigating reports of published Chameleon experiences showing as unpublished/Inactive in the Chameleon dashboard. Will update when we have more information

Report: "App + API slowness"

Last update
resolved

For approximately 2.5 hours from 4:30am PDT to 7am PDT our App and API saw longer response times than normal. Operations that includes writes (vs read only) were impacted most.

Report: "Processing delays - Events and Companies"

Last update
resolved

This incident is now resolved. Our key metrics are back to normal and all queued jobs have been processed.

monitoring

We're seeing timeouts on approximately 4% of requests coming into Chameleon with the url prefix of https://fast.chameleon.io/observe/ and are still working to get back to fully operational

identified

We've identified a problem in how Heroku handles DNS resolution from Private spaces and are in contact with the support team. A large percentage of our servers are connecting to our database properly but the rest are failing to resolve the cluster address down to the addresses of the cluster instances. We're working on a possible workaround at the moment

investigating

We're seeing elevated error rates and retries related to processing Events and updates to Company properties. We're investigating the source of the problem and will report back here with more information.

Report: "HelpBar search results error rates"

Last update
resolved

We're marking this as resolved because the HelpBar search and DNS lookup are working as expected but we will plan to remove the temporary fix when Heroku gets to the bottom of the networking issue.

monitoring

We've identified the issue as isolated to resolving ip DNS via IPv6. Our engineers have deployed a temporary fix while Heroku works on the resolution of this incident. As of now they have not posted on their status page (https://status.heroku.com/).

identified

HelpBar search results are seeing elevated error rates as some of the calls into our search service are not correctly resolving the DNS on the remote host. We're working with the Heroku team to identify the cause of this issue and will report back when we know more. In the meantime, we're keeping track of the hosts that are not correctly resolving the DNS and rotating them out of the Heroku cluster.

Report: "Elevated rates of error - Timeouts"

Last update
resolved

We've resolved the second half of yesterday's incident which involved an single member of our application server cluster being reported as connectable and available to handle requests but internally the connections were being dropped by the heroku routing layer. This resulted in the a client-facing 503 error in approximately 4% of requests when contacting urls with the prefix https://fast.chameleon.io/observe/*.

Report: "Chameleon app / API slowdown"

Last update
resolved

While we're confident this incident is over for the immediate future, and while everything is catching back up (for a few types of data updates, mainly to companies) as everything returns to normal, we will be monitoring a few things more closely over the next few days.

monitoring

This incident is now on it's way to resolution; from our testing and metrics the Chameleon dashboard (app.chameleon.io) is now loading and Experiences are being displayed (e.g. Launchers, Tours). We will provide a final update here when all of our metrics/signals have returned to normal.

identified

As an update to the ongoing issue we're experiencing, our database provider's team (MongoDB Atlas) has identified the underlying issue is actively working on the resolution. Chameleon, for the most part, will continue to be unavailable until this is resolved. The primary impact is that new Chameleon Experiences will not be displayed to your users and the Chameleon dashboard (app.chameleon.io) is not loading.

identified

app.chameleon.io continues to timeout when loading -- the underlying cause appears to be a recent maintenance event at our MongoDB instance which has left it only partially available and are working with their team toward resolution

investigating

We're investigating a slowdown in serving API requests to Chameleon

Report: "Database failover causing slowness in Chameleon Builder"

Last update
resolved

An unexpected increase in traffic to Chameleon's database servers caused a brief slowdown with some requests taking up to 30 seconds, and some requests timing-out. This caused issues when trying to access the Chameleon Builder inside customer apps or use interfaces inside the Chameleon customer dashboard. The issue resolved itself at approximately 9:30am PST, when Chameleon's database servers automatically detected the issue and adjusted server resources accordingly.

Report: "iFrame access errors"

Last update
resolved

At around 11am PST today the Chameleon team released an update to our iFrame support that was intended to provide better support for multiple-layers of iFrames that show Chameleon Tours. For a small number of customers that had iFrames configured with subdomain origins may have seen errors reported from the Chameleon script attempting to access iFrames improperly. The change was reverted at 2:53pm PST with the release of the new feature coming at a later date.

Report: "Schedule publishing delay"

Last update
resolved

This incident has been resolved.

identified

We're restoring scheduled publishing / unpublishing to realtime and will be caught back up shortly

Report: "Email delivery delays"

Last update
resolved

As of 7PM pacific time Friday night transactional email services was restored and the pending deliveries were delivered albeit delayed.

identified

The Postmark team continues to work on this => update here https://ea5d5bbc.sorryapp.com/ We have seen some transactional emails being delivered but not yet back to 100%.

identified

Postmark confirmed it's upstream from them at the deliverability service -- emails will be sent as the upstream provider comes back from this incident.

investigating

Transactional emails (Magic links, Data export completion, Invites, etc.) are not being delivered at the moment. We've identified an issue with out email provider and are in contact with them and are following along here => https://status.postmarkapp.com/ which also appears to be having issues. Support email and everything else Chameleon is not affected by this -- reach out to us via hello@chameleon.io with anything that needs our urgent attention.

Report: "TLS Certificate errors"

Last update
resolved

The monitoring tool has confirmed they have resolved the issue.

monitoring

We're continuing to monitor this but have no indication that browsers are rejecting our valid TLS certificates -- the issue appears to be in how our monitoring tool interprets the validity window. We will keep this open until they report back and we see the intermittent errors no longer showing up.

investigating

Our CDN provider is investigating an issue with occasionally serving a TLS certificate that is either expired or only valid starting a small distance in the future. Our TLS certificates are provided by Lets Encrypt.

Report: "Reference error causing console logging"

Last update
resolved

This incident has been resolved.

monitoring

We've deployed the update for the cause of this issue and are watching the error rates drop.

identified

We've reproduced an issue that causes an error to be logged to the console when an Automatic Tour / Microsurvey is displayed -- this currently only logs the error but does not impact the Tour being displayed nor the behavior of the Tour itself.

Report: "Builder loading error"

Last update
resolved

This incident has been resolved.

monitoring

We're seeing an error that prevents the Builder from opening, have deployed and fix and are monitoring the update

Report: "Manually delivered tours not displaying"

Last update
resolved

This incident has been resolved. Please note that shortlinks **do not require any changes** to continue working as they previously did.

monitoring

We're testing this fix on staging and with a limited subset of accounts -- will report back when it's live for everyone

identified

We're investigating / identified an issue that caused Manually delivered Tours not to display. Typically they show immediately on page load after the short link is clicked. Quick reminder that Manually delivered tours are ones where your "Copy the short link" and paste that short link in an email or help center content.

Report: "Issues with opening builder from app.chameleon.io"

Last update
resolved

With the update to the Chrome extension to version 3.1.9 you should now be able to open the Chameleon Builder from the extension directly in your app again.

monitoring

We will continue to leave this open until we're confident you can open the Builder in all of the ways we expect you to

identified

The most recent update here is that you can now open the Builder from the Chameleon dashboard https://app.chameleon.io/ and Clicking "Create Tour" in the upper right. As the newly submitted Chrome extension is approved and is auto-updated you will also be able to open the Builder from the Chrome extension. Thanks for bearing with us today

identified

As an update to this incident, some people have reported their CSV export links were not working, the fix is to login and click the link from your inbox again.

identified

We're able to reproduce an issue that prevents some users from opening the Builder directly from the Chameleon dashboard on (app.chameleon.io) -- in the meantime please navigate to your web application and open the Builder from the chrome extension.

Report: "Chameleon is moving to chameleon.io"

Last update
resolved

We've completed our switchover to chameleon.io! As a parting thought, to be directly logged into https://app.chameleon.io simply visit https://www.chameleon.io and click the link in the upper right for "Dashboard" (or "Sign in" if you were already logged out of Chameleon and generate a new Magic link). We have taken great lengths to be your partner over the years, here's to many more!

monitoring

The last phase of the switch to chameleon.io is nearly complete, we're monitoring the suite of changes against a subset and will release more broadly throughout the next hour.

Report: "Status page updates"

Last update
resolved

This incident has been resolved.

identified

We're updating internal settings related to our status page (https://status.chameleon.io) -- be back soon

Report: "Chameleon Dashboard retrieving an HTTP 500 error"

Last update
resolved

This incident has been resolved

investigating

We are currently investigating reports of the Chameleon Dashboard being inaccessible and retrieving an HTTP 500 error. In case you experience a similar behavior outside of the Dashboard, please let us know by sending an email to support@trychameleon.com.

Report: "High error rates in Chameleon APIs"

Last update
resolved

A series of overlapping updates flooded a specific write queue rendering it 'blocked' until the writes completed, meanwhile a failover caused all of these writes to retry all at once when the new instance came back online.

monitoring

Database cluster health has returned to normal. We will continue to monitor this incident. Will post a more in-depth look at this when we have a more complete idea of the root cause.

monitoring

Our database appears to have failed over two times back to back and is taking some time to fully warm a new node for the replica set

investigating

We have seen increased error rates in responding to requests to identify users, loading the Chameleon Builder etc

Report: "Certain external APIs and Admin services slow/unresponsive"

Last update
resolved

This incident has been resolved and response times/rates are back within normal range. Any tracked Events that arrived during this time were queued and are currently processing (eta is 20 minutes to be back to real time). The team will add any additional information to this incident if there is anything new to report.

monitoring

We are continuing to monitor this incident. The main cause has a work around in place but the team are still seeing elevated error rates and things return to normal. Will update again in approximately 20 minutes

identified

A datastore responsible for rate-limiting and queueing is responding much slower than normal causing slow responses to API requests both internal and external. Will update shortly with more information.

Report: "Login maintenance"

Last update
resolved

While performing this maintenance, login is redirecting back to login :|

Report: "Chrome Extension Errors (v 3.1.3)"

Last update
resolved

The Builder version 3.1.4 has been published

identified

We have pushed a new version of our Chrome extension (v 3.1.4) to address and issue in a recently published version (v 3.1.3) that caused the Chrome extension to not load the "homepage" when you initially open the extension.

Report: "Segmentation and analytics outage"

Last update
resolved

This incident has been resolved.

monitoring

The main impact of this incident is now Resolved for the Chameleon Data API. For a period of ~6 hours today the autoscaler (and and turn the manual scaler) was not able to add additional resources to meet the typical morning/afternoon needs of the Data API. Any data that was received by Chameleon is now available for display on the dashboard and for access via API. As mentioned previously, all other aspects of Chameleon remained online during this incident (we used Chameleon to update our customers in the Chameleon dashboard 🎉 http://is.am/4qxa). We will hold this incident open until AWS and Heroku mark it as resolved, but since the impact to Chameleon is resolved, this will be the last email/slack alert you receive about this. Thank you for your patience today.

identified

The referenced issue on heroku is here => https://status.heroku.com/incidents/2390 - we're still keeping up with updates as they come and will report back here when we have a resolution.

identified

Due to the API and console outage at AWS us-east-1 and downstream for the Heroku API etc. the Data API will likely remain unavailable until those issues are resolved. Data sent to Chameleon, your Chameleon Experiences (live to your end-users), editing and publishing Experiences are not impacted at this time.

investigating

We are currently seeing Segmentation and Analytics data fail to load on the Chameleon dashboard and via API requests. We're currently investigating the cause of the issue, and will continue to provide updates as more information becomes available. URLs affected but this: https://api.trychameleon.com/v3/analyze/* and https://ultraviolet.trychameleon.com/*

Report: "[Security Information] Apache Log4j"

Last update
resolved

Chameleon nor any of our vendors are vulnerable; we have evaluated our systems and the systems of key vendors to check for the presence of Apache Log4j in reference to the Log4Shell vulnerabilities (CVE-2021-45046, CVE-2021-44228). We will continue to use best in class tools and continue to monitor the situation for any updates.

Report: "Tour Steps ignored URL rules"

Last update
resolved

This is a retrospective about an incident we discovered at 9:50am PT and immediately recovered from (the incident had started at 7:57am PT). A small subset of Tours with a specific legacy configuration [1] were displayed on pages that did not match the URL Rules. This resulted in Tours being erroneously displayed to users who would not have otherwise have seen the Tour. All other rules were correctly applied including Segmentation, Element matching etc. Upon learning of this issue we immediately rolled back the change and invalidated all relevant caches. We are following up with our customers directly to provide details about how this incident impacted their Tours and Users. Please feel free to reach out for more information. [1] We added support for multiple URL matching conditions (~11 months ago) and continued to support the legacy configuration of having only one URL match, when we deployed an update to how URL matching was handled the legacy configuration was omitted from the new code (and should not have been). Upon learning of this issue we immediately rolled back the change and invalidated all relevant caches.

Report: "Missing asset failover issue"

Last update
resolved

A missing asset correctly fell back to a previous version of the asset, however the previous version became unavailable during this time. Once the issue was discovered we republished the original and will investigate why the failover script was not available.

Report: "Input text color change issue"

Last update
postmortem

Our postmortem is [here](https://drive.google.com/file/d/1Jwq-xbo-UGRoj33PSOXIoDcPIRkUTPls/view?usp=sharing) please reach out to us with any questions or comments.

resolved

On March 16, 2022 from 12:51 PM PDT to 3:15 PM PDT, some customers saw the text color in text input fields changed to white (for cases where customer did not otherwise set the input text color in their own CSS). Once we discovered this issue, we shortly deployed the fix.

Report: "Heroku incident information"

Last update
resolved

Heroku security incident information (https://status.heroku.com/incidents/2413). Chameleon uses Heroku for the hosting of our application servers. At this time we have performed the checks and recommended steps as outlined by Heroku, GitHub and our own internal policies. As a precaution we revoked tokens where appropriate and have no indication that any of our private repositories were accessed. Our ongoing operations are not impacted by this.

Report: "Migration update"

Last update
resolved

For a brief period of time ~20 accounts may have seen an empty white box appear as an unintended part of a migration being performed. The issue was spotted and fixed within a few minutes.

Report: "Experiences Sandbox: Slow loading"

Last update
resolved

Surge is reporting their outage resolved (https://twitter.com/surge_sh/status/1520103571190165504) and we're seeing the same.

identified

An upstream provider is experiencing an outage noted here https://twitter.com/surge_sh. This issue is causing our Experiences Sandbox https://app.trychameleon.com/setup/experiences to either load slowly or not at all (If you get stuck please use the browser back button to go back to setup). The issue is isolated to the Sandbox in the "setup flow" on the Chameleon dashboard.

Report: "Help center updates"

Last update
resolved

This incident has been resolved.

monitoring

We're updating internal settings related to our help center -- be back soon

Report: "Developer hub updates"

Last update
resolved

This incident has been resolved.

identified

We're updating internal settings related to our developer hub (https://developers.chameleon.io) -- be back soon