Historical record of incidents for The Tie
Report: "Unexpected downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
There is currently an outage with our central database, impacting the collection and distribution of new data across most datasets. We've identified the cause and are working with our infrastructure provider to implement a solution to bring things back online as soon as possible.
Report: "Unexpected downtime for many data components"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
There is currently an unexpected outage with our database that is causing many components on the Terminal to show no data or errors. We're working with our database provider to quickly resolve this issue and will share updates as they come. In terms of API impact: Unlocks API is affected. Sentiment API is not affected. News API is unaffected except for the trending news endpoints.
Report: "Database Provider Outage"
Last updateThis incident has been resolved.
Our database service provider is experiencing a global outage for all clients, which is impacting the majority of data on our platform. We are closely monitoring the situation and communicating with them.
Report: "Partial coin data outage"
Last updateThis incident has been resolved.
Market and sentiment data are temporarily not refreshing on the Terminal and in the Sentiment API but we're currently resolving the issue and they should continue updating soon
Report: "Sentiment outage"
Last updateThe incident has been resolved and data is flowing in again.
We are experiencing an outage for sentiment data. Values are being carried forward as we investigate the issue.
Report: "Terminal APIs Partial Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Twitter sources outage for news"
Last updateThis incident has been resolved and Twitter sources are back on.
Tweets for news have stopped streaming. We are investigating the issue. Sentiment data is not impacted by this issue.
Report: "Market data is currently not updating"
Last updateThis incident has been resolved.
Our market data provider is experiencing an outage currently, so our backup price feed and other metrics such as market cap, supply, etc. are currently not receiving updates. This affects price charts throughout the Terminal (stale values are being carried forward during the outage). However, our live market prices that update every second remain operational, as they're produced by a separate system.
Report: "Unusually low tweet volumes for some coins"
Last updateThe incident has been resolved, however please note that metrics containing longer look-back windows may still be impacted. Please do not hesitate to reach out to your account manager or to api+support@thetie.io if you have any questions or require further assistance.
A solution has been implemented and we are monitoring the results. This issue will be marked as resolved once daily tweet volumes return to normal levels in about 24 hours.
The issue has been identified and a fix is being implemented.
Some coins are exhibiting unusually low tweet volumes, which is also impacting their sentiment scores. We are investigating the cause.
Report: "Sentiment and Market Data Outage"
Last updateThere was an outage for sentiment and market data arising from a database issue starting at Sep 20 23:06 UTC. The problem was resolved at Sep 21 06:33 UTC, allowing data to continue flowing into the database again. We are investigating the root cause of the issue.
Report: "Sentiment data outage"
Last updateThis incident has been resolved.
Tweets have resumed flowing through our data pipeline and the calculation of all the metrics has resumed. Metrics containing a look back period will take time to recover to normal values (up to one look back period). However, we are working on accelerating this process by backfilling as much of the history as we can.
We restored the following sentiment metrics: Daily Sentiment Hourly Sentiment Long Term Sentiment Positive Tweets Percent Neutral Tweets Percent Negative Tweets Percent Because these are restored using a smaller but still representative sample of Twitter conversation, we are continuing to hold volume-related metrics static to avoid large discrepancies.
There's an issue with our Twitter Firehose connection that is preventing sentiment metrics from being calculated. Until it's resolved, we are carrying forward the last valid values we observed, which are from June 17th, 19:15 UTC (15:15 Eastern). Given the nature of the issue, we expect this to take time to resolve, as it's dependent on a response from Twitter. This does not impact tweets in the News feed and trending news.
Report: "Major AWS Outage affecting the Terminal and data streams"
Last updateThis incident has been resolved.
Most services have been recovered on AWS's side, and we are seeing most of the affected parts of the Terminal recovered. Continuing to monitor until everything proves stable.
Amazon Web Services (our infrastructure host) is currently experiencing a major outage that is affecting: 1. many dashboard components (ones built with the custom component builder) 2. market data is being carried forward with static data 3. data fetches for several pages (unlocks, companies, individual coin page, NFTs, etc) We are eagerly waiting an update from them and will report back when things are resolved.
Report: "Experiencing an outage with Twitter news sources and trending news"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing a data outage related to Twitter sources and the detection of trending news. Our team is actively working to recover this stream.
Report: "Unexpected downtime"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Currently experiencing unexpected downtime due to our database but are actively resolving.
Report: "News performance degradation"
Last updateThis incident has been resolved.
The infrastructure powering news queries and alerts is currently at capacity, causing slow load times and some downtime. We are investigating the cause and working to reduce load.
Report: "Temporary news downtime"
Last updateThis incident has been resolved.
We are upgrading our news infrastructure and it is taking a little longer than anticipated to complete, causing temporary downtime.
Report: "News query performance degradation"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Performance is temporarily suffering on news queries - this is affecting load times on frontend news feeds, News API calls, and alert delivery lag times.
Report: "Experiencing some timeouts due to server load"
Last updateThis incident has been resolved.
We're experiencing some request timeouts due to unexpected load. Should be resolved shortly
Report: "Ethereum data delayed"
Last updateThis incident has been resolved.
Our Ethereum blockchain database is currently delayed by about 7 hours and is catching up to real-time. As a result, the on-chain metrics and information you see are delayed accordingly. We'll post an update when it's caught up.
Report: "Increased error rates for Sentiment API"
Last updateThis incident has been resolved.
Extra load seems to have cooled off, but we're still monitoring. This was caused by an influx of API queries, which created a clog.
There are increased error rates for the Sentiment API due to a problem with the database.
Report: "Downtime on the SigDev Terminal & API"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently experiencing some downtime on the SigDev terminal and API.
Report: "Partial Data Outage"
Last updateThis incident has been resolved.
A fix has been implemented and we're monitoring the results.
We're experiencing issues with our servers that aggregate our data, which is causing partial outages. Investigating
Report: "News feed is down"
Last updateNews system is caught up and functioning normally
Our news system is temporarily backed up - should be resolved quickly
Report: "NFT Black Hole is affected by OpenSea API outage"
Last updateOpenSea's API appears recovered and our NFT data is restoring back to normal.
OpenSea's API is currently experiencing downtime, which is affecting the data quality of the NFT Black Hole tool. We're monitoring their API status and will update this incident when their issues are resolved.
Report: "Numerical scaling issue with Daily and Long Term Sentiment data"
Last updateThe problem is now resolved for data going forward. Still pending a backfill to correct the previous data.
Issue has been identified and the problem should resolve shortly. We'll post another update when current values are back to normal. On Monday we will backfill the incorrect data.
We noticed extreme values for Daily Sentiment and Long Term Sentiment metrics, which look to be the result of a raw data scaling/transformation issue. We're looking into the cause of this to resolve as soon as we can.
Report: "On-Chain data performance issue - updates paused"
Last updateResolved
Upgrade is done - monitoring
We are continuing to work on a fix for this issue.
We are having performance issues with our on-chain data and are currently updating the database - we temporarily paused updates to make this upgrade and should be back online soon.