Historical record of incidents for Sharetribe
Report: "Delay in Event processing"
Last updateThere was an increased delay in the Event processing pipeline between 9:25 - 9:40. The issue is now resolved and the Event processing pipeline has caught up.
Report: "Elevated API Errors"
Last updateWe experienced an elevated level of API errors around 16:15-16:20 UTC.
Report: "Elevated latency in Marketplace and Integration APIs"
Last updateAround 18:20 to 18:25 UTC we encountered elevated latency in Marketplace and Integration APIs, caused by a database component failover.
Report: "Elevated error rates in Marketplace and Integration APIs"
Last updateFrom 21:15 to 21:25 UTC on September 20th we saw elevated error rates in Marketplace and Integration APIs. The incident was caused by faulty autoscaling of our search system.
Report: "Search temporary unavailable"
Last updateThe search engine was partially unavailable between 00:19 and 01:08 UTC. This affected all the API endpoints that use search, e.g., the Marketplace API `/listings/query` endpoint. During this time, the endpoints that use search returned 5xx responses. Our system couldn't recover from the partial outage without manual intervention. We keep investigating why the system didn't recover automatically and how we can improve it to prevent similar outages in the future.
Report: "Delays in Event processing"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating an issue causing delays in Event processing
Report: "Elevated API error rates"
Last updateThis incident has been resolved.
We rolled back some recent changes, which we suspect caused the issue. All systems are operating normally. We will keep monitoring the situation.
Increased delay in processing background jobs
We are seeing elevated error rates in API responses. We are investigating the issue.
Report: "Delay in background job processing"
Last updateUp to 30 minutes of delay in background job processing between 23:31 - 23:59 UTC.
Report: "Delay in background job processing"
Last updateUp to 30 minutes of delay in background job processing between 19:07 - 19:39 UTC.
Report: "Delay in event processing"
Last updateThis incident has been resolved.
Our mitigation was effective, the event extraction service is catching up.
There is a delay in processing new events. New events will become available in queries with some delay. We've deployed mitigation.
Report: "Delay in search indexing"
Last updateThis incident has been resolved.
There is currently approximately 20 minutes delay in user data indexing and the indexing is catching up.
We've mitigated the issue and it only affects users data, but not listings. This, therefore, affects only the Integration API and Console.
We're experiencing high delay in search indexing. Listing and user queries may match against stale data.
Report: "We are investigating an issue making the system partly unresponsive"
Last updateThe attack has subsided. The mitigations we put in place were successful in keeping Sharetribe systems fully operational.
We are under a DDoS attack but have managed to mitigate the attack and system is currently operating normally. We continue to monitor the issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
The system is under a potential DDoS attack. We are investigating the situation and trying to bring the system back up.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are investigating an issue making the system partly unresponsive
Report: "Markertplace APIs and Integration APIs are unresponsive"
Last updateToday, Friday 29th Dec at 4:20 UTC - 5:22 UTC we had a major system wide outage, affecting all the APIs, Console and CLI. At 5:22 UTC the system became partially operational. At 5:34 the system because fully operational. At 4:21 UTC our team was alerted and started immediate investigation. We identified a memory issue in our main database and after identifying it, the issue was resolved. To prevent this happening in the future, we took actions and allocated more resources to our database instances. In addition, we will continue investigating the root cause and we will implement additional alert system to identify the issue quicker in the future.
This incident has been resolved.
We are continuing to monitor for any further issues.
We are continuing to monitor for any further issues.
The APIs are up and running again; we will continue investigating the root cause and monitoring the system.
We are currently investigating an issue with Marketplace API and Integration APIs being unresponsive.
Report: "Slowness in search indexing"
Last updateThis incident has been resolved.
We have identified the issue and implemented a fix. We keep monitoring the situation.
We are currently seeing slowness in search indexing. We are investigating the issue.
Report: "Delay in search indexing"
Last updateThis incident has been resolved.
The indexing process is catching up, there is currently approximately 30 minutes of unindexed data.
Fix is implemented and indexing process is catching up with the data. This process may still take a while.
The issue has been identified and a fix is being implemented.
Search indexing is experiencing delays, listing queries may not match against most recent data.
Report: "Delay in processing events"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We've developed a fix and will deploy it asap.
The issue has been identified and a fix is being implemented.
We are investigating an issue preventing new events from being recorded.
Report: "Delay in search indexing"
Last updateAll data has been indexed and is up-to-date.
Our data indexing pipeline is experiencing higher load than expected and there is delay in processing data. Listing and user search APIs may not return up-to-date matches.
We are currently investigating this issue.
Report: "Elevated error rates when interacting with Stripe API"
Last updateAt 10:50:40 UTC, we saw elevated error rates when the system interacted with Stripe API. We assume this was caused by recent code changes, which are now rolled back. The rollback was completed at 12:24:25 UTC. We continue investigating the root cause.
Report: "Search indexing delay"
Last updateIndexing has caught up and operating normally.
The indexing process is going through a large backlog. We're monitoring and expect that the process will catch up in a few minutes.
Search indexing is delayed affecting listing and user search API endpoints.
Report: "[TEST] Testing integration to #flex-development Slack channel"
Last updateWe've now integrated https://status.sharetribe.com to #flex-development Slack Channel. This is a test that the channel gets notified when incidents are reported.
Report: "Delay in event sending"
Last updateDue to an error in event processing, events were not sent between Thu 9 23:56 - Fri 10 00:28 UTC. The issue is now resolved and all events have been sent.
Report: "Elevated latency in Marketplace API"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Marketplace API high request error rate and latency"
Last updateThe Marketplace API was unable to process requests due to temporary network issue.
Report: "Marketplace API high request error rate"
Last updateThe Marketplace API was unable to process requests due to temporary issue with an internal service. All APIs are operating normally now.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Delayed processing of background jobs"
Last updateThe issue is now resolved. We've updated our monitoring to catch such incidents early.
We have identified the issue and the backlog of jobs is processed. We are monitoring to confirm and working on fixing a blind spot in our monitoring that prevented us from noticing the issue in a timely manner.
The processing of background jobs is impaired and we're working through a backlog of jobs. Activities such as sending emails and automatic transaction transitions are affected.
Report: "Increased delay in search indexing"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are investigating increased delay in our search indexing pipeline. Recent changes to listings are not reflected in search results.
Report: "Elevated error count"
Last updateThis incident has been resolved.
We've identified the problem and deployed a fix. We're monitoring to confirm recovery.
We are currently investigating the issue
Report: "Background job processing delay"
Last updateEmail delivery has normalized. We will continue monitoring but we consider this incident resolved.
Background jobs have been processed. There may still be delays in some email delivery due to temporary restrictions in receiving email providers.
We've resolved the major bottleneck and are processing remaining jobs as fast as possible.
Because of unusually high volume of outgoing emails we have a delay processing all background jobs. These include email sending and automatic transitions in a transaction process. We are working on resolving the issue.
Report: "Elevated latency"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
We've increased our capacity and are seeing API response latency going back to normal.
We're experiencing elevated latency due to unexpected high load and are increasing capacity.
Report: "Elevated latency in Flex API"
Last updateThe issue has been resolved and all APIs are operating normally.
We've deployed mitigation and latency has normalized.
We're investigating elevated API response latency in Flex API.
Report: "Elevated API response times"
Last updateThis incident has been resolved.
We've identified the cause, deployed a fix and are now monitoring to confirm the issue is resolved.
We are investigating an issue with elevated API response times for certain queries to the API.