Historical record of incidents for Embedly
Report: "app.embed.ly downtime"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "X (FKA Twitter) incident"
Last updateThis incident has been resolved.
We've identified an issue impacting x, formerly known as twitter, embeds. A fix is being deployed shortly, and we'll continue to monitor.
Report: "app.embed.ly maintenance"
Last updateThis incident has been resolved.
We are performing scheduled maintenance on app.embed.ly. Users won't be able to login for the maintenance period.
Report: "We are aware of a brief API outage lasting for about 4 minutes 15/11/2023, 1:51PM PST."
Last updateThis incident has been resolved.
The issue has been resolved, but we are watching closely.
Report: "Intermittent YouTube provider latency"
Last updateThis incident has been resolved.
Embed.ly is currently monitoring a situation where we can observe higher than usual intermittent latency coming from the YouTube upstreams which may impact Embed/Extract performance.
Report: "i-cdn.embed.ly, i.embed.ly image links"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are investigating an issue related to 525 errors upstream from our Display image links.
Report: "Images Missing"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We've identified an issue with image fetching with our Cards or Extract product. We're working behind the scenes to fix this bug as soon as possible.
Report: "High Latency"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "API/Cards affected by Cloudflare Outage"
Last updateThis incident has been resolved.
Cloudflare reported that the root cause has been addressed, and we're seeing recovery in API/Cards service health as well. We will keep monitoring.
Cloudflare is experiencing downtime at the moment, affecting our ability to properly route requests to our API/Cards services. We are waiting on an update from Cloudflare (https://www.cloudflarestatus.com/) regarding expected resolution.
Report: "Twitter Metadata Availability"
Last updateTwitter made a policy change to restrict the availability of tweet metadata, like thumbnail images, but most twitter functionality has been restored to the best of our ability.
This incident has been resolved.
We're working on a fix for Twitter metadata which can be missing from Embed and Extract content. This is related to a change in Twitter policy, and we're working quickly to resolve this issue if possible.
Report: "5/15/20 - Service Degraded"
Last updateThis incident has been resolved. If you have any further issues please contact us.
We are continuing to monitor for any further issues.
Core Embed/Extract functionality has been restored.
A fix has been implemented and we are monitoring the results.
We've begun to recover API functionality. You may encounter periods of degraded service performance as we move towards full recovery. Thank you for your continued patience as we work through this incident.
We're working around the clock to resolve this service outage and will continue investing all of our available resources towards restoring normalcy over the weekend. Thank you for your patience as we work through this incident.
We are actively working with AWS to resolve cloud infrastructure issues, which is still preventing us from reliably serving API (extract, embed) and Cards functionality.
We are actively working with AWS to resolve cloud infrastructure issues, which is still preventing us from reliably serving API (extract, embed) and Cards functionality.
We are currently experiencing networking failures in our cloud infrastructure, severely degrading our ability to provide API and Cards services. More updates to follow.
Report: "YouTube Embeds"
Last updateWe have recovered from an outage affecting YouTube embeds which we determined was caused by upstream rate-limiting from YouTube's servers. We now have a solution in place, but are continuing to monitor for any further issues. Please keep us posted at https://support.embed.ly about any further issues you may experience.
We are currently investigating an issue with YouTube embed functionality
Report: "Cloudflare System Status"
Last updateThis incident has been resolved.
We are currently tracking a Cloudflare incident that may impact our API performance for some customers: https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr
Report: "Elevated response times"
Last updateA connectivity issue between backend components was causing elevated response times in the Display product for about an hour this morning and has been resolved.
Report: "Display API"
Last updateEverything is back to normal.
We are looking into an issue with 502s for uncached URLs to the Display API.
Report: "High latency"
Last updateThis incident has been resolved.
We are reviewing a spike in our data service that caused API latency. Response times are returning to normal, but we will continue to monitor.
We are experiencing high latency on the Embed, Extract APIs at this time. We are currently reviewing the issue and hope to back to normal soon.
Report: "CDN Outage"
Last updateA bad config on our CDN caused an outage from ~11:15PST to ~11:35 PST. This has been resolved.
Report: "Display Usage Reporting"
Last updateThis incident has been resolved.
Due to a problem with an upstream provider, display endpoint usage in dashboard is missing data. The problem is being corrected up stream and the data should be filled in over the next couple of days.
Report: "Bad Deploy"
Last updateThis incident has been resolved.
Rollback complete, everything is returning to normal.
A bad deploy went out, bringing the API down. We are rolling back now.
Report: "Image Latency for Embedly Display"
Last updateThis incident has been resolved.
We're currently monitoring issues related to the DDoS attack affecting DNS providers in the US. This may cause higher than normal latency for image processing through i.embed.ly
Report: "app dashboard unavailable"
Last updateThis incident has been resolved.
We're reviewing an issue with our app dashboard access is temporarily unavailable.
Report: "Increased response times"
Last updateGood to go.
We are seeing response times going back to normal. We will continue monitoring.
We are bringing up more API worker boxes to help with response times.
Report: "API latency"
Last updateAn API update has been pushed to fix this issue.
We experienced some increased latency on API calls caused by a regression.
Report: "API increased response times"
Last updateThis incident has been resolved.
We found an issue with one of our fetcher boxes that was slowing responses down. We have removed and created a new box. Response should be returning to normal.
We are experiencing some performance issues with the API. Currently reviewing, expect slower response times.
Report: "Vimeo URLs intermittent"
Last updateVimeo links are now working through the API. Any previously called links will take about 12 hours to refresh.
Vimeo recently integrated some aggressive rate limiting for their oEmbed API and video detail pages. Embedly got caught up by their rate limiting and were unable to serve their embeds. Vimeo white listed Embedly, so things are returning back to normal. Due to Embedly caching, all URLs called in the last 12 hours will continue to error till they are cleared. For cards, this is 24 hours. You can manually clear Embedly's cache by adding force=true to your API request like so: http://embed.ly/docs/explore/oembed?url=http%3A%2F%2Fvimeo.com%2F18150336&force=true ( This should only be done on a manual basis) Please reach out to support@embed.ly if you have any questions. Shout out to Brad and Peter at Vimeo for getting us white listing.
We are working with Vimeo to resolve a rate-limiting issue that is causing intermittent failures for Vimeo URLs. This effects calls to the Embed, Extract API for Vimeo URLs. All Vimeo embeds previously called should work correctly.
Report: "Timeouts and Drop in Response times"
Last updateResponse times are back to normal.
We have replaced a bad cache box and will be monitoring performance. It should get better over time as the cache builds back up.
We are investigating an issue with our caching layer that is causing issues with load and timeouts.
Report: "High Latency"
Last updateEverything has returned to normal. 2 bad things happened at once: 1. One of services that checks URLs against the safe browse index died. This is generally not a big deal as it's meant to quietly rebuild itself while traffic keeps humming along. 2. We saw a drastic change in traffic patterns. This made the first problem manifest into a big deal. We have corrected the first problem and almost double capacity for the night.
Root cause found, latency is moving in the correct direction.
Still on it, it's the engineer's birthday too. Poor @anrope.
We are experiencing some increased latency, we are looking into this now.
Report: "API Latency above 2 seconds"
Last updateThis incident has been resolved.
We have isolated the traffic spike and responses should be returning to normal.
We are bringing up more boxes to handle the load.
We are seeing a large traffic spike of about 1000 req/sec.
We are looking into some latency issues, our reporting indicates latency is above 2 seconds. It's normally at 500 milliseconds.
Report: "Display Issue"
Last updateAll good in the hood.
The new box is deployed and latency is returning to normal. We have removed the worker box and will be investigating the root cause.
We are deploying a new display worker box to offload traffic. This should be up in next 10 minutes.
We are experiencing some issues with our Display Endpoint. Seeing 502 and 504 errors.
Report: "Bad build"
Last updateGood to go.
We introduced a bad build that caused issues with our API worker boxes. New build in progress. ETA 1 minute.
Report: "High Latency"
Last updateThings have returned to normal. <3
Things are still sluggish, but response times have fallen to acceptable levels. We are going to continue monitoring response times for awhile. The engineers believe the root cause in our deployment system, but I'm not sure I believe those guys.
Things are starting to improve, still looking into the root cause.
We are noticing some high latency. The team is working on a fix now.
Report: "502 Bad Gateway"
Last updateThis incident has been resolved.
Everything seems back to normal, but still monitoring for now.
Bad push, reverting and waiting for a fix to hit all the boxes.
We are seeing a lot of 502 errors, looking into it now.
Report: "APP Usage Data"
Last updateUsage data is now working on our dashboard.
Usage data is currently not appearing in your App Dashboard. We will be putting in a fix after the weekend to resolve this issue.
Report: "CDN IFrames are down."
Last updateEmbeds should all be working correctly. We are continuing to monitor to ensure embeds will work.
A fix has been implemented and we are monitoring the results.
Continuing to work through our CDN to get everything back up and running.
Our devops guy is playing with admin panels and configuring servers. I don't want to bug him, but we are working on it.
We have put in a temporary fix for http sites, but it is still down for https. We hope to have https back soon.
Our CDN provider terminated our account, they are currently working on re-activating that account now. Yes, we did pay our bill.
We are looking into this.