Historical record of incidents for URL2PNG
Report: "broad internet outage"
Last updateA broad outage is impacting many providers at this time. See link: https://downdetector.com/
Report: "Upstream outage at various providers (broad impact)"
Last updateThere is an upstream outage at various providers that started yesterday, which is limiting the availability of some web sites to be captured. This outage also includes our web site and customer dashboard at times. See https://status.salesforce.com/incidents/10014315
Report: "production traffic is impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "truncated viewport"
Last updateStarting at 10:30AM EDT, api requests with a defined viewport may have resulted in an image 4px shorter than expected. Image width was not impacted. As of 2:40PM EDT, this issue has been resolved.
Report: "production traffic impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Production traffic partially impaired on API - Capture"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Production traffic impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Production traffic impaired"
Last updateStarting at 21:00 UTC fresh captures started experiencing increased latency, with some request timing out and resulting in an error response. The majority of requests during this window did capture the desired page, despite the initial error response. At 21:15 UTC all traffic and error levels have returned to normal.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Production traffic impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Upstream Global CDN Outage"
Last updateThis incident has been resolved.
Fastly, our upstream CDN provider, suffered a service disruption starting at Jun 08, 2021 - 09:58 UTC. At Jun 08, 2021 - 10:57 UTC a fix was deployed on Fastly's systems and they are monitoring the situation presently. At this time all systems are operation on our end, but we are reviewing/monitoring traffic patterns during the return of global traffic and playing catch up with delayed requests now being processed and populating our outer most cache layer. Incident Report for Fastly: ( Global CDN Disruption - https://status.fastly.com/incidents/vpk0ssybt3bj )
Report: "Impaired production traffic"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "limited production traffic impaired"
Last updateThis incident has been resolved.
The cold cache layer behind our CDN became impaired, resulting in several short windows of high latency responses and a small number of error responses within those windows. We have repaired the cold cache layer infrastructure, and it is actively rebuilding and returning to normal operations. All requests are processing normally at this time, but the system will remain in monitoring until 12pm EST.
We are currently investigating this issue.
Report: "Production traffic impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results. (Fresh captures impaired from 23:37- 23:40 UTC. Cached results were not impacted.)
Report: "Increased latency on fresh captures (cached assets not impacted)"
Last updateThis incident has been resolved.
Capture times returning to normal. Related incident: https://status.fastly.com/incidents/8bk8y5q158pk
We are currently investigating this issue.
Report: "Upstream network/ip transit issues impacting traffic in some regions"
Last updateThis incident has been resolved.
At this time there is a global scale outage across a number of providers. Capturing against the internet is expected to be error prone until global routing is corrected/stabilized.
https://status.fastly.com/incidents/p1lwmmv2j2gq
Report: "url2png.com site/dashboard outage"
Last updateThis incident has been resolved.
Report: "CDN communication issues for www.url2png.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Production traffic impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "production traffic impaired"
Last updateThis incident has been resolved.
Report: "disruption to production traffic"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "upstream connectivity issues (NYC)"
Last updateThis incident has been resolved.
Digital Ocean has reported connectivity issues in their NYC1 region: https://status.digitalocean.com/incidents/lvyhmpwhvdt4 We did briefly observe an impact to instances in that region (and sites hosted from the region). At this time the impact appears limited. This incident will be kept in monitoring until the parent incident is fully resolved.
Report: "elevated rate of 500s in production"
Last updateMaking as resolved at this time. We will post an additional update once public statements are available from AWS on their ongoing DNS challenges today. - https://news.ycombinator.com/item?id=21327980
A fix has been implemented and we are monitoring the results.
The upstream dns/network issue has gotten considerably worse. At this time we have deployed a fleet wide work around. Error rates should continue to drop with this in place. We are actively monitoring the situation and will update as we deploy any additional resolutions.
We are continuing to work on a fix for this issue.
Related upstream event: https://status.digitalocean.com/incidents/1z3kmlvz69v6
The issue has been identified and a fix is being implemented.
Some request types are experiencing an elevated rate of 500s. Initial evidence points to transient upstream connectivity issues, but we are investigating as current status updates from AWS/DigitalOcean do not account for our regions at this time.
Report: "api.url2png.com routing outage"
Last updateThis incident has been resolved.
Around 12:44 UTC we experienced a routing outage for api.url2png.com. At 12:58 UTC we cut over to alternative routing and service has been restored. We will continue to monitor the situation.
Report: "beta api end point non-responsive"
Last updateFull captures resumed as of Jun 02 23:25:08 UTC. Google declared their outage to have ended as of 2019-06-02 11:45 (UTC).
If you have important capture traffic running on the beta api endpoint, we recommend you shift to the production api endpoint heading forward. Keep in mind, capturing assets still impacted by this outage will likely result in errors and/or abnormal asset capture.
This outage is related to a broader outage of upstream providers. See https://status.cloud.google.com/incident/compute/19003 however the impact is broader than just url2png, sites and services being served by GCP seem unavailable for many customers.
We are currently investigating this issue.
Report: "elevated rate of 504s on fresh captures"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "slow capture times being experienced on beta.url2png.com"
Last updateThis incident was resolved somewhat prior, but beta continued to experience infrequent network related issues. At this time we are seeing more stable capture traffic patterns. Customers are advised to move production traffic away from the beta endpoint and reach out to support about any blocking issues related to rendering or beta-specific features.
We are still seeing some infrequent networking issues, but capture times have returned to normal.
Our upstream provider is experiencing compute/network outages in regions important to this service end point (beta). We have launched additional capacity in other unaffected regions, but are still investigating and seeing network related impacts across regions.
Report: "api.url2png.com was impaired, rejecting new capture requests"
Last updateapi.url2png.com rejected new capture requests from approximate 17.50:00 to 17.59:00 UTC. All jobs are processing as normal since that time. Assets already in the CDN were not impacted during this time.
Report: "beta.url2png.com edge experiencing issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "slow capture times being experienced on api.url2png.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Additional infrastructure changes will be applied over the next 24 hours to mitigate the recent change in capture performance.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "production impaired"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "increased error rate on beta.url2png.com"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "beta api / general internet impaired by Google Cloud Global Load Balancer outage"
Last updateFor a brief period of time today Google suffered a global outage in their load balancer offering. This impacted requests issued to the url2png beta platform directly, as well as captures against sites hosted behind Google Cloud Global Load Balancers on the general internet. At this time, no outstanding issues are noted. For additional details see: Google Cloud Networking Incident #18012 The issue with Google Cloud Global Load balancers returning 502s has been fully resolved. https://status.cloud.google.com/incident/cloud-networking/18012
Report: "beta worker instances are being rebalanced"
Last updateThis incident has been resolved.
beta workers are being over provisioned to allow for rotation/rebalancing efforts. Until completed there may be a minor increase in latency on some capture requests.
Report: "Connectivity issues at upstream compute provider"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
An upstream compute provider is experiencing network connectivity issues in their NYC data center(s), impacting our ability to reliably capture from these locations and against sites hosted in these locations. At this time we are monitoring the situation.
Report: "Investigating an increase in 500 responses to capture requests"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Upstream connectivity in NYC3 impacted ability to capture pages"
Last updateAn incident with our compute provider specific to NYC systems, starting at Mar 05, 2018 - 03:57 UTC and being resolved at Mar 05, 2018 - 07:15 UTC prevent consistent capture from workers in that region. At this time, the incident seems fully resolved.
Report: "increased render times expected while we bring additional capacity online"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "api.url2png.com limited availability"
Last updateFor a brief period between 15:30 UTC to 15:55 UTC, api.url2png.com failed to serve fresh captures on some incoming job requests. Assets already being served from the CDN layer were not affected. This incident is now resolved.
Report: "An increased rate of failures is being experienced, we are currently investigating."
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "url2png :: api.url2png.com impacted - limited new renders being served"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "beta end point experiencing routing issues for some accounts"
Last updateRouting infrastructure was scaled to handle a burst in traffic.
A fix has been implemented and we are monitoring the results.
Report: "beta endpoint is experiencing an increased frequency of 500s during page capture"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Increased frequency of 503 responses"
Last updateIncreased frequency of 429/503 responses due to incoming network traffic volumes.
Report: "URL2PNG detected an interruption of service for the api.url2png.com end point"
Last updateThis incident has been resolved.
Service has been restored for most accounts, but we are actively monitoring api.url2png.com traffic levels.
Report: "Google Cloud Storage Incident #17002 related outage"
Last updateThe last related error was witnessed over an hour ago, and Google is indicating "All services available". Marking incident as resolved.
The beta endpoint should be functional again, with the last recorded error being about 29 minutes ago. Waiting on full resolution of https://status.cloud.google.com/incident/storage/17002 before closing out this incident.
The beta endpoint is currently being impacted by a related Google Cloud Storage Incident: https://status.cloud.google.com/incident/storage/17002
The beta.url2png.com endpoint is currently being impacted by a related Google Cloud Storage Incident: https://status.cloud.google.com/incident/storage/17002
Report: "Elevated disconnects on api.url2png.com"
Last updateService restored
The issue has been identified and a fix is being implemented.
Report: "Slow responses"
Last updateThis incident has been resolved.
load balancers appear happier after some manual pruning, we'll continue to monitor the siutation
Appears we are seeing some instability on our amazon edge nodes. Reprioritizing and relaunching instances.
Report: "First times requests slow/broken"
Last updateThis incident has been resolved.
Looks like one of our front end servers chewed up all its disks on today's logs! That is a lot of screenshots! Should be resolved now.
First time requests are not being returned properly
Report: "Intermittent request failures"
Last updateThis incident has been resolved.
We identified a rogue bot overwhelming a layer of our infrastructure. Additional resources have been deployed to mitigate this issue.
We are currently investigating this issue.
Report: "Large number of failed requests"
Last updateThis incident has been resolved.
We are currently investigating this issue.
Report: "Intermittent 500s on new requests"
Last updateWe quickly spotted and resolved a regional issue creating new screenshots. Stop by and say hi if you are at devopsday Boston.
Report: "Slower than normal response times"
Last updateThis incident has been resolved.
Additional resources hosted elsewhere have been brought on line, rate limit restrictions have been eased. Enterprise customer remain unaffected by this outage.
We are running on limited resources and some requests may take longer to fulfill. We are working to spin up more resources elsewhere.