Historical record of incidents for hCaptcha
Report: "Increased error rates for Google SSO due to Google outage"
Last updateDue to an incident at Google, we are observing increased error rates for users of this method. Contact support@hcaptcha.com if you need assistance with alternate methods. You may track status here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4
Report: "Inbound support email deliverability issues"
Last updateDue to an issue at an upstream mail gateway, some inbound support emails may have been delayed or rejected earlier today. This did not affect outbound email, and the issue has now been resolved.
Report: "Upstream CDN issues causing increased JS SDK load error rates"
Last updateThis incident has been resolved. As a reminder, please confirm that you are loading the JS SDK only via the recommended host: https://js.hcaptcha.com/1/api.js
A fix has been implemented and we are monitoring the results.
Report: "Elevated P99 latency on some requests"
Last updateP99 latency has returned to norm as of 20:05 UTC.
Due to an incident at an upstream CDN provider starting at 18:00 UTC, we are observing elevated P99 latency (+100-200ms) on a minority of traffic in the Eastern US and Western Europe regions, caused by some traffic being rerouted to more distant POPs. We have put overrides in place and will continue to monitor performance; there is no impact to availability and no action is required.
Report: "China region endpoints: elevated error rates"
Last updateAt Mar 22 01:21:56 2024 GMT, some users of dedicated mainland China region endpoints experienced an expired certificate error on some endpoints. The root cause has been identified and addressed. Global traffic was unaffected.
Report: "Reminder: Please ensure your server's root CA certificates are up to date"
Last updatehCaptcha APIs use several SSL certificate authorities, maintaining both primary and backup certificates; our CAA record is authoritative. We also automatically rotate certificates every three months as part of our security best practices. We received several reports today from customers running servers with outdated root CA entries. They either needed to update these after our most recent automatic certificate rotation, or had locked their validation for our endpoints to a specific certificate chain rather than relying on CA validation and our CAA records. Please ensure your servers calling the siteverify endpoint have an updated root CA store. This is an important security practice, as root CAs are occasionally compromised and removed from OS vendors' stores. Similarly, if you would like to enforce additional restrictions on validating our TLS certificates, please rely on the CAA record rather than hard-coding a specific intermediate chain.
Report: "Elevated scores for some traffic"
Last updateSome sites consuming hCaptcha risk scores briefly experienced an elevated score distribution, affecting approximately 3% of traffic. This incident was fully resolved by 01:28 UTC. This issue and its root cause have now been addressed, and adjustments scheduled to prevent any similar incidents in the future.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Login error rates will increase on dashboard logins in some regions during subsystem maintenance.
Report: "Elevated error rates on token verification"
Last updateToken verification endpoint error rates increased for several minutes on approximately 0.1% of sites during a token-related update. This was most commonly visible in an increased probability of a sitekey mismatch error.
Report: "Elevated error rates for some mainland China users"
Last updateError rates for some users located in mainland China and using dedicated regional routing endpoints were elevated during a rerouting event triggered by failures at an upstream provider. Mainland China users accessing standard global and first-party endpoints were unaffected.
Report: "Certificate issue on some China endpoints"
Last updateDue to an upstream caching issue, some endpoints for mainland China users failed to update during certificate rotation and briefly served requests using an expired TLS certificate. This affected only users of dedicated China endpoints; China traffic using global endpoints was unaffected.
Report: "Elevated challenge rate on some sites"
Last updateChallenge rates were briefly elevated for some traffic segments, with a duration of between 0 and 20 minutes depending on region. This issue has now been resolved.
Report: "Challenge images failed to load for some WAF customer sites"
Last updateA behavior change in an upstream deployment workflow caused some paths related to WAF customers' asset host paths to temporarily de-route. While most requests (>99%) were re-routed automatically with no user-visible impact, some WAF customer sites using custom asset hosts failed to load images for visual challenges. This issue has now been resolved.
Report: "Elevated challenge rate for some users"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
Report: "Dashboard login services under maintenance"
Last updateThis incident has been resolved.
You may see errors or logouts as dashboard login services are updated over the next hour. If so, please clear your cookies and cache and retry login.
Report: "Dashboard SSO logins under maintenance"
Last updateMaintenance is now complete.
Dashboard SSO services are being updated, and login error rates may increase for SSO users over the next few minutes.
Report: "Elevated error rates"
Last updateFrom 12:11 PM PT through 12:42 PM PT, users in some regions experienced elevated error rates and timeouts due to an upstream service issue at one of our CDN partners, primarily affecting Americas traffic. We are continuing to monitor their resolution, and will re-route traffic onto our other providers as necessary in the event of recurrence.
Report: "Asset redirection, Elevated error rates on some image URIs"
Last updateUsers in some regions briefly experienced asset redirection leading to "too many redirect" messages or 500 errors due to a cache policy update; this was not visible to most users due to caching and automatic client-side retry logic, but generated increased errors for client services monitoring browser sub-request status codes.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
AWS continues to have a major multi-AZ outage affecting some data resources. We are continuing work to migrate these resources.
Due to a multi-AZ outage at AWS affecting RDS and other resources, database services are currently being migrated.
Error rates will increase while database maintenance is performed.
Report: "Dashboard under maintenance"
Last updateFree user accounts in some regions experienced elevated error rates on login while maintenance was performed.
Report: "New account creation error rates may increase during dashboard maintenance"
Last updateThis incident has been resolved.
Account creation may not succeed in some regions while maintenance is underway: no other features are impacted.
Report: "Elevated error rates on new signups"
Last updateThis incident has been resolved.
Signups of some new account types in some regions may return an error over the next 10 minutes as maintenance is performed.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Metrics requests for some sitekeys will be under maintenance for approximately 30 minutes as cache service updates are made to metrics systems.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Logins and signups on the Dashboard may see elevated error rates over the next 15 minutes as changes are made to associated database resources.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Error rates for logins are currently increased due to an ongoing cloud provider service incident. hCaptcha CAPTCHA service is unaffected.
Report: "Elevated error rates on some requests in Brazil"
Last updateThis incident has been resolved.
Due to an issue with an upstream provider, 0.35-0.87% of requests in Brazil (primarily in Sao Paolo and nearby regions) are experiencing elevated error rates. We expect to resolve this shortly.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
We are continuing to work on a fix for this issue.
Dashboard logins and authenticated API requests will be unavailable for approximately 3 minutes.
Report: "Elevated latencies on some requests"
Last updateDue to reachability issues with an internal system component starting at approximately 9pm PT, some edge nodes were unable to retrieve all data for some requests. The end-to-end system continued to function for all requests with no increase in error rates, however with higher latency than normal. Internal metrics did not immediately identify this because it occurred only on some combinations of requests and request parameters, and the particular timeout behavior did not trigger the internal latency SLA cutoff. An audit has been scheduled to verify that all internal requests meet the system’s internal latency enforcement, and work has been scheduled to expand external monitoring to more combinations of requests and request parameters.
Latencies (but not error rates) increased on requests in some regions, primarily on free accounts.
Report: "Elevated error rates on siteverify"
Last updateThis issue was caused by a rollout of new code that contained a bug for some combinations of feature flags and request data. This should have been caught by other safeguards, but due to an unrelated change in the system those safeguards no longer triggered in this case. Improvements to the affected systems have been scheduled.
Approximately 0.009% of requests to siteverify with some challenge modes received an `invalid-passcode` response, starting around 1:20PM PT and continuing for ~30 minutes. This affected less than 1% of sites.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
Report: "Dashboard updates may take longer than usual"
Last updateThis incident has been resolved.
Updates made to sitekeys and account settings make take longer than usual to be reflected in the live production service for some users while queue maintenance is underway. This will affect approximately 5% of users in the EU region.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Some users may be unable to log in to the dashboard over the next ~10 minutes during maintenance.
Report: "Sitekey creation temporarily paused for free users during dashboard maintenance"
Last updateThis incident has been resolved.
For free (non-enterprise) accounts, self-serve sitekey creation will be paused for 10-15 minutes. You may see an error message during this window.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
The dashboard will be under maintenance for approximately five minutes, and may be unavailable for users in some regions during this time.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
The dashboard is currently under maintenance, and will not be available for login in some regions over the next 10 minutes.
Report: "Sitekey settings changes slow to update"
Last updateThis incident has been resolved.
Sitekey settings changes made via the dashboard are currently propagating slowly in some regions.
Report: "Some files served with missing content-type for some users"
Last update~0.1% of users received pre-compressed files without an appropriate content-type header for several minutes, causing the checkbox to fail to render on some browsers where those files were not already cached.
Report: "Sitekey config changes and stats updating slower than usual"
Last updateThis incident has been resolved.
Performance should now be normal in most regions.
Sitekey config changes and metrics updates may take longer than usual to take effect as we perform maintenance on related queue infrastructure.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Users in some regions will be unable to reach the dashboard for several minutes during maintenance.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
The dashboard is being upgraded, and some features may be temporarily unavailable for several minutes.
Report: "EU region: siteverify slow for some Free/Publisher sites"
Last update~0.13% of verification traffic to free sites saw increased latency and error rates for several minutes as our EU origins autoscaled to deal with unusually high request volume. This has now been resolved.
Report: "Signups paused"
Last updateThis incident has been resolved.
Signups are temporarily paused while the dashboard is under maintenance.
Report: "Signups temporarily paused"
Last updateDashboard maintenance is now complete.
New user signups are temporarily paused on the dashboard during maintenance. This restriction will be lifted in approximately 10 minutes. All other dashboard features are working normally.
Report: "Estimated earnings not updating for some users"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
On December 23rd a change was made to the dashboard that caused estimated earnings to stop updating for some users. This does not affect actual earnings or other service functionality.
Report: "Scaling up database up"
Last updateThis incident has been resolved.
We have verified the operation was successful.
Reprovisioning is complete.
We are currently reprovisioning a database server used for user data. This will not affect the captcha service.
Report: "Issues with captcha completion workflow"
Last update**Immediate cause:** A transient database connectivity issue caused some data to be inaccessible to the sync process responsible for distributing it to edge nodes, causing a new constraint check within siteverify to fail for some users as this code change deployed across our network and edge services restarted without access to the required data. **Root cause:** The root cause of this issue was an unintentional dependency on database access added during a recent code change to more stringently enforce some per-user constraints on the siteverify call, which has now been addressed. **Resolution:** The database connectivity issue was resolved, and the constraint check was improved to handle the no-data-available case correctly. **Ongoing work:** All parts of our system are designed to continue functioning fully even during internal issues like database connectivity failures. We are implementing an additional set of CI/CD safeguards to more aggressively verify that this behavioral contract is maintained for all code changes.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
Currently investigating an issue with siteverify calls.
Report: "Slow Dashboard"
Last updateThis incident has been resolved.
We are experiencing some database slowdowns that may cause logins and other actions on the dashboard and accounts endpoints to take longer than usual. The captcha service and related endpoints are unaffected.
Report: "Dashboard slowness"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We are currently investigating this issue.
Report: "Some dashboard features temporarily inaccessible"
Last updateDue to the interaction of a recent database migration and code changes to the Dashboard APIs, some user features were temporarily hidden after login. This has now been resolved.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Some regions will be unable to access the dashboard during maintenance.
Report: "Dashboard under maintenance"
Last updateThis incident has been resolved.
Dashboard statistics and sitekey-related data may fail to load for some users during maintenance. Other functions will remain available.