Historical record of incidents for Wikimedia
Report: "Captcha-protected actions on all wikis are not working"
Last updateThis incident has been resolved.
We are working on a fix.
Report: "Captcha-protected actions on all wikis are not working"
Last updateWe are working on a fix.
Report: "Certain logging in and editing actions have errors"
Last updateWe are aware that users are having trouble editing or logging in Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "gerrit.wikimedia.org (Code Review) down"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "gerrit.wikimedia.org (Code Review) down"
Last updateWe are currently investigating this issue.
Report: "Wikis editing unavailability"
Last updateThis incident has been resolved.
The fix appears to be effective, and edit metrics are returning to normal, along with user reports of recovery. We will continue to monitor.
We have identified an issue that we believe is responsible for the impact to login sessions and edits, and are pursuing a fix.
We are investigating an issue impacting edits and / or logins, with impact beginning at approximately 2:55 UTC.
Report: "Wikis editing unavailability"
Last updateThe fix appears to be effective, and edit metrics are returning to normal, along with user reports of recovery. We will continue to monitor.
We have identified an issue that we believe is responsible for the impact to login sessions and edits, and are pursuing a fix.
We are investigating an issue impacting edits and / or logins, with impact beginning at approximately 2:55 UTC.
Report: "Wikis Editing unavailability"
Last updateWe are investigating an issue impacting edits and / or logins, with impact beginning at approximately 2:55 UTC.
Report: "Partial media unavailability"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing issue affecting media upload and download. We are investigating.
Report: "Partial media unavailability"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are experiencing issue affecting media upload and download. We are investigating.
Report: "Reduced availability for some users"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Some users are experiencing connectivity issues to Wikimedia project pages. Staff are investigating the issue
Report: "Reduced availability for some users"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
Some users are experiencing connectivity issues to Wikimedia project pages. Staff are investigating the issue
Report: "Wikis read-only: datacentre switchover"
Last updateMaintenance completed.
Scheduled maintenance is currently in progress. We will provide updates as necessary.
The SRE team will run a planned data center switchover, moving all wikis from our data center in Texas to the data center in Virginia. This is an important periodic test of our tools and procedures, to ensure the wikis will continue to be available even in the event of major technical issues.The switchover process requires a brief read-only period for all Foundation-hosted wikis, which will start at 14:00 UTC on Wednesday March 19th, and will last for a few minutes while we execute the migration as efficiently as possible. All our public and private wikis will be continuously available for reading as usual, but no one will be able to save edits during the process.
Report: "Trouble with images"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of issues with accessing images, and we are investigating.
Report: "Trouble with images"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of issues with accessing images, and we are investigating.
Report: "Editing issues"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Editing issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
The issue has been identified and a fix is being implemented.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Wikimedia Gerrit git repository is unavailable"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are aware that developers are having trouble updating and checking out code from gerrit.wikimedia.org and our engineers are working on it.
Report: "www.wikipedia.org portal unavailable on some locations"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware that many users are having trouble accessing the portal www.wikipedia.org for excessive redirects, and we are investigating.
Report: "Unscheduled maintenance to prevent database errors"
Last updateFor some minutes between 13:07 to 13:14 UTC, wikis in the s3 section ( https://noc.wikimedia.org/conf/highlight.php?file=dblists/s3.dblist ) were in read only mode (reads could happen, but edits were not allowed) due to unscheduled maintenance in order to fix ongoing database errors. The maintenance is over and the issue should be now fixed. More info at: https://phabricator.wikimedia.org/T385457
Report: "Occasional errors with Wikimedia commons media access"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Errors for logged in and some logged-out users"
Last updateThe remediation was successful.
A primary database server for cached data experienced an issue serving requests. We have corrected the issue and services are recovering.
Report: "Database issues in the Dallas TX datacenter"
Last updateTraffic back to baseline, root cause investigation is still ongoing but the incident is resolved.
We are continuing to monitor for any further issues.
We have repooled the primary datacenter for read traffic and are monitoring the situation while investigating the root cause.
We shifted read traffic to the secondary datacenter, which helped with the overloaded database. We are still investigating the root cause.
Reading and editing is affected. We are investigating the issue.
Report: "Trouble with some wikis"
Last updateFor approximately one hour after 5am UTC some uncached requests, such as those coming from logged in users, edits and api calls observed high latencies and/or errors due to an overload on MediaWiki nodes at the primary datacenter.
Report: "Site slow for users in the Asia region (eqsin)"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to work on a fix for this issue.
We are investigating issues with connectivity to our sites from the Asia region.
Report: "Eventgate failure in codfw"
Last updateThis incident has been resolved.
Eventgate has returned to normal operation, we are continuing to monitor the situation.
We are currently investigating this issue; in the mean time, events will not propagate properly, and some operations may fail.
Report: "Database issues"
Last updateThis incident has been resolved. We're continuing to address the root cause to reduce future recurrences.
We are monitoring the situation and continuing to investigate the root cause of brief recurring lag spikes.
We're currently experiencing a recurrence of lag on our MariaDB S4 primary instance.
Report: "Database issues"
Last updateWe've made changes to reduce primary database load, and things are stable for now.
We have mitigated the issue, but we are still searching for the root cause. We are monitoring the situation. Short recurrences are likely.
We're currently experiencing a recurrence of lag on our MariaDB S4 primary instance. We are investigating to determine the root cause.
Report: "Database issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We're currently experiencing a recurrence of lag on our MariaDB S4 primary instance.
Report: "Database issues"
Last updateThis incident has been resolved.
We are continuing to investigate this issue.
We're currently experiencing lag on our MariaDB S4 primary instance
Report: "Database issues"
Last updateThis incident has been resolved.
We're currently experiencing lag on our MariaDB S4 primary instance
Report: "Problems with our secondary datacenter"
Last updateThis incident has been resolved.
Everything is back to normal.
A fix has been implemented and we are monitoring the results.
We are currently investigating this issue.
Report: "Trouble with some wikis"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Issues with multiple wikis"
Last updateThis incident has been resolved.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Editing issues"
Last updateThe sites remain stable and we believe the incident to be resolved.
Things appear stable and we are monitoring.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Issues with some wikis"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of issues with saving edits to some wikis, and we are working on a fix.
Report: "Trouble with some wikis"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are continuing to investigate this issue.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Connectivity trouble"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware that many users are having trouble accessing Wikipedia and other Wikimedia sites, and we are investigating.
Report: "Wikipedia slow to respond"
Last updateThis incident has been resolved.
A workaround has been applied. Services have returned and are operational again. We are monitoring the situation and reviewing the incident.
We are aware that users are having trouble accessing and editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Some pages show errors"
Last updateFixes were successful and the issue has been resolved.
The fixes have rolled out and we are monitoring recovery.
The problem has been identified and fixes are being rolled out.
We are aware that some users are having trouble accessing Wikipedia and other Wikimedia sites, and we are investigating.
Report: "Trouble with some wikis"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Editing and reverting issues"
Last updateThis incident has been resolved. A summary will be posted here later.
We are aware that some users are having trouble editing or reverting on Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Connectivity trouble"
Last updateThis incident has been resolved.
We are receiving large floods of traffic that degrade performance for many users, and are working on fixes.
We are aware that many users are having trouble accessing Wikipedia and other Wikimedia sites, and we are investigating.
Report: "Network issues in our Amsterdam's site"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
Report: "Edits are not showing on some wikis"
Last updateThis incident has been resolved.
Some wikis, among others German Wikipedia, are not showing edits due to a problem with the databases. The issue is resolved, but the databases will take some time (in rare cases up to a few hours) to catch up. The edits made during this time are not lost, just slow to show up.
Report: "Trouble with images"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware that many users are having trouble accessing Wikipedia Images and other Wikimedia files, and we are investigating.
Report: "Wikipedia Mobile App articles interruption"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are currently investigating an issue that is preventing articles to load on the mobile apps.
Report: "Trouble with some wikis"
Last updateThis incident has been resolved.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Trouble accessing all wikis for customers of Vodafone Germany"
Last updateThis incident has been resolved.
We are continuing to monitor for any further issues.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Editing issues"
Last update[https://wikitech.wikimedia.org/wiki/Incidents/2023-05-25\_eqiad/LVS](https://wikitech.wikimedia.org/wiki/Incidents/2023-05-25_eqiad/LVS)
This incident has been resolved. A summary will be posted here later.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Connectivity trouble"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are continuing to investigate this issue.
We are aware that many users are having trouble accessing Wikipedia and other Wikimedia sites, and we are investigating.
Report: "Editing issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "Trouble with some wikis"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
We are aware of issues with accessing some wikis, and we are investigating.
Report: "Editing broken for 15 minutes"
Last updateWikipedia's session store suffered an outage for about 15 minutes. This caused users to be unable to log in or edit pages. Services are back online and review is in progress.
Report: "API errors"
Last update[https://wikitech.wikimedia.org/wiki/Incidents/2023-01-09\_API\_errors\_-\_db1143](https://wikitech.wikimedia.org/wiki/Incidents/2023-01-09_API_errors_-_db1143#Graphs)
We served an elevated number of 5xx errors for about ten minutes, roughly between 21:04 and 21:14 UTC. Users may have seen errors during that time. wbsgetsuggestions on Wikidata did not deliver suggestions. This was resolved by depooling one specific database server.
Report: "Trouble with some wikis"
Last updatevendor maintenance work has been completed and our Singapore datacenter has been repooled and is in service again
Readers in the Asia Pacific region should experience some minor performance degradation while we continue monitoring the situation. Otherwise, everything is up and running.
We 've identified an issue with our networking connections to our Singapore datacenter. We 've deployed mitigations and are monitoring the issue. Services should be available again globally.
We are aware of issues with accessing some wikis in specific geographical areas in Asia, and we are investigating.
Report: "Editing issues"
Last updateThis incident has been resolved.
A fix has been implemented and we are monitoring the results.
The issue has been identified and a fix is being implemented.
We are currently investigating this issue.
Report: "Editing issues"
Last updateThis incident has been resolved.
We are aware that users are having trouble editing Wikipedia and other Wikimedia wikis, and we are investigating.
Report: "User login / session issues"
Last updateNo further anomalies have been detected, edit rates and login failures have been within norm for the past 10 minutes.
The SRE team has applied a corrective measure, users should be able to log in and edit pages while logged in.
At the moment, due to a problem with our sessions service, users cannot login and sessions are being invalidated.