Simwood Incident History

This incident has been resolved.

We have some out of hours house-keeping to do here but this incident is otherwise resolved. We will be monitoring for a few hours before closing to be certain.

CDRs are now being processed at full speed and catching up rapidly.

Portal and API (v4) are now fully operational. CDRs remain lagged.

We have lost hardware in Slough. Voice services have failed over as expected and are unaffected. Most other services have recovered too although we are currently dealing with some Redis master failures elsewhere on the network. This will manifest in portal logins and writing to selected API end-points. Separately CDR processing is several million delayed so API balances etc. will not be current. We are recovering Redis clusters as P1 and will advise progress.