Back office & API Issues

The Issue At approximately 9:27am BST we were alerted to some potential issues regarding logging into our CMS. Upon investigation, we discovered an issue with our database server that is used to control logins to Showoff and cache some GA stats, which affected a a small number of sites.

Upon further investigation, it was discovered that the server had become in-accessible. A reboot was performed, which did not bring the server back online and we were not able to remove onto the server to further diagnose.

Resolution At 10:15 am BST the decision was taken to re-build this server from a backup and migrate our back office to use this new database server to bring systems back online. At approximately 11:15 am BST the back office started to become available again and logins were allowed. By 11:40am BST all back office access had been restored and we started to migrate our frontend clusters to use this new server. By midday, all clusters were pointing to this new server and all services had been brought back online.

Lessons Learnt To mitigate this issue in future, we’ll be migrating away from the server that failed, to a different more resilient architecture. This has always been planned as part of our hosting modernisation but in light of this outage, we’ll accelerate those plans to remove this database server from our infrastructure.

All systems are operational again and we'll continue to monitor the platform

Our monitoring systems have picked up a potential issue. The team have been notified and we are investigating this further.