This morning we experienced an unexpected production database outage for approximately 10 minutes.
This caused an outage on our web platform, affected both the UI and web services. Card spending was unaffected however.
The root cause was our main production database running out of storage, which caused an AWS autoscale event to add more.
The abrupt consumption of the previously ample available storage was caused inadvertently by some new features we had added to our platform. The consumption of the available space was so rapid that low storage alerts were not delivered in time for us to schedule a fix.
These features have now been rolled back and all services are running normally.
We are currently investigating how this error was made and making sure that the appropriate lessons are learned.