[Outage] Issue with Box Services

For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

Incident Report for Box

Postmortem

NOTICE AND DISCLAIMER: CUSTOMER UNDERSTANDS AND AGREES THAT THIS IS PRELIMINARY INFORMATION SUBJECT TO FURTHER REVIEW AND ANALYSIS BY BOX. TO THE BEST OF OUR KNOWLEDGE, THIS IS THE CURRENT STATE AND WILL BE UPDATED AS MORE INFORMATION IS CONFIRMED.

LAST UPDATED: December 20, 2019

Summary Impact

Between 12:53 PM PST and 1:17 PM PST on December 3, 2019, some users may have experienced difficulties while working in Box. During this time, access to uploads, download and public API was impacted, and the Box All Files and Login pages were unavailable for 3 minutes.

Analysis

The issue occurred because one metadata SQL pod failed and the impact of this cascaded to the database service layer, which caused it to max out connections to other databases. We were able to resolve the issue by flipping over to a passive database and restarting the failed database instance. In addition, we are working to improve the handling of database failures to prevent similar issues from occurring in the future.

Defect Remediation

The following remediation actions have been completed or are planned:

Improve the global track connection closure so it wont be blocked on a long running query on a connection.
Configure the MySQL fleet to support extra reserved connections beyond the 1 reserved connection enabled by default.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,
The Box Team

Posted Dec 05, 2019 - 15:43 PST

Resolved

No additional impact has been observed and this incident is now considered Resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.

Posted Dec 03, 2019 - 14:34 PST

Monitoring

Our teams have validated that all affected services are recovered at this time. We are continuing to monitor for any additional impact.

Posted Dec 03, 2019 - 13:36 PST

Update

Our team has identified the cause of this issue and is actively taking steps to remediate impact. Our monitoring indicates initial recovery is underway. We will continue to provide additional updates as they become available.

Posted Dec 03, 2019 - 13:30 PST

Identified

Posted Dec 03, 2019 - 13:27 PST

Investigating

We are currently investigating an issue with Box services. We will provide more information as it becomes available.

Posted Dec 03, 2019 - 13:10 PST

This incident affected: Box Platform / API (Content API, Content Preview, Authentication (OAuth 2.0 / JWT), Search, Uploads/Downloads, Webhooks), Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search, Preview, Sharing (Shared Links), Email Notifications, Admin Console & Functionality, Governance (Retention), Workflows and Automations, Comments and Tasks, Accessible Site (a.box.com)), Desktop Applications (Login/SSO, Box Sync, Box Drive, Box Edit / Tools), Mobile Applications (Login/SSO, Preview, Search, Uploads/Downloads, Mobile-Optimized Website (m.box.com)), and Box Notes (Web Application).