We recently addressed issues affecting Desktop Applications (Box Sync, Box Drive), Box Web Application (Uploads/Downloads), and Box Platform / API (Uploads/Downloads). We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 3:30pm PDT and 4:17pm PDT on Jan 9th, 2023, some users may have experienced difficulties while working on the Box platform. During this time, users may have experienced slow login times as well as failures when attempting to upload and download files.

Analysis

The issue occurred as an unexpected result of a resource utilization priority conflict during the deployment process of an internal tool. Specifically, an internal service being deployed to our production environment had a higher resource utilization priority than another critical system process, which subsequently failed when affected nodes’ resources were exhausted. We were able to resolve the issue by reverting the change.

In addition, we are improving our change risk assessment criteria and health alerts for this kind of tool, and working to design a more progressive deployment process for this internal tool to prevent similar issues from occurring in the future.

Corrective Actions

The following corrective actions have been completed or are planned:

• Improving our risk assessment criteria for deploying these types of tools.

• Extending our capacity guidelines to provide more clarity around system resource consumption and better resource distribution.

• Increasing our alerting to detect early warning signs of systems running at a higher capacity than expected.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team