We recently addressed issues affecting certain parts of the Box Web App and public API, including Logins, Uploads and Downloads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 11:35 AM PT and 12:42 PM PT on June 1, 2023, some users may have experienced difficulties while working in Box. During this time, some users experienced slowness and intermittent failures interacting with certain parts of the Box webapp and public API, including Logins, Uploads and Downloads. The issue occurred as a result of a recent code change in our ongoing effort to improve performance and stability. We were able to resolve the issue by rolling back the code change. We are working to improve our deployment and monitoring processes to prevent similar issues from occurring in the future. 

‌

Analysis

Just prior to the start of the issue, we deployed a change that introduced a new database query in an attempt to simplify complex legacy queries and reduce database load. This new query was more expensive than anticipated, and caused the database to overload on larger tables in the production environment. Customers may have experienced failed logins, uploads, and downloads as a result. This change was tested in our pre-production environments for a week prior to release. However, our pre-production environments have different load characteristics, making it difficult to identify the problem the query would cause under production load. We are putting additional processes in place to thoroughly validate any database query changes prior to production deployment in order to prevent this from happening again.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Rework the solution to improve expensive database queries.
• Improve the deployment process to include more incremental stages, streamline the verification process, and expedite rollback speed.
• Formalize a process that tests DB effectiveness to better assess query impact when we adjust/create them

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team