[Major] Issues with Uploads, Downloads, Public API and Logins

We recently addressed issues affecting the Box Web App and public API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On April 19, 2023 between 21:02 and 22:02 (Pacific Time), some users may have experienced difficulties while working in Box. During this time, these users experienced slowness and intermittent failures interacting with certain parts of the Box Web App and public API, including Logins, Uploads and Downloads. The issue occurred as a result of an internal process overloading the database pod responsible for serving approximately 1% of our customers. We were able to resolve the issue by stopping the internal process that was stressing the database. In addition, we have already started the process of upsizing and splitting the database as part of our routine maintenance for busy database pods to prevent similar issues from occurring in the future.

‌

Analysis

At the beginning of 2022, the database pod on which this issue took place was identified as a candidate for splitting due to organic growth of the database. The split successfully completed in May 2022. In December 2022, the database pod was once again identified as a candidate for splitting due to additional organic growth since the previous split. The second split kicked off in December and was estimated to complete in 8 weeks. However, a latent bug caused duplicate processing of some of the data on the pod; as a result, the split took longer than originally planned. Load on the database pod continued to increase while the split was taking place and occasionally reached levels that were not sustainable, resulting in impact such as what was seen on April 19. While waiting for the split to be completed, the database pod was migrated onto more powerful hardware, providing some relief. The database split ultimately completed at the end of April 2022.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Improve our database split process to reduce duplicative processing, improve efficiency in performing splits, and better scoping completion timeline.

• Identify even earlier database pods that require a database split

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team