***UPDATED: JUNE 28, 2021 09:30 AM PST***

We recently addressed issues affecting Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 6:55pm PDT and 8:01pm PDT on June 2, 2021, some users may have experienced difficulties while working in Box. During this time, Box Notes was intermittently unavailable. The issue occurred due to a latent bug that caused several servers to stop serving traffic unexpectedly without notifying operators. We were able to resolve the issue by moving traffic to another data center. In addition, we are working towards a fix for the bug and improving our metrics for early detection of this condition to prevent similar issues from occurring in the future

Analysis

A bug in the way servers make themselves available to take requests caused several servers to stop taking requests. As a result the remaining instances of this particular type of server received more requests than they could safely handle. The servers that were not in traffic continued to report themselves as healthy so operators were unaware of the problem until an alarm was raised due to the increased errors on the still in-traffic servers. Believing that all servers where still in service and that the errors where due to a problem on the servers that were reporting the errors, operators re-started one of the still active servers. This increased the load even further on the remaining servers and caused Box Notes to be temporarily unavailable. 

Several problems were discovered in the course of this issue.

1. The code bug that caused servers to stop taking requests.

‌ 2. The lack of an alert when otherwise healthy servers stopped taking traffic unexpectedly.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• The code bug has been identified and a fix is planned.

• An alert will be implemented to detect the out of traffic servers.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team

Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.