We recently addressed issues affecting Shield and Classification. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 5:16 PM PDT and 7:40 PM PDT on November 23, 2022, some users may have experienced difficulties while working in Box. During this time, it was not possible to access Shield Admin pages, the Shield Admin dashboard and the Classification Label admin pages. Additionally, some users may have experienced issues manually classifying files and folders. The issue occurred as a result of an inadequate capacity configuration that prevented a service from starting gracefully after rebooting. We were able to resolve the issue by increasing the capacity of the affected service. In addition, we are working to extend our capacity guidelines and to improve our alerting capabilities to prevent similar issues from occurring in the future. 

‌

Analysis

The service affected by this incident recently experienced a significant increase in traffic as a result of the release of a new feature. Although the service remained stable after the feature’s release, the temporary degradation of another supporting service resulted in an increased number of retries, which in turn resulted in increased processing activity. As a result of this increased CPU usage, the impacted service eventually rebooted, but was unable to effectively reboot because the memory currently allocated for that service was inadequate during this window of increased processing activity. We were able to resolve the issue by increasing the capacity of the affected service.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Extending our capacity guidelines to provide more clarity around system restart considerations.
• Increasing our alerting to detect early warning signs of systems running at a higher capacity than expected.
• Improving the documentation of our existing alerts to allow us to respond more quickly to system reboot issues.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team