[Critical] Issue with Multiple Box Services

We recently addressed issues affecting Box services. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

On February 6, 2025, between 5:00 PM PST and 7:00 PM PST, Box conducted a planned zonal resilience test as part of ongoing efforts to enhance system reliability. During the test, all traffic from one active service zone was diverted to other healthy zones. As a result, some users may have experienced slowness, login failures, or other difficulties while using Box. The impact lasted until 7:04 PM PST, extending four minutes beyond the planned maintenance window. During the test, some backend services experienced temporary overloading, making the impact more severe than anticipated. We mitigated the issue by redistributing traffic across all active zones.

‌

Analysis

During a planned zonal resilience test, all traffic from one active service zone was diverted to other zones. However, one zone unexpectedly experienced a disproportionate increase in traffic compared to the others. While the backend services in that zone generally had enough capacity to handle the load, a cache instance became overloaded due to a key that was accessed significantly more than others. This overload triggered a ripple effect, impacting database health, which in turn caused requests to pile up at the Edge layer, ultimately leading to request rejections. During the test, we successfully identified and root-caused the issue, mitigating it by ending the test and redistributing traffic more evenly across all active zones.

The issue revealed the following areas of improvements:

• Insufficient monitoring for single-instance failures during resilience testing.
• Uneven traffic distribution among zones, leading to localized overload.
• Lack of mechanisms to detect and prevent key issues from overloading cache instances.

‌

Corrective Actions

Box has initiated the following corrective actions:

• Enhancing zonal resilience mechanics to ensure more balanced traffic distribution during a zonal failure.
• Improving monitoring to detect single-instance failures more effectively during resilience testing.
• Optimizing cache access patterns to prevent key issues from overloading cache instances.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team