[Critical] Issues with multiple Box services

We recently addressed issues affecting Box services. We would like to take the opportunity to further explain these issues and the steps we have taken to prevent them from happening in the future.

On December 16, 2025 between 4:25 PM PST and 7:23 PM PST, some users may have experienced slowness and temporary unavailability while working in Box. The issue occurred when we attempted to roll back a diagnostic change that was suspected of causing a minor degradation. The rollback unintentionally resulted in decreased capacity in our middleware service, leading to a degraded site experience. It also introduced a distributed deadlock condition that prolonged our incident response and mitigation efforts. Ultimately, we were able to resolve the deadlock and return to normal operations by temporarily throttling inbound traffic.

We have implemented measures to increase system capacity and minimize the possibility of a similar deadlock occurring in the future. We are also working on adding safeguards to avoid the rollback failure that served as the proximate cause of this incident.

Analysis

Between 11:11 AM PST and 1:38 PM PST, there was a production deployment of code that was aimed at gathering diagnostic information from our middleware service, in our ongoing efforts to improve system performance and resilience.

At 3:30 PM PST, we observed a brief, mostly imperceptible degradation and decided to roll back the change as a precaution. Due to failure in the rollback procedure, we observed a significantly reduced capacity in our middleware service that eventually led to a distributed deadlock condition between critical databases.

Box has made significant investments in reliability and mitigation measures. In this case, the issue was regional in scope and cross-region recovery capabilities are still in development. We therefore completed recovery by remediating the impacted region directly rather than executing a regional failover.

We restored middleware service capacity by 5:21 PM PST, but the deadlock condition persisted. After several lower-impact attempts to dislodge the deadlock were unsuccessful, we took a broader mitigation step and throttled all traffic at 6:54 PM PST. This resulted in a 26-minute full outage, after which systems stabilized and gradually recovered, with the site returning to full functionality at 7:23 PM PST.

Corrective Actions

Box has initiated the following corrective actions:

• Complete the planned capacity increase to reduce the chance of similar deadlocks and shorten recovery time if they occur
• Strengthen rollback safeguards so changes can be safely and reliably reverted, preventing a repeat of the rollback failure that triggered this incident

Thank you again for being a valued Box customer. We are continuously working to improve Box and want to make sure we are delivering the best in class product and user experience. Should you have any additional questions please do not hesitate to contact us.

Sincerely,

The Box Team