For more information about our Incident Response and Communications please read this support article.
We also maintain a list of Known Product Issues separate from this site here.
[Critical] Issues with Multiple Box Services
Postmortem
We recently addressed issues affecting multiple Box services. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.
On May 7, 2025 between 01:21 AM and 01:00 PM PT, some users may have experienced difficulties while working in Box, particularly when interacting with Box services. The issue occurred due to a code change that inadvertently increased the memory requirements in our database backend for certain queries. This issue triggered automated remediation tooling, however, it was unable to keep up with the consistently increasing load. This resulted in degraded availability for some of our services that perform write operations for a small subset of users. We were able to resolve the issue by rolling back the change once it was identified. We have implemented enhanced monitoring to quickly catch and mitigate similar issues in the future.
Analysis
The Search Team is actively working to reduce overall quicksearch latency for customers. Quicksearch is a fast search request executed in the Box Web Application by typing a query term (without pressing Enter to continue to FullSearch). As part of these efforts, the team implemented validation logic that inadvertently issued a large number of parallel queries against our databases, causing them to experience memory exhaustion. This issue was not caught in testing due to differences in our production data compared to test data.
Corrective Actions
Box has initiated the following corrective actions:
- Creating test datasets that reflect production data distribution.
- Enforcing configuration of bounded execution pools for asynchronous processing, preventing future uncontrolled surges and improving system resilience.
- Conducting a round of rollback exercises using standardized pipelines while ensuring rollback links are centrally accessible and reliable.
- Improving logging and monitoring on database nodes and critical dependency traffic, to enable earlier detection of abnormal load patterns.
We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.
Sincerely,
The Box Team