We recently addressed issues affecting the Search service. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 1:05 PM PT on February 9, 2023 and 3:00 AM PT on February 11, 2023, some users may have experienced difficulties while working in Box. During this time, some users were unable to search for items that were uploaded or changed during the impacted timeframe. The issue occurred as a result of an automation intended to optimize our messaging infrastructure. The automation put in place did not drive the optimization at a controlled rate, which caused overutilization of available resources and resulted in end-user impact. We resolved this issue by stopping the optimization automation. In addition, we have added guardrails to limit the optimization process to a subset of infrastructure to prevent similar issues from occurring in the future. 

Analysis 

On the day of the incident, we observed increased disk activity in our messaging infrastructure. This was the result of an automated optimization process to balance out disk space evenly across the underlying hosts in our messaging infrastructure. The optimization process ran across all the hosts, resulting in increased disk activity that affected our search service. As a result of this issue, our search service temporarily failed to index any file changes made after the onset of the issue. As a result, we stopped the optimization automation, and as a result, we observed incremental recovery until all the messages were processed. The issue was completely resolved February 11, 2023 at approximately 3:00 AM PT.

Corrective Actions

The following corrective actions have been completed or are planned:

We have restricted our disk space optimization process to limit the hosts to process at a given time.

We have identified additional alerts and checks to detect and diagnose issues with messaging infrastructure proactively.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team