We recently addressed issues affecting Search. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 12:13 AM and 2:55 AM PDT on June 14, 2024 and between 8:04 AM and 10:44 AM PDT on June 17, 2024, some users may have experienced slow or failed Search queries while working in Box. The issue occurred due to a change in a query pattern coming into the system that caused increased load on the search backend. We were able to resolve the issue by adding additional capacity to the search backends. In addition, we are working on updating our resource allocation to prevent load on a few search backend nodes from affecting other nodes.

‌

Analysis

On June 14, some search nodes received a change in a query pattern that caused high load. This led to increased query latency and query failures, which eventually impacted users of Search. We were able to add more query nodes to handle the higher load. On June 17, the query pattern re-occurred with increased volume. In the second occurrence, we were able to again address the increased volume with additional capacity. We are reconfiguring resource allocation in the search backend nodes to better handle changes to query patterns. In addition, we’re working on solutions to prevent added load on a few nodes from cascading.

‌

Corrective Actions

Box has initiated the following corrective actions:

• Increase resources/right size clusters
• Optimize resource utilization to better handle the new query pattern
• Isolate search nodes to ensure better quality of service to all users
• Decrease the amount of time it takes to add new resources

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team