For more informtion about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Minor] New files and folders not available to search
Incident Report for Box
Postmortem
Notice and disclaimer: Customer understands and agrees that this information is subject to change. To the best of our knowledge, this is the current state and will be updated as information is confirmed.
Last Updated: December 10, 2019

Summary Impact

Between 9:41 PM PST on November 15th and 2:51 AM PST on November 18th, customers were not able to search for newly uploaded or modified documents. This was caused by a confluence of two factors. The first happened on November 8th, when a new version of the search service was deployed and introduced a threading model change from a shared library. The second happened in the hours leading up to the issue, when the search service experienced increased load due to an internal maintenance process. The increased load in conjunction with the threading model change caused the search service to experience thread starvation, slowing down and, ultimately, preventing indexing of newly uploaded documents and modifications to existing documents.

Analysis

Multiple search service components use a shared library to perform critical actions. There was a change to the threading model used by this library. When this change propagated to multiple search components they all became susceptible to spikes in load. The threading model change operated without issues for a period of time; however, when a maintenance workflow exerted a high load on the search service, the underlying issue with the threading model change manifested. To restore the search service, the team rolled back to an old version of the service which did not contain the threading model change. To address the root cause, the search team fixed the threading model used by this library.

Defect Remediation

The following remediation actions have been completed or are planned:

  • Change to threading model of core library of search service.
  • Changes in health check monitoring and remediation process to actively respond in cases similar to this incident.
  • Process changes to service restoration in order to expedite recovery.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,
The Box Team

Posted Nov 20, 2019 - 17:06 PST

Resolved
The indexing queue has cleared and we are now considering this issue resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.
Posted Nov 18, 2019 - 06:52 PST
Update
The indexing queue has cleared and we are now considering this issue resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.
Posted Nov 18, 2019 - 06:51 PST
Monitoring
We are taking steps to remediate impact to search. At this time, we expect it will take ~20 hours to go through our backlog of items. We will keep this post open until we have confirmed we have fully resolved this issue. If this estimate changes we will share another update. To recap the current state: Files and folders created after 9:41 PST on Friday 11/15/2019 will not be returned in search results. In addition, any edits to files/folders that were created before 9:41 PST on Friday 11/15/2019 will not be reflected in search results.
Posted Nov 17, 2019 - 21:26 PST
Update
We are continuing to investigate investigate issues related to search. There are no significant updates to provide at this time. Please see previous posts to this incident for more information. If you are having an issue with search please contact us at https://support.box.com
Posted Nov 17, 2019 - 19:55 PST
Update
We are continuing to investigate investigate issues related to search. There are no significant updates to provide at this time. Please see previous posts to this incident for more information. If you are having an issue with search please contact us at https://support.box.com
Posted Nov 17, 2019 - 17:24 PST
Update
We are continuing to investigate issues related to search. The next update will be provided in one hour if no significant progress is made before then. To recap the current state: Files and folders created after 9:41 PST on Friday 11/15/2019 will not be returned in search results. In addition, any edits to files/folders that were created before 9:41 PST on Friday 11/15/2019 will not be reflected in search results.
Posted Nov 17, 2019 - 16:16 PST
Update
We are continuing to investigate issues related to content search with Box. The next update will be provided in one hour if no significant progress is made before then.
Posted Nov 17, 2019 - 15:18 PST
Update
We are continuing to investigate issues related to search. The impact extends from the original post to include any edits made on files or folders created/uploaded prior to 9:41 PM PST on Friday 11/15/2019. Those changes may not be reflected in search results.
Posted Nov 17, 2019 - 14:16 PST
Investigating
We are currently investigating an issue related to search. Files and Folders created after 9:41 PM PST on Friday (11/15/2019) may not be returned when searching content in box. This affects the web application search, content search in the admin console, and the API search endpoint.
Posted Nov 17, 2019 - 13:20 PST
This incident affected: Mobile Applications (Search), Box Web Application (Search), and Box Platform / API (Search).