We recently addressed issues affecting File Request. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 09:23 am PDT and 10:33 am PDT on March 24, 2023, some users may have experienced difficulties while working in Box. During this time, users may have experienced failures of certain File Request operations. The issue occurred due to a scheduled change to add a new server in the service discovery cluster, which eventually impacted the File Request service. We were able to remediate this issue by reverting the change and performing a manual restart of the File Request service. Additionally, we have hardened the server deployment strategy to prevent similar issues from occurring in the future. 

‌

Analysis

File Request transitively relies on service discovery tools via using a distributed database system. An attempt to add a new server in the service discovery cluster triggered a chain reaction that ultimately caused all the file request database operations to timeout.

After the addition was reverted, the file request service had to be manually restarted in order to reset the primary leader information cached by the database client in the service.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Proactively upgrade service discovery tools to more stable versions.
• Add additional server(s) to existing service discovery cluster to increase cluster fault tolerance.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team