[Major] Customers may have been experiencing failures when using Uploads or Public API

We recently addressed issues affecting file uploads and downloads. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 03:10 AM PT and 03:27 AM PT on March 15, 2023, some users may have experienced difficulties while working in Box. During this time, certain file uploads and downloads may have failed. The issue occurred due to an automated security patching tool restarting a server, which caused some service discovery tools to fail. The issue resolved itself once the server restarted. As a result, we have hardened the server patching tool’s configuration to prevent similar issues from occurring in the future. 

‌

_Analysis _

The service discovery cluster requires a fixed number of minimum healthy servers running at any given time. The automated patching tool removed the primary service discovery server from the working server pool, at which point the cluster became unhealthy. None of the remaining servers in the working pool assumed the primary role responsibility as expected, which resulted in a temporary interruption of the service discovery function. Service discovery functionality resumed after the original primary server was patched and returned to the working pool.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Add additional server(s) to existing service discovery cluster to increase cluster fault tolerance.
• Modify automated patching tool configuration to limit impact on existing service discovery cluster.
• Migrate clients to use an updated and stable service discovery cluster version.
• Add additional monitoring to check service discovery cluster readiness.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team