We recently addressed issues affecting uploads, downloads, notes, and public API. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 6:55 pm PDT and 7:07 pm PDT on October 16, 2022, some users may have experienced difficulties while working in Box. During this time, some Box users may have experienced issues with uploads, downloads, notes and public API. The issue occurred as a result of a latent issue with platform workload initialization controllers failing to detect new workloads.

‌

Analysis

This issue was the result of a controller component that provides service-to-service authentication failing to detect that new workloads needed to be processed. This controller monitors an internal API for new workloads that match certain criteria indicating that the workloads are initializing and waiting for internal credentials. In this case, the impacted controller failed to detect new workloads waiting in the API results that should have been processed. We were able to resolve the issue by restarting the controllers and the new workloads were processed. In addition, we also added alerts to prevent similar issues from occurring in the future. The existing alerting in place was monitoring the controllers for errors, but because the controllers were idle, no alerts were immediately raised.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Improve alerting on the workloads if they time out while waiting for the controller to process them.
• Replace the controller with a more robust method of authenticating workloads.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team