We recently addressed issues affecting the streaming user events API and Box Drive. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 2:00 AM PDT and 2:21 AM PDT on September 12, 2023, some users may have experienced difficulties while working in Box. During this time, customers may have experienced transient failures in the Events API and delays in synchronizing files via Box Drive. The issue occurred because of a sudden surge in customer traffic that overloaded an isolated part of our infrastructure. The issue was resolved once the infrastructure automatically scaled up to handle the load.

‌

Analysis

The issue occurred due to our services experiencing a sudden surge in traffic volume from Box Drive. This surge caused the services managing our user events API to run in a degraded state. All these services had automatic scaling enabled, and as they received increased load the services automatically scaled up to accommodate the surge. Due to some limitations in detecting and automatically scaling up new servers, some customer requests experienced failures or delays during this time. The system stabilized once the scale up was complete.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Tune request parameters like timeouts and retry policies across all the services involved.
• Re-evaluate our auto-scaling configurations to scale up the servers sooner when they receive increased traffic.

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team