[Major] Subset of Customers Experiencing Latency in Web App and API Errors

For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

Incident Report for Box

Postmortem

Notice and disclaimer: Customer understands and agrees that this information is subject to change. To the best of our knowledge, this is the current state and will be updated as information is confirmed.

Last Updated: December 5, 2019

Summary Impact

On November 13th between 9:13 AM PST and 12:03 PM PST, some users may have experienced difficulties while working in Box. During this time, customers observed increased latency in the Box web app.

The issue occurred because of a sudden spike in legal holds related user activity. We were able to resolve the issue by lowering the number of concurrent threads processing the legal holds specific activity. In addition, We have improved scheduling and rate limiting based on overall load for legal holds related activities to prevent similar issues from occurring in the future.

Analysis

Increased legal holds related activity causes spikes in number of jobs that are required to be processed to fulfill the user requests. This sudden spike causes the system to overload and customers may experience latencies while using Box web app.

Defect Remediation

The following remediation actions have been completed or are planned:

Enable scheduling for asynchronous legal holds jobs based on the system load.
Enable rate limiting so that legal holds jobs are spread over a period to avoid sudden spikes in processing.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,
The Box Team

Posted Nov 15, 2019 - 16:36 PST

Resolved

No additional impact has been observed and this incident is now considered Resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.

Posted Nov 13, 2019 - 13:21 PST

Monitoring

Our teams have validated that all affected services are recovered at this time. We are continuing to monitor for any additional impact.

Posted Nov 13, 2019 - 12:39 PST

Update

The delay in processing Governance requests is continuing to recover. Functionality of remaining affected services should be restored as this time. Our teams are continuing to take action to push to a fully recovered state.

Posted Nov 13, 2019 - 12:22 PST

Update

Our internal monitoring is showing initial recovery across affected services. We are continuing to take action to remediate impact.

Posted Nov 13, 2019 - 12:14 PST

Update

Our teams are continuing to take action to remediate impact. We will provide additional updates as they become available.

Posted Nov 13, 2019 - 11:45 PST

Identified

Our team has identified the cause of this behavior and is actively taking steps to remediate impact. We will continue to provide additional updates as they become available.

Posted Nov 13, 2019 - 11:27 PST

Update

Our teams are continuing to investigate this issue and will provide additional updates as they become available. We have amended this status post to include additional components.

Posted Nov 13, 2019 - 11:08 PST

Update

We are continuing to investigate this issue. We have amended this status post to include additional components.

Posted Nov 13, 2019 - 10:46 PST

Investigating

We are currently investigating an issue affecting a specific subset of Box instances. Users may experience slowness or latency when accessing the Box Web Application or utilizing the Box API. We will post additional information as it becomes available.

Posted Nov 13, 2019 - 10:43 PST

This incident affected: Box Platform / API (Content API, Content Preview, Authentication (OAuth 2.0 / JWT), Search, Uploads/Downloads), Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search, Preview, Sharing (Shared Links), Email Notifications, Admin Console & Functionality, Governance (Retention), Workflows and Automations, Comments and Tasks, Accessible Site (a.box.com)), and Box Notes (Web Application).