For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Critical] Issues with File Requests, Tasks, and Notifications

Incident Report for Box

Postmortem

We recently addressed an issue that impacted the performance of Box features in your enterprise. We would like to take the opportunity to further explain this issue and the steps we have taken to keep them from happening in the future.

Between November 1, 2022 at 4:40 PM PDT and November 2, 2022 at 8:08 PM PDT, users in your enterprise may have experienced difficulties while working in Box. This issue occurred as a result of missing a critical step in a regular database maintenance procedure, which resulted in missing indices on the file table. As a result, the File Requests service experienced some latency. Additionally, Workflow Tasks users experienced an outage that manifested as UI errors. The issue was remediated by adding back indices to the file table. Because this remediation was time consuming, some users in your enterprise may have experienced degraded performance until November 2, 2022 at 8:08 PM PDT.

Corrective Actions

The following corrective actions have been completed or are planned:

  • Adding validations to ensure no steps are missing during maintenance procedures;
  • Creating additional alerting to identify similar issues ahead of time;
  • Improving our rollback plan to facilitate reversion of similar changes to minimize remediation time;
  • Adjusting our procedure to improve testing of changes on a node before deploying the change to production;

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted Nov 04, 2022 - 10:26 PDT

Resolved

After further monitoring, this incident is now considered resolved. File Requests, Tasks, and Notifications have been restored to full functionality. Please contact Box Support at https://support.box.com/ if you continue to experience any issues.
Posted Nov 03, 2022 - 17:47 PDT

Update

We have resolved the issues impacting File Requests, Tasks, and Notifications. We are continuing to monitor to ensure no further impact is observed.
Posted Nov 03, 2022 - 14:25 PDT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Nov 02, 2022 - 20:39 PDT

Update

Our team is continuing remediation efforts to restore full functionality to File Requests, Tasks, and Notifications.
Posted Nov 02, 2022 - 10:34 PDT

Update

We are continuing to work on a fix for this issue.
Posted Nov 02, 2022 - 01:07 PDT

Identified

Starting at approximately 4:33 PM PDT on Nov. 1st, users may have began experiencing intermittent failures and a high level of latency when utilizing File Requests, Tasks, and Notifications. We have identified the root cause and are currently in the process of mitigating the issue.
Posted Nov 01, 2022 - 23:48 PDT
This incident affected: Box Web Application (Comments and Tasks).