We recently addressed issues affecting Enterprise Reporting. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

‌

Between 10:00 AM PST on January 31, 2023, and 4:00 PM PST on February 1, 2023, some users may have experienced difficulties while working in Box. During this time, some enterprise admins may have experienced delays running Box admin reports. The issue occurred due to backend resource exhaustion in our reporting system. We were able to resolve the issue by removing idle reporting resources and adjusting our internal configurations governing recycling of idle reporting resources. In addition, to prevent similar issues from occurring in the future, we will be provisioning additional monitoring to detect an excess buildup of idle reporting resources before they impact the customer experience. 

‌

Analysis

The Enterprise Reporting product dynamically creates compute clusters to run reports requested by customers and then destroys them once they are idle in order to free up capacity for new clusters to be launched. The root cause of this issue was that the process responsible for destroying these reporting compute clusters was unable to keep up with the rate at which clusters were scheduled for destruction, creating a backlog that eventually resulted in temporary exhaustion of the compute resource available to run customer reports. As a result, some enterprise admins may have experienced delays in running reports during the impacted timeframe. Additionally, this issue took longer to detect than it otherwise may have because our monitoring initially failed to distinguish the resulting customer report backlog from the expected higher customer reporting traffic that typically occurs at the end of each month.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Update the reporting cluster destruction process so that it can keep pace with the rate at which clusters are provisioned
• Add direct monitoring for a backlog of reporting clusters that are scheduled for destruction
• Add direct monitoring for exhaustion of available reporting compute resources

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team