For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Major] Issue Accessing Admin Console

Incident Report for Box

Postmortem

We recently addressed issues affecting the Admin Console. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 10:04 AM PDT and 11:19 AM PDT on Wednesday, March 17, 2021, some users may have experienced difficulties while working in Box. During this time, some customers may have experienced intermittent errors while accessing their Admin console. The issue occurred due to traffic contention in a database that supports our Policy Engine service. We were able to resolve the issue by migrating our Policy Engine service to a different datacenter and restarting the databases that were under load. In addition, we are improving our Policy Engine service architecture to prevent similar issues from occurring in the future.

Analysis

Prior to the Admin Console degradation, a server in the database used by the Policy Engine service experienced errors due to a database bug, which has been fixed in a newer database version. This server was decommissioned, however it led to high load on a new server in the database. The new server failed to process requests coming from the Policy Engine service, which led to degradation in Admin Console. The issue was solved by migrating the Policy Engine service to a different datacenter. We have identified improvements to prevent a recurrence of similar issues in the future.

Corrective Actions

The following corrective actions have been completed or are planned:

Improve failover mechanisms for the Policy Engine service - planned
Improve alerts and monitoring for the Policy Engine service - planned
Improve alerts and monitoring for the Policy Engine database - planned
Upgrade the database to a newer version - planned

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team

Posted Mar 31, 2021 - 15:42 PDT

Resolved

We have completed our monitoring period and are considering the issue fully resolved. If you encounter any new issues, please contact our Support team at https://support.box.com.

Posted Mar 17, 2021 - 12:00 PDT

Monitoring

Our team has taken steps to remediate this behavior. We are currently monitoring for any additional impact. If you are still encountering any issues accessing the Admin Console, please contact Box Support at https://support.box.com.

Posted Mar 17, 2021 - 11:33 PDT

Investigating

We are currently investigating an issue where some customers may be unable to access the Admin Console. We will provide additional information as it becomes available.

Posted Mar 17, 2021 - 11:16 PDT

This incident affected: Box Web Application (Admin Console & Functionality).