For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

[Critical] Box Logins with SSO and Box Notes
Incident Report for Box
Postmortem

We recently addressed issues affecting SSO Logins and Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between between 07:42 AM and 08:47 AM PDT on September 14, 2022 , some users may have experienced difficulties while working in Box. During this time, customers may have experienced issues with SSO Logins and Box Notes. The issue occurred due to a failure in the network connecting one of our points of presence (POPs) to Box’s Application Data Center (ADC). We were able to resolve the issue by implementing backbone routing changes to bypass the affected network path. In addition, we are working towards migrating to the public cloud to leverage their diverse and global network to prevent similar issues from occurring in the future. 

Analysis

This issue was the result of a network failure affecting one of our POPs. These POPs possess two redundant transport circuits that are on diverse physical paths and are provided by two separate providers providing redundant connectivity. During this incident, the impacted POP experienced simultaneous failures of both of its redundant circuits due to separate service provider incidents.

Simultaneous failure of both transport circuits is a rare occurrence; in this case, the root cause was identified by the service providers as a “fiber cut“. This dual failure isolated the impacted POP from the rest of the network. In response, we implemented a temporary backbone routing change to resolve the issue at that time, which temporarily diverted the impacted traffic to a sub-optimal path. Later the connectivity was restored to the original preferred path when one of the circuit recovered. After that, the second circuit also recovered, thereby restoring the redundant connectivity to the POP.

Corrective Actions

The following corrective actions have been completed or are planned:

  • We have migrated 100% of our edge traffic to GCP. By doing this we have further reduced the dependence on on-premise POPs.
  • This now enables us to leverage Google’s global POP presence, further improving our global reach close to the customer location.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,

The Box Team

Posted Sep 21, 2022 - 09:15 PDT

Resolved
From approximately 7:38 AM to 9:08AM US Pacific time, we had an issue impacting SSO logins and Box Notes. Our systems automatically detected and corrected the underlying issue. There is no current impact and no further updates will be provided here. If you are currently seeing any issues, please let us know at https://support.box.com.
Posted Sep 14, 2022 - 09:15 PDT
Investigating
We are currently investigating an issue where users were unable to login via Single Sign On and/or access Box Notes. Our team is currently investigating the root cause. Currently the issue looks to be mitigated and our team is investigating the cause.
Posted Sep 14, 2022 - 08:13 PDT
This incident affected: Box Web Application (Login/SSO) and Box Notes (Web Application).