[Outage] Issue with Box Services

For more information about our Incident Response and Communications please read this support article.

We also maintain a list of Known Product Issues separate from this site here.

Incident Report for Box

Postmortem

NOTICE AND DISCLAIMER: CUSTOMER UNDERSTANDS AND AGREES THAT THIS INFORMATION IS SUBJECT TO CHANGE. TO THE BEST OF OUR KNOWLEDGE, THIS IS THE CURRENT STATE AND WILL BE UPDATED AS INFORMATION IS CONFIRMED.

LAST UPDATED: August 17, 2020

Summary Impact

Between 4:02 pm and 4:15 pm PDT on July 8, 2020, some users may have experienced difficulties while working in Box. During this time, users may have been unable to connect to Box services. The issue occurred as a result of routing updates associated with a scheduled network maintenance to increase data center capacity. We were able to detect the issue and resolve it by rolling back the configuration changes. In addition, we are working to improve our network resiliency by increasing our testing and validation processes and increasing our service isolation to prevent similar issues from occurring in the future.

Analysis

Upon investigation, analysis identified that the tooling used to generate BGP (Border Gateway Protocol) routing configurations for new data center expansion space unintentionally generated a configuration that overlapped with existing data center space. The subsequent deployment of the new configuration overwrote the BGP routes of the existing space, isolating traffic for critical services in the existing space. While the tooling that generated the configuration has been in use for several years, it lacked a validation test to look for conflicts with existing configurations. It was further identified that application configurations were not properly isolating services to limit the area of impact.

Defect Remediation

The following remediation actions have been completed or are planned:

Update tooling to validate and test candidate routing changes do not conflict with existing configurations
Increase application service isolation to limit area of impact

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,
The Box Team

Posted Jul 09, 2020 - 11:05 PDT

Resolved

No additional was observed and this incident is now considered resolved. Please contact Box Support at https://support.box.com if you experience any additional issues.

Posted Jul 08, 2020 - 18:18 PDT

Monitoring

Our teams validated that all affected services are recovered at this time. We are continuing to monitor for any additional impact.

Posted Jul 08, 2020 - 18:07 PDT

Update

Our team continues to take steps to fully remediate impact. Our monitoring indicates continued recovery and we will provide additional updates as they become available.

Posted Jul 08, 2020 - 17:33 PDT

Identified

Our team identified the cause of this issue and our monitoring indicates continued recovery. We will continue to provide additional updates as they become available.

Posted Jul 08, 2020 - 17:16 PDT

Update

We're continuing to see recovery and will continue to provide additional updates as they become available.

Posted Jul 08, 2020 - 17:00 PDT

Update

Our monitoring indicates initial recovery is underway, but customers may be experiencing slowness or 429 errors. We will continue to provide additional updates as they become available.

Posted Jul 08, 2020 - 16:43 PDT

Investigating

We are currently investigating an issue with Box services. We will provide more information as it becomes available.

Posted Jul 08, 2020 - 16:13 PDT

This incident affected: Box Platform / API (Content API, Content Preview, Authentication (OAuth 2.0 / JWT), Search, Uploads/Downloads, Webhooks), Box Web Application (Login/SSO, Uploads/Downloads, Collaboration, Search, Preview, Sharing (Shared Links), Email Notifications, Admin Console & Functionality, Governance (Retention), Workflows and Automations, Comments and Tasks, Accessible Site (a.box.com)), Desktop Applications (Login/SSO, Box Sync, Box Drive, Box Edit / Tools), Mobile Applications (Login/SSO, Preview, Search, Uploads/Downloads, Mobile-Optimized Website (m.box.com)), Partners / Integrations (Microsoft Online, Apple iWork, Box for Salesforce, Google Apps, NetSuite), Box Notes (Web Application), and Box Relay, FTP, Box Website.