We recently addressed an issue affecting JWT Logins, Uploads, Downloads, Box Notes and API requests. We would like to take the opportunity to further explain this issue and the steps we have taken to keep it from happening in the future.

Between 09:48 AM PDT and 10:15 AM PDT on July 8, 2022, some users may have experienced difficulties while working in Box. During this time, some users experienced high latency and errors on JWT Login, Upload, Download, API and Box Notes features. The issue occurred because of an error in applying a configuration change to our traffic rate limiting infrastructure. We were able to quickly identify and resolve resolve the issue by rolling back the affecting change. Below is a summary of our analysis and corrective actions to prevent similar impact going forward. 

‌

Analysis

The configuration change that caused the impact is related to our API infrastructure. Part of the infrastructure is a component that controls the allowable request rate for various API endpoints based on a number of factors. These factors are controlled by a configuration file. Our engineering team was updating this configuration file to add a new rate limit entry when they applied an entry missing a required parameter. As a result, the rate limiter system could not load its configuration, which caused the system to fail.

As a result of this incident we have identified a few key areas for improvement. First, we are improving our procedures regarding changes to the rate limiting system, adding more detail specifying required parameters and sample configurations for use in validating requests. We have also conducted additional training with the team to reinforce the procedure. Additionally, we are investing in better automated validation of requests when they are submitted, ensuring they are well formed (i.e., they contain all required parameters). This investment will reduce the likelihood of an incorrect configuration being applied to the system. We believe the combination of automated configuration validation and improved documentation and training significantly improves our rate limiting system and mitigates similar future mistakes.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Improving our procedures regarding changes to the rate limiting system by adding more detail and sample configurations for use in validating requests.
• Conducting additional training to reinforce the procedure for making changes to the rate limiting system.
• Improving automated validation of requests to minimize the likelihood of incorrect configurations in the future.

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter. 

Sincerely,
The Box Team