Notice and disclaimer: Box is providing this preliminary information subject to further review and analysis. To the best of our knowledge, this is the current state and we will update as more information is confirmed.

 On March 20, 2021 between 12:34 PM PST and 8:18 PM PST, some users may have experienced difficulties while working in Box. During this time, some Admin Events API and Admin Console events experienced delay due to large backlog in processing events into the data store that powers the Events API. The issue occurred due to an issue in the queueing software (Kafka) that resulted in a replay of old messages instead of the most current messages. We were able to resolve the issue by moving the API traffic to the secondary cluster, which was not impacted.

Analysis

Upon investigation of the issue, it was determined that there was a large backlog of messages that needed to be processed.  Upon further investigation, it was determined that the Kafka queue offset pointer had erroneously reset to the start of the queue partition.  Researching on the Apache Kafka site and release notes, it was determined that this was an issue that was identified in the version of Kafka Box is running and that there is an updated version that addresses this issue.

Corrective Actions

To prevent this issue from occurring in the future the following corrective actions are planned:

1. Upgrade Kafka software version to a version not susceptible to this issue

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.  

 Sincerely,

The Box Team