We recently addressed issues affecting Box Notes. We would like to take the opportunity to further explain these issues and the steps we have taken to keep them from happening in the future.

Between 2:15 PM and 2:30 PM PDT on June 25, 2024, some users may have experienced difficulties while working in Box Notes. During this time, users would have seen an error banner indicating that the system was reconnecting and, upon reconnecting, some of the users’ recent edits may have been reverted. The issue occurred as a result of a code change enabling values to be written to cache on the Notes servers unintentionally. We were able to resolve the issue by disabling the code that the change was applied to and updating it to no longer write to cache. In addition, we are working to improve the resiliency of our system in the related areas as well as gain higher visibility through logging to detect and prevent similar issues from occurring in the future. 

‌

Analysis

The issue occurred because a process that reads values from the Notes database and caches them in memory on the Notes server was unintentionally enabled on servers not taking live traffic (passive). Meanwhile, servers taking live traffic (active) were allowed to make updates to those values in the Notes database. However, the values in cache on the passive servers were not updated or cleared. When traffic was redirected to the passive servers, those stale values were returned from cache causing a mismatch in what was returned by the server and what was expected by clients. Clients attempted to send updates to the servers to correct the mismatch, but a bug in the client caused the updates to be invalid and rejected by the servers. This resulted in an error banner being shown and the client being unable to recover from the mismatch.

‌

Corrective Actions

The following corrective actions have been completed or are planned:

• Update code to prevent process from running on passive servers
• Update code to clear values from cache when servers transition from passive to active
• Fix client bug causing invalid updates to be sent to server
• Add logging to detect when values are returned from cache unintentionally
• Improve developer documentation to prevent similar changes from being made in the future

‌

We are continuously working to improve Box and want to make sure we are delivering the best product and user experience we can. We hope we have provided some clarity here and we would be happy to answer any questions you may still have regarding this matter.

Sincerely,

The Box Team