KB10886: How to minimize the risk of loss of report caches and history list messages in the MicroStrategy Intelligence Server

Strategy Intelligence Server stores report caches (including results of reports saved into users' history lists) and history lists into disk files. These files may be stored locally on the same hard drive as the Intelligence Server executable, or on a remote file server.
 
The Intelligence Server depends on these files being written successfully to disk, to preserve all necessary caches and history list messages. If the file system fails for any reason, there is always a likelihood that caches and history list messages will be lost.
 
System administrators may take certain precautions to minimize the damage in the event of a filesystem outage.
 
NOTE: Do not shut down Intelligence Server when a filesystem outage is observed.
 
This is a crucial point because administrators may wish to stop all Intelligence Server activity until the filesystem can be restored, and shutting down the Intelligence Server might seem like the best way to achieve this. However, this is the worst thing to do in terms of preserving caches.
 
If the Strategy Intelligence Server is shut down, it will attempt to write to disk all of the caches and inbox files that exist in memory but have not been updated on disk. Then, when the Intelligence Server process terminates, all of the memory is released -- including memory holding unsaved caches. Thus, shutting down Intelligence Server when the cache/inbox filesystem is down will guarantee the loss of caches.
 
The recommended procedure is to place all projects in full idle status (request idle plus execution idle), and leave the server running until the filesystem is back up. Only when full read/write access has been restored should the Intelligence Server be restarted. Idling the projects increases the likelihood that caches still in memory will remain in memory as long as the Intelligence Server is still up.
 
In this demonstration, a filesystem outage has been simulated by making the cache folder for the Strategy Tutorial project read-only. Any attempt to write caches into the directory will fail. Executing a report and saving into the history list results in the following cache being created:
 

 
Note the cache status, "R, L, U, D." The "D" stands for Dirty, meaning that the version of the cache in memory is more recent than the version on disk. This is an indication that the cache file was not written to disk successfully. Shutting down the Intelligence Server now would mean that the cache would never be written to disk, and would therefore be unavailable for history list messages after the server comes back up.
 
This cache monitor screen shot illustrates what happened to the above dirty cache when the Strategy Intelligence Server is restarted while the filesystem is unavailable. It was not written to disk and it could not be recovered after restart:
 

 
Here, the Intelligence Server was restarted after the filesystem was made available once again, and the cache monitor shows that the cache is now "filed" ("F") on disk:
 

 
Notes:

• Idling the projects will not guarantee that all caches will be saved correctly. There is no guaranteed method of preserving all caches when the filesystem goes down but will minimize the loss, however. Shutting down Intelligence Server immediately guarantees that many caches will be lost. Idling the projects will preserve many of them that would otherwise be lost.
• Caches are lost only when requests coming into the Intelligence Server cause caches to be loaded into cache RAM. When cache RAM is full, caches that have not been recently used will be unloaded from memory. If the file write fails at this point, the cache is irretrievable. Idling the projects prevents these requests; in theory, cache RAM should not change while projects are idle.
• It is critical to catch the filesystem failure as early as possible. Administrators may monitor the DSSErrors.log file for messages similar to the following. These messages may indicate a problem with the file system.

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

terminate 

instancename

 
 
Prevention:
The Strategy Intelligence Server's backup frequency controls how long the server will wait to write cache and inbox files to disk. Setting this to a higher number may improve performance under high job concurrency, but it increases the amount of time caches that exist in memory only (i.e., "dirty" status). The default setting of 0 is the safest in terms of preserving caches, because the files will be written to disk immediately.
 
The backup frequency setting is located under Strategy Intelligence Server Configuration > Server Definition > Advanced, illustrated below:
 

 
Administrators are also encouraged to back up the contents of the Caches and Inbox folders on a regular basis. If, for any reason, the cache files, cache index files and inbox files are out of sync, administrators may restore an earlier cache/inbox state with the following steps:

1. Shut down Strategy Intelligence Server.
2. Delete the existing cache and inbox folders.
3. Copy the backup files into the cache file location.
4. Restart Strategy Intelligence Server.

Note: It is not possible to merge specific caches from a backup into another set of caches.