KB483889: Side effects of cache or cube folders not shared correctly in a MicroStrategy Intelligence Server cluster environment

Starting with the release of Strategy ONE (March 2024), dossiers are also known as dashboards.
There are two ways to configure cache or cube folder sharing for a Strategy Intelligence Server cluster environment: localized or centralized.
For more information, see KB441125: How to configure Cache, Cube, Inbox, and Session Recovery file sharing for a MicroStrategy Intelligence Server cluster.
However, when the cache or cube folder are not configured correctly, you may encounter the following issues:

• Cube or cache file cannot be shared in a cluster environment
• The Intelligence Server may hang when starting up
• System may hang when starting the Intelligence Server

To prevent the above situations in the future, see Testing Correctly Configured Cache and Cube Folder.

Cube or cache file cannot be shared in a cluster environment 

There are three situations in which you may find a cube or cache file cannot be shared in a cluster environment.

Situation 1

If all nodes cannot access each other's cube folder, you will see that only cubes published in the local node appear in the Cache Monitor in Strategy Developer.
When starting the Intelligence Server, you may see the following log in the DSSErrors.log of two nodes:

[HOST:NODE2][Cube Server][Error] Warning: this node may not be able to use cubes created by NODE1. NODE1 may be down or \\NODE1\ClusterCube\Server_NODE1_PB92AE4004A57063CF8B0FBBB7D18DA5E does not exist or not readable

[HOST:NODE1][Cube Server][Error] Warning: this node may not be able to use cubes created by NODE2. NODE2 may be down or \\NODE2\ClusterCube\Server_NODE2_PB92AE4004A57063CF8B0FBBB7D18DA5E does not exist or not readable

[HOST:NODE2][COM][Error] Cube file \\NODE1\ClusterCube\Server_NODE1_PB92AE4004A57063CF8B0FBBB7D18DA5E\5F95A9534D06A5725CE8D59754A72DF2.cube does not exist[HOST:NODE2][Cube Server][Error] Error when loading cube instance category of ID 5F95A9534D06A5725CE8D59754A72DF2 from Server NODE1, hr = -2147072546.

Situation 2

Additionally, you may find that a cube published on one node cannot be used on the other nodes. For example, if a cube is published on Node 1, and you run a dossier based on that cube while logged into Node 2, you may meet error:
"Document Execution Failed: One or more dataset reports returned an error."
 

Situation 3

Finally, you may find that a cache created on one node cannot be hit by users logged onto other nodes. New cache will be created on the local node. For big reports or documents, performance will be significantly impacted. New cache can also be a waste of disk space.
You can see similar logs in the DSSErrors.log:

[HOST:NODE2][Document Server][Error] Warning: this node might not be able to use document caches created by NODE1. NODE1 May be down or \\NODE1\ClusterCaches\ServerNODE1_PB92AE4004A57063CF8B0FBBB7D18DA5E\RWDCache does not exist or not readable

The Intelligence Server may hang when starting up

The Intelligence Server may hang when starting up in Linux with a mounted file system. If you cannot access the folder, the parameter of command mount will influence performance. 
To troubleshoot this issue, you can enable Cache Trace in the Diagnostic Configuration Tool:

• Cube Server > Trace
• Report Server > Cache Trace  
• Document Server > Cache Trace

There are two cases in which this situation may occur:

Case 1

On the Isilon OneFS Network File System (NFS), the Intelligence Server may hang when starting up because it is trying to acquire a read lock to the cube index file during the load of the index file. If you add "nolock" and soft mounting options, the issue will resolve itself.
To help troubleshoot, add a log, as shown below, in the cube server trace (/report server cache trace/document server cache trace/):
 

[Cube Server][Trace] CDSSCubeManager::LoadIndex0: Start to acquire read lock for file Z:\BIN\x64\Cube\test\Server_NODE2_PB92AE4004A57063CF8B0FBBB7D18DA5E\CubePool0.idx, please wait for the result... If no result for long time, please refer to KB483342.

[Cube Server][Trace] CDSSCubeManager::LoadIndex0: Succeeded in acquiring read lock for file Z:\BIN\x64\Cube\test\Server_NODE2_PB92AE4004A57063CF8B0FBBB7D18DA5E\CubePool0.idx.

[Cube Server][Trace] CDSSCubeManager::LoadIndex0: Failed to acquire read lock for file Z:\BIN\x64\Cube\test\Server_NODE2_PB92AE4004A57063CF8B0FBBB7D18DA5E\CubePool0.idx

Case 2

The Intelligence Server will hang if the NFS folder cannot be accessed. When attempting to access the cube/cache file path, the stat() function is called. If you are using a soft mount, the Intelligence Server requests a file from the NFS and the NFS client daemons will attempt to contact the server repeatedly until the server times out, the NFS retry threshold is met, or if a connection is established. If you are using a hard mount (by default), a NFS request is issued repeatedly until the request is satisfied. This will cause a performance issue and the Intelligence Server will hang.
To troubleshoot, add a log, as shown below, in the cube server trace (/report server cache trace/document server cache trace/kernel user trace for inbox folder):

[Cube Server][Trace] Start to check cube file path Z:\BIN\x64\Cube\test, if no result for a long time, please check whether the path is accessible.[Cube Server][Trace] Path Z:\BIN\x64\Cube\test is accessible.

System may hang when starting the Intelligence Server

There is also a hang case in the Windows system.
 

Case 1 The whole system hangs when the folder is not configured correctly. After properly reconfiguring, the issue is resolved. However, the root cause is unknown because the issue cannot be reproduced in the local environment.

Testing Correctly Configured Cache and Cube FolderYou can test to check whether the cube/cache folder is configured correctly.

In the system, check for the following items:

• Read/write access to the NFS folder
• Enough disk space on the NFS server
• Network connectivity between all servers and NFS severs

In Strategy, you can check in the following: 
 

• Cube folder: publish a cube on one node and check the status of the cube on Cache Monitor in Strategy Developer. It should be "ALF" on all nodes and from the File Name, you can see the server that published the cube.

• There should be no errors in the DSSErrors.log and cube trace.
  
   
  
• Cache folder: run a normal report on one node and cache will be created. Run the same report for the other nodes. In the Report details, Report Cache Used should be Yes.
  

• Strategy Web: log into one node and run a document. Cache will be created. Log into the other nodes and run the same document. Logs, as show below, is printed:
  
  

  [Document Server][Cache Trace] RWDCacheManager::LookupCache: cache with ID 7C95DAC948D29DCA977F9894C7A71528 found for DI=0x718E2860

If you find issues with the cluster environment, you can check the DSSErrors.log and the cube/cache trace to troubleshoot.