KB17216: Steps to follow to collect troubleshooting information in the event of a MicroStrategy Intelligence Server Universal hang when running on a UNIX/Linux platform

The steps below outline how to collect troubleshooting information in the event of a Strategy Intelligence Server Universal 9.x and 10.x hang when running on a UNIX/Linux platform.
 
This Information must be collected from ALL NODES if running in a clustered Intelligence Server configuration to capture the state completely:

1. Find the Strategy Intelligence Server 9.x or 10.x process id (henceforth {MPID})
   ps -efl | grep MSTRSvr
   
2. (LINUX ONLY) Find the files locked by Strategy Intelligence Server 9.x or 10.x process. This has to be done on each node where {HOST} represents the name of the machine (obtained by running uname -n).
   cat /proc/locks | grep {MPID} > ${HOST}_proclocks
   
3. Find the index number (inode) information associated with the inbox files (e.g. ls -i /mstr/Intelligence Server/Inbox/orion13):
   ls -i > ${HOST}_inode
   
4. Find the file descriptors associated with the Strategy Intelligence Server 9.x or 10.x process:
   Linux and AIX
   ls -al /proc/{MPID}/fd > ${HOST}_fds
   Solaris
   pfiles {MPID} > ${HOST}_fds
   
5. Obtain the mount information for the system
   mount > ${HOST}_mount
   
6. Obtain the NFS statistics for the system
   Linux
   nfsstat -a > ${HOST}_nfsstat
   AIX, Solaris, and HP-UX
   nfsstat > ${HOST}_nfsstat
   
7. Obtain a process stack view of the Strategy Intelligence Server 9.x or 10.x; do this twice five minutes apart
    
   
1. Obtain a process stack for {MPID}
2. Obtain a snap shot of the current processes with signals
3. Wait for approximately 5 minutes
4. Obtain another process stack for {MPID}
5. Obtain a second process snapshot with signals
8.  
   Linux, Solaris, HP-UX: (on SUSE Linux replace 'pstack' with 'gstack'). Users should note that the pstack or gstack utilities may not be installed by default on the operating system. Users would need to contact their system administrators to obtain access to these utilities:
   NOTE: It is important that the pstack output be provided to Strategy Technical Support. The 'ps axms' output is optional and in some cases 'ps axms' may not work on all installed versions of Linux, Solaris, or HP-UX. If the OS does not support these flags try using 'ps -ef' or 'ps -af'.
   

   
   date > ${HOST}_pstack1; pstack {MPID} >> ${HOST}_pstack1
   date > ${HOST}_psaxms1; ps axms >> ${HOST}_psaxms1

   
    
   [WAIT 5 MINUTES]
    
   

   
   date > ${HOST}_pstack2; pstack {MPID} >> ${HOST}_pstack2
   date > ${HOST}_psaxms2; ps axms >> ${HOST}_psaxms2

   
     
   Note: The

    pstack/gstack

   command should be run by root (preferred) or by the user that owns the Intelligence Server process. Running by a different user may not produce any output for the 

   pstack

   command.
    
   AIX:
   

   
   date > ${HOST}_pstack1; procstack {MPID} >> ${HOST}_pstack1
   date > ${HOST}_psaxs1; ps axs >> ${HOST}_psaxs1

   
    
   [WAIT 5 MINUTES]
    
   

   
   date > ${HOST}_pstack2; procstack {MPID} >> ${HOST}_pstack2
   date > ${HOST}_psaxs2; ps axs >> ${HOST}_psaxs2

   
   
9. Obtain the system messages files from the machine (Linux: /var/log/messages; AIX: errpt -a)
10. Obtain the DSSErrors logs (and backups) from the machine

Additional steps to follow for the hanging Strategy Intelligence Server 9.x instance
Users may also be able to terminate and create a core file of the Strategy Intelligence Server process instead of shutting down the process normally. This would provide another data point for Strategy Technical Support to get a wider view of the root cause of the issue. Users can do so by running the following commands that sends a SEGV command to the server process:
 
Solaris, Linux, HPUX: kill -s SEGV {MPID}
AIX: kill -SEGV {MPID}
 
Note that creating a core file requires a large amount of disk space to allow for the core file to be created, and for processes running with a large amount of memory, this process may take a while. Users should consult the following Strategy Technical Note if a core file is not generated correctly:  
KB13610: System settings to check to ensure that core files for the MicroStrategy Intelligence Server Universal process are correctly created by the Unix/Linux operating system platform
If a core file could not be generated users may use the 'kill -9 {MPID}' if the mstrctl command to force the process termination.
WARNING:
Disclaimer: The utility application is provided "as is" and without warranty of any kind. Strategy Expressly disclaims all warranties, express, implied or statutory, including, without limitation, the implied warranties of merchantability, fitness for a particular purpose, satisfactory quality and noninfringement.
Limitation of Liability: Strategy shall have no liability to licensee for any damages of any kind, including, but not limited to, liability for direct, indirect, special, incidental or consequential, damages (which shall include, but not be limited to, loss of data or information, loss of revenue or anticipated profits or lost business).
 
17216