KB13315: Overview of the Memory Contract Manager (MCM) behavior in MicroStrategy Intelligence Server 2021

Memory Contract Manager (MCM):
The memory contract manager is a consolidated memory management component within Strategy Intelligence Server 2021 that grants or denies memory from different tasks within Strategy Intelligence Server. Users can refer to the following Strategy Knowledge Base technical note for a detailed explanation of the design of the MCM governor:
 

• KB7295: What is the Memory Contract Manager in MicroStrategy Intelligence Server 10.x and later? PART1
• ​KB7295: What is the Memory Contract Manager in MicroStrategy Intelligence Server 10.x and later? PART2

 
 
SmartHeap:
SmartHeap is a third party component used by Strategy Intelligence Server 2021 to provide improved memory management performance. This component is used on both the Microsoft Windows and Linux Platform.
 
From the manufacturer's website, "SmartHeap is a fast reliable, ANSI-compliant malloc/operator new library. SmartHeap supports multiple memory pools, includes a fixed-size allocator, and is thread-safe. SmartHeap also includes comprehensive memory debugging APIs to detect leakage, overwrites, double-frees, wild pointers, out of memory, references to previously freed memory, and other memory errors."
 
Specifics of the Smartheap behaviour are discussed in the following Strategy Knowledge Base article:
 

• KB16977: What is SmartHeap and how does it work with the MicroStrategy Intelligence Server?
  

MCM governing for the Strategy Intelligence Server:
In both the Strategy Intelligence Server on Microsoft Windows, and the Strategy Intelligence Server Universal on Unix/Linux platforms, users will observe that the server process virtual bytes or process size does not reduce significantly, even during periods of low server activity. To a large extent, this observed behaviour of the server process over time is due to the smartheap behaviour of caching previously allocated memory for future use.
 
The core Intelligence Server code allocates and releases memory for the specific operations it performs. However, smartheap, which manages the memory allocation and deallocation for the server code simply marks this memory for reuse, and will utilize this cached memory to satisfy future memory allocation requests rather than allocating from the operating system again.
 
Users can expect to see a difference in the exact amount of memory cached and Intelligence Server memory profile depending on the specific operating system used. This is primarily because the default tuning of the smartheap parameters differs between various operating systems.
 
Also the default behaviour of the MCM components will differ between platforms. One difference lays in available counters to monitor memory usage. 
 
Unix/Linux systems generally provide two useful counters - Process 'Size' and 'Resident Set Size', that can be used to monitor process memory usage. The 'Size' counter can be considered roughly equivalent to the Virtual Bytes counter in Microsoft Windows. However there is no easily accessible operating system counter of the committed memory used by a process equivalent of the Private Bytes Counter on Microsoft Windows. Further, the MCM component in the Strategy Intelligence Server universal on Unix/Linux does not perform the calculation to identify the largest free blocks or use this in determining whether memory requests may be rejected.
 
To calculate this committed memory usage in Unix, Strategy Intelligence Server Universal on Unix/Linux uses a smartheap provided counter internally called 'PoolSize', which increments and decrements with memory allocation and de-allocation. This is roughly equivalent to the committed memory, and the MCM component uses this counter to decide whether a request can be granted or denied. For all MCM calculations, this counter is treated as the equivalent of the Microsoft Windows 'Private Bytes' counter. This counter is logged in the DSS Errors log as the Smart Heap Pool Size or Memory Oscillator component, when a MCM request is rejected as shown below:
 

2019-12-09 08:42:39.000 [HOST:env-1744981laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] Memory Oscillator(bytes)

 
In both cases, i.e. with and without the Smart Heap component, the MCM component within the Intelligence Server Universal keeps track of the committed memory and treats it in the same manner as the pivate bytes counter in calculating the High and Low watermarks as when running on a Microsoft Windows platform. The process size counter is used as the equivalent virtual bytes counter on Unix/Linux platforms.
 
Calculating appropriate values for MCM governing:
The first important point to note is that in the context of the Intelligence Server universal on Unix/Linux, the Total Virtual memory is an important parameter and it is equal to (RAM + page file space). It is usually logged in the DSSErrors.log when a request is rejected in the following manner for both Windows and Unix/Linux:
 

System Total (Bytes)= 30181662216

 
The amount of memory available to the Strategy Intelligence Server process to use is the sum of (Total Virtual memory - memory used by all processes on the machine including Strategy Intelligence Server). In the MCM request reject logs, this counter appears as 'System Available (Bytes)' in the logs for both Windows and Unix/Linux, as shown below:
 

System Available (Bytes)= 8213233224

 
Users should note that some of the other processes could be Strategy processes as well such as M8PrcSvr and M8MulPrc. However since these are separate processes, they are not governed by MCM within the Intelligence Server.
 
The memory in use by all the other processes on the machine excluding Strategy Intelligence Server is also logged along with any MCM request reject messages:
 

For Unix/Linux: All Other Process Data(Bytes) = 36264755 
For Windows: All Other Process PB = 10022375424

Additionally, it is possible in Unix to assign a maximum 'Heap Limit' to a process. This is the maximum memory address space that the process can use as a heap, independent of the System virtual memory availability. This limit can be checked by running the command 'ulimit -a' and noting the value returned for 'data', as shown below:
 

For Unix/Linux: Heap Limit(Bytes)= 18446744073709551615 
For Windows: System UAS (User Address Space): 140737488224256

 
 
Note: The heap limit applies to the individual processes and in many ways can be considered similar to the 'User Address Space' concept in Microsoft Windows. However while the 'User Address Space' applies to all processes in the same manner, the heap limit can be modified for different processes and users. By default in most operating system platforms, the heap limit for the processes is set to 'unlimited'.
 
As mentioned earlier, MCM also checks the 'Size' parameter for the Strategy Intelligence Server process which represents the amount of virtual memory currently allocated by the process, as shown below:
 

For Unix/Linux: I-Server Private Memory Size = 8276877600 
For Windows: I-Server PB = 7970549760

 
To calculate the appropriate values for MCM to govern the Intelligence Server usage of system memory users must consider the two cases of memory depletion that MCM attempts to guard against:

1. Prevent the Intelligence Server committing memory in excess of the system commit limit
2. Prevent the Intelligence Server process from exceeding the User Address Space.

 
For Unix/Linux systems, there is no 'User Address Space' concept as in Microsoft Windows. As discussed earlier, the heap limit in Unix/Linux systems is typically set to an unlimited value. As a result, in most Unix/Linux systems, the default MCM governing setting of a virtual byte limit of 90% will evaluate as 90% of 2^64 bytes which is also a very large number.
 
In most practical cases, the total amount of memory (physical + swap) will be much below this value, and the MCM parameters must be set to avoid the condition (a) - i.e. prevent the Intelligence Server from allocating all the available memory on the system. Because the default Intelligence Server governing configuration does not check for this condition, users may notice that MCM will fail to prevent the the server from running out memory unless specific governing is implemented.
 
The specific MCM governing settings that must be changed to ensure that the Intelligence Server does not run out of memory are the 'Reserved memory for processes external to the Intelligence Server' settings as shown below. Users can either specify an absolute value, or a percentage of the total memory to be reserved for other processes.
 

 
When calculating the value to be used, users should know how much physical memory and swap memory is available on the system (or the total of both). Users should also know exactly how much memory they would like to limit the memory usage of the Intelligence Server process to use.
 
Consider an example of how the MCM governor works for a simple scenario. The log files for a rejected request are reproduced below:
Windows:

2019-10-22 22:52:12.471-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Error][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] Contract Request Failed. ***** Start of Info Dump *****
2019-10-22 22:52:12.471-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Error][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] Source= XMLDOM, Handle=1, Size=69632.
ContractedTotal=200724480, AvailableTotal=0.
2019-10-22 22:52:12.472-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Error][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] Memory I-Server PB = 7970549760.
I-Server VB (Virtual Byte)= 15743385600.
System UAS (User Address Space) = 140737488224256.
System Available (Bytes)= 9842229248.
System Total (Bytes)= 27835154432.
All Other Process PB = 10022375424.
Largest Free and its bottom = 0, 2147483648.
High and Low Watermark (byte) = 7467908480, 7094512980.
2019-10-22 22:52:12.473-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Error][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] Single Alloc Limit = 18446744073709551615.
Memory Reserve = 19484608102.
Virtual Byte Limit (percent) = 90.
Doubling Limit = 1 MB
2019-10-22 22:52:12.474-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Info][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] SmartHeap Cache Releasable = true
2019-10-22 22:52:12.475-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:11208][Kernel][Error][UID:54F3D26011D2896560009A8E67019608][SID:0F69D1F3DCBC05A8BA5FAEDAAE31F20E][OID:8885C1254CEF4EBC1AC080B4C4161EBC] Contract Request Failed. ***** End of Info Dump *****.
       

 
Unix/Linux

2019-12-09 08:42:38.999 [HOST:env-174498laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] Contract Request Failed. ***** Start of Info Dump *****.
2019-12-09 08:42:38.999 [HOST:env-174498laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] Source= cache - hashtable, Handle=1, Size=4096.
ContractedTotal=641773568, AvailableTotal=0.
2019-12-09 08:42:39.000 [HOST:env-174498laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] Memory Oscillator(Bytes) = 45294759088.
I-Server Private Memory Size(Bytes) = 49550954496.
Heap Limit(Bytes) = 18446744073709551615.
System Available(Bytes) = 10496536576.
System Total(Bytes) = 64387756032.
All Other Process Data(Bytes) = 4340264960.
High and Low Watermark(Bytes) = 54729592628, 51993112970.
2019-12-09 08:42:39.000 [HOST:env-174498laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] MCM Settings:
Single Alloc Limit(Bytes) = 4194304000.
Memory Reserve(Bytes) = 9658163404.
Virtual Byte Limit(percent) = 95.
Doubling Limit(MB) = 1
SmartHeap Cache Releasable = true
2019-12-09 08:42:39.000 [HOST:env-174498laio2use1][SERVER:CastorServer][PID:4270][THR:140172129986304][Kernel][Error][UID:0][SID:0][OID:0] Contract Request Failed. ***** End of Info Dump *****.

Note that the Virtual Byte limit set in the governing settings for this Strategy Intelligence Server is the default - 90%, and the heap value is essentially unlimited. The total virtual memory shows that the (physical memory + swap space) for this machine is 64.39GB, and the available memory on the machine is 10.5GB. Notice that the Strategy Intelligence Server committed memory level is at 45.29GB, and the total virtual memory used by Strategy Intelligence Server is only 49.55GB.
 
Users can also observe that the Strategy Intelligence Server has a reserved memory for other processes set. This can be see in the log files as shown below:
 

For Unix/Linux:Memory Reserve(Bytes) = 9658163404
For Windows:Memory Reserve = 19484608102

 
When Strategy Intelligence Server rises to a certain value, PBLimit will be hit. The condition is IServerPB*1.05> {SystemTotal - Max( Memory Reserve, All other Process). This flag will influence the calculation of counter values used by MCM to allow or reject server requests. The following message can be seen in the DSSErrors.log:

2019-10-22 22:52:12.367-04:00 [HOST:TEC-W-003923][SERVER:CastorServer][PID:16368][THR:5724][Kernel][Info][UID:0][SID:0][OID:0] MCM has reached PBLimit

Before PBLimit is reached, High Watermark = SystemTotal - Max( Memory Reserve, All other Process). Max( Memory Reserve, All other Process) represents the maximum value of counters: Memory Reserve and All other Process. After PBLimit is reached and when Strategy Intelligence Server goes into request reject state, High Watermark = the Memory Oscillator counter value for the first failed request . High Watermark will keep this value unchanged until the Intelligence Server shuts down. The Low Watermark is always set at 95% of the High Watermark, and the MCM component will continue to reject requests until the Oscillator counter falls below the Low Watermark, just as the Intelligence Server in Microsoft Windows.
How to judge whether MCM should reject a request or not：
If PBLimit is not reached,

• If the previous request is succeeded, the request will be failed if the following condition is qualified: IServerPB*1.05 + Contract Total +Size > High Watermark
• If the previous request is failed, the request will be failed if the following condition is qualified: IServerPB*1.05 + Contract Total +Size > Low Watermark. For the sample log on Unix/Linux, the Low Watermark is 51.99GB and the result of (IServerPB*1.05 + Contract Total +Size) is 52.67GB. Thus, MCM rejects the request. Note that the default 90% virtual byte limit MCM governing setting will not be useful in this case.

If PBLimit is reached,

• For Windows, the condition is still useful as when PBLimit is not reached. 
• For Unix/Linux, the counter IServerPB in the conditions need to be changed to Memory Oscillator.

Note: For Windows, IServerPB represents the value of "I-Server PB," For Unix/Linux, IServerPB represents the value of "I-Server Private Memory Size(Bytes)." The equation is not always satisfied for all circumstances, it's just an approximate.