KB483585: Memory Contract Manager (MCM) Denial When Other Processes Use Too Much Memory

Description

Memory Contract Manager (MCM) was designed to govern the memory usage of Intelligence server and prevent memory depletion. MCM denial normally happens in the two following scenarios:

• One single memory request exceeded to configured limit
• The accumulated memory usage of Intelligence server reached the configured limit

A third scenario for MCM denial is when other processes use too much memory, so there is not enough memory for Intelligence server

Why is this happening?

In the following example, Available Total was around 21.7 GB at 2019-04-10 10:17. The first MCM denial happened right after that when AvailableTotal dropped to zero, System Total (Bytes) was 38.5 GB, and All Other Process PB was 20.6GB. This means other processes caused the sudden memory consumption, so there was not enough memory for the Intelligence server. Later on, the memory used by other processes was reduced to 9.0GB. Intelligence server still did not have enough available memory and the MCM denial occurred.

2019-04-10 10:17:34.273+10:00 [HOST:xxxxxx][PID:2544][THR:5220][Kernel][Info] Contract Request Succeeded: Source=RW Document, Handle=38079, Size=13606912, ContractedTotal=484003840, AvailableTotal=23334786839
2019-04-10 10:17:38.465+10:00 [HOST:xxxxxx][PID:2544][THR:3804][Kernel][Error] Contract Request Failed. ***** Start of Info Dump *****.
2019-04-10 10:17:38.494+10:00 [HOST:xxxxxx][PID:2544][THR:3804][Kernel][Error] Source= MDb::MPConnection, Handle=19245, Size=8192. ContractedTotal=463925248, AvailableTotal=0.
2019-04-10 10:17:38.521+10:00 [HOST:xxxxxxx][PID:2544][THR:3804][Kernel][Error]
Memory Oscillator = 16678898592.
I-Server PB = 18524151808.
I-Server VB (Virtual Byte)= 21929562112.
System UAS (User Address Space) = 140737488224256.
System Available (Bytes)= 667193344.
System Total (Bytes)= 41338650624.
All Other Process PB = 22147305472.
Largest Free and its bottom = 0, 2147483648.
2019-04-10 10:17:38.530+10:00 [HOST:xxxxxx][PID:2544][THR:3804][Kernel][Error] Single Alloc Limit = 18446744073709551615.
Memory Reserve = 4133865062.
Virtual Byte Limit (percent) = 90.
High and Low Watermark (byte) = 16678898592, 15844953575.

In the following example, when MCM denial occurred, the watermark was set to the current memory size that Intelligence server could use and was never changed. When MCM denial occurred, Intelligence server used 15.5 GB. Since the watermark was frozen at 15.5 GB, even when more memory was available, Intelligence server could only use 15.5 GB.

2019-04-10 10:17:44.164+10:00 [HOST:xxxxxx][PID:2544][THR:4772][Kernel][Error] Source= MDb::MPConnection, Handle=16062, Size=8192.
ContractedTotal=463388672, AvailableTotal=0.
2019-04-10 10:17:43.414+10:00 [HOST:xxxxxx][PID:2544][THR:3604][Kernel][Error] Memory Oscillator = 15837379168.
I-Server PB = 17707270144.
I-Server VB (Virtual Byte)= 21118144512.
System UAS (User Address Space) = 140737488224256.
System Available (Bytes)= 13971275776.
System Total (Bytes)= 41338650624.
All Other Process PB = 9660104704.
Largest Free and its bottom = 0, 2147483648
2019-04-10 10:17:43.414+10:00 [HOST:xxxxxx][PID:2544][THR:3604][Kernel][Error] Single Alloc Limit = 18446744073709551615.
Memory Reserve = 4133865062.
Virtual Byte Limit (percent) = 90.
High and Low Watermark (byte) = 16678898592, 15844953575.

Workaround

Restart Intelligence server to reset the watermark to a normal value. This issue is under investigation. Please refer to KB442303 for more information on troubleshooting the MCM denial issue.