After upgraded to 2021 the connection to Intelligence Server is refused in Web Administrator in a two Intelligence Server nodes cluster environment. The Intelligence Server becomes hang status. Issue is only recovered after restart the Intelligence Server.
Below error is noticed in DSSErrors.log right before the issue happen.
[OID:xxxxxxxxxxxxxxxxxxxxx][Pool.h:199] Pool::Add: Maximum working cache limit 6144000KB is hit, please check Server Configuration > Governing Settings > Default > Memory Settings > Maximum RAM for working set cache setting. You need increase this setting otherwise performance may become worse
[OID:0][MSICmdUtilXMLReqEx.cpp:1069] Unable to reply to the client(xxx). Typically, it is due to connection getting reset at the client's end. Possible reasons could be: 1) the client is not able to recover from some error or exception; 2) the client fails to hear the server's response in time and choose to close the connection. XML Command headline: <grtex sid="xxxxxxxxx"
Hit working cache limit leads to hanging behavior on the Intelligence Server as it needs to swap out older messages based on LRU(least recently used) rules. While the swapping can be expensive as in involves dumping them from memory to disk especially if there is high concurrency at the time.
