KB486851: Intelligent Server reports java.lang.OutOfMemoryError messages after configuring data sources to use JDBC connections

Symptom

The Intelligent Server service crashes and logs the following records in the DSSErrors.log

[HOST:<host_name>][SERVER:CastorServer][PID:30082][THR:140604326401792][Database Classes][Error][UID:xxx][SID:xxx][OID:xxx][ConnectionKeeper.cpp:77] JdbcStatementWrapper::ExecQuery failed. Disconnected.  Error type: Unknown exception. Jdbc operation attempted: MIX_JdbcStatement_execute. Error Message:unable to create native thread: possibly out of memory or process/resource limits reached.

Which would suggest the application of the technical notes:

• https://community.Strategy.com/s/article/KB44815-Understanding-the-quot-ulimit-quot-and-how-it-affects
• https://community.Strategy.com/s/article/Troubleshooting-errors-in-Java-code-type-OutOfMemoryError
• https://www2.Strategy.com/producthelp/Current/InstallConfig/en-us/Content/Recommended_system_settings_for_UNIX_and_Linux.htm

These articles recommend increasing the number of descriptors (nofiles) to 65535, but that would be insufficient.
Additionally, there are error messages such as the following:

2024-06-12 17:09:57.201-03:00 [HOST:<host_name>][SERVER:CastorServer][PID:30082][THR:140604345313024][Query Engine][Error][0x80042300][UID:xxx][SID:xxx][OID:xxx][CDSSQueryEngine.cpp:6408] QueryEngine encountered error: JdbcStatementWrapper::ExecQuery failed. Disconnected. 
Error type: Unknown exception. Jdbc operation attempted: MIX_JdbcStatement_execute. Error Message:unable to create native thread: possibly out of memory or process/resource limits reached.. 
2024-06-12 17:09:57.254-03:00 [HOST:<host_name>][SERVER:CastorServer][PID:25303][THR:139673487382784][MultiProcess][Error][UID:0][SID:0][OID:0][Multiprocess.cpp:365] Quiting child process because of error: ::pthread_create failed. System Error (EAGAIN): --- Resource temporarily unavailable

That would suggest that the system is reaching the process limits assigned to the service account defined to run the Intelligent Server service and reported in our technical note https://community.Strategy.com/s/article/KB46831-Scheduled-subscriptions-sometimes-fail-when-the?language=ja
However, the error message would remain even after the application of the suggested solution.
 

Cause

The threads and child processes for the Intelligent Server process are consuming a high amount of resources for the service account that runs the Intelligent Server service, which includes OpenRefine Server, Search Engine, Process Servers, and JDBC processes. Each JDBC process can create between 20 and 50 threads, while the ODBC process only creates between 4 and 10 threads.
Therefore, when switching from ODBC to JDBC, it is easy to reach the thread limit. As a result, the thread quota is low and may not be sufficient when the Intelligent Server service is under a very high workload.
 

Solution

1. Increase the number of process quotas allowed for the user running the Intelligence Server process five times to 20480 or larger. To accomplish this change the client must consult the System Administrator to increase the maximum number of threads, This change might need to be made in different ways depending on the configuration of the machine where the Intelligence Server is running, one possible way is to modify the /etc/security/limits.conf file and add the following line (in this example, the Intelligence Server is run by a user named mstruser and the desired number of threads is 20480):
   

   
    mstruser soft nproc 20480
   

   
   
2.  Limiting the JDBC process count by lowering the job priority quota of the database instance that uses the JDBC connection:
   
1. Running Strategy Developer
2. Accessing the Project Source
3. Accessing Administration Project
4. Expanding Configuration Managers
5. Expanding Database Instances
6. Right-click on the database instance and select "Prioritization"
7. In the tab "Job Prioritization", in the "Number of connections by priority" section, select a lower number for the High, Medium, and Low connections.