A number of scenarios exist in which "system call" errors may appear in the DSSErrors.log file on a Strategy Intelligence Server 2021 GA and later machine, accompanied by symptoms including but not limited to the following:
- Report execution failures
- Unresponsiveness (hanging) or unexpected shutdowns of Strategy Intelligence Server
Examples of typical "system call" errors follow. This is not an exhaustive list.
Query Engine Encountered Error: unknown failed. Unable to connect. Error type: System call error. Error communicating with
connection process. Process hung.[][][]. Error in processing method of component.QueryEngineServer, Project 21,
Error code=-2147212544
QueryEngine encountered error: Unknown failed. Unable to connect. Error type: System call error. Error communicating with
connection process. No memory to be contracted. Contract request rejected.
Connection::Connect failed. Unable to connect. Error type: System call error. Could not spawn process.Cannot open named pipe
for read/write. System Error (ENXIO): No such device or address.
Connection::Connect failed. Unable to connect. Error type: System call error. Could not spawn process.Failed in fork System
Error (ENOMEM): Not enough space. Connection String: DSN=EM_STATS;UID=em_stats;.
QueryEngine encountered error: Unknown failed. Unable to connect. Error type: System call error. Error communicating with
connection process. Synchronous call timed out after --- milliseconds.Error in Process method of Component: QueryEngineServer,
Project ---, Job 9181, Error Code= -2147212544.
The primary common feature of "system call" errors is that the application has placed a call to an operating system function, and that call failed. Usually the call is made to obtain some system resource: to allocate memory, fork a new thread or open a file handle. Failure generally means that the resource is nearly exhausted on the machine.
"System call" errors, then, usually indicate a resource shortage.
The error "Synchronous call timed out after
n milliseconds," where
n is a number that may vary, usually means that the CPUs are too busy. When a database connection in Strategy Intelligence Server is active ("Busy" in the database connection monitor), the Intelligence Server periodically checks to be sure the connection is healthy and expects the database connection process to respond within a certain amount of time. If the processor(s) is/are overloaded, the database connection process may not be able to respond in time and the query will fail with the "Synchronous call timed out" error.
"Synchronous call timed out" may also mean that the database connection process is handling a very large result set, compromising its ability to reply quickly to the Intelligence Server. This could potentially occur more frequently for queries against OLAP Cube data sources (SAP BI, Microsoft Analysis Services or Hyperion Essbase) because such queries require additional processing in the Query Engine to render the multidimensional results from the database into the tabular format used for report display and manipulation. CPU demands increase with the size of the report.
Time out errors are not always specific to high CPU activity, however. If physical RAM is completely occupied, there may be high virtual memory paging on the system. This can cause processes to perform slowly and respond late to inter-process communications.
Report execution or metadata connectivity failures
The error message often points to the cause of the problem. References to memory, contract request denials ("No memory to be contracted"), or memory-related UNIX error codes such as ENOMEM point to memory shortages. Failure to open named pipes in UNIX could indicate insufficient file handles available to the process, or inadequate permissions on the system's temp directory. "Process hung" or timeouts suggest excessive CPU activity.
Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. | Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. |
Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. | Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. |
Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. | Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. |
Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. | Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. |
Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. | Cube does not uses all the attribute elements.
MessagePossible causeNo memory to be contracted. Contract request rejected.Addressing space available to Strategy Intelligence Server is low. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure the Intelligence Server's use of its address space.Could not spawn process. Failed in fork System Error (ENOMEM): Not enough space.System memory outside of Strategy Intelligence Server is too low for new database connections to be opened. Review Memory Contract Manager configuration, and use Performance Monitor logging to measure system memory levels.Error communicating with connection process. Process hung.
Error communicating with connection process. Synchronous call timed out after -- milliseconds.The CPU is overcommitted, and database connection processes are not able to respond back to Strategy Intelligence Server rapidly enough.
Or, virtual memory use is high and RAM pages are being swapped to and from disk at a high rate. This can also cause processes to perform slowly and prevent timely replies from database connection processes to Strategy Intelligence Server.Could not spawn process. Cannot open named pipe for read/write. System Error (ENXIO): No such device or address.Problems with file system access to create named pipes. This could be related to insufficient permissions on the system tmp directory in UNIX, incorrect tmp directory location, or a shortage of file handles due to restrictive ulimit settings. |
In all such cases, a standard set of troubleshooting data should be collected:
- Performance Monitor logging for CPU usage (both systemwide and for the Intelligence Server process) and memory usage (both systemwide and for Intelligence Server).
In Windows environments, systemwide CPU and memory counters are found under the Memory and Processor performance objects. If the machine has more than one CPU, all of the numbered processor instances should be selected in addition to _Total in the Processor object. Counters specific to the Intelligence Server are found under the Process performance object, where the MSTRSvr.exe instance should be selected. Under Process, % Processor Time, Private Bytes and Virtual Bytes should be taken.
In UNIX environments, the Strategy Diagnostics and Performance Logging Tool should be used. Systemwide data are under the Memory and Processor counters, and Intelligence Server data are under Process. It is also recommended to supplement system-level data using an operating system-specific performance tool.
- Along with memory and CPU performance monitor counters, data from the Strategy Intelligence Server Jobs object can be helpful to measure report concurrency. "Executing Reports" and "Open Reports" are the most important counters.
- Machine specifications should be checked for number of CPUs, clock speed, and amounts of physical and virtual memory. The Strategy Service Manager shows which CPUs are available to the Strategy Intelligence Server. In some cases, the machine may physically have enough CPU resources but some processor(s) may be disabled due to the conditions of the licensing contract.
- In UNIX environments, the result of the command ulimit -a. User limits may have been set too low for Strategy Intelligence Server Universal to run effectively.
Causes for "Error communicating with connection process. Process hung" other than resource contention
The message "Error communicating with connection process. Process hung" may occur in some cases where no resource contention is in evidence.
- Connection caching: Strategy may keep database connections open after report execution completes, so that the connection can be reused without incurring the overhead of logging into the database again. A database or network timeout setting may close the connection without Strategy's knowledge, in which case a subsequent attempt to use the cached connection would fail with the "process hung" message.
- Insufficient value for the database connection property "maximum connection attempt time":
- Metadata and warehouse sharing the same Data Source Name (DSN): If the metadata in the warehouse share the same DSN, connection caching and connection lifetime settings may become interchanged. It is generally recommended to keep the metadata and warehouse on separate physical database servers for performance reasons.
Unresponsiveness (hanging) or unexpected shutdowns of Strategy Intelligence Server
Note: This discussion concerns only shutdowns or hangs of Strategy Intelligence Server that are accompanied by system call errors in the DSSErrors.log. Without these errors, the cause is very likely something else and should be investigated as a separate issue.
System call errors may affect the responsiveness or stability of Strategy Intelligence Server. In many cases, the Intelligence Server's behavior may be a side effect of the system's state as a whole. In particular, high CPU use may prevent the Intelligence Server from handling user requests quickly.
High memory use can lead to two types of unexpected shutdown:
- Memory request idle timeout, in which the Memory Contract Manager is unable to satisfy memory requests for too long a time. In this case, the shutdown is normal behavior of the Memory Contract Manager; however, the system call errors may highlight the cause of memory unavailability. This issue appears in DSSErrors.log as a long sequence of memory contract denials leading up to the shutdown.
In general, the same types of performance and system configuration data should be collected as in the case of generic system call errors. The system call errors may be useful to distinguish whether the shortage is in the Intelligence Server's addressing space or in memory used for other processes (including Strategy database connection processes).
If hanging or sluggishness in Strategy Intelligence Server is caused by high CPU use, the poor response is likely to be accompanied by "process hung" or "synchronous call timed out" messages in DSSErrors.log. Performance monitoring should be used to check CPU activity levels during the time when the system is responding slowly. Memory paging counters can identify if the problem is due to excessive virtual memory swapping instead of execution of user processes. Memory paging cannot be tracked in the Strategy Diagnostics and Performance Logging tool, but is available through the Windows performance monitor and UNIX-
Further reading
