KB438764: Amazon Network File System(NFS) Performance

This article addresses how you can optimize and/or Tune the Amazon Network File System to handle High IO related performance issues.
 
Problem:
When Strategy utilizes Redhat Enterprise Linux as the Operating System for running Intelligence Server, the server will experience high I/O demand during common activities such as cube cache publishing and load operations. In some instances, hundreds of gigabytes of cube data must be loaded from disk immediately on server startup.
A range of Network File Share issues may manifest in peak I/O: hanging Strategy processes, cube/cache unavailability, mount points failing, Redhat Kernel Panics, and automatic machine restarts due to hung tasks. All of these symptoms can be traced to the NFS module in Redhat, which is responsible for I/O operations, during its interaction with mounted shared network drives. As the I/O footprint rises, instability with the Redhat kernel and NFS module exhibited instability and triggered multiple outages.
The common system-level error messages will be similar to this block in the case of a hung kernel task reboot:
 <hostname> kernel: : INFO: task MSTRSvr:43561 blocked for more than 300 seconds.
 <hostname> kernel: : Not tainted 2.6.32-642.11.1.el6.x86_64 #1
 <hostname> kernel: : "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Solution:

1. Disable kernel hung task - By default, in the Linux kernel the kernel.hung_task_panic tunable is enabled and the kernel.hung_task_timeout_secs tunable is set to a default 300, this parameters will cause the underlying OS to panic and reboot in case of Strategy process hang detected for 5 minutes. To ensure that the node does not panic and reboot, the kernel.hung_task_panic tunable is recommended to be disabled.
2. 2. Virtual Memory configuration - Virtual memory is typically consumed by processes, file system caches, and the kernel. Virtual memory utilization depends on a number of factors, which can be affected by the following parameters and our recommendations are:
   
• dirty_ratio Defines a percentage value. Writeout of dirty data begins (via pdflush) when dirty data comprises this percentage of total system memory. The default value is 

  20

  . This value is recommended to be 5.

• dirty_background_ratio -  Defines a percentage value. Writeout of dirty data begins in the background (via pdflush) when dirty data comprises this percentage of total memory. The default value is 

  10

  . This value is recommended to be 

  3.

KB438764