Understanding TIBCO ActiveMatrix BusinessWorks? Engine Memory and CPU consumption

Table of Contents

• CPU and thread utilization are governed by various factors:
• Troubleshooting the "OutOfMemory" exceptions and possible remedies:
• Document References
• Troubleshooting
• Information to be sent to TIBCO Support
• For Unix Systems
• For Windows Systems

One of the key features of the TIBCO ActiveMatrix BusinessWorks? Engine is its ability to optimize system resources (CPU, memory, and disk) and therefore generate higher throughput. Memory consumption increases as the number of jobs being processed increases. If the JVM max heap size is sufficient, then the ActiveMatrix BusinessWorks engine will process all jobs and when completed, memory usage should return to normal.

Sometimes, the ?java.lang.OutOfMemory? error in the ActiveMatrix BusinessWorks engine is reported by users. ?OutOfMemory? exceptions occur when there is not enough memory for the ActiveMatrix BusinessWorks engine to process jobs. Occasionally, users report that ActiveMatrix BusinessWorks engine(s)? performance is slow, CPU utilization is high and there is a noticeable increase in memory consumption. The following explains some of the issues surrounding excessive ActiveMatrix BusinessWorks resource usage.

CPU and thread utilization are governed by various factors:

• Load on the engine. The number of jobs being processed when the CPU spike is observed.

• JVM heap consumption.

• Process-intensive jobs (process design, conditional looping resulting in a high amount of mathematical computations, custom Java code).

• A particular activity in the process is taking a long time to execute resulting in high CPU usage.

• Engine tuning parameters: Engine.ThreadCount

  Engine.ThreadCount sets the number of engine worker threads. The number of threads limits the number of process instances that can be executed concurrently. The typical number of worker threads ranges between 4 and 32. (Default value=8)

  This property should be tuned carefully by calculating the available CPU and memory resources on your system under a typical processing load.

  Specifying too low a value for the thread pool can cause higher memory utilization as it will queue up the requests waiting to be processed and lower the engine throughput even though spare CPU resources exist.

  Specifying too high a value can cause CPU thrashing (context switching) behavior.

Troubleshooting the "OutOfMemory" exceptions and possible remedies:

• If there are "OutOfMemory" exceptions reported along with high CPU usage, then consider tuning the parameters Flow Limit, Max Jobs, and the JVM heap size. If the engine runs out of memory due to increased load then possibly the ActiveMatrix BusinessWorks engine is running out of resources. If the load is constant but memory utilization is increasing, then there is possibly a memory leak in the process. In case of a memory leak, isolate the process and the activity which is causing the leak. For the "OutOfMemory" exceptions due to resource availability, consider tuning the following parameters for the ActiveMatrix BusinessWorks engine:

  1. JVM heap size

     It is important to set the heap size to a suitable value. It should be set such that it allows the heap to be contained within the physical memory to avoid swapping to disk

  2. MemorySavingMode

     This parameter significantly improves the memory footprint for processes that manipulate a large amount of data for a small part of their lifetime. It allows the engine to release references to unused data objects (say, huge XML data) so that it can be garbage collected by the JVM, thus improving performance.

  3. Tuning Max Jobs, Activation Limit, and Flow Limit

     ?Max Jobs? allows you to specify the maximum number of concurrent process instances that can be stored in memory. ?Activation Limit? controls the paging mechanism. If the pace of incoming events still exceeds the number of threads available to run them, consider using ?flow limit?. For more details on these options, please refer to the Performance Tuning section in the ActiveMatrix BusinessWorks documentation.

• If there are "OutOfMemory" exceptions due to resource constraints, then consider hardware scaling.

  1. The number of processors/machine.

     If the engine is using too much CPU even after tuning the thread pool and the message load is high consider using a multi-CPU machine.

  2. For better CPU utilization in heavy load scenarios, you can also consider distributing your ActiveMatrix BusinessWorks instances using Distributed Queue.

It is best practice to do some performance and stress tests and to tune the ActiveMatrix BusinessWorks engine parameters such as java.heap.size, Flow limit, Max Jobs and Thread Count. You can tune the value of the Engine.ThreadCount property between 4 and 32 depending on your CPU utilization and load (We recommend 8 threads per CPU).

Document References

For details, please refer to the following ActiveMatrix BusinessWorks documentation :

Administration --> Chapter 10 Performance Tuning

Troubleshooting

To find out the cause of high CPU and memory usage:

- On Windows, you can check this using Task Manager ?Processes Tab.

Monitoring the amount of allocated virtual memory size and memory usage allows you to see whether a process is consuming more memory over a given period of time.

- On Unix, you can run commands like top, prstat, vmstat, ps aux

Sample output of the top command:-

 PID USERNAME LWP PRI NICE  SIZE  RES  STATE  TIME   CPU  COMMAND  7664  tibuser 460   1   0  1219M 1071M sleep  46.0H 0.18% tibcoadmin_TIBD 21428  tibuser 999  39   0  1609M 1316M sleep  25:38 0.10% bwengine 28813  tibuser 999   9   0   595M  514M sleep 445:14 0.04% bwengine 20447  tibuser 342  15   0   710M  457M sleep 145:18 0.03% bwengine 27449  tibuser  51  59   0   267M  251M sleep  71:09 0.02% adr3u 27752  tibuser 227  59   0   432M  268M sleep 585:46 0.01% bcengine

You can find the pid of the ActiveMatrix BusinessWorks engine by using

 ps ?ef | grep bwengine 

and run the prstat command to monitor the CPU usage as:

 prstat ?p  5 > stats.log 

(where pid is the process id for the bwengine showing High CPU)

- You can use the getMemoryUsage() Hawk microagent method to check the heap allocation for the process causing high memory usage.

- Monitoring Tools like JConsole: With jre1.5 or higher, JMX monitoring tools like JConsole can be enabled and used to monitor the ActiveMatrix BusinessWorks engine.

- Garbage collection (GC)

Setting java.extended.properties=-verbose:gc in tra file will print out GC output to the console. From this output the amount of memory released after GC and Full GC.

- If "java.lang.OutOfMemory" exceptions due to lack of heap space are observed and the heap size is sufficiently high then collect Heap Dump to check which object/class is causing the out-of-memory exception by adding the following property in the .tra.

 java.extended.properties=-verbose:gc -XX:+HeapDumpOnOutOfMemoryError                                      -XX:HeapDumpPath=/tmp/dumps.log

• Once the ActiveMatrix BusinessWorks engine causing the high CPU usage has been identified, check the activities used in the process and identify if there are activities that show high CPU time.

• Check the Admin GUI for the execution time of these activities by collecting engine statistics. The statistics will show for each job and activity how much eval time and elapsed time is taken.

• Check the Admin GUI- Active processes to see if the jobs are hanging at a particular activity.

• Check the ActiveMatrix BusinessWorks application logs to see if there are any exceptions being thrown.

• Check the memory consumption of the engines.

• If the logs show any errors related to RV, then enable tracing for RV and check for any errors. (See Chapter 12 Protocol Monitor (rvtrace) in the TIBCO Rendezvous manual).

• If the issue is reproducible, please run the engine from the command line, redirecting its output to the file and collect thread dumps.

Information to be sent to TIBCO Support

1. Operating system and hardware configuration.

2. Details on the number of CPUs and type.

3. Confirm the Admin/TRA/ActiveMatrix BusinessWorks versions with hotfixes, if any.

4. Details on message rate and message size and volume.

5. The multi-file project and the deployed .ear file.

6. The deployed application .tra file: /tra/domain//application//.tra

7. The ActiveMatrix BusinessWorks engine logs.

8. CPU and memory statistics (For example, ps, vmstat, top, dependent on operating system)

9. Details on incoming message rate and message size.

10. From the Admin GUI, click on the particular process and collect the statistics of the Activities. This will have columns such as 'Name of the activity', 'Called process', 'Execution Count', 'Elapsed Time (ms), CPU time (ms), 'Errors', 'Status, 'Function'

11. From the Admin GUI, collect engine statistics. Click on Service Instance-Engine Control Tab?collect statistics ---Start and save this to a .csv file.

12. From the Admin GUI, screenshots of the active processes (if any) show the jobs hanging/stuck at a particular activity.

13. Thread dump, if the issue is reproducible, restart the ActiveMatrix BusinessWorks engine from the command line, re-direct the output to a file, and capture thread dumps as follows:

For Unix Systems

• Run ?/tra/domain//application//.sh > debug.out 2>&1? and note the PID for the engine

• When the issue occurs execute ?kill -3 ? for the engine 2 or 3 times at intervals of 5 minutes. Then send us the re-directed output (debug.out).

For Windows Systems

• Run \tra\domain\\application\\.cmd > debug.out 2>&1

• When the issue occurs, press the key combination ?Ctrl + Break? when the focus is on the running application at least 2 or 3 times at intervals of 5 minutes. Then send us the re-directed output (debug.out).