Project

General

Profile

Out of memory error

Matthew M
Added over 4 years ago

We ran into this exception once and could not establish any connection, restart tigase server seems fix it.

We are still trying to investigate into the problem, but if Tigase team could share some insights that what

could the the cause, that would be very helpful.

tigase.conf:

JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} -server -Xms100M -Xmx200M -XX:PermSize=32m -XX:MaxPermSize=256m -XX:MaxDirectMemorySize=128m "

Exception

RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=57393] throws
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:403)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
at java.lang.Thread.run(Thread.java:744)

It happens in the late mid night where we don't have any high load at all. Well this brings up another question,

what's the best practice to monitor the "load" of the Tigase server and persist the load data? In this case, we might

be able to examine the historical load metrics to associate with event of the out-of-memory exceptions.

  1. I found this to get server statistics, do you think we can get the any data related to server load? Such as connection / thread / memory usage?

https://projects.tigase.org/projects/tigase-http-api/wiki/Usage_examples

  1. I also found this option of --stats-history http://www.tigase.org/content/stats-history

how the historical stats could be retrieved?

  1. Since the "--stats-history" cache the metric data in the memory, is that an easy to add a log statement to log the data so that we can persist

the data via log files? What would be best java class to look into to add log statement for metrics?

Thanks!

For example,

maybe print to a log so that late on we can associate the load history with this kind of exception?

I recall tigase has a "statistic"


Replies (3)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Out of memory error, is usually hard to find out cause afterwards if you do not have any monitoring data.

Please note, --stats-history can consume lots of memory and by default it actually does consume lots of memory, so ironically it could be the cause of the OOM in your case. I do not remember how much it was the max usage if you have all the Tigase possible components loaded it may even get as large as 1GB.

The best way to collect Tigase metrics is through JMX. Tigase exposes all it's metrics via JMX and you can pull them using a simple command line tool attached to the topic as a code example.

Added by Matthew M over 4 years ago

Thanks for the tips and groovy script! It looks great!

I hardly groovy before but would love to get to know more about it, seems pretty cool.

I wonder

  1. what's the full command line look like to run this groovy script (in Mac) ?

  2. what directory / path I should run this in order to properly "import" all the java package? (Is it in tigase server project) ?

Added by Wojciech Kapcia TigaseTeam over 4 years ago

This is not a tigase-server project but it depends on tigase-server.jar. You can run it from anywhere as long as you provide correct paths to both tigase-server.jar as well as StatsDumper.groovy (assuming it's not in the current directory):

$groovy -cp <path_to_tigase-server.jar> <path_to_StatsDumper.groovy> <options>

passing -h as option will give you list of parameters

    (1-3/3)