Project

General

Profile

Tigase using too much memory

Kulshreshth Dhiman
Added over 4 years ago

Hi,

we have allocated 8gb as max memory in tigase.conf file, but we are observing that the memory goes beyond 12GB. There are around 60k connections and around 30k messages sent per day. It doesn't go out of memory, but we are unable to find out the reason for this huge memory usage. Actual memory used is around 5GB and we are using java 1.7, garbage collector is G1GC. Please guide us.

Regards,

Kulshreshth


Replies (6)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

A few questions:

  1. How do you know/monitor memory usage?

  2. If this is system top command which column do you take into consideration?

  3. Do you run Tigase inside a VM? We had experience with some VMs and JVMs that memory settings were not respected and JVM used much more RAM that it was allowed to (actually all the available RAM).

  4. Do you check what memory usage is reported by JVM or Tigase in Tigase's metrics?

Added by Kulshreshth Dhiman over 4 years ago

Hi,

We are using top to measure memory usage. We are running Tigase on network optimized AWS instance. Here is top command output,

top - 15:52:52 up 39 days, 50 min,  2 users,  load average: 0.91, 2.26, 1.74
Tasks: 121 total,   1 running, 120 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.9%us,  2.0%sy,  0.0%ni, 93.2%id,  0.0%wa,  0.0%hi,  0.8%si,  0.2%st
Mem:  15404752k total, 13644236k used,  1760516k free,     5420k buffers
Swap:        0k total,        0k used,        0k free,    25036k cached

PID     USER  PR  NI  VIRT  RES  SHR  S %CPU %MEM    TIME+  COMMAND                                                                                  
11946 root      20   0    17.7g 7.8g  5792 S  65.9    52.9         23:31.91 java

But from the tigase stats we see that Used Heap is around 2-2.5G,

Other message router stats are attached.

Also, if heap memory is around 2gb, what are the other processes that might be taking memory like threads, sockets etc? Below is the memory used by sockets which is also low.

cat /proc/net/sockstat
sockets: used 53977
TCP: inuse 54807 orphan 916 tw 1349 alloc 54809 mem 20655
UDP: inuse 5 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

Here is JVM settings from tigae.conf

JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} -server -Xms2G -Xmx8G -XX:PermSize=128m -XX:MaxPermSize=512m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -Djava.rmi.server.hostname=IP -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

Tigase server went down due to out of memory error after running for 20 hours when we reduced Xmx to 4G. What could be the possible reasons for this behavior.

Regards,

Kulshreshth

Added by Kulshreshth Dhiman over 4 years ago

Hi,

  • We are looking at RES column in top which is increasing beyond defined limit(Xmx).

Thanks

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Actually everything looks correct. You assign to Java (Tigase) process 8GB of RAM and the top command shows that Java process allocated 7.8GB which is close to the max but still below. Java shows that it uses about 2 - 2.5GB of heap which also is correct. You have to be aware that usually Java process allocates all the memory it is allowed to allocate because it has own, internal memory management mechanisms. So even if it allocates 8GB of RAM it does not mean that the whole memory is used by the actual JVM application. In the case above, Tigase uses only 2 - 2.5GB of all allocated memory, it should also show about 5 - 6 GB of used heap which gives you plenty of room.

So the above comments are technical stuff related to memory management. Another, more or less separate issue is whether the Tigase uses too much memory. There is not simple answer to this question. It depends on many factors. One of them is an average contact list size (roster size). You say you have 60k user connections. If each user has approx 200 contacts in his roster then the server keeps in memory 12 million contacts and some metadata about these contacts. This requires memory of course. You also say, you have approx 30k messages a day. But if all of these 30k are sent at the same time, then there can be time when you gave 30k messages per second and then nothing for the rest of the day. Still 30k messages a day on average but at some point Tigase requires a lot of memory to process 30k messages at the same time. Another thing is, do you have any other traffic like presence traffic. For 60k users and 200 roster, assuming each user logins only once a day and logs out once a day, this can generate more than 50M presence packets a day. On average this is just 600 presences a second, but usually users login at more or less the same time and logout at more or less the same time, so there could be a peak traffic times.

There are also other factors which may affect memory usage, such as group chat, stream compression, etc....

From our LT we know that Tigase can handle up to 160k online users with average roster size of 150 on real HW machine with 16GB of RAM.

Added by Kulshreshth Dhiman over 4 years ago

Hi,

We disabled zlib compression from the server and this solved our memory issue.

Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie

Cpu(s): 5.5%us, 2.5%sy, 0.0%ni, 90.7%id, 0.0%wa, 0.0%hi, 1.0%si, 0.3%st

Mem: 15404752k total, 4349240k used, 11055512k free, 90876k buffers

Swap: 8388604k total, 6312k used, 8382292k free, 706828k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

22286 root 20 0 14.5g 2.6g 12m S 86.5 17.5 30:50.15 java

Earlier, JVM memory was constant but RES was continuously increasing. Now JVM memory and RES in top are close to each other.

Zlib plugin uses ByteBuffer. Please check if there is memory-leak in this plugin.

Thanks

Kulshreshth

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Hm, I remember we investigated a similar problem a while ago. Apparently this problem depends on the environment. It's not a problem with Tigase own code but some combination with lib library installed on the operating system and certain JVM installed.

Andrzej worked on this problem as far as I remember so he may provide more details and suggest a solution which does not require disabling compression on the server.

    (1-6/6)