Project

General

Profile

High java heap memory usage after upgrading to the JDK 8

Ehsan Mozaffari
Added about 3 years ago

Hi

As new versions of Tigase uses JDK8 I have installed TigaseSNAPSHOTb3983 to utilize XEP-0184 for offline messages.

Unfortunately, the heap memory usage has been increased in my server after the upgrade.

I have been struggling to figure out what is the reasons for this issue since last week.

My server is TigaseSNAPSHOTb3983 UBUNTU VMware with 4GB RAM, but, we had no problem with JDK 7 on tigase server b3821.

I am suspicious to my "tigase.conf" parameters because in the tigase monitoring console the read-outs for heap memory is "Mem:27.1/-1%".

Some times it comes over 50% and without even one user.

Please check the attached screen shot.

The following are my tigase.conf and init.properties

#osgiEnabled=(true|false)
#osgiEnabled=false
OSGI=${osgiEnabled}
ENC="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
DRV="-Djdbc.drivers=com.mysql.jdbc.Driver:org.postgresql.Driver:org.apache.derby.jdbc.EmbeddedDriver"
#GC="-XX:+UseBiasedLocking -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelCMSThreads=2 -XX:-ReduceInitialCardMarks"
#EX="-XX:+OptimizeStringConcat -XX:+DoEscapeAnalysis -XX:+UseNUMA"
#JMX_REMOTE_IP="-Djava.rmi.server.hostname=0.0.0.0"
JAVA_HOME="/root/jdk1.8.0_60/"
CLASSPATH=""
PRODUCTION_HEAP_SETTINGS=" -Xms1G -Xmx1G " # heap memory settings must be adjusted on per deployment-base!
JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} ${JMX_REMOTE_IP} -server ${PRODUCTION_HEAP_SETTINGS} -XX:MaxDirectMemorySize=128m "
TIGASE_OPTIONS=" --property-file etc/init.properties "
config-type = --gen-config-all
--virt-hosts=x5.test.net,xmpp.test.net
--admins = admin@xmpp.test.net
--user-db = mysql
--user-db-uri=jdbc:mysql://dbtest.net/tigasedb?user=tigase&password=tigase12
--debug=server,db
--monitoring=jmx:9050
--sm-plugins=+jabber:iq:auth,+urn:ietf:params:xml:ns:xmpp-sasl,+urn:ietf:params:xml:ns:xmpp-bind,+urn:ietf:params:xml:ns:xmpp-session,-jabber:iq:register,+jabber:iq:roster,+presence,+starttls,+amp,+vcard-temp,+http://jabber.org/protocol/commands,+jabber:iq:private,+urn:xmpp:ping
--amp-security-level=NONE

Thanks in advance for your kind.


Replies (7)

Added by Wojciech Kapcia TigaseTeam about 3 years ago

You are using both different JDK and Tigase, and we have changed default settings for heap. Please make sure on both configuration Xmx/Xms settings are exactly the same.

Added by Hamid Alimohammadi about 3 years ago

Hi Wojciech,

Thank you for the reply.

Ehasn and I have been struggling to update JDK to Oracle 1.8 for TigaseSNAPSHOTb3983, but in Tigase Monitor readouts in the memory dashboard there is a permanent -1 which implies something is wrong: Mem:11.7%/-1%

Sorry but we are a little confused. To my understanding from the reply, in our installation, using Oracle jdk1.8.0_60 and Tigase 3983 are ok, but we are using wrong default Heap settings (kindly correct me if I misunderstood).

If the issue is related to Heap settings;Is there any link in Tigase docs or any hint for new default Heap size?

Thank you again for help

Added by Wojciech Kapcia TigaseTeam about 3 years ago

Hamid Alimohammadi wrote:

Ehasn and I have been struggling to update JDK to Oracle 1.8 for TigaseSNAPSHOTb3983, but in Tigase Monitor readouts in the memory dashboard there is a permanent -1 which implies something is wrong: Mem:11.7%/-1%

Actually not, it's not wrong. This indicates that the (in this case) direct memory information is not available.

Sorry but we are a little confused. To my understanding from the reply, in our installation, using Oracle jdk1.8.0_60 and Tigase 3983 are ok, but we are using wrong default Heap settings (kindly correct me if I misunderstood).

Yes, latest nightlies require latest JDK8. Please share etc/tigase.conf from both installations. We've changed the defaults recently hence if you were using defaults this may be a reason.

If the issue is related to Heap settings;Is there any link in Tigase docs or any hint for new default Heap size?

We've switched from setting initial Xmx=200M to default JVM behaviour which calculates max heap based on the resources. Of course while deploying Tigase it's recommended to adjust HEAP memory settings (Xmx) accordingly.

Added by Hamid Alimohammadi about 3 years ago

Hi again Wojciech,

Sorry for my late reply.

Wojciech Kapcia wrote:

Please share etc/tigase.conf from both installations. We've changed the defaults recently hence if you were using defaults this may be a reason.

The followings are etc/tigase.conf from both installation:

tigase-server-7.0.1-b3810 using Oracle JDK 1.7.0_79:

#osgiEnabled=(true|false)
#osgiEnabled=false
OSGI=false
ENC="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
DRV="-Djdbc.drivers=com.mysql.jdbc.Driver:org.postgresql.Driver:org.apache.derby.jdbc.EmbeddedDriver"
#GC="-XX:+UseBiasedLocking -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelCMSThreads=2 -XX:-ReduceInitialCardMarks"
#EX="-XX:+OptimizeStringConcat -XX:+DoEscapeAnalysis -XX:+UseNUMA"
#JMX_REMOTE_IP="-Djava.rmi.server.hostname=0.0.0.0"
JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
CLASSPATH=""
JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} ${JMX_REMOTE_IP} -server -Xms64769344 -Xmx1038090240 -XX:PermSize=32m -XX:MaxPermSize=256m -XX:MaxDirectMemorySize=128m "
TIGASE_CONFIG="etc/tigase.xml"
TIGASE_OPTIONS=" --property-file etc/init.properties "

tigase-server-7.1.0-SNAPSHOT-b3983 using Oracle JDK jdk1.8.0_60

#osgiEnabled=(true|false)
#osgiEnabled=false
OSGI=${osgiEnabled}
ENC="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
DRV="-Djdbc.drivers=com.mysql.jdbc.Driver:org.postgresql.Driver:org.apache.derby.jdbc.EmbeddedDriver"
#GC="-XX:+UseBiasedLocking -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelCMSThreads=2 -XX:-ReduceInitialCardMarks"
#EX="-XX:+OptimizeStringConcat -XX:+DoEscapeAnalysis -XX:+UseNUMA"
#JMX_REMOTE_IP="-Djava.rmi.server.hostname=0.0.0.0"
JAVA_HOME="/root/jdk1.8.0_60/"
CLASSPATH=""
PRODUCTION_HEAP_SETTINGS=" -Xms65011712 -Xmx1038090240 " # heap memory settings must be adjusted on per deployment-base!
JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} ${JMX_REMOTE_IP} -server ${PRODUCTION_HEAP_SETTINGS} -XX:MaxDirectMemorySize=128M -XX:MaxNewSize=346030080 -XX:NewSize=21626880"
TIGASE_OPTIONS=" --property-file etc/init.properties "

Added by Wojciech Kapcia TigaseTeam about 3 years ago

And with the above settings you observer such discrepancy in heap memory usage? Could you share detailed statistics about heap usage?

Could you also run tigase-server-7.0.1-b3810 against jdk7 and jdk8 and compare memory usage?

Added by Hamid Alimohammadi about 3 years ago

Yes that right with above settings we observe such discrepancy in heap memory usage.

I have made screenshots for detailed statistics about heap usage and also snapshot for what you asked for comparing tigase b3810 against jdk7 and jdk8.

In the new test for tigase-server-7.0.1-b3810 against jdk8 we noticed that memory readouts is: Mem:12.6%/100 while in tigaseSNAPSHOTb3983 against jdk8 it was: Mem:11.7%/-1%

Please check the heap usage snapshots at file attachment.

Added by Wojciech Kapcia TigaseTeam about 3 years ago

Hamid Alimohammadi wrote:

Yes that right with above settings we observe such discrepancy in heap memory usage.

Can you try running Tigase with the exact same JVM options? In the second case you've added @ -XX:MaxNewSize=346030080 -XX:NewSize=21626880@.

    (1-7/7)