Project

General

Profile

How to narrow down tigase debugging to spot a dead thread in tigase

Ehsan Mozaffari
Added almost 4 years ago

Hi dude!

I have trapped on my xmpp server CPU load when clients were connect to tigase server.

The problem is CPU load without any pattern is increase everyday multiple times.

So too many SEVER and WARNING errors are appearing in logs/tigase-console.log

I am totally detest java.lang.NullPointerException error and all I wanted to know is the root cause of such this error.

I've used some java development tools such as: jstack , jconsole , top command thread monitoring and I realize some threads life time approached to 15 mins and they stuck in LOCKED state and throw exceptions. They almost always are 2 threads in this states. This way has many problems, one problem is, you don't know the exact time of thread, states and CPU load as well as the client side users maybe sent wrong commands.

There are many stanza which are seen by me after cpu load and it's complicated to figure out and find out the error which caused this problem in logs/tigase-cosole.log and logs/tigase.log.0.

I have been read the administration parts of tigase forum and docs and found this link

[[[http://docs.tigase.org/tigase-server/7.0.0/Administration_Guide/html_chunk/_debuging_tigase.html]]]

I added this options into init.properties --debug=server,xmpp.XMPPIOService but it didn't solve my problems.

So the main question is:

Is it possible to add some options to etc/init.properties --debug to show threads state,life time,CPU usage in tigase.log.0 into each stanza.

I really appreciate in advance for your time spending on my question.


Replies (4)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

  1. Could you tell us what is your problem from the end-user point of view?

  2. Could you please tell us more about your installation and your use-case? How many users connect to your server, how many messages are sent, etc...

  3. Can you provide us your config files for review?

  4. What Tigase version do you use?

  5. Do you have any custom code?

  6. What database do you use?

  7. What OS do you have Tigase running on?

Such problems you describe are often caused by configuration mistakes or by custom code.

Added by Ehsan Mozaffari almost 4 years ago

Thank you for replying.

I am sorry because my post has not enough information that you need.

7.CentOs 6.5

6.Mysql

5.No

4.b3810

3.The Tegase init.properties file has been attached.

2.There are about 27 online test clients. I used manual installation method (*.gz.tar) to install Tigase server in single mode (not clustered). We are an android application for mobile-phones which is developed by our software team. Regarding the number of messages, as I mentioned we have 27 test concurrent online users that occasionally send messages. On average, each user has 80 buddies in his roster.

1.Although the android developers receives some errors(please check the following read outs), my understanding is that these few errors from 27 clients shouldn't cause CPU-Load in Tigase.

In the server-side I see these exceptions and Threads that consume too much CPU_Load (Using jStack, jconsole , top , tigase.log.0 , Virtual VM )


1948tigase2004082m904m12mR13.323.623:16.77java 23:16.77 java "socketReadThread-6" prio=10 tid=0x00007f79a00ef800 nid=0x79c runnable [0x00007f7989d9d000]

2015-08-05 15:03:06.558 [socketReadThread-6]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_43075, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_43075 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout

1962tigase2004082m904m12mS13.323.616:41.60java 16:41.60 java "socketReadThread-13" prio=10 tid=0x00007f79a01bb800 nid=0x7aa runnable [0x00007f7988f8f000]

2015-08-05 15:03:57.000 [socketReadThread-13]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357 Socket[unconnected], jid: 441111111112@xmpp.test.net/*, service stopped - resumption disabled, sending unacked packets
2015-08-05 15:03:57.000 [socketReadThread-13]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Connection stopped: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-05 15:03:57.000 [socketReadThread-13]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Attempt to stop incorrect service: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-05 15:03:57.000 [socketReadThread-13]  ClientConnectionManager.xmppStreamClosed()  FINER: Stream closed: c2s@xmpp.test.net/hostIP_5222_test.ip1_10357

1938tigase2004082m904m12mR13.323.619:03.74java 19:03.74 java "socketReadThread-1" prio=10 tid=0x00007f79a005e800 nid=0x792 runnable [0x00007f798a7a7000]

2015-08-05 15:05:43.510 [socketReadThread-1]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395 Socket[unconnected], jid: 441111111112@xmpp.test.net/*, service stopped - resumption disabled, sending unacked packets
2015-08-05 15:05:43.510 [socketWriteThread-1]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395 Socket[unconnected], jid: 441111111112@xmpp.test.net/*, service stopped - resumption disabled, sending unacked packets
2015-08-05 15:05:43.511 [socketReadThread-1]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Connection stopped: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-05 15:05:43.511 [socketReadThread-1]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Attempt to stop incorrect service: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-05 15:05:43.511 [socketReadThread-1]  ClientConnectionManager.xmppStreamClosed()  FINER: Stream closed: c2s@xmpp.test.net/hostIP_5222_test.ip1_10395

11301tigase2004058m516m12mS26.513.53:42.02java 3:42.02 java "socketReadThread-7" prio=10 tid=0x00007fade00fc800 nid=0x2c25 runnable [0x00007fadb2464000]

2015-08-06 10:53:33.342 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip5_44223, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip5_44223 Socket[unconnected], jid: 441111111113@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 10:54:07.504 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-06 10:54:07.504 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Attempt to stop incorrect service: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-06 11:08:30.945 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null


2015-08-06 12:48:53.086 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip5_45542, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip5_45542 Socket[unconnected], jid: 441111111114@xmpp.test.net, service stopped - StreamManagement disabled
2015-08-06 13:02:27.017 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:02:27.061 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:02:27.101 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:02:27.111 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47462 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:18:18.239 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:18:18.252 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:18:18.265 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout
2015-08-06 13:18:18.266 [socketReadThread-7]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip4_47514 Socket[unconnected], jid: 441111111111@xmpp.test.net/*, service stopped - checking resumption timeout

11297tigase2004066m468m10mR20.712.212:21.58java 12:21.58 java "socketReadThread-5" prio=10 tid=0x00007fade00c2000 nid=0x2c21 runnable [0x00007fadb2868000]

2015-08-08 09:25:21.283 [socketReadThread-5]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 10:56:14.525 [socketReadThread-5]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 10:57:08.901 [socketReadThread-5]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 11:00:06.579 [socketReadThread-5]  StreamManagementIOProcessor.serviceStopped()  FINEST: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625 Socket[unconnected], jid: 441111111112@xmpp.test.net/*, service stopped - resumption disabled, sending unacked packets
2015-08-08 11:00:06.579 [socketReadThread-5]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Connection stopped: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-08 11:00:06.579 [socketReadThread-5]  ConnectionManager.serviceStopped()  FINER: [[c2s]] Attempt to stop incorrect service: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625, type: accept, Socket: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625 Socket[unconnected], jid: 441111111112@xmpp.test.net/*
2015-08-08 11:00:06.579 [socketReadThread-5]  ClientConnectionManager.xmppStreamClosed()  FINER: Stream closed: c2s@xmpp.test.net/hostIP_5222_test.ip2_10625


11299tigase2004066m468m10mR20.712.211:46.81java 11:46.81 java "socketReadThread-6" prio=10 tid=0x00007fade00df800 nid=0x2c23 runnable [0x00007fadb2666000]

2015-08-08 09:03:10.257 [socketReadThread-6]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 09:03:10.257 [socketReadThread-6]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null

11301tigase2004066m468m10mR20.712.222:36.98java 22:36.98 java "socketReadThread-7" prio=10 tid=0x00007fade00fc800 nid=0x2c25 runnable [0x00007fadb2464000]

2015-08-08 09:50:15.081 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 10:26:19.015 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null
2015-08-08 10:53:16.272 [socketReadThread-7]  ConnectionManager.serviceStopped()  FINER: [[s2s]] Connection stopped: CID: null, null, type: accept, Socket: nullSocket[unconnected], jid: null


Then other process is waited, and consume more cpu process such as:

PID CPU Mem JStack Info
11297tigase2004066m468m10mR20.712.212:21.58java 20         12:21.58 java        "socketReadThread-5" prio=10 tid=0x00007fade00c2000 nid=0x2c21 runnable [0x00007fadb2868000]
11299tigase2004066m468m10mR20.712.211:46.81java 20         11:46.81 java        "socketReadThread-6" prio=10 tid=0x00007fade00df800 nid=0x2c23 runnable [0x00007fadb2666000]
11301tigase2004066m468m10mR20.712.222:36.98java 20         22:36.98 java        "socketReadThread-7" prio=10 tid=0x00007fade00fc800 nid=0x2c25 runnable [0x00007fadb2464000]
11333tigase2004066m468m10mS15.012.27:05.08java  11         7:05.08 java     "socketWriteThread-6" prio=10 tid=0x00007fade02cf000 nid=0x2c45 runnable [0x00007fadb0444000]
11331tigase2004066m468m10mS11.312.27:21.20java  11         7:21.20 java     "socketWriteThread-5" prio=10 tid=0x00007fade02b1800 nid=0x2c43 runnable [0x00007fadb0646000]
11335tigase2004066m468m10mS11.312.213:57.32java 15  13:57.32 java       "socketWriteThread-7" prio=10 tid=0x00007fade02ec000 nid=0x2c47 runnable [0x00007fadb0242000]
11354tigase2004066m468m10mS5.612.218:13.33java  5.6 12.2    18:13.33 java   "pool-15-thread-1" prio=10 tid=0x00007fade8030000 nid=0x2c5a waiting on condition [0x00007fadaef2f000]
11355tigase2004066m468m10mS5.612.218:19.81java  5.6 12.2    18:19.81 java   "pool-15-thread-2" prio=10 tid=0x00007fade8032000 nid=0x2c5b runnable [0x00007fadaee2e000]
11356tigase2004066m468m10mS5.612.218:23.68java  5.6 12.2    18:23.68 java   "pool-15-thread-3" prio=10 tid=0x00007fade8034000 nid=0x2c5c waiting on condition [0x00007fadaed2d000]
11357tigase2004066m468m10mS5.612.218:20.66java  5.6 12.2    18:20.66 java   "pool-15-thread-4" prio=10 tid=0x00007fade8035800 nid=0x2c5d waiting on condition [0x00007fadaec2c000]
11360tigase2004066m468m10mS5.612.218:19.36java  5.6 12.2    18:19.36 java   "pool-15-thread-6" prio=10 tid=0x00007fade8043800 nid=0x2c60 waiting on condition [0x00007fadae929000]
11361tigase2004066m468m10mS5.612.218:28.94java  5.6 12.2    18:28.94 java   "pool-15-thread-7" prio=10 tid=0x00007fae48027800 nid=0x2c61 waiting on condition [0x00007fadae828000]
11363tigase2004066m468m10mS5.612.218:18.01java  5.6 12.2    18:18.01 java   "pool-15-thread-9" prio=10 tid=0x00007fade8047000 nid=0x2c63 waiting on condition [0x00007fadae626000]
11364tigase2004066m468m10mS5.612.218:26.49java  5.6 12.2    18:26.49 java   "pool-15-thread-10" prio=10 tid=0x00007fae48038000 nid=0x2c64 waiting on condition [0x00007fadae525000]
11365tigase2004066m468m10mR5.612.218:16.58java  5.6 12.2    18:16.58 java   "pool-15-thread-11" prio=10 tid=0x00007fade8048800 nid=0x2c65 waiting on condition [0x00007fadae424000]
11366tigase2004066m468m10mS5.612.218:23.93java  5.6 12.2    18:23.93 java   "pool-15-thread-12" prio=10 tid=0x00007fae48039000 nid=0x2c66 waiting on condition [0x00007fadae323000]
11367tigase2004066m468m10mS5.612.218:19.70java  5.6 12.2    18:19.70 java   "pool-15-thread-13" prio=10 tid=0x00007fade804a000 nid=0x2c67 waiting on condition [0x00007fadae222000]
11290tigase2004066m468m10mS3.812.28:35.13java   3.8 12.2    8:35.13 java    "ResultsListener-socketReadThread-2" prio=10 tid=0x00007fade0068800 nid=0x2c1a waiting on condition [0x00007fadb2f6f000]
11292tigase2004066m468m10mS3.812.28:36.21java   3.8 12.2    8:36.21 java    "ResultsListener-socketReadThread-3" prio=10 tid=0x00007fade0086000 nid=0x2c1c waiting on condition [0x00007fadb2d6d000]
11294tigase2004066m468m10mS3.812.28:31.68java   3.8 12.2    8:31.68 java    "ResultsListener-socketReadThread-4" prio=10 tid=0x00007fade00a3000 nid=0x2c1e waiting on condition [0x00007fadb2b6b000]
11308tigase2004066m468m10mS3.812.28:40.58java   3.8 12.2    8:40.58 java    "ResultsListener-socketReadThread-11" prio=10 tid=0x00007fade016f000 nid=0x2c2c waiting on condition [0x00007fadb1d5d000]
11314tigase2004066m468m10mS3.812.28:37.31java   3.8 12.2    8:37.31 java    "ResultsListener-socketReadThread-14" prio=10 tid=0x00007fade01c6800 nid=0x2c32 waiting on condition [0x00007fadb1757000]
11324tigase2004066m468m10mS3.812.28:38.04java   3.8 12.2    8:38.04 java    "ResultsListener-socketWriteThread-2" prio=10 tid=0x00007fade0258800 nid=0x2c3c waiting on condition [0x00007fadb0d4d000]
11334tigase2004066m468m10mS3.812.28:36.72java   3.8 12.2    8:36.72 java    "ResultsListener-socketWriteThread-7" prio=10 tid=0x00007fade02ea000 nid=0x2c46 waiting on condition [0x00007fadb0343000]
11336tigase2004066m468m10mS3.812.28:38.00java   3.8 12.2    8:38.00 java    "ResultsListener-socketWriteThread-8" prio=10 tid=0x00007fade0307000 nid=0x2c48 waiting on condition [0x00007fadb0141000]
11338tigase2004066m468m10mS3.812.28:37.96java   3.8 12.2    8:37.96 java    "ResultsListener-socketWriteThread-9" prio=10 tid=0x00007fade0324000 nid=0x2c4a waiting on condition [0x00007fadaff3f000]
11342tigase2004066m468m10mS3.812.28:36.80java   3.8 12.2    8:36.80 java    "ResultsListener-socketWriteThread-11" prio=10 tid=0x00007fade035e800 nid=0x2c4e waiting on condition [0x00007fadafb3b000]
11350tigase2004066m468m10mS3.812.28:37.98java   3.8 12.2    8:37.98 java    "ResultsListener-socketWriteThread-15" prio=10 tid=0x00007fade03d3000 nid=0x2c56 waiting on condition [0x00007fadaf333000]
11359tigase2004066m468m10mS3.812.218:18.80java  3.8 12.2    18:18.80 java   "pool-15-thread-5" prio=10 tid=0x00007fade8042000 nid=0x2c5f waiting on condition [0x00007fadaea2a000]
11362tigase2004066m468m10mS3.812.218:21.43java  3.8 12.2    18:21.43 java   "pool-15-thread-8" prio=10 tid=0x00007fade8045000 nid=0x2c62 waiting on condition [0x00007fadae727000]
11368tigase2004066m468m10mS3.812.218:16.04java  3.8 12.2    18:16.04 java   "pool-15-thread-14" prio=10 tid=0x00007fade804b800 nid=0x2c68 waiting on condition [0x00007fadae121000]
11369tigase2004066m468m10mS3.812.218:11.81java  3.8 12.2    18:11.81 java   "pool-15-thread-15" prio=10 tid=0x00007fade804e000 nid=0x2c69 waiting on condition [0x00007fadae020000]
11370tigase2004066m468m10mS3.812.218:19.42java  3.8 12.2    18:19.42 java   "pool-15-thread-16" prio=10 tid=0x00007fade804f800 nid=0x2c6a waiting on condition [0x00007fadadf1f000]
11371tigase2004066m468m10mS3.812.218:28.54java  3.8 12.2    18:28.54 java   "pool-15-thread-17" prio=10 tid=0x00007fae4803a800 nid=0x2c6b waiting on condition [0x00007fadade1e000]
11286tigase2004066m468m10mS1.912.28:36.29java   1.9 12.2    8:36.29 java    "ResultsListener-socketReadThread-0" prio=10 tid=0x00007fade002e800 nid=0x2c16 waiting on condition [0x00007fadb3373000]
11288tigase2004066m468m10mS1.912.28:33.07java   1.9 12.2    8:33.07 java    "ResultsListener-socketReadThread-1" prio=10 tid=0x00007fade004b800 nid=0x2c18 waiting on condition [0x00007fadb3171000]
11296tigase2004066m468m10mS1.912.28:33.40java   1.9 12.2    8:33.40 java    "ResultsListener-socketReadThread-5" prio=10 tid=0x00007fade00c0000 nid=0x2c20 waiting on condition [0x00007fadb2969000]
11298tigase2004066m468m10mS1.912.28:39.02java   1.9 12.2    8:39.02 java    "ResultsListener-socketReadThread-6" prio=10 tid=0x00007fade00dd800 nid=0x2c22 waiting on condition [0x00007fadb2767000]
11300tigase2004066m468m10mS1.912.28:36.28java   1.9 12.2    8:36.28 java    "ResultsListener-socketReadThread-7" prio=10 tid=0x00007fade00fa800 nid=0x2c24 waiting on condition [0x00007fadb2565000]
11302tigase2004066m468m10mS1.912.28:36.35java   1.9 12.2    8:36.35 java    "ResultsListener-socketReadThread-8" prio=10 tid=0x00007fade0117800 nid=0x2c26 waiting on condition [0x00007fadb2363000]
11304tigase2004066m468m10mS1.912.28:35.38java   1.9 12.2    8:35.38 java    "ResultsListener-socketReadThread-9" prio=10 tid=0x00007fade0134800 nid=0x2c28 waiting on condition [0x00007fadb2161000]
11306tigase2004066m468m10mS1.912.28:33.53java   1.9 12.2    8:33.53 java    "ResultsListener-socketReadThread-10" prio=10 tid=0x00007fade0152000 nid=0x2c2a waiting on condition [0x00007fadb1f5f000]
11310tigase2004066m468m10mS1.912.28:35.28java   1.9 12.2    8:35.28 java    "ResultsListener-socketReadThread-12" prio=10 tid=0x00007fade018c000 nid=0x2c2e waiting on condition [0x00007fadb1b5b000]
11316tigase2004066m468m10mS1.912.28:34.13java   1.9 12.2    8:34.13 java    "ResultsListener-socketReadThread-15" prio=10 tid=0x00007fade01e3800 nid=0x2c34 waiting on condition [0x00007fadb1555000]
11318tigase2004066m468m10mS1.912.28:34.92java   1.9 12.2    8:34.92 java    "ResultsListener-socketReadThread-16" prio=10 tid=0x00007fade0200800 nid=0x2c36 waiting on condition [0x00007fadb1353000]
11320tigase2004066m468m10mS1.912.28:36.45java   1.9 12.2    8:36.45 java    "ResultsListener-socketWriteThread-0" prio=10 tid=0x00007fade021e000 nid=0x2c38 waiting on condition [0x00007fadb1151000]
11322tigase2004066m468m10mS1.912.28:35.18java   1.9 12.2    8:35.18 java    "ResultsListener-socketWriteThread-1" prio=10 tid=0x00007fade023b000 nid=0x2c3a waiting on condition [0x00007fadb0f4f000]
11326tigase2004066m468m10mS1.912.28:37.59java   1.9 12.2    8:37.59 java    "ResultsListener-socketWriteThread-3" prio=10 tid=0x00007fade0275800 nid=0x2c3e waiting on condition [0x00007fadb0b4b000]
11328tigase2004066m468m10mR1.912.28:37.32java   1.9 12.2    8:37.32 java    "ResultsListener-socketWriteThread-4" prio=10 tid=0x00007fade0292800 nid=0x2c40 waiting on condition [0x00007fadb0949000]
11330tigase2004066m468m10mS1.912.28:34.64java   1.9 12.2    8:34.64 java    "ResultsListener-socketWriteThread-5" prio=10 tid=0x00007fade02af800 nid=0x2c42 waiting on condition [0x00007fadb0747000]
11340tigase2004066m468m10mS1.912.28:37.39java   1.9 12.2    8:37.39 java    "ResultsListener-socketWriteThread-10" prio=10 tid=0x00007fade0341800 nid=0x2c4c waiting on condition [0x00007fadafd3d000]
11344tigase2004066m468m10mS1.912.28:38.48java   1.9 12.2    8:38.48 java    "ResultsListener-socketWriteThread-12" prio=10 tid=0x00007fade037b800 nid=0x2c50 waiting on condition [0x00007fadaf939000]
11346tigase2004066m468m10mS1.912.28:36.70java   1.9 12.2    8:36.70 java    "ResultsListener-socketWriteThread-13" prio=10 tid=0x00007fade0399000 nid=0x2c52 waiting on condition [0x00007fadaf737000]
11348tigase2004066m468m10mS1.912.28:35.21java   1.9 12.2    8:35.21 java    "ResultsListener-socketWriteThread-14" prio=10 tid=0x00007fade03b6000 nid=0x2c54 waiting on condition [0x00007fadaf535000]
11352tigase2004066m468m10mS1.912.28:33.69java   1.9 12.2    8:33.69 java    "ResultsListener-socketWriteThread-16" prio=10 tid=0x00007fade03f0000 nid=0x2c58 waiting on condition [0x00007fadaf131000

Therefore, I see first of all, SocketReadThread-5 was destructed then 7 and after that 6.


init.properties (1.99 KB) init.properties config file-Aug 8
tigase monitor snapshot Aug 8-Tigase b3810.PNG (52.4 KB) tigase monitor snapshot Aug 8-Tigase b3810.PNG Snap shot Aug 8
Errors.txt (21 KB) Errors.txt Jstack, top and tigase.log.0 outputs
Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

A few unanswered questions:

  1. Which Tigase version do you use?

  2. What database do you use?

From the monitor screenshot it looks like there are over 70 connections to the server. Still very low number which should not cause any load.

A few more questions:

  1. Are you running Tigase on a VM?

  2. What are parameters of the VM?

  3. Can you also share tigase.conf file?

Added by Ehsan Mozaffari almost 4 years ago

Thank you again for the reply.

Regarding your last question:

I attached tigase.conf

Tigase-server-7.0.1-b3810-dist-max and mysql 5.1.

The tigase is running on a CentOS 6.5 guest on top of an ESxi and the allocated resources are: 4GB RAM - 4 total core numbers( just to add I've also tested on real server and I got the same errors so it seems the problem is not related to VM)

tigase.conf (722 Bytes) tigase.conf

    (1-4/4)