Project

General

Profile

sess-man queue size is to high

Igor Khomenko
Added almost 4 years ago

Hi there,

we run some load tests for Tigase. It's simple test which sends 1500K 1-1 messages per second.

After some time we see that sess-man component queue size is to high and increasing.

Is there some recommendations which parameters we can tune and check to decrease this bottle neck?

thanks


Replies (9)

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

Best option would be to check which (if any) plugin causes the bottleneck - in the session manager statistics you have list of all plugins and number of statistics related to processing.

Added by Igor Khomenko almost 4 years ago

Should I see something like this?

sess-man/Processor: amp=, Queue: 5359, AvTime: 28, Runs: 1062830, Lost: 73910,

What is Queue,AvTime,Runs, Lost?

Added by Igor Khomenko almost 4 years ago

So, the main issues is with IN queue (NORMAL)

sess-man/In queue wait: NORMAL=44490,

Out queue is empty

Added by Igor Khomenko almost 4 years ago

So, I see that only amp plugin queues messages, but it's queue is much less than sess-man/In queue wait

Any suggestions?

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

Igor:

  1. What version of the Tigase XMPP Server are you testing?

  2. What DB do you use?

  3. Is it a clustered setup or a single mode installation?

  4. By 1500k/sec do you really mean 1.5k/sec or 1.5M/sec?

The metrics you are asking about mean:

  1. Queue - current size of the plugins queue, that is a number of currently waiting packets in the queue

  2. AvTime - is an average processing time of a single packet in this plugin in ms

  3. Runs - basically how many packets the plugin already processed from the server startup time

  4. Lost - Each queue has maximum size to prevent OOM. Once the queue is full the plugin (or component) drops any new incoming packets which do not fit into the queue. This counter shows how many packets were dropped (lost)

Tigase version 5.2.x had a code causing performance issues in SM and some plugins. Looks like you might be affected. Unless you use 7.0.0, please rerun your tests with the most recent version of the Tigase server. The fix is relatively simple, so it could be back ported if it hasn't been yet.

Added by Igor Khomenko almost 4 years ago

We use Tigase 5.2.0, MySQL db, single mode installation.

We test 1500 messages per second, we have Tsung script which produces 1500 messages per second

According to your information, looks like the amp plugin is a bottle neck because sometimes I see about 1000 messages in a Queue and Lost has a big value

Do you think it could be an issue with Tigase 5.2.0 ? Because I see that you fixed an issue with amp plugin in 5.2.3 http://www.tigase.net/blog-entry/tigase-xmpp-server-523-release

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

As Artur already said - this is very possible.

Added by Igor Khomenko almost 4 years ago

I went through this topic https://projects.tigase.org/boards/15/topics/3104 and disabled all logs first

so, it works now much faster, will continue our tests, thanks guys

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

Yes, disabling all logs will improve performance significantly, that's for sure. However, if you are using the version 5.2.0 you are affected by the performance problem described above. I suggest to update to at least 5.2.3 version if possible.

    (1-9/9)