Project

General

Profile

Too many SM packets per second

Stanislav Ruzani
Added over 5 years ago

Hi,

I have just checked statistics on our Tigase server instance, and I found something strange - SMPacketsNumberPerSec was too high, about 250-300 packets per second.

This is very strange, as we are using server only for presence notifications.

Normally we have about 2000 connections at the same time, and we get about 2 presences per second and about 1-2 messages per second. Unfortunately the SMPacketsNumberPerSec is very high and it seems our server needs huge amount of cpu.

Do you have any idea what may be wrong? What does the SMPacketsNumberPerSec mean exactly?

Thanks a lot!


Replies (5)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 5 years ago

Tigase on a standard server class machine (current XEON CPU) can handle at least 10k XMPP packets per second per CPU core. At 250 - 300 packet per second you really do not need lots of CPU. At such a traffic the CPU should be used way below 5%.

On the other hand, you are right, the traffic on your installation is much higher than what we would expect at this number of users connections. There might be many reasons for this. To investigate the problem, you have to analyze Tigase metrics to answer following questions:

  1. How much CPU is Tigase using on your machine? (There is a metric for this)

  2. If the CPU is high, what threads are using the CPU the most?

  3. What kind of packets are the majority of the traffic? (Presences? Messages?, IQs?)

  4. Is the high traffic all the time, or there are spikes?

The * SMPacketsNumberPerSec* means only how many XMPP packets were processed in the last second. It does not average over longer time. So it may show 0 in one second and 1000 in another. A better indication of the traffic over longer period of time would be to look at "packets per minute"/60 or better yet "packets per hour"/3600. This would give you an average traffic per second in last minute or last hour. Constant monitoring might provide you a better insight into the server.

Added by Stanislav Ruzani over 5 years ago

We have quite powerful machine (8 physical CPU's, 64Gig of RAM), this is not a problem. These numbers also stay quite stable for longer time, so filtering is not an issue. The issue is that I don't understand this:

PresencesNumberPerSec float RO 1.3340657

SM presences rec Tot: 3043851 / 0 last sec

SM presences sent Tot: 6111432 / 1 last sec

SMPacketsNumberPerSec float RO 175.01642

So we have 1-2 messages per second, 0 or 1 presences per second.. And 175 SM Packets per second? Where does it come from? What other XMPP traffic this could be?

Our application doesn't use anything else - we are just sending presences when the app connects to the server and then once in a while (every 5-10 minutes) one or two messages. What is then the remaining traffic? I mean - at least 170 messages are "unknown" we are not using any other XMPP features, it is used really only for signaling.

I have also activated server + net debugs, this creates quite a lot of logging data, could this slow things down?

Thank you for your support!

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 5 years ago

How do you get the metrics above? Where do they come from? I am just looking at all our metrics in the Tigase server and I do not see any of them except the packets per second.

If you look at the Tigase statistics for Session Manager, at the highest level you should be able to find metrics with IN_QUEUE in name. These are counters for each packet type which goes through the Session Manager. If you take these every minute and compare results you should have a clear picture what kind of packets generate traffic.

Added by Stanislav Ruzani over 5 years ago

Hi,

the metrics are taken from tigase' HTTP Server statistics page (which is running on standard port 9080), item tigase.stats/type=StatisticsProvider.

In the meantime I disabled network debugging and the cpu load went significantly down. It still doesn't explain the large amount of those "SMPackets", but as long as the server load is low, it is okay for us.

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 5 years ago

Yes, the logging slows down the server a lot.

I am not sure how many metrics are available via HTTP, I suggest to use either JMX or XMPP to load all Tigase statistics. If the mentioned earlier IN_QUEUE metrics are showing via HTTP, then it might be enough. But also via JMX or XMPP you could automate statisgics gathering and you could run some calculations to see what kind of packets cause the traffic.

    (1-5/5)