Project

General

Profile

WARNING: [[cl-comp]] XMPP Limits exceeded on connection null...

kellogs .
Added almost 5 years ago

Hi,

On a two clusters installation, on one cluster I get loads of:

WARNING: [[cl-comp]] XMPP Limits exceeded on connection null, type: accept, Socket: nullSocket[addr=/192.168.101.250,port=54170,localport=5277], jid: null stopping, packets dropped: [from=null, to=null, DATA=<cluster to="sess-man@server1.domain.com" xmlns="tigase:cluster" type="set" from="sess-man@server1.domain.com" id="cl-9331"><control><visited-nodes><node-id>sess-man@server1.domain.com</node-id><node-id>sess-man@server2.domain.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key" [...]

This is a WARNING message, so according to https://projects.tigase.org/issues/1070 we are running the fixed tigase post-4ccaf048e8040924694ae35756108d9eef772900

On the other server in our cluster, there are loads of:

2014-05-26 23:03:06.164 [in_50-cl-comp]    ClusterConnectionManager.writePacketToSocket()  WARNING: No cluster connection to send a packet: from=null, to=null, DATA=<cluster from="sess-man@server2.domain.com" to="sess-man@server1.domain.com" id="cl-42891" xmlns="tigase:cluster" type="set"><control><visited-nodes><node-id>sess-man@server2.domain.com</node-id></visited-nodes><method-call name="packet-forward-sm-cmd"><par name="user-session-found-key">sess-man@server2.domain.com</par></method-call><first-node>sess-man@server2.domain.com</first-node></control><data><presence from="+391838@node.com/tsung" to="+391857@node.com" id="4984" xmlns="jabber:client" type="unavailable"/></data></cluster>, SIZE=557, XMLNS=tigase:cluster, PRIORITY=CLUSTER, PERMISSION=NONE, TYPE=set

the load we are tsunging with is some 50 users / second each making 4 regular sized requests and one 3 kB , with 10 seconds in between the 5 stanzas. The logs litter very early into the test.

Do we need to set higher traffic limits ? How, if so ?

Thank you!


Replies (7)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

What version of the Tigase XMPP Server do you use?

For the first problem, there was indeed a time when the limits for cluster connections were set too low and this caused problems you describe.

The second problem indicates cluster and/or network level misconfiguration on your service.

Added by Wojciech Kapcia TigaseTeam almost 5 years ago

kellogs . wrote:

Do we need to set higher traffic limits ? How, if so ?

Well, as Artur pointed out, for some time, you shouldn't need to worry about adjusting the setting for hi-load services (clustering, external component connections, etc) but if you need then you can use --cm-ht-traffic-throttling setting.

Added by kellogs . almost 5 years ago

first problem seems solved with the --cm-ht-traffic-throttling setting, thanks!

second one, uhm... just wanted to clarify if it is network / OS related or tigase related.

Restarted test, after 30 - 60 secs it started spitting those entries in logs. Stopped test, the entries were pouring through nevertheless. Stopped the "tail -f logs/tigase-console.log" command and relaunched it. In one server the entries were still pouring while on the other they came to a stop. So, it looks as if one of the clusters was stuck in a loop.

Restarted test again but changed user count to 20 from 200k. Problem did not manifest. Then upped to 200 users and it did not manifest again.

Version is customized tigase based on

commit 48635d0a30a0a28e062b8da33a3d6f8453418866

Author: Wojciech Kapcia wojciech.kapcia@tigase.org

Date: Wed Feb 12 17:28:44 2014 +0100

tigase-server-5.2.0 release

Any pointers are welcome!

Added by kellogs . almost 5 years ago

Hmm.. seems not OS related. I can telnet just fine onto port 5277 of both cluster machines during those "seizures"

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

It's not just about being able to telnet. Most important is DNS configuration for both cluster nodes and make sure they can and they do connect to each other. Also make sure you do not use the cluster node hostname as a vhost.

Added by kellogs . almost 5 years ago

kellogs . wrote:

Restarted test again but changed user count to 20 from 200k. Problem did not manifest. Then upped to 200 users and it did not manifest again.

Only seems high volume related

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

Maybe you are right. The error message:

WARNING: No cluster connection to send a packet: from=null, to=null, DATA=<cluster from="sess-man@server2.domain.com" to="sess-man@server1.domain.com"

So it looks like there is no connection between cluster nodes. There are 2 possibilities I can see right now:

  1. There is no connection between cluster nodes because it was never there, so simply, there is no connectivity between cluster nodes (however, I am not certain if one cluster node would even generate a packet to another cluster node if there was never connection between them)

  2. There was a connection between cluster nodes but it was lost/disconnected/broken, maybe this is related to a load, maybe you have some software, firewall, selinux or something like this activated which detects excessive load and considers it a DOS attack and blocks traffic?

    (1-7/7)