Project

General

Profile

Hm, receiver is not set yet - On TCP Connections

Subir Jolly
Added almost 5 years ago

Guys,

We are running bosh as well as tcp tests using tsung. We have been communicating with you guys about other issues under: https://projects.tigase.org/boards/4/topics/1293

But in this case, we are running tcp tsung tests on port 5222. After logging in a couple thousand users using tsung tests, we start getting a lotttt of these messages:

tigase.log.2:2014-02-11 12:38:38.843 [pool-11-thread-66] ClientConnectionManager.processSocketData()  INFO: Hm, receiver is not set yet (misconfiguration error)..., ignoring: from=c2s@app2.wa-chat.ord1s.rsapps.net/192.168.64.98_5222_192.168.64.100_42422, to=null, DATA=<message xmlns="jabber:client" id="35463" to="test319@testdomain22.com" type="chat"><body>CData size: 72</body></message>, SIZE=179, XMLNS=jabber:client, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=chat, connection: c2s@app2.wa-chat.ord1s.rsapps.net/192.168.64.98_5222_192.168.64.100_42422, type: accept, Socket: c2s@app2.wa-chat.ord1s.rsapps.net/192.168.64.98_5222_192.168.64.100_42422 Socket[addr=/192.168.64.100,port=42422,localport=5222], jid: null

Please tell me what I can do to fix this. Or is this even an issue?

Thanks,

Subir


Replies (6)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

This happens when a client (Tsung) sends XMPP data (message, presence, or anything) before sending stream open data. From our experience this usually happens when Tsung is overloaded.

Added by Subir Jolly almost 5 years ago

Could it also be possible if tsung creates multiple connections using the same port from the same ip? Tigase overwrites the previous connections and the previous users get disconnected. Could it be because of that?

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

Is it possible at all? I do not know how this can be done. Once a connection for some IP, PORT is crested the OS marks that port as used so it cannot be used to open a new connection.

Added by Subir Jolly almost 5 years ago

I am not sure. I was just guessing.

Added by Sean Drummond almost 5 years ago

While doing more tests with xmpp I was able to get 33,600 users online all sending messages (each user had a buddy list of 5) all on one tigase server. This was accomplished by having 48 tsung servers with logins approximately 1 / second total. When trying to scale out a little is when I started getting the "Hm, receiver is not set yet...". By scale out I mean increase logins to more than once per second or by adding a second server.

So logins less than once per second I have no issues.

If logins are more than once per second I get that error at the following thresholds:

1) With 1 server at 1000 users logged in

2) With 2 servers clustered 1700 users logged in

Once the server has 4000 users logged in tigase stops responding and I have to restart.

Although the issue may lie with tsung why would tigase lockup and stop responding to all requests?

If we leave the test running, when we get to about 6000 connections we get the following error as well

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

Sean Drummond wrote:

Once the server has 4000 users logged in tigase stops responding and I have to restart.

Although the issue may lie with tsung why would tigase lockup and stop responding to all requests?

Indeed. But it is hard to tell why, what without having look at it. Because even "stop responding" can mean many things. Is it possible to open a connection to the Tigase? Any kind of connection: Bosh, standard XMPP? If yes, we can dig deeper and check what part is locked and why. Maybe internal queues are full and this "stop responding" really means it is very slow with responding (minutes). Of course such a slow response is still unacceptable but it gives us hints what needs to be fixed/corrected/improved.

If you cannot open a TCP/IP connection at all, it may mean some OS level resources are used up - file descriptors, ports, network buffers, etc.... So fix would be to adjust OS level settings.

Or maybe the server crashed with OOM? If this is the case maybe internal queues are too large and should be configured smaller.

If we leave the test running, when we get to about 6000 connections we get the following error as well

[...]

This particular error means that the authentication request came in too late and the user's session has been already destroyed. This is because once a connection to the server is opened Tigase creates a timeout for this connection within which the connection must be authenticated. If the login does not occur within the timeout, the session is destroyed and the connection is closed.

If the server is overloaded with requests and they are queued inside the Tigase, the user session can be destroyed and connection closed but the request to authenticate can be still in a queue. Once it gets for processing, it is too late.

One thing about the error message is, it is not for Bosh connection, it is for standard XMPP connection.

    (1-6/6)