Cluster mode in 7.0.4
I seem to have a weird issue. I have setup a cluster with two nodes and ACS. Using mysql as database.
The machines connect to each other on port 5277, but they don't send data at all between each other. I tried a tcpdump and there was no data between them. I go to the cluster_nodes in the DB there are both there. And it seems it using the db to notify each other about there presence.
But If I do a broadcast message through the API, the message are only received by the clients on the node that I called with the rest api, but the other node doesn't received it. But if one of my users do something in the client like log off and log on, or sends a message the message suddenly appears. If I look in the msg_history table, they look like offline messages.
I have another cluster with the exact same setup, and it is working perfectly. They are using the same init.properties except using another db.
Have you guys seen this?
Added by Artur Hefczyc about 2 years ago
It certainly unusual behaviour. It is hard to tell what is going on without deeper investigation. I would suspect some kind of network configuration issue, like, for example, incorrect hostname settings on on or both cluster nodes or maybe DNS misconfiguration.
Added by Tue Topholm about 2 years ago
There is no DNS issues, all machines can resolve each other.
I have one of the nodes working and while I was debugging I saw this log on the node that doesn't work:
2017-03-06 15:46:38.641 [in_3-sess-man] OnlineUsersCachingStrategy.getNodesForPacketForward() FINEST: Packet not suitable for forwarding: firstname.lastname@example.org, to=null, DATA=<message from="yousee.tv" type="chat" xmlns="jabber:client" id="1bf3fe68-1532-47aa-b1ac-1aaa0cf9f872" to="email@example.com"><body>sodavand</body><amp xmlns="http://jabber.org/protocol/amp"><rule/></amp></message>, SIZE=217, XMLNS=jabber:client, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=chat
And this one is from node that can send message to both:
2017-03-06 15:47:43.197 [in_3-sess-man] OnlineUsersCachingStrategy.getNodesForPacketForward() FINEST: Selected nodes: [firstname.lastname@example.org], for packet: email@example.com, to=null, DATA=<message to="firstname.lastname@example.org" id="35f5c900-fb3d-4cda-83e5-56f2d1f232b0" from="rest.gaijin.dk" type="chat" xmlns="jabber:client"><body>sodavand</body><amp xmlns="http://jabber.org/protocol/amp"><rule/></amp></message>, SIZE=222, XMLNS=jabber:client, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=chat
Added by Artur Hefczyc about 2 years ago
It's really hard to tell without investigation.
Even if machines can resolve each other that does not mean all the DNS configuration is correct. The Tigase clustering offers auto-configuration feature, so when you bring a new cluster node to the system you do not need to change any config files. It is all automatic, however, for this to work all the settings must be correctly.
If one node can send messages to users on both nodes and the other node cannot send messages to the other node, this, even more, makes me think that some DNS settings might be off.
Added by Wojciech Kapcia about 2 years ago
Tue Topholm wrote:
The machines connect to each other on port 5277, but they don't send data at all between each other.
How do you verify they are connected? Using netstat or somesuch and monitor opened sockets?
Are the number of opened and established connections the same? Do the IPs and ports match?
I tried a tcpdump and there was no data between them. I go to the cluster_nodes in the DB there are both there. And it seems it using the db to notify each other about there presence.
While the database is used to propagate information about cluster node (automatic cluster discovery) it's not used for presence broadcast.
Can you log in to each node using XMPP client and in the Service discovery check the Cluster component for the list of connected nodes?
Or you can navigate to: http://node_hostname:8080/admin/ and then
Get list of all connected cluster nodes and check that both nodes are listed there?