Project

General

Profile

Asterisk core dump using Tigase XMPP server

Carol Dupont
Added about 5 years ago

Hi,

We have successfully deployed an installation of 18 Asterisk 11.4 servers (about 300 sip peers monitored) in combination of a central Tigase XMPP server (version 5.1.5). Things seems to works fine for a a dozen hours but this morning we encountered a crash of 6 of those Asterisk servers at the same time. A few seconds prior to this, we found out that Tigase had a NullPointerException. I guess Asterisk servers should have been able to protect themselves against this error but we are looking at every possible cause to this problem. So far, it doesnt look like this is a known issue on Asterisk side although the xmpp module got some fixes about xmpp since 11.4 (Current Asterisk version is 11.7). Core dumps of the Asterisk servers seems to point out that there was a xml parse error that made them crash. It look like they got an incomplete message that cause them to failed...

Below is the Tigase exception we caught. Anyone can provide any insight on what may has caused it? Seems that TYPE=null is something that should not have happened. Could this exception have killed a Tigase thread causing it to send incomplete messages to the Asterisk servers making them crash? We will probably upgrade to Asterisk 11.7 and Tigase 5.2 in case this could prevent further crashes...

Let me know if you need more info, config files, etc. Thank you!

2014-02-13 11:30:02  WorkerThread.run()                  SEVERE:   tigase.server.xmppsession.SessionManager$ProcessorWorkerThread,(pep Queue Worker 0) Exception during packet processing: from=c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43250, to=sess-man@pbxip-4507461000a.local.local, DATA=<pubsub from="qc@xmpp.abc.idsvoip.com/qc" xmlns="http://jabber.org/protocol/pubsub"><publish node="message_waiting"><item id="3731@default"><mailbox eid="2c:59:e5:9a:67:50" xmlns="http://asterisk.org"><NEWMSGS>0</NEWMSGS><OLDMSGS>0</OLDMSGS></mailbox></item></publish></pubsub>, SIZE=287, XMLNS=http://jabber.org/protocol/pubsub, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=null
java.lang.NullPointerException
    at tigase.xmpp.impl.PepPlugin.processFromUserToServerPacket(PepPlugin.java)
    at tigase.xmpp.XMPPProcessorAbstract.processFromUserPacket(XMPPProcessorAbstract.java)
    at tigase.xmpp.XMPPProcessorAbstract.process(XMPPProcessorAbstract.java)
    at tigase.server.xmppsession.SessionManager$ProcessorWorkerThread.process(SessionManager.java)
    at tigase.util.WorkerThread.run(WorkerThread.java)
2014-02-13 11:30:13  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:13  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.1.91.10_39626, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.1.91.10_39626 Socket[addr=/10.1.91.10,port=39626,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:17  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:17  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43332, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43332 Socket[addr=/192.168.5.10,port=43332,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:28  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:28  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.1.91.10_39646, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.1.91.10_39646 Socket[addr=/10.1.91.10,port=39646,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:44  WorkerThread.run()                  SEVERE:   tigase.server.xmppsession.SessionManager$ProcessorWorkerThread,(pep Queue Worker 0) Exception during packet processing: from=c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43432, to=sess-man@pbxip-4507461000a.local.local, DATA=<pubsub from="qc@xmpp.abc.idsvoip.com/qc" xmlns="http://jabber.org/protocol/pubsub"><publish node="message_waiting"><item id="4804@default"><mailbox eid="2c:59:e5:9a:67:50" xmlns="http://asterisk.org"><NEWMSGS>0</NEWMSGS><OLDMSGS>0</OLDMSGS></mailbox></item></publish></pubsub>, SIZE=287, XMLNS=http://jabber.org/protocol/pubsub, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=null
java.lang.NullPointerException
    at tigase.xmpp.impl.PepPlugin.processFromUserToServerPacket(PepPlugin.java)
    at tigase.xmpp.XMPPProcessorAbstract.processFromUserPacket(XMPPProcessorAbstract.java)
    at tigase.xmpp.XMPPProcessorAbstract.process(XMPPProcessorAbstract.java)
    at tigase.server.xmppsession.SessionManager$ProcessorWorkerThread.process(SessionManager.java)
    at tigase.util.WorkerThread.run(WorkerThread.java)
2014-02-13 11:30:44  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:44  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43432, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_192.168.5.10_43432 Socket[addr=/192.168.5.10,port=43432,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:44  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:44  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39440, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39440 Socket[addr=/10.6.91.10,port=39440,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:49  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:49  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39451, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39451 Socket[addr=/10.6.91.10,port=39451,localport=5222], data parsing error, stopping connection
2014-02-13 11:30:52  XMPPDomBuilderHandler.error()       WARNING:  XML content parse error.
2014-02-13 11:30:52  XMPPIOService.processSocketData()   WARNING:  c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39452, type: accept, Socket: c2s@pbxip-4507461000a.local.local/192.168.5.10_5222_10.6.91.10_39452 Socket[addr=/10.6.91.10,port=39452,localport=5222], data parsing error, stopping connection


Replies (3)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam about 5 years ago

The NPE is something we will look at and fix, here is the ticket for your reference: #1728. However, the only side effect of that happening is a missing/lost, unprocessed packet.

The packet processing logic is separate to network code, therefore it is impossible for Tigase to send part of a message. Sending part of a message could happen due to network problems and connection lost during transfer but this is highly unlikely.

However, you also report XML parse errors in Asterisk, if you look at the Tigase logs you provided above, they also report XML parser error and disconnection. Looks like this happens a lot. From our experience there are 2 possible causes of XML content parse errors:

  1. Something was misconfigured and connects to Tigase (and Asterisk) ports but it does not "speak" either XMPP or SIP and sends some data which is garbage for your services.

  2. Somebody scans your ports or even attacks your services and attempts to connect using different protocols.

For Tigase this is not a big problem as it contains logic handling it nicely.

Upgrading Tigase to 5.2.0 does make sense as it was greatly improved, however, from the information you provided I do not see anything which suggest you must do the upgrade in rush. Looks like the current version of Tigase handles the case OK.

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam about 5 years ago

Andrzej - the other Tigase team member just spotted the PubSub packet processed by Tigase and causing NPE is incorrect. One thing is that Tigase 5.2.0 has better data checking and this packet would be ignored and not caused NPE, another thing is, you should investigate where this incorrect packet was generated and sent to Tigase server.

Added by Carol Dupont about 5 years ago

Thank you for your quick replies Artur,

We will raise the issue with Asterisk. We experienced another crash during the night on multiple servers at the same time. We dont believe in an attack on our services as all servers are on private network. But there might be a configuration error in Asterisk that is causing garbage beeing sent. What we have discovered since first posting this here is that a lot of the Asterisk servers are giving us this warning : res_xmpp.c: JABBER: socket read error. So it might be connected to our issue.

Still we deployed the same setup on a much smaller scale (only 3 asterisk servers) on a public network and we havent seen a single problem with this installation. We have turned on XMPP debugging in Asterisk in hope that we catch whatever is causing the failures. We will also upgrade Tigase and Asterisk servers as soon as possible and hope the issue goes away.

Thanks!

    (1-3/3)