Project

General

Profile

Logging in/out packets traffic

Mauro Carrio
Added almost 5 years ago

Hi, I'm facing the following issue: some mobile clients are losing messages stanzas randomly. We have implemented a custom "ack" answer from server when a message or subscription presence is handled by the server. and also have implemented Stream Management for outgoing messages server to clients with resume enabled. But in some cases on an active conversation (wifi connected stable network) some messages dont seem to be received by the clients. But really I can't still ensure where is the problem.

I was thinking about logging in a table database all in/out packets traffic of the three users most affected by this issue. How could be the best approach to achieve this? Or any other idea?

Thanks a lot.


Replies (13)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

Is your installation a very busy one? How many XMPP packets per second do you have on your system? Are you sure Tigase has enough resources to handle the load on your system?

I do not think logging traffic to DB is necessary. It could kill your database. My suggestion is to enable Tigase server debug logging and track these lost messages in the log file. You can also enable separate logs for a specific user - this can be done using ad-hoc command: "Activate log tracker for a user".

Added by Mauro Carrio almost 5 years ago

No it doesn't. We are now in development stage.

I'll try the adhoc command for this.

Thanks!

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

In such a case I suggest to enable extensive debug logging. Then you would have a trace of everything that happens to messages, how far they got, etc....

To be honest we have never experienced this kind of message loss either for a stand-alone clients or for mobile clients. Such a message loss does happen sometimes over a mobile network in certain cases but not over a wifi stable network.

If you have stream management extension implemented and enabled you should see in the log file whether the ACK is received, you know the whole data flow.

What kind of software do you use on the mobile client side?

Added by Mauro Carrio almost 5 years ago

First thanks for your suggestions.

How do I enable extensive debugging?

This is a part of my init.properties:

--user-db=mysql
--admins=admin@mydomain.com
--user-db-uri=jdbc:mysql://db001/db_tigase?user=xx&password=pass&useUnicode=true&characterEncoding=UTF-8
config-type=--gen-config-def
--auth-db-uri = jdbc:mysql://db001/db_tigase?user=xx&password=pass&useUnicode=true&characterEncoding=UTF-8
--user-repo-pool-size=12
--virt-hosts=mydomain.com
--debug=server,xmpp.XMPPIOService
--sm-plugins=gv-push-service,gv-message-ack,gv-presence-ack,gv-simple-presence-v2,-presence
--monitoring=jmx:9050,http:9080,snmp:9060

--tigase.cache=false
--stanza-ack=white-char,xmpp

c2s/processors[s]=urn:xmpp:sm:3

# LOGS
basic-conf/logging/java.util.logging.FileHandler.limit=100000000
basic-conf/logging/java.util.logging.FileHandler.count=50
#basic-conf/logging/java.util.logging.FileHandler.pattern=/var/log/tigase/tigase.log

The software used on mobile clients depends on the platform, on Android we're using asmack and on iOS a propietary library. And in both platforms we're experiencing lost messages and we've not a pattern to reproduce this issue yet.

A couple of questions:

1) In order to execute AddUserTracker adhoc command, should I use tclmt using admin user? Or maybe each user I want to activate log should execute the command through its xmpp session?

2) I've seen that in some cases using SASL plain authentication (with iq-auth it didnt happen), when a user logs in to Tigase having offline messages stored or sending from other user messages during login, sometimes not always Tigase delivers some of them after the autentication before binding the resource and it causes asmack to raise a runtime error. And of course this messages do not appear on screen. We are working on this from the client side but is this a correct behavior of the server?

Thanks again!

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

--debug=server

Should be usually good enough. You can see whether an XMPP packet (message or presence) gets to ClientConnectionManager. If there is any network level error it should be printed anyway. With the setting you should be able to see the message as it travels through the Tigase server form one component to another. It should be first received by ClientConnectionManager*, then it should get through MessageRouter, SessionManager, *Message plugin, then again MessageRouter, ClientConnectionManager, network and your client.

If you really want more extensive and low level logging then:

--debug=server,net,io

I doubt the above would be helpful unless this is TLS or zlib related. By the way, do you use zlib compression?

You know that we (Tigase) have an Android client as well. I wonder if you could install our client and try to reproduce the problem with lost messages.

ad 1) The user tracker is activated via ad-hoc command. Ad-hoc commands are admin stuff so you have to execute them from an admin account. You can do it from a regular XMPP client which supports service disco and ad-hoc commands (I recommend Psi) or our command line tool (which is just a simplified command line client by the way) tclmt. You could also try http://tigase.im which is our web client with support for admin ad-hoc commands as well.

ad 2) Sending offline messages to logged in user is triggered by the user's first available presence. Please check whether your client sends presence status before resource binding or maybe the client sends resource binding together with presence status and somehow response to resource bind arrives after presence was processed and offline messages are delivered? Again with the server level logging on the server you should be able to easily find out.

Added by Mauro Carrio almost 5 years ago

No, I'm not using zlib compression, should I?

Yes, I know about your Android library, in fact I use it to make some tests. We have scheduled xmpp library migration but that will not be done very soon.

So about the ad-hoc command (add-user-tracker), I'm not being able to execute from Psi, because I suppose Tigase is not exposing commands. And executing it from tclmt with admin account this is the error I'm getting:

$ ./bin/tclmt.sh -ip 10.10.10.10 -u admin@mydomain.com -p 1111 remote sess-man add-user-tracker

Feb 6, 2014 11:52:12 AM tigase.xml.db.XMLDB setupNewDB
INFO: Create empty DB.
awaiting response...
java.lang.NullPointerException: Cannot invoke method getType() on null object
        at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:77)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:45)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:32)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at Script1.run(Script1.groovy:52)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:315)
        at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:41)
        at tigase.tclmt.Script.execute(Script.java:94)
        at tigase.tclmt.CommandManager.executeScript(CommandManager.java:72)
        at tigase.tclmt.Tclmt.execute(Tclmt.java:178)
        at tigase.tclmt.Tclmt.main(Tclmt.java:226)

Added by Wojciech Kapcia TigaseTeam almost 5 years ago

If you connect as admin all commands should be exposed by default. As for the provided command - a user JID for which monitoring is to be enabled is missing.

Also please check if the admin ad-hoc scrips are loaded correctly (can be checked in the logs) and if they are listed in the TCLMT in the interactive mode.

Added by Mauro Carrio almost 5 years ago

I'm connected with admin account on Psi+. This is the xmpp response when it asks for commands:

<iq type="get" to="mydomain.com" id="aaf3a">
<query xmlns="http://jabber.org/protocol/disco#items" node="http://jabber.org/protocol/commands"/>
</iq>


<iq from="mydomain.com" type="result" id="aaf3a" to="admin@mydomain.com/Psi+">
<query xmlns="http://jabber.org/protocol/disco#items" node="http://jabber.org/protocol/commands"/>
</iq>

Added by Mauro Carrio almost 5 years ago

Yes, the scripts were loaded succesfully. But listing them with tclmt interactive mode, it doesnt show the add-user-tracker command:

# ./bin/tclmt.sh -ip XX.XX.XX.XX -u admin@mydomain.com -p 1111 list
Feb 6, 2014 1:43:14 PM tigase.xml.db.XMLDB setupNewDB
INFO: Create empty DB.
Available commands:
        add-script                Adds script                        
        add-user                  Adds user                          
        announce                  Sends announcement to online users 
        change-user-password      Changes user password              
        comp-repo-item-add        Adds component repository item     
        comp-repo-item-remove     Removes component repository item  
        comp-repo-item-update     Updates component repository item  
        comp-repo-reload          Reloads component repository items 
        config-list               Lists configuration of component   
        connection-time           Gets maximum and avarage connection time for all connected users
        del-script                Remove script                      
        delete-user               Deletes user                       
        example-python            Lists remote commands (Python)     
        get-active-users-list     Gets list of active users for domain
        get-idle-users-list       Gets list of idle users for domain 
        get-online-users-list     Gets list of online users for domain
        get-registered-user-list  Gets list of registered users for domain
        get-user-roster           Gets user roster                   
        help                      Display help                       
        list                      Lists commands                     
        list-remote               Lists remote commands              
        load-errors               Gets load errors catched by the server during execution
        remote                    Executes basic remote commands     
        ssl-certificate-add       Adds SSL certificate for domain    
        user-stats                Gets user statistics               

In TCLMT script folder there is no AddUserTracker.groovy file, may be this is the problem. I ckecked in script folder on Tigase installation and the file exists.

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

What version of Tigase do you use? Have you built it on your own or used a binary installer?

Added by Wojciech Kapcia TigaseTeam almost 5 years ago

Try querying particular component (for user logger it would be @sess-man@), i.e.:

<iq type="get" to="sess-man@mydomain.com" id="get-1">
<query xmlns="http://jabber.org/protocol/disco#items" node="http://jabber.org/protocol/commands"/>
</iq>

For the TCLMT you should use command list-remote (as it is remote command) and desired component (in this case again @sess-man@), i.e.:

$ ./bin/tclmt.sh -ip XX.XX.XX.XX -u admin@mydomain.com -p 1111 list-remote sess-man

Added by Mauro Carrio almost 5 years ago

Artur, the version is Tigase-5.2.0-RC1. I've build on my own, and running over Fedora 15.

Wojciech, I got no response from the server querying the component sess-man. And executing list-remote from TCLMT I got this:

./bin/tclmt.sh -ip XX.XX.XX.XX -u admin@mydomain.com -p 1111 list-remote sess-man
Feb 7, 2014 1:27:29 PM tigase.xml.db.XMLDB setupNewDB
INFO: Create empty DB.
awaiting response...
Error executing script list-remote
        java.lang.NullPointerException: Cannot invoke method getChildrenNS() on null object

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 5 years ago

Please try the most recent version of Tigase and the command line tool. Also, if you build binaries on your own, make sure you also copy all the admin scripts to the installation directory from the source code repo. If you run Tigase from within your source code repository, you should know that admin scripts are not located in the place where the running Tigase expects them to be.

The same applies to the command line tool.

    (1-13/13)