Project

General

Profile

Offline message and user db

Steffen Larsen
Added over 4 years ago

Hi Guys,

As you might know I am using AMP for offline message storage and delivery.

I am also using my own authentication mechanism, which is just a simple token based auth. login which is implementing AuthRepository.

--user-db=mysql
--user-db-uri=jdbc:mysql://xxx:3306/tigase?user=tigase&password=xxx&useUnicode=true&characterEncoding=UTF-8&autoCreateUser=true
--auth-db=com.mypackage.TokenAuthentication

I am getting this in my log, which seems like the user can't be found in the db. But why does it have to exist?

Is it because it have to compare when the user was online for the last time for AMP messages?

2014-07-15 08:56:07.222 [presence Queue Worker 6]  UserRepositoryMDImpl.getData()  WARNING: Couldn't obtain user repository for domain: xxx, not even default one!
2014-07-15 08:56:07.224 [amp Queue Worker 2]  MessageAmp.process()            INFO:     Something wrong, DB problem, cannot load offline messages. tigase.db.UserNotFoundException: User: 400621339@xxx/mnUwZzjcuObjGNVPE6/EsTdFmt4= was not found in database.
2014-07-15 08:56:43.375 [hostnames]        UserRepositoryMDImpl.getData()     WARNING:  Couldn't obtain user repository for domain: vhost-manager, not even default one!

Do I have to implement UserRepository also in my TokenAuth? I thought it just used the default one, when non was specified in the init.properties (I am only setting auth-db).

Cheers and thanks in advance, guys! :-)

/Steffen


Replies (14)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

The first and third line of the attached log excerpt means Tigase could not connect to the database when it was starting up. You should have some errors and/or exceptions early in the log file after the server started up. The second line is most likely a consequence that Tigase cannot connect to the database. Maybe you have a mistake in your DB connection string, or maybe the tigase user does not have permission to connect to the DB?

Added by Steffen Larsen over 4 years ago

I have a hard time believing that the server can't reach the db, because I can see that the cluster nodes are stamping in (cluster_nodes are updated (last_update field).

But I'll check again.

But I believe that using --auth-db should nothing to do with the user repository, right? when nothing is defined it should use the default one (mysq, with the give uri). right?

/Steffen

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Steffen Larsen wrote:

I have a hard time believing that the server can't reach the db, because I can see that the cluster nodes are stamping in (cluster_nodes are updated (last_update field).

But I'll check again.

Hm, just thinking... Tigase tries to open a number of connections - db connections pool, maybe some of the objects maintaining connection was not initialed correctly? Let's say the Tigase attempts to open 10 connections but it can only open 5?

But I believe that using --auth-db should nothing to do with the user repository, right? when nothing is defined it should use the default one (mysq, with the give uri). right?

yes, this is correct.

Added by Steffen Larsen over 4 years ago

Could be, can I check it via JMX or other monitoring?.. or maybe on the mysql server?

I'll go through all of the logs tomorrow.. and see if there is any suspicious entries.

/Steffen

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

You can probably check it on mysql server, unfortunately I do not know how. Wojciech would know but he is on holiday, will be back in 2 weeks.

Added by Steffen Larsen over 4 years ago

Ok Thanks Artur,

I think that the easiest way is to check the processes via. the SQL: "SHOW FULL PROCESSLIST"

This gives me the hosts the query towards the database and its queries + time spend.

/Steffen

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Of course Tigase server logs provide details on what exactly went wrong.

Added by Steffen Larsen over 4 years ago

Jesus, this is somewhat embarrassing!

It seems like when we have done an incorrect export of SQL (when porting to a new machine). The stored procedures was not correctly imported into the new server instance :-P

Sometime the answer is just much more simpler than the one you actually are chasing.. Typically.

Thanks for keeping up with me and my mumbling. :-)

/Steffen

Added by Steffen Larsen over 4 years ago

I am sill having problems with offline storage (AMP). Ive looked through the db and can't find no init errors or anything.

I get this in the log:

2014-07-28 13:57:59.003 [amp Queue Worker 0]  MessageAmp.process()            INFO:     Something wrong, DB problem, cannot load offline messages. tigase.db.UserNotFoundException: User: 100664232@txxxx/9+CHxiBABuFk64zPdvUG76qEHF0= was not found in database.

Is it because that the AMP module does not support delivering to direct resources or?

I can see that the user 100664232 is in the db (tig_users.user_id)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

AMP has a separate set of tables. A dedicated, optimized DB schema to store large number of messages. So the UserNotFoundException from AMP's MsgRepository really means, there are no messages waiting for the user and there was never anything stored to offline storage for this user.

I think we should change handling this case and instead of returning an exception, the method should just return an empty set.

Added by Steffen Larsen over 4 years ago

Hi Artur,

I found it in the code and made a fix for that, because it was polluting my log files.

BUT, I simply can't make the offline AMP messages work. I check the tigase-console, and there should be no problem with the database connection.

I've been testing different scenarios:

Normal (no AMP)
<message xmlns="jabber:client" type="chat" to="testtest@braintrust.dk" from="braintrust.dk"><body>kjh kjhk j</body></message>

AMP (no resource, before expiry) - WORKS
<message xmlns="jabber:client" type="chat" to="testtest@braintrust.dk" from="braintrust.dk"><body>test</body><amp xmlns="http://jabber.org/protocol/amp"><rule match-resource="exact" action="drop" value="2014-07-21T02:56:15Z" condition="expire-at"/></amp></message>

AMP (no resource, after expiry) - WORKS when online
<message xmlns="jabber:client" type="chat" to="testtest@braintrust.dk" from="braintrust.dk"><body>Offline test</body><amp xmlns="http://jabber.org/protocol/amp"><rule match-resource="exact" action="drop" value="2014-08-21T02:56:15Z" condition="expire-at"/></amp></message>

AMP (no resource, after expiry) - DOES NOT WORK when offline
<message xmlns="jabber:client" type="chat" to="testtest@braintrust.dk" from="braintrust.dk"><body>offline tests again</body><amp xmlns="http://jabber.org/protocol/amp"><rule match-resource="exact" action="drop" value="2014-08-21T02:56:15Z" condition="expire-at"/></amp></message>

AMP (with resource, after expiry) - DOES NOT WORK when offline
<message xmlns="jabber:client" type="chat" to="testtest@braintrust.dk/testtest" from="braintrust.dk"><body>offline test.. to resource</body><amp xmlns="http://jabber.org/protocol/amp"><rule match-resource="exact" action="drop" value="2014-08-21T02:56:15Z" condition="expire-at"/></amp></message>

I'd expect that hone the testiest user is offline, that the offline message would be stored in the database (msg_history table???), but nothing is there.

Another question I have, would AMP and broadcast work together? I have implemented broadcast with AMP in my groovy script and hopefully that would work as well, when I get this "small" bug fixed. :-)

-Cheers!

/Steffen

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Steffen Larsen wrote:

Steffen,

I found it in the code and made a fix for that, because it was polluting my log files.

BUT, I simply can't make the offline AMP messages work. I check the tigase-console, and there should be no problem with the database connection.

I've been testing different scenarios:

[...]

I'd expect that hone the testiest user is offline, that the offline message would be stored in the database (msg_history table???), but nothing is there.

I need more details here as I do not fully understand what exactly is wrong. You gave us some examples which work and some which don't. However, 'works', 'does not work' can have different meaning depending on specification and interpretation. Therefore, to be able to investigate the problem I need following details:

  1. What do you do, step by step, (who is online, who is offline, what a user sends, etc, ... with example packets)

  2. What exactly happens

  3. What do you think should happen instead

The 2 last points are very important.

Another question I have, would AMP and broadcast work together? I have implemented broadcast with AMP in my groovy script and hopefully that would work as well, when I get this "small" bug fixed. :-)

What is "broadcast"?

Added by Steffen Larsen over 4 years ago

Hi Artur,

Sorry about the late response here, but Ive been super busy (but you already know that) :-)

I'll do a detailed step by step guide later when I got the time, but basically most of the stuff works now.

The strange this is though that I can't see the messages in the table msg_history, even though it stores them for later delivery (maybe it stores it temporarily in memory.. I need to see the Amp module source later).

But my last question about broadcast. By broadcast I meant when i am sending to all of the domain (with my own AMP patched version of BroadcastToOnline.groovy script) and it need to work with that.

So when I use the broadcast script it needs to store the messages until the they are delivered to the users.

Of course that might be a problem if I don't know my finite number of users (we have our own auth. mechanism) and thereby do not know when the last one have connected and have to get the message delivered.

Have you guys ever tried to use broadcasting to a domain with AMP?

Cheers and thanks for all your help Artur!

/Steffen

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

AMP should not change anything. This is just a payload which is processed by a separate component and/or a client.

    (1-14/14)