Project

General

Profile

'received message by server' status of a message

Igor Khomenko
Added over 4 years ago

What is the best way to know that your chat message was received by server?

I know some popular XEPs which provide a way to implement 'received by client' logic.

Message Delivery Receipts http://www.xmpp.org/extensions/xep-0184.html

Chat Markers http://xmpp.org/extensions/xep-0333.html

But, for example, if my opponent is offline, I would like to know that my message was delivered to server and wasn't lost somewhere by some reason.

Does Tigase support this? Or how we can implement it?


Replies (15)

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Stream management XEP-0198 takes care of that. Tigase server does support it.

Added by Igor Khomenko over 4 years ago

Great, thanks

I checked XEP-0198

It's clear how to use Basic Acking Scenario http://xmpp.org/extensions/xep-0198.html#scenarios-basic

But I have some doubts how to use properly Efficient Acking Scenario http://xmpp.org/extensions/xep-0198.html#scenarios-efficient

For example, user sent 5 messages and then sent **** element and received **** . How he knows which message wasn't delivered to server? which one from 5 messages?

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

For example, user sent 5 messages and then sent **** element and received **** . How he knows which message wasn't delivered to server? which one from 5 messages?

He knows that ALL of them were delivered. The server responds only to messages delivered. If 2 out of 5 were delivered the server would confirm delivery of 2 only.

Added by Matthew M over 4 years ago

Hi Tigase team,

We have a similar need to "confirm" the server has received message (and processed correctly) from the client.

XEP--0198 is a great protocol, but somehow I feel it's like a overkill for this plain simple "sent-confirmation". The client has to keep track of its sending history which is more complicated.

What if we write a simple plugin in tigase to echo back a customized IQ to confirm that the message is received, such as

<iq ...> <sent-ack message-stanza-id = 'abcd'> </iq....>

In this case, the client logic could be very simple: just wait for the "sent-ack" on each outgoing message stanza, consider the message is lost after some timeout.

Do you think is there any downside on this approach? Thanks!

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

(and processed correctly) from the client.

What does it mean?

XEP--0198 is a great protocol, but somehow I feel it's like a overkill for this plain simple "sent-confirmation". The client has to keep track of its sending history which is more complicated.

What if we write a simple plugin in tigase to echo back a customized IQ to confirm that the message is received, such as

In this case, the client logic could be very simple: just wait for the "sent-ack" on each outgoing message stanza, consider the message is lost after some timeout.

This is exactly what the XEP-198 does. It sends you ack for received messages. As an optimization it may send you ACK for more than one message. More over XEP-0198 is implemented on connections level, not as a plugin, which makes it very efficient and low resources consuming. There is no traffic in SM related to ACS.

Do you think is there any downside on this approach? Thanks!

Your approach would work certainly but it has a few downsides.

  1. You are slowing down your client, because you essentially require confirmation for each packet before sending another. So if there more packets to send by the client you have to queue them somehow on the client, wait for confirmation. This all make take time and slow things down. It would impact user experience.

  2. You are essentially doing the same as XEP-0198 but less efficient way

  3. As you implement it as a plugin, it increased load on the server, the overall traffic is doubled as each packet needs confirmation which goes through the whole server processing phases.

  4. Your solution would work for 1k, 10k online users, however will it work for 100k online users? for 1M. You are generating significant overhead on the server, remember that.

  5. Your solution does not solve the other way around confirmations. What about delivery form the server to a client? How do you want to implement this?

Added by Igor Khomenko over 4 years ago

Artur Hefczyc wrote:

For example, user sent 5 messages and then sent **** element and received **** . How he knows which message wasn't delivered to server? which one from 5 messages?

He knows that ALL of them were delivered. The server responds only to messages delivered. If 2 out of 5 were delivered the server would confirm delivery of 2 only.

Let me explain a bit more,

This is pseudo-XML:

C: <enable/>
S: <enabled/>
C: <message text="1" id="111"/>
C: <message text="2" id="222"/>
C: <message text="3" id="333"/>
C: <message text="4" id="444"/>
C: <message text="5" id="555"/>
C: <r/>
S: <a h='3'/>

as you can see, user had sent 5 messages, but server said that it received only 3 messages.

How user understand which messages he should resend?

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

This is an impossible case. If we are talking about TCP/IP connection, then it is not possible that there were 2 messages lost out of 5 and the connection is still good to send and .

The only possible response from the server, in the scenario above is:

<a h='5'/>

Added by Gabriel Rossetti over 4 years ago

Could XEP-0079 not be used for this? In XEP-0184's introduction (http://www.xmpp.org/extensions/xep-0184.html#intro) it mentions that XEP-0079 is server-level confirmation (I may be misinterpreting this though).

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Yes, AMP (XEP-0079) can be used for this as well. It is fully supported on the server. However, AMP has slightly different purpose and works on a higher level of the protocol. The confirmation is just kind of a side-effect. AMP is for "Advanced Message Processing", so you can specify different conditions and actions that are applied to an XMPP packet. One of them is confirmation. However, AMP does not say anything what happens if packet is lost, there is no retry or error reporting, etc.... The only thing you have is, if you receive conformation than you know that everything was OK, if you do not receive confirmation than you do not know anything. It could be OK (confirmation lost) or it is still being processed, or something went wrong...

The stream management (XEP-0184) is low-level, almost TCP/IP connection level, packet delivery confirmation with a way to send retries and recover from connection loss.

Added by Gabriel Rossetti over 4 years ago

Thank you Artur, that was very informative.

So from what I understand in the replies before mine, it can ack several msgs at once but there is no way in this case to know which one failed (if one failed), the best then is to ack every msg if you want to be sure but this doesn't seem optimal. I say this because if you send messages in an asynchronous manner, you have no way of knowing for which message the (asynchronous) ack is for, it doesn't have the ID of the msg it is ack'ing, or am I missing something here? I read the spec and didn't see anything about being able to match an ack to the msg, so how is this useful? To illustrate:

C: <enable/>
S: <enabled/>
C: <message text="1" id="111"/>
C: </r>
C: <message text="2" id="222"/>
C: </r>
C: <message text="3" id="333"/>
S: <a h='2'/>
C: </r>
C: <message text="4" id="444"/>
C: </r>
C: <message text="5" id="555"/>
C: <r/>
S: <a h='2'/>

I sent 5 messages asynchronously and requested an ack for each one, in this example one of the 5 failed, from what I understand from the spec it could be any of the 5 (not just one of the 1st 3) since for example msg 3 could have been processed at a later time but another msg could have been lost. This just tells me that something went wrong and I should reconnect and start over again. Now imagine these are not simple chat msgs but commands being sent to the server, and one failed to arrive, I need to therefore have a sort of rollback procedure in place...

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

This was explained in one of the comments above. The TCP/IP protocol guarantees, that if data packets are delivered, they arrive in a correct order. Therefore, in the example you provided above, it is not possible that messages: 1, 2, 3 and 5 were delivered and message 4 was lost due to network error. Therefore, if you receive <a h='2'/> twice, it means 4 packets were received by other side. In this case, it means messages 1-4 were already received. As for the message '5' we do not know yet whether it was received already or not, no confirmation was received and no error was signaled.

You just have to think of this confirmation protocol as a protocol which counts packets and sends you information on how many packets were already received by other side. And this is actually all that you need to know.

Added by Gabriel Rossetti over 4 years ago

This is not exactly correct, TCP/IP does guarantee the packet ordering, but XMPP is an application layer protocol, an XMPP message is split into one or more TCP packets. TCP/IP is a transport layer protocol that guarantees that all the packets composing your XMPP msg will arrive in order, not that several XMPP msgs sent in an order will be received in that order. Here is a good explanation:

http://en.wikibooks.org/wiki/A-level_Computing/AQA/Computer_Components,_The_Stored_Program_Concept_and_the_Internet/Structure_of_the_Internet/TCP_IP_protocol_stack

So in my example above it is possible that you send several msgs and that one is lost or arrives later, but each msg is guaranteed to arrive whole.

I have seen this happen while chatting, not sure if it was an XMPP based protocool or not, I recieved msgs in the wrong order and then it re-orders the msgs. I assume they had a client side timestamp or msg sent order and the client received and displayed them and when it noticed that they were not ordered correctly (because it recieved a new msgs that should have been before the previous msgs it recieved) it fixed that, but this is application layer, TCP/IP had nothing to do with it.

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Believe or not I do know quite a bit about TCP/IP and other network protocols. After all networking and implementing network protocols is something I have been doing for last 20 years or so. But, thank you for the link anyway. There is always room to learn something new.

Your interpretation is also not entirely correct. The TCP/IP layer does NOT guarantee that packets composing of your XMPP message arrive in a correct order. This is not true because TCP/IP has no understanding of application layer protocols such as XMPP, HTTP, FTP or any other. The TCP/IP protocol only guarantees that the data submitted in a buffer to TCP/IP layer will be delivered in the same order as they are submitted, even though they can travel through different paths, different routers and can arrive at different order.

You are correct that on XMPP application level, a server or a client may reorder messages. But this is "application layer" reordering. And we are not concerned about this. The XEP-0184 works (or should work) - at least in the Tigase server, between the application level and TCP/IP level. Actually in Tigase is on as low level, close to network socket as possible. So in the example above we really are talking about messages submitted to the network socket. Once they submitted in that order, they arrive in that order, and ACK responses refer these packets.

Added by Gabriel Rossetti over 4 years ago

Artur Hefczyc wrote:

Believe or not I do know quite a bit about TCP/IP and other network protocols. After all networking and implementing network protocols is something I have been doing for last 20 years or so. But, thank you for the link anyway. There is always room to learn something new.

:-), I didn't mean it like that; I just wanted to illustrate what I was referring to, no hard feelings. There is something about XEP-0184 that is nagging me so I am trying to understand better, especially since I will have to explain to someone else that will probably ask the same questions, thanks in advance for your patience and answers.

Your interpretation is also not entirely correct. The TCP/IP layer does NOT guarantee that packets composing of your XMPP message arrive in a correct order. This is not true because TCP/IP has no understanding of application layer protocols such as XMPP, HTTP, FTP or any other. The TCP/IP protocol only guarantees that the data submitted in a buffer to TCP/IP layer will be delivered in the same order as they are submitted, even though they can travel through different paths, different routers and can arrive at different order.

Ok, that is what I meant more or less, I know it knows nothing about XMPP, but since the msg is submitted to it as raw bytes thus it is put in its buffer and thus it guarantees that the packets composed of those bytes (+ header) are delivered in the correct order. You explained it better though.

TCP is considered reliable; it does guarantee the order but does not guarantee the delivery (which is where our problem comes from). When you replied to Igor above, you said:

This is an impossible case. If we are talking about TCP/IP connection, then it is not possible that there were 2 messages lost out of 5 and the connection is still good to send and .

Why is that impossible? (it is essentially the same example as I gave but in mine I lost only one msg). If you look here (again, just so you see what I am referring to) http://www.stratus.com/blog/openvos/?p=749 , imagine that the first or second scenario occurs, my TCP stack knows something happened, I received an ack <a h='2'/> when I expected (application layer) an ack @@. In this scenario I have no idea which msg got lost. I re-read XEP-0198 and I made a mistake in my example above, the counter is the total number of msgs received since the beginning and not the number of messages received since the last ack receipt request (@@), so it should have looked like this:

C: <enable/>
S: <enabled/>
C: <message text="1" id="111"/>
C: <message text="2" id="222"/>
C: </r>
C: <message text="3" id="333"/>
S: <a h='2'/>
C: <message text="4" id="444"/>
C: <message text="5" id="555"/>
C: <r/>
S: <a h='4'/>

In this example I should have received at the end and instead I received , I thus know something happened but I do not know which msg was lost. In this scenario I have to do something like (at the application layer, client side):

1) request a new ack

2) if nothing has changed, wait a little and ask again, do this a couple of times maybe?

3) send the 3 unacked msgs again

and on the server side I will (hopefully) receive the 3 msgs (suppose they are commands) this time and I need to either ignore the ones I received (if the reception order is not important) or rollback the previous 2 msgs and process the 3 new msgs.

so I think in XEP-0198 it may not be optimal to ask for an ack for each msg but it is from what I see the only way to really be sure the msg arrived and not have to have a complicated multi-msg rollback mechanism. I wonder why they didn't define it something like this instead:

C: <enable type="id_list"/> (could either be "simple and it just sends the increment like currently or "id_list" and you get the list of received msg IDs since the last ack request only)
S: <enabled/>
C: <message text="1" id="111"/>
C: <message text="2" id="222"/>
C: </r>
C: <message text="3" id="333"/>
S: <a ids="111, 222"/>
C: <message text="4" id="444"/>
C: <message text="5" id="555"/>
C: <r/>
S: <a ids="333, 555">
Wait a little...
C: <r/>
S: <a ids="333, 555">

Here you know that msg "444" was lost

You are correct that on XMPP application level, a server or a client may reorder messages. But this is "application layer" reordering. And we are not concerned about this. The XEP-0184 works (or should work) - at least in the Tigase server, between the application level and TCP/IP level. Actually in Tigase is on as low level, close to network socket as possible. So in the example above we really are talking about messages submitted to the network socket. Once they submitted in that order, they arrive in that order, and ACK responses refer these packets.

Ok, thanks

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam over 4 years ago

Long posts generally require long time to respond ;-)

But here the answer is short. It comes down to a correct understanding of the sentence:

bq. TCP is considered reliable; it does guarantee the order but does not guarantee the delivery.

The above means that all data is delivered in order as long as the connection is established. The only way data is not delivered "not guarantee of the delivery" is when the connection breaks. But then neither the "data" in transit or any subsequent data is delivered. With TCP/IP you have guaranteed that no data in the middle of the stream is lost.

Your case above, should be translated to TCP/IP level and it would look like this:

  1. Messages '111', '222', '333' is delivered by TCP/IP and passed to the application level because from the TCP/IP point of view everything is OK (note, for TCP/IP this is just a data buffer lets call it b1)

  2. Message '444' on the TCP/IP level is lost (note, for TCP/IP this is just another data buffer, lets call it b2)

  3. Message '555' is on transit on the TCP/IP level (note, again, for TCP/IP this is yet another data buffer, lets call it b3)

  4. Message '555' arrives to the destination point on the TCP/IP level

  5. TCP/IP nows that that there was a data buffer lost in transit - data buffer b2

  6. TCP/IP does not pass the data buffer b3 (message '555') to application level and waits for the missing buffer b2 to be delivered

  7. A TCP/IP level timeout expires. The data buffer b3 is discarded and connection broken is reported to application level

I think, now it should be clear that the case presented by you and Igor are impossible if we use TCP/IP. And... this is precisely why we use TCP/IP. The problem you describe can only happen with UDP protocol and this is why UDP is not suitable for XMPP.

    (1-15/15)