Project

General

Profile

Too many close_wait connections

Bogdan Shaparenko
Added over 3 years ago

Hi,

We configure HTTP component, and have many connections to tigase, but when our application close connection - tigase doesn't.

Help me please understand what's the problem.

lsof -i -n -P |grep 8083

ava      19147  chat 3724u  IPv6 177089053      0t0  TCP 172.31.19.232:8083->172.31.18.195:32833 (CLOSE_WAIT)
java      19147  chat 3725u  IPv6 177089364      0t0  TCP 172.31.19.232:8083->172.31.18.195:32994 (CLOSE_WAIT)
java      19147  chat 3726u  IPv6 177089070      0t0  TCP 172.31.19.232:8083->172.31.18.195:32844 (CLOSE_WAIT)
java      19147  chat 3727u  IPv6 177089166      0t0  TCP 172.31.19.232:8083->172.31.18.195:32922 (CLOSE_WAIT)
java      19147  chat 3728u  IPv6 177090536      0t0  TCP 172.31.19.232:8083->172.31.18.195:33136 (CLOSE_WAIT)
java      19147  chat 3729u  IPv6 177091439      0t0  TCP 172.31.19.232:8083->172.31.18.195:33700 (CLOSE_WAIT)
java      19147  chat 3730u  IPv6 177094199      0t0  TCP 172.31.19.232:8083->172.31.18.195:34070 (CLOSE_WAIT)
java      19147  chat 3731u  IPv6 177098787      0t0  TCP 172.31.19.232:8083->172.31.18.195:35102 (CLOSE_WAIT)
java      19147  chat 3732u  IPv6 177089405      0t0  TCP 172.31.19.232:8083->172.31.18.195:33067 (CLOSE_WAIT)
java      19147  chat 3734u  IPv6 177090440      0t0  TCP 172.31.19.232:8083->172.31.18.195:33134 (CLOSE_WAIT)
java      19147  chat 3735u  IPv6 177091750      0t0  TCP 172.31.19.232:8083->172.31.18.195:33278 (CLOSE_WAIT)
java      19147  chat 3736u  IPv6 177091710      0t0  TCP 172.31.19.232:8083->172.31.18.195:33264 (CLOSE_WAIT)
java      19147  chat 3737u  IPv6 177090648      0t0  TCP 172.31.19.232:8083->172.31.18.195:33212 (CLOSE_WAIT)
java      19147  chat 3738u  IPv6 177091709      0t0  TCP 172.31.19.232:8083->172.31.18.195:33213 (CLOSE_WAIT)
java      19147  chat 3740u  IPv6 177090848      0t0  TCP 172.31.19.232:8083->172.31.18.195:33281 (CLOSE_WAIT)
java      19147  chat 3741u  IPv6 177090850      0t0  TCP 172.31.19.232:8083->172.31.18.195:33326 (CLOSE_WAIT)
java      19147  chat 3743u  IPv6 177092853      0t0  TCP 172.31.19.232:8083->172.31.18.195:33754 (CLOSE_WAIT)
java      19147  chat 3745u  IPv6 177097352      0t0  TCP 172.31.19.232:8083->172.31.18.195:34934 (CLOSE_WAIT)
java      19147  chat 3746u  IPv6 177098831      0t0  TCP 172.31.19.232:8083->172.31.18.195:35153 (CLOSE_WAIT)
java      19147  chat 3747u  IPv6 177093859      0t0  TCP 172.31.19.232:8083->172.31.18.195:33902 (CLOSE_WAIT)
java      19147  chat 3748u  IPv6 177093108      0t0  TCP 172.31.19.232:8083->172.31.18.195:33835 (CLOSE_WAIT)
java      19147  chat 3751u  IPv6 177094314      0t0  TCP 172.31.19.232:8083->172.31.18.195:34125 (CLOSE_WAIT)
java      19147  chat 3752u  IPv6 177094318      0t0  TCP 172.31.19.232:8083->172.31.18.195:34202 (CLOSE_WAIT)
java      19147  chat 3755u  IPv6 177095475      0t0  TCP 172.31.19.232:8083->172.31.18.195:34472 (CLOSE_WAIT)
java      19147  chat 3757u  IPv6 177101012      0t0  TCP 172.31.19.232:8083->172.31.18.195:35575 (CLOSE_WAIT)
java      19147  chat 3758u  IPv6 177097987      0t0  TCP 172.31.19.232:8083->172.31.18.195:35176 (CLOSE_WAIT)
java      19147  chat 3762u  IPv6 177101164      0t0  TCP 172.31.19.232:8083->172.31.18.195:35757 (CLOSE_WAIT)
java      19147  chat 3763u  IPv6 177101163      0t0  TCP 172.31.19.232:8083->172.31.18.195:35701 (CLOSE_WAIT)

lsof -i -n -P |grep 8083 | wc -l

3519

cat /etc/sysctl.conf

fs.file-max=360000
net.ipv4.ip_local_port_range=1024 65000
net.ipv4.tcp_keepalive_time=60
net.ipv4.tcp_keepalive_probes=3
net.ipv4.tcp_keepalive_intvl=90

cat /etc/security/limits.conf

chat               soft    nofile         500000
chat               hard    nofile         500000

cat tigase-consile.log

2016-01-21 15:40:13.459 [in_1-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-21 18:34:36.625 [in_7-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 08:26:49.028 [in_3-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 10:11:23.588 [in_7-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 12:06:28.062 [in_0-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 13:38:36.412 [in_5-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 14:16:34.856 [in_7-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 14:25:20.469 [in_6-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 16:15:10.247 [in_7-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 17:17:51.320 [in_5-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-22 18:40:20.984 [in_5-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ
2016-01-23 19:16:18.819 [in_0-c2s]         TLSIO.writeBuff()                  WARNING:  Infinite loop detected in writeBuff(buff) TLS code, tlsWrapper.getStatus(): NEED_READ

Replies (4)

Added by Wojciech Kapcia TigaseTeam over 3 years ago

What exact requests are you making? What is your (Tigase) configuration (I assume it's not default, at least judging by the ports). Because you are referring to HTTP component yet you provide excerpt from the logs with are related to the c2s connection (regular XMPP client socket connections over port 5222).

Bottomline - please provide more details and comments.

Added by Bogdan Shaparenko over 3 years ago

Hi,

my config

config-type = --gen-config-def
--admins = 1-1@chat.test

# setup DBs
--user-db = mysql
--user-db-uri = --------------------

--data-repo-pool-size = 30

# set domain
--virt-hosts = chat.test
--ssl-def-cert-domain = chat.test

--sm-plugins=-jabber:iq:private,-jabber:iq:register,-jabber:iq:auth,-jabber:iq:version,-msgoffline,-message,-vcard-temp,+message-carbons,+messagetocustomobject,+lastrequestat

--comp-name-1 = ext
--comp-class-1 = tigase.server.ext.ComponentProtocol
--external = muc.chat.test:muc-secret:listen:5271:10.0.10.20:ReceiverBareJidLB

# new config for HTTP component
--comp-name-3=http
--comp-class-3=tigase.http.HttpMessageReceiver
http/http/server-class=tigase.http.jetty.JettyStandaloneHttpServer
http/rest/api-keys[s]=----------------------
http/http/port[I]=8083

# enable WebSockets
ws2s/connections/ports[i]=5290,5291
ws2s/connections/5291/socket=ssl
ws2s/connections/5291/type=accept

bosh/connections/ports[i]=5280,5281
bosh/connections/5281/socket=ssl
bosh/connections/5281/type=accept

# enable XEP-0198
c2s/processors[s]=urn:xmpp:sm:3
c2s/processors/urn\:xmpp\:sm\:3/resumption-timeout[I]=60
c2s/processors/urn\:xmpp\:sm\:3/ack-request-count[I]=1

# enable monitoring
--monitoring = http:9080

--cm-traffic-throttling = xmpp:600:0:disc,bin:0:0:disc
--cm-ht-traffic-throttling = xmpp:0:0:disc,bin:0:0:disc

There are no errors or warnings in logs except c2s connection

Added by Bogdan Shaparenko over 3 years ago

Did you have a chance to check provided details? Please help us with this.

Added by Andrzej Wójcik IoT 1 CloudTigaseTeam over 3 years ago

Following line in your etc/init.properties file:

http/http/server-class=tigase.http.jetty.JettyStandaloneHttpServer

forces Tigase HTTP API Component to use Jetty HTTP Server as server responsible for accepting, processing and closing connections opened on port 8083 of your server.

This means that Jetty HTTP Server is not closing properly opened connections for some reason. To diagnose this issue I would suggest to enable debugging of Tigase HTTP API component and Jetty HTTP Server.

To enable debugging of Tigase HTTP API coponent you need to add following line to etc/init.properties file:

--debug=http

As for enabling debugging of Jetty HTTP Server I think you would need to use alternative loggers as described at https://projects.tigase.org/projects/tigase-server/wiki/Alternate_loggers_in_Tigase and properly configure logback.xml file.

    (1-4/4)