Project

General

Profile

Test tigase with tsung

Xuan Zhou
Added almost 4 years ago

I had did , and I set ulimit to 350000, but it's running too slow, why ?

2015-05-12 10:24:52.136 [jabber:iq:register Queue Worker 7] ThreadExceptionHandler.uncaughtException() SEVERE: Uncaught thread: "jabber:iq:register Queue Worker 7" exception

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:714)

at java.util.Timer.<init>(Timer.java:176)

at java.util.Timer.<init>(Timer.java:146)

at com.mysql.jdbc.ConnectionImpl.getCancelTimer(ConnectionImpl.java:331)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2098)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2261)

at tigase.db.jdbc.DataRepositoryImpl.checkConnection(DataRepositoryImpl.java:363)

at tigase.db.jdbc.DataRepositoryImpl.getPreparedStatement(DataRepositoryImpl.java:182)

at tigase.db.DataRepositoryPool.getPreparedStatement(DataRepositoryPool.java:139)

at tigase.db.jdbc.TigaseCustomAuth.addUser(TigaseCustomAuth.java:335)

at tigase.db.AuthRepositoryMDImpl.addUser(AuthRepositoryMDImpl.java:72)

at tigase.xmpp.RepositoryAccess.register(RepositoryAccess.java:473)

at tigase.xmpp.impl.JabberIqRegister.process(JabberIqRegister.java:315)

at tigase.server.xmppsession.SessionManager$ProcessorWorkerThread.process(SessionManager.java:2434)

at tigase.util.WorkerThread.run(WorkerThread.java:128)

2015-05-12 10:24:59.765 [IPMonitor Timer] IPMonitor$1.run() WARNING: Many disconnects for IP: 192.168.0.201 - 956

jabber_register.xml (1.29 KB) jabber_register.xml tsung test-script
init.properties (3.02 KB) init.properties tigase config file

Replies (15)

Added by Xuan Zhou almost 4 years ago

xuan zhou wrote:

I had did , and I set ulimit to 350000, but it's running too slow, why ?

2015-05-12 10:24:52.136 [jabber:iq:register Queue Worker 7] ThreadExceptionHandler.uncaughtException() SEVERE: Uncaught thread: "jabber:iq:register Queue Worker 7" exception

java.lang.OutOfMemoryError: unable to create new native thread

at java.lang.Thread.start0(Native Method)

at java.lang.Thread.start(Thread.java:714)

at java.util.Timer.(Timer.java:176)

at java.util.Timer.(Timer.java:146)

at com.mysql.jdbc.ConnectionImpl.getCancelTimer(ConnectionImpl.java:331)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2098)

at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2261)

at tigase.db.jdbc.DataRepositoryImpl.checkConnection(DataRepositoryImpl.java:363)

at tigase.db.jdbc.DataRepositoryImpl.getPreparedStatement(DataRepositoryImpl.java:182)

at tigase.db.DataRepositoryPool.getPreparedStatement(DataRepositoryPool.java:139)

at tigase.db.jdbc.TigaseCustomAuth.addUser(TigaseCustomAuth.java:335)

at tigase.db.AuthRepositoryMDImpl.addUser(AuthRepositoryMDImpl.java:72)

at tigase.xmpp.RepositoryAccess.register(RepositoryAccess.java:473)

at tigase.xmpp.impl.JabberIqRegister.process(JabberIqRegister.java:315)

at tigase.server.xmppsession.SessionManager$ProcessorWorkerThread.process(SessionManager.java:2434)

at tigase.util.WorkerThread.run(WorkerThread.java:128)

2015-05-12 10:24:59.765 [IPMonitor Timer] IPMonitor$1.run() WARNING: Many disconnects for IP: 192.168.0.201 - 956

And when I test for a while, the ssh can't connect anymore,

the error is:

Connecting to 192.168.0.203:22...

Connection established.

To escape to local shell, press 'Ctrl+Alt+]'.

Last login: Tue May 12 10:31:15 2015 from 192.168.0.12

-bash: fork: retry: Resource temporarily unavailable

-bash: fork: retry: Resource temporarily unavailable

-bash: fork: retry: Resource temporarily unavailable

-bash: fork: retry: Resource temporarily unavailable

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

How much memory/RAM does the machine have? Have you adjusted JVM memory settings (Xmx, Xms) in @etc/tigase.conf@?

Added by Xuan Zhou almost 4 years ago

Wojciech Kapcia wrote:

How much memory/RAM does the machine have? Have you adjusted JVM memory settings (Xmx, Xms) in @etc/tigase.conf@?

The server has 32G memory/RAM and I have adjusted JVM memory settings (Xmx, Xms) in etc/tigase.conf

[tigase@camera3 etc]$ cat tigase.conf 
#osgiEnabled=(true|false)
#osgiEnabled=false
OSGI=${osgiEnabled}
ENC="-Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"
DRV="-Djdbc.drivers=com.mysql.jdbc.Driver:org.postgresql.Driver:org.apache.derby.jdbc.EmbeddedDriver"
GC="-XX:+UseBiasedLocking -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:ParallelCMSThreads=2 -XX:-ReduceInitialCardMarks"
EX="-XX:+OptimizeStringConcat -XX:+DoEscapeAnalysis -XX:+UseNUMA"
JMX_REMOTE_IP="-Djava.rmi.server.hostname=192.168.0.203"
JAVA_HOME="${JAVA_HOME}"
CLASSPATH=""
JAVA_OPTIONS="${GC} ${EX} ${ENC} ${DRV} ${JMX_REMOTE_IP} -server -Xms4096M -Xmx4096M -XX:PermSize=512m -XX:MaxPermSize=1024m -XX:MaxDirectMemorySize=1024m"
TIGASE_CONFIG="etc/tigase.xml"
TIGASE_OPTIONS=" --property-file etc/init.properties "
[tigase@camera3 etc]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256356
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 350000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 8192
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[tigase@camera3 etc]$ 

Added by Xuan Zhou almost 4 years ago

xuan zhou wrote:

Wojciech Kapcia wrote:

How much memory/RAM does the machine have? Have you adjusted JVM memory settings (Xmx, Xms) in @etc/tigase.conf@?

The server has 32G memory/RAM and I have adjusted JVM memory settings (Xmx, Xms) in etc/tigase.conf

[...]

[...]

[root@camera1 ~]# sysctl -p
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 360000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 90
[root@camera1 ~]# 
Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

You have 32GB machine but you assigned only 4GB of memory to Tigase. This is why it is slow and throws out of memory errors. Try to increase memory to at least 16GB or more depending on how many users you are trying to connect. It might be also a problem with performance related to a slow DB network or something else. You need to look at the Tigase server statistics to find potential bottlenecks.

Added by Xuan Zhou almost 4 years ago

Now, I have adjusted tigase settings in init.properties file like this:

--packet.debug.full=true

--test=true

--sm-threads-pool=custom:2000

--sm-plugins=jabber:iq:register=8,jabber:iq:auth=16,urn:ietf:params:xml:ns:xmpp-sasl=16

--max-queue-size = 25000

--queue-implementation=tigase.util.PriorityQueueStrict

--new-connections-throttling=5222:2000

and it's faster, but has errros in console sometimes:

2015-05-13 09:12:21.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2665
2015-05-13 09:12:21.515 [in_17-message-router]  MessageRouter.processPacket()  WARNING: Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_54269, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 09:12:31.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2822
2015-05-13 09:12:41.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2689
2015-05-13 09:12:51.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2688
2015-05-13 09:13:01.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2627
2015-05-13 09:13:11.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2752
2015-05-13 09:13:21.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2512
2015-05-13 09:13:21.644 [in_27-message-router]  MessageRouter.processPacket()  WARNING: Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_11293, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 09:13:23.492 [in_5-message-router]  MessageRouter.processPacket()  WARNING:  Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_9102, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 09:13:30.395 [in_26-message-router]  MessageRouter.processPacket()  WARNING: Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_45196, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 09:13:31.147 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2534
2015-05-13 09:13:41.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2814
2015-05-13 09:13:51.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2754
2015-05-13 09:14:01.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2624
2015-05-13 09:14:11.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2610

I'm trying to regist a new user every 0.0025 second until 200000 users for a test. why errors sometimes? How can I reach 500000 users online in the same time?

Added by Xuan Zhou almost 4 years ago

xuan zhou wrote:

Now, I have adjusted tigase settings in init.properties file like this:

--packet.debug.full=true

--test=true

--sm-threads-pool=custom:2000

--sm-plugins=jabber:iq:register=8,jabber:iq:auth=16,urn:ietf:params:xml:ns:xmpp-sasl=16

--max-queue-size = 25000

--queue-implementation=tigase.util.PriorityQueueStrict

--new-connections-throttling=5222:2000

and it's faster, but has errros in console sometimes:

[...]

I'm trying to regist a new user every 0.0025 second until 200000 users for a test. why errors sometimes? How can I reach 500000 users online in the same time?

CPU and memory usage is not high .

Added by Xuan Zhou almost 4 years ago

Artur Hefczyc wrote:

You have 32GB machine but you assigned only 4GB of memory to Tigase. This is why it is slow and throws out of memory errors. Try to increase memory to at least 16GB or more depending on how many users you are trying to connect. It might be also a problem with performance related to a slow DB network or something else. You need to look at the Tigase server statistics to find potential bottlenecks.

200000 users try to regist, but 198130 users registed finally.

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

Please remove following from your configuration:
--sm-threads-pool=custom:2000
--max-queue-size = 25000
--queue-implementation=tigase.util.PriorityQueueStrict

Are you running Tsung on the same machine?

Added by Xuan Zhou almost 4 years ago

yes, like this;

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "tsung-1.0.dtd">
<tsung loglevel="notice" dumptraffic="false" version="1.0">

    <clients>
        <client host="localhost" use_controller_vm="true" maxusers="200000">
        </client>
    </clients>

    <servers>
        <server host='192.168.0.203' port='5222' type='tcp' />
    </servers>

    <!-- register 200000 users in less than 15 minutes -->
    <load>
        <!--
        <arrivalphase phase="1" duration="60" unit="second">
            <users maxnumber="30" interarrival="1" unit="second"></users>
        </arrivalphase>

        <arrivalphase phase="1" duration="60" unit="second">
            <users maxnumber="100" interarrival="0.1" unit="second"></users>
        </arrivalphase>

        <arrivalphase phase="1" duration="60" unit="second">
            <users maxnumber="1000" interarrival="0.01" unit="second"></users>
        </arrivalphase>
        -->
        <arrivalphase phase="1" duration="15" unit="minute">
            <users maxnumber="200000" interarrival="0.0025" unit="second"></users>
        </arrivalphase>
    </load>

    <options>
        <option type="ts_jabber" name="global_number" value="200000"></option>
        <option type="ts_jabber" name="userid_max" value="200000"></option>
        <option type="ts_jabber" name="domain" value="192.168.0.203"></option>
        <option type="ts_jabber" name="username" value="tsung"></option>
        <option type="ts_jabber" name="passwd" value="tsung"></option>
    </options>

    <sessions>
        <!-- 用户注册 -->
        <session probability="100" name="jabber-example" type="ts_jabber">
            <request>
                <jabber type="connect" ack="local"></jabber>
            </request>
            <request>
                <match do="abort" when="match">error</match>
                <jabber type="register" ack="local" id="new"></jabber>
            </request>
            <request>
                <jabber type="close" ack="local"></jabber>
            </request>
        </session>
    </sessions>
</tsung>

Added by Xuan Zhou almost 4 years ago

Wojciech Kapcia wrote:

[...]

Are you running Tsung on the same machine?

It's still print errors on tigase-console.log:

2015-05-13 13:36:07.610 [in_18-message-router]  MessageRouter.processPacket()  WARNING: Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_43257, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 13:36:09.610 [in_27-message-router]  MessageRouter.processPacket()  WARNING: Packet with TO attribute set to NULL: from=c2s@camera3.netpower/192.168.0.203_5222_192.168.0.201_39376, to=null, DATA=<stream:features type="error"><ver xmlns="urn:xmpp:features:rosterver"/><starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls"/><mechanisms xmlns="urn:ietf:params:xml:ns:xmpp-sasl"><mechanism>PLAIN</mechanism><mechanism>ANONYMOUS</mechanism></mechanisms><register xmlns="http://jabber.org/features/iq-register"/><compression xmlns="http://jabber.org/features/compress"><method>zlib</method></compression><auth xmlns="http://jabber.org/features/iq-auth"/><error type="wait" code="404"><recipient-unavailable xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/></error></stream:features>, SIZE=573, XMLNS=null, PRIORITY=NORMAL, PERMISSION=NONE, TYPE=error
2015-05-13 13:36:11.137 [IPMonitor Timer]  IPMonitor$1.run()                  WARNING:  Many disconnects for IP: 192.168.0.201 - 2831

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

xuan zhou wrote:

Wojciech Kapcia wrote:

[...]

Are you running Tsung on the same machine?

It's still print errors on tigase-console.log:

[...]

Can you verify in Tsung that it's working correctly and not dropping connections?

Added by Xuan Zhou almost 4 years ago

Wojciech Kapcia wrote:

xuan zhou wrote:

Wojciech Kapcia wrote:

[...]

Are you running Tsung on the same machine?

It's still print errors on tigase-console.log:

[...]

Can you verify in Tsung that it's working correctly and not dropping connections?

I don't know how to verify, do you have some ideas?

Avatar?id=6023&size=32x32

Added by Artur Hefczyc TigaseTeam almost 4 years ago

200000 users try to regist, but 198130 users registed finally.

If you try to register 200k users over XMPP then database can be a bottleneck for sure, as each registration request requires database write access. Therefore, some users failed to register due to a timeout and out of memory most likely happened due to excessive queuing in Tigase.

Have you tried to register all users concurrently or one by one?

Added by Xuan Zhou almost 4 years ago

Artur Hefczyc wrote:

200000 users try to regist, but 198130 users registed finally.

If you try to register 200k users over XMPP then database can be a bottleneck for sure, as each registration request requires database write access. Therefore, some users failed to register due to a timeout and out of memory most likely happened due to excessive queuing in Tigase.

Have you tried to register all users concurrently or one by one?

You can see the test script above (jabber-regist.xml) , I tried every 0.025 seconds a user registration.

    (1-15/15)