Project

General

Profile

CPU Hight-loads around midnights

Ehsan Mozaffari
Added over 3 years ago

Hi

I've been monitoring my tigase server for a while using Tigase Monitor and I've noticed there are cpu high loads around mid-night (around 11:00PM - 1:00AM local time).

As those cpu high loads occurs in these intervals that the users' activities are minimal I am wondering if this is related to any auto schedule in tigase server functions.

Thank you in advance.


Replies (8)

Added by Ehsan Mozaffari over 3 years ago

Just to add, during the last week I have been observing Threads and processes that causes CPU High loads by jstack, top ,jhat ,jmap and etc

I have been trying to analyze the behavior of threads that may help to conclude some meaningful overview for CPU High loads (please check the attached snapshot)

Thank you for the support.

graph_image.php.png (153 KB) graph_image.php.png Analytical graph for CPU

Added by Wojciech Kapcia TigaseTeam over 3 years ago

Ehsan Mozaffari wrote:

As those cpu high loads occurs in these intervals that the users' activities are minimal I am wondering if this is related to any auto schedule in tigase server functions.

While there is a mechanism that allows performing some actions at a given interval it shouldn't cause such spikes in usage.

Do you have any custom components or special configuration? Are there any entries in the logs or entries in the statistics indicating more traffic?

Added by Ehsan Mozaffari over 3 years ago

Thank you for replying.

We don't have any custom component or special configuration. ( Kindly check the attached configuration file)

The traffic usage has been normal.

Wojciech Kapcia wrote:

While there is a mechanism that allows performing some actions at a given interval it shouldn't cause such spikes in usage.

I'm a little suspicious to our Database for null entries. Regarding the mechanism you mentioned above; Are these actions related to database?

Actually, we have had these CPU-loads issue since some months ago.

Some times Tigase consumes up to 80 percent in our servers.

The following is a post for CPU Highloads that we had posted before.

[[[https://projects.tigase.org/boards/15/topics/5479?r=5542#message-5542]]]

Added by Wojciech Kapcia TigaseTeam over 3 years ago

Ehsan Mozaffari wrote:

Thank you for replying.

We don't have any custom component or special configuration. ( Kindly check the attached configuration file)

The traffic usage has been normal.

Just as a side-note - assuming you're on one of the latest versions this can be removed

--cluster-nodes =x3.lopl.net,x2.lopl.net
--cluster-connect-all=true

Wojciech Kapcia wrote:

While there is a mechanism that allows performing some actions at a given interval it shouldn't cause such spikes in usage.

I'm a little suspicious to our Database for null entries. Regarding the mechanism you mentioned above; Are these actions related to database?

Actually, we have had these CPU-loads issue since some months ago.

Anything particular that may have happened at that time or it "just started on it's own"?

Some times Tigase consumes up to 80 percent in our servers.

Can you try to capture the situation with the Tigase Monitor (including bottom part) and post screenshots of whole overview window while it's happening?

The following is a post for CPU Highloads that we had posted before.

[[[https://projects.tigase.org/boards/15/topics/5479?r=5542#message-5542]]]

That post indicated more towards problems with deadlocks, however it may be related.

Added by Ehsan Mozaffari over 3 years ago

Wojciech Kapcia wrote:

While there is a mechanism that allows performing some actions at a given interval it shouldn't cause such spikes in usage.

I'm a little suspicious to our Database for null entries. Regarding the mechanism you mentioned above; Are these actions related to database?

Actually, we have had these CPU-loads issue since some months ago.

Anything particular that may have happened at that time or it "just started on it's own"?

I think, when we was starting using stream management component it was started!

Wojciech Kapcia wrote:

Some times Tigase consumes up to 80 percent in our servers.

Can you try to capture the situation with the Tigase Monitor (including bottom part) and post screenshots of whole overview window while it's happening?

I've attached the screen captures which happened yesterday afternoon and last night.

Wojciech Kapcia wrote:

The following is a post for CPU Highloads that we had posted before.

[[[https://projects.tigase.org/boards/15/topics/5479?r=5542#message-5542]]]

That post indicated more towards problems with deadlocks, however it may be related.

I have checked threads states with thread dump analyzer such as jstack,samurai,TDA,jvirtualvm and I haven't seen any deadlock so far.

Added by Wojciech Kapcia TigaseTeam over 3 years ago

OK, one question that comes to my mind - are there any other processes on that machine? Have you verified that this is in fact Tigase process that utilizes the CPU at that time?

Added by Ehsan Mozaffari over 3 years ago

There are not other processes on Tigase machine and that machine belongs completely to tigase without even local MySQL [Real dedicated HP server with Oracle jdk1.7.0_79 (latest version)]

I wrote a Perl application that combines jstack with top and it executes by crontab from 9PM to 2 AM so it obviously shows the socketReadThreads were utilizing High CPU-usage at that time and other threads of tigase such as ResultsListener-socketReadThreads and pool-y-threads eat almost always 5.9 percent per each thread. So, overall of CPU consumption reach to 70% to 90%.

I would be grateful if I can use debugging switches to narrow down which functions or methods of threads that I mentioned above into init.properties.

I have attached my perl and shell programs.

Thank you for your time.

cpu-java.sh (388 Bytes) cpu-java.sh Shell script of cpu-load.pl
cpu-load.pl (1.68 KB) cpu-load.pl Perl application in order to find cpu-load by jstack and top
cpl_23-Aug_0-6-10.log (133 KB) cpl_23-Aug_0-6-10.log Sample output of cpu-load.pl
top_23-Aug_0-6-10.log (82.8 KB) top_23-Aug_0-6-10.log Sample output of cpu-load.pl
jstack_23-Aug_0-6-10.log (690 KB) jstack_23-Aug_0-6-10.log Sample output of cpu-load.pl

Added by Wojciech Kapcia TigaseTeam over 3 years ago

Ehsan Mozaffari wrote:

I would be grateful if I can use debugging switches to narrow down which functions or methods of threads that I mentioned above into init.properties.

Adding net and io packages could shed more light on the issue.

    (1-8/8)