Project

General

Profile

Any way to find out, if MySQL server is down, through tigase HTTP API?

Pratap Patil
Added almost 4 years ago

I have a tigase server hosted on one AWS EC2 machine and MySQL on AWS RDS machine.

Then there is one application server which creates tigase users using the tigase HTTP API written in groovy.

This combination works just fine. But when the MySQL server goes down there is no way to find it out. The tigase HTTP API return 401 and logs the following line in logs

AuthRepositoryMDImpl.plainAuth()  WARNING: Couldn't obtain user repository for domain

My question is that is there any way to find out of through tigase HTTP API that the MySQL server has gone down? Is there any groovy script for the same?


Replies (5)

Added by Pratap Patil almost 4 years ago

With MySQL hosted on AWS RDS which has a "Reboot with failover", explained here http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RebootInstance.html, which does the following:

When you force a failover of your DB instance, Amazon RDS automatically switches to a standby replica in another Availability Zone and updates the DNS record for the DB instance to point to the standby DB instance. As a result, you will need to clean up and re-establish any existing connections to your DB instance

When a MySQL DB replica is made available in another Availability Zone tigase is not able to flush the DB instances.

Is there any HTTP API to force tigase to flush DB connections and re-establish them?

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

Pratap Patil wrote:

When a MySQL DB replica is made available in another Availability Zone tigase is not able to flush the DB instances.

Is there any HTTP API to force tigase to flush DB connections and re-establish them?

When the database connection is severed then the connection pool is removed and then there are subsequent attempts are made. Does it happen that even when the database is up (different machine under same DNS)? Does restarting Tigase forces proper resolution of DNS to new machine?

Added by Pratap Patil almost 4 years ago

Here are my detailed observations by experimenting with AWS RDS (MySQL).

  1. First we open start tigase and RDS and they are successfully able to communicate with each other

  2. Then I force reboot RDS with change of MySQL machine (replica), this has a downtime (~ 2mins) which is acceptable.

  3. Once the new machine is up AWS makes the DNS changes and makes the new machine available on same address by updating DNS records. I tested this by accessing MySQL via command line interface.

  4. However, tigase is not able to recover till I do a stop and restart tigase server.

When the database connection is severed then the connection pool is removed and then there are subsequent attempts are made.

If this is the case then it seems like AWS RDS does not sever the existing connection, which may be the reason why they explicitly put it in the document to do a clean-up.

However, is there a way to reset the connections remotely via the HTTP API?

Added by Wojciech Kapcia TigaseTeam almost 4 years ago

Initial error:

AuthRepositoryMDImpl.plainAuth()  WARNING: Couldn't obtain user repository for domain

would suggest that the connection is terminated (Tigase tries to access db, fails therefore creates new pool) however either JDBC driver fails to re-resolve the DNS to new IP.

There is no special command to re-establish the pool via HTTP API.

Added by Pratap Patil almost 4 years ago

Thanks, Wojciech Kapcia. That helps me a lot. I'll debug further.

    (1-5/5)