Can't find root certificate in chain (Let's Encrypt)

Jim Carter
Added about 1 year ago

This is for Tigase-7.1.3-b4482 .

I'm getting in tigase.log.0 the message:
java.lang.RuntimeException: Can't find root certificate in chain!

I can't spot the fault in $TIGASE_DIR/certs/ . In other
forum posts, one user had the cert, key, and intermediate, but no root
cert; I do have the root cert. Another user had the wrong root cert;
mine appears to be the one that signed the intermediate cert and is
bit-for-bit identical with other copies of the same cert. I've tried
(with the same results) two orders: my cert, key, intermediate, root;
and key, cert, intermediate, root.

The listed path is a symlink to a chain file in /etc/ssl/private/ which
is symlinked in several steps to
/etc/certbot/archive/ but the result is the same if
I make a copy in $TIGASE_DIR/certs/ (for fiddling with the key vs cert
order). The same file is used by Apache, and the corresponding separate
key, cert and/or chain files (without key) are used by Postfix, Dovecot
and slapd, with no complaints from the servers or clients.

I've attached the complaint stanza from tigase.log.0 . I hope the needed
clues are not nearby but deleted, but I don't see anything relevant;
I can post the whole log file (2.6Mb, do you want it compressed?)

I've also attached the chain file, with the following edits: the private
key payload is mutilated, and the subject and issuer of each cert is
squeezed in, with altered DNs so my organization is not indexed on
Google. The certs themselves are unaltered, of course. I've also
attached .

Can you see something I'm missing?

Update: As a wild leap in the dark, I constructed a new chain file from
my internal net's PKI using unaltered certs. Order was key, cert,
intermediate, root; the root cert was delimited by BEGIN/END CERTIFICATE
(without TRUSTED); there were text excerpts ahead of each cert and key.
Wonder of wonders, it was loaded error free and the clients can connect
(only onsite, i.e. if they trust the internal net's root cert). This
file is attached with similar alterations as the failing one. The
"only" difference is that the DN of the cert issuer's intermediate cert
has an apostrophe in the failing case, and has only alphanumerics and @
for the internal net PKI, succeeding.

But other users of Tigase have got to be using "Let's Encrypt"
successfully because they don't report my symptom in forum postings.

As my clients are nonlocal and do not have my internal root cert
installed, do you have any suggestions for how to use a host cert signed
by "Let's Encrypt"? I hated that apostrophe from when I junked StartCom
and switched to Let's Encrypt. (5.49 KB) The chain file that fails (8.09 KB) The chain file that gets loaded
tigase.log (1.73 KB) tigase.log Error message stanza (2.13 KB) Tigase configuration

Replies (3)

Added by Wojciech Kapcia TigaseTeam about 1 year ago

First of all we use Let's encrypt certificates on our installation ( and they work without problem).

Jim Carter wrote:

Can you see something I'm missing?

I'd say that you lack CN=DST Root CA X3 root certificate:

$ java -cp ~/dev/tmps/tigase-server-dists/tigase-server-7.1.4-SNAPSHOT-b4484/jars/tigase-server.jar tigase.cert.CertificateUtil -lc  | grep -E "(Issuer|Subject):"
  Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
  Subject: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
  Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.

You could also try using Let’s Encrypt Authority X3 (Signed by ISRG Root X1):

$ java -cp ~/dev/tmps/tigase-server-dists/tigase-server-7.1.4-SNAPSHOT-b4484/jars/tigase-server.jar tigase.cert.CertificateUtil -lc | grep -E "(Issuer|Subject):"
  Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
  Subject: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
  Issuer: CN=ISRG Root X1, O=Internet Security Research Group, C=US
  Subject: CN=ISRG Root X1, O=Internet Security Research Group, C=US
  Issuer: CN=ISRG Root X1, O=Internet Security Research Group, C=US

Added by Jim Carter about 1 year ago

@Wojciech, thanks for the suggestion. I really honestly do have the
DST root cert in the chain file. However, your suggestion led me to a

OpenSuSE package ca-certificates-2+git20170807.10b2785-4.2.noarch
provides three directories (with no content):
/var/lib/ca-certificates/openssl/ /var/lib/ca-certificates/pem/
and /usr/share/pki/trust/ . The actual certificates live in the
last one and are provided by package
ca-certificates-mozilla-2.22-1.1.noarch . SuSE script
/usr/sbin/update-ca-certificates (from the ca-certificates...
package and "inspired by" a Debian script of the same name) munges
the Mozilla originals and puts the result into the
/var/lib/ca-certificates/ subdirs. All the certs are present in
both directories but with subtle differences from each other and from
the original. In particular, the .../openssl/ certs have the
following items appended (showing a diff of the -text representations
in which /openssl/ has these extras, while the certs in the other
two dirs don't have them and have identical -text representations but
are not identical in PEM form)

Trusted Uses:
TLS Web Server Authentication
No Rejected Uses.
Alias: DST Root CA X3
(The "alias" is identical to the CN in the subject/issuer DN.)

When the /openssl/ DST cert is used in Tigase's chain file, Tigase
claims that it cannot find the root cert that signed the Let's Encrypt
intermediate cert. When the /pem/ cert is used, Tigase is happy and
clients can connect. So my problem is now solved.

(A chain file with the ISRG root and the corresponding intermediate cert
also works, on clients which have received firmware updates so they
trust the ISRG root :-)

I tried verifying both chain files using "openssl verify -no-CApath \
-no-CAfile -purpose sslserver -show_chain \
-untrusted letsenc.crt -trusted DSTroot.crt"
OpenSSL of course accepted as valid the provided certs whether the
root cert came from the /openssl/ or /pem/ directories.

It does cross my mind that both the /pem/ and /openssl/ root certs
should probably be considered to be without fault, and so Tigase is
rejecting a valid cert, which is a bug. More likely, the can of worms
which is Java PKI probably can neither parse nor ignore /openssl/
extra attributes, so it rejects the cert as invalid. I have a feeling
that the developer who deals with that will get more insight into
what's happening by referring to this thread, than if I try to create
an issue about it, since I don't do Java and the only thing I know
about Java PKI is sulfurous comments by developers in forum postings.
If your Linux distro doesn't have the /openssl/ style root certs, the
one in the posted chain file can be extracted.

Added by Wojciech Kapcia TigaseTeam about 1 year ago

Thank you for the throughout explanation.

Our Cert Utilities use trusted certificates provided by the JVM (which in turn should use system certificates). So this looks somewhat like JVM is not picking correct paths.