Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Creating JSK for a Java based web server

Daniel Nashed  27 October 2019 23:27:38
This weekend I have been looking into setting up a Flexnet server, which is based on a Java process that needs a certificate.
The server uses the Java Key Store format (JKS) which isn't obvious to create form an existing certificate.

It's easier when you have an existing store. But for a server you usually need a new JKS file.

I found a quite straight forward but not obvious path.

This might be also interesting for other applications. That's why I am posting it here. Having that information would have saved me an hour of research.

-- Daniel

Convert PEM into P12

In many cases you have a PEM file with the key, leaf certificate, intermediate certs and the trusted root.

This can be converted to a pkcs12 (aka as p12) file. This format can be used by many web-servers. But like Domino needs it's own keyring format, Java needs the JKS format.

openssl pkcs12 -export -out lls.p12 -in lls.pem -password pass:mypassword

Import p12 into a new JKS

Once you created a p12, the Java keytool can convert the p12 into a JKS.
This step creates a new JKS file with all the information from the p12 file :-)

keytool -importkeystore -deststorepass mypassword -destkeystore lls.jks -srcstorepass mypassword -srckeystore lls.p12 -srcstoretype PKCS12

That's a quite straight forward way, which can be used to automate the process.

Show Certs

Once you have created the JKS, you can dump all information from the JKS file to check it's all included.

keytool -list -keystore lls.jks -storepass mypassword -v

"Locale" issue on Linux CentOS & RHEL

Daniel Nashed  26 October 2019 18:40:14

"Locale" issue on Linux CentOS & RHEL

I ran into this with Domino on Docker. I noticed that the locale settings have not been detected correctly by Domino when running on CentOS with the German locale.

It turned out that this is a more general issue on Docker with other locales than LANG=C or LANG=en_US.UTF-8

There are differences between CentOS/RHEL 7 and the new version 8 where the language support in glibc has changed!

The issue in version 7 only impacts Docker environments. But with the new language handling in CentOS/RHEL8 this also impacts native applications on Linux.

When the locale is broken you see following error messages when checking the "locale":

export LANG=de_DE.UTF-8


locale: Cannot set LC_CTYPE to default locale: No such file or directory

locale: Cannot set LC_MESSAGES to default locale: No such file or directory

locale: Cannot set LC_ALL to default locale: No such file or directory















CentOS 7 / RHEL7

It took a while to figure out why this is happening. And it turned out that for Docker the CentOS base image does only contain English in the glibc package.

There is a setting in /etc/yum.conf which causes that only the English locale is installed -apparently to save a bit of storage (it's not really much..).

The solution is to remove the following line from /etc/yum.conf


Afterwards to reinstall the glibc-common. In some cases that doesn't work and you need to update it instead

yum reinstall -y glibc-common  
or  yum update -y glibc-common

For our Docker project this means we have to change the setting during the Docker build process and than update the glibc-common package.

In CentOS /RHEL 7 this only affects Docker and I have seen the discussion about this issue in the Docker community.
But I didn't find any solution. So I am documenting it in my blog to make it searchable.


With CentOS/RHEL 8 they have completely changed the way language support in glibc is handled.

This does not only affect Domino on Docker but also natively installed.

There is an interesting article about it on the RedHat website:

They are mentioning a glibc-all-langpacks
which can be used instead of the smaller package glibc-minimal-langpack which is installed by default.

But even when installing this package you don't get all the languages installed. Instead this is the packet which sort of manages the other languages you can install.

There is an alternate solution to install glibc langauge packs (also mentioned in the article) but that's not recommended because it is slower as they say.

So what you need to do when you want to install additional languages is to first install the language packs named for example langpacks-en and than install glibc-all-langpacks

yum install -y langpacks-en langpacks-de glibc-all-langpacks

Only with this combination your locale works as expected and the "locale" command does not complain about missing files.

For Docker we will take care about this in the Docker Project. We will have to see which languages we install by default in future with CentOS 8 (it's not yet supported by Domino).

But for CentOS 7 we will just update the glibc-common package as described above.

-- Daniel

DNUG Domino Day 28.11.2019 in Köln

Daniel Nashed  18 October 2019 10:28:05

DNUG Domino Day 2019 in Köln

Auch in diesem Jahr haben wir wieder einen Domino Day organisiert.
Dieses Jahr im Herzen von Köln am Mediapark am Donnerstag, 28. November 2019.

Im Fokus der Veranstaltung steht das anstehende Release von Domino V11 und auch die neuesten Informationen von HCL zu Lizenzen und Support.
Es geht aber genau so um Neuerungen bei allen anderen Produktion, die in den Themenbereich der Fachgruppe fallen.

Die Anzahl der Teilnehmer ist auf 70 Plätze beschränkt. Daher macht es Sinn sich frühzeitig anzumelden!

Anmelde-Link -->

Ich denke wir haben wieder interessante Themen dabei und es hat sich einiges seit der letzten Konferenz/dem letzten Domino Day getan.
Und wir haben extra auch eine Session zum Thema Lizenzierung und Neuerungen/Änderungen im Bereich Support&Downloads aufgenommen, da ich dazu im meinem Blog und auch offline viele Fragen bekommen habe.

Gegen Ende gibt es vor einer Drink Receiption von TIMETOACT, noch eine Frage-Runde mit HCL, wo Ihr alle offenen Fragen loswerden könnt, die Ihr bis dahin noch nicht beantwortet bekommen habt.

Danke an Christoph Adler und Manfred Lenz als meine Fachgruppen-Kollegen bei der Unterstützung :-)

Ich freu mich viele von Euch zu sehen!



Aktuelle Agenda

09:00 – 09:15 Uhr
FG-Domino: Daniel Nashed, Christoph Adler, Manfred Lenz

09:15 – 10:15 Uhr
Keynote: Strategy & Roadmaps Notes / Domino / Nomad (ggf. auch Sametime / Connections / VoP)

10:15 – 10:30 Uhr

10:30 – 11:30 Uhr
HCL Nomad
Detlev Poettgen, Christoph Adler

11:30 – 12:00 Uhr
HCL Update – Lizenzen, FlexNet (Support & Downloads) & Co.
Uffe Sorensen – HCL

12:00 – 13:00 Uhr

13:00 – 14:00 Uhr
Notes V11 & VOP 1.0.8 – What’s new
Manfred Lenz, Christoph Adler

14:00 – 14:30 Uhr

14:30 – 15:30 Uhr
Domino V10 & 11 Session – What’s new & Lessons learned & Docker
Daniel Nashed

15:30 – 16:15 Uhr
AppDevPack & DQL – What’s new & Lessons learned
Stefan Neth

16:15 – 17:00 Uhr
Fragen & Antworten
DNUG-Fachgruppe & HCL

ab 17:00 Uhr
Ausklang & Drink Reception
sponsored by TIMETOACT

Donnerstag, 28.11.2019
9 – 17 Uhr

Im Mediapark 5
Raum Barcelona, 1. OG
50670 Köln

    CentOS 8 Released

    Daniel Nashed  5 October 2019 15:55:47
    RHEL 8 is available for a while. And traditionally CentOS takes a couple of month before it is also updated to the same code base.

    I have downloaded and installed CentOS 8. The first version was the full version. There wasn't a minimum base image yet.

    Be aware that neither CentOS 8 nor RHEL 8 nor SLES 15 SP1 are currently supported!
    There are even packages which have older versions in CentOS 8 than the last updated of CentOS 7!
    For example I tried to install the latest Docker CE version 19.09. It needs a newer version than what is currently shipped on CentOS 8.
    So I would stay with your current releases for now!

    Of course I have looked into SLES 15 SP1, RHEL 8 and CentOS 8 with Domino to see if it works.
    I don't think it makes sense to look into Domino 10 support for those platforms. I would expect an updated Linux version support for Domino 11.

    On the Docker side I ran into an issue preparing demons on a different machine.
    When you pull a centos:latest image today, you will get centos 8 which isn't working with the current dockerfiles which ships with the master version of the Docker project.
    We have updated the develop branch of the project already with a changed dependency:

    FROM centos:7 will continue provide the latest image of CentOS 7.

    So the project has been updated with another currently experimental dockerfile to build Domino on CentOS 8 for testing.

    But I can only recommend that you stay on CentOS 7 for now because that's the tested and supporter version.

    This is true for native and also Docker image versions!

    -- Daniel

    HCL Nomad V1.0.4 released

    Daniel Nashed  5 October 2019 15:32:28
    There is a new iPad Application which has been released.
    This is the first version from HCL and Nomad V1.0.4 replaces the offering known as IBM Domino Mobile Apps.

    The new version also comes with some interesting new features

    - Open your personal mail file in Nomad
    -> this was prevented by internal policy and is now allowed by default
    - GPS geo location support via LotusScript
     --> this will be available in the Notes 11 designer
    - Mobile Device Management pre-configuration
    - Free panaganda MarvelClient for iOS integration!

    - And there is also an option to disable DNS lookups to improve compatibility with some VPN solutions.
    - @platform([specific]) now returns the following type of text-list: iOS; 13.1; iPad; iPad7,5

    There is an interesting blog post from Andrew Mandy and Andrew Davis providing also information about what is coming next :-)

    Specially the free Marvel Client (MC) integration is great.
    MC is integrated into Nomad 1.0.4 and allows you to check and update your Nomad clients.

    In combination with the MDM configuration profiles this means zero manual configuration.

    I have online-updated my MC of my existing installation and see my iPad in the MC reporting database.

    There is a FAQ regarding MarvelClient for iOS, whch you might want to check.

    Well done integration! This is really what we need!

    Congrats & Thanks HCL and panagenda

    iOS 13 Native Mail App Issues with Traveler

    Daniel Nashed  3 October 2019 15:41:43
    iOS 13 introduced a couple of changes. Some of them are done with good intention, but they broke existing functionality.
    There are currently two issues that are known. Both are described in detail in the following technote:

    The technote will be updated with new information about the known issue and also new issues that might arise with iOS 13.

    The second issue just occurs when you have calendar ghosting disabled, which is enabled by default.
    But the issue with the duplicate sent mail folder entries cannot be worked around.

    Beside those two known issues, Traveler should work with iOS 13 and the native mail app.
    You should updates to iOS 13.1.2 which is another update in a short time we got for iOS 13.
    "13" isn't a good number of Apple as it sounds...

    -- Daniel

    1) Duplicate Sent folder entries

    As of iOS/iPadOS 13.0, Apple devices add an entry to the Sent folder for any emails sent from the Mail app.
    When the Sent folder is synced, the server entry is added and the device does not remove the original, resulting in a duplicate. HCL development has opened an Apple bug for this issue (FB7337231).

    Workaround: No workaround is available

    2) Accepting a meeting invitation from the iOS device does not send the response to the server

    If ghosting is disabled on the Traveler server, responding to a calendar notice from an iOS 13 device does not send the response.
    The meeting accept is reflected on the app but not in the user's notes calendar.  
    iOS 13.0 and 13.1 do not send MeetingResponse requests to the Traveler server unless the event is ghosted to the calendar.
    HCL Development has opened an Apple bug for this issue (FB7328175).

    Ghosting is enabled at the Traveler server by default.  Check the Traveler server notes.ini parameters for NTS_CALENDAR_GHOSTING_SYNCML and NTS_IOS_CALENDAR_INITIAL_GHOST.  If found, make sure that they are set to true.

    RNUG -- Russian Notes User Group Event in Moscow

    Daniel Nashed  2 October 2019 08:48:42
    This is going to be a very special event! I am really looking forward to be there next week!
    I have never been to Moscow and I am looking forward to the event and also visiting the city.

    The venue looks great and there are many known speakers including fellow IBM Champions/HCL Masters from around the world.

    My sessions will be about Domino Performance and also Domino on Docker.

    And we are having a Domino on Linux Round Table session as well to get feedback from the Russian market.

    I have blogged about some tests which I have done with a local Linux which seems to be quite popular -->

    So this will be an exiting event for participants and also us speakers!

    -- Daniel

    Image:RNUG -- Russian Notes User Group Event in Moscow

    Creating Internal use X.509 Certs

    Daniel Nashed  28 September 2019 13:58:30
    For one of my test servers I needed a proper certificate. A self-signed cert works in many cases. But creating your internal CA has benefits. You can have the CA root trusted in your brwoser etc.
    I needed a certificate for a local test server today and used the script I developed for the Docker project.

    A while ago I updated the script to add also additional SANs (Subject Alternate Names) and it will also add the SANs to a CSR request if you use the script with an external CA.
    Even when just generating a certificate with just a DNS name, this name also needs to be added to the SAN.
    This was implemented from the beginning but now you can add more SANs.

    After you configured the script, generating a proper certificate is just invoking this script.
    The CA directory contains the CA root that you add to your browser afterwards.

    Here is the example and here is the link to the script -->

    The script creates the private key, generates the CSR, depending on the configuration the reuqest is signed and everything is merged together into a single PEM.
    That PEM is imported into a matching keyring file -- if the kyrtool is installed and you are running as "notes".

    -- Daniel

     ./ "traveler-nashcom-loc" "/CÞ/O=NashCom/CN=traveler.nashcom.loc" "traveler.nashcom.loc,trav2.nashcom.loc,trav2.nashcom.loc"

    (Using config file /local/cfg/certmgr_config)
    Generating key [/local/certmgr/key/traveler-nashcom-loc.key]
    Generating RSA private key, 2048 bit long modulus
    e is 65537 (0x10001)
    Creating certificate Sign Request (CSR) [/local/certmgr/csr/traveler-nashcom-loc.csr]
    Removing [/local/certmgr/pem/traveler-nashcom-loc_all.pem]
    Signing CSR [/local/certmgr/csr/traveler-nashcom-loc.csr] with local CA
    Signature ok
    Getting CA Private Key
    Removing [/local/certmgr/csr/traveler-nashcom-loc.csr]

    Keyfile /local/certmgr/kyr/traveler-nashcom-loc.kyr created successfully

    Using keyring path '/local/certmgr/kyr/traveler-nashcom-loc.kyr'
    Successfully read 2048 bit RSA private key
    SECIssUpdateKeyringPrivateKey succeeded
    SECIssUpdateKeyringLeafCert succeeded

     traveler-nashcom-loc -> OK
     KeyLen       :  2048 bit
     Subject      :  /CÞ/O=NashCom/CN=traveler.nashcom.loc
     DNS NAME     :  traveler.nashcom.loc, DNS
     Valid Until  :  Sep 25 10:12:07 2029 GMT

      HCL Traveler 10.0.1 FP2 Released

      Daniel Nashed  20 September 2019 15:09:12
      The first "HCL" Traveler updated has shipped.

      This version contains updated APNS push certificates, because the current shipped cert expires mid of October.

      Beside that fix there are a couple of other fixes which might be relevant for your environment.

      Here is a list for all changes. There are no big surprises. But there also some backend changes for the updated Verse client which is coming soon.

      IMPORTANT: Required reading for Administrators - Upgrading from IBM Verse for iOS to HCL Verse 10.0.7 for iOS  

      Here is a technote for important changes with the upcoming first HCL Verse release.
      They are changing the name of the application which is has some impact.

      I got also questions about iOS 13 support. There are no known issues with Traveler and

      HCL did only test those two versions. Earlier versions should still work but it is highly recommended to update!

      Traveler 10.x is supported on Domino 9 and the installer detects the underlaying Domino version.
      But it is still recommended to update to Domino 10.0.1 with a current fixpack (currently FP3).

      The new Traveler version is available on the HCL Flexnet Download portal.

      Here is a search link:

      And here are the file-names and the SHA256 hashes.

      I am currently looking a lot into those download options and filenames because of Docker and we are adding the new version to the software.txt file..

      -- Daniel

      HCL Traveler v10.0.1 FP2 for Linux M

      SHA256 CheckSum        4669eb49ad354d7bc8f67bf02693d6d1eccdf69b3a0a0317657d14370c793109

      HCL Traveler v10.0.1 FP2 for Windows ML
      SHA256 CheckSum        a8ea719ce0ede272b51b23fdb43b3dbf7c31839534b4f69796851063e21727f9


      Docker Support for Domino 10.0.1 FP3

      Daniel Nashed  12 September 2019 22:51:32
      The first HCL Software packages are released. Domino 10.0.1 FP3 is the first download which is only available from HCL.
      So we added support for HCL Flexnet downloads.

      The file names for the software have different names for older fixes and Domino itself, than the files from IBM Passport Advantage and FixCentral Downloads.
      So we added support for more than two file names per software file. The first file is now the HCL Flexnet download, which will also be used to generate a download hint.


      10.0.1FP3           [NA]  Domino_10.0.1FP3_Linux64.tar  (-)

      I didn't find a direct link to a software download that worked. If you select the link you get, it will not work for someone else. But at least the search will return a single file.
      The changes allow to build Domino 10.0.1 FP3. But because some partners and customers still did not mange to download FP3, I did not make FP3 the default version yet.

      If you want to build with FP3, download the software and run

      ./ domino 10.0.1 FP3

      This will build an image with FP3 but does not mark it as latest yet.

      We also added the Domino 11 beta version to the software list and if you have access to the beta version, you can build a Domino 11 beta image.
      The Docker project is prepared for the beta and you will see HCL instead of IBM branding.

      To build Domino 11 Beta1 images run:

      ./ domino 11.0.0.beta1

      All those changes are currently submitted to the develop branch.

      -- Daniel


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]