Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino 12 Beta 1 - HTTPS Review & Ratings

Daniel Nashed  January 24 2021 07:03:46 AM

Now that Domino V12 Beta1 is available, I took the opportunity to look into the current state of TLS and ciphers in Domino.

Rating of the results

You can see from the rating that we are at a very high score.
Even without TLS 1.3 support, this already looks great!
And looking into details you see, that you could switch to ECDSA keys already for most web servers.

If are are interested in the details and want to test your own server -- even if not exposed to the internet .. continue reading and see the details from the commented results.

-- Daniel


Image:Domino 12 Beta 1 - HTTPS Review & Ratings




Test Setup

- Requested a new TLS certificate from Let's Encrypt using the new CertMgr integrated into Domino V12.
- After receiving the certificate, I redirected HTTP to HTTPS (if you have HTTP enabled HSTS isn't working).
- I set the following notes.ini variables to enable HSTS and OCSP for Let's Encrypt production:
set config OCSP_RESPONDER=http://ocsp.int-x1.letsencrypt.org/
set config SSL_ENABLE_OCSP_STAPLING=1
set config HTTP_HSTS_INCLUDE_SUBDOMAINS=1
set config HTTP_HSTS_MAX_AGE=17280000

Test Tool "testssl.sh"

Even my server is reachable over the internet, I have been using a very cool shell script available on GitHub -- which will also work for internal hosts.
Beside that I have looked into a SSL Labs test result, confirming my findings.

https://github.com/drwetter/testssl.sh

yum install -y git bind-utils
mkdir -p /local/github
cd /local/github/
git clone https://github.com/drwetter/testssl.sh.git
cd testssl.sh



Details looking into testssl.sh output

Let's have a closer look of the current state.

Domino V12 introduces support for ECDSA crypto. So I requested a ECDSA certificate from Let's Encrypt using CertMgr, creating a ECDSA NIST-P 256 key.

Note:
  A  ECDSA NIST-P 256 is comparable to a RSA 3072 bit key. This should be fully sufficient and provides good security with best performance.
  The ratings for NIST-P 256 and NIST-P 384 are the same in my tests.
  A  NIST-P 384 key would be comparable to a RSA key size of 7680 bit!
 So my recommendation would be to stay with NIST-P 256!


Ciphers for ECDSA

When using a ECDSA key the standard cipher list is ignored and the following two ciphers are enabled automatically -- and all other ciphers are disabled.
So there are no additional settings you have to configure. And by the way TLS 1.0 is disabled by default in Domino V12 Beta1 already.

xc02c   TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
xc02b   TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256


You can see from the client simulation in the detailed report below, that those ciphers are available in most platforms.

Only very old clients don't support them.

New curves supported

Domino V12 supports two new curves X25519 and X448 which are leveraged with ECDH.

Specially Curve X25519 is an important addition and you can see in the test report, that most client simulations negotiated a connection leveraging X25519.

Curve X25519 is not just faster, it is also an independent implementation, not influenced by the NSA -- in contrast to the NIST-P curves.
It has become "the de facto alternative to P-256" also because it's an independent implementation and is the preferred Curve used for Domino.

(See details here: https://help.hcltechsw.com/domino/earlyaccess/wn_new_curves_ecdhe.html)

Detailed look into the test results


You can see that the testssl.sh script delivers comparable results to the SSL Labs website.
The big benefit is that it works also for intranet hosts, without any external connections.
And you can get the output different formats.

Running the scan for HTTPS on the standard is simple. You just specify the hostname:


./testssl.sh pluto.csi-domino.com


The scan is also faster and uses your locally installed OpenSSL and DNS lookup tools.

Let's have a look into the details.

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (124f6f5 2021-01-22 12:21:01 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~183 ciphers]
 on localhost:./bin/openssl.Linux.x86_64
 (built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")


 Start 2021-01-24 06:27:14        -->> 159.69.147.90:443 (pluto.csi-domino.com) <<--

 rDNS (159.69.147.90):   pluto.csi-domino.com.
 Service detected:       HTTP


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)-- Only TLS 1.2 is offered
 TLS 1.3    not offered and downgraded to a weaker protocol
 NPN/SPDY   not offered
 ALPN/HTTP2 not offered

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            not offered
 Strong encryption (AEAD ciphers) with no FS       not offered
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK) -- Forward Secrecy is offered


 Testing server's cipher preferences

 Has server cipher order?     yes (OK) -- Server provides the cipher order
 Negotiated protocol          TLSv1.2
 Negotiated cipher            ECDHE-ECDSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)-- openssl negotiated the right cipher.
 Cipher per protocol

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (IANA/RFC)
-----------------------------------------------------------------------------------------------------------------------------
SSLv2
 -
SSLv3
 -
TLSv1
 -
TLSv1.1
 -
TLSv1.2 (server order)
 xc02c   ECDHE-ECDSA-AES256-GCM-SHA384     ECDH 256   AESGCM      256      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
 xc02b   ECDHE-ECDSA-AES128-GCM-SHA256     ECDH 256   AESGCM      128      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

The server cipher order lists the two ciphers mentioned above, with a favor to the more secure cipher

TLSv1.3
 -


 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4

 FS is offered (OK)           ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256
 Elliptic curves offered:     prime256v1 -- Curve offered is what we expect


 Testing server defaults (Server Hello)

 TLS extensions (standard)    "renegotiation info/#65281" "status request/#5"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID: yes
 TLS clock skew               Random values, no fingerprinting possible
 Signature Algorithm          SHA256 with RSA
 Server key size              EC 256 bits (curve P-256)-- yes we got a ECDSA NIST-P 256 key
 Server key usage             Digital Signature
 Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication
 Serial / Fingerprints        048635D3302F808EBEFDC34DF8244BDAEDD5 / SHA1 C264144A2B2FAE743441A4B51BC46722D8A1CDEC
                              SHA256 C69718BF1A8BFD1436F4173659C21295E8232293D3918017280C2A4EF30BCA4D
 Common Name (CN)             pluto.csi-domino.com
 subjectAltName (SAN)         pluto.csi-domino.com
 Trust (hostname)             Ok via SAN and CN (same w/o SNI)
 Chain of trust               Ok
 EV cert (experimental)       no
 Certificate Validity (UTC)   89 >= 30 days (2021-01-24 05:26 --> 2021-04-24 06:26) -- You can see that I just got the new certificate requested, because I had to compare P-256 and P-384 ;-)
 ETS/"eTLS", visibility info  not present
 Certificate Revocation List  --
 OCSP URI                     http://r3.o.lencr.org -- our OCSP configuration worked
 OCSP stapling                offered, not revoked -- certificate looks good
 OCSP must staple extension   --
 DNS CAA RR (experimental)    not offered
 Certificate Transparency     yes (certificate extension)-- Let's Encrypt offers certificate transparency, which is the newer standard replacing "HTTP Public Key Pinning (HPKP)"
 Certificates provided        3
 Issuer                       R3 (Let's Encrypt from US)
 Intermediate cert validity   #1: ok > 40 days (2021-09-29 21:21). R3 <-- DST Root CA X3
                              #2: ok > 40 days (2021-09-30 16:01). DST Root CA X3 <-- DST Root CA X3
 Intermediate Bad OCSP (exp.) Ok


 Testing HTTP header response @ "/"

 HTTP Status Code             401 Unauthorized  WWW-Authenticate: Basic realm="/"
 HTTP clock skew              +1 sec from localtime
 Strict Transport Security    200 days=17280000 s, includeSubDomains
 Public Key Pinning           --
 Server banner                Lotus-Domino
 Application banner           --
 Cookie(s)                    (none issued at "/") -- maybe better try target URL of 30x
 Security headers             X-Content-Type-Options: nosniff
                              Cache-Control: no-cache
 Reverse Proxy banner         --


 Testing vulnerabilities

All vulnerability test passed  -- But this is not new in Domino V12:

 Heartbleed (CVE-2014-0160)                not vulnerable (OK), no heartbeat extension
 CCS (CVE-2014-0224)                       not vulnerable (OK)
 Ticketbleed (CVE-2016-9244), experiment.  not vulnerable (OK), no session ticket extension
 ROBOT                                     Server does not support any cipher suites that use RSA key transport
 Secure Renegotiation (RFC 5746)           supported (OK)
 Secure Client-Initiated Renegotiation     not vulnerable (OK)
 CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)
 BREACH (CVE-2013-3587)                    no gzip/deflate/compress/br HTTP compression (OK)  - only supplied "/" tested
 POODLE, SSL (CVE-2014-3566)               not vulnerable (OK), no SSLv3 support
 TLS_FALLBACK_SCSV (RFC 7507)              No fallback possible (OK), no protocol below TLS 1.2 offered
 SWEET32 (CVE-2016-2183, CVE-2016-6329)    not vulnerable (OK)
 FREAK (CVE-2015-0204)                     not vulnerable (OK)
 DROWN (CVE-2016-0800, CVE-2016-0703)      not vulnerable on this host and port (OK)
                                           no RSA certificate, thus certificate can't be used with SSLv2 elsewhere
 LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK): no DH EXPORT ciphers, no DH key detected with <= TLS 1.2
 BEAST (CVE-2011-3389)                     not vulnerable (OK), no SSL3 or TLS1
 LUCKY13 (CVE-2013-0169), experimental     not vulnerable (OK)
 Winshock (CVE-2014-6321), experimental    not vulnerable (OK) - doesn't seem to be IIS 8.x
 RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)


 Running client simulations (HTTP) via sockets

 Browser                      Protocol  Cipher Suite Name (OpenSSL)       Forward Secrecy
------------------------------------------------------------------------------------------------
 Android 4.4.2                TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Android 5.0.0                TLSv1.2   ECDHE-ECDSA-AES128-GCM-SHA256     256 bit ECDH (P-256)
 Android 6.0                  TLSv1.2   ECDHE-ECDSA-AES128-GCM-SHA256     256 bit ECDH (P-256)
 Android 7.0 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Android 8.1 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Android 9.0 (native)         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Android 10.0 (native)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Chrome 74 (Win 10)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Chrome 79 (Win 10)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Firefox 66 (Win 8.1/10)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Firefox 71 (Win 10)          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 IE 6 XP                      No connection
 IE 8 Win 7                   No connection
 IE 8 XP                      No connection
 IE 11 Win 7                  TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 IE 11 Win 8.1                TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 IE 11 Win Phone 8.1          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 IE 11 Win 10                 TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Edge 15 Win 10               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Edge 17 (Win 10)             TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Opera 66 (Win 10)            TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Safari 9 iOS 9               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Safari 9 OS X 10.11          TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Safari 10 OS X 10.12         TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Safari 12.1 (iOS 12.2)       TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Safari 13.0 (macOS 10.14.6)  TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Apple ATS 9 iOS 9            TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Java 6u45                    No connection
 Java 7u25                    No connection
 Java 8u161                   TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Java 11.0.2 (OpenJDK)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 Java 12.0.1 (OpenJDK)        TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256)
 OpenSSL 1.0.2e               TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     256 bit ECDH (P-256) --> OpenSSL 1.0.2 is not using the newer X25519 curve we prefer
 OpenSSL 1.1.0l (Debian)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 OpenSSL 1.1.1d (Debian)      TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)
 Thunderbird (68.3)           TLSv1.2   ECDHE-ECDSA-AES256-GCM-SHA384     253 bit ECDH (X25519)

You can see that only very very old OS/Browser/Software combination doesn't connect with those settings.
And I would see it more as a feature that really old software, which isn't patched any more cannot connect to a server.
Many clients already support the X25519 curve. All others use NIST-P 256 which is perfectly OK


 Rating (experimental)

 Rating specs (not complete)  SSL Labs's 'SSL Server Rating Guide' (version 2009q from 2020-01-30)
 Specification documentation  https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
 Protocol Support (weighted)  100 (30)
 Key Exchange     (weighted)  100 (30)
 Cipher Strength  (weighted)  90 (36)
 Final Score                  96
 Overall Grade                A+

The rate we get is in line what SSL Labs returned. Not only from the overall rate, but also from the detailed ratings.
So a final score of 96 from 100 points looks very good to me!

 Done 2021-01-24 06:27:56 [  43s] -->> 159.69.147.90:443 (pluto.csi-domino.com) <<--

The whole test took 43 seconds. You can run this against all your internal servers and see what you get back.
This can be run also against SMTP servers and other services!

Here is the command line for SMTP/STARTTLS:
./testssl.sh --starttls=smtp domino.nashcom.de:25




Notes, Domino, Traveler V12 Beta1 released

Daniel Nashed  January 22 2021 12:17:54 AM


Notes, Domino and Traveler V12 Beta 1 is available

Here is the official blog post..

https://blog.hcltechsw.com/domino/start-your-engines-hcl-domino-v12-beta-is-here-are-you-ready/

Download

The software is available on Flexnet like we had the early access code drops. But now we got the clients and server installers in addition to the HCL Domino Docker image.

Documentation

In contrast to earlier years, the documentation isn't in a PDF.
The documentation has been added to the standard documentation and you see the new features in "what's new sections" as part of the standard documentation.
I spoke with customers and partners who have been confused about finding the trial image documented next to the new functionality for example.


One of my favorite features: Automated Certificate Management

Certificate Management has been already part of the early access code drops.
With Beta1 Let's Encrypt / ACME DNS-01 challenges have been added.

So you can request certificates from and for servers, which have no incoming HTTP connection.
Instead a DNS-TXT record is used for validation (see details here --> https://letsencrypt.org/docs/challenge-types).

You can get free certificates from Let's Encrypt and other providers supporting the ACME standard (e.g. ZeroSSL: https://zerossl.com/features/acme/,  BuyPass: https://www.buypass.com/ssl/products/acme).

Currently there are two reference implementations vor DNS Automation for Cloudflare, Inc. and Hetzner GmbH here in Germany.
Beta 1 already includes a framework for implementing your own integration for your favorite provider.

If you are interested Certificate Management, you should really look into the features in this first beta and provide feedback.

And of course also test the other new functionality Thomas has mentioned.


Domino Docker Project Update

I have been already testing Domino V12 Beta1 with our Docker project and updated the images for Domino and Traveler.

The new version names are already included in the develop branch --> https://github.com/IBM/domino-docker/tree/develop

So you could also use the community image already, when you download the Domino Linux web-kit to get your image build locally.


Enjoy the beta and hopefully we see us in the Beta forum.

-- Daniel

Introducing Domino Container Script for Podman an Docker

Daniel Nashed  January 17 2021 05:31:08 PM

This is a brand new script, which is the logical extension to the Nash!Com Domino Start script.
A while ago I introduced Docker support for the start script inside the Domino container.
It comes with an entry-point script we are also leveraging in out Domino Docker Github community script.

This new script is intended to manage and run your Docker and Podman containers.
It includes also a systemd script to run a Domino server on Podman in production using the new systemd options introduced in Podman 1.7+.
Beside that it includes a simple framework to build and manage your own add-on images


Components & functionality

- Predefined systemd script for running a Domino server on Podman 1.7+
- Support for Docker 18+
- sysconfig script to configure your container
- Environment file to configure the server inside the container
- domino_container script as your single point of service for all type of operations
- Build environment for custom images on top of the HCL Domino image or 3rd party images like your Domino on Docker project

It will allow to manage your server from the domino_container script including jumping into the container from command-line.
And you can also leverage it to update your container even running with systemd on Podman.

I moved one of my production servers to a new provider and decided to use Podman with this systemd scripts.
But also when using Docker this script allows you to completely manage your server.


This will replace the management script in the Docker project over time.
The script is more flexible and also provides a simplified way to build add-on image.


build_image directory
With the new approach each image has it's own directory including a Container file (Podman uses this name instead of Dockerfile) and a build.cfg.
You can clone this directory and define your own add-on images which you can build from the directory using the domino_container script.
The build logic is in the script and the install logic for the image is in the install_dir.

The out of the box example is already prepared to install server-tasks and extension managers. It can be used as a template and be customized and extended for any type of customer or partner image.

Installation
Like the start script, the container_script has it's own install script (install_domino_container), which writes the files to the right directories.
Once you installed it, you can run "domino_container" with all the options specified below.
I hope this provides a quick overview and is what you need to run your Domino servers on Docker & Podman.

Any feedback is welcome. It's a first version...
-- Daniel

Current commands

domino_container help
(Using config file /etc/sysconfig/domino_container)

Nash!Com Container Script Version 1.0.0
(Running on Podman Version 2.0.5)

Usage: domino_container

       start | stop | restart | status | statusd | inspect | console | config | env
       domino | bash | update | attach | remove | removeimage | cleanup | port
       install | load | build | version | help

start [live]    start an existing container (the 'live' option shows start script output)
stop  [live]    stops container (the 'live' option shows start script output)
restart         restarts or starts the server
status          shows container status (running, exited, notexisting)
statusd         shows the systemd status
info            shows status and basic information about container and image
inspect         shows detailed information about container and image
console         open Domino console inside container
logs            shows container logs (output from entry point script/start script)
attach          attach to entry point script
domino          pass a command to start script (e.g. domino nsd)
bash [root]     invokes a bash in the running container. optionally run as root instead of notes user
remove|rm       removes the container (if not running)
removeimage|rmi removes the current container (you have to remove the container first)
config|cfg      edit configuration
update          updates the container if referenced image has changed (stops Domino, stops the container, runs a new image)
install         install Podman
load            load HCL Domino Docker image
build           builds a current image -- even image tags might not have changed to ensure OS patches are installed
port            show used tcp/ip ports for container
enable|on       enable systemd service
disable|off     disable systemd service
clean           cleanup container and systemd if configured
env             edit environment file
version         shows script version information
help            you already figured out what help does ;-)



Domino on Linux Start Script Sametime 11 support

Daniel Nashed  January 17 2021 11:28:42 AM
Now that we have the Sametime Premium Meeting Server, there are more customers looking into Domino on Linux.

In earlier days I had some issues getting the start script in combination with SLES working.
So ST wasn't never on my support list.  The SLES issue was going way when switching from init.d scripts to systemd and replacing the SUSE specific "rc" logic -- which caused the issues earlier. I removed that logic completely, because systemd doesn't need it on SUSE platforms.

With this change I started to look into Sametime on Linux again and I have now added Sametime Community server support to the start script.
The new version 3.5.0 will be available on our Domino Docker github repo in the develop branch first.

The missing part for ST 11 was adding the lib path for two specify ST directories.

  LD_LIBRARY_PATH=$Notes_ExecDirectory:$LD_LIBRARY_PATH:$Notes_ExecDirectory/STOpenSSL:$Notes_ExecDirectory/sticc

Those directories are now added for all installations and should not cause an issue on servers where no ST is installed.

I also added a "resetstlogs" to include the ststart script "resetlogs" functionality.

So the Domino service will leverage the start script and you can take benefit on a central entry point for all operations like start/stop/console, .. including the "resetstlogs" command.

-- Daniel

    Y2K21 Notes property box - Created date is empty

    Daniel Nashed  January 4 2021 09:11:42 PM

    The following has been already reported to HCL development and there is a SPR# NNAIBWX3MD.

    This came up quite early in the HCL Ambassador discussion channel over the holidays.


    The property box is not showing a created data for documents created in 2021 and later unless there is a $Created date item like in incoming SMTP mails.

    This is just the properties box not showing the date. The functionality is completely working fine on all currently shipping versions and client platforms.


    You can still see it in for example formulas with @created, in NotesPeek or any other tool you might have installed.

    All functionality like replication etc. is working as expected. This is really just the properties box not showing the date.



    Image:Y2K21 Notes property box - Created date is empty

    Thomas Hampel just confirmed it in his blog and I would expect a technote to be published soon.

    https://blog.thomashampel.com/blog/tomcat2000.nsf/dx/y2k21-problem-in-notes-property-box-created-date-is-empty.htm

    -- Daniel


    Docker 20.10 and Docker Desktop 3.0 released

    Daniel Nashed  December 31 2020 03:38:04 PM


    I never said Docker is completely dead, even I see today better integration and more features in Podman.
    This new release of Docker CE is helping specially on CentOS 8 and you don't need to work-around compatibility issues with an older version of containerd shipped with CentOS 8.

    At the same time Docker updated Docker desktop to a new major version -- which also includes Docker version 20.10.

    The Docker Community project will continue to use Docker as the build platform, but also supports Podmand.

    I have just installed both today. Docker Desktop 3.0 did hang on installing the new version in my Win10 WSL2 environment.
    But re Windows reboot helped to get it installed -- actually this was a tip in a forum so I might not be the only one running into this issue.


    On CentOS 8, I had to uninstall Docker completely as the Docker official instructions reference --> https://docs.docker.com/engine/install/centos/

    In addition to their recommendations I also had to remove the docker0 network like this:

    firewall-cmd --permanent --zone=trusted --remove-interface=docker0

    So my installation steps have been:

    1. Uninstall docker-ce via "yum remove ..." as described on the CentOS installation page
    2. Remove the interface from the configuration -- if you added it earlier to get it working like me
    3. Reboot
    4. Use the new script to install Docker 20.10.

    It's working fine now on my CentOS 8 machine without any "hacks".

    And in my case I removed all existing VMs and images. The container data was on /local/.. and the container description was in my scripts.
    So I was able to uninstall Docker and remove all container data as suggested on the Docker homepage.
    Just uninstalling the software and removing the docker0 network should already work fine.

    There are enhancements like rootless Docker containers (not inside the container which is already a best practice for a long time but running the container without root permissions).

    I am not a big fan of that, because it still might cause issues. So having systemd start with root permissions isn't a big issue.

    There are a couple of other improvement, which might be interesting. I got the following link with a good summary https://medium.com/nttlabs/docker-20-10-59cc4bd59d37.

    Still Docker 20.10 is only adding mainly functionality already available in other environments like Podman.

    Docker is a great platform to start with. And specially Docker desktop integrated with Win10+WSL2 has some benefits.
    Also the documentation from Docker is more detailed.
    So as a starting point Docker is still a good platform to work with.

    Our last "Docker" workshop has been using CentOS with Docker 19.x. Sounds like I should now prepare the lab with CentOS 8 and Docker 20.10 ..

    -- Daniel

    SPAM from servers with IN-ARPA *.cloudapp.azure.com

    Daniel Nashed  December 31 2020 10:55:27 AM
    Today I got a lot of SPAM today gray-listed from hosts in those domains.
    The worrying part is that this is split on different data centers apparently.
    So I got mails from the following sub-domain which have been listed in my SpamGeek with a high gray-list value and now got an explicit SPAM rating.
    Not expecting to have any customers sending me mails from there... But if they do, they need some really positive other rating based on content ..

    Moving applications into the cloud at providers with shared services can cause that your outgoing mail traffic is impacted by others misbehaving on the same shared service.

    I looked up one of the sub domains in SenderBase. None of those hosts have been rated with a "Good" rating -- currently 7 hosts are rated "Poor".
    And I know companies setting their CISCO ESA (which is using the SenderBase) to even not accept "Neutral" reputation (which is something I would never recommend).

    CISCO ESA (aka. IronPort) are one of the leading solutions used by many larger customers.
    And if you are having a "Poor" rating you are in serious outgoing e-mail trouble!

    -- Daniel

    *.northcentralus.cloudapp.azure.com
    *.northeurope.cloudapp.azure.com
    *.eastus.cloudapp.azure.com
    *.westeurope.cloudapp.azure.com


    Image:SPAM from servers with IN-ARPA *.cloudapp.azure.com


    Domino on Podman in production

    Daniel Nashed  December 24 2020 09:01:28 AM

    This week and I am migrating my Domino server to a new provider updating to CentOS 8.

    Because I am working a lot with Domino in the Docker/Podman/Kubernetes space, I am looking into it also for my own server.


    The big benefit is that I can apply my private "NashCom" image derived from the standard image to all servers needed without installation. And switching versions is means just a restart.

    My own image will be based on the Docker Project version and use the add-on functionality to build your own image on top of it.

    In my case to add all my tools starting with the "nsh.." prefix.. And also the Domino V12 DSAPI filter for Let's Encrypt/ACME integration.



    Podman


    I am looking into the latest Podman (
    http://docs.podman.io) version that comes with CentOS 8.
    CentOS 8 has better Podman support than Docker support


    And in contrast to Docker, Podman is not using a daemon. For Docker I added my management script to our Docker project -->
    https://github.com/IBM/domino-docker/blob/master/docs/management.md
    With Podman I will extend this concept and make it a full start script, which can be used in combination with systemd.


    Podman has added systemd functionality and you can generate your own systemd services with Podman -->
    https://www.redhat.com/sysadmin/podman-shareable-systemd-services
    I have looked into it and will bring this concept into a Domino start script that is Podman aware.



    Domino Start Script support


    The Domino start script itself runs inside the container  already. So you have most start script options available inside the container on all container platforms.

    This new functionality is intended for the outside host part in combination with Podman and systemd.


    Probably I will end up with a separate script based on the current management script and just integrate it with the classical start script.

    So that you could for example pass start script configuration variables from the host to the container running in Podman.


    The management script is already a kind of Docker/Podman start script. But it wasn't design for systemd integration yet.

    Not sure if I will keep it in the Docker project or add it to the start script to make it automatically installable etc.

    My start script is also part of the Docker project. So it will be probably just moved at some point.


    Not sure yet where it will be located. But it will also be able to work with the HCL out of the box Domino Docker image, not just with the Domino Docker project image.



    HCL Domino image support


    Both images are moving into the same direction anyway. We are adding new Domino V12 functionality into our Domino Docker project while the HCL image is taking ideas from our script and bring them into code.


    For example the "one touch scriptable setup" is now part of Domino V12 Early Access and works cross platform, not just for Linux or containers alone!

    This isn't based on scripts or Java attached to Domino. It's part of the server code.


    So as soon new functionality is added to the official HCL image, we adopt those new features and replace existing features in our script for newer Domino versions.

    On the other side we will still continue to complement what is there is additional functionality needed in the community.


    For now in the early access phase we don't have access to the Linux web kit as the base. But this will change as soon the betas are available early next year.

    So you will see our Domino Docker project to support Domino V12 right from the start of the first beta.


    Yes this isn't intended for production at that stage. But you can bet on me still upgrading at least a cluster mate to V12 early on.

    In fact I am running Domino V12 already in my hosted lab on a Docker host using the HCL V12 Early Access December drop.



    Podman limitations


    Podman isn't designed for large scale Domino deployments. You could run multiple "partitions" on the same machine and I will support creating more than one Domino systemd service.

    But you will need a separate IP address for each Domino instance if you want to expose NRPC on the same standard port 1352 on each instance.

    For all internet protocols like HTTP, SMTP and others you could have reverse proxy in front of it -- but today NRPC still needs it's a dedicated IP address.


    Also in many cases using the "host network" on Podman is the right choice. I will for example need it to get the real IP addresses for incoming SMTP traffic.

    And also I want to see the original IP addresses for incoming HTTP traffic.


    There are ways to pass the original IP address via X-FORWARD-FOR headers on HTTP.

    And for SMTP there are also ways to pass the original IP address. But this is a complexity you don't need for a single Domino instance running on Podman.


    The benefit running Podman and other container solutions is the easier management and deployment of the application in a container.


    IMHO if you are looking into larger scale deployment neither Podman nor Docker is the right choice.

    In that case you should look into Kubernetes (K8s) or other platforms based on it like OpenShift, Rancher or VMware Tanzu just to name some of the big players in that space.


    Docker vs Container

    When starting to look into the code I noticed that there are many variables prefixed "DOCKER_". And the default is Dockerfile to build a new image.
    Podman changed the name from Dockerfile to Containerfile which is the better term.
    The Domino Docker project will stay with the term "Docker" but the new Podman start script project will neither use Docker nor Podman for the naming unless something is platform specific.
    So the new naming convention will be CONTAINER_ instead of DOCKER_
    I think this new project is a good starting point to move to the new terminology. This really reflects better what it is today.


    Feedback


    If you are looking into Podman or other container technologies in combination with Domino I really want your feedback.

    What are using today and why? And what do you would like to use in future? Do you have any specific requirements or challenges?


    Our focus in the Domino Docker GitHub project is to make the image itself work nicely in all environments.

    This start script initiative is to help to bring containers into smaller environments without running a full blown Kubernetes.

    Of course I am also looking into that and I have a zoo of different K8s implementations running.


    Still it looks like Podman can be a good starting point for a smaller environment with one server per host.


    Thoughts?



    -- Daniel

    Sametime Meetings Premium internally without Google STUN servers

    Daniel Nashed  December 23 2020 08:52:51 PM


    We ran into this today. A customer is using ST Meetings Premium in their intranet and have no connection outside.

    This isn't only a challenge for software installation and updates, but also causes issues with ST meetings if the Google STUN servers cannot be reached.


    Side note: The update from Pre-Release to Premium by the way just took 1 hour for the meeting server, community server and proxy server all togehter!

    A STUN server is actually a TURN server, taking care that server and client can talk to each other if a NAT environment is involved.

    This can hit you even in an intranet depending on the network segmentation with IPv4 addressing.


    In case NAT is involved you would need an internal TURN server. If no NAT is involved you still have to disable the Google STUN servers.


    Disable STUN Servers in intranet without NAT


    To disable the STUN servers you have to modify the hidden internal configuration file .env.  (the file has a leading dot which makes it hidden on Linux).
    The line to disable looks like this by default:


    JVB_STUN_SERVERS=stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302


    In addtion you have to set to set the DOCKER_HOST_ADDRESS to the exposed, accessible IP address in your  custom.env file.

    This should be the primary interface of your Docker host!


    # IP address of the Docker host. Check LAN configuration of the system. If the host has multiple interfaces,

    # choose the one which is considered the routable IP for the host.

    DOCKER_HOST_ADDRESS=192.168.96.15



    After you made the change, you have to restart your environment.

    The easiest way is to use  "
    docker-compose down" and "docker-compose up -d".

    In one case my containers have not been removed cleanly and I had to use a "
    docker-compose rm".

    Tip:
    To ensure the changes are applied you should remove the jitsi-config folder before your bring up the server again.


    Side note: This will overwrite your private key and certificate for the web-server usually you have replaced it with your own key and you have to copy it again.
    In case the server created a self signed, you are getting a new one and all users get a security warning.

    You should always deploy a real certificate! And keep a copy outside the directory to copy it back again.
    The keys are located in jitsi-config/jibri_web/keys and they are called cert.crt and cert.key


    Test if you can access the Google STUN servers


    There is a client you can install to check the connection.


    Here is an example how to install and test.
    You can use the same test on your own TURN server later.


    yum install -y epel-release
    yum install -y coturn-utils


    turnutils_uclient -v -p 19302 stun.l.google.com




    Installing your own TURN server


    In case you have NAT connections internally you can install your own TURN server.

    If running on CentOS or RHEL this is very easy.  The most used TURN server project is
    https://github.com/coturn/coturn
    I have installed the server locally without any modification.


    Installation


    yum install coturn


    Enable and start systemd service


    systemctl enable coturn

    systemctl start coturn


    The configuration is located here -->
    /etc/coturn/turnserver.conf

    The default used port 3478 and 3479 and you have to open those ports in your firewall for TCP and UDP!!! (can be OS firewall and/or corporate internal firewalls)


    Afterwards you can change your STUN server configuration in .env like this and start the ST meeting server again:


    JVB_STUN_SERVERS=192.168.96.213:3478



    Reference for the general info an topology configurations.
    https://help.hcltechsw.com/sametime/11.5/admin/topology.html

    Conclusion


    A NAT environment can also be an intranet environment. It doesn't need to be an outside connection.

    So you either disable the STUN server as described if no NAT is involved or install your own TURN server.


    Thanks to the HCL Sametime Support team to quickly help us to figure out what was going wrong!!
    Most of it is documented but not exactly with this background and in one place.


    -- Daniel

    SUSE Leap -- this distribution getting more fans soon?

    Daniel Nashed  December 14 2020 05:01:48 PM

    On the server side for Domino and other HCL products RHEL, SLES and CentOS have been always the safe bet and fully supported.

    With the changes in the CentOS space I am rethinking if a strong fucus on CentOS is still the right step for me.

    I have been looking into SUSE Leap again over the weekend.

    SUSE Leap is also a great stable platform with a regular release cycle. In contrast to Tumbleweed which is a rolling release.
    See some details here:
    https://software.openSUSE.org/distributions

    The Domino installation on SUSE Leap isn't completely smooth and it isn't officially supported today.

    But it looks like an interesting option to have in future. So I spent some time looking into it.


    The install shows some warnings, because of the way it checks for the platform


    SUSE Leap for Domino on Docker


    Adding the SUSE Leap 15.2 base image option into our Domino Docker community image (
    https://hub.docker.com/r/openSUSE/leap) sounded like a good start point.
    I had some challenges like using zypper instead of yum and different default package selection.


    Again this isn't anything that is really supported. But I added it to get some first hand on experience in combination with Domino.

    The same way we added Centos8 and UBI support you can now use the alternate "dockerfile_leap" for building your image (currently only added to the develop branch).


    While working on this I ran into some memory issues with wget and I replaced it with curl now completely for the install script.
    But moving away from wget and replacing it with curl was already on the list.


    SUSE Leap on WSL2


    The more I look into the distribution, the more I like it.

    For example for WSL2 on Windows there is no offering from RedHat.
    And I wasn't completely happy with Ubuntu and Debian (personal choice and arguable of course).


    So I installed ->
    https://www.microsoft.com/en-us/p/openSUSE-leap-152/9mzd0n9z4m4h

    And I think this will be my new Windows WSL 2 desktop Linux!

    I will also continue to look into it in more detail also for other projects.

    For Domino it's today not an option. It's not on the "supported list". But the way how "supported" is defined is in the flow today at HCL as far I understood.

    And SUSE Leap sounds like one of the logical choices to me -- beside Astra Linux for the Russian market.


    -- Daniel



    Links

      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]