Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Community HCL SafeLinx Container with HCL Nomad Web support

Daniel Nashed  3 July 2022 21:18:40

If you are looking for a simple way to run HCL Nomad, here is a simple to use Docker container for SafeLinx and Nomad Web.

I just merged it to the main branch and wrote some basic documentation.

https://opensource.hcltechsw.com/domino-container/safelinx/


Oliver wrote a nice blog post about the container, which gives you a different view --> https://oliverbusse.notesx.net/hp.nsf/blogpost.xsp?documentId=22F6
Thanks Oliver for your great write up!

-- Daniel


Image:Community HCL SafeLinx Container with HCL Nomad Web support


Domino ZFS Snapshot Backup

Daniel Nashed  18 June 2022 18:54:39

Image:Domino ZFS Snapshot Backup


ZFS is one of my favorite file-systems. And I posted before about using it as a backup target.
The integration is pretty simple with Domino backup, because it is a simple file backup.

Now that we have the new VSS Writer for Domino 12.0.2 on Windows, it is time to look into ZFS snapshots.

For now this is native Linux for now, because you need OS level calls to create snapshots and more important for mounting the backup to the server for a restore operation.
But for native Domino on Linux this is pretty cool! ZFS has many advantages including sending and receiving snapshots in remote locations. This also includes encryption!

ZFS has quite a history and with the move to OpenZFS it's now available in Linux distributions.
One of the best integration is SUSE Linux Leap 15.3 and higher, where ZFS can be installed out of the box.

Here is a must watch video if you are interested in ZFS and there is also a presentation:

https://papers.freebsd.org/2020/linux.conf.au/paeps_the_zfs_filesystem/


Now working on the DNUG lab and final presentation preparations, I thought it would time to get this implemented.
I will demo it at #DACHNUG 49 conference next week.
And depending on feedback, I will make the configuration available via a DXL file.

If you are at #DACHNUG 49 conference next week, stop by at the DNUG Lab booth.

I have setup another server native on Linux running on ZFS, which runs the OpenZFS snapshot integration.

We can look into all details live. I have prepared many different integrations running servers in the lab environment.


-- Daniel

Additional note:

There is some optimization potential, if the Domino Backup application would provide the full restore file including the .DELTA file.
I have worked around this in multiple configurations and I think it would make sense the default restore file would already contain the .DELTA extension.

Image:Domino ZFS Snapshot Backup

openSUSE Leap 15.4 released -- works well with Domino and Docker images

Daniel Nashed  16 June 2022 10:04:37


Image:openSUSE Leap 15.4 released -- works well with Domino and Docker images


openSUSE Leap is one of the platforms I really care about. Not just because they are German and it was the first distribution I used very long time ago, when software was distributed on floppy disks.

They do a lot of things right and I have a mix of servers.

I have not used the on-line update function. And I would wait for that for a while.

But they already released the Docker base image and I had to install a new lab machine on my notebook for travel anyway.

Here is where you get the full ISO. And also the Network Image (173.0 MiB) will work.
In earlier versions, there have been issues using the smaller images. You had to configure the repositories manually.

This is now working very well and it the best setup wizard on Linux I know of.

https://get.opensuse.org/leap/15.4/

Here is the current kernel version as of last night's update:

Linux localhost 5.14.21-150400.22-default #1 SMP PREEMPT_DYNAMIC Wed May 11 06:57:18 UTC 2022 (49db222) x86_64 x86_64 x86_64 GNU/Linux

OpenSSL 3.0.1 support

SUSE added OpenSS 3.0.1. But in contrast to Redhat who moved to Openssl 3.0.1 with RHEL/CentOS Stream 9.0 completely, SUSE offers it in a separate package "openssl-3".
This might be helpful for some software. Also OpenSSL 1.1.1 has been updated and I wonder why SUSE Leap 15.3 is not getting this update  ( my Leap 15.3 server is still at OpenSSL 1.1.1d  10 Sep 2019)-

Here are the current versions as of today.

openssl-3

openssl-3 version
OpenSSL 3.0.1 14 Dec 2021 (Library: OpenSSL 3.0.1 14 Dec 2021)

openssl version

OpenSSL 1.1.1l  24 Aug 2021 SUSE release 150400.5.14


ZFS Support

I have not managed to get ZFS installed last night. And I probably wait a bit until the official repo is listed.

Building Docker images

An easy start to look into it, would be a Docker container.

I have updated our DNUG Lab environment last night via

./build.sh domino 12.0.2 -capi -verse -from=opensuse/leap:15.4

And I added new tags leap15.4 and leap15.3 in the develop branch to make it easier to select. -from=leap continues to point to whatever SUSE decides to be latest.




New arrivals in the DNUG LAB for next week: Minio for DAOS T2 and Domino Backup

Daniel Nashed  14 June 2022 23:22:08

There are is always one more thing to add ...
I just introduced a Docker based Minio S3 server on the SUSE Leap server.

The underlying file system is ZFS with deduplication enabled.

Domino Backup S3 with ZFS

This is an example for my backup and storage optimization session.
I just took the S3 backup integration and configured a Bucket for Domino Backup.

After 3 full backups of the server the storage looks like this:

zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs-pool  19.5G   195M  19.3G        -         -     0%     0%  2.80x    ONLINE  -


DAOS T2 and DAOS shared encryption  

Now that I have S3 storage available, a DAOS T2 repository was easy to add.

DAOS T2 is always encrypts NLOs when pushing them to S3.
Therefore I added AES 256 shared key encryption to the server.  

So some reason the bucket stats don't update.

But all the data is there. Backup and DAOS T2 along in the same ZFS pool ..


Image:New arrivals in the DNUG LAB for next week: Minio for DAOS T2 and Domino Backup


How to report security related problems to a vendor?

Daniel Nashed  14 June 2022 11:04:03


Reporting potential security issues is very important for software quality.
Every software has bugs -- As we have seen even in the Linux world in the couple of last month. Yes and there are even Linux kernel security bugs.

Reporting security issues in the open source world is a separate topic.
But how do you report security issues or potential security issues to a commercial vendor?


Customer Support


If you
are a customer with support, you should always open a support ticket.
Those support teams know best about the actual problem and how to flag tickets for fast security reviews in the right team.



How do you report if you don't have support?


First of all, if security is important to you, you should have maintenance for all your software products to update to the latest versions to get security fixes!
But if you run into an issue and have no support, there are usually special accounts at software companies to report bugs in a safe way.


14.06.2022: Update from Martin (huge thanks)

There is a blog post describing how to open tickets. There is a separate category for security and reporting security issues.
And there is even a guest form, in case you have no support account available.


 https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0010164



Security TXT


There is an initiative which some companies are following -->
https://securitytxt.org/
This "standard" allows each Domain to provide information about their security incident reporting process.

Take a look for example at  -->
https://www.ibm.com/.well-known/security.txt


P
roduct Security Incident Response Team

There is an other standard term you should know about -- "PSIRT".

Many companies have special teams and accounts to report security issues to.


In case you are having issues with your support account or your maintenance expired, this would be probably the best way to report a security incident.
For example HCL Software has the following web page with all the details about security incident reporting:


https://www.hcltechsw.com/resources/psirt


Why is reporting security incidents in private is important?


First all, a security concern needs to be evaluated by a vendor.

If you are not a professional researcher, it can be quite difficult to get it right and there can be false positives, due do a misconfiguration or misinterpretation of logs etc.


If you report in public -- like in a blog --  this can have negative effects for the product you care about and want to help to improve.



Blog posts


So even it might not be a bug, others known less then yourself might get the wrong impression.

Also if it turns out to not be a bug, it is difficult to correct the first impression someone had about this issue.


Adding another blog post with an update on an existing post where you raised the concern, would be even less desirable.

Because readers of your blog might only read your initial post -- not the updated information in a follow-up post.



Getting the fame for finding a bug


Money should not be the main incentive to report a bug.
But getting proper credit for a bug you found is something that even ethical hackers are striving for.


If you want the credit, you let the software company name you in their CVE instead of being the first one to blog about it.


A blog post should be the last step in the process after the problem has been confirmed, the bug is fixed and the fix is available.

Unless there s a simple work-around, there is no point in making it public early.



I thought this would be common knowledge. But I had some discussions in the last week, which really surprised and disappointed me.

This lead to this blog post and I hope this helps others if not the one I tried to discuss with in private.


-- Daniel

SafeLinx Nomad Server Community project?

Daniel Nashed  12 June 2022 10:53:46

Wouldn't it be cool to have a SafeLinx Docker image with Nomad Web included with auto configuration?
Maybe having a docker-compose.yml with just some basic parameters to get SafeLinx and Nomad up and running?


Docker container configuration:

A configuration could look like this:

CONTAINER_HOSTNAME=nomad.acme.com
DOMINO_ORG=acme
LDAP_HOST=ldap.acme.com

And just running "docker-compose up" could get SafeLinx and Nomad Web up and running ..


Certificate for the SafeLinux server

But what about getting a certificate for your server?

If your server is behind a load balancer, you can get away with automatic created certificates just for the container.
So it could include a small CA creating EDCDA keys for you.


CertMgr auto certificate updates

If SafeLinx isn't behind a reverse proxy updating official certificates and keys could be just be dropping PEM files into a mount and let the container do all the work for you..

Maybe it would be a good idea to teach SafeLinx to auto update certificates from a CertMgr server directly if the existing private key matches the new certificate retrieved via HTTPS SNI?
So wishful thinking would be just to just specify like CERTMGR_HOST=certmgr.acme.com to let the container update certificates automagically?


Hmmmmm ....

I really wanted a Nomad Web configuration for our new DNUG Lab environment, we want to showcase at DNUG.
And configuring it via the old fashioned remote admin GUI wasn't an option for me...

OK as you know once I have an idea and start building, I am like in a coding tunnel until it is all done ..
So at #DACHNUG 49 I will demo the new HCL SafeLinx Community image in combination with Domino CertMgr functionality in my Domino 12.0.x security session.

There isn't any documentation yet and I am working on some fit & finish. But it does already exactly what I described above and available in the develop branch of the Domino community image.

Building the image works very similar to the Domino, Traveler and Volt image builds.
And it builds in less then 2 minutes. The software download information is included in the software.txt like for any other image.


./build.sh safelinx +nomadweb


A docker-compose.yml with .env setup file example file is also included.

docker-compose up


Creating network "safelinx_safelinx_net" with driver "bridge"
Creating volume "safelinx_data" with default driver
Creating safelinx ... done
Attaching to safelinx
safelinx    |
safelinx    | HCL SafeLinx Community Server
safelinx    |
safelinx    | Configuration
safelinx    | ------------------------------------------------------------
safelinx    | DOMINO_ORG       : [acme]
safelinx    | NOMAD_HOST       : [nomad.acme.com]
safelinx    | CONFIG_BASE      : [o=local]
safelinx    | CERTMGR_HOST     : []
safelinx    | (CHECK_INTERVAL) : [30]
safelinx    | TRUSTED_ROOTS    : [/opt/hcl/SafeLinx/datastore/trusted_roots.pem]
safelinx    | LDAP_HOST        : [ldap.acme.com]
safelinx    | LDAP_PORT        : [389]
safelinx    | LDAP_SSL         : [0]
safelinx    | LDAP_USER        : []
safelinx    | LDAP_BASEDN      : [acme]
safelinx    | ------------------------------------------------------------
safelinx    |
safelinx    |
safelinx    | Configuring SafeLinx
safelinx    |
safelinx    | NomadServer Available
safelinx    | LDAP-Server Available
safelinx    | LDAP-Authentication Available
safelinx    | nomad-web-proxy0 Available
safelinx    |
safelinx    | Generated PEM import password: x3+SfroADK48vI2SHAzinLLHxAohqh/cMuoyJOX0WS4=
safelinx    |
safelinx    | Write down the password, if you plan to import password protected PEM files (e.g. from HCL Domino CertMgr)
safelinx    |
safelinx    |
safelinx    | Waiting for mounted cert ...
safelinx    |
safelinx    | Startup: Timeout waiting for initial certificate
safelinx    |
safelinx    | Creating new certificate for nomad.acme.com
safelinx    |
safelinx    | Signature ok
safelinx    | subject=O = acme, CN = nomad.acme.com
safelinx    | Getting CA Private Key
safelinx    |
safelinx    | Export Password: pZtC9IJh1h8RyMrSCFp23igSZtyo6msOLqwtkMC6phw=
safelinx    |
safelinx    |
safelinx    |
safelinx    | HCL SafeLinx Version 1.3.0.0 (5724-R20)
safelinx    |
safelinx    |
safelinx    |
safelinx    | Certificate
safelinx    | -----------
safelinx    |
safelinx    | SAN         : DNS:nomad.acme.com
safelinx    | Subject     : O = acme, CN = nomad.acme.com
safelinx    | Issuer      : O = acme, CN = SafeLinxCA
safelinx    | Expiration  : Jun  9 08:14:06 2032 GMT
safelinx    | Fingerprint : C0:AB:7F:F5:3C:56:00:9E:EA:0C:6B:54:CA:68:44:13:3D:7B:3E:24
safelinx    | Serial      : 1FBAA17407B2CEFB2DA48C413797934983A2D044
safelinx    |
safelinx    |


Bash command of the week: Find unmatched quotes in a shell script - very very helpful

Daniel Nashed  11 June 2022 09:56:13

There is always someone who might already have done, what you are looking for -- Specially on Linux
I found the following genius line via Google when I was looking for a unmatched quote in a bash script

This like gives you the line numbers where you have unmatched quotes:

tr -cd "\"\n" < install_dir_safelinx/entrypoint.sh | awk 'length%2==1 {print NR, $0}'

This really made my day!! Very very cool!!
-- Daniel

#DACHNUG 49 conference lab mission completed

Daniel Nashed  10 June 2022 20:52:42

Image:#DACHNUG 49 conference lab mission completed


This will be the most complete Domino lab environment you have seen prepared for a conference.


I took mot of the new Domino 12.0.x features -- including 12.0.2 EA1 into three servers.

Come and see Domino 12.0.2 live in action, get your own demo account for the conference, ask questions.


You will see some special configurations described in the last couple of month on my blog.
And I just finished a first version of a SafeLinx Nomad Web container image, which we will use and showcase at the conference.


I just got the OK from the board, that we get our own mini booth for the lab.

So beside the sessions I plan to spend a lot of time at the lab booth.


If anything is missing on the list you want to see about Domino 12.0.x.. It's still time to add it.. Let me know ..

We can walk thru all the features and specially my favorite topics Domino Backup, CertMgr and the new ICAP Antivius integration.


Have a great weekend and I hope to see many of you at #DACHNUG 49 soon.


-- Daniel



Domino Lab Setup


-- 3 Servers --
  • Domino 12.0.2 EA1 first/additional server with OneTouch setup
  • Servers hosted @ Hetzner --> Installation via Hetzner Cloud and DNS REST API using a Notes application demoed at Domino 12.0 launch event
  • Linux servers run the current HCL Domino community image
  • using dominoctl for containers -- Container start script for Docker and Podman (with systemd service)
  • OneTouch templating and automation on Linux
  • Access to Windows server via SSH tunnel with Ed25519 key for RDP access


linus.lab.dnug.eu
  • CentOS Stream 9 with Podman
  • OneTouch templating and automation on Linux
  • Traveler 12.0.2 on Podman
  • c-icap server with ClamAV integration providing ICAP for mailscan behind NGINX to offload TLS
  • Domino and Fail2Ban Integration
  • SpamGeek with SPF

ray.lab.dnug.eu
  • SUSE Leap 15.3 with Docker
  • CAPI 12.0.1 development environment in Domino container
  • Verse 2.2.0a in Domino container image
  • SafeLinx 1.3 with Nomad Web 1.0.3 in a separate Docker container
  • KeyCloak server on Docker
  • Minio server on Docker
  • NGINX for port 443 to dispatch to different applications via SNI


bill.lab.dnug.eu
  • Windows 2022
  • Veeam Backup & Replication 11


-- Main features configured --


Security
  • CertMgr with HTTP-01 & DNS-01 (free provider: deSEC e.V.)
  • ECDSA keys
  • TOTP
  • SAML with KeyCloak
  • Domino CA with Lotus Script
  • Internet lockout with IP based blocking
  • ID Vault


Message Security
  • MailScan with c-ICAP and ClamAV
  • DKIM with RSA and Ed25519
  • DKIM and SPF configured in DNS

TCO: Backup and storage optimization etc
  • VSS Writer Backup with Veeam
  • Linux Domino Backup to ZFS
  • Domino 3 way cluster with cluster repair
  • DAOS
  • NIFNSF
  • Translog
  • DBMT best practices configuration
  • DDM setup


SUSE Leap @ Hetzner

Daniel Nashed  6 June 2022 06:44:28

CentOS Stream 9 is an awesome Linux distribution.
But it still has no good ZFS support. And there are also other benefits using SUSE.


Sadly Hetzner does not allow to create virtual servers using SUSE Leap.

But they added the DVD ISO image for SUSE Leap 15.3.


You can just boot from the ISO and install your server on your own...


Works like a charm.. I just have to redo it, because I missed up ZFS and btrfs snapshots ..


Hetzner does some really cool things. They use DHCP and usually Linux and also Windows comes up with the right IP address configured.

Even if you attach a private network, the network is automatically detected.


The other DNUG Lab server is using Podman. SUSE Leap comes with Docker.


-- Daniel


Image:SUSE Leap @ Hetzner

DNUG Lab @DACHNUG conference running on Domino 12.0.2

Daniel Nashed  3 June 2022 18:51:05

Image:DNUG Lab @DACHNUG conference running on Domino 12.0.2

This is going to be awesome. This is my new weekend project and it will be a full featured lab @DACHNUG conference this month.


I already setup a cluster running Domino 12.0.2 EAP1 with most of the new features.  And it is running in production style with best practices.


We will have a full lab environment to show & tell and for hands-on during DACHNUG conference.

And it will continue to be a lab environment the different DNUG focus groups will work with.


We are planning to add SafeLinx and Nomad web next week.


And I just created a registration database leveraging the Domino CA process to create users in ID Vault.

All other new features like TOTP and DKIM outbound are added step by step over the couple of next days.


It is built all on best practices also in the back-end with a SSH tunnel from the Linux (CentOS Stream 9) machine to the Windows 2022 server securing the RDP port.


This environment will have all cool new features.

See previous post for more details..


Have a great weekend!


-- Daniel


Image:DNUG Lab @DACHNUG conference running on Domino 12.0.2

Links

    Archives


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]