Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Paranoid SSH configuration -- 3FA

Daniel Nashed  24 May 2020 10:04:01

This weekend I moved one of my servers to a new provider while upgrading it to current CentOS 8.1 release.
Beside disabling root access, implementing fail2ban and certificate based authentication, I checked additional multi-factor authentication.

In current SSH versions you can define how many authentication mechanisms need to succeed for a login.
Once that was setup I added time based authentication (TOTP) via Google Authenticator (see: https://en.wikipedia.org/wiki/Time-based_One-time_Password_algorithm).

For TOTP I used a YubiKey 5 NFC (https://www.yubico.com), which comes with an iPhone app protected by the key.
So in fact this adds another layer of security and you could see it as a 4th factor.

Beside setting up a key in .ssh/authorized_keys for your account you have to enable authentication methods and have to define which methods should be used in which order.


The required setting is not in the default /etc/ssh/sshd_config

AuthenticationMethods "publickey,password" "publickey,keyboard-interactive"

This line ensures that you are first authenticated with a certificate, before you have to use a password for your account.
In addition you have to make sure all the authentication options below are enabled:

# needed fur certificate based authentication
PubkeyAuthentication yes

# needed for password authentication
PasswordAuthentication yes

# needed for Google Authenticator 2FA / TOTP
ChallengeResponseAuthentication yes

# ensure PAM authentication is enabled -- also needed for Google Authenticator
UsePAM yes

Beside those settings I am disabling a couple of other options and move the SSH port to a none standard port, to ensure I have less login attempts
(and less logs, don't forget to change firewall rules for the port you chose).

# security best practices
Port 222
PermitRootLogin no
PermitEmptyPasswords no
GSSAPIAuthentication no
IgnoreRhosts yes
MaxAuthTries 4
MaxSessions 4


Beside those settings you have to install the Google Authenticator

yum install google-authenticator

And configure it as a PAM module in /etc/pam.d/sshd

auth required pam_google_authenticator.so


For reference here is the project page  --> https://github.com/google/google-authenticator-libpam
You have to take care, not all options might already work in the version included in CentOS.

So once your configuration is done and you registered your use with the procedure below, you can login specifying 3 different factors.
In my case for TOTP I am using my


Example Login over SSH

login as: notes
Authenticating with public key "nsh@nashcom.loc"
 (#1 authentication with certificate)
Further authentication required

Keyboard-interactive authentication prompts from server:

Password: xxxx
 (#2 authentication with password)
Verification code: xxxx
 (#3 authentication with time based token -> in my case I use the key to protect my app #4 factor)


Example configuration for an individual user


google-authenticator


Do you want authentication tokens to be time-based (y/n) y

Warning: pasting the following URL into your browser exposes the OTP secret to Google:

https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/nsh@centos8.nashcom.loc%3Fsecret%3D5DOMINOUZKRL4V7NOTES3JUYIXQ%26issuer%3Dcentos8.nashcom.loc
Failed to use libqrencode to show QR code visually for scanning.

Consider typing the OTP secret into your app manually.

Your new secret key is: 5DOMINOUZKRL4V7NOTES3JUYIXQ

Enter code from app (-1 to skip): 711236

Code confirmed

Your emergency scratch codes are:

11780937

21101653

12110617

57721107

55818118


Do you want me to update your "/home/nsh/.google_authenticator" file? (y/n) y


Do you want to disallow multiple uses of the same authentication

token? This restricts you to one login about every 30s, but it increases

your chances to notice or even prevent man-in-the-middle attacks (y/n) y


By default, a new token is generated every 30 seconds by the mobile app.

In order to compensate for possible time-skew between the client and the server,

we allow an extra token before and after the current time. This allows for a

time skew of up to 30 seconds between authentication server and client. If you

experience problems with poor time synchronization, you can increase the window

from its default size of 3 permitted codes (one previous code, the current

code, the next code) to 17 permitted codes (the 8 previous codes, the current

code, and the 8 next codes). This will permit for a time skew of up to 4 minutes

between client and server.

Do you want to do so? (y/n) y


If the computer that you are logging into isn't hardened against brute-force

login attempts, you can enable rate-limiting for the authentication module.

By default, this limits attackers to no more than 3 login attempts every 30s.

Do you want to enable rate-limiting? (y/n) y

iOS 13.5 has finally been released!

Daniel Nashed  21 May 2020 23:49:02
I was already missing my Apple mail client, because I disbled it on all of my devices thru a Traveler setting.

The security page at Apple isn't updated yet --> https://support.apple.com/en-us/HT201222

They didn't really think this fix was important. I would have expected to get a intermediate fix before rolling out 13.5.
But apparendly they put all the energy into the back-end APIs needed for the "Corona" apps, which I would see as important, that the security fixes.

-- Daniel

Specifiying Notes.ini .location via =

Daniel Nashed  19 May 2020 22:56:37

The syntax of specifying the notes.ini via =/local/notesdata/notes.ini is a cross platform feature I was aware of.
But I didn't know how deep this is hooked into the Notes/Domino code.

For the Domino core server tasks and client programs this is build-in by skipping the parameter when it starts with a =
And the core takes automatically care of it.

I will update all my applications step by step to skip parameters starting with a " = "
This enables to from any directory where it finds the binary and also the Notes dll/libs.
So you don't have to be in the data directory and could just run from any location.
Also when you build applications that call Notes binaries like the kyrtool you can specify the notes.ini with his notation.

So nothing completely new for us using this functionality. But important for C-API developers to just skip the parameter to support this functionality.
Just added it to one of my latest projects...

-- Daniel

Notes 11.0.1 IF1 released with important fixes!

Daniel Nashed  12 May 2020 19:37:52
There is a new IF released today.
I have been waiting for this fix, because I quite often crashed my client when deleting the trash folder.
Or even deleting all documents I searched for. Also in a calendar view a delete caused a crash.

I had reverted my client back to 11.0.0 just for that. This fix came out faster then the current Apple iOS mail security bug ;-)

-- Daniel

https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0079166


SARBNFCXK Fixed a crash that occurred when deleting the last document from a folder. This regression was introduced in 11.0.1
RKRYBMTPUA Fixed an issue where the client would hang when using the calendar timezone drop down control with two finger scroll
SBLEBN2HGN Fixed an issue where certain infobox fields would result in a crash when clicking on the drop-down list of choices - section border styles and action bar button Notes icons in particular. This regression was introduced in 11.0.1
MSKABN2HED Fixed an issue in the Admin client where, when trying to manage a vault server to do an action like add/remove a vault server, the error “NULL Parameter Error” would be received and the action would fail. This regression was introduced in 11.0.1
PDARBNFC7N Mac only: Fixed a problem where a second repeating meeting could not be created after creating a repeating meeting. Error about too many dates scheduled would be shown. This regression was introduced in 11.0.1



Volume Shadow Copy (VSS) for Backup and Updates

Daniel Nashed  30 April 2020 23:48:55



As you might now, I am working on a backup solution for Domino.
Snapshot backup is the master class for backup. And many modern backup solutions support snapshot backup.

Not all implementations work as we would expect. Some use VSS on Windows or Linux to take a snapshot of the data to buy time to backup the snapshot later on.
Other software leverages ESXi storage API which generates a underlying VM snapshot. And this also takes time and has impact on machine performance.

And both types of snapshots need the application to play an active role and bring databases into consistent state.

For VSS the application needs to register as a VSS writer and bring the application into a consistent state.
Domino doesn't support VSS (or file-system snapshots on Linux) and needs it's to leverage the Domino Backup API to bring databases into consistent mode.
My backup solution brings all databases into backup mode at the same time to initiate a snapshot afterwards.

Using a snapshot, the database still needs the backup API to restore a database - even no delta was recorded during the very short time between:

a.) bringing all databases, b.) taking the snapshot and c.) bringing all databases back on-line


I am leveraging VSS to create a snapshot after bringing all databases into backup mode via the Domino backup API.

But without any backup solution, you could still stop the server for a very short time, leverage snapshot solution to take a backup and bring the server back on-line.
To leverage VSS to create a snapshot Microsoft introduced the diskshadow tool, which is available on Windows servers only (not on a Windows 10 workstation).

This could be even a very convenient way also to take a snapshot before updating your Domino server.

Below you find the txt files and command files used for diskshadow I am using in a similar way for snapshots in Domino!Backup.

-- Daniel


-- backup_start.txt --

set context persistent nowriters  
set metadata c:\backup\metadata_domino.cab
set verbose on
begin backup
add volume d: alias backup_d
create  
expose %backup_d% b:

-- backup_stop.txt --

end backup
delete shadows volume d:


-- command to start backup and expose the snapshot to a new drive letter --

@echo off
diskshadow -s c:\backup\backup_start.txt
echo:
echo Snapshot created and exposed at B:
echo:

-- command to stop the backup and delete the meta data --

@echo off
diskshadow -s c:\backup\backup_stop.txt
del c:\backup\metadata_domino.cab
echo:
echo Snapshot removed
echo:



Critical Security issue in iOS Apple mail app -- all versions affected!

Daniel Nashed  24 April 2020 13:07:37

German BSI announced that there is a very critical security issue in the mail app shipped with iOS. Also the current iOS 13.4.1 is affected!


The exploit can be used to control the mail app and depending and in combination with other not described edge conditions they could be even in control of the whole device. There is no fix yet.

Here is the original post with all the technical details ->
https://blog.zecops.com/vulnerabilities/unassisted-ios-attacks-via-mobilemail-maild-in-the-wild/

Now with those details it's not 100% clear to me if the issue will also impact Traveler. We have to see how Traveler transfers the over Active Sync.
Domino stores attachments in an object and if the sizes don't match it could be that we are running into an error already when Traveler reads the attachments.

That's something only the Traveler team could tell us.
For now we have assume we are also affected.

See details in those German links.


https://www.bsi.bund.de/DE/Presse/Pressemitteilungen/Presse2020/Warnung_iOS-Mail_230420.html
https://www.heise.de/mac-and-i/meldung/Mail-Bugs-BSI-warnt-vor-iOS-4708945.html

I got the first question from a customer how to disable Traveler sync for the native Apple iOS app.

There are settings which define the user agents for each of the device types.

And there is a setting along with it to allow or to deny a device type. Let me dump some of you as an example.


So if you set the following notes.ini setting and restart your Traveler servertask, those devices could not sync any more.


set config NTS_USER_AGENT_ALLOWED_APPLE=false


Update 24.04.2020 15:50: I got the feedback from HCL that those type of variables are checked once per minute and we haven't been patient in testing. I can confirm on my server that it works after 1 minute. So no restart is needed for this change!

-- Daniel


NTS_DEVICE_TYPE_USER_AGENT_ANDROID_PROTOCOL = "^(?=.*Android)(?!.*(BlackBerry (Windows Phone))).*"

NTS_DEVICE_TYPE_USER_AGENT_ANDROID_SYNCML = "^Lotus Traveler Android"

NTS_DEVICE_TYPE_USER_AGENT_APPLE = "(^Apple-(iPhone iPod iPad Touchdown)) (^Mozilla.*(iPhone iPod iPad))"

NTS_DEVICE_TYPE_USER_AGENT_APPLE_COMPANION = "^TravelerCompanion.*CFNetwork.*Darwin"

NTS_DEVICE_TYPE_USER_AGENT_APPLE_TO_DO = "^TravelerToDo"

NTS_DEVICE_TYPE_USER_AGENT_BB_10 = "(BB10) (Toggle) (OP/) ((RIM BlackBerry PlayBook).*/10\.) (WorkConnect)"

NTS_DEVICE_TYPE_USER_AGENT_BB_2 = "(RIM Tablet) ((RIM BlackBerry PlayBook).*/2\.)"

NTS_DEVICE_TYPE_USER_AGENT_BB_GENERIC = "RIM BlackBerry PlayBook BB10"

NTS_DEVICE_TYPE_USER_AGENT_IBM_APPLE = "^Traveler-iOS-"

NTS_DEVICE_TYPE_USER_AGENT_SECUREPIM_GENERIC = "^SecurePIM"

NTS_DEVICE_TYPE_USER_AGENT_WINPHONE_10 = "(WinPhone/10) (Windows Phone 10.) (Windows Phone OS 10.) (MSFT-WIN-4/)"

NTS_DEVICE_TYPE_USER_AGENT_WM_GENERIC = "(^IBM SyncML Client$) (^Lotus Traveler WM)"


NTS_USER_AGENT_ALLOWED_ANDROID = true

NTS_USER_AGENT_ALLOWED_APPLE = true

NTS_USER_AGENT_ALLOWED_BB = true

NTS_USER_AGENT_ALLOWED_IBM_APPLE = true

NTS_USER_AGENT_ALLOWED_OTHER = true

NTS_USER_AGENT_ALLOWED_OUTLOOK = true

NTS_USER_AGENT_ALLOWED_OUTLOOKEAS = true

NTS_USER_AGENT_ALLOWED_REGEX = ".*"

NTS_USER_AGENT_ALLOWED_SECUREPIM = true

NTS_USER_AGENT_ALLOWED_WINPHONE_10 = true

Domino on Linux application installation script

Daniel Nashed  19 April 2020 16:25:09

This weekend I looked into an easy way to install one of my new applications.
For Windows I could end up writing an agent which self deploys the binaries, because usually an unrestricted agent with the right signer should be able to write to a Domino binary directory also from remote.
But I am not sure every customer would like the idea? Any thoughts?


On Linux we always need root permissions. So I wrote a shell script, which can install my application but is also customizable.


1. Directory for server-tasks
2. Directory for extension-managers

3. Directory copied into data directory (including sub-directories)

4. Notes.ini file with parameters to update, add and remove from lists like exmgr_addins and servertasks.

So you can see the target audience is mostly the C-API developer deploying extensions which need native code.

This could be also extended for Java / OSGI applications and others.

Plain NSF/NTF based applications should be deployed via client/admin client.

The syntax for the notes.ini variables:

nshback_configdb=nshbackup.nsf
extmgr_addins+=libnshextlog.so


The += adds to a list and the -= removes from a list.


There is also a configuration file which currently only holds the name of the application. But it is the place to add more parameters in future.
It could be another extra in my start script and could help others to install their applications.

It's not a full installer, but should fit for many applications. I could add custom logic as an option.


Here is the structure of the installer:


.

├── install.sh

├── notes.ini

├── config.txt

├── extmgr

│   └── libnshextlo.so

├── notesdata

│   └── nshbackup.ntf

└── servertasks

   ├── nshbackup

   └── nshrestore



It automatically detects your Domino binary directory and data directory. And it also finds out about the owned of your data directory (checking the notes.ini) at start-up.

Once identified you are prompted to confirm the settings.  In most cases you can install pressing 3 x "enter" :-)


I also added a silent mode ( -s ) which will not ask for parameters and just installs if binary and data directory can be identified automatically.


./install.sh


---------------------------------------------

Domino!Backup Installer

---------------------------------------------


Domino Binary Directory : /opt/hcl/domino

Domino  Data  Directory : /local/notesdata



Your configuration


Domino Binaries :  /opt/hcl/domino       [OK]

Domino Data     :  /local/notesdata      [OK]

Domino User     :  notes                 [OK]

Domino Group    :  notes                 [OK]


[I]nstall [D]ata Dir [B]inary Dir [Q]uit :


Installing '/opt/hcl/domino/notes/latest/linux/nshbackup'

Installing '/opt/hcl/domino/notes/latest/linux/nshrestore'

[/local/notesdata/nshbackup.ntf] copied

notes.ini [extmgr_addins=libnshxban.so,libextlo.so] updated


Installation done



A typical config.txt could look like this. Those are the first parameters I added.


INSTALLER_NAME="Domino!Backup Installer"


#SILENT_INSTALL=yes


# Overwrite Domino data directory

#DOMINO_DATA_PATH=/local/notesdata


# Overwrite Domino binary directory

#LOTUS=/opt/hcl/domino


# Editor used for configuration

#EDIT_COMMAND=mcedit


# Specific locale used by the installer

#INSTALL_LOCALE=C


Would this be useful for Domino on Linux environments?
I think I will also update my start script installed, leveraging this logic and make it available for free thru my start script.

-- Daniel



Windows 10 is not going to sleep after latest updates

Daniel Nashed  15 April 2020 21:57:15


Not that my notebook gets much rest those days..  But there is a current issue with the sleep mode. If you put it to sleep it might happen that it wakes up after a while and keeps running during the the night.

I found out that I am not the only one with this issue after recent updates. And there is an easy fix for that.  

It turned out that PowerdownAfterShutdown was set to "
0" and needs to be set to "1"


[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon]
"PowerdownAfterShutdown"="1"



After that change my notebook didn't wake up at night any more.

Update: 18.04.2020:

The change didn't help completely. My notebook is still waking up.
I also looked into the power management of mouse and keyboard and turned out wake-up by  external keyboard and mouse.

Still testing what else could cause it. Looks like it didn't happen outside the docking station. So I will try to disable network-card wakeup ..


Update 21.04.2020:
After those changes didn't help I did some additional research.
First of all I went thru all the different hardware devices in the device manager, which could wake-up my notebook.

Not all devices have the option to allow to wake up a machine.

Than I found some other settings which affects it.

It turns out that some of the scheduled tasks -- even I turned off nightly automatic update and I didn't schedule my backup this way, are scheduled to run at night and by default will wake-up the machine.

The first important commands are to figure out what woke up your machine the last time:

powercfg -lastwake

Wake History Count - 1
Wake History [0]
  Wake Source Count - 1
  Wake Source [0]
    Type: Wake Timer
    Owner: [SERVICE] \Device\HarddiskVolume3\Windows\System32\svchost.exe (SystemEventsBroker)
    Owner Supplied Reason: Windows will execute 'NT TASK\Microsoft\Windows\UpdateOrchestrator\Universal Orchestrator Start' scheduled task that requested waking the computer.

powercfg -devicequery wake_armed

NONE


powercfg /waketimers

Timer set by [SERVICE] \Device\HarddiskVolume3\Windows\System32\svchost.exe (SystemEventsBroker) expires at 07:04:24 on 02.05.2020.
  Reason: Windows will execute 'NT TASK\Microsoft\Windows\UpdateOrchestrator\Backup Scan' scheduled task that requested waking the computer.

Next I disabled "UsoSvc" (Update Orchestrator Service)

sc stop UsoSvc
sc config UsoSvc start= disabled


But to make sure I opened the task scheduler and went thru all services to disable that they can wake-up my machine.

I went thru the update and backup tasks under \Microsoft\Windows and checked the settings.
The important setting is under "Conditions" tab and is called "Wake the computer to run this task"


In my case my admin user wasn't able to change the settings. The only work-around was to use the psexec.exe toll from Sysinternals to run the task scheduler.
This might not affect you. But in my case this was the only solution.

You just need to download it from the Microsoft site and start it like this:

psexec.exe -i -s %windir%\system32\mmc.exe /s taskschd.msc


There are quite some steps that can be performed when you run into wake-up at night issues.
I never looked into this before and I never had to. But maybe this helps some of you to safe some time ...

Last night my notebook was sleeping longer than me ..

This morning it reported the following:

powercfg /waketimers
There are no active wake timers in the system.

powercfg -lastwake
Wake History Count - 1
Wake History [0]
  Wake Source Count - 1
  Wake Source [0]
    Type: Fixed Feature
    Power Button


-- Daniel


    Notes & Domino & Traveler 11.0.1 available for download

    Daniel Nashed  31 March 2020 21:31:33


    Notes & Domino & Traveler 11.0.1 is available for download!

    I have not seen Sametime 11 FP1 yet. But it should be available shortly ..

    Here is a very quick summary of my personal highlights ..

    Traveler 11.0.1

    For Traveler a long waited new feature is the cross Domain ID-Vault Support.

    This is based on a back-end enhancement in Domino 11.0.0 which needed also a code change on the Traveler side to support it.
    And you also need a notes.ini parameter to enable it -> IDV_ENABLE_CROSS_DOMAIN=1

    You only need this new version on the Traveler server which is requesting side. There is no change needed on the Domino server hosting the ID vault or the mail-server.

    For details see this technote --> https://help.hcltechsw.com/traveler/11.0.0/Plan_Domino_domains.html


    Domino 11.0.1

    The other great feature is SAN x.509 Cert support in combination with SNI support.

    So now finally we can have one IP address with multiple SSL/TLS enabled websites.

    I have tested this feature already and it is the reason why I have to finally move my server from CentOS 6.10 to CentOS 8 to update to Domino 11.0.1 :-)

    There are a couple of other features in detail, which are well described here --> https://help.hcltechsw.com/domino/11.0.1/whats_new_in_domino11.0.1.html


    Domino 11.0.1 HCL Docker Image

    There is also a ready to go Docker image from HCL.
    It's a first version which is planned to be the new delivery model for Domino beta releases as well.
    The image needs to be manually configured. And is currently only supported on Docker CE 19.x

    But I hope that we will see more from HCL as a full image. And I would also wish extensibility and auto configuration.

    See details here -> https://help.hcltechsw.com/domino/11.0.1/inst_dock_domino_overview.html

    We have also updated the develop branch of our Community Domino Docker image to Domino 11.0.1.
    And we are looking into updating Traveler as well before we push the changes to the master branch ..

    https://github.com/IBM/domino-docker/tree/develop


    -- Daniel







    Domino Docker Project update -- OpenShift support & Podman+Docker+K8s support for "arbitrarily assigned user ID"

    Daniel Nashed  15 March 2020 12:49:55


    A couple of days ago I got the request from Daniele Vistalli, a fellow HCL Master, that he needs support to run Domino on Kubernetes (K8s) with a so called "arbitrarily assigned user ID".
    He and the team around him are doing incredible work to finalize Factor-y's MSP offering which included Domino on K8s as part of their platform.
    Running with distinct UIDs is an important security aspect, when offering cloud services separating data between containers -- and even more important tenants.

    On OpenShift the concept of a "arbitrarily assigned user ID" is a strong requirements for images to run. They assign a new UID for the container to run for security best practices.
    In general a container is quite safe already without it. But to reduce exposure to potential security issues, they don't only forbid to run a container with root but also assign an unique UID to each container on the fly.

    Quote from a RedHat technote:
    "When OpenShift starts a container, it uses an arbitrarily assigned user ID.
    This feature helps to ensure that if an application from within a container manages to break out to the host,
    it won’t be able to interact with other processes and containers owned by other users, in other projects."
    (1)

    K8s doesn't have this strong requirement. But in Docker and K8s you can specify a UID manually when the container is initiated.
    On Docker the command line option is e.g. --user 1234. And there is a K8s equivalent.
    But this will cause issues with "whoami" and other code trying to figure out about the user.
    So we had to add code to modify the /etc/passwd in a safe way (application runs with root permissions having the sticky bit set -- like bindsock does).

    OpenShift and also Podman in current versions automatically modify /etc/passwd and add the UID with it's numeric value as a user like this:

    notes:x:1000:1000::/home/notes:/bin/bash
    1025570000:x:1025570000:0:1025570000 user:/:/sbin/nologin

    "The OpenShift run-time CRI-O (starting from OpenShift 4.2 onward) now inserts the random user for the container into /etc/passwd." (1)

    So for OpenShift and also when you use Podman (which also used CRI-O) the platform already takes care of adding the UID into /etc/passwd.

    For older versions and also for Docker/K8s, the work-around is to modify the "notes" user in /etc/passwd with the group 0.
    Podman is also using CRI-O and works similar to current OpenShift also adding the UID to /etc/passwd.

    So depending on the container platform there are different approaches.
    I have tested the the different images for CentOS 7 and RedHat UBI 8 on all the platforms. And it looks good so far.
    The changes are checked into the development tree --> https://github.com/IBM/domino-docker/tree/develop

    Beside those changes I also moved all the scripts into a central location /domino-docker/scripts and made sure only root has write permissions to this folder.
    This is also the preparation for future extensibility, that partners can have their own hook points to be executed for example during server configuration or startup.

    If you want to get your hands on OpenShift, there is a free 30 days trial offering directly from RedHat --> https://manage.openshift.com.
    You are up and running in minutes and it has a very clean graphical interface and provides the "oc" command line, which offers all the K8s commands.


    -- Daniel


    (1) Reference: https://access.redhat.com/articles/4859371

    Archives


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]