Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

DNUG Focus Group Administration Event including our Docker session next week

Daniel Nashed  8 July 2020 07:14:18

It's still a week to register! The focus group administration prepared an interesting agenda (but beside one slot from Heather it will be all in German).

The new brand new HCL Domino Volt, Domino V11 Directory Sync, Verse on Premise and Docker&Containers are focus topics next week.

Thomas Hampel (HCL) and me present about Docker and other containers.
We have been both working for quite a while on our Domino on Docker project long before we had an official HCL Domino on Docker image.
Now Docker will be the platform for new products like ST meetings and also the platform for the upcoming Domino V12 beta.

So it is time to explain Docker and other container environments with a bit shifted focus.
And also speak about offerings from HCL on Docker and other container platforms.

We will show how you can take benefit container platforms today and how others are using them.
You will also take home what could be your best starting point for a first deployment and hands-on.

-- Daniel

    TrendMicro Virus Scanner Support for Domino 11

    Daniel Nashed  25 June 2020 13:12:17
    We had the question for AIX a while ago and it was more difficult because the version for AIX is a different implementation.

    But for Windows and Linux there is an official support statement as well.
    My customer sent me a different link not showing Domino 11 support.

    But my contact had the correct link:

    Not having anti-virus or backup supported for a new Domino version, can be a big road blocked for upgrades.
    In this case, Trend did already do the verification.

    There are no back-end API changes that should not make it work.
    But there are changes in Domino 11 like removing the IBM GSkit and replacing it with OpenSSL components and internal crypto, could theoretically have impact.
    So they need to test the code again on Domino 11.

    We had the same questions for IBM TDP/Spectrum Protect, which is also supported on Domino 11 since a while!
    Actually the development team was happy that IBM GSKit was removed, because they had an compatibility issue with it. TDP is also using it.

    If you have other software that is not Domino 11 ready and this is a road blocker, I would be interested to hear...

    -- Daniel

    Borg Backup Ubuntu on WSL2

    Daniel Nashed  14 June 2020 18:57:55

    Some really cool software isn't available for Windows.

    BorgBackup -> is on my list for very efficient backup.
    I have tested it a while ago with my Domino Backup application. It's a bit more difficult for backing up one database after another.
    But for backup of my source code and related files, this is a perfect match.

    Already when playing with NSF backup the results for compression and deduplication have been amazing!

    I can further optimize it removing or excluding binaries and objects. But this already looks great!

    BorgBackup can use remote repositories. So I can send the data over SSH to a Linux machine.
    And I could still use another backup tool to backup that repository. Actually I would probably use more than one repository.

    Install BorgBackup

    Sadly BorgBackup isn't included in the Ubuntu distribution for WSL2.

    But you can download borg_linux64 and just use it.

    BorgBackup is definitely a great tool to look into, if you are running on Linux!

    See my backup results below from the first and second run today.

    /mnt/n is my drive n: automatically mounted by WSL inside Ubuntu.

    Because my source code directories will not change a lot, this is great incremental backup.

    But it also worked well with Domino NSF if you have the right storage optimization strategy.
    If you use DBMT with 10% pre-alloc and a compact not more than every 2 weeks, you should have a quite good deduplication rate
    (assumed you have DAOS, NIFNSF and other storage optimization in place).

    Oh now that I have Linux tools for my

    -- Daniel

    borg create --stats notes@ /mnt/n/notesapi/work
    notes@'s password:
    Enter passphrase for key ssh://notes@
    Archive name: Test
    Archive fingerprint: 56db944f75a460d6bb14ce236fb5cde69106a0d2936b874454be1611cb360cff
    Time (start): Sun, 2020-06-14 17:43:17
    Time (end):   Sun, 2020-06-14 17:47:50
    Duration: 4 minutes 33.05 seconds
    Number of files: 10466
    Utilization of max. archive size: 0%
                           Original size      Compressed size    Deduplicated size
    This archive:              591.78 MB            305.64 MB            737.01 kB
    All archives:                1.18 GB            611.28 MB            257.61 MB

                           Unique chunks         Total chunks
    Chunk index:                    8503                21049

    Run Domino & C-API Dev on WSL2 Ubuntu Linux

    Daniel Nashed  14 June 2020 18:34:24

    Run Domino & C-API Dev on WSL2 Ubuntu Linux

    Now that I have Ubuntu running on my Windows machine thru WSL2, I have a couple of ideas what to do with it.

    Having Windows combined with a real Linux implementation has some interesting new opportunities.
    Yes some of it isn't supported at all. But it makes my development life a bit easier.

    Domino installation

    First of all I tried to install Domino.
    The installation failed on first attempt and it turned out I just had to change the InstallAnywhere shell script to use bash  --> #!/bin/bash
    After that change the installation worked like a charm.

    Domino configuration with X11

    Usually I am using the remote server setup via -list 1352. But I wanted to see if I get X11 working.
    There are multiple ways to do it and since I am not a big X11 fan, I took the easy part.
    I exported my display with a line I found in a forum. It gets the name server for the Ubuntu host, which is the IP of my machine.

    export DISPLAY=$(awk '/nameserver / {print $2; exit}' /etc/resolv.conf 2>/dev/null):0

    When I started a X application, my MobaXterm just asked me if it would be OK to open the session -- Done.

    Running /opt/hcl/domino/bin/server let me use the graphical local setup instead of the remote setup.

    Domino Start Script

    The Domino start script works in general. But I had to tweak it a bit.

    The install script will try to configure systemd. But systemd isn't currently supported by WSL2 Ubuntu.
    So I switched it manually back to the old init.d style
    (comment out DOMINO_SYSTEMD_NAME=domino.service in /etc/init.d/rc_domino and /etc/sysconfig/rc_domino_config

    The remote server console still did not work, because for some reason the reading a variable generates  SIGCHLD (17) signals, which stop the console.
    If you remove signal 17 from the "trap statement" it will work.

    I might update the start script and start script installer to detect a WSL environment and configure accordingly.

    Domino C-API Environment

    Now that I have a Domino server and the Ubuntu instance can directly access my other file-systems, I could potentially compile my applications directly from the Ubuntu window.

    The gcc/gcc++ compiler was already installed. But it's the wrong version, which is not supported by Domino!

    But for a development environment to do a first build, it's still a good idea -- as long as your finally compile with the gcc/gcc++ 8.x compiler that ships with CentOS 7.4.

    Ubuntu 20.04 LTS
    ships with gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2) which is a new major version.

    I tested it anyway and beside one bigger change, it still works.
    Linking with the noteslibs for startup will fail because the code generated with the newer compiler is position independent compiled & linked.

    To get Domino Linux 64 binaries working you have to use -no-pie = disable position independent code

    I am still testing but it looks good so far with just this one change.
    And this makes my life a lot easier when testing Linux applications.
    I can run everything on the same machine (even I have many virtual machines running anyway).

    -- Daniel

    Docker, WSL and VMware Workstation running at the same time on Windows 10

    Daniel Nashed  13 June 2020 19:16:31

    This is awesome news! When I updated this week to Windows 10 Pro 2004, I looked into what's new.
    WSL2 got some updates and there is a new Windows Sanbox, which is also pretty cool to test stuff.

    You can enable it and start it just like an app. This is similar to starting a Linux desktop like Ubuntu.

    WSL2 is around for a while but I never looked into it. The what's new and the Sandbox made me look into it.

    Once you enabled WSL2 and downloaded one of the distributions form the Microsoft store (Ubuntu in my case), you can really run Linux on your Windows machine.

    Docker on Windows

    But the really cool part is that Docker now supports WSL2 and leverages it instead of leveraging a Linux installed on a Hyper-V VM.

    I never liked the Docker integration for Windows. But this is a game changer!
    Our Domino on Linux project including all build scripts and the management script just runs in an Ubuntu bash.
    And you can look into your containers also with your Docker desktop environment on Windows.

    So that means we are now also supporting our Domino Docker project on this new platform.

    VMware Workstation 15.5 on the same machine

    Of course my development environment will stay on CentOS 7 with Docker CE.
    And this continues to run on my VMware workstation installed on the same machine!

    This was the biggest surprise I had, when I did some research.

    I didn't use the Linux sub-system or Docker because this conflicted with my VMware workstation.

    But since version 15.5 -- thanks to VMware and Microsoft working together -- both virtualization technologies can work hand in hand.

    All the installation was really straight forward. I updated to Windows 2004, installed WSL2 and updated my VMware Workstation, before installing the current Docker version.

    Enclosed you find links with background information and also how to set it up!

    -- Daniel

    Image:Docker,  WSL and VMware Workstation running at the same time on Windows 10

    References you should check if your want to install it on your own

    Docker using WSL2

    What's new in Windows 2004

    Windows Sandbox

    VMware Workstation 15.5 Now Supports Host Hyper-V Mode

    Windows Subsystem for Linux (WSL): The Ultimate Guide

    What's new in WSL"

    Notes Client all languages in one package

    Daniel Nashed  12 June 2020 14:22:51
    To include  dictionaries was very high on my personal priority list for a while.
    I don't want to install a German client just to have German spell-checking.

    And this is true for may customer environments who need multiple languages.

    And I customer asked me about multi language kits today.

    We have many different kits today per languages + MUI packs.

    What we have today is complicated and confusing.

    I found an AHA idea asking for packs that include all languages.

    And I really like the idea to have one English version with all dictionaries and one kit that contains all languages!

    The idea has 7 votes today ->

    If you think this would simplify deployments and just adds a bit more of space, add yourself to the wish-list item ;-)

    And I would be very interested to hear your feedback.
    I am not really a client guy. But I run into those kind of things quite often in customer deployments.

    -- Daniel

    Traveler 11.0.1 FP1 released

    Daniel Nashed  3 June 2020 05:42:56

    If you are using a multi Domino domain Traveler environment, this Fixpack is important for you!
    And you should read the referenced technotes in very detail!

    Traveler support for multi Domino domains updates

    When Active Sync 16 support was introduced, Traveler started to use a calendar API which wasn't completely multi Domino domain aware.

    Notices for invitations have been send from the Traveler server instead of the home mail server of the user.

    There is a new notes.ini setting to control this behavior  -->

    So for a multi Domino domain environment you really have to install the FP and enable this parameter in most of the cases to ensure the right mail routing and formatting of addresses for those notices.

    User session mode and trusted Server required!
    (some few admins might still use it)

    You should also be aware that the old Traveler access mode should be completely avoided with Traveler 11 and the introduced changes.
    In case you are still running the old mode as a work-around via setting
    NTS_USER_SESSION ini paramter to FALSE, you really have to switch to the new mode and add your Traveler servers to the Trusted Server list on your mail-servers!
    We rarely see customers still using this old mode and now is the final call to switch to the new mode!

    The new mode uses the trusted server functionality and creates a real Notes user session on the mail server and is enabled by default.

    Traveler HA recommendations

    Another note is important when updating Traveler HA pools! You should ensure that all servers in a pool are updated from an older release to Traveler 11 in a very short sequence of time.

    Mixing different ActiveSync Versions in a Traveler pool is not desirable and should be avoided!

    So it would be best to update one Traveler server after another!

    The new fixpack now also supports the quite new Microsoft SQL Server 2019.

    See this technote for details:

    And find the fixes here (search does still not work with a partner account).

    Domino Antivius powered by CalmAV

    Daniel Nashed  31 May 2020 23:03:04

    This project was only on hold for 12 years. -- LOL. When I wrote SpamGeek, I always wanted a matching anti-virus.
    But I never found a good and free integration for Linux. Two weeks ago I discovered that "clamd" the service behind ClamAV offers a nice TCP/IP interface.
    It doesn't provide channel encryption but should be OK when invoked on (port 3310) like the Tika server used by Notes/Domino 10 and higher to index attachments.

    See the ClamAV website for details:

    First implementation & ClamAV
    Clamd is native available on Linux and also for Windows so I wrote a native integration on TCP/IP level.
    For now it is a servers for W32/W64/Linux64 which can scan databases. If all works well, I will integrate it with my SpamGeek application.
    Beside scanning attachments it can send the full MIME stream to clamd. I am still experimenting with the different clamd scan options like heuristics.
    The task by default skips larger attachments to scan whole databases. The limit can be increased.

    And you can also use a remote clamd server specifying an IP address via notes.ini or command-line for testing.
    The task prints the virus name along with a Notes:// link. And also generates a SHA1 used for a link to VirusTotal lookups.
    You can move potentially infected mails from inbox to a virus Notes folder.

    I am planning to give this away for free for smaller environments with up to 20 user. And I am not sure yet if I want to make this available for larger environments.
    This has been build for my own needs first. But I am sure this would be a great fit for many small customer or business partner environments.

    Beta available on request
    Installing the servertask is pretty simple and by default it just scans and reports.
    The more complex part is the clamd configuration. And there isn't a systemd service for clamd. I might write one.
    But for testing the servertask would be already available by mail.

    Here is the current syntax and a scan example from my info mail account.
    What do you think? Would this be helpful?
    -- Daniel

    -f   Move infected from Inbox to Virus Folder
    -m   Also scan MIME (sent complete EML to Clamd)
    -e   Try to scan encrypted documents (will only work for owner of mail-file)
    -v   Verbose
    -s Remote Server to scan
    -a   Maximum scan size per attachment  a=KB, A=MB (Default 1 MB)
    -b   Maximum scan size for MIME stream b=KB, B=MB (Default 1 MB)

    nnshdomav.exe mail/nashcom5.nsf -c
    31.05.2020 22:41:29   nshdomav: Domino Antivius 0.5.1 using Clamd: []
    31.05.2020 22:41:29   nshdomav: ClamAV 0.102.3/25828/Sat May 30 14:36:41 2020
    31.05.2020 22:41:29   nshdomav: [mail/nashcom5.nsf] Scanning 335 documents
    31.05.2020 22:41:30   nshdomav: [] -> [Win.Trojan.Agent-35842] (79 ms)  [notes:///mail/nashcom5.nsf/0/8AC7E8B7F5F9A4A7C12576850069BACA] []
    31.05.2020 22:41:31   nshdomav: [] -> [Win.Trojan.Generic-42] (187 ms)  [notes:///mail/nashcom5.nsf/0/15BAA48400CFC7DCC125772B00796FA5] []
    31.05.2020 22:41:32   nshdomav: [] -> [Win.Trojan.Bublik-23] (546 ms)  [notes:///mail/nashcom5.nsf/0/EB27A50C0325766EC12579EA003C1839] []
    31.05.2020 22:41:32   nshdomav: [] -> [Win.Trojan.Bublik-23] (454 ms)  [notes:///mail/nashcom5.nsf/0/9633B7A3A0D41D58C12579EA003BE637] []
    31.05.2020 22:41:33   nshdomav: [] -> [Win.Trojan.Bublik-23] (468 ms)  [notes:///mail/nashcom5.nsf/0/9A4A7171E3EA8529C12579EA003AE57B] []
    31.05.2020 22:41:33   nshdomav: [] -> [Win.Trojan.Agent-1138832] (422 ms)  [notes:///mail/nashcom5.nsf/0/6C54495460287EC022228391C9073494] []
    31.05.2020 22:41:35   nshdomav: [VERSANDDETAILS 12-05-2020·] -> [Win.Trojan.Fareit-7784794-0] (453 ms)  [notes:///mail/nashcom5.nsf/0/0F4CC616428497B7BCFA697A41FAB71E] []
    31.05.2020 22:41:35
    31.05.2020 22:41:35   nshdomav: Virus Attachments  :          7
    31.05.2020 22:41:35   nshdomav: Virus Att Warn     :          0
    31.05.2020 22:41:35   nshdomav: Virus MimeStream   :          0
    31.05.2020 22:41:35   nshdomav: Databases          :          1
    31.05.2020 22:41:35   nshdomav: DatabaseOpenErrors :          0
    31.05.2020 22:41:35   nshdomav: Attachment Erorrs  :          0
    31.05.2020 22:41:35   nshdomav: Attachments        :        623
    31.05.2020 22:41:35   nshdomav: Skipped Large      :         10
    31.05.2020 22:41:35   nshdomav: Docs with Attm     :        335
    31.05.2020 22:41:35   nshdomav: Docs No Attm       :          0
    31.05.2020 22:41:35   nshdomav: Docs encrypted     :          0
    31.05.2020 22:41:35   nshdomav: Socket resets      :          0
    31.05.2020 22:41:35   nshdomav: Runtim   (sec)     :          5
    31.05.2020 22:41:35   nshdomav: ScanTime (sec)     :          4
    31.05.2020 22:41:35   nshdomav: Total Size Attm    :    73.5 MB
    31.05.2020 22:41:35   nshdomav: Total size scanned :    45.2 MB
    31.05.2020 22:41:35   nshdomav: Total size skipped :    28.3 MB
    31.05.2020 22:41:35   nshdomav: Shutdown

    C-API Visual Studio 2017 Domino 10 and higher

    Daniel Nashed  31 May 2020 15:53:29

    Quite a long post, but I think important for many of my ISV friends. I will update my post once I get an official technote from HCL about this new compiler recommendation.
    This week I figured out that already Domino 10 is build with Visual Studio 2017 instead of the old Visual Studio 2010 SP1 (VS2010 SP1) compiler.
    For Linux we got the official info, that beginning with Domino 10 we have to switch to the GCC compiler CentOS 7.4.
    But there wasn't any official statement for compiling C-API applications for Win64 and the old requirement was to build with VS2010 SP1.
    Also existing applications compiled with VS2010 SP1 continue to work in most cases with Domino 10 & 11 in contrast to Linux (glibc incompatibility).

    I just found out this week HCL switched to Visual Studio 2017 already beginning with Domino 10.

    It makes a lot if sense to switch to the new compiler and rely on the new Universal C run-time (see below).
    I never ran into issues with applications compiled with VS2010SP1 but I did some research and testing over the weekend.
    This included installing Win2012 R2 and Win2019 from scratch to have no inherited run-time installed. And than tested with Domino 9 and Domino 11.

    Universal C run-time in Windows

    Microsoft introduced the new Universal run-time which is automatically installed with Win2012+ (just tested installing from scratch with Win2012 R2).
    So there are no additional run-time components to install with the new compiler.

    The links inside this article are broken. But the article itself is helpful:

    Also see the official download page (but it's included already if you are on a support Domino Windows version).

    Applications compiled with VS2010 SP1 on new installed Domino 10/11 servers might fail on older Windows server versions!

    Domino 9 installs a couple of older Visual Studio runtimes (VS 2005 and VS 2010).
    But the current HCL Domino versions only use the Visual Studio 2017 universal run-time.

    If you run on an existing environment that has been upgraded, you should be on the safe side, because Domino 9 installed the needed run-times already for you.
    But with a breand new Domino 11 server installed, you will miss the old VS2010 run-time!

    The first missing file reported is "msvcr100.dll" and you would have to install "Microsoft Visual C++ 2010 SP1 Redistributable Package (x64)"

    So this should still work but needs to be taken care of in those edge cases.

    Dynamically Linking recommended

    Domino itself is dynamically liked with /MD compile switch and this is also the official Microsoft recommendation I found:
    "We strongly recommend against static linking of the Visual C++ libraries, for both performance and serviceability reasons"

    So you should make sure your are also compiling in the same way if there are no other good reasons to statically link.

    Moving to Visual Studio 2017

    Over the weekend I updated my production build environment.
    I have installed Visual Studio 2017 and have looked into existing makefiles and build scripts.

    In my environment I use makefiles. Because this is how I started like 20 years ago and this is what makes a lot of sense for cross platform development.
    I have separate makefiles for all the platforms but I looked into standardizing them including some build-commands having W32+W64 on the same machine.

    It turned out that we can't continue to use the makefiles as they have been used with Visual Studio 2010.
    So if you used the same makfile style Lotus introduced long time about in the sample applications, you need some changes!

    SDK Changes

    VS2010 was using "Microsoft SDKs\Windows\v7.x", which contained win32.mak.
    This file defined a couple of variables used in the sample makefiles like "conlibsmt" and "cvarsmt".

    This makefile include is not present in the Microsoft 10 SDK used in Visual Studio 2017.

    Without the includes you have to use your own defines. And this is also what you find in the build command line shipped with the C-API toolkit in the "cmp" directory.
    The cmp does contain the the original build switches used to build Domino 9. They should still be fine with the new compiler. At least in my tests they worked well.
    See my example makefile below. This is my current working makefile for testing. And I will have to convert makefiles for all of my applications step by step.

    Use of new features with current WINVER

    For one of my applications I had to increase the minimum OS version when compiling with VS2010 when working on a punycode research project (needed for DNS resolution for none ASCII chars).

    I set /DWINVER=0x0600 which would be the equivalent of Windows Vista / Server 2008.

    The default WINVER compiled by VS2010 SP1 is 0x0500 (Windows 2000) and for VS2017 the default is 0xA00 (Windows 10).

    The table here does not show the current server versions

    But I looking into other build numbers it should be the following mapping:
    Windows 8 Windows Server 2012
    Windows 8.1 Windows Server 2012 R2
    Windows 10 Windows Server 2016

    So if we want to ensure compatibility with Win 20012 we should probably stay with /DWINVER=0x0602 (Windows 2012).


    The step to move to Visual Studio 2017 is a very positive and good move!
    And using the Universal C run-time in future will make things easier in future.

    Dynamically linking in this context makes also a lot of sense. And we are on the safe side using the same compilers and link options than HCL.
    -- Daniel

    Appendix: Simple Example Makefile W64 C-API Servertasks with Visual Studio 2017

    This is the example from my punycode tests where I changed the WINVER earlier.
    You will usually not link with Normaliz.lib Ws2_32.lib Kernel32.lib.. But that was needed in my case.

    # C-API makefile
    # Nash!Com / Daniel Nashed
    # Windows 64-bit version using
    # Microsoft Visual Studio 2017



    # Link command

    n$(PROGRAM).exe: $(PROGRAM).obj
            link /SUBSYSTEM:CONSOLE $(PROGRAM).obj notes0.obj notesai0.obj notes.lib msvcrt.lib user32.lib Normaliz.lib Ws2_32.lib Kernel32.lib /PDB:$*.pdb /DEBUG /PDBSTRIPPED:$*_small.pdb -out:$@
            del $*.pdb $*.sym
            rename $*_small.pdb $*.pdb

    # Compile command

            cl -nologo -c -D_MT -MT /Zi /Ot /O2 /Ob2 /Oy- -Gd /Gy /GF /Gs4096 /GS- /favor:INTEL64 /EHsc /Zc:wchar_t- /DWINVER=0x0602 -Zl -W1 -DNT -DW32 -DW -DW64 -DND64 -D_AMD64_ -DDTRACE -D_CRT_SECURE_NO_WARNINGS -DND64SERVER -DPRODUCTION_VERSION /DUSE_WIN32_IDN  $*.c


            del *.obj *.pdb *.exe *.dll *.ilk *.sym *.map

    End of support for Notes/Domino 9.0.1 Announcement end of Q2/2021

    Daniel Nashed  30 May 2020 14:32:57

    Update: 03.06.2020

    The feedback I got from HCL was that the EOS date in the referenced documents was not intended. This is true for 9.0.x (and also 10.0.x).
    I still belive moving forward makes a lot of sense as outlined in this blog post.
    On the server side moving to a new major release isn't nearly as complex compared to other vendors.
    And on the client side it is getting a lot easier once you are on a current release. You can use the free offerings to automate updates (And there is a free Panagenda offering to bring you to version 10.0.1 first).


    There is a new announcement about Notes/Domino 9.0.1 life cycle, which has been in the market for quite a while.

    The "EOM" date for end of Q2 defines the withdraw of the product. That means it will have another year of support until the EOS Date.

    Notes/Domino 11.0.1 has shipped and there is Notes/Domino 12 planned for Q1/2021.

    Having 3 code streams in support would be quite a lot, especially because Notes/Domino 9.0.1 was an IBM release.

    So this is good and important statement which helps us to get customers upgraded to the right Domino versions!

    There are way better reasons to move to current releases than a support statement. And it s very sad that often this is the only reason that counts for management.

    We have seen environments still running on 8.5.x releases and those versions still work.

    But also upgrading to a newer version is far less complex than what other products offer.

    That often means that new functionality is not adopted.

    We still see that features like Translog, DAOS and other storage optimization, ID-Vault and other security features, DDM and other technologies are not adopted.

    So this is the upgrade chance for a Domino 11 upgrade project while talking benefit of current features ;-)

    See the full HCL Life Cycle document below. Thank's Vlad, for the link! I did not see it yet.


    • [IBM Lotus Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]