Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    SELinux Support for Domino

    Daniel Nashed  22 January 2020 14:47:54


    There is a AHA idea to have Domino support SELinux --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-1121
    My impression was that SELinux is already supported with current Domino releases.
    I asked HCL and it turned out that SELinux is not tested, thus it is currently not supported.
    It would be extra test effort for every distibution and version to run with SELinux.

    Security-Enhanced Linux (SELinux) is a security architecture for Linux® which is integrated in the kernel and allows a separate security layer.
    It has been originally developed by the NSA and is today integrated in the kernel.

    You also have distinct between different SELinux modes. I was very sure the strict mode would not be supported.

    But I thought the default mode "enforce" mode with "target" policy would be supported -- but it is currently not.
    Below is a short introduction directly from RedHat. And if you are interested in details there is a video of a great presentation linked below.

    When I talk to Domino admins they either don't know about SELinux but are told to disable it.
    But there are companies who really have to enable SELinux.
    In fact I have customers who run it today in enforce/target mode without knowing -- because it's default.

    I would be very interested to hear your feedback. Do you want to use it? Do you have to use it? Are you using it?

    You can either comment here,on the AHA idea or both. And if you find SELinux important to have supported, you can vote on the AHA idea.
    But on top of the vote please leave a comment which requirements you have in detail?
    Is enforced with targeted policy OK? Do you need a profile for Domino (that would be a lot of work and has impact on deployment, troubleshooting etc).

    To check if SELinux is enabled and in which mode, you can use the following command:

    sestatus
    SELinux status:                 enabled
    SELinuxfs mount:                /sys/fs/selinux
    SELinux root directory:         /etc/selinux
    Loaded policy name:             targeted
    Current mode:                   enforcing
    Mode from config file:          enforcing
    Policy MLS status:              enabled
    Policy deny_unknown status:     allowed
    Memory protection checking:     actual (secure)
    Max kernel policy version:      31

    -- Daniel


    References

    Video
    https://www.youtube.com/watch?v=_WOKRaM-HI4

    Public Documentation
    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/using_selinux/index


      Notes & Domino 10 - Language Support Statement

      Daniel Nashed  17 January 2020 20:39:19
      Many of us are waiting for an update on the language support especially for Notes clients.
      It turned out to be much more work than they thought and there is still a lot of work to do.
      But they are getting there and for the next updates and with the new process, this should be much faster in future.

      German and Japanese was already available in Notes/Domino 11 Beta 2 and has shipped with the GA release.
      We can expect more languages to be shipped with GA but this time we still have to wait.

      I would have wished we would have got that info already around GA time. But even it takes some more time, this statement is good news.

      They are shipping Notes & Domino 10.0.1 languages first, because of customer deployments and I would expect they will ship the same languages with 11.0.1 hopefully.

      See details here...

      https://www.cwpcollaboration.com/blogs/important-languages-support-update-domino-notes-vop-and-volt

      -- Daniel



      Building Domino C-API applications for RHEL/CentOS 8 - Lib dependency

      Daniel Nashed  16 January 2020 23:35:17


      Now that Domino 11 officially supports RHEL/CentOS 8.x I am testing my applications.

      I ran into a changed lib version for Domino C-API applications which compile and link with makefiles derived from the examples shipped with the SDK.
      Those examples use the lib "-lnsl".

      RHEL/CentOS 8 changed libnsl.so to a new version (libnsl.so.1 --> libnsl.so.2).

      The lib is usually not needed for most applications unless you are using cetain TCP/IP operations as far I found out.

      The error you see, when starting the application is the following:


      /opt/hcl/domino/notes/latest/linux/nshtest: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory


      So when building your applications you can usually remove the dependency.
      Domino itself doesn't have the dependency.

      Just to remind you: The support build environment for Domino 10 and Domino 11 on Linux is still CentOS 7.4 with the gcc/gcc++ compiler shipped with that version!

      This is a about the run-time environment and the changed lib dependencies. And again, this is only for specific add-on applications.
      I added the exact error message to let Google to pick it up...

      Here is the original link line form the examples:

      # set LIBS to list all the libraries ld should link with.

      LIBS = -lnotes -lm
      -lnsl -lc -ldl -lpthread -lresolv

      Here are the libs installed by default:

      RHEL 7

      /usr/lib64/libnsl.so.1
      /usr/lib64/libnsl.so

      RHEL 8

      /usr/lib64/libnsl.so.2
      /usr/lib64/libnsl.so.2.0.0

      Domino 11 support for RHEL and CentOS 8.x

      Daniel Nashed  11 January 2020 12:39:57
      The Domino 11 system requirements technote has been updated and does now include RHEL 8.x and CentOS 8.x -- SLES 15.x is still not on the supported list.

      So within a major release, all the minor releases are supported.
      RHEL 8.1 is already available and for CentOS we are waiting for the update.
      We have tested Domino 11 already with those Linux versions at beta time and also with Docker.

      For Docker the default will stay with CentOS 7 Latest for now.
      There is a separate dockerfile for centos8, which you can use with the standard build command. Just add "dockerfile_centos8" to the build line ;-)
      This isn't documented in the project but is now that CentOS 8 is officially supported also blessed by the Docker project :-)

      No, I don't know if HCL will also support Domino 10.0.1 FP4 on Version 8.x.

      -- Daniel

      Reference to technote:
      https://support.hcltechsw.com/csm?id=kb_article&sys_id=5d9fe4311b7d885083cb86e9cd4bcb6d


      Domino 11.0.0 on Linux Command or option is ambiguous

      Daniel Nashed  8 January 2020 15:13:09


      We ran into this during an update and I thought this was just me. But I got a mail from a partner today with the same issue.

      I would really recommend installing Domino 11 into a new binary directory and switch the path to the new default /opt/hcl/domino.
      But in case you cannot do that, you might run into an issue, because of you locale.
      I blogged about "res" files before and they have a separate "known issue" section in my start script.

      On Windows the string resources are part of the binaries. On Linux and UNIX in general Domino uses string resources as separate files.
      This is an abstraction layer for translation. And in earlier times Domino had string resources for different languages.
      Today we only have English res files and also the Windows servers are just having English.

      But the res files are still around on Linux and can cause issues.

      In this case on Domino 11.0.0 when res files are installed into different directories, it could happen that you see wrong messages on the server and it happened that the server console does not accept any commands:


      [006527:000010-00007FE24B326700] Command or option is ambiguous



      When you install into the new default location
      /opt/hcl/domino you will not run into this.
      But if you are updating your server the res files might get confused.
      Below is an example how the res file directories look like.

      Depending on the language of your installation you might have a different directory name for the res files.
      Either the POSIX locale (default) C is used.

      Or you have a directory in your language for example en_US.UTF-8. This is usually the locale which the root user used during installation!
      Your Notes user could have a different locale, so in some cases you don't see messages but hex strings on the console.

      In our new case something else has happened. There are two directories and the server us using the old directory on startup.
      C is the fallback locale if your locale has no corresponding directory name.

      My personal best practice is to use the POSIX locale for installation to get res files installed into the "C" directory.
      And I create a sym-link for the other locales which I might need:

      /opt/ibm/domino/notes/latest/linux/res/


      drwxr-xr-x 2 root root 4096 20. Dez 16:24 C

      lrwxrwxrwx 1 root root    1 20. Dez 16:35 de_DE.UTF-8 -> C

      lrwxrwxrwx 1 root root    1 20. Dez 16:35 en_US.UTF-8 -> C



      This also avoids, that the installer updates files in a different directory, than what is used later on by your server.

      In any case, if you run into it, just check the date of the file and correct the directories accordingly.

      -- Daniel

        Docker Project Update - Domino 11 & Traveler 11 Support

        Daniel Nashed  6 January 2020 08:20:59


        During the Domino 11 beta we already updated the Domino and also the Traveler versions in the "develop" branch.

        We used it for our own testing and it was also documented for the beta community. There was a special build option for the beta.


        When Domino 11 and Traveler 11 was released 20.12.2019 we updated the develop branch to the V11 GA version and removed all the beta downloads and references.

        This weekend we have also updated the master branch to V11 -->
        https://github.com/IBM/domino-docker.

        The project is updated to build Domino 11 by default and tag it as "latest".
        Also Traveler 11 will build on "latest" Domino version. This means the Traveler 11 image will be now based on the Domino 11 image by default.

        The configuration files in the management directory have also been updated to use V11.

        You can still build Domino 9 and Domino 10 images with different FP versions, if you specify the version explicitly - the software.txt still contains all versions.


        A Docker image is intended to auto update and always run the latest version.

        So if you start from scratch, you will be using V11. If you have existing containers, you can update them to V11 in seconds once you build the new image.


        Yes the Docker image does update data directories for Domino and Traveler servers as well!

        The first start after an update will detect the changed version and will apply template changes.

        All templates and other files are added to a compressed tar file, which is part of the image.


        We are also using a new V11 notes.ini SERVER_UPGRADE_NO_DIRECTORY_UPGRADE_PROMPT=1 parameter introduced in Domino 11 for those kind of upgrades.

        This parameter updates the design of the Domino Directory on admin server without prompting and without logging.


        So once you switched to a new image, the upgrade of your server will happen automatically.

        If you want to see detailed logs about an installation or an upgrade check the /domino-docker directory in your container for details.

        This directory contains all the Domino Docker specific information (only the entry point script and the health check script are still located in the /).


        Below you find an output log for an upgrade from V11 Beta2 to the GA version.


        Enjoy and we are really looking forward to your feedback.


        Thomas Hampel

        Daniel Nashed


        -- Example Domino 11 Update --


        Once you built the new image, it's a single command if you are using the management script on a local Docker host...

        The following is using our Docker management script. But of course this will work similar in other environments running the image as well.



        ./docker_domino inspect

        (Using config file /local/cfg/config_domino)


        Info: New Image Version available!


        ------------------------------------------------------------------------------------------

        Status         :  running

        Health         :  healthy

        Started        :  10.12.2019 15:45:59

        Name           :  noteslab-domino11

        Image          :  hclcom/domino:11.0.0.BETA2

        Version CNT    :  11.0.0.BETA2

        Version IMG    :  11.0.0

        Image Size     :  1474 MB


        Domino Ver CNT :  11.0.0.BETA2

        Domino Ver IMG :  11.0.0

        BuildTime CNT  :  23.11.2019 18:28:42

        BuildTime IMG  :  20.12.2019 17:50:05


        Hostname       :  noteslab-domino11

        Volumes        :  notesdata_noteslab1

        NetworkMode    :  default

        IPAddress      :  172.17.0.2


        Platform       :  linux

        Driver         :  overlay2

        ------------------------------------------------------------------------------------------

        Container ID   :  b895e7c101c9

        Image-ID CNT   :  31d3c36e8a75

        Image-ID IMG   :  ba3fc0807b95

        ------------------------------------------------------------------------------------------

        Docker Ports   :

                         1352/tcp -> 0.0.0.0:1352

                         25/tcp -> 0.0.0.0:25

                         443/tcp -> 0.0.0.0:443

                         80/tcp -> 0.0.0.0:80

        ------------------------------------------------------------------------------------------



        ./docker_domino update

        (Using config file /local/cfg/config_domino)


        Info: New Image Version available!


        Updating Container [noteslab-domino11] ...

        Stopping Container [noteslab-domino11] before update ...

        noteslab-domino11

        Removing Container [noteslab-domino11] ...

        noteslab-domino11

        Creating & starting new Container [noteslab-domino11] ...

        d08ecb2c9cb3a7244beb009694b232b14f51a37dd4cccf83a82c14b204067eed


        Successfully updated Container [noteslab-domino11]


        RNUG Conference in St. Petersburg in August 2020

        Daniel Nashed  4 January 2020 19:54:08
        Image:RNUG Conference in St. Petersburg in August 2020



        RNUG 2019, the first large Domino conference October in Moscow was my big conference surprise of the year!

        I never have been to Moscow. Getting the visa took a while and I was a bit concerned how Moscow will be.
        But I was very positive surprised by Moscow and also the very well organized event!
        It was impressive what the team did an quite a short time and I was very happy to be a the event!

        Beside my performance session and the Domino on Docker session together Thomas Hampel from HCL, we had a very interesting Linux round table with a very productive discussion about Domino support for Astra Linux.
        The result was that HCL will look into Astra Linux support for the Russian market.

        Even language was a challenge, everyone tried their best to communicate and beside the simultaneous translation in the main room, there have been interpreters helping us.

        This year the conference will be in St. Petersburg in August and I am already looking forward to this event!

        St. Petersburg has big history and I am sure we will have another perfect conference!
        And the town and the whole area will be probably great in August.

        Because it is a tourist area, getting a visa will be much easier. There is a evisa you can apply online --> https://stpetersburg.russia-evisas.com.

        So I am really looking forward to the conference already.
        The exact date isn't announced yet. But the website shows all the pictures from last years conference --> https://en.rnug.ru/


        -- Daniel

        Domino Start Script Update V 3.3.0

        Daniel Nashed  2 January 2020 16:52:42


        The new version of the Domino Start Script is available.

        Basically there are 3 new features/changes

        Updated AIX support including supporting the install script.

        Updated Container support which is needed for other container implementations. In Version 3.2.2 only Docker was properly detected.
        I have been using this feature already for the Domino on Docker script and it is now included in 3.3.0.

        A bigger back-end change is the way the server console works.

        Until now the command read from console is echoed into the notes.input file with >>.
        For some reasons we never really figured out in detail, even the text was written into the input file (which is used as an input file with <, when the server is started),
        in some cases the server process does not accept those commands any more until the server is restarted.

        While troubleshooting a customer came up with another approach to send the command over to the server, which worked more reliable in our tests.

        The server command is send to the running server via  server -c "command". To get this reliably working,
        I have added some sanity checks for the input read from the console and also I am checking if the server is running.

        This new behavior is enabled by default and you can revert back to the existing functionality with a new config parameter.

        In addition this allows now also a live console when using the server controller (by figuring out the output file from notes.ini),.

        The input redirection file is still there but not used any more. This functionality has been like this from the very beginning and I wasn't sure I want to change it.
        But it appears to work reliably. Let me know what you find out in your environment.

        -- Daniel


        V3.3.0 01.01.2020


        New Features
        ------------

        Updated container support for other container environments than Docker (detecting other container run-time environments)

        Updated support for AIX including install script


        Changes
        -------

        Changed live console functionality

        Up to now the live console wrote into the notes.input file which is connected to the server process (< input file).
        With the new functionality the commands are send to the server via server -c  "command".
        This change is intended to solve an issue with a stall console in some situations.
        In addition this allows live console functionality also in combination with the server controller.
        The script detects the current server controller file (via notes.ini setting DominoControllerCurrentLog).
        You can switch to the previous behavior via DOMINO_CONSOLE_SERVERC=NO.

        Removed legacy configuration from rc_domino_script, which was confusing


        Traveler Install requires configured server to detect Domino version

        Daniel Nashed  31 December 2019 21:48:11
        Now that the second customer ran into it, let me explain why this happens and how to prevent it.

        Traveler 10.0.1.2 and Traveler 11 can be installed on top of Domino 9/10/11.

        All 3 different Domino releases need different Traveler binaries to be installed.
        Therefore it is essential that the Traveler setup can detect your Domino version.

        The current Java routine only works if the Domino server is configured and has a vaild server.id.

        Traveler 10.0.1.2 does assume Domino 11 if no platform can be determined.

        In that case on a Domino 10.0.1 server you would get the wrong binaries installed, which will not work..

        If you are installing a new server, you should configure Domino first before installing Traveler.
        In some scenarios you are installing a new machine to take over the data of an existing Traveler server.
        To get your server up and running you either have to use a silent install specifying the Domino release explicitly (like we are doing in the Docker project) or you have to copy the basic Domino configuration before installing Traveler.

        You find the detected version in the Traveler install log -> /local/notesdata/IBM_TECHNICAL_SUPPORT/traveler/logs/TravelerInstall.log

        It looks like this:

        Domino returned version string: Release 9.0.1FP10HF382 | November 19, 2018

        DominoVersionAction
        Domino version detected: 9.0.1

        In case you want to use the silent install option, the parameters would be as follows:

        INSTALLED_DOMINO_VERSION=10.0.1
        BYPASS_DOMINO_VERSION_CHECK=true

        I hope this helps

        -- Daniel

        20 Years Domino on Linux

        Daniel Nashed  31 December 2019 11:14:30

        The end of the year is always a good time to look back.

        You might now that I am looking into Domino on UNIX for quite some time.

        But I just ran into an old CD looking for something else when I realized that we have Domino on Linux now for 20 years!


        Linux changed the IT world


        I think Linux changed the IT world dramatically! Without Linux, which also made open source popular, we might not be where we are today. Linux is running the Internet.

        And I am pretty sure the IT world look completely different without Linux today!

        It is a remarkable and outstanding effort Linus Torvalds did at that time and does today!


        Linux is everywhere and even Microsoft is looking into implementing Linux technology in their products and releases software for Linux (for example the SQL server which is even available as a Docker container).


        Last week I got a link on Facebook to Linus Torvalds Master of Science Thesis (see link).

        And I found two interesting videos (linked below), which might be interesting to look into ..

        In combination of finding this old first Domino on Linux disk, this lead to this blog post ..


        -- Daniel






        My first contact with Linux


        When I started in IT in 1990 after school there wasn't any Linux in the commercial world.
        At that time I worked with TeX instead of Word which I downloaded at the university in D├╝sseldorf, where a friend studied mathematics and IT.

        We got Linux and TeX from the internet (downloaded via FTP to floppy disks) and I wrote all my papers with TeX instead of office (actually I should look into this again today ).

        I have been using TeX (a very special digital typographical systems) on Windows with the at that time famous emTeX implementation.
        And I looked also into Linux as a operating system for fun for education purposes and fun.


        HP-UX my first commercial contact with UNIX on Domino


        The first Notes production environment I set up was running on version 4 on HP-UX.

        There have been just a handful of installations in Europe from what I recall -- even HP moved to Windows NT instead of HP-UX.


        My first production environment was a jointed effort with Spanish guy, who was working for Lotus Professional services in the UNIX competence center in Paris, visiting me in Essen (Germany) to install our production environment.

        We had a lot of fun installing this first server and he was the first one I could sync up with about Notes on UNIX.

        At that time the first UNIX start script was born. Which I later on adopted also to Solaris and AIX. And finally years later for Linux.


        Domino on Linux


        When I started with Notes around 1995, Linux wasn't used in the commercial world.
        For me this changed end of 1999 when IBM started to look into Linux (actually one of the developer ported it already earlier at home as a skunkworks).


        The first versions Domino on Linux wasn't very scalable due to some kernel resource limitations.

        We needed one lightweight process for each users, because we had no pthreads yet and also no sys-epoll implementation.

        Later on Linux 64 became one of the most popular Domino platforms beside Windows 64. And it is very scalable.


        Domino was early on running on Linux and I have been always a big fan for Domino on Linux.



        25 years of Linux in 5 minutes


        https://www.youtube.com/watch?v=qFTIc5frqw8


        TED - The mind behind Linux | Linus Torvalds


        https://www.youtube.com/watch?v=o8NPllzkFhE


        Linux: a Portable Operating System

        Master of Science Thesis


        Linus Torvalds, Helsinki January 31, 1997


        https://www.cs.helsinki.fi/u/kutvonen/index_files/linus.pdf

        Archives


        • [IBM Lotus Domino]
        • [Domino on Linux]
        • [Nash!Com]
        • [Daniel Nashed]