Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

    Notes & Domino & Traveler 11.0.1 available for download

    Daniel Nashed  31 March 2020 21:31:33


    Notes & Domino & Traveler 11.0.1 is available for download!

    I have not seen Sametime 11 FP1 yet. But it should be available shortly ..

    Here is a very quick summary of my personal highlights ..

    Traveler 11.0.1

    For Traveler a long waited new feature is the cross Domain ID-Vault Support.

    This is based on a back-end enhancement in Domino 11.0.0 which needed also a code change on the Traveler side to support it.
    And you also need a notes.ini parameter to enable it -> IDV_ENABLE_CROSS_DOMAIN=1

    You only need this new version on the Traveler server which is requesting side. There is no change needed on the Domino server hosting the ID vault or the mail-server.

    For details see this technote --> https://help.hcltechsw.com/traveler/11.0.0/Plan_Domino_domains.html


    Domino 11.0.1

    The other great feature is SAN x.509 Cert support in combination with SNI support.

    So now finally we can have one IP address with multiple SSL/TLS enabled websites.

    I have tested this feature already and it is the reason why I have to finally move my server from CentOS 6.10 to CentOS 8 to update to Domino 11.0.1 :-)

    There are a couple of other features in detail, which are well described here --> https://help.hcltechsw.com/domino/11.0.1/whats_new_in_domino11.0.1.html


    Domino 11.0.1 HCL Docker Image

    There is also a ready to go Docker image from HCL.
    It's a first version which is planned to be the new delivery model for Domino beta releases as well.
    The image needs to be manually configured. And is currently only supported on Docker CE 19.x

    But I hope that we will see more from HCL as a full image. And I would also wish extensibility and auto configuration.

    See details here -> https://help.hcltechsw.com/domino/11.0.1/inst_dock_domino_overview.html

    We have also updated the develop branch of our Community Domino Docker image to Domino 11.0.1.
    And we are looking into updating Traveler as well before we push the changes to the master branch ..

    https://github.com/IBM/domino-docker/tree/develop


    -- Daniel







    Domino Docker Project update -- OpenShift support & Podman+Docker+K8s support for "arbitrarily assigned user ID"

    Daniel Nashed  15 March 2020 12:49:55


    A couple of days ago I got the request from Daniele Vistalli, a fellow HCL Master, that he needs support to run Domino on Kubernetes (K8s) with a so called "arbitrarily assigned user ID".
    He and the team around him are doing incredible work to finalize Factor-y's MSP offering which included Domino on K8s as part of their platform.
    Running with distinct UIDs is an important security aspect, when offering cloud services separating data between containers -- and even more important tenants.

    On OpenShift the concept of a "arbitrarily assigned user ID" is a strong requirements for images to run. They assign a new UID for the container to run for security best practices.
    In general a container is quite safe already without it. But to reduce exposure to potential security issues, they don't only forbid to run a container with root but also assign an unique UID to each container on the fly.

    Quote from a RedHat technote:
    "When OpenShift starts a container, it uses an arbitrarily assigned user ID.
    This feature helps to ensure that if an application from within a container manages to break out to the host,
    it won’t be able to interact with other processes and containers owned by other users, in other projects."
    (1)

    K8s doesn't have this strong requirement. But in Docker and K8s you can specify a UID manually when the container is initiated.
    On Docker the command line option is e.g. --user 1234. And there is a K8s equivalent.
    But this will cause issues with "whoami" and other code trying to figure out about the user.
    So we had to add code to modify the /etc/passwd in a safe way (application runs with root permissions having the sticky bit set -- like bindsock does).

    OpenShift and also Podman in current versions automatically modify /etc/passwd and add the UID with it's numeric value as a user like this:

    notes:x:1000:1000::/home/notes:/bin/bash
    1025570000:x:1025570000:0:1025570000 user:/:/sbin/nologin

    "The OpenShift run-time CRI-O (starting from OpenShift 4.2 onward) now inserts the random user for the container into /etc/passwd." (1)

    So for OpenShift and also when you use Podman (which also used CRI-O) the platform already takes care of adding the UID into /etc/passwd.

    For older versions and also for Docker/K8s, the work-around is to modify the "notes" user in /etc/passwd with the group 0.
    Podman is also using CRI-O and works similar to current OpenShift also adding the UID to /etc/passwd.

    So depending on the container platform there are different approaches.
    I have tested the the different images for CentOS 7 and RedHat UBI 8 on all the platforms. And it looks good so far.
    The changes are checked into the development tree --> https://github.com/IBM/domino-docker/tree/develop

    Beside those changes I also moved all the scripts into a central location /domino-docker/scripts and made sure only root has write permissions to this folder.
    This is also the preparation for future extensibility, that partners can have their own hook points to be executed for example during server configuration or startup.

    If you want to get your hands on OpenShift, there is a free 30 days trial offering directly from RedHat --> https://manage.openshift.com.
    You are up and running in minutes and it has a very clean graphical interface and provides the "oc" command line, which offers all the K8s commands.


    -- Daniel


    (1) Reference: https://access.redhat.com/articles/4859371

    Important Information Traveler 11 for iOS in multi-domain environments

    Daniel Nashed  26 February 2020 00:20:40

    Starting with Traveler 11 Active Sync is supported and enabled by default.
    It turned out that AS 16 support comes with a new back-end API which has side effects in multi-domain environments.
    If your Traveler server is inside your main domain, there are no side effects.

    For a Traveler server just hosting one domain, the easiest way might be to move your Traveler server into your main domain.
    If your Traveler environment hosts users in multiple domains, you might want to wait until a solution has been found.
    Or you could disable AS16 for now using the described work-around --> NTS_AS_PROTOCOL_VERSIONS=2.5,12.0,12.1,14.0,14.1

    If you haven't updated your multi-domain environment to Traveler to Version 11, you might want to wait until the limitations has been solved.
    This might require also a Domino side fix, because the current C&S API doesn't provide the multi-domain functionality.

    See details here:

    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0076165

    -- Daniel

    SLES 15 is now supported for Domino 11 & Sametime 11

    Daniel Nashed  12 February 2020 17:49:41
    The system requirement technotes have been updated...

    Now SLES 15 is listed as supported for Domino and Sametime.


    SLES 15 is different from the previous versions. I tried to look into it with the GA version and looked into it again with SP1.

    You should only run it with SP1. And if you can do the online configuration and not just off-line from DVD.. They changed the installer .. What can I say ...


    I have tested Domino 11 already with SLES 15 SP1 and it just works.
    And I spent a lot of time getting Sametime installed on CentOS over the weekend and I spent the whole evening installing it on SLES 12 SP5.

    I would not expect much more difficulties with SLES 15. But SLES 12 was already a challenge ..


    But the good news I figured out why my start script wasn't working with SLES. The rc init.d functions from SuSE always broke the ST status website.
    Now that we don't have init.d I made a fix for the start script to not use the rc init.d code from SuSE in combination with systemd. That finally fixed the issues I had with Sametime in combination with my start script on SLES. Still testing.. If someone needs the changed version let me know ...

    Update 15.2.2020:

    On SLES 12 I was able to install Mongo 3.6. And we had some missing OpenSSL *.so version dependencies which have not been resolved. Setting symbolic links helped.
    But on SLES 15 SP1 -- which is the current version you would run, when using SLES 15 -- Mongo DB is supported starting at version 4.2.1. This version isn't supported by ST 11 yet. And even a standard Mongo DB 4.2.x installation from the original repro failed with the same OpenSSL version dependencies.

    So for now you should not try to install ST 11 on SLES 15 until this is resolved!

    -- Daniel



    HCL Domino 11.0 Detailed System Requirements  


    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074573

    HCL Sametime 11 System requirements


    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074454

    Converting Timedate Strings in @Forumula Language with different date formats

    Daniel Nashed  11 February 2020 12:19:32

    Maybe I am thinking to complicated.. But I did not find an easier solution ..
    I had an interesting issue yesterday when adding multi timezone and date-time format to my backup solution, because I cannot store everything in Notes native TIMEDATES.
    There are some text logs involved, when storing information about a backup. To ensure I have an unified format, I decided to change the internal storage to UTC (in Notes terms: GMT).

    --------------------

    Update 12.02.2020: Yes I was thinking to complicated, because I missed the function @Date which takes the date components as numeric values in a fixed order.


    @Date( time-date )
    @Date( year ; month ; day )
    @Date( year ; month ; day ; hour ; minute ; second )

    After feedback from Simon (huge thanks) and also with some discussions with my friend Rudi Knegt, who is also an old Lotus Formula fan like me, I now have the following solution to convert my date. I actually need the date as text and as time for comparison. That's why I did the conversion of the full date this way at the end. So my use-case need the text for selection and the date for comparison.

    X:="2020.02.11 01:02:03 GMT"; Z:=" ";C:=@Explode(X;Z); D:=@ToNumber(@Explode(C[1];".")); @if(@Elements(C)<3;"invalid timedate";@Elements(D)<3;"invalid date";@ToTime(@Text(@Date(D[1];D[2];D[3]))+Z+C[2]+Z+C[3]));


    By the way. Without knowing the other parameter options @Date function we had another idea taking benefit of the wrong text to date conversion. If you convert a date with the day>12 the result can be used to replace the result with the date format I would have needed.
    That solution would not have needed the Lotus Script code in my first solution below. But with the @Date function it's more straightforward.


    My final solution in my actual code looks a bit different and just needs one explode. But for demonstration purposes in this context it makes sense.

    Thanks! This is true community spirit!

    -- Daniel

    --------------------


    With my German locale I ran into limitations when converting the text back into a date.
    It turned out that even when I use a string that is YMD formatted, the day/month order is still from my German locale and causes conversion issues for the first 12 days in a month.
    This wasn't what I expected because there are those 3 different settings internally: DMY, MDY and YMD.

    Example with German settings:

    @ToTime ("2020.
    02.10 11:22:33 GMT") --> 02.10.2020 13:22:33
    @ToTime ("2020.
    10.02 11:22:33 GMT") --> 10.02.2020 12:22:33
    @ToTime ("2020.
    02.14 11:22:33 GMT") --> 14.02.2020 12:22:33


    In my case my internal format is always "yyyy.mm.dd hh:mm:ss GMT" and I have to convert to a correct TIMEDATE.
    I had to use Lotus Script in the DB init event to get the international settings and store them in an environment variable to be used in my form.

    With that client specific setting, I am converting the date to the current settings. From there on I can use @ToTime to convert it correctly.

    D:=@Word(x; " ";1);T:=@Word(x; " ";2);Z:=@Word(x; " ";3);@ReplaceSubstring(DateFormat; "Y":"M":"D":"T":"Z";@Word(D;".";1):@Word(D;".";2):@Word(D;".";3):T:Z);


    If someone finds a more straight forward way to convert this given format with different international settings, I would love to make it easier.
    Specially reading the international settings via Lotus Script and passing it to my formula is ugly ..

    -- Daniel


    Example:

    With --> x:="2020.02.10 11:22:33 GMT"; DateFormat:= "D.M.Y T Z";

    The RESULT is --> 10.02.2020 11:22:33 GMT

    Form there on I can use @ToTime to convert it correctly independent from the date settings of the client...

    DateFormat:=@Environment( "DominoBackupDateFormat");

    Sub Initialize
           Dim session As New NotesSession
           Dim international As NotesInternational
           Dim DateFormat As String
           
           Set international = session.International
           
           If international.IsDateDMY Then
                   DateFormat = "D.M.Y T Z"
           Elseif international.IsDateMDY Then
                   DateFormat = "M.D.Y T Z"
           Elseif international.IsDateYMD Then
                   DateFormat = "Y.M.D T Z"
           Else
                   DateFormat = "D.M.Y T Z"
           End If
           
           Call session.SetEnvironmentVar( "DominoBackupDateFormat", DateFormat )
           
    End Sub


    Domino SMTP error limit before terminating connections

    Daniel Nashed  10 February 2020 09:11:15

    My friend Harvey and I noticed a lot of brute force delivery attempts on servers to figure out about email addresses.
    By default Domino doesn't limit the number of errors until a connection is closed.
    But there is a notes.ini entry and setting in config document to define the number of errors.

    https://help.hcltechsw.com/domino/11.0.0/conf_definingthemaximumerrorlimitbeforeaconnectionterm_t.html

    For larger servers you have to be careful, because that could also prevent newsletters from being delivered, if there are too many users which don't exist.
    On my own server I set the value now to 4 ;-)

    And if you have more errors -- this includes invalid recipients -- the connection is closed with a temporary error.
    But that also means that the mail is not received and the other server would try again.
    So you have to be a bit careful with this parameter. I would sent it do a higher value like 20 on larger servers.

    421 domino.nashcom.de SMTP service not available, closing transmission channel


    Example from log:
    10.02.2020 06:21:53   SMTP Server: 185.143.223.xxx connected
    10.02.2020 06:21:54   SMTP Server: Mail for morris@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for mom@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for buy@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for abcdefg@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for az@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for schmidt@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for babbar@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for edith@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
    10.02.2020 06:21:54   SMTP Server: Mail for juliet@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.



    Image:Domino SMTP error limit before terminating connections



    Notes/Domino 10.0.1 Fix Pack 4 Released

    Daniel Nashed  9 February 2020 15:55:48

    Notes & Domino 10.0.1 FP 4 has been released before the weekend.

    Now that Notes/Domino 11 is released, my focus is more in the Notes/Domino 11 release.

    But for customers running the 10.0.1 code stream, this is an important update!


    I have downloaded the Linux version on and updated our Docker image to support FP4 (currently in the develop tree) last week.

    But the default for Domino on Docker is 11.0 of course.


    Find the official URLs for FP4 including the system requirements.


    By the way the official URL for the support website is
    support.hcltechsw.com -- There have been multiple names around, but I got the confirmation from support, that this is the one we should use and this is what they will use for all the links.
    Some other links will not work without authentication.


    -- Daniel


    Notes/Domino 10.0.1 Fix Pack 4 Release Notice


    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0075554


    Notes 10.0.1 Fix Pack 4 System Requirements


    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074997


    Domino 10.0.1 Fix Pack 4 System Requirements


    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074995

    /proc/sys error messages Domino on Linux

    Daniel Nashed  6 February 2020 08:05:15

    The Domino start script now leverages "server -c" commands instead of writing into the server's redirected input file since the latest version as blogged earlier.
    If you use the server binary for that purpose, some side effects could occur, which are not new. But you might not have noticed them.

    When the server process starts, it tried to set kernel parameters to optimize certain network parameters.
    Domino itself uses a none privileged user (default: notes).  To be able to set the kernel parameters a helper binary called "tunekrnl", which has the sticky bit set, is used.
    The sticky bit runs the process as root and allows those restricted operations.

    But even with root permissions in some Linux server environment those changes are not allowed.

    For example in paravirtualized environments, where you don't have your own "root" server like when using virtual servers from HostEurope, your server doesn't allow those operations.
    Another case would be a Docker environment where you are also not allowed to change those kernel parameters.

    In those cases you see error messages similar to this:

    Error messages on console

    Error - can't open /proc/sys/fs/file-max.
            errno: 13
            Permission denied
    Error - can't open /proc/sys/net/ipv4/tcp_fin_timeout.
            errno: 13
            Permission denied
    Error - can't open /proc/sys/net/ipv4/tcp_max_syn_backlog.
            errno: 13
            Permission denied
    Error - can't open /proc/sys/net/ipv4/tcp_tw_reuse.
            errno: 13
            Permission denied
    Error - can't open /proc/sys/net/ipv4/ip_local_port_range.
            errno: 13
            Permission denied

    As long this was just on server start, I ignored those messages. But in a remote console for every command you want to get rid of them.

    In our Docker project we are already removing the tunekrnl during install. This avoids those error messages.

    If you are on virtualization platform, which provides you are full virtualization stack -- like ESX, you should not receive an error.
    In that case you should check the permissions of the file and not remove it!


    It should look like this (the file needs to be owned by root and the sticky bit should be set -- the s in the 4. position)

    ll tunekrnl
    -r-sr-xr-x. 1 root daemon 71768 Nov 25 08:33 tunekrnl


    If the settings are wrong, here is the way to change them. The following is a more paranoid setting. The file is owned by root and the group notes and you are allowing user's from the group notes to execute the file with root permissions.

    chown root:notes /opt/hcl/domino/notes/latest/linux/tunekrnl
    chmod 4550 /opt/hcl/domino/notes/latest/linux/tunekrnl


    By the way : There is one other file in Domino which needs root permissions to run. bindsock  is used to allow processes to listed on restricted ports below 1024. So bindsock is a helper binary to allow those operations.

    -- Daniel



    Traveler 11 HTTP/2 Push changes in detail - Review before updating

    Daniel Nashed  5 February 2020 09:21:07

    Detlev Pöttgen and I ran into this with the first deployments and we put together the details behind it (so you will find the same blog post on his blog and my blog).
    There will be a documentation update from HCL side. Here is what we found out in detail with the feedback we got from the Traveler team.



    Beginning with Traveler 11 the new push API is used -- > https://developer.apple.com/news/?id=11042019a
    This is the new recommended push service from Apple which every service should use.
    The older API will be available until November 2020!

    There is no change needed to change to the new API and Traveler uses the new push API by default.


    But your infrastructure also needs to be ready for this change!

    If you have to disable the new API to go back to the "legacy API", because you can't change your infrastructure right now, there is just one notes.ini parameter that you need to set:


    NTS_PUSH_APNS_HTTP2=false


    But you should only use this as a very temporary solution and switch to the new push API as soon as you can!

    The statement in release documentation is quite short and doesn't go into the details of what this might mean for your environment. Let me explain the changes in detail:


    New Protocol HTTP/2


    If you are behind a proxy, you have to check if your proxy supports the HTTP/2 protocol!  You might run into connectivity issues depending on the proxy.


    Port change from 2197 to 443


    The new port used is the standard HTTPS port 443 instead of  the APNS "legacy" port 2197.

    You have to check your firewall if the port is open! Usually network admins are more happy with the standard port 443 but it might not be open in your environment by default!


    The new HTTP/2 Push services is also available on port 2197 on the new servers to allow a more smooth migration.

    There are NTS parameters you could use to change the default port 443 to the old legacy port if you really need to.


    There are specific settings for each different push service and they look like this:


    Example for the Verse app: NTS_PUSH_APNS_APPLE_VERSE_IBM_PRODUCTION_SERVER_PORT


    If you don't set the parameter explicitly the NTS_PUSH_APNS_HTTP2 will take care of changing the port to 443 for all push Apple services.

    So this is more a work-around which you should only use for example if it takes time to change your firewall.



    Change from gateway.push.apple.com to api.push.apple.com


    Also the target servers have changed. Before Apple used gateway.push.apple.com and switched to api.push.apple.com
    Usually there isn't any change needed in your infrastructure. I checked which servers are currently behind the DNS entries and they are coming from the same netblock at Apple (see references below).



    Conclusion


    You really have to check your environment to see if you are prepared for new APNS HTTP/2 API.

    Not having the right prerequisites isn't a reason to not update to Traveler 11. You could use the legacy API for a couple of weeks or some of the settings above might help you to get it working for your environment.

    If your environment uses APNS Push, you have to migrate your environment to Traveler 11
    before November 2020!


    Daniel



    References:



    Developer Information for APNS


    https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/

    Current DNS Settings
    :


    gateway.push-apple.com.akadns.net

    Aliases: gateway.push.apple.com

    17.188.142.26
    17.188.140.27
    17.188.128.157
    17.188.132.21
    17.188.130.28
    17.188.133.27
    17.188.136.22
    17.188.138.21


    api.push-apple.com.akadns.net

    Aliases: api.push.apple.com

    17.188.161.182
    17.188.161.11
    17.188.161.203
    17.188.160.13
    17.188.164.15
    17.188.162.16
    17.188.163.207
    17.188.161.13



    Whois Extract for Apple Net-Block


    NetRange:       17.0.0.0 - 17.255.255.255
    CIDR:           17.0.0.0/8
    NetName:        APPLE-WWNET
    Organization:   Apple Inc. (APPLEC-1-Z)




      Quick change to default font with smart icon

      Daniel Nashed  2 February 2020 15:50:16

      The Notes standard Client has a short list of last used fonts. It's not exactly working like the last languages in the propery box of the document.
      And it isn't available in the basic client, which is still what I am using, because of performance ..

      Most of the times I have to set back the font from something else to Default Sans Serif.

      So after all those years I came up with a simple smart icon, which sets the current selected text to Default Sans Serif

      @Command([TextSetFontFace];@GetMachineInfo ([EnvVariable];"NAMEDSTYLE1_FACE"))


      By the way the button is using a not so well documented @function which gets a notes.ini variable in @formula that doesn't have to start with a $.

      For me this will be a huge time saver.  

      I would wish I could set the standard for past to text only instead of having a 3 keys short cut for plain text.

      You can take this one step further and change the font for the whole body field with one click...


      @Command([EditGoToField];"Body");
      @Command([EditSelectAll]);
      @Command([TextSetFontFace];@GetMachineInfo ([EnvVariable];"NAMEDSTYLE1_FACE"));
      @Command( [EditDeselectAll] );

      -- Daniel


      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]