Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

SLES 15 is now supported for Domino 11 & Sametime 11

Daniel Nashed  12 February 2020 17:49:41
The system requirement technotes have been updated...

Now SLES 15 is listed as supported for Domino and Sametime.


SLES 15 is different from the previous versions. I tried to look into it with the GA version and looked into it again with SP1.

You should only run it with SP1. And if you can do the online configuration and not just off-line from DVD.. They changed the installer .. What can I say ...


I have tested Domino 11 already with SLES 15 SP1 and it just works.
And I spent a lot of time getting Sametime installed on CentOS over the weekend and I spent the whole evening installing it on SLES 12 SP5.

I would not expect much more difficulties with SLES 15. But SLES 12 was already a challenge ..


But the good news I figured out why my start script wasn't working with SLES. The rc init.d functions from SuSE always broke the ST status website.
Now that we don't have init.d I made a fix for the start script to not use the rc init.d code from SuSE in combination with systemd. That finally fixed the issues I had with Sametime in combination with my start script on SLES. Still testing.. If someone needs the changed version let me know ...

Update 15.2.2020:

On SLES 12 I was able to install Mongo 3.6. And we had some missing OpenSSL *.so version dependencies which have not been resolved. Setting symbolic links helped.
But on SLES 15 SP1 -- which is the current version you would run, when using SLES 15 -- Mongo DB is supported starting at version 4.2.1. This version isn't supported by ST 11 yet. And even a standard Mongo DB 4.2.x installation from the original repro failed with the same OpenSSL version dependencies.

So for now you should not try to install ST 11 on SLES 15 until this is resolved!

-- Daniel



HCL Domino 11.0 Detailed System Requirements  


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074573

HCL Sametime 11 System requirements


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074454

Converting Timedate Strings in @Forumula Language with different date formats

Daniel Nashed  11 February 2020 12:19:32

Maybe I am thinking to complicated.. But I did not find an easier solution ..
I had an interesting issue yesterday when adding multi timezone and date-time format to my backup solution, because I cannot store everything in Notes native TIMEDATES.
There are some text logs involved, when storing information about a backup. To ensure I have an unified format, I decided to change the internal storage to UTC (in Notes terms: GMT).

--------------------

Update 12.02.2020: Yes I was thinking to complicated, because I missed the function @Date which takes the date components as numeric values in a fixed order.


@Date( time-date )
@Date( year ; month ; day )
@Date( year ; month ; day ; hour ; minute ; second )

After feedback from Simon (huge thanks) and also with some discussions with my friend Rudi Knegt, who is also an old Lotus Formula fan like me, I now have the following solution to convert my date. I actually need the date as text and as time for comparison. That's why I did the conversion of the full date this way at the end. So my use-case need the text for selection and the date for comparison.

X:="2020.02.11 01:02:03 GMT"; Z:=" ";C:=@Explode(X;Z); D:=@ToNumber(@Explode(C[1];".")); @if(@Elements(C)<3;"invalid timedate";@Elements(D)<3;"invalid date";@ToTime(@Text(@Date(D[1];D[2];D[3]))+Z+C[2]+Z+C[3]));


By the way. Without knowing the other parameter options @Date function we had another idea taking benefit of the wrong text to date conversion. If you convert a date with the day>12 the result can be used to replace the result with the date format I would have needed.
That solution would not have needed the Lotus Script code in my first solution below. But with the @Date function it's more straightforward.


My final solution in my actual code looks a bit different and just needs one explode. But for demonstration purposes in this context it makes sense.

Thanks! This is true community spirit!

-- Daniel

--------------------


With my German locale I ran into limitations when converting the text back into a date.
It turned out that even when I use a string that is YMD formatted, the day/month order is still from my German locale and causes conversion issues for the first 12 days in a month.
This wasn't what I expected because there are those 3 different settings internally: DMY, MDY and YMD.

Example with German settings:

@ToTime ("2020.
02.10 11:22:33 GMT") --> 02.10.2020 13:22:33
@ToTime ("2020.
10.02 11:22:33 GMT") --> 10.02.2020 12:22:33
@ToTime ("2020.
02.14 11:22:33 GMT") --> 14.02.2020 12:22:33


In my case my internal format is always "yyyy.mm.dd hh:mm:ss GMT" and I have to convert to a correct TIMEDATE.
I had to use Lotus Script in the DB init event to get the international settings and store them in an environment variable to be used in my form.

With that client specific setting, I am converting the date to the current settings. From there on I can use @ToTime to convert it correctly.

D:=@Word(x; " ";1);T:=@Word(x; " ";2);Z:=@Word(x; " ";3);@ReplaceSubstring(DateFormat; "Y":"M":"D":"T":"Z";@Word(D;".";1):@Word(D;".";2):@Word(D;".";3):T:Z);


If someone finds a more straight forward way to convert this given format with different international settings, I would love to make it easier.
Specially reading the international settings via Lotus Script and passing it to my formula is ugly ..

-- Daniel


Example:

With --> x:="2020.02.10 11:22:33 GMT"; DateFormat:= "D.M.Y T Z";

The RESULT is --> 10.02.2020 11:22:33 GMT

Form there on I can use @ToTime to convert it correctly independent from the date settings of the client...

DateFormat:=@Environment( "DominoBackupDateFormat");

Sub Initialize
       Dim session As New NotesSession
       Dim international As NotesInternational
       Dim DateFormat As String
       
       Set international = session.International
       
       If international.IsDateDMY Then
               DateFormat = "D.M.Y T Z"
       Elseif international.IsDateMDY Then
               DateFormat = "M.D.Y T Z"
       Elseif international.IsDateYMD Then
               DateFormat = "Y.M.D T Z"
       Else
               DateFormat = "D.M.Y T Z"
       End If
       
       Call session.SetEnvironmentVar( "DominoBackupDateFormat", DateFormat )
       
End Sub


Domino SMTP error limit before terminating connections

Daniel Nashed  10 February 2020 09:11:15

My friend Harvey and I noticed a lot of brute force delivery attempts on servers to figure out about email addresses.
By default Domino doesn't limit the number of errors until a connection is closed.
But there is a notes.ini entry and setting in config document to define the number of errors.

https://help.hcltechsw.com/domino/11.0.0/conf_definingthemaximumerrorlimitbeforeaconnectionterm_t.html

For larger servers you have to be careful, because that could also prevent newsletters from being delivered, if there are too many users which don't exist.
On my own server I set the value now to 4 ;-)

And if you have more errors -- this includes invalid recipients -- the connection is closed with a temporary error.
But that also means that the mail is not received and the other server would try again.
So you have to be a bit careful with this parameter. I would sent it do a higher value like 20 on larger servers.

421 domino.nashcom.de SMTP service not available, closing transmission channel


Example from log:
10.02.2020 06:21:53   SMTP Server: 185.143.223.xxx connected
10.02.2020 06:21:54   SMTP Server: Mail for morris@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for mom@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for buy@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for abcdefg@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for az@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for schmidt@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for babbar@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for edith@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.
10.02.2020 06:21:54   SMTP Server: Mail for juliet@csi-domino.com rejected for policy reasons.  Recipient could not be found in the Domino Directory.



Image:Domino SMTP error limit before terminating connections



Notes/Domino 10.0.1 Fix Pack 4 Released

Daniel Nashed  9 February 2020 15:55:48

Notes & Domino 10.0.1 FP 4 has been released before the weekend.

Now that Notes/Domino 11 is released, my focus is more in the Notes/Domino 11 release.

But for customers running the 10.0.1 code stream, this is an important update!


I have downloaded the Linux version on and updated our Docker image to support FP4 (currently in the develop tree) last week.

But the default for Domino on Docker is 11.0 of course.


Find the official URLs for FP4 including the system requirements.


By the way the official URL for the support website is
support.hcltechsw.com -- There have been multiple names around, but I got the confirmation from support, that this is the one we should use and this is what they will use for all the links.
Some other links will not work without authentication.


-- Daniel


Notes/Domino 10.0.1 Fix Pack 4 Release Notice


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0075554


Notes 10.0.1 Fix Pack 4 System Requirements


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074997


Domino 10.0.1 Fix Pack 4 System Requirements


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0074995

/proc/sys error messages Domino on Linux

Daniel Nashed  6 February 2020 08:05:15

The Domino start script now leverages "server -c" commands instead of writing into the server's redirected input file since the latest version as blogged earlier.
If you use the server binary for that purpose, some side effects could occur, which are not new. But you might not have noticed them.

When the server process starts, it tried to set kernel parameters to optimize certain network parameters.
Domino itself uses a none privileged user (default: notes).  To be able to set the kernel parameters a helper binary called "tunekrnl", which has the sticky bit set, is used.
The sticky bit runs the process as root and allows those restricted operations.

But even with root permissions in some Linux server environment those changes are not allowed.

For example in paravirtualized environments, where you don't have your own "root" server like when using virtual servers from HostEurope, your server doesn't allow those operations.
Another case would be a Docker environment where you are also not allowed to change those kernel parameters.

In those cases you see error messages similar to this:

Error messages on console

Error - can't open /proc/sys/fs/file-max.
        errno: 13
        Permission denied
Error - can't open /proc/sys/net/ipv4/tcp_fin_timeout.
        errno: 13
        Permission denied
Error - can't open /proc/sys/net/ipv4/tcp_max_syn_backlog.
        errno: 13
        Permission denied
Error - can't open /proc/sys/net/ipv4/tcp_tw_reuse.
        errno: 13
        Permission denied
Error - can't open /proc/sys/net/ipv4/ip_local_port_range.
        errno: 13
        Permission denied

As long this was just on server start, I ignored those messages. But in a remote console for every command you want to get rid of them.

In our Docker project we are already removing the tunekrnl during install. This avoids those error messages.

If you are on virtualization platform, which provides you are full virtualization stack -- like ESX, you should not receive an error.
In that case you should check the permissions of the file and not remove it!


It should look like this (the file needs to be owned by root and the sticky bit should be set -- the s in the 4. position)

ll tunekrnl
-r-sr-xr-x. 1 root daemon 71768 Nov 25 08:33 tunekrnl


If the settings are wrong, here is the way to change them. The following is a more paranoid setting. The file is owned by root and the group notes and you are allowing user's from the group notes to execute the file with root permissions.

chown root:notes /opt/hcl/domino/notes/latest/linux/tunekrnl
chmod 4550 /opt/hcl/domino/notes/latest/linux/tunekrnl


By the way : There is one other file in Domino which needs root permissions to run. bindsock  is used to allow processes to listed on restricted ports below 1024. So bindsock is a helper binary to allow those operations.

-- Daniel



Traveler 11 HTTP/2 Push changes in detail - Review before updating

Daniel Nashed  5 February 2020 09:21:07

Detlev Pöttgen and I ran into this with the first deployments and we put together the details behind it (so you will find the same blog post on his blog and my blog).
There will be a documentation update from HCL side. Here is what we found out in detail with the feedback we got from the Traveler team.



Beginning with Traveler 11 the new push API is used -- > https://developer.apple.com/news/?id=11042019a
This is the new recommended push service from Apple which every service should use.
The older API will be available until November 2020!

There is no change needed to change to the new API and Traveler uses the new push API by default.


But your infrastructure also needs to be ready for this change!

If you have to disable the new API to go back to the "legacy API", because you can't change your infrastructure right now, there is just one notes.ini parameter that you need to set:


NTS_PUSH_APNS_HTTP2=false


But you should only use this as a very temporary solution and switch to the new push API as soon as you can!

The statement in release documentation is quite short and doesn't go into the details of what this might mean for your environment. Let me explain the changes in detail:


New Protocol HTTP/2


If you are behind a proxy, you have to check if your proxy supports the HTTP/2 protocol!  You might run into connectivity issues depending on the proxy.


Port change from 2197 to 443


The new port used is the standard HTTPS port 443 instead of  the APNS "legacy" port 2197.

You have to check your firewall if the port is open! Usually network admins are more happy with the standard port 443 but it might not be open in your environment by default!


The new HTTP/2 Push services is also available on port 2197 on the new servers to allow a more smooth migration.

There are NTS parameters you could use to change the default port 443 to the old legacy port if you really need to.


There are specific settings for each different push service and they look like this:


Example for the Verse app: NTS_PUSH_APNS_APPLE_VERSE_IBM_PRODUCTION_SERVER_PORT


If you don't set the parameter explicitly the NTS_PUSH_APNS_HTTP2 will take care of changing the port to 443 for all push Apple services.

So this is more a work-around which you should only use for example if it takes time to change your firewall.



Change from gateway.push.apple.com to api.push.apple.com


Also the target servers have changed. Before Apple used gateway.push.apple.com and switched to api.push.apple.com
Usually there isn't any change needed in your infrastructure. I checked which servers are currently behind the DNS entries and they are coming from the same netblock at Apple (see references below).



Conclusion


You really have to check your environment to see if you are prepared for new APNS HTTP/2 API.

Not having the right prerequisites isn't a reason to not update to Traveler 11. You could use the legacy API for a couple of weeks or some of the settings above might help you to get it working for your environment.

If your environment uses APNS Push, you have to migrate your environment to Traveler 11
before November 2020!


Daniel



References:



Developer Information for APNS


https://developer.apple.com/documentation/usernotifications/setting_up_a_remote_notification_server/sending_notification_requests_to_apns/

Current DNS Settings
:


gateway.push-apple.com.akadns.net

Aliases: gateway.push.apple.com

17.188.142.26
17.188.140.27
17.188.128.157
17.188.132.21
17.188.130.28
17.188.133.27
17.188.136.22
17.188.138.21


api.push-apple.com.akadns.net

Aliases: api.push.apple.com

17.188.161.182
17.188.161.11
17.188.161.203
17.188.160.13
17.188.164.15
17.188.162.16
17.188.163.207
17.188.161.13



Whois Extract for Apple Net-Block


NetRange:       17.0.0.0 - 17.255.255.255
CIDR:           17.0.0.0/8
NetName:        APPLE-WWNET
Organization:   Apple Inc. (APPLEC-1-Z)




    Quick change to default font with smart icon

    Daniel Nashed  2 February 2020 15:50:16

    The Notes standard Client has a short list of last used fonts. It's not exactly working like the last languages in the propery box of the document.
    And it isn't available in the basic client, which is still what I am using, because of performance ..

    Most of the times I have to set back the font from something else to Default Sans Serif.

    So after all those years I came up with a simple smart icon, which sets the current selected text to Default Sans Serif

    @Command([TextSetFontFace];@GetMachineInfo ([EnvVariable];"NAMEDSTYLE1_FACE"))


    By the way the button is using a not so well documented @function which gets a notes.ini variable in @formula that doesn't have to start with a $.

    For me this will be a huge time saver.  

    I would wish I could set the standard for past to text only instead of having a 3 keys short cut for plain text.

    You can take this one step further and change the font for the whole body field with one click...


    @Command([EditGoToField];"Body");
    @Command([EditSelectAll]);
    @Command([TextSetFontFace];@GetMachineInfo ([EnvVariable];"NAMEDSTYLE1_FACE"));
    @Command( [EditDeselectAll] );

    -- Daniel


    Notes 10.0.1 G2 Languages available for download

    Daniel Nashed  28 January 2020 21:59:33
    As posted earlier, HCL is delivering the other Group languages for 10.0.1 first, before shipping Notes 11.0 languages.
    The Group 2 languages are available on Flexnet download.

    I can see the downloads for the different kits for:

    - Arabic
    - Czech
    - Dutch
    - Polish
    - Swedish
    - Russian

    If you are looking for the Group 1 languages, they all have been shipped as MUI packs 15.8.2019.

    HCL Notes v10.0.1 Multilingual User Interface Group 1
    HCL Notes Client 10.0.1 Basic Configuration Multilingual User Interface Windows Group 1
    HCL Notes v10.0.1 Multilingual User Interface Group 1

    Notes 11 is available in English, German and also Japanese. The other languages are now following step after step as HCL as posted here --> https://www.cwpcollaboration.com/blogs/important-languages-support-update-domino-notes-vop-and-volt


    -- Daniel

      Restoring deleted folders in a Notes database

      Daniel Nashed  25 January 2020 11:46:46

      Feature Request: Trash for Folders


      In many customer environments, restoring deleted folders is number one reason for restoring databases.

      For documents you already have soft-deletions and the trash (default 48 hours which can be increased).

      And you can restore the document to the original folder.


      There is an AHA idea to have a trash for folder deletes, which is really popular (  
      https://domino-ideas.hcltechsw.com/ideas/NTS-I-31 ).
      Actually I could come up with an extension manager which hooks into the folder delete and rename it to DeletedFolders\MyFolderName.
      But this should be there out of the box and fully integrated into Notes and into the mail-file!



      Restoring deleted folders


      Last week I had to recover folders for a customer and there is actually a straight forward way, if you know how replication works in detail.


      Idea


      Thanks to Friedhelm Klein from Timetoact, who came up with a great idea in one of his Admin Camp presentations!

      When you restore a database you just have to update the folder design notes in the restored database to make the sequence time/date newer than the deletion stub in the original database.

      An update beats a deletion and you don't have to remove any deletion stubs in the original database on your own. The replication will also take care for putting back all still existing documents into the folder!


      Out of the box without any extra tools you just need the following steps:


      1. Restore the database with replication disabled

      2. Run convert -u twice on the database -> this will update the folder design of all custom folders from the inbox and modify the folder design element

      3. Enable replication and replicate (push) the changes from the restore database to the original database. You might want to use the -NOPIRC flag depending on when the folder was deleted

      4. Disable replication or delete the restore database


      Those steps are quite straightforward.

      But if you could also add this to your restore routine. I added it to my "nshrestore" helper application, which is also the restore part of my backup solution. So with the restore, you will be able to request automatically folder restores.
      I am still testing and have to integrate it to the UI. But it works like charm!


      C-API based solution


      load nshrestore.exe mailrestore.nsf -k mailtest.nsf

      ..
       Restoring deleted folder [Test1] Deleted: 25.01.2020 13:03:32

        Restoring deleted folder [Test1\Test A] Deleted: 25.01.2020 13:03:32
        Restoring deleted folder [Test1\Test A\Test AA] Deleted: 25.01.2020 13:03:32
        Restored 3 folder(s) from [mailrestore.nsf] to [mailtest.nsf]

      How does it work?
      • I am searching for deleted NOTE_CLASS_VIEW
      • If the Note returns a deletion stub, I am looking for the original document by UNID on the restored database.
      • In case this is really a folder (DESIGN_FLAG_FOLDER_VIEW, etc), I am updating the folder note twice
      • Afterwards I push replicate the restore database, with the local server

      This restores the original folders :-)


      Conclusion


      So even with no add-on tools you can do this today. And there are options for further automation.
      It's a bit tricky to get information from a deletion stub. But beside that, it was quite straightforward.


      And I would really wish we don't have to do those kind of restores less often, once the trash for folders will make it into Domino!


      -- Daniel


        SELinux Support for Domino

        Daniel Nashed  22 January 2020 14:47:54


        There is a AHA idea to have Domino support SELinux --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-1121
        My impression was that SELinux is already supported with current Domino releases.
        I asked HCL and it turned out that SELinux is not tested, thus it is currently not supported.
        It would be extra test effort for every distibution and version to run with SELinux.

        Security-Enhanced Linux (SELinux) is a security architecture for Linux® which is integrated in the kernel and allows a separate security layer.
        It has been originally developed by the NSA and is today integrated in the kernel.

        You also have distinct between different SELinux modes. I was very sure the strict mode would not be supported.

        But I thought the default mode "enforce" mode with "target" policy would be supported -- but it is currently not.
        Below is a short introduction directly from RedHat. And if you are interested in details there is a video of a great presentation linked below.

        When I talk to Domino admins they either don't know about SELinux but are told to disable it.
        But there are companies who really have to enable SELinux.
        In fact I have customers who run it today in enforce/target mode without knowing -- because it's default.

        I would be very interested to hear your feedback. Do you want to use it? Do you have to use it? Are you using it?

        You can either comment here,on the AHA idea or both. And if you find SELinux important to have supported, you can vote on the AHA idea.
        But on top of the vote please leave a comment which requirements you have in detail?
        Is enforced with targeted policy OK? Do you need a profile for Domino (that would be a lot of work and has impact on deployment, troubleshooting etc).

        To check if SELinux is enabled and in which mode, you can use the following command:

        sestatus
        SELinux status:                 enabled
        SELinuxfs mount:                /sys/fs/selinux
        SELinux root directory:         /etc/selinux
        Loaded policy name:             targeted
        Current mode:                   enforcing
        Mode from config file:          enforcing
        Policy MLS status:              enabled
        Policy deny_unknown status:     allowed
        Memory protection checking:     actual (secure)
        Max kernel policy version:      31

        -- Daniel


        References

        Video
        https://www.youtube.com/watch?v=_WOKRaM-HI4

        Public Documentation
        https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/using_selinux/index


        Archives


        • [IBM Lotus Domino]
        • [Domino on Linux]
        • [Nash!Com]
        • [Daniel Nashed]