Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Domino on Linux/Unix Start Script - New install location /opt/nashcom/startscript

Daniel Nashed  18 July 2019 09:43:16

 From the feedback I got in the recent survey most admins want the start script in it's own location under /opt/nashcom.
This change is needed to allow more flexibility and easier configuration. And in addition it is a preparation for Domino 11 where the /opt/ibm/domino standard install location will change.

So the new installation path for the start script will be /opt/nashcom/startscript.
All new configuration will point to this location by default. But the install routine will also preserve the existing standard location /opt/ibm/domino if present.
It replaces the old script with a symbolic link to the new script location for compatibility with earlier configuration files.

The configuration itself will remain in the existing location. Those are the standard location on Linux and there is no reason to change it.
But there will be additional files in the new directory over time. One is a sample configuration file for each new version of the start script for reference.

The install shell script will use the new location and can either overwrite your existing configuration (upd option) or just install the new code.
In case you decide to not update your existing configuration files, the old standard path will leverage the sym-link to the old start script location.

The main reason to switch to a new location is that some files like the systemd service "domino.service" file do not allow to use variables for configuration and need a fixed location for the scripts invoked.
So the following main files are used by the start script. Actually only the start script itself moved. But it needed changes in all those files. So let me got thru the components to explain.

/opt/nashcom/startscript/rc_domino_script
Main start script with all the logic. It is invoked by rc_domino or the systemd service.
It is referenced in multiple places and when changing the location, those files need to be changed as well!


/etc/sysconfig/rc_domino_config (required)
Standard configuration --> this configuration is replaced when you use the "upd"

/etc/sysconfig/rc_domino_config_notes (optional)
Server specific configuration which overwrites standard configuration values.
This configuration is never touched by the install routine. And you could have your specific configuration here.
You can use those files in any way that fit your environment needs.
I am personally using it in way that I have the standard configuration for all servers defined in the main configuration.
And have specific configuration here --> for example the shutdown commands for a Traveler server.

/etc/init.d/rc_domino
Main entry point for all operations with your Domino server.
Even systemd is used instead of init.d this location stays the same for compatibility reasons.
But there is a symbolic link /usr/bin/domino to ensure you can use the short cut "domino" to invoke all operations.

This script is still intended to be the main entry point for all operation.
It will figure out if you are the right user and switch if needed before invoking rc_domino_script.
And it contains logic for starting and stopping a server via systemd (systemctrl command).

You can run this script either for normal operations with the "notes" user or for start/stop operations via root user (or with sudo).

/etc/systemd/system/domino.service

Domino systemd service configuration --> this configuration is also replaced when you use the "upd"

It contains some basic configuration values (security limits, locale, ..) and references rc_domino_script for start/stop operations.
Here a full and fixed path is needed for rc_domino_script which was the main reason for a new start script location.


Updating the start script and your configuration

When you update your start script to the new version only the scripts are overwritten (rc_domino_script, rc_domino).
Only when specifying "upd" the configuration will be replaced by a current default configuration.
This will work in case you had standard locations for the Domino binary and Domino data directory.
Else you have to review and change the locations of the binary directory in your configuration afterwards.


New planned logic for locating the binary directory

The binary directory is still configured via LOTUS variable.

The plan is to change the logic for detecting the binary directory. Today you have to specify it manually.
And it would use /opt/ibm/domino if not defined. The current default configuration points to this location.

To keep installation easier and be prepared for a new standard location, the idea is to not have LOTUS configured by default but search in the new and old standard location for binary files.
So the plan is to search for /opt/hcl/domino/bin/server first before checking for /opt/ibm/domino/bin/server. The first binary directory found is used.

You can still specify your binary location and different binary locations would only need one configuration variable (LOTUS) to be changed.
Before you had to modify rc_domino, rc_domino_config and domino.service because the start script was inside the Domino binary directory.

How would you like this change? Does this make sense for a default logic?

Updating Your start script

If you are using a standard configuration the install script can handle the start script update for you.

a.) Either decide to replace your current start script configuration via "upd" option, review the new standard configuration and make your changes (rc_domino_config, domino.service).
b.) Or keep your legacy configuration and stay with the sym link to the old start script location. It will continue to work but you still have the old reference and no new default configuration settings.
You could run either way, but I would recommend replacing and reviewing your configuration. I am thinking about a back of the configuration files when updating.

Current Status

I just finished the new version and I am currently testing it on normal Domino environments and also the IBM Domino Docker Script.

The changes are already submitted to the develop Branch of the github repository for preview --> https://github.com/IBM/domino-docker/tree/develop
And you could get a copy of the start script from there in the latest version I am working on --> https://github.com/IBM/domino-docker/raw/develop/dockerfiles/domino/install_dir/start_script.tar

I am looking for feedback if those changes work for you. Either testing or just giving feedback to what I just wrote here.
Either comment or feel free to send me an email for feedback.

Finding the time a message was send from the return receipt

Daniel Nashed  17 July 2019 07:16:16

We had an interesting and sort of funny support call this week.

An user got a reply from a service database to confirm his request sent in by mail.

The original message sent was a return receipt. But the sender said that there wasn't any received message, with that subject which could have generated a return receipt.


We checked all the logs for an outgoing previous message, without any luck.

Than I had the idea to check the $REF field of the return receipt to find out the time the original message was send.


The $REF is the UNID of the original document (response doc). The last part of the UNID is the Note part which is actually a TIMEDATE.


F
409DB34F:1C3EEDFB-N
C1257ACC:004FDD93

The internal representation of a TIMEDATE looks like this:


typedef struct tagTIMEDATE {
 DWORD Innards[2];
} TIMEDATE;


You can take the internal binary representation of a TIMEDATE and convert it back to a human readable format.

So I wrote a small C-API routine (appended below) to convert the hex representation of the TIMEDATE and print the timedate.


Example:


C1257ACC:004FDD93 = [06.12.2012 15:32:20]


In our case it turned out that the original message was from December 2012 :-).

And the return receipt was generated when the user was cleaning up the mail-file opening a message which had never opened before generating a return receipt.


By the way you can do the same for a replica ID which is actually the TIMEDATE the first instance of a database was created. So you can figure out the creation time of a database.
This can be both helpful in some situations for troubleshooting.


-- Daniel


ConvertUnid2Timedate (char *unid_text)

{

 TIMEDATE timedate;

 char str[255];

 char *p;

 
 strcpy (str, unid_text);

 
 p = strstr (str, ":");


 if (p == NULL) p = strstr (str, ".");

 
 if (p == NULL)

 {

   AddInLogMessageText("nshsem: invalid timedate!", 0, "");

   return 0;

 }

 
 *p = '\0';

 p++;
 
 timedate.Innards[0] = hex2int(p);

 timedate.Innards[1] = hex2int(str);

 
 AddInLogMessageText("nshsem: %s = [%z]", 0, unid_text, &timedate);


 return 0;

}

    Domino Backup which type of integration would you expect?

    Daniel Nashed  9 July 2019 19:00:53
    There have been a couple of other projects including Domino on Docker which took some time.
    So it took some time until I was able to look into the Backup project again.
    I have build a configuration, log and restore job database. It can be all in one database or in separate databases.

    The integration with standard backup software for single database or snapshot backup is implemented via command-files.

    So for each backup or restore of a database a OS level command is called.
    I wrote some shell scripts / batch files which are invoked with a set of parameters.
    Those scripts can be changed to fit for any backup solution which allows command-line integration.

    The idea is to keep it very flexible. Here is an example for backup of a notes database including logging into a central log-file to show which parameters are specified.

    Die Shell Script/Batch files are invoked and the output of the command is parsed from the backup application.
    The command called will get all possibly important information for taking a backup.
    This includes references to the backup type/name and backup mode.
    And also the start timedate of the backup for reference.

    The sample implementation below on Windows just uses xcopy (which isn't simple because xcopy isn't as flexible as I expected).
    On Linux the command line options are more flexible.

    The idea is to allow integration into backup applications (also allow snapshot backups) or offer integration point for your own type of backups.
    For example on Linux you could leverage rsync to a different location.

    What do you think of this approach? Is this what you need?


    -- Backup --

    @echo off
    SET LOGFILE=c:\nshback\log\nshbackup.log
    echo ---BACKUP DB--- >> %LOGFILE%
    echo FullFileName    : %1 >> %LOGFILE%
    echo ShortFileName   : %2 >> %LOGFILE%
    echo BackupDbRef     : %3 >> %LOGFILE%
    echo BackupNode      : %4 >> %LOGFILE%
    echo BackupName      : %5 >> %LOGFILE%
    echo BackupMode      : %6 >> %LOGFILE%
    echo BackupStartDT   : %7 >> %LOGFILE%

    xcopy /Y "%~1" "c:\nshback\backup\db\%7\%~2*" >> %LOGFILE%
    echo: >> %LOGFILE%
    echo NshBackupResult: c:\nshback\backup\db\%7\%~2
    echo NshBackupRef: TestReference
    echo Return: PROCESSED   (%~1)


    -- Restore --

    @echo off
    SET LOGFILE=c:\nshback\log\nshrestore.log
    echo ---RESTORE--- >> %LOGFILE%
    echo FullFileName    : %1 >> %LOGFILE%
    echo ShortFileName   : %2 >> %LOGFILE%
    echo BackupDbRef     : %3 >> %LOGFILE%
    echo BackupNode      : %4 >> %LOGFILE%
    echo BackupName      : %5 >> %LOGFILE%
    echo BackupMode      : %6 >> %LOGFILE%
    echo BackupStartDT   : %7 >> %LOGFILE%
    echo RestoreFileName : %8 >> %LOGFILE%
    echo DeltaFileSize   : %9 >> %LOGFILE%

    echo xcopy /Y "c:\nshback\backup\db\%7\%~2" "%~8.DAD*" >> %LOGFILE%
    xcopy /Y "c:\nshback\backup\db\%7\%~2" "%~8.DAD*" >> %LOGFILE%

    if "%9" == "0" goto skip_delta

    xcopy /Y "c:\nshback\backup\db\$7\%~2.DELTA" "%~8.DELTA*" >> %LOGFILE%

    :skip_delta

    echo: >> %LOGFILE%
    echo NshBackupResult: %~8
    echo Return: PROCESSED (%~1)


    HCL Support & Software Download

    Daniel Nashed  2 July 2019 14:28:56

    On Monday I was a bit surprised to get an e-mail from IBM about support cases are getting closed because they are getting transferred to HCL.
    Before 1.7 HCL did not have access to the data, so it sounds like they started to migrate yesterday.
    HCL worked on it yesterday to get all the support cases imported and the user accounts created.

    Yesterday afternoon I got links to my existing support cases.
    Today I got an e-mail how to access my account by just using the password reset function on the new support site --> https://hclpnpsupport.hcltech.com.

    I was surprised that not only the open cases but also my last closed cases have been transferred and I was able to open new tickets!

    What will take another 1-2 days is the software download via FlexNet as far the support line told me.
    But you can still download software on the IBM side right now.

    I have been told that not all accounts might have been migrated yet.
    So you might need to have to give it 1-2 days.

    If you did not receive an e-mail and/or you have an urgent case you can contact the support center directly (see details on the support homepage --> https://hclpnpsupport.hcltech.com).

    -- Daniel

    Domino Deletion Log Update for US timezones

    Daniel Nashed  27 June 2019 06:41:45
    There has been an issue in my Deletion Log Tool for US locales. I was surprised this did not come up earlier, because it is a quite general issue.
    The problem was that for US timezones the month and the date have different orders (MDY instead of DMY).

    The @TextToTime used to convert the date stored in the deletion log file was converted incorrectly.

    I had to change the code to use the @Date function instead, specifying the components of the date separately, which wasn't obvious (see previous blog post for details).

    The new version now correctly converts those dates also for US locales. I have updated the version to 0.9.3 (you need to request it again).

    Interestingly I am not getting much feedback for the tool. This was the first issue reported since I shipped the first version when Domino 10 came out.
    I only had around 50 requests for the deletion log database so far.  Are you using deletion log at all? Are you annotating it on your own?

    If you are not using deletion logging, I would be interested to understand why.
    I think this was one of the big missing features to figure out what was going on with deletions but I hardly get any questions from customers about the feature.

    -- Daniel

      Date Conversion Question Lotus Script / Fromulas @TextToTime Internationally

      Daniel Nashed  25 June 2019 06:29:49
      I ran into an issue with my delete log application when converting timedates from the internal format  like [20190612T154744,66-05] to a date when the server is in the US.

      What I have assumed is that if I specify a dot instead of the slash the @TextToTime function would work with day.month.year.
      But it appears it is still working month.day.year instead.

      Does anyone have an idea how to do this conversion operation safely internationally?
      Is there any save string format which will always corrected in the same way?

      In C-API it would be simple. You can fill in the corresponding values in the TIME structure and convert.
      But I did not find anything like that in @Formulas or LotusScript.

      Before I start spending time on it, probably someone has solved this already ...

      -- Daniel


      Sametime 10 Limited use for Windows 32bit has been released

      Daniel Nashed  15 June 2019 00:41:01
      Samtime 10 Limited has been released. It's the first new version that we see for a long time. And there is already planning for Sametime 11 and 12.

      But the next step will be to finish the Sametime 10 64bit Support which will run on Domino 10 for Windows and Linux.
      For now you have to install Sametime 10 Limited use on Domino 9.0.1 32bit.

      See all details in the following blog post which answers many of the open questions:

      https://www.ibm.com/blogs/collaboration-solutions/2019/06/14/what-is-sametime-limited-use/


      -- Daniel

      New Start Script version 3.2.2 with a Tika Stop server work-around

      Daniel Nashed  14 June 2019 06:39:09

      There are a couple of new features to the start script and fixed one issue with restartcompact for systemd.
      And there is also an addition for the Docker environment. I have a separate blog post about the current Tika issues -->
      http://blog.nashcom.de/nashcomblog.nsf/dx/domino-10-attachment-indexing-with-apache-tika.htm

      There is shutdown control I added into the start script, because the external Java process does currently not terminate.

      By default without any config parameter the shutdown check will be 30 seconds.
      The default in the config in the configuration file is 20 seconds. So if you don't deploy the new config settings, the script will wait 10 seconds longer before killing the Tika process.


      Beside that hard coded logic for the Tika process, I added another delayed shutdown command that is invoked during the shutdown after some time.

      And there is also a new tika process stop/kill command, which I added.


      When you kill the tika server process, it will be restarted when the next process tries to index an attachment.


      The new version is available by mail request (-->
      http://www.nashcom.de/nshweb/pages/startscript.htm).

      I am still planning for the next major release. It might not have many new features but will have changed default locations (probably /opt/nashcom), maybe will move to github (not sure because than I am loosing a bit of control but make the feedback process and download easier).

      And I might move to a new documentation format. So probably the documentation will be online only in MD format. Still experimenting in converting MD files back to txt.

      Also I sort of like the old text file better than the new MD format for the purpose of the start script documentation. Maybe I will just bring it up in two formats and get feedback.


      -- Daniel


      New Features


      New Commands:


      "systemlog" shows the last log lines from systemd service.

      This is helpful to see output of the start script.



      "tika" stop|kill


      Shows the Tika server status and can be used to terminate the process.


      Without additional parameters this command shows the status of the Tiker server process.

      tika stop --> stops the process.

      tik kill  --> kills the process.


      Added "locale" output to server start logging.

      This can help to troubleshoot issues with locale setting on your server.



      New configuration options:



      DOMINO_TIKA_SHUTDOWN_TERM_SECONDS


      Tries to shutdown the Tika index server during shutdown.

      IT happens that the Tika server does not terminate, which prevents the Domino server from shutting down properly.

      Default 30 seconds.


      DOMINO_SHUTDOWN_DELAYED_SCRIPT


      Script which can be executed delayed during shutdown.

      DOMINO_SHUTDOWN_DELAYED_SECONDS
      specifies the number of seconds after shutdown start.


      DOMINO_SHUTDOWN_DELAYED_SECONDS
      (default 20 seconds)


      Shutdown Delay for delayed shutdown command.

      Default is 20 seconds if script is defined.


      Docker Support:


      Now the entry point script checks if the script is already started with the right user and will not switch to the user.

      There are configuration options on the Docker side to run the entry point script directly with the right user.

      When you switch the user to "notes" at the end of your dockerfile, the container is started with "notes".

      This provides better security.


      Problems Solved


      restartcompact and restartfixup did not work correctly with systemd.

      For systemd the rc_domino script needs to stop the service run compact/fixup and restart the service.

      Domino 10 Attachment Indexing with Apache Tika

      Daniel Nashed  14 June 2019 06:38:17

      Tika is the new engine used to extract text from attachments for fulltext indexing in Notes/Domino 10 (client and server).
      It replaces the previous key view package which had a different integration (it was leveraging the kvoop binary).

      The Tika server is a stand alone Java based open source server component (
      http://tika.apache.org/) which is accessed by the server processes via curl requests (you can see it when looking into a NSD ;-)

      Tika is just a single jar file which is started in a separate JVM instance. It is loosely coupled with Domino.

      So it is living in it's own "stand box" with communication over TCP/IP. It's not based on any Domino code and doesn't share memory nor other resources like MQs etc.


      Because it is quite new in Domino there are some open issues. Some of them have been already addressed in FP2 but there are some fixes that had to wait for FP3.

      The good news is that those limitations can be worked around.


      I hope this helps you in your deployments. The parameters are just needed on server side but they would also work on the Notes client if really needed.

      It's the same back-end code which is executed on the client and the indexing works in the same way on the client.


      -- Daniel


      Tika Shutdown Issues on Linux


      One of the issues we ran into is a server shutdown hang. On Linux currently the Tika process does not terminate, which will cause my start script to wait for this process to terminate.

       
      I have added new logic to my start script , which is automatically killing the process at shutdown (default without any parameter after 30 seconds, default in the config file 20 seconds).

      This ensures that your server shutdown will continue to work in time and this logic stays active also for newer versions just in case.

      So we give the Tika process time to terminate before stopping it on OS level.


      In addition the start script has an option to stop/kill the process during run-time. In normal cases killing a Domino process isn't a good idea.

      But because Tika is loosely coupled, you can kill it without having any Domino server impact. Also the child died signal is not causing the server to panic in this case.


      See version 3.2.2 update in a separate post for details of the new start script version -->
      http://blog.nashcom.de/nashcomblog.nsf/dx/new-start-script-version-3.2.2-with-a-tika-stop-server-work-around.htm

      Here are the details from HCL support for this issue. This fix is planned for FP3 and Domino 11.


      SPR - JPAIB6ZLKG reports Java Tika Process not terminating when Domino Shutdowns cleaning (NON-WINDOWS Platforms Only)



      Tuning Tika


      Beside that there is a performance issue one customer with very large attachments (mostly PDF) did run into.

      By default the memory for the Tika process might be too small. And the server is not honoring the JVM size parameter.


      But there is a general JVM Options Override parameter, which can be used to pass options to the Tika server.

      There is more control planned without this generic parameter for the next fixpack. But for now this option can be used to increase the memory.


      In addition there is a newer Tika server version 1.20 which might give you better results from performance side as well.

      Domino is using version 1.18 but because it is a single jar and the server is loosely coupled, you could replace the jar file with the current version from the Tika project website.


      It's not fully supported by IBM/HCL today but works in our test. They are looking into updating the Tika server version for Domino 11 and maybe also for Domino 10 FP3. But that isn't committed.


      In addition to this change, you might want to increase the Tika server timeout to give the server more time to respond for larger attachments and higher load on the server.


      Here are the parameters that could help you. Note this examples use the newer Tika version. The existing version is the same name without the additional version number.


      Windows:

      TIKA_JVM_OPTIONS_OVERRIDE= -Xmx1536m -jar "C:\Program Files\IBM\Domino\tika-server-1.20.jar"


      Linux:

      TIKA_JVM_OPTIONS_OVERRIDE= -Xmx1536m -jar "/opt/ibm/domino/notes/latest/linux/tika-server-1.20.jar"


      DEBUG_TIKA_TIMEOUT=60000


      More strict Server Certificate handling in iOS 13 macOS 10.15

      Daniel Nashed  5 June 2019 08:12:23

      Beginning with iOS13 and macOS 10.15 the more stricter handling could have quite high impact and lead to untrusted certificates specially for internal certificates.

      An official CA should have taken care of this already and certificates should be correctly issued.

      But this can impact internal certificate authorities (CAs). Existing certificates are not impacted by the stricter key usage checking and the existing expiration will not change.

      For new certificates you have to have a careful check of the following Apple statement which I commented in blue.


      I know certificate stuff can be kind of boring and complicated. But you have to look into those changes!!


      -- Daniel


      https://support.apple.com/en-us/HT210176

      Let me comment the new announcement
      in green.

      -- snip --


      All TLS server certificates must comply with these new security requirements in iOS 13 and macOS 10.15:

      TLS server certificates and issuing CAs using RSA keys must use key sizes greater than or equal to 2048 bits. Certificates using RSA key sizes smaller than 2048 bits are no longer trusted for TLS.


      This isn't new and nobody should use have a key size below 2048.


      TLS server certificates and issuing CAs must use a hash algorithm from the SHA-2 family in the signature algorithm. SHA-1 signed certificates are no longer trusted for TLS.


      Certificates and intermediate CA certs already needed to be SHA-2 before.
      The root CA's certificate is verified in a different way and should still not be effected. Even the statement isn't that clear about it.

      But if that would not be OK any more, many CAs would be in trouble.


      TLS server certificates must present the DNS name of the server in the Subject Alternative Name extension of the certificate. DNS names in the CommonName of a certificate are longer trusted.


      The SAN name (Subject Alternate Name - DNS name)  is something that is already required by browsers etc. And I don't see this as a big surprise.


      Additionally, all TLS server certificates issued after July 1, 2019 (as indicated in the NotBefore field of the certificate) must follow these guidelines:


      TLS server certificates must contain an ExtendedKeyUsage (EKU) extension containing the id-kp-serverAuth OID.


      This is a big deal for you own CAs. EKU isn't used by all private CAs. And for openssl it's not simple to configure the extended key usage.

      The OID and name is the following: TLS Web Server Authentication (1.3.6.1.5.5.7.3.1).

      To check your certificate you can look into what your browser tells you about it. Or you use openssl to dump your certificate.


      Here is a command-line example and the result should look like this:


      openssl x509 -text -noout -in domino.crt

      ...

       X509v3 extensions:

                X509v3 Extended Key Usage:

                    TLS Web Server Authentication

                X509v3 Subject Alternative Name:

                    DNS:domino.nashcom.loc

      ...


      The name and the OID are the following:

      TLS Web Server Authentication (1.3.6.1.5.5.7.3.1)

      To create a certificate properly your openssl command-line gets more complicated or you need a configuration file.

      But in combination with a SAN you want to write the file dynamically...


      Here is an example command-line from one of my scripts:


      -- snip --

      openssl x509 -passin pass:$CA_PASSWORD -req -days $CLIENT_VALID_DAYS -in $CSR_FILE -CA $CA_CRT_FILE -CAkey $CA_KEY_FILE \

              -out $CRT_FILE -CAcreateserial -CAserial $CA_DIR/ca.seq  -extfile <(printf "extendedKeyUsage = serverAuth \n subjectAltName=DNS:$SANS") > /dev/null


      -- snip --


      TLS server certificates must have a validity period of 825 days or fewer (as expressed in the NotBefore and NotAfter fields of the certificate).


      Wow this is a big deal! That's a bit more than two years and two month! So this will impact customers getting 3 year certificates and also long vaild internal certificates.

      In some cases customers generate internal certificates for much longer periods which would not work any more when you issue them after July, 1st, 2019!


      So you might want to create those certificates before 1. July 2019 ;-)


      Connections to TLS servers violating these new requirements will fail and may cause network failures, apps to fail, and websites to not load in Safari in iOS 13 and macOS 10.15.


      -- snip --


      So the two biggest surprises are the reduced certificate expiration days! Two years and two month is not really much!

      And I am curious how the certificate vendors react on it.


      -- Daniel

      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]