Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Docker Support for Domino 10.0.1 FP3

Daniel Nashed  12 September 2019 22:51:32
The first HCL Software packages are released. Domino 10.0.1 FP3 is the first download which is only available from HCL.
So we added support for HCL Flexnet downloads.

The file names for the software have different names for older fixes and Domino itself, than the files from IBM Passport Advantage and FixCentral Downloads.
So we added support for more than two file names per software file. The first file is now the HCL Flexnet download, which will also be used to generate a download hint.

Example:

10.0.1FP3           [NA]  Domino_10.0.1FP3_Linux64.tar  (-)
https://hclsoftware.flexnetoperations.com/flexnet/operationsportal/DownloadSearchPage.action?search=Domino_10.0.1FP3_Linux64.tar+&resultType=Files&sortBy=eff_date&listButton=Search

I didn't find a direct link to a software download that worked. If you select the link you get, it will not work for someone else. But at least the search will return a single file.
The changes allow to build Domino 10.0.1 FP3. But because some partners and customers still did not mange to download FP3, I did not make FP3 the default version yet.

If you want to build with FP3, download the software and run

./build.sh domino 10.0.1 FP3

This will build an image with FP3 but does not mark it as latest yet.

We also added the Domino 11 beta version to the software list and if you have access to the beta version, you can build a Domino 11 beta image.
The Docker project is prepared for the beta and you will see HCL instead of IBM branding.

To build Domino 11 Beta1 images run:

./build.sh domino 11.0.0.beta1

All those changes are currently submitted to the develop branch.


-- Daniel

Fail2Ban Support for Domino on Linux -- Intrusion Detection

Daniel Nashed  13 August 2019 17:18:23


Introduction


Domino supports Internet password lockout, which is meanwhile working for all internet protocols (it came in thru a fix I think somewhere in the 8.5.x code stream and isn't really documented).

This does already help to protect individual accounts. But it doesn't currently help for the same IP trying to hack different accounts.


There is a
AHA idea to improve it. And I think it is an important functionality for Domino. But blocking IPs with suspicious login attempts isn't always simple for an application.
On the one side someone behind a remote proxy could be blocked if there are too many people having bad password attempts at the same time.

On the other side if your server is behind a secure proxy, you don't have full control to block IPs


As long you have remote IPs hitting your server "directly", you could block them on your server.

This will work for many infrastructures and there is already a quite flexible solution for Linux.

Fail2Ban offers a wide range of "filters" for different applications which parse log files to find out which IP is not behaving correctly and blocks them in the local Linux firewall.

The idea is to have Fail2Ban read thru the Domino console log (better notes.log from my start script because it never wraps around) to find failed password attempts.

Fail2Ban is designed to track and block those IPs in the local Linux firewall.


Here is a sample line for an invalid login attempt. All other protocols use the same format.


[10780:00015-00007F4E8FFA6700] 08.08.2019 22:52:04   http: john.doe@acme.com [1.2.3.4] authentication failure using internet password


Once you have the right filter defined, it's quite easy to install and use Fail2Ban.

I wrote a filter for Domino and also have a default configuration which also includes a configuration for sshd.


The following is a installation description provides all you need to be up and running.
It also includes information about operations like status checking, unblocking users and troubleshooting.

The scripts used will be added to my start script in the "extra" directory.

It's only a solution for Linux and right now only for a local server without a proxy.

For Linux this offers also protection for other services like sshd.


Proxy Support


A friend is using NGINX in front of the Domino HTTP stack on the same machine. And he asked if I could help to get fail2ban working in combination with a proxy in front.

From Domino point of view traffic appears to come from the proxy IP address. But I found a solution which isn't what I expected but it works.

Via notes.ini HTTP_LOG_ACCESS_XFORWARDED_FOR=1 you can configure to write an additional field "ForwaredFor" into domlog.nsf.


The log entry (see above) still lists the proxy IP. There is another
AHA idea to enhance the logging.
But for now I wrote a small extension manager, which captures the domlog.nsf update and writes the original requesting IP in the same format into log. So Fail2Ban can capture the right IP address data.


Remote Proxies


This works for a locally installed proxy, but for a remote proxy you will have to pass the information to the proxy. This could be done with event monitoring configurations (run a program, start an agent, etc) based on the log information.

This would a more tricky configuration. The basic configuration is pretty simple.


Below you find all the instructions and additional information.

Enjoy and let me know what you think.


-- Daniel



Current Implementation and Feedback


This installation instruction below uses CentOS 7.6. But once you installed Fail2Ban it will also work with other distributions.
I have it also tested with CentOS 6.10 which works a bit different because init.d is used instead of systemd.


The current implementation checks for all protocols (http, smtp, ldap, imap, pop3).

It is a single filter which would count failed login attempts for all protocols together and than blocks the IP for all protocols.

This seems to be the most reasonable configuration. But depending on your needs you might want to have separate filter definitions and configurations.


The current script can be easily adopted to individual protocols.

But to keep it simple and also because I think this should be the most reasonable way in most cases.

I am looking for feedback if this is what you need. Alternatively I could have a separate filter for all protocols like "domino_http.conf".

But it is far easier to just have one definition and one rule set.



-- Installation --


First of all you have install Fail2Ban. It's included in the epel repository, which can be enabled via yum


yum install epel-release


Next you can install the package


yum install fail2ban



Disable SELinux


Before you can run the log filter, you have to disable SELinux (you could also create a profile for the service, but Domino is also not supported with SELinux enabled).

Check the status via


getenforce


The result should be "disabled". If not you can change it the following way.


vi /etc/selinux/config


Change the line


SELINUX=disabled


The next reboot disables SELINUX

You can temporary disable SELinux if you don't want to reboot now (you should reboot at least later to ensure your server will still boot!).


setenforce 0



The application leverages python and works in combination with firewalld used by default in CentOS 7.
You can enable and start the systemd services via systemd commands. A configuration change needs a restart.


systemctl enable fail2ban

systemctl start fail2ban
systemctl restart fail2ban



-- Domino Configuration --


Copy new configuration file jail.local and Domino filter configuration domino.conf (contains filters for multiple protocols)

If you have an existing configuration copy entries manually. The jail.local is a good starting point and also contains an enabled sshd configuration.
You should review the configuration and change parameters as needed. The default configuration and the service configuration contain the same values but can be customized per service.

Copy the two configuration files


cp jail.local /etc/fail2ban/jail.local

cp domino.conf /etc/fail2ban/filter.d/domino.conf


You should review the configuration. But some details might need to be adjusted.

The domino.conf file contains a "datepattern" which is very important for the pattern matching.
fail2ban parses the date first and removes it from the original string line before the regex expessions are used to match the string and get the HOST IP address.


The Script contains two definitions for the mostly used date format. The format widely used in Europe and the US settings.
You could also change the Domino log format to the one Fail2Ban understands (see notes.ini settings in the domino.conf files).
But I would recommend to change the datepattern in the domino.conf file instead.

Example:


# European Date 31.12.2019 22:11:01
datepattern = %%d.%%m.%%Y %%H:%%M:%%S


The second important parameter is in jail.local.

The logpath defines the log file to check By default the standard location used by the Domino start script is configured.
Please use the Domino Start Script log, because the file doesn't rotate like console.log!

Example:

logpath  = /local/notesdata/notes.log


Afterwards restart the service


systemctl restart fail2ban



-- Operations --


Check status for a jail


fail2ban-client status domino

Status for the jail: domino
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     8
|  `- File list:        /local/notesdata/notes.log
`- Actions
|- Currently banned: 1
|- Total banned:     1
`- Banned IP list:   192.168.100.107



List IP Tables for banned IPs


iptables -L -p


Chain f2b-domino (1 references)
target     prot opt source               destination
REJECT     all  --  192.168.100.107      0.0.0.0/0            reject-with icmp-port-unreachable
RETURN     all  --  0.0.0.0/0            0.0.0.0/0



Unban IP for a specific rule


To unban an IP before it expires use the fail2ban-client

Example:


fail2ban-client set domino unbanip 192.168.100.107



-- Troubleshooting --


Check the log file:


cat /var/log/fail2ban.log


Check Python Errors:


abrt-cli list



-- Testing Rules --


In case you want to test rules to see that for example the date format matches, you can use the following regex test tool included in Fail2Ban

Example:

fail2ban-regex /local/notesdata/notes.log /etc/fail2ban/filter.d/domino.conf


You can use the following filters:


--print-all-matched
--print-all-missed


-- Appendix Configuration Files --


Just copy the following configuration files.

The configuration is a basic configuration, which can be changed for your needs.

You find the code currently in the start script extras directory in the IBM Domino Docker script
.
I had to change the download location because pasting it into the Domino blog template made some code disappear.
It is really time to find something better than the old blog template...

https://github.com/IBM/domino-docker/tree/develop/start_script/extra/fail2ban


Domino Portable Edition - Building the smallest Domino server

Daniel Nashed  3 August 2019 18:06:51
Sadly I haven't been at the HCL Factory tour #3 in Chelmsford.
I would like to have seen the demo first hand. But I got live tickers over iMessage when Thomas Hampel took a lot of spare time to build it.

Thomas did show the smallest Domino server running on a small pocket size Linux based machine leveraging Docker CE on Ubuntu with the official
Domino on Docker script.

You should have a look at his
blog post for details.

The reason for Ubuntu is the missing hardware support on SLES/RHEL/CentOS for the CPU used in this tiny device.

But the beauty is that Docker works the same on Ubuntu and runnning our Docker image is supported.
Thomas could have build the image also on the Docker host installed on that machine.

On the other side it was faster to export and import the Docker image from a build machine and it shows how you could do it if you have Docker installed on a different device or on the cloud where you have to import the image.


That's Thomas for the time you spend on it also for the step by step documentation and the pictures!
It is really great to see what is possible. The tricky part was the Linux install. The Docker part works unmodified.

I was a big fan of the foundation server. And having something like this again but as a software solution for different standard hardware like the Intel NUC would be very cool!


Now you have a smaller box than the NUC you and me used for our DNUG Docker workshop earlier this year ;-)


Have a great weekend!


-- Daniel


    HCL Support and Software Download

    Daniel Nashed  1 August 2019 10:52:32


    After HCL took over from IBM the first weeks have been a bit challenging.
    Creating support tickets and the takeover of existing open and recently closed support tickets worked quite well in most cases.
    But still I get questions from customers how to access the HCL support website and to register.

    Software Download

    The software download is a separate portal, because HCL is using FlexNet for software download.
    You have to register a separate account with the same name.
    In the beginning I had issues registering and my entitlements haven't been there. So I wasn't able to see software.
    But we finally got it solved when we found the root causes.

    Also HCL got all the data and software at day one and they are adding the software step by step.
    They are also improving the experience for the support portal and will provide a tips & tricks document soon.
    But let me give you some trips in advance which might help you.

    We had a telco with the responsible team at HCL for the downloads. They are looking into all the feedback and take it as priority.
    But let me share some current information.

    IBM Passport Advantage still works

    For customers the IBM Passport Advantage software download should still work!
    All the software should be still there -- I checked a couple of current software downloads.

    Partner Download doesn't have all the current software any more

    For partners who got the value pack IBM removed some of the current software and there isn't a new download yet.
    But the partner team at HCL is working on it and we will get a solution this month.

    What should you do today

    Customers should register to both portals. The support portal and the download portal.
    I had some issues with the software portal URL and you should only use the one listed in this blog post!
    The password reset emails had a different URL listed, which does not work for login from external.

    To register to the support portal and to the FlexNet software download, you have to use the password reset functionality for both.
    You should have got a welcome message and HCL registered existing IBM customers with their primary email addresses listed.

    So after the password reset you should see your existing cases and be able to open new cases.
    I just have done that with a customer and also hist contact data was there. But you should review and update your profile!

    In case you cannot register to the FlexNet download site, open at ticket in the support portal for software download issues.
    They have a team looking into those kind of issues separately.

    Here are the two URLs you will need:

    HCL Support:

    https://hclpnpsupport.hcltech.com/csm

    HCL Software Download:

    https://hclsoftware.flexnetoperations.com/flexnet/operationsportal/startPage.do

    The software is still being uploaded. It takes some time because of the new system. HCL is working on the most current software first. And if you need something, that is not listed you can open a ticket.
    I would personally have a look into the new software portal and see if all your licensed software is listed.
    But for now continue to work with IBM Passport Advantage and Fixcental.

    HCL is looking into our feedback how to improve the software search and download experience.
    But with some tips it works well today.

    Searching for Downloads

    You can select different categories of software depending on your licensed products.
    But if you really looking for a specific file, you can use the search. The search also works for the product codes we had on the IBM side.

    Under Download/Search you can search for software. But you have first to switch from "Download Packages" to "Files".

    From what I figured out so far the search is a OR search and I am sometimes getting odd result sets.
    We have asked for a description and best practices for searches and how we can narrow down searches.
    Detailed instructions will be added to the main start screen as a support document. There will be tips and search instructions.

    But you should find software by description and product code already.

    Checksums for Downloads

    I was missing the checksums for the downloads and finally found them by clicking on the "+" of the download. This opens details for the download and shows a MD5 and SHA256 checksum on the right.



    There are ongoing enhancements to the support portal and they looking into our feedback.

    It might take some weeks until everything is fully available. But also as we have seen in other areas, HCL is really keen on feedback and puts it into action items.
    Yes the first days have been somehow "bumpy" but there are improvements every day.

    You can expect that I post detailed information about how to download Notes/Domino/Traveler updates as soon as they appear.

    -- Daniel

    Domino Docker on Astra Linux

    Daniel Nashed  28 July 2019 08:42:35

    This morning got up early and took another look into Astra Linux.

    First of all I did another install without GUI to see how things would work with a server that is comparable to a CentOS minimum install.
    The installation was very smooth and straight forward even on command-line.
    The only part I had to setup manually was the network configuration and I had to install the SSH server (apt install openssh-server). That was the base for my installation.

    Domino Installation

    Than I took another look into installing Domino native to see which packages needed to be installed for Domino.

    What I needed on top of the standard installation are just the following packgages: gdb, bc and policykit-1

    apt update
    apt install gdb bc policykit-1

    Afterwards it was a straight forward installation.
    I just had to create the missing /etc/sysconfig directory for the start script (mkdir /etc/sysconfig).
    The mentioned astra-ptrace-lock service (see last post) wasn't installed by default. So no changes where needed here.

    So I got the server up and running in minutes including installing the OS.

    Domino on Docker Installation

    That made me curious to see how Docker would run on this machine and what I need to do to get it up and running.
    It turns out that Docker CE is part of the distribution and can be installed out of the box.

    First of all I installed Docker in one single step:

    apt install docker.io

    Installing the package installed and started Docker 17.12.1-ce !! All settings looked good and are requirements are met (overlay2 driver is configured, ext4 is a supported file-system)!

    Than I just downloaded our IBM Domino on Docker project

    git clone https://github.com/IBM/domino-docker

    I pointed the download directory to my local server hosting Domino software by changing the configuration:

    vi build.sh
    DOWNLOAD_FROM=http://192.168.96.170/software

    And finally built the Domino CE image

    ./build.sh domino-ce

    To see if the image is working I created a new server leveraging our management script and took a look into the container.

    cd management
    ./docker_domino-ce run
    ./docker_domino-ce bash root
    domino console

    This got me an up and running Domino 10.0.1 FP2 server in 5 minutes :-)

    Conclusion

    The installation just worked without any tweaks needed. I left out optimization steps and the installation for native Domino in detail here (which works exactly like on any other Linux).
    So Domino on Astra Linux isn't a supported configuration but looks technical good also running with a "minimum installation" without a graphical interface.

    On the other side running Domino on Docker on Astra Linux with Docker CE is a fully supported configuration.
    The image is running on centos.latest inside Docker container. So technically the base OS for running Domino is CentOS.
    Like CentOS native this looks like a great free deployment option for Domino on Docker to me!

    Did anyone try this already? What are your experiences?

    -- Daniel

    Domino on Astra Linux Feedback

    Daniel Nashed  27 July 2019 23:02:15


    For the
    Russian Notes UserGroup #RNUG  in Moscow 10 and 11 October I am preparing myself to see what is special in the Russian market.
    I am really looking forward to be at the conference!


    Vlad explained to me that Astra Linux is widely used on the Russian market. It is a Debian derived distribution and is used by the Russian government!
    There is a strategic move away from Microsoft in government organisations. So Astra Linux will probably more important in future.


    According to Wikipedia (
    https://en.wikipedia.org/wiki/Astra_Linux) it is very secure and can be used to store top secret information. This alone makes it quite impressive!

    So I had a look into it this morning and installed Domino including my start script on it.


    It is not a supported distribution! But I would be interested to get your feedback.

    Who is using it? Would you like to use it? Would this help to provide better services on the Russian market?

    What are the issues/special cases you ran into?


    I can only support my start script on Astra Linux. Support for Domino would be something for HCL.

    So I am really interested to understand who is using it and getting feedback from the field.


    For CentOS support it was quite easy to get support from HCL. CentOS is source code compatible to RHEL and HCL changed their build environment to CentOS 7.4.

    For Astra Linux it would be a lot more difficult to get official support.

    So I am really interested in your feedback (either as a comment or via email).


    Below you find my first impressions an findings installing it on my own.


    -- Daniel


    Image:Domino on Astra Linux Feedback


    Download/Install


    After downloading the current AstraLinux Common Edition ISO from
    https://astralinux.ru/ it was very straight forward to install it.
    The GUI can be changed to English and supports different locates including German.


    I installed the graphical version which was kind of easier for the first tests.

    The installation is pretty straight forward and it comes with a couple of extra security questions.

    I did not enable extra security like the hardened kernel and the more strict security settings.
    But most of it was still enabled by default. So I had to re-enable ptrace to allow NSD and memcheck to access processes.

    The extra hardened kernel seemed not to make any difference. The grub boad-loader selected the kernel by default.


    Some minor differences


    When adding a new user with adduser I noticed that the -U option to create a matching group wasn't available but generating a group before with addgroup and using that group-id when creating the user via --gid worked as expected.


    The installation of Domino went well even the installer said the environment isn't supported.

    Also the configuration via remote setup just worked.


    There are no packages that you have to install to get it up and running (bc, perl have been already installed in the selections I made).

    I installed sysstat and the gdb (GNU Debugger needed by NSD) using the graphical package manager.


    Start Script


    My start script could not completely installed automatically because there wasn't a /etc/sysconfig directory.

    So I just created an empty directory, because the start script does always look for configuration data in it's config file located in this directory.


    With this minor tweak also the start script works out of the box.


    NSD Debugging requires ptrace, which is disabled by default.

    This isn't new for me. We had the same limitations on our first Docker installations. For Docker you have to allow the container to use ptrace on it's running processes.


    Even I haven't enabled it, ptrace wasn't allowed and NSD and memcheck could not attach to processes.

    It turned out that ptrace was disabled, even the graphical installation and configuration said it was disabled.


    So I used the following settings:


    Check if ptrace lock is enabled


    systemctl is-enabled astra-ptrace-lock

    enabled


    Disable ptrace lock


    astra-ptrace-lock disable


    Check again that it is disabled:


    systemctl is-enabled astra-ptrace-lock

    disabled


    You have to boot your server to have this change effective

    But afterwards NSD and memcheck run as expected.


    Performance Tuning


    Most other distributions have changed the I/O scheduler already from "cfg" at least to "deadline".

    But Astra Linux still uses "cfg" as the default. So you have to add "elevator=noop" to your grub kernel boot line.

    It was surprisingly easy to change the grub configuration. I just went into the graphical grub configuration and made the change.

    The tool automatically wrote the right grub configuration. After a reboot the disk used "noop" as the I/O scheduler.


    You can check the settings via:


    cat /sys/block/sda/queue/scheduler

    [noop]
    deadline cfq



    Summary


    With a couple of smaller changes to my standard install procedure, I was able to install Domino 10.0.1 FP2 on Astra Linux.

    Astra Linux looks like an interesting and secure distribution. So I look forward to your feedback and requirements.
    I have no idea how many customers and partners from Russia are reading my blog.
    But I would really like to hear from you and I am looking forward to see many of you In Moscow in October.

      Cross Certifing a Notes ID only works with a Safe.ID via C-API/Lotus Script/Java

      Daniel Nashed  24 July 2019 08:03:35

      For the Docker project I looked into cross certifying IDs via C-API and the Lotus Script/Notes Java classes.
      It turned out that this only works well for safe.ids. The information from the ID files which is needed for cross certification requires to open the Notes.ID.

      In Lotus Script/Java in a client you are prompted for the password. On C-API there is an error returned for a normal ID --  Wrong Password. (Passwords are case sensitive - be sure to use correct upper and lower case.).

      This behavior isn't documented and they don't mention that it only works well for safe.ids. There is no way to specify the password for the Notes.ID to be cross certified.
      I have a support ticket open and they created an enhancement request.
      They also created an AHA idea for me ->
      https://domino.ideas.aha.io/ideas/DOMINO-I-875

      In Lotus Script/Java there isn't a way to check in advance if a Notes.ID is a safe.id. In C-API you can check before calling REGCrossCertifyID () if the ID is a safe.id.
      This call has been around for a long time and I am surprised nobody ever reported this limitation.

      For the Docker project it makes sense to pass a safe.id anyway. So for us it's not really a show stopper.

      It looks like the limitation is that reading the public key from the Notes.ID already needs the password.


      In the client you can create a cross certificate from a signature. But this isn't available in any exposed API either.
      Having a way to create a cross certificate from a signature would helpful for auto registration applications ;-)
      A signed request could be used to cross certify the ID...

      -- Daniel

      Domino on Docker Project Updates

      Daniel Nashed  23 July 2019 14:33:20

      Thomas and me are working on the Domino on Docker project which has been around for a while. We are constantly updating it with more functionality.
      Beside the main functionality of providing an automated installation we have a management script that can help to build custom Domino docker images for (e.g.) including applications.
      We are working on making the resulting image more flexible. The first version allowed only to automatically setup a first server in a new Domain, but customers already have an environment and either want to setup an additional server in an existing domain or at least have a cross certified environment.


      Whats new:

      1. Additional server setup

      You can now specify an existing server.id and existing server to get the system databases from. You still need to register the second server.id manually in your Domino Directory, however the ID file does not need to be copied anymore.
      Just specify the environment variable
      ServerIDfile to point to a location (local or http/https) from where the server.id file can be downloaded and the container startup routine will take care of automatically setting up your second server.

      2. Add your own data into a container at initial startup

      The big challenge is how to bring in data into a new container automatically. Distributing server.id files, templates, or even full applications.

      We looked at different approaches which included "Docker secrets", shared volumes and other options.
      For improving flexibility we decided to use configurable http/https download links which can be used to download a server.id or an additional data-directory.zip which is automatically expanded at first server start.
      This would be for example a way for business partners to deploy their software on top of the image. Or for a customer to deploy their applications or specific adoptions.
      All you have to do is to specify an environment variable
      CustomNotesdataZip (attention, case sensitive!) pointing to a zip file that will be downloaded and extracted into the container at runtime.

      3. Scriptable configuration

      Now that you have provided your own templates - how do you turn them into an application, how do you change ACLs, or server settings at run-time?

      We have added a method to automatically configure a server based on a
      config JSON file. This can be used to create databases, change groups, change server settings etc.
      The configuration is applied before starting up the (new) Domino server for the first time and also allows to sign applications, change the ACL of databases.
      ...there is even more configuration options to come.


      4. More flexible deployment options

      In previews versions there was image specific data in the /local directory.
      So we moved that data to a separate directory to optionally allow /local to be mapped to a volume instead of having multiple volumes for /local/notesdata, /local/translog and /local/daos.
      Mounting /local to a single volume will work fine, but if you want to build a
      high performance Domino server we are recommending to have separate volumes for those different parts. We even added directories for nif and ft to allow separate volumes for those parts as well.
      The Docker volume mapping is comparable to creating mount points. It's about providing most flexibility with best practices in mind.


      5. Preparation for new binary location

      The project now now includes a new start script version 3.3.0 which is already prepared for changing the program directory default location ( /opt/ibm/domino ) with Domino 11.
      The start script and all docker image script files have been prepared to support a different binary location in future. All places in the scripts use standard variables. And we will keep the LOTUS variable to point to the binary location.


      Feedback & Future planning

      One of the next features will be to allow cross certification with existing IDs. The certifier.id is currently staying on the first installed machine. So the idea is to cross certify a provided safe.id.
      This is specially helpful to create test environments. A small servertask will take care of creating cross certifying a safe.id and adding it to the LocalDomainAdmin group.

      Another idea is to integrate this functionality into the toolchain which sets up the server, we have not decided yet.
      We are looking for your feedback so leave a comment with your suggestions for improvement or
      create an issue in our domino-docker project

      Domino on Linux/Unix Start Script - New install location /opt/nashcom/startscript

      Daniel Nashed  18 July 2019 09:43:16

       From the feedback I got in the recent survey most admins want the start script in it's own location under /opt/nashcom.
      This change is needed to allow more flexibility and easier configuration. And in addition it is a preparation for Domino 11 where the /opt/ibm/domino standard install location will change.

      So the new installation path for the start script will be /opt/nashcom/startscript.
      All new configuration will point to this location by default. But the install routine will also preserve the existing standard location /opt/ibm/domino if present.
      It replaces the old script with a symbolic link to the new script location for compatibility with earlier configuration files.

      The configuration itself will remain in the existing location. Those are the standard location on Linux and there is no reason to change it.
      But there will be additional files in the new directory over time. One is a sample configuration file for each new version of the start script for reference.

      The install shell script will use the new location and can either overwrite your existing configuration (upd option) or just install the new code.
      In case you decide to not update your existing configuration files, the old standard path will leverage the sym-link to the old start script location.

      The main reason to switch to a new location is that some files like the systemd service "domino.service" file do not allow to use variables for configuration and need a fixed location for the scripts invoked.
      So the following main files are used by the start script. Actually only the start script itself moved. But it needed changes in all those files. So let me got thru the components to explain.

      /opt/nashcom/startscript/rc_domino_script
      Main start script with all the logic. It is invoked by rc_domino or the systemd service.
      It is referenced in multiple places and when changing the location, those files need to be changed as well!


      /etc/sysconfig/rc_domino_config (required)
      Standard configuration --> this configuration is replaced when you use the "upd"

      /etc/sysconfig/rc_domino_config_notes (optional)
      Server specific configuration which overwrites standard configuration values.
      This configuration is never touched by the install routine. And you could have your specific configuration here.
      You can use those files in any way that fit your environment needs.
      I am personally using it in way that I have the standard configuration for all servers defined in the main configuration.
      And have specific configuration here --> for example the shutdown commands for a Traveler server.

      /etc/init.d/rc_domino
      Main entry point for all operations with your Domino server.
      Even systemd is used instead of init.d this location stays the same for compatibility reasons.
      But there is a symbolic link /usr/bin/domino to ensure you can use the short cut "domino" to invoke all operations.

      This script is still intended to be the main entry point for all operation.
      It will figure out if you are the right user and switch if needed before invoking rc_domino_script.
      And it contains logic for starting and stopping a server via systemd (systemctrl command).

      You can run this script either for normal operations with the "notes" user or for start/stop operations via root user (or with sudo).

      /etc/systemd/system/domino.service

      Domino systemd service configuration --> this configuration is also replaced when you use the "upd"

      It contains some basic configuration values (security limits, locale, ..) and references rc_domino_script for start/stop operations.
      Here a full and fixed path is needed for rc_domino_script which was the main reason for a new start script location.


      Updating the start script and your configuration

      When you update your start script to the new version only the scripts are overwritten (rc_domino_script, rc_domino).
      Only when specifying "upd" the configuration will be replaced by a current default configuration.
      This will work in case you had standard locations for the Domino binary and Domino data directory.
      Else you have to review and change the locations of the binary directory in your configuration afterwards.


      New planned logic for locating the binary directory

      The binary directory is still configured via LOTUS variable.

      The plan is to change the logic for detecting the binary directory. Today you have to specify it manually.
      And it would use /opt/ibm/domino if not defined. The current default configuration points to this location.

      To keep installation easier and be prepared for a new standard location, the idea is to not have LOTUS configured by default but search in the new and old standard location for binary files.
      So the plan is to search for /opt/hcl/domino/bin/server first before checking for /opt/ibm/domino/bin/server. The first binary directory found is used.

      You can still specify your binary location and different binary locations would only need one configuration variable (LOTUS) to be changed.
      Before you had to modify rc_domino, rc_domino_config and domino.service because the start script was inside the Domino binary directory.

      How would you like this change? Does this make sense for a default logic?

      Updating Your start script

      If you are using a standard configuration the install script can handle the start script update for you.

      a.) Either decide to replace your current start script configuration via "upd" option, review the new standard configuration and make your changes (rc_domino_config, domino.service).
      b.) Or keep your legacy configuration and stay with the sym link to the old start script location. It will continue to work but you still have the old reference and no new default configuration settings.
      You could run either way, but I would recommend replacing and reviewing your configuration. I am thinking about a back of the configuration files when updating.

      Current Status

      I just finished the new version and I am currently testing it on normal Domino environments and also the IBM Domino Docker Script.

      The changes are already submitted to the develop Branch of the github repository for preview --> https://github.com/IBM/domino-docker/tree/develop
      And you could get a copy of the start script from there in the latest version I am working on --> https://github.com/IBM/domino-docker/raw/develop/dockerfiles/domino/install_dir/start_script.tar

      I am looking for feedback if those changes work for you. Either testing or just giving feedback to what I just wrote here.
      Either comment or feel free to send me an email for feedback.

      Finding the time a message was send from the return receipt

      Daniel Nashed  17 July 2019 07:16:16

      We had an interesting and sort of funny support call this week.

      An user got a reply from a service database to confirm his request sent in by mail.

      The original message sent was a return receipt. But the sender said that there wasn't any received message, with that subject which could have generated a return receipt.


      We checked all the logs for an outgoing previous message, without any luck.

      Than I had the idea to check the $REF field of the return receipt to find out the time the original message was send.


      The $REF is the UNID of the original document (response doc). The last part of the UNID is the Note part which is actually a TIMEDATE.


      F
      409DB34F:1C3EEDFB-N
      C1257ACC:004FDD93

      The internal representation of a TIMEDATE looks like this:


      typedef struct tagTIMEDATE {
       DWORD Innards[2];
      } TIMEDATE;


      You can take the internal binary representation of a TIMEDATE and convert it back to a human readable format.

      So I wrote a small C-API routine (appended below) to convert the hex representation of the TIMEDATE and print the timedate.


      Example:


      C1257ACC:004FDD93 = [06.12.2012 15:32:20]


      In our case it turned out that the original message was from December 2012 :-).

      And the return receipt was generated when the user was cleaning up the mail-file opening a message which had never opened before generating a return receipt.


      By the way you can do the same for a replica ID which is actually the TIMEDATE the first instance of a database was created. So you can figure out the creation time of a database.
      This can be both helpful in some situations for troubleshooting.


      -- Daniel


      ConvertUnid2Timedate (char *unid_text)

      {

       TIMEDATE timedate;

       char str[255];

       char *p;

       
       strcpy (str, unid_text);

       
       p = strstr (str, ":");


       if (p == NULL) p = strstr (str, ".");

       
       if (p == NULL)

       {

         AddInLogMessageText("nshsem: invalid timedate!", 0, "");

         return 0;

       }

       
       *p = '\0';

       p++;
       
       timedate.Innards[0] = hex2int(p);

       timedate.Innards[1] = hex2int(str);

       
       AddInLogMessageText("nshsem: %s = [%z]", 0, unid_text, &timedate);


       return 0;

      }

      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]