Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Notes/Domino 9.0.1 Feature Pack 9 shipped

Daniel Nashed  20 August 2017 21:39:16
Notes and Domino 9.0.1 Feature Pack 9 is available.

The client side and server-side introduces fixes and also new features.

The official "flixlist" can be found here --> http://www.lotus.com/ldd/fixlist.nsf/0/12d957b7c277fc728525816300434c53

Here are the highlights and some important comments.


JVM Update in Notes Client & Domino Server

The security fixed version introduced with a JVM patch for FP8 is included in FP9:

Notes/Domino - Java 1.8 SR4 FP5

But this is still just the runtime in the client. The compile time support for Java 1.8 has to wait for FP10 because it also needs an update Eclipse version.

The Designer compile is Java 1.6 SR16 FP45


Notes Client Updates

Some of the changes need an updated mail-template. Some changes need notes.ini parameters or other settings.
But some other new functionality is enabled by default.


High resolution support for the Notes client

The Windows client now scales correctly text and icons with high resolution displays (higher the HD) and also with custom DPI settings.
This was a long waited feature request. FP8 already had some improvement but the main change is shipped with FP9.


Full fidelity for fonts in Notes emails

There is a change in the fonts used for the MIME body for messages to render fonts better in other clients.

In my tests it turned out that this causes interoperability issues with Notes clients.
When using sans-serif fonts in outgoing messages the messages are displayed with a serif font depending on your Notes client configuration.

It happens when you are using "Disable Embedded Browser for MIME mail" via preference (notes.ini BrowserRenderDisable=1).

Also in any case when the message is put into edit mode.

IBM is currently looking into this issue and I will post an update what I find out.


Improved handling of non-English characters in internet messages

Notes and Domino now support RFC2231, a standard internet protocol for handling non-English characters in internet messages
 to improve message fidelity in communications with other applications that support this standard.


Improved name lookup in Notes

Searching by through typeahead or in the ambiguous name dialog, returns the same results as searching by .
For example, searching for don smith or smith don returns the same results, including variants such as Donald, Donovan, Smithfield.

This is also a long waited feature. But it needs the updated pernames.ntf template provided with 9.0.1 FP9 and add a notes.ini setting AllowWildcardLookup=1.



Domino Server Updates

Enhancement Request To Be Able To Increase The Amgr Queue Beyond 100 (SPR #RSTNA4SL7C APARID: LO87242)

The Agent Manager's Eligible queue is now able to change from the lowest value possible at 100, to 255 which is the highest value possible via an INI AMGRMaxQueue.

I worked with one of those customers who has the requirement to run many agents in a very tight schedule. In there environment even on a fast server not all agents have been scheduled.
This parameter allows to increase the queue. But sadly the parameter is just a BYTE and cannot be increased about 255.
I would have wished they would have increased the limit to a higher value (which would have been a bigger change) and I would have wished that the parameter would be 255 by default.
But it should already help for most environments.


Databases and views can be opened more quickly in databases that are enabled for transaction logging

It takes less time to open databases and views that are at ODS 52 or higher and enabled for transaction logging.
Previously, performance for opening databases or views could be slow in frequently updated databases.
This improvement is due to the implementation of less contention with update operations.

So it is really important that your servers have translog enabled! We have still seen customers not running without translog.
And it is important that you update your ODS to 52 via create_r9_databases=1 and compact -C.

I have not seen any performance values. But this should improve also performance for NIFNSF enabled databases.

Enabling and managing inline view indexing

A view index is an internal filing system that NotesĀ® uses to build the list of documents to display in a database view or folder.
By default, view indexes are updated on a server at scheduled intervals.
To update a view index immediately after documents change instead, administrators can enable inline view indexing.
When you enable inline view indexing, a critical view is always kept up-to-date for your users.

I have not looked into it in detail yet. But it is a bigger change and the release information does not contain all the details.

Here is the documentation update with all the details -->https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/admn_inline_index_enabling.html
The documentation contains information how to implement it and also how to query information, statistics and other details.

I hope those additional comments on top of the release notes are helpful for your first look into FP9.


-- Daniel


Blog Certificate updated and Let’s Encrypt Update

Daniel Nashed  8 August 2017 11:30:13
My certificate expired after 90 days because I did not track it. And the Let's Encrypt original client configuration did not work any more when I was looking into renewal today.
The client was Python based and there is a newer client --> https://certbot.eff.org/ which is officially recommended by Let's Encrypt.

It's still complicated to use and you need to have Python installed.
But since I first implemented it there are many other ACME clients that properly integrate with Let's Encrypt -> https://letsencrypt.org/docs/client-options/.
There are even two simple shell script based clients which both do not require root permission and work in combination with Domino.

I have installed the "getssl" script (https://github.com/srvrco/getssl) and it was quite easy to implement, even for a server with multiple certificates (SAN cert).

And I also updated my shell script to automatically generate a Domino keyring file now with the getssl script.
But it still needs a manual restart of all servertasks that use the certificate. So it is not a completely automated process yet.

The gettssl script works with the Domino html root and port 80.  
With some additional checks I could potentially automate certificate updates on my server completely.
For now there is a manual step required.

Is anyone using Let's Encrypt Certificates with Domino? Which ACME client are you using?

Let's Encrypt Certificates are a good alternative if certificate updates would be automatically installed.
Right now it's a simple shell script. I could polish it and make it available if there is demand for it.

What do you think? Any feedback is welcome!

-- Daniel



SLES 12 SP2 Issues with Domino running with Systemd

Daniel Nashed  24 July 2017 12:01:20
There is a new feature introduced in SLES 12 SP2 which could lead to issues with larger Domino or Traveler servers.

The default nproc size is still set to 7400. So in most cases this tunable does still not need to be set in your Domino service file.


But there is a new security feature introduced in SLES 12 SP2 which will cause processes fail to start or not able to span more threads.


The error you might see is the following:


Jul 20 11:02:41 dom-srv kernel: cgroup: fork rejected by pids controller in /system.slice/domino.service

The new feature limits processes by default to use more than 512 threads.


Here is the relevant extract from SLES 12 SP2 readme:


-- snip --


2.3.2 Support for PIDs cgroup Controller #

The version of systemd shipped in SLES 12 SP2 uses the PIDs cgroup controller. This provides some per-service fork() bomb protection, leading to a safer system.
However, under certain circumstances you may notice regressions. The limits have already been raised above the upstream default values to avoid this but the risk remains.
If you notice regressions, you can change a number of TasksMax settings.

To control the default TasksMax= setting for services and scopes running on the system, use the system.conf setting DefaultTasksMax=. This setting defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum.

For thread- or process-heavy services, you may need to set a higher TasksMax value. In such cases, set TasksMax directly in the specific unit files. Either choose a numeric value or even infinity.
Similarly, you can limit the total number of processes or tasks each user can own concurrently. To do so, use the logind.conf setting UserTasksMax (the default is 12288).
nspawn containers now also have a TasksMax value set, with a default of 16384.


-- snip --

The best solution for Domino is to increase the limit directly in the domino.service file.

In addition to this new setting I also updated the config file with an updated value for LimitNPROC= 8000 in addition to the new setting TasksMax= 8000.

8000 Threads should be sufficient for all Domino server environments.


So in case you are running a larger scale environment with SLES 12 SP2 you really should check those settings in  your service file!


-- Daniel



-- snip --


[Unit]

Description=IBM Domino Server (notes)

After=syslog.target network.target


[Service]

Type=forking

User=notes

LimitNOFILE=60000

LimitNPROC= 8000

TasksMax= 8000

PIDFile=/local/notesdata/domino.pid

ExecStart=/opt/ibm/domino/rc_domino_script start

ExecStop=/opt/ibm/domino/rc_domino_script stop

TimeoutSec=100

TimeoutStopSec=300

KillMode=none

RemainAfterExit=no

#Environment=LANG=en_US.UTF-8

#Environment=LANG=de_DE.UTF-8


[Install]

WantedBy=multi-user.target

-- snip --

    Microsoft fixes Notes Client Windows 10 Creators Crash

    Daniel Nashed  28 June 2017 22:16:28
    Today I got feedback from IBM that the fix that Microsoft releases does solve the blue screen issue with Notes and the customized home page issue.

    There have been multiple situations in which the client crashed or caused a blue screen because of some Windows UI calls in Notes after the Windows creators update.

    I am interested to get feedback if the fix does solve all your Notes Client on Windows creators update.

    Here is a link for the update:

    https://support.microsoft.com/en-in/help/4022716/windows-10-update-kb4022716

    -- Daniel

      Traveler 9.0.1.18 with new Security Mode for Mail-File Access

      Daniel Nashed  22 June 2017 10:07:40
      Traveler 9.0.1.18 comes with a couple of minor fixes and a big change in the way Traveler Server access mail-databases.
      In 9.0.1.15 IBM introduced a new check if the Traveler server is listed in Trusted Servers (Server Security Tab) to show a warning if not.

      Now we know what IBM was preparing for. The server now acts as the user instead of the server. That's only possible if listed in Trusted Servers.

      You still need the Traveler server to be listed in the ACL of the mail databases. Trusted Servers means that to server itself can make the session on a database look like it would be the user session.
      But the remote server still needs access to the database.

      I have done a quick test. Without the proper ACL an error is logged and also the user status reports an error.

      The IBM Traveler server encountered an internal error validating your User ID CN=John Doe/O=Acme/CÞ.  Please contact your server administrator.
      [CN=notes.acme.de/OU=Srv/O=Acme-Net, mail/johndoe.nsf] is not reachable, status(0x4ac) "Unexpected internal error".

      The new method for accessing mailfiles solves a couple of limitations. See details from the documentation below.


      -- Daniel

      What's new?


      Traveler Server Run as User


      Starting with IBM Traveler 9.0.1.18, the run as user feature will now be enabled by default. When running as the user, the Traveler server will access the user's mail file as the user ID instead of the server ID. This feature resolves several long standing issues with accessing the user's mail file as the server ID, including:

      • Honor ACL controls on mail file and corporate lookup for the user.
      • Prevent event notices and automated responses from being sent from the server ID.
      • Prevent the server ID from being assigned as the owner of the mail profile when there is no owner defined.

      Note:
      For run as user feature to function properly, the Traveler server must be listed as a trusted server in the user's Mail Server document. To disable run as user, set this notes.ini parameter: NTS_USER_SESSION=false



      APAR # Abstract
      LO90096 Info update continues to be ghosted on mobile device after the event is processed.
      LO91797 Empty comments displayed on iOS native Calendar application when event processed in iNotes.
      LO91836 Invalid this and future reschedule generated by iOS native Calendar application.
      LO91875 Ghosted event not displayed on mobile device.
      LO91956 Maill attachment does not sync to mobile device when contains angle brackets < and >.
      LO91997 IBM Traveler web administrator may show iOS Verse 9.4 device as not supporting security capabilities.
      LO92010 Better handling of special character in mail header fields.
      LO92080 Ignore a reply message with out a valid action defined.
      LO92085 Hard delete processed notices vs soft delete to prevent from filling up trash folder.
      LO92209 Second meeting room may be lost if event updated from mobile device.
      LO92210 Unable to turn off iOS Verse application password via Domino policy document setting.
      LO92257 Two instances of a previously processed event may show on mobile device if the daylight savings rules change for the time zone.
      LO92303 SQL Syntax error adding index TSGUDTSTAMPCREATEIDXSQL9 on DB2.




      Notes Client/Windows Crash with Windows 10 Creators update

      Daniel Nashed  3 June 2017 16:36:03
      Just got that question today at DNUG. There is an issue with the Notes Client with the current Windows 10 Update - aka Creators Update (Build 1703).

      According to the responsible person who is at DNUG today, this happens because of changed Windows graphics APIs.
      IBM is working on a fix which will be available in FP9.

      FP9 will also have full High Resolution support! We saw a demo with FP9 which really looked great!

      Here are the two relevant SPRs:


      SPR LHEYALMCEP : Domino Designer crashes the OS after Windows 10 Creators update [For Designer BSOD issue]
      SPR AYAVALMCJK : Windows 10 Creators update and OS crashes while using Notes/Designer

      IBM said that you should remove the following registry setting to avoid the blue screen after the Notes Client start.
      (Updated: By mistake I wrote notes.ini parameter but correct is registry setting which might not exist).

      -- snip --

      Delete this registry entry and the crash should go away

      PageHeapFlags, VerifierFlags from
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notes.exe  

      -- snip --

      Ram also said that the problem is not impacting all configurations and the other bug is not happening very often.

      So I hope that if you updated your Windows 10 environment already, your are not running into this issue or the temporary work-around helps you.

      Another Update I got in comments  of my posts is that this happened often with customized welcome pages.

      Note: I updated the blog post and got a replication/save conflict. So I deleted and added the post again because the blog template does not like replication/save conflicts.
      Some comments might be lost but I added them to the post anyway. Thanks for your feedback!

      --. Daniel

      Security Bulletin: IBM Domino TLS server Diffie-Hellman key validation vulnerability (CVE-2016-6087)

      Daniel Nashed  1 June 2017 07:27:46
      There is a vulnerability in the TLS stack which could lead an exploit which could lead a less secure connection.
      The good news is that the fix is already included in FP8. So you should upgrade to 9.0.1 FP8 if you have a public facing Domino Server with HTTPS.

      See the details and reference below.

      -- Daniel

      A vulnerability in the IBM Domino TLS server's Diffie-Hellman parameter validation could potentially be exploited in a small subgroup attack which could result in a less secure connection.
      An attacker may be able to exploit this vulnerability to obtain user authentication credentials.

      Vulnerability Details

      CVEID: CVE-2016-6087 / DESCRIPTION: IBM Domino could allow an attacker to steal credentials using multiple sessions and large amounts of data using Domino TLS Key Exchange validation.

      CVE-2016-6087 is tracked as SPR# DKEN9WGMYE.


      http://www.ibm.com/support/docview.wss?uid=swg22002808

      Important Security Fix for IMAP

      Daniel Nashed  22 April 2017 11:13:16
      In case you are running IMAP on a server that is reachable over the internet you should look into this fix ASAP.

      It might not be that critical for internal services.

      See details about this vulnerability here --> http://www.ibm.com/support/docview.wss?uid=swg22002280

      All versions of Domino are affected!


      NIFNSF Supported Maximum Size above 64 GB! -> 1 TB is officially supported!

      Daniel Nashed  21 April 2017 19:02:31
      After getting that question offline and having a discussion on my blog, I checked with IBM if they plan support NIFNSF sizes above 64 GB.
      Since it is kind of a database container and needs a database handle someone could think that the maximum limit is also 64 GB.

      That would give us at least 64 GB room for the NIF index -- which would be already a big improvement.


      But from what I recall from some comments at Connect some years ago the maximum limit was not around 64 GB when they designed it.


      On the other side it is difficult to test and such large view / folder index sizes. And you will not run into many situations where you need such a large size.


      From what I heard, IBM is about to publish a supported official size for the NIFNSF indexs that is far beyond 64 GB.
      Stay tuned for the official statement. For now I can tell you that it will work above the 64 GB limit!


      On the other side a databases with that index size will reach other limits like application responsiveness issues because of the nature of complex views with many documents.


      But it is good to know that it was designed to support larger sizes and also the counters in the database will continue to work as I have tested earlier for DAOS.


      Once we get an official statement I will update my post and share the link.

      25.04.2017 Update:

      The technical documentation has been updated -->
      https://www.ibm.com/support/knowledgecenter/SSKTMJ_9.0.1/admin/admn_moving_views_out_of_databases.html

      Here is the official statement for NIFNSF .NDX File size :-)

      ".NDX files have a limit of 1 TB. The real determination of how large the views can grow is based on application responsiveness or if any other limits are reached before the 1 TB .NDX file limit is reached."


      -- Daniel

      Disclaimer Attachment Issue not yet fixed in IF1

      Daniel Nashed  14 April 2017 20:28:40
      As Rob Kirkland commented in one of my last blog posted, the fix in IF1 does not solve the iusse.

      We both checked with IBM and got the reply that the SPR just changes back the default and disables the change introduced in FP8 for Google calender integration.

      IBM is working on a fix hopefully makes it into FP9.

      So for now you should keep the notes.ini Parameter MIMEDisclaimersNoEncode=0 disabled.

      Thanks to Rob to bring this up!

      -- Daniel

      TPONAKFJLP

      After upgrade to FP8, with disclaimers enabled, .pdf attachments have content-transfer-encoding of binary  




      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]