Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Weekend Project "Domino on Docker Update"

Daniel Nashed  13 April 2019 19:09:53
There are a couple of updates pending for the offical IBM Docker project --> https://github.com/IBM/domino-docker
  • Support for 10.0.1 FP1
  • Support for Domino Community Edition
  • Preparation for supporting add-on products like "Traveler", "VOP", "AppDevPack" and later maybe "Sametime"
  • Making software downloads more reliable and provide better error checking
  • Install Log checking for installed software
  • Official implementation of the Domino on Dock management script.

The main challenge is proper version support without having multiple dockerfiles.
So I am working on a way to have the version tags and the installation files outside the dockerfile.

Lables and variables can be passed to the docker build command and overrwrite default settings.

Every FP will be an own full image installed on top of centos/latest.
Add-on products or customized versions are always installed as an own layer .


We discussed this week about having multiple layers like "10.0.1 -> 10.0.1 FP1 --> 10.0.1 FP1 IF1 but that would make image management more difficult for admins.

The build script could take care of building a "10.0.1" image before building a 10.0.1 FP1 image that is based on the other image.

But there isn't much benefit, beside a bit space reduction.

The add-on products and customization will have a separate layer and will use the current Domino image. For example Domino --> Traveler.


Does this make sense for you?  I first also thought that having the different versions build on each other would make sense. But we don't see the benefits. What do you think?


There will be a new file "software.txt" containing version numbers, download-file names and hashes.


Filenames of the downloads are the biggest challenge. The community edition, for example, has complete different filenames...

This map and download file will allow specifying versions without adding the download filename in the dockerfile or in the build file.


The community edition will be installed as a different product "DOMINO_CE" instead of "DOMINO" because also the FPs have different names and hashes ..
And of course, the directories inside the extracted software tar have a different directory structure (e.g. linux64/DominoEval/.. ).
The guys building it, have no idea how we are using it.


But I think I figured out a good way to organize versioning :-)


The new version will be also prepared for upcoming Interims Fixed and Hotfixes.

And I will propably also add JVM patches in the next step.

There is also a management and customization script to configure, build, run, mange, update Domino Docker containers.


-- Daniel







Domino 10.0.1 FP1 SAML, iNotes, Traveler, ID-Vault working together

Daniel Nashed  9 April 2019 13:08:48
There is an issue in Domino 9.0.1 FP10 and earlier which Milan Matejic posted about and there is also an official technote related to it.

https://milanmatejic.wordpress.com/2019/04/05/saml-ibm-notes-traveler-encrypted-e-mails-issue/

When I read the post I was sort of confused because there are also changes in Domino 10.0.1 FP1 which sound very similar. I had a pending post for FP1 for more detailed information for the notes.ini parameters added.

I checked offline with Milan and we figured out that the new notes.ini settings introduced in FP1 address the same issue he got an hotfix for in 9.0.1 FP10.


With the hotfix Milan got the same notes.ini parameter DISABLE_SAML_FLAG=1 which needs to be set on the server.

The same functionality is available in 10.0.1 FP1 along with another new functionality.

The SPR descriptions for the notes.ini are a bit short. So I asked for a more detailed description (see below).


This addeses issues with iNotes and Traveler. And could also fix issues when another application is using the same C-API calls.

In this case the user wasn't able to decrypt a mail. Also the second parameter could be useful! So if you have SAML enabled for Notes Clients not for HTTP only specially for Traveler you should have both parameters in place!

I did not run into this before, because most of my customers only use SAML for HTTP authentication and not for Notes-Client authentication.

Here is the official technote


User ID files may fail to synchronize with ID Vault for users who are enabled for Web Federated Login or Notes Federated Login

https://www.ibm.com/support/docview.wss?uid=swg21990021


And below you find descriptions from the SPR fixlist along with a more detailed description.

The notes.ini entry can also be helpful if you are leveraging the underlaying C-API calls in your own application and needs to be set on the machine where the code is executed!


-- Daniel



SPR# RGAU9VLHT3
- On the domino server set the following notes.ini (DISABLE_SAML_FLAG=1) to allow for an ID vault sync with a SAML user via the SecIdfget function.

------------------------------------------


DISABLE_SAML_FLAG=1
- There has been a limitation in the public C-API call SecidfGet() that sends the server the client's capabilities.

In this case it was sending to the server that the client could hande SAML as authentication for ID Vault ID download.

The SecidfGet() API does not support SAML for download. It only works with password. But the server will pick SAML over password if the user is enabled for SAML in their effective policy.

iNotes and Traveler are using API when attempting to download the ID file from the Vault, if the ID file is not found in the mail file or the password does not work against the ID file in the mail file.

If the user was configured for SAML in the policy then SECidfGet() would fail and the user would not get the ID file pulled from the Vault and iNotes could not do secure mail operations.

A Notes applications that called this API would have the same issue and would need to set this Notes.ini on the local machine, in order to get it to work.




SPR# LIBAB59NUY
- The ability to enable the upload of a notes ID to the mail file via iNotes can now be enabled on the server using the notes.ini of ENABLE_IDUPLOAD_FOR_SAML=1.

------------------------------------------


ENABLE_IDUPLOAD_FOR_SAML=1
- In this case the customer was attempting to import their ID file into their mail file.
However with SAML enabled for the user the ID file was only being loaded into memory as SAML loads the id file into memory.
So the user would have to import the ID file for each session. With the Notes.ini set, the import will attach the ID file to the Mail file and also push it to the ID Vault


Traveler server not connecting to Microsoft SQL Server using only TLS 1.2

Daniel Nashed  9 April 2019 09:32:46


There is a new issue that Detlev Pöttgen has already reported in detail about on his blog including the work-around
--> https://www.netzgoetter.net/internet/blogs/netzgoetter.nsf/dx/traveler-ha-running-on-ms-sql-server-issue-with-tls-1.2.htm

Current MS SQL Servers do only support TLS 1.2 and the JDBC Driver used on the Traveler Server does not open a TLS 1.2 session by default.

This impacts Traveler configuration with the travelerutil and also operating your server -- if the SQL server is updated to a version that supports TLS 1.2 only!

(Original IBM technote for reference --> https://www.ibm.com/support/docview.wss?uid=ibm10871764 ).

Detlev's post contains the details for the work-around and change the default to allow TLS 1.2.

The error message shown is:


com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption


-- Some more technical details --

I was very surprised when looking into it. The underlying issue is a IBM JVM issue in combination with the way the application uses the TLS connection.
I recall from previous tests (when we got TLS 1.0 and 1.2 for Domino 9.0.1 Fixpacks) that the IBM JVM was using TLS 1.2 already.
So it isn't a general issue but an issue with the wrong default settings in combination with the way the application is written.

In this case what is probably happening is that the current Microsoft SQL JDBC driver is using the "TLS" constant which by default only uses TLS 1.0 on the IBM JVM!!

The Oracle JVM by default uses TLS 1.0/1.1/1.2 when specifying "TLS" but the IBM JVM just uses TLS 1.0 in contrast.

https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/matchsslcontext_tls.html

So this isn't directly a Traveler issue nor can be fixed by Traveler (unless there is a way to tell the JDBC driver explicitly to use TLS 1.2 -- I did not find any setting yet).

The following table still seems to apply. I haven't tested in detail.
But it's not clear how they will address this issue, because it's more a JDBC driver IBM JDK issue.

IMHO the IBM JVM should be fixed to use TLS 1.2 per default because this might cause also issues in other applications implemented in the same way.

-- Daniel



https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.1.0/com.ibm.java.security.component.71.doc/security-component/jsse2Docs/disablesslv3.html
Protocol label
Protocol enabled before the fix
Protocol enabled after the fix
SSL SSL V3.0
  • Server: TLS V1.0, TLS V1.1 and TLS V1.2 protocols.
  • Client: TLS V1.0 protocol
SSLv3 SSL V3.0 None, the connection fails.
TLS TLS V1.0 (defined in RFC2246) TLS V1.0 (defined in RFC2246)
TLSv1 TLS V1.0 (defined in RFC2246) TLS V1.0 (defined in RFC2246)
TLSv1.1 TLS V1.1 (defined in RFC4346) TLS V1.1 (defined in RFC4346)
TLSv1.2 TLS V1.2 (defined in RFC5246) TLS V1.2 (defined in RFC5246)
SSL_TLS Enables all SSL V3.0 and TLS V1.0 protocols TLS V1.0
SSL_TLSv2 Enables all SSL V3.0 and TLS V1.0, V1.1, and V1.2 protocols Enables all TLS V1.0, V1.1, and V1.2 protocols



Domino Backup Feedback, Requirements and my new project

Daniel Nashed  4 April 2019 13:01:08
Domino Backup

As a consultant I see many Domino environments every year and the topic "Domino Backup" is coming up regularly.

Today there are still Domino servers not being backuped with Domino supported backup applications!

And I even see servers without Domino transaction logging enabled!!


It's really a best practice and requirement to enable transaction logging for any type of Domino server for data consistency and concurrent access to databases

-- even you are not leveraging point in time recovery and incremental backup when transaction logging is configured in archive style mode!


And the only fully supported way to backup a database online is to have a backup agent which leverages the Domino Backup C-API!
Open file backup and Microsoft Volume Shadow Copy (see reference below) and snapshot backup that does not leverage the backup API  is not supported and can cause data loss!


Therefore it is essential to have a supported backup solution for your environment.

Usually you are not having a separate solution for Domino but have selected your strategic backup solution.


Most larger vendors like IBM, EMC and NetApp support Domino.
On the other side  for Backup Exec does not support Domino any more

(I have my personal opinion about Backup Exec and how they implemented their Domino and specially their DAOS Support. But that's another story).


Getting feedback about current solutions


There is a AHA idea to collect a list of support backup solutions for Domino (see AHA idea in references below).

And I would volunteer to collect them thru comments here and emails and check with IBM/HCL if they want to technote them or if I post them on my blog.
So feel free to send me the solution you are using and I am also interested in your feedback.

That might be a good starting point for others to check which backup solution might fit them.

I think we have to distinct between larger enterprise solutions and solutions for smaller environments.


Enterprise products I work with


I have personally worked mainly with IBM TSM/TDP (now called IBM Spectrum Protect) and the EMC Networke. Both work like a charm on command-line. I don't like their GUIs at all -- LOL.

And I helped customers with Linux deployment scripts and also backup/restore scripts including DAOS restore.

Beside those two, NetApp has a very interesting backup solution (see references below) which allows fully supported snapshots by bringing all databases into backup mode before initiating the snapshot.

They implemented their own snapshot agent. But it only works with their back-end storage.

Requirements


With storage optimization in Domino the disk size which needs to be backed up with a Domino aware backup agent is getting smaller.

DAOS Files with dedupliate attachment data will be stored outside the NSF in a separate file-system can be backed up separately with a file agent.

View indexes can be moved into separate .NDX files leveraging NIFNSF introduced in Domino 9.0.1 FP8. And FT Index Files can be moved to a separate disk already since Domino 8.5.3.


So independent from your backup approach (file backup or snapshot) this can reduce your on-line Domino backup server storage and time already dramatically.

We even have customers putting their DAOS files into a central storage system like a NetApp via NFS and gain additional optimization by block level deduplicating NLO files from multiple servers (see references below).

And HCL is looking into further optimization for Domino 11 by introducing a tiered DAOS storage model where older NLOs can be moved to any S3 compliant storage on prem or in the cloud.


So we are mainly having the requirement to backup NSF files on-line. All other parts can be backed up without a Domino aware solution or for index files, need no backup!


Current Requirements


Today most servers are virtualized and more and more customers are moving to snapshot type of backups. But not everyone might be using NetApp storage.
In addition we have smaller environments or hosted environments where those enterprise solutions are too expensive or too complicated to implement.


There are also requests on the AHA website (see AHA idea in references below) for an affordable backup solution for smaller environments.
Having a way to store backups into simple tar files or rsync them to a different server are common requests I hear also from partners for their own hosted servers.

And also for my own Domino servers I could benefit from a simple backup solution.

I looked into what is already available today on OpenNTF. But that wasn't fully fitting my requirements. The current project "just" generates a consistent file-copy of the NSF leveraging the backup C-API.

My intention would be to have flexible Domino backup application which should fit for any type of backup solution -- either file or snapshot based!


I spent some time over the last weekends and build a first version which integrates with Backup solutions invoking command-line scripts per database or snapshot begin and end scripts in case of snapshot backup.


My current idea is to make this application available for free for smaller environments (e.g. 10 users, 100 database per server) and for example the Domino Community server and have a commercial version available for larger environments.

This would allow any type of integration and one of the reference implementations would be a Linux & resync or tar backup with all required scripts.

But I also want to work with vendors (already spoke with one who support snapshot backup) to see if my solution could complement their existing applications and enable them for Domino.


Here is a very brief overview of the current implementation. I am really interested to get your feedback if this is an approach that could be helpful.




My Domino Backup Project "nshback"
  • On-line Domino Backup leveraging the Domino Backup C-API
  • Support for circular log with full backup
  • Support for archive translog and incremental backup
  • Point in time restore leveraging transaction logs
  • Incremental backup (either in combination with archive-style translog or just backup databases that have changed since the last backup)
  • Support for snapshot backup of file-system (bring all databases into backup mode, take snapshot and backup delta occurred during backup)
  • Store backup logs in CSV files along with the backup (for disaster recovery) and also store it in a central NSF (for invoking restore requests directly from the NSF)
  • Flexible restore options that allow to: bring database online, point in time recovery, disable replication, change replica ID, change title, disable all agents, etc.
  • Support for Win64 and Linux64
  • Command-Line integration for backup tools for file and snapshot backup

Those features are already implemented. The remaining challenging part is how to organize and maintain the backups and to have proper backup retention.

This does highly depend also on the back-end storage used for the backups.


I am looking into S3 compliant storage right now for the mentioned two tier DAOS storage leveraging S3 storage planned for Domino 11.

So I am also looking into S3 integration for this backup solution.
I have found an interesting project (
https://hub.docker.com/r/minio/mc/) which offers a command-line interface for Windows and Linux to S3 storage which could be easily integrated leveraging scripting.

Open points


I am still working on the right way to restore all databases in disaster recovery mode.
But that will also depend on the back-end used for storing the backups. Also the round trip from backup to restore needs a manual step today.

That's also mainly because of the flexibility for backup solutions.
I might start with a full reference implementation for back-end storage leveraging one tar/zip file for each backup and continue from there with integrating with other infrastructure.


While implementing the incremental/delta backup functionality I figured out that there is a C-API call which is not completely helpful to figure out if a database needs a new backup.

I have worked around it by comparing the DBIID of the database (which is actually the datetime when this instance was created. But there are two internal fields in a database header that show when a database was last backed up (see AHA idea in references below).

But that's just an enhancement which would allow more granular control.



What do you think???


I am interested in all type of feedbacks, either via comments or per mail. I really want to understand what the community needs.

And I also want to understand what currently works good for your with your current backup solutions.

I am not saying that you should move from your current solution. It's more that I would like to introduce new options that might help in certain cases.


-- Daniel


Appendix - References and additional information


1.) AHA Idea - Requesting to provide a public and official list of third party tools for backup compliant with Domino


https://domino.ideas.aha.io/ideas/DOMINO-I-706

2.) AHA Idea - A simple backup client for the NSF


https://domino.ideas.aha.io/ideas/DOMINO-I-131

3.) AHA Idea - Need Backup C-API Call to get backup start and end date for a database


https://domino.ideas.aha.io/ideas/DOMINO-I-529


4.) Is Microsoft Volume Shadow Copy supported for Domino backups?


http://www.ibm.com/support/docview.wss?uid=swg21196479

5.) Blog Post - Domino with NetApp Storage


http://blog.nashcom.de/nashcomblog.nsf/dx/domino-with-netapp-storage.htm

6.) NetApp Snap Creator® Framework 4.3.1


https://mysupport.netapp.com/documentation/docweb/index.html?productID=62378

    Notes G1 Kit Finally Available

    Daniel Nashed  3 April 2019 15:29:36
    As indicated yesterday, the fixed G1 Kit is finally available today.
    Here is an official statement from IBM DACH about what happened in detail.


    https://dnug.de/tom-zeizels-blog-notes-v10-wir-sprechen-deutsch9506

    This post explains in more detail why we had to wait for the G1 version and what happened in the background.

    I google translated the essential part of the text for our none German speaking yellow bleeding community below and checked that the text still makes sense.

    Obviously HCL had bigger challenges than my simple auto translate from on the road.


    -- Daniel



    -- extract translated text from the original post --


    Software translation packages are centralized at IBM. However, none of the persons involved in the development partnership went to HCL and HCL had to re-establish the topic. Just to remind you: HCL has not only taken over almost all ICS / Lotus / Iris developers in the Boston area (USA), but has also created more than 150 new jobs. First goal of the assumption of the then ex-IBMer was of course to minimize skill losses.


    And that was the problem with the language packs. There was no skill transfer here and the new colleagues probably underestimated the topic a bit. In any case, some IBM Champions discovered very early that there were problems with the first release of Language Pack 1. IBM / HCL then withdrew it very quickly in order to do no harm to the customers. The problem immediately received tremendous attention in the development team, but also in the management. It quickly became clear that the roots of some of the problems were already in the older versions, but nobody really noticed them there. So you had to do more than just exchange a few words.


    This has taken a few weeks, but it is also thanks to the support of some champions who have helped testing, certainly the best quality language package 1 for a long time. And the high attention paid to the management problem has also highlighted the fundamental importance of these packages. For this reason, Richard Jefts, the responsible general manager at HCL, has also announced that in the future HCL will always deliver the language pack 1 together with the original English release and not some 90 days later. I think that the problem has had something good, too - and it shows that HCL as a modern organization also learns quickly.


    -- extract translated text from the original post --



    Update: Thanks to Christoph Adler, here is a detailed download list with all the part numbers


    -- IBM Notes 10.0.1 Language Kit G1 is available for download --

    IBM Notes Client v10.0.1 Multilingual User Interface for Windows (Group 1) Multilingual (CC0I9ML)

    IBM Notes v10.0.1 Windows German (CC0HRDE)
    IBM Notes v10.0.1 Windows Japanese (CC0KCJA)
    IBM Notes v10.0.1 Windows Brazilian (CC0HZBP)
    IBM Notes v10.0.1 Windows Italian (CC0HYIT)
    IBM Notes v10.0.1 Windows Japanese (CC0HXJA)
    IBM Notes v10.0.1 Windows Simplified Chinese (CC0HTSC)
    IBM Notes v10.0.1 Windows Traditional Chinese (CC0HWTC)
    IBM Notes v10.0.1 Windows French (CC0HVFR)
    IBM Notes v10.0.1 Windows Spanish (CC0HUES)
    IBM Notes v10.0.1 Windows Korean (CC0HSKO)

    IBM Notes, Domino Designer and Admin V10.0.1 for Windows German (CC0I0DE)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Japanese (CC0KDJA)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Simplified Chinese (CC0I2SC)
    IBM Notes, Domino Designer and Admin V10.0.1 for Windows Korean (CC0I1KO)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows French (CC0I4FR)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Spanish (CC0I3ES)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Brazilian (CC0I8BP)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Italian (CC0I7IT)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Japanese (CC0I6JA)
    IBM Notes, Domino Designer and Admin v10.0.1 for Windows Traditional Chinese (CC0I5TC)

    IBM Notes v10.0.1 Mac 64 bit German and Italian (CC0IDML)
    IBM Notes v10.0.1 Mac 64 bit French, Spanish, Brazilian Portuguese (CC0ICML)
    IBM Notes v10.0.1 Mac 64 bit Korean and Japanese (CC0IBML)
    IBM Notes v10.0.1 Mac 64 bit Simplified Chinese and Traditional Chinese (CC0IAML)

      Notes 10.0.1 G1 NL kits slipstream are on the way

      Daniel Nashed  2 April 2019 22:36:38

      As many of you know the G1 Language Kits have been pulled back and had more issues than just the missing ShortCuts.

      It took a while to get those addressed. And I know first hand how much work HCL spent getting those addressed.


      The move from a global team in IBM being responsible to a new team in HCL had more challenges than expected.
      But as I mentioned in my last post, I am very confident that it is getting much better in the next versions. And we will get at least the G1 languages shipped immediately with the GA version of Notes 11.


      The new versions of the G1 kits will work properly with FP1. And in case you have installed the previous version, you should completely reinstall the client!

      There is a brand new technote describing an issue when upgrading the first shipped version-->
      https://www.ibm.com/support/docview.wss?uid=ibm10879631

      I am interested in your feedback once the new version is available. Either by comment or direct mail


      We are all waiting for the G1 versions for client deployments.

      And my friend Christoph Adler is waiting for it as well for his DNUG client workshop next week -->
      https://dnug.de/erinnerung-workshop-zum-ibm-notes-client/
      The workshop will cover up to date information about the Notes client and the new free Marvel Client essentials and the free upgrade tool to Notes 10.


      -- Daniel

      HCL releases 10.0.1 FP1 and AppDev Pack 1.0.1

      Daniel Nashed  31 March 2019 12:52:29
      Just in time before of Q1 HCL releases the following two updates:

      1. Notes/Domino 10.0.1 FP1

      This is a FixPack not a FeaturePack. In contrast to 9.0.1 with the FeaturePacks IBM/HCL went back to a full release cycle which makes a lot of sense.
      We will see an new version every year with quaterly FixPacks and if needed Interims Fixes (IFs) for the released versions.
      This is the first fixpack. The release notes have been already published but they are not final. There could be additional fixes that are not listed yet.

      There are a couple of importan fixes and also some new notes.ini settings for SAML to customize the current behavior.

      http://www.lotus.com/ldd/fixlist.nsf/WhatsNew/b1df4042fb8a980c852583b40067a7be

      2. IBM Domino AppDev Pack 1.0.1

      With 1.0.1 HCL ships the IAM component for authentication as a supported product and also adds support for Windows.

      See this blog posts for the official statement with details an links.

      https://www.ibm.com/blogs/collaboration-solutions/2019/03/29/exciting-news-ibm-domino-appdev-pack-1-0-1-is-now-available-on-windows-and-verse-on-premises-1-0-7-has-ical-import


      I got many questions when the G1 Language Kit will be released.
      From what I understood, it's ready to be reshipped. And it has been tested with FP1.


      -- Daniel


        My two cents about HCL

        Daniel Nashed  30 March 2019 11:53:15

        It has been a surprise for us that HCL is taking over Notes/Domino and most other collaboration products completely from IBM.
        The deal is not yet effective (hopefully beginning of June) and it should be "business as usual" until then.
        But we already see what we can expect from HCL taking over the complete portfolio including product strategy, sales and customer relationship.

        From what I have seen so far from the products and platforms division of HCL (which is more like a company inside a company) is amazing. They have the chance and the need to do things different!
        I have been at two events this year. One was the Factory Tour 2.0 in Milan and the other one was Admin Camp in Germany this week.

        The developers I meet in both events are more than just listing to customers and partners.
        They are actively seeking for feedback and want to work directly with customers and partners on delivering what we need.
        Beside the Jam events and the on-line version https://domino.ideas.aha.io/ the developers are looking for 1:1 feedback at events and also talking for example to the Champion community.

        Ideas they showed us in Milan got updated for Admin Camp based on feedback they got in Milan.
        And we had active discussions about topics like security, S3 support for a Domino tiered DAOS store where objects are moved to a S3 compliant, cheaper storage when not needed for a longer time.

        I really see the passion and energy coming back from the old Lotus times. This part of HCL is more behaving like a startup than a 42 year old company.
        And the statement that I found "YOUNG enough to be BRAVE. OLD enough to be SMART." fits from what I have seen so far.
        Also quote "relationship beyond the contract" is part of the corporate strategy to work close and open with customers and partners.

        Interesting is how the same people we worked with before are now allowed to speak more open to us and engage directly with us!
        So the Client Advocacy Program and other changes are not a coincident but part of their corporate strategy.

        Today HCL cannot speak about all details but they are already working on Notes/Domino/Traveler/Sametime 11 because those products have been part of the first deal with IBM.
        It is going to be an interesting but also challenging time. There will be changes (from what I see mostly positive), There will be mistakes because a lot of things are new. But they will learn fast and take actions.

        For example the Language Kit G1 which had to be pulled back because it wasn't completely ready.
        They fixed it and it will be available hopefully next week along with Fixpack 1.

        Even before shipping again, they already announced that for Notes/Domino 11 the language kits for G1 will ship at the same time, the product ships!
        And that we will even get beta versions for language kits!

        They took over from a global IBM team doing all the languages and had a learning curve because of some unexpected surprises but they improved the process to get it right for the next version!
        On the one side we are waiting already for a while on the other side this was exactly the right move forward! They are changing the process in the back-end based on that.

        We can expect more information about Notes/Domino 11 and other products at the next Engage event and also at DNUG Conference here in Germany.
        They have been both explicitly mentioned in HCL presentations at AdminCamp. And DNUG conference is around the planned time HCL takes officially over.

        You can already have a look into the factory tour slides for details especially for the Advocacy and the Partner Program
        Not all slides from the Factory tour have been posted but you should have a look what is there --> http://www.cwpcollaboration.com/agenda.html

        Business Partners should have a close look into the business partner slides! What HCL is planning for the program differs a lot from IBM in a positive way.
        The session was also a feedback session for HCL. So you can expect details to change.. But you should already become a HCL partner!

        Here is also the entry point for HCL Collaboration product website  -> http://www.cwpcollaboration.com  where you can expect more information as the deal is officially effective.


        -- Daniel













          Domino on Linux Platforms

          Daniel Nashed  4 March 2019 15:11:21
          Now that we have official Domino support for CentOS I would expect everyone who needs a free server OS to look into CentOS.
          Still -- I got the second person in two weeks looking into Domino on Ubuntu.

          Just to be clear -- everything that is not RHEL, SLES or CentOS LTS with a matching release is completely unsupported by IBM/HCL and also by my Start Script and other solutions.
          I will not answer any questions for anyone using Ubuntu or other distributions!

          My start script is completely free of charge and I am trying to answer every question.
          But I cannot spend additional time to look into other distributions.

          And I also don't see a need for looking into other distribution since CentOS is supported.
          CentOS is also the base for the official Domino on Docker script. And is also used as the current development platform at HCL.

          There are huge differences between Linux versions and distributions and it doesn't make sense to use anything untested and unsupported!

          CentOS administration isn't that difficult. But maybe I should post a quickstart HOWTO for Domino on CentOS?
          A setup from a minimal image just takes a couple of minutes and there are not many command line steps to follow.
          I have added a sample rule for the firewall configuration xml to my start script which can be used to open the NRPC port.
          Firewall and network configuration might be the only two a bit more complicated steps beside systemd (but systemd is something that you will have to deal with in any case).

          -- Daniel


          Traveler Optimization for Slow Backend Mail Server Connections

          Daniel Nashed  3 March 2019 09:30:57
          In the last couple of month we have been working on performance bottlenecks for customers with higher latency network connections between Traveler and the back-end mail-servers.
          It took a while until we got all the fixes implemented after very detailed analysis (for example I wrote an extension manager to track object reads).

          The good news is that those fixed are included in the current release and most of the settings are now even enabled by default in the latest releases.


          [Side note about Traveler accepted latency]


          IBM/HCL recommend that the connection between your mail-servers and Traveler servers should have less than 50 ms latency!
          But you don't have always a choice. On the other hand I have seen corporate network connections with latency around 5/6 ms today!
          Even internet connections between two different provides I use are around 6 ms!

          See technote for recommendations and troubleshooting steps:


          https://www.ibm.com/support/docview.wss?uid=swg21961707

          My first observation was that the attachments for richtext messages are sent multiple times over the network during sync, which lead to the first fix already implemented in the Traveler 9.0.1 code stream.

          After we got the fix, I figured out that also MIME messages have been effected in similar way -- It was just harder to track.


          Specially on WAN networks transferring attachments multiple times causes additional network utilization and in combination with higher latency also causes slower sync.

          Not just when the attachment is syned, because attachments might be pre-streamed in some cases.

          The changes are very low-level in the back-end how Traveler uses the Domino APIs. So the overhead was only trackable below the Traveler interface to Domino (C-API calls).


          The two many changes have been implemented in Traveler 10.0.1 / 10.0.0 and one fix needed a changed notes.ini to not pre-stream the attachment.

          In our first hotfixes the parameter needed to be disabled NTS_ATTACHMENT_PRESTREAM=false but since 10.0.1 the parameter is disabled by default.


          The pre-stream of attachments was needed for blackberry devices which need the exact size before syncing a document. Unless you have blackberry devices the new default should work for you.


          The two main fixes are the following:


          Traveler 10.0.1.1
          TRAV-3279         MIME message processing reads attachments multiple times


          Traveler 10.0

          TRAV-3004         Avoid streaming attachments just to calculate size.



          In addition Traveler 10.0 introduces two other optimization fixes for slow network connections:


          TRAV-3165         Reduce Dispatch logging to reduce network utilization.

          TRAV-2952         Master Monitor queue bottlenecked by slow response from mail servers.



          Network Session Optimization


          There is one additional notes.ini Parameter which is helpful to optimize back-end connections between Traveler and the Domino mail-servers.

          I have worked in two larger environments with a high number of Domino mail-servers in the same Traveler HA pool.


          Usually you should use separate Traveler pools for servers in different locations and best practices would be to have a Traveler pool in the same data center than your mail-servers. But this isn't possible in all customer environments.

          In combination with a high number of users on different mail-servers and a single Traveler HA pool, we have seen many open network connections per mail-server.

          You can see up to 40 ESTABLISHED network sessions with mail-servers for a longer time.


          The following (finally officially documented) NTS parameter helps to optimize and properly recycle those Domino NRPC network sessions between Traveler and your mail-servers.

          If you are experiencing a high number of open NRPC sessions per Domino back-end mail-servers, you should have a look into this parameter.


          NTS_DOMINO_THREADS_OPTIMIZE_RECYCLE=false


          Controls whether IBM Traveler threads that use Domino API calls are Domino initialized and terminated when IBM Traveler is done with the thread and
          the thread is destroyed (true) or when each usage of the thread for a user's device is done but the thread is not destroyed (false).
          True saves the overhead of doing the initialization and termination for each user's device but NRPC connections are cached per thread and only released upon the termination.
          If your IBM Traveler server is talking over NRPC to a large number of mail servers (for example, more than 100) and the IBM Traveler server is running out of TCP/IP network ports,
          you may want to change this value to False to force more frequent thread terminations which release NRPC connections more frequently.

          Archives


          • [IBM Lotus Domino]
          • [Domino on Linux]
          • [Nash!Com]
          • [Daniel Nashed]