Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

DNUG Domino Day 28.11.2019 in Köln

Daniel Nashed  18 October 2019 10:28:05

DNUG Domino Day 2019 in Köln

Auch in diesem Jahr haben wir wieder einen Domino Day organisiert.
Dieses Jahr im Herzen von Köln am Mediapark am Donnerstag, 28. November 2019.

Im Fokus der Veranstaltung steht das anstehende Release von Domino V11 und auch die neuesten Informationen von HCL zu Lizenzen und Support.
Es geht aber genau so um Neuerungen bei allen anderen Produktion, die in den Themenbereich der Fachgruppe fallen.

Die Anzahl der Teilnehmer ist auf 70 Plätze beschränkt. Daher macht es Sinn sich frühzeitig anzumelden!

Anmelde-Link --> https://www.eventbrite.de/e/domino-day-2019-tickets-70041755777

Ich denke wir haben wieder interessante Themen dabei und es hat sich einiges seit der letzten Konferenz/dem letzten Domino Day getan.
Und wir haben extra auch eine Session zum Thema Lizenzierung und Neuerungen/Änderungen im Bereich Support&Downloads aufgenommen, da ich dazu im meinem Blog und auch offline viele Fragen bekommen habe.

Gegen Ende gibt es vor einer Drink Receiption von TIMETOACT, noch eine Frage-Runde mit HCL, wo Ihr alle offenen Fragen loswerden könnt, die Ihr bis dahin noch nicht beantwortet bekommen habt.

Danke an Christoph Adler und Manfred Lenz als meine Fachgruppen-Kollegen bei der Unterstützung :-)

Ich freu mich viele von Euch zu sehen!

Ciao

Daniel

Aktuelle Agenda

09:00 – 09:15 Uhr
Begrüßung
FG-Domino: Daniel Nashed, Christoph Adler, Manfred Lenz

09:15 – 10:15 Uhr
Keynote: Strategy & Roadmaps Notes / Domino / Nomad (ggf. auch Sametime / Connections / VoP)
HCL

10:15 – 10:30 Uhr
Kaffeepause


10:30 – 11:30 Uhr
HCL Nomad
Detlev Poettgen, Christoph Adler

11:30 – 12:00 Uhr
HCL Update – Lizenzen, FlexNet (Support & Downloads) & Co.
Uffe Sorensen – HCL

12:00 – 13:00 Uhr
Mittagspause


13:00 – 14:00 Uhr
Notes V11 & VOP 1.0.8 – What’s new
Manfred Lenz, Christoph Adler

14:00 – 14:30 Uhr
Kaffeepause


14:30 – 15:30 Uhr
Domino V10 & 11 Session – What’s new & Lessons learned & Docker
Daniel Nashed

15:30 – 16:15 Uhr
AppDevPack & DQL – What’s new & Lessons learned
Stefan Neth

16:15 – 17:00 Uhr
Fragen & Antworten
DNUG-Fachgruppe & HCL

ab 17:00 Uhr
Ausklang & Drink Reception
sponsored by TIMETOACT
Termin

Donnerstag, 28.11.2019
9 – 17 Uhr
Ort

STARTPLATZ Köln
Im Mediapark 5
Raum Barcelona, 1. OG
50670 Köln
Anmeldung


    CentOS 8 Released

    Daniel Nashed  5 October 2019 15:55:47
    RHEL 8 is available for a while. And traditionally CentOS takes a couple of month before it is also updated to the same code base.

    I have downloaded and installed CentOS 8. The first version was the full version. There wasn't a minimum base image yet.

    Be aware that neither CentOS 8 nor RHEL 8 nor SLES 15 SP1 are currently supported!
    There are even packages which have older versions in CentOS 8 than the last updated of CentOS 7!
    For example I tried to install the latest Docker CE version 19.09. It needs a newer containerd.io version than what is currently shipped on CentOS 8.
    So I would stay with your current releases for now!

    Of course I have looked into SLES 15 SP1, RHEL 8 and CentOS 8 with Domino to see if it works.
    I don't think it makes sense to look into Domino 10 support for those platforms. I would expect an updated Linux version support for Domino 11.

    On the Docker side I ran into an issue preparing demons on a different machine.
    When you pull a centos:latest image today, you will get centos 8 which isn't working with the current dockerfiles which ships with the master version of the Docker project.
    We have updated the develop branch of the project already with a changed dependency:

    FROM centos:7 will continue provide the latest image of CentOS 7.

    So the project has been updated with another currently experimental dockerfile to build Domino on CentOS 8 for testing.

    But I can only recommend that you stay on CentOS 7 for now because that's the tested and supporter version.

    This is true for native and also Docker image versions!

    -- Daniel

    HCL Nomad V1.0.4 released

    Daniel Nashed  5 October 2019 15:32:28
    There is a new iPad Application which has been released.
    This is the first version from HCL and Nomad V1.0.4 replaces the offering known as IBM Domino Mobile Apps.


    The new version also comes with some interesting new features

    - Open your personal mail file in Nomad
    -> this was prevented by internal policy and is now allowed by default
    - GPS geo location support via LotusScript
     --> this will be available in the Notes 11 designer
    - Mobile Device Management pre-configuration
    - Free panaganda MarvelClient for iOS integration!

    - And there is also an option to disable DNS lookups to improve compatibility with some VPN solutions.
    - @platform([specific]) now returns the following type of text-list: iOS; 13.1; iPad; iPad7,5

    There is an interesting blog post from Andrew Mandy and Andrew Davis providing also information about what is coming next :-)


    https://www.cwpcollaboration.com/blogs/update-strategy-and-release-of-nomad-104-for-apple-ipad

    Specially the free Marvel Client (MC) integration is great.
    MC is integrated into Nomad 1.0.4 and allows you to check and update your Nomad clients.

    In combination with the MDM configuration profiles this means zero manual configuration.


    I have online-updated my MC of my existing installation and see my iPad in the MC reporting database.


    There is a FAQ regarding MarvelClient for iOS, whch you might want to check.


    https://www.panagenda.com/marvelclient-for-nomad/

    Well done integration! This is really what we need!

    Congrats & Thanks HCL and panagenda
    !


    iOS 13 Native Mail App Issues with Traveler

    Daniel Nashed  3 October 2019 15:41:43
    iOS 13 introduced a couple of changes. Some of them are done with good intention, but they broke existing functionality.
    There are currently two issues that are known. Both are described in detail in the following technote:

    https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0073141


    The technote will be updated with new information about the known issue and also new issues that might arise with iOS 13.

    The second issue just occurs when you have calendar ghosting disabled, which is enabled by default.
    But the issue with the duplicate sent mail folder entries cannot be worked around.

    Beside those two known issues, Traveler should work with iOS 13 and the native mail app.
    You should updates to iOS 13.1.2 which is another update in a short time we got for iOS 13.
    "13" isn't a good number of Apple as it sounds...

    -- Daniel


    1) Duplicate Sent folder entries

    As of iOS/iPadOS 13.0, Apple devices add an entry to the Sent folder for any emails sent from the Mail app.
    When the Sent folder is synced, the server entry is added and the device does not remove the original, resulting in a duplicate. HCL development has opened an Apple bug for this issue (FB7337231).

    Workaround: No workaround is available


    2) Accepting a meeting invitation from the iOS device does not send the response to the server

    If ghosting is disabled on the Traveler server, responding to a calendar notice from an iOS 13 device does not send the response.
    The meeting accept is reflected on the app but not in the user's notes calendar.  
    iOS 13.0 and 13.1 do not send MeetingResponse requests to the Traveler server unless the event is ghosted to the calendar.
    HCL Development has opened an Apple bug for this issue (FB7328175).

    Workaround:
    Ghosting is enabled at the Traveler server by default.  Check the Traveler server notes.ini parameters for NTS_CALENDAR_GHOSTING_SYNCML and NTS_IOS_CALENDAR_INITIAL_GHOST.  If found, make sure that they are set to true.

    RNUG -- Russian Notes User Group Event in Moscow

    Daniel Nashed  2 October 2019 08:48:42
    This is going to be a very special event! I am really looking forward to be there next week!
    I have never been to Moscow and I am looking forward to the event and also visiting the city.

    The venue looks great and there are many known speakers including fellow IBM Champions/HCL Masters from around the world.


    https://en.rnug.ru

    My sessions will be about Domino Performance and also Domino on Docker.

    And we are having a Domino on Linux Round Table session as well to get feedback from the Russian market.


    I have blogged about some tests which I have done with a local Linux which seems to be quite popular -->
    http://blog.nashcom.de/nashcomblog.nsf/dx/domino-on-astra-linux-feedback.htm

    So this will be an exiting event for participants and also us speakers!


    -- Daniel


    Image:RNUG -- Russian Notes User Group Event in Moscow


    Creating Internal use X.509 Certs

    Daniel Nashed  28 September 2019 13:58:30
    For one of my test servers I needed a proper certificate. A self-signed cert works in many cases. But creating your internal CA has benefits. You can have the CA root trusted in your brwoser etc.
    I needed a certificate for a local test server today and used the script I developed for the Docker project.

    A while ago I updated the script to add also additional SANs (Subject Alternate Names) and it will also add the SANs to a CSR request if you use the script with an external CA.
    Even when just generating a certificate with just a DNS name, this name also needs to be added to the SAN.
    This was implemented from the beginning but now you can add more SANs.

    After you configured the script, generating a proper certificate is just invoking this script.
    The CA directory contains the CA root that you add to your browser afterwards.

    Here is the example and here is the link to the script --> https://github.com/IBM/domino-docker/blob/develop/management/manage_certs.sh

    The script creates the private key, generates the CSR, depending on the configuration the reuqest is signed and everything is merged together into a single PEM.
    That PEM is imported into a matching keyring file -- if the kyrtool is installed and you are running as "notes".

    -- Daniel

     ./manage_certs.sh "traveler-nashcom-loc" "/CÞ/O=NashCom/CN=traveler.nashcom.loc" "traveler.nashcom.loc,trav2.nashcom.loc,trav2.nashcom.loc"

    (Using config file /local/cfg/certmgr_config)
    Generating key [/local/certmgr/key/traveler-nashcom-loc.key]
    Generating RSA private key, 2048 bit long modulus
    ...........................................+++
    ...+++
    e is 65537 (0x10001)
    Creating certificate Sign Request (CSR) [/local/certmgr/csr/traveler-nashcom-loc.csr]
    Removing [/local/certmgr/pem/traveler-nashcom-loc_all.pem]
    Signing CSR [/local/certmgr/csr/traveler-nashcom-loc.csr] with local CA
    Signature ok
    subject=/CÞ/O=NashCom/CN=traveler.nashcom.loc
    Getting CA Private Key
    Removing [/local/certmgr/csr/traveler-nashcom-loc.csr]

    Keyfile /local/certmgr/kyr/traveler-nashcom-loc.kyr created successfully


    Using keyring path '/local/certmgr/kyr/traveler-nashcom-loc.kyr'
    Successfully read 2048 bit RSA private key
    SECIssUpdateKeyringPrivateKey succeeded
    SECIssUpdateKeyringLeafCert succeeded

    --------------------------------------------
     traveler-nashcom-loc -> OK
    --------------------------------------------
     KeyLen       :  2048 bit
     Subject      :  /CÞ/O=NashCom/CN=traveler.nashcom.loc
     DNS NAME     :  traveler.nashcom.loc, DNS
     Valid Until  :  Sep 25 10:12:07 2029 GMT
    --------------------------------------------

      HCL Traveler 10.0.1 FP2 Released

      Daniel Nashed  20 September 2019 15:09:12
      The first "HCL" Traveler updated has shipped.

      This version contains updated APNS push certificates, because the current shipped cert expires mid of October.


      Beside that fix there are a couple of other fixes which might be relevant for your environment.


      Here is a list for all changes. There are no big surprises. But there also some backend changes for the updated Verse client which is coming soon.


      https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0072728



      IMPORTANT: Required reading for Administrators - Upgrading from IBM Verse for iOS to HCL Verse 10.0.7 for iOS  


      Here is a technote for important changes with the upcoming first HCL Verse release.
      They are changing the name of the application which is has some impact.


      https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0069584


      I got also questions about iOS 13 support. There are no known issues with Traveler 10.0.1.1 and 10.0.1.2.

      HCL did only test those two versions. Earlier versions should still work but it is highly recommended to update!


      Traveler 10.x is supported on Domino 9 and the installer detects the underlaying Domino version.
      But it is still recommended to update to Domino 10.0.1 with a current fixpack (currently FP3).



      The new Traveler 10.0.1.2 version is available on the HCL Flexnet Download portal.


      Here is a search link:


      https://hclsoftware.flexnetoperations.com/flexnet/operationsportal/DownloadSearchPage.action?search=Traveler_10.0.1FP2&resultType=Files&sortBy=relevance&listButton=Search

      And here are the file-names and the SHA256 hashes.


      I am currently looking a lot into those download options and filenames because of Docker and we are adding the new version to the software.txt file..


      -- Daniel


      HCL Traveler v10.0.1 FP2 for Linux M

      Traveler_10.0.1FP2_LINUX_ML.tar.gz        
      SHA256 CheckSum        4669eb49ad354d7bc8f67bf02693d6d1eccdf69b3a0a0317657d14370c793109



      HCL Traveler v10.0.1 FP2 for Windows ML

      Traveler_10.0.1FP2_WIN_ML.zip
      SHA256 CheckSum        a8ea719ce0ede272b51b23fdb43b3dbf7c31839534b4f69796851063e21727f9


           
       

      Docker Support for Domino 10.0.1 FP3

      Daniel Nashed  12 September 2019 22:51:32
      The first HCL Software packages are released. Domino 10.0.1 FP3 is the first download which is only available from HCL.
      So we added support for HCL Flexnet downloads.

      The file names for the software have different names for older fixes and Domino itself, than the files from IBM Passport Advantage and FixCentral Downloads.
      So we added support for more than two file names per software file. The first file is now the HCL Flexnet download, which will also be used to generate a download hint.

      Example:

      10.0.1FP3           [NA]  Domino_10.0.1FP3_Linux64.tar  (-)
      https://hclsoftware.flexnetoperations.com/flexnet/operationsportal/DownloadSearchPage.action?search=Domino_10.0.1FP3_Linux64.tar+&resultType=Files&sortBy=eff_date&listButton=Search

      I didn't find a direct link to a software download that worked. If you select the link you get, it will not work for someone else. But at least the search will return a single file.
      The changes allow to build Domino 10.0.1 FP3. But because some partners and customers still did not mange to download FP3, I did not make FP3 the default version yet.

      If you want to build with FP3, download the software and run

      ./build.sh domino 10.0.1 FP3

      This will build an image with FP3 but does not mark it as latest yet.

      We also added the Domino 11 beta version to the software list and if you have access to the beta version, you can build a Domino 11 beta image.
      The Docker project is prepared for the beta and you will see HCL instead of IBM branding.

      To build Domino 11 Beta1 images run:

      ./build.sh domino 11.0.0.beta1

      All those changes are currently submitted to the develop branch.


      -- Daniel

      Fail2Ban Support for Domino on Linux -- Intrusion Detection

      Daniel Nashed  13 August 2019 17:18:23


      Introduction


      Domino supports Internet password lockout, which is meanwhile working for all internet protocols (it came in thru a fix I think somewhere in the 8.5.x code stream and isn't really documented).

      This does already help to protect individual accounts. But it doesn't currently help for the same IP trying to hack different accounts.


      There is a
      AHA idea to improve it. And I think it is an important functionality for Domino. But blocking IPs with suspicious login attempts isn't always simple for an application.
      On the one side someone behind a remote proxy could be blocked if there are too many people having bad password attempts at the same time.

      On the other side if your server is behind a secure proxy, you don't have full control to block IPs


      As long you have remote IPs hitting your server "directly", you could block them on your server.

      This will work for many infrastructures and there is already a quite flexible solution for Linux.

      Fail2Ban offers a wide range of "filters" for different applications which parse log files to find out which IP is not behaving correctly and blocks them in the local Linux firewall.

      The idea is to have Fail2Ban read thru the Domino console log (better notes.log from my start script because it never wraps around) to find failed password attempts.

      Fail2Ban is designed to track and block those IPs in the local Linux firewall.


      Here is a sample line for an invalid login attempt. All other protocols use the same format.


      [10780:00015-00007F4E8FFA6700] 08.08.2019 22:52:04   http: john.doe@acme.com [1.2.3.4] authentication failure using internet password


      Once you have the right filter defined, it's quite easy to install and use Fail2Ban.

      I wrote a filter for Domino and also have a default configuration which also includes a configuration for sshd.


      The following is a installation description provides all you need to be up and running.
      It also includes information about operations like status checking, unblocking users and troubleshooting.

      The scripts used will be added to my start script in the "extra" directory.

      It's only a solution for Linux and right now only for a local server without a proxy.

      For Linux this offers also protection for other services like sshd.


      Proxy Support


      A friend is using NGINX in front of the Domino HTTP stack on the same machine. And he asked if I could help to get fail2ban working in combination with a proxy in front.

      From Domino point of view traffic appears to come from the proxy IP address. But I found a solution which isn't what I expected but it works.

      Via notes.ini HTTP_LOG_ACCESS_XFORWARDED_FOR=1 you can configure to write an additional field "ForwaredFor" into domlog.nsf.


      The log entry (see above) still lists the proxy IP. There is another
      AHA idea to enhance the logging.
      But for now I wrote a small extension manager, which captures the domlog.nsf update and writes the original requesting IP in the same format into log. So Fail2Ban can capture the right IP address data.


      Remote Proxies


      This works for a locally installed proxy, but for a remote proxy you will have to pass the information to the proxy. This could be done with event monitoring configurations (run a program, start an agent, etc) based on the log information.

      This would a more tricky configuration. The basic configuration is pretty simple.


      Below you find all the instructions and additional information.

      Enjoy and let me know what you think.


      -- Daniel



      Current Implementation and Feedback


      This installation instruction below uses CentOS 7.6. But once you installed Fail2Ban it will also work with other distributions.
      I have it also tested with CentOS 6.10 which works a bit different because init.d is used instead of systemd.


      The current implementation checks for all protocols (http, smtp, ldap, imap, pop3).

      It is a single filter which would count failed login attempts for all protocols together and than blocks the IP for all protocols.

      This seems to be the most reasonable configuration. But depending on your needs you might want to have separate filter definitions and configurations.


      The current script can be easily adopted to individual protocols.

      But to keep it simple and also because I think this should be the most reasonable way in most cases.

      I am looking for feedback if this is what you need. Alternatively I could have a separate filter for all protocols like "domino_http.conf".

      But it is far easier to just have one definition and one rule set.



      -- Installation --


      First of all you have install Fail2Ban. It's included in the epel repository, which can be enabled via yum


      yum install epel-release


      Next you can install the package


      yum install fail2ban



      Disable SELinux


      Before you can run the log filter, you have to disable SELinux (you could also create a profile for the service, but Domino is also not supported with SELinux enabled).

      Check the status via


      getenforce


      The result should be "disabled". If not you can change it the following way.


      vi /etc/selinux/config


      Change the line


      SELINUX=disabled


      The next reboot disables SELINUX

      You can temporary disable SELinux if you don't want to reboot now (you should reboot at least later to ensure your server will still boot!).


      setenforce 0



      The application leverages python and works in combination with firewalld used by default in CentOS 7.
      You can enable and start the systemd services via systemd commands. A configuration change needs a restart.


      systemctl enable fail2ban

      systemctl start fail2ban
      systemctl restart fail2ban



      -- Domino Configuration --


      Copy new configuration file jail.local and Domino filter configuration domino.conf (contains filters for multiple protocols)

      If you have an existing configuration copy entries manually. The jail.local is a good starting point and also contains an enabled sshd configuration.
      You should review the configuration and change parameters as needed. The default configuration and the service configuration contain the same values but can be customized per service.

      Copy the two configuration files


      cp jail.local /etc/fail2ban/jail.local

      cp domino.conf /etc/fail2ban/filter.d/domino.conf


      You should review the configuration. But some details might need to be adjusted.

      The domino.conf file contains a "datepattern" which is very important for the pattern matching.
      fail2ban parses the date first and removes it from the original string line before the regex expessions are used to match the string and get the HOST IP address.


      The Script contains two definitions for the mostly used date format. The format widely used in Europe and the US settings.
      You could also change the Domino log format to the one Fail2Ban understands (see notes.ini settings in the domino.conf files).
      But I would recommend to change the datepattern in the domino.conf file instead.

      Example:


      # European Date 31.12.2019 22:11:01
      datepattern = %%d.%%m.%%Y %%H:%%M:%%S


      The second important parameter is in jail.local.

      The logpath defines the log file to check By default the standard location used by the Domino start script is configured.
      Please use the Domino Start Script log, because the file doesn't rotate like console.log!

      Example:

      logpath  = /local/notesdata/notes.log


      Afterwards restart the service


      systemctl restart fail2ban



      -- Operations --


      Check status for a jail


      fail2ban-client status domino

      Status for the jail: domino
      |- Filter
      |  |- Currently failed: 0
      |  |- Total failed:     8
      |  `- File list:        /local/notesdata/notes.log
      `- Actions
      |- Currently banned: 1
      |- Total banned:     1
      `- Banned IP list:   192.168.100.107



      List IP Tables for banned IPs


      iptables -L -p


      Chain f2b-domino (1 references)
      target     prot opt source               destination
      REJECT     all  --  192.168.100.107      0.0.0.0/0            reject-with icmp-port-unreachable
      RETURN     all  --  0.0.0.0/0            0.0.0.0/0



      Unban IP for a specific rule


      To unban an IP before it expires use the fail2ban-client

      Example:


      fail2ban-client set domino unbanip 192.168.100.107



      -- Troubleshooting --


      Check the log file:


      cat /var/log/fail2ban.log


      Check Python Errors:


      abrt-cli list



      -- Testing Rules --


      In case you want to test rules to see that for example the date format matches, you can use the following regex test tool included in Fail2Ban

      Example:

      fail2ban-regex /local/notesdata/notes.log /etc/fail2ban/filter.d/domino.conf


      You can use the following filters:


      --print-all-matched
      --print-all-missed


      -- Appendix Configuration Files --


      Just copy the following configuration files.

      The configuration is a basic configuration, which can be changed for your needs.

      You find the code currently in the start script extras directory in the IBM Domino Docker script
      .
      I had to change the download location because pasting it into the Domino blog template made some code disappear.
      It is really time to find something better than the old blog template...

      https://github.com/IBM/domino-docker/tree/develop/start_script/extra/fail2ban


      Domino Portable Edition - Building the smallest Domino server

      Daniel Nashed  3 August 2019 18:06:51
      Sadly I haven't been at the HCL Factory tour #3 in Chelmsford.
      I would like to have seen the demo first hand. But I got live tickers over iMessage when Thomas Hampel took a lot of spare time to build it.

      Thomas did show the smallest Domino server running on a small pocket size Linux based machine leveraging Docker CE on Ubuntu with the official
      Domino on Docker script.

      You should have a look at his
      blog post for details.

      The reason for Ubuntu is the missing hardware support on SLES/RHEL/CentOS for the CPU used in this tiny device.

      But the beauty is that Docker works the same on Ubuntu and runnning our Docker image is supported.
      Thomas could have build the image also on the Docker host installed on that machine.

      On the other side it was faster to export and import the Docker image from a build machine and it shows how you could do it if you have Docker installed on a different device or on the cloud where you have to import the image.


      That's Thomas for the time you spend on it also for the step by step documentation and the pictures!
      It is really great to see what is possible. The tricky part was the Linux install. The Docker part works unmodified.

      I was a big fan of the foundation server. And having something like this again but as a software solution for different standard hardware like the Intel NUC would be very cool!


      Now you have a smaller box than the NUC you and me used for our DNUG Docker workshop earlier this year ;-)


      Have a great weekend!


      -- Daniel


      Archives


      • [IBM Lotus Domino]
      • [Domino on Linux]
      • [Nash!Com]
      • [Daniel Nashed]