Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ... | ||
 Daniel Nashed
| Responsive sizing for the old Domino blog templateDaniel Nashed 27 March 2023 10:39:43The old Domino blog template isn't really responsive and specially on mobile devices it cuts off text.There is a simple change for the default.css which you can find in Resources/Style Sheets Thanks to Theo Heselmans for fixing my blog with this tip!!
DNUG Lab at #DACHNUG50 in Siegburg, June 2023 -- including K8s clusterDaniel Nashed 26 March 2023 11:54:55For last year's conference we setup a full featured permanent Lab environment --> https://dnug.de/dnug-lab/ This environment is intended for members and workshops to have a fully implemented environment to test and learn about current Domino and companion product features. For example the development focus group will use this environment for collaboration and workshops. Last month we added a full production style Kubernetes cluster including our own Harbor registry running on the K8s cluster. The Harbor registry is the same type of registry HCL uses to provide container images -> https://hclcr.io/. The new Domino 12.0.2 container image has been added recently to harbor registry. DNUG will also move the Sametime environment into this cluster soon. And we are running it as a production environment, which can be used by multiple DNUG focus groups. Some of my presentations already leveraged configurations from this environment and it is also a kind of reference implementation for members to look into. RKE2 + Rancher + Longhorn storage We setup a Kubernetes environment with master node and two additional worker nodes running on Ubuntu 22.04 at Hetzner cloud. The environment uses RKE2 + Rancher + Longhorn enterprise storage. We also plan to have a DNUG Lab booth at the conference again. This year I will bring a current Mac Mini with M2 instead of the notebook running Linux desktop. #DACHNUG50 – Die Jubiläumskonferenz -- 13.-15.06.2023 – Rhein-Sieg-Forum / Siegburg The conference will be the 50th anniversary conference! The location is between Cologne and Frankfurt and very easy to reach from both airports. After last years conference in the very southern not that easy to reach -- but very nice location in Konstanz last year, this years location should fit for everyone. I hope also Austrian and Swiss members will find their way to the conference! Call for abstracts is open DNUG has a new event management application for submitting abstracts--> https://em.dnug.de Hopefully there will be already a Notes/Domino V14 code drop around that time. In that case my topic would be the first code drop... Here is a link to conference page --> https://dnug.de/dachnug50/ I hope I meet many of you face to face at DNUG conference in June. We are planning to have a Domino on Kubernetes workshop on the first day, too. Probably everyone will be install an own RKE2 + Rancher + Longhorn environment -- there is still planning in progress and we are also looking for feedback. We will deploy the latest available Domino version (release or code drop ..) and of course use our new Harbor registry ... -- Daniel
Running Notes & Domino on Apple SiliconDaniel Nashed 26 March 2023 10:49:00The native Mac client is supported on Apple Silicon. But specially the developers among us would like to use a Notes Design client on a Mac at least. With Intel Macs you could run a VM with Windows 10/11 to have an officially supported environment. On a Mac with M1/M2 this becomes more a bigger challenge. Binaries intended for Intel/AMD x64 cannot run natively on a ARM CPU. There is always emulation involved - which costs performance impact and overhead. The following is not officially supported. And if you are just running a normal client you should always prefer the native Mac client! Even it isn't a native ARM application, the native client is the best option today. You could also run Nomad Web in Chorme, which works extremely fast -- It runs on Linux desktops too by the way.. Note: To find out which architecture an application is using, there is a useful & free tool--> https://apps.apple.com/de/app/silicon-info/id1542271266. Run Domino on Linux as container I would not try to run Domino on Linux with one of the virtualization environments I am describing below. In case you need Domino on Linux check my previous blog post. Windows 11 ARM edition When running Windows on a Mac you need virtualization software as described below. I have not tried a virtual machine running a classical Intel/AMD x64 personally. And it would make most sense to use Windows ARM to run on your Apple Silicon Mac. When looking into technical details of the Windows ARM implementation it's clear that you should only run Windows 11 ARM VMs. Arm64EC - Build and port apps for native performance on Arm This article describes the technical details of the Microsoft ARM implementation --> https://learn.microsoft.com/en-gb/windows/arm/arm64ec And it is the up to date Microsoft technology for running native ARM and Intel/AMD x64. With this information I would not even try to install an older Windows version. Virtualization Software There are a couple of solutions out there, which are reported to work quite well. This isn't a complete list. It's just what I heard this week, when asking around what is working well for others. Parallels Parallels (https://www.parallels.com) is the most recommended solution and is reported to work well. But it requires a commercial license for more than 100 euro Oracle VirtualBox VirtualBox isn't working well as far the feedback I got. VMware Fusion VMware Fusion player is free for none commercial use (https://customerconnect.vmware.com/en/evalcenter?p=fusion-player-personal-13). And from what I heard back it is working well for Apple Silicon in the current version as well. UTM A very interesting open source solution I ran into is UTM (https://mac.getutm.app/). It has a broad support for different types or hardware. And it can also create virtual machines for ARM hardware, which I use for a Windows 11 ARM VM. This is what I used to bring up a Notes & Domino server for testing and development. Conclusion & Recommendations It's not a supported environment. But it runs with decent performance and stability. I got some feedback that Java code might be problematic. My Mac isn't my main development environment. I am using it in parallel to Windows. And I also have a ESXi server running Windows 11. So I have multiple options in parallel. This is a starting point and there is an official AHA idea --> https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2387 to get this type of configuration supported for development and test environments. You can see also that emulation comes with overhead and performance impact (see the screen print from UTM above). A native ARM VM has way better performance. But still there is emulation involved inside the Windows 11 ARM VM for Notes & Domino x64 programs. The article from Microsoft makes clear that this is the currently best option we have. More and more software will support ARM natively or in an hybrid mode.
Tips running Domino Container Images on Apple SiliconDaniel Nashed 26 March 2023 09:36:12Even it is not officially supported, I want to provide some feedback and tips to run the Domino Container Image on Macs with M1 and M2 CPUs. I got question about it three times this week. Kind of a nice coincident is that I got my first Mac since years last Monday. Of course I first looked into Docker Desktop to see how the Domino Community image would work. I crashed and burned on the first run, because building the container image stuck when running the Domino Install Anywhere Java edition installer inside the build image. When you run AMD64 platform code (Intel and AMD), your Mac has to emulate the foreign CPU. This causes performance impact in general -- even with a very powerful Apple M2 chip. Apple added a way for applications to run AMD64 code. It's called Rosetta -- like it was called when Apple switched from PowerPC CPUs to Intel CPUs ages ago. But even Apple Silicon is around for some time and the second generation of Apple Silicon is released, not all software can completely handle the emulation properly. You can see that Docker Desktop warns you that the performance might not be optimal when emulating an Intel/AMD x64 CPU. "Image may have poor performance, or fail, if run via emulation" Improve Performance and get Container Images build First of all you should run the latest Docker Desktop version (V 4.17.0 99724 today). Each version will bring performance improvements. Not all features have been fully released yet. One feature, required to run AMD64 images with decent performance is still in beta. When you enable "Use Rosetta x64/amd64 emulation on Apple Silicon" performance for those image types will be much better. This allows you to build a Domino 12.0.2 image on a Mac Mini M2 in around 14 minutes. This is far away from what it takes on a native x64/amd64 CPU on Linux. But it has worked for me reliably in my current tests. Usually when running your first Intel GUI application you are prompted to install Rosetta. Docker Desktop is a native ARM binary already. So this would not trigger the installation. I used the command line "softwareupdate --install-rosetta" to install it. This will install Rosetta 2 support on your Apple Silicon Mac. Another functionality, which should bring better performance, already released. By default "gRPC FUSE" is still used. The "VirtioFS" provides better performance and stability. You also have to enable the "Virtualization framework", if not already enabled. Rancher Desktop Docker Desktop isn't free for many of you, working for larger companies. I am working on my own on open source projects only. This still allows me to use Docker Desktop for free. The Rancher Desktop project (https://rancherdesktop.io/) is completely free and includes Mac Silicon support as well. I am usually running it on Windows in combination with WSL (https://learn.microsoft.com/en-us/windows/wsl/), but I have also installed it on my Mac in parallel to Docker Desktop. Rancher Desktop with containerd back-end uses nerdctl instead of the docker command. The HCL Domino Community image supports Rancher Desktop as a build and run-time environment. In general it works in the current version 1.8.1. There are no changes needed. But the build time is like double the time needed with Docker Desktop using the mentioned options. Conclusion & Recommendations Using this type of emulation is not fully supported and isn't providing the same performance compared to native environments. If you have other environments like a ESXi server or Proxmox server to run x64 Linux native, this might be the better choice. You could also use hosted servers with remote SSH access, if you don't have a local virtualization infrastructure. But sometimes if you are on the road, there might not be a good remote internet connection. So I can see the need to at least run Domino in a container locally. I would at least build the image remotely and add it to a private registry. Another option would be the new HCL Domino 12.0.2 container image, which is a special build of the community image (using Redhat UBI 8.6 as the base). It would make sense to document the settings and current state of Apple Silicon in the community project (https://github.com/HCL-TECH-SOFTWARE/domino-container). But I first wanted to raise the awareness here and see what feedback I get. I hope this helped and gave you some ideas how to run it. If you have other tips, I would really appreciate your feedback. There is also an AHA idea to get official support for development environments running Domino inside a container on Apple Silicon. https://domino-ideas.hcltechsw.com/ideas/DOMINO-I-2387 If this is something you need, please vote and leave your comments directly on the AHA idea to raise awareness.
Preparing DNUG Domino Automation Hands-On WorkshopDaniel Nashed 4 March 2023 12:18:46Mid of this month (15.3) we are having our first hands-on workshop for Domino automation.I am in the middle of lab and slide preparation and just realized how much new and interesting functionality we got in Domino 12 which makes life of a Domino administrator easier. And there is more coming in Domino V14 to simplify updates and administration from what I heard. There are already 140 slides and I am keep adding interesting related material. There is so much to look into. And I probably need a part two .. LOL OTS, CertMgr automation, automation testing, deployment in different environment (Linux native, containers, Windows, ..) there is so much interesting stuff I worked on in the last three years. Marc Thomas from Panagenda will cover the Notes client. The first round of the workshop is in German --> https://dnug.de/event/dnug-online-domino-2/ But depending on the feedback, we might provide one in English as a repeat. For German speakers.. We are 15 people so far. And there is still room for like 5 more.. -- Daniel
Windows Updates are the highest challenge in patienceDaniel Nashed 25 February 2023 13:01:17After upgrading my ESXi to version 8.0 I am looking again into updating my Windows 11 machine. Turns out Intel NUCs have no supported TPM 2.0 module. And the software module only works with vSphere and not a stand-alone ESXi server. But I found a way convincing the Windows 11 installer specifying registry keys during boot to install. The installation was very quick. But the updates took quite long. Now I am waiting since at least 30 minutes for the machine coming up after reboot.. Most admins can't avoid using Windows completely. But I can't understand why someone would prefer to run an application like Domino on Windows. The complete setup for a Linux machine takes minutes. Updates are installed automatically if configured and the resources needed are much lower. If you manually update it's just a "yum update" (RHEL/CentOS), "apt update" (Ubuntu/Debian), zypper update (SUSE). Some Linux distributions like current Ubuntu and SUSE even tell you which services need to restart. I have auto updated enabled on all of my Domino on Linux servers hosted and just get a daily mail with updates installed. Automation is so much easier in the Linux world.. Most of my Domino servers in production run on Docker or Podman with native volumes. A Domino update is just a "dominoctl update" in my environment. I would really like to hear from you how you run your environment and if admins are looking into modern container basted deployments and why (or why not...) Blog post done, restart after first reboot is still not finished .. Hmmm... -- Daniel
Notes UIDoc reopen without saving the underlying documentDaniel Nashed 23 February 2023 22:48:20I am working on a form that has embedded passthru HTML to display in-line base64 encoded image data from a text field (don't ask why ..). The image did not always display immediately. So I tried all kind of refresh and reload options without any luck. Until I found an example in the Notes help which does exactly what I need. I tried to tweak and simplify it without any good result. It only works with this exact sequence of events for me. Now let the crow wisdom find an easier, but still reliable way. I am pretty happy with this already.
Every attempt to simplify it, failed. But this is great to have and works like a charm. -- Daniel doc.SaveOptions = "0" Call uidoc.Close (True) Set uidoc_new = workspace.EditDocument (True, doc, , , , True) Delete uidoc Call uidoc_new.Document.RemoveItem ("SaveOptions")
Udpating to Veeam Backup & Replication 12Daniel Nashed 19 February 2023 16:25:48The new Veeam Backup & Replication 12 version is available and I updated my production and lab environment already. The update took a moment, but was almost a simple "next.. next" experience coupled with agreeing to license terms. When you run the installer from ISO it will directly detect you are in an update scenario and shows what it found, what needs update. The Domino backup VSS Writer will work unchanged, because it is using the standard VSS interface. Also the restore integration continues tot work exactly the same. Here is a link for what's new in Backup & Replication 12. https://www.veeam.com/whats-new-backup-replication.html There is nothing explicitly important for Domino Backup. But there is a lot of interesting new functionality added. The user interface stays unchanged. But there is a best practices wizard, which helps you to ensure your environment is configured right. Just updated and tested the DNUG Lab environment with the latest Veeam version, too. -- Daniel
Windows Sandbox customization -- this is freaking awesome!Daniel Nashed 27 January 2023 14:20:50As I know I love the Microsoft Sandbox and I am using it for many different purposes. Today I was researching a name resolution problem and coincidently ran into this Windows Sandbox configuration document. https://learn.microsoft.com/en-us/windows/security/threat-protection/windows-sandbox/windows-sandbox-configure-using-wsb-file There are some very convenient pieces you can tune. If you are using the Sandbox for security reasons, you might want to disable copy & paste. But even more interesting is to map directories from your host to the sandbox. This can be read-only or read/write and is much faster and convenient than copying over all the installers on every setup. Another very usefully option is a command you can run when the sandbox starts. I just created a configuration and setup to automatically install and configure ad complete Domino server leveraging Domino One Touch Setup including the JSON templating we added to the Linux container image earlier. To customize the Sandbox you just create a file with the extension .wsb and launch it. You can only have one Sandbox at a time. But you can run multiple configurations ... Here is my current sandbox.wsb file
Nomad Web server connection optionsDaniel Nashed 25 January 2023 09:57:35Nomad Web is a modern HCL client offering in form of a Progressive Web Application (PWA) running in your web browser. In addition to Windows or Mac, it also works on Ubuntu and other Linux distributions! So there is finally a client offering for Linux clients again! The Nomad Web application is installed on a server providing the required files for download. Those files can be stored on a SafeLinx or Domino/Nomad Web server. Your browser downloads the application and runs it locally in your browser. It is basically a cross compile using the Notes basic client code. Special connectivity requirements: WebSockets Nomad Web clients cannot directly connect to your Domino servers using NRPC with a standard TCP/IP connection. Because the client is in a browser it uses modern web technologies to connect to the server. This brings new advantages but also new challenges. Standard HTTPS connections are not a stateful network connection. You can send multiple HTTPS request over the same connection and have a TLS session. But it isn't a TCP/IP network session in the way NRPC would require it. Modern web technology supports so called web-sockets to allow stateful network connections for web applications. Nomad Web tunnels the NRPC session with all it's transactions via WebSockets to Domino. But because Domino itself does not understand WebSocket NRPC connections, you need a server component to translate the network packages. SafeLinx Server Until the Nomad Server was released recently a HCL SafeLinx server was the only network component allowing to bridge the protocols. You don't need to separate license a SafeLinx server. But it is a separate server component, which is not always intuitive to deploy. Therefore the HCL Domino Container Community project provides an easy to configure SafeLinx container --> https://opensource.hcltechsw.com/domino-container/safelinx/ The container is easy to configure specifying just a couple of environment variables instead of using the old fashioned Java admin client application. SafeLinx offers a connection module specially designed for Nomad Web bridging the WebSocket protocol to NRPC. It also allows you to define target Domino servers and the corresponding internet host name. Safelinx handles the TLS connection and tunnels the NRPC connection to the right target host. In addition to a static configuration mapping Domino server names to host names to connect to, SafeLinx can leverage a LDAP connection to a Domino server to map server names dynamically. The SafeLinx container image uses this type of configuration to avoid complex configurations. "Server Name Indication" SafeLinx receives all the traffic over the same HTTPS connection using a single TLS/SSL certificate on a single IP address to dispatch all the traffic acting as a secure reverse proxy. The first NRPC package connecting the client to the server contains the target Domino server name in the first network package. SafeLinx uses this Domino name to map the session to the right Domino server using it's FQDN (lookup in it's own configuration or via LDAP from a Domino server). The resulting stateful WebSocket connection is handled by the SafeLinx server. This means you can use a single SafeLinx server to connect to multiple Domino servers in parallel. SafeLinx ensures the dispatching and handles the stateful WebSocket connection tunneling the NRPC socket connection for you. Nomad Web Server Because not every customer wants to install a separate SafeLinx server, HCL came up with a new server component called "Nomad Web Server". A Nomad Web Server consists mainly on two parts. Let's have a look at the two binary files shipped with the Windows version: nwsp-win.exe Is a Node.js application compiled into a single executable. Node.js provides native WebSocket protocol already and a Node.js application is a low overhead way to implement a way to bride protocols. All connections are going thru this component and will be routed to the target Domino server directly. You can run this component separately from your Domino server and configure all settings in a YAML file. In this case the YAML configuration contains settings for the TLS certificate/key and also mapping configuration for your Domino servers. Very similar to what SafeLinx provides with a static configuration. nnomad.exe The more convenient way is to use this component directly on a Domino server in combination with a Nomad servertask. This server task is started on a Domino server running on the same host. Both components talk to each other using a private TLS connection. The Nomad servertask provides configuration information to connect to the own Domino server it is running on. And also provides connectivity information to other Domino servers in the Domain. This is comparable to what SafeLinx provides using the LDAP lookup. Both components work hand in hand and glue together. You can even leverage existing TLS Credentials in your Domino Certificate store (certstore.nsf). The only configuration needed in this case is the hostname for the certificate specified in a notes.ini setting: NOMAD_WEB_HOST=domino.acme.com This would also work for wild-card certificates like this: NOMAD_WEB_HOST=*.acme.com The TLS Credentials document just must be assigned to the Domino server to have Domino decrypt the private key. And it needs to be a unique match in your certstore.nsf. Using a Load Balancer or Secure Reverse Proxy in front of Nomad Web with SNI Nomad Web Server and SafeLinx work very similar in handling the connection and establishing the session. Both also handle the mapping to the right Domino server in the same way analyzing the first NRPC package. But what if you want to put NGINX or another load balancer in front of your server? Note: I would even advice you to add a robust load balancer like NGINX as a first line of defense in front of any Node.js application like Nomad Server. Because the websocket protocol is HTTPS based, most modern load-balancers and reverse proxies can handle the HTTPS session and even dispatch traffic over server name indication (SNI). This means you can run those HTTPS sessions on the same IP and TLS port 443 you are using for other connections. The only special requirement for WebSockets is a configuration which supports the WebSocket upgrade header. You find a sample configuration for NGINX in the Nomad Web Server documentation referenced below. Conclusion and additional tips This blog post is mainly intended to give you and overview and not an instruction how to setup Nomad Web. Specially for intranet environments the Nomad Server is an easy to install component, which helps you to deploy Nomad Server quite quickly. We added the Nomad Server package to the Domino community image as an add-on, which can be automatically build into the Domino server image --> https://opensource.hcltechsw.com/domino-container/. Still even for intranet deployments I would always add a secure load balancer in front of the Nomad server. SafeLinx in contrast is already a secure load-balancer written in C on a more robust stack. But both options provide you with the required WebSocket connectivity for Nomad Web. References for more details: Nomad Web Documentation https://help.hcltechsw.com/nomad/1.0_web/nomad_web.html Nomad Server Documentation https://help.hcltechsw.com/nomad/1.0_admin/nomadserver_domino.html WebSocket Wikipedia https://en.wikipedia.org/wiki/WebSocket RFC6455 The WebSocket Protocol https://datatracker.ietf.org/doc/html/rfc6455 The WebSocket API (WebSockets) https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
| Recent EntriesLinksArchives
|
![[IBM Lotus Domino]](../lotus-domino.gif){kind=small}
![[Domino on Linux]](../domino-linux.gif){kind=small}
![[Nash!Com]](../nashcom.gif){kind=small}
![[Daniel Nashed]](../daniel-nsh.gif){kind=small}