Windows Sandbox customization -- this is freaking awesome!

Daniel Nashed  27 January 2023 14:20:50

As I know I love the Microsoft Sandbox and I am using it for many different purposes.
Today I was researching a name resolution problem and coincidently ran into this Windows Sandbox configuration document.

https://learn.microsoft.com/en-us/windows/security/threat-protection/windows-sandbox/windows-sandbox-configure-using-wsb-file

There are some very convenient pieces you can tune.

If you are using the Sandbox for security reasons, you might want to disable copy & paste.

But even more interesting is to map directories from your host to the sandbox.
This can be read-only or read/write and is much faster and convenient than copying over all the installers on every setup.

Another very usefully option is a command you can run when the sandbox starts.

I just created a configuration and setup to automatically install and configure ad complete Domino server leveraging Domino One Touch Setup including the JSON templating we added to the Linux container image earlier.

To customize the Sandbox you just create a file with the extension .wsb and launch it.
You can only have one Sandbox at a time. But you can run multiple configurations ...

Here is my current sandbox.wsb file

• Comments [0]

Nomad Web server connection options

Daniel Nashed  25 January 2023 09:57:35

Nomad Web is a modern HCL client offering in form of a Progressive Web Application (PWA) running in your web browser.
In addition to Windows or Mac, it also works on Ubuntu and other Linux distributions! So there is finally a client offering for Linux clients again!
The Nomad Web application is installed on a server providing the required files for download.
Those files can be stored on a SafeLinx or Domino/Nomad Web server.

Your browser downloads the application and runs it locally in your browser.
It is basically a cross compile using the Notes basic client code.

Special connectivity requirements: WebSockets

Nomad Web clients cannot directly connect to your Domino servers using NRPC with a standard TCP/IP connection.
Because the client is in a browser it uses modern web technologies to connect to the server.
This brings new advantages but also new challenges.
Standard HTTPS connections are not a stateful network connection.
You can send multiple HTTPS request over the same connection and have a TLS session.
But it isn't a TCP/IP network session in the way NRPC would require it.

Modern web technology supports so called web-sockets to allow stateful network connections for web applications.

Nomad Web tunnels the NRPC session with all it's transactions via WebSockets to Domino.
But because Domino itself does not understand WebSocket NRPC connections, you need a server component to translate the network packages.


SafeLinx Server


Until the Nomad Server was released recently a HCL SafeLinx server was the only network component allowing to bridge the protocols.
You don't need to separate license a SafeLinx server. But it is a separate server component, which is not always intuitive to deploy.

Therefore the HCL Domino Container Community project provides an easy to configure SafeLinx container --> https://opensource.hcltechsw.com/domino-container/safelinx/

The container is easy to configure specifying just a couple of environment variables instead of using the old fashioned Java admin client application.
SafeLinx offers a connection module specially designed for Nomad Web bridging the WebSocket protocol to NRPC.

It also allows you to define target Domino servers and the corresponding internet host name.
Safelinx handles the TLS connection and tunnels the NRPC connection to the right target host.

In addition to a static configuration mapping Domino server names to host names to connect to, SafeLinx can leverage a LDAP connection to a Domino server to map server names dynamically.
The SafeLinx container image uses this type of configuration to avoid complex configurations.


"Server Name Indication"

SafeLinx receives all the traffic over the same HTTPS connection using a single TLS/SSL certificate on a single IP address to dispatch all the traffic acting as a secure reverse proxy.

The first NRPC package connecting the client to the server contains the target Domino server name in the first network package.
SafeLinx uses this Domino name to map the session to the right Domino server using it's FQDN (lookup in it's own configuration or via LDAP from a Domino server).

The resulting stateful WebSocket connection is handled by the SafeLinx server.

This means you can use a single SafeLinx server to connect to multiple Domino servers in parallel.
SafeLinx ensures the dispatching and handles the stateful WebSocket connection tunneling the NRPC socket connection for you.


Nomad Web Server


Because not every customer wants to install a separate SafeLinx server, HCL came up with a new server component called "Nomad Web Server".

A Nomad Web Server consists mainly on two parts.
Let's have a look at the two binary files shipped with the Windows version:

nwsp-win.exe

Is a Node.js application compiled into a single executable. Node.js provides native WebSocket protocol already and a Node.js application is a low overhead way to implement a way to bride protocols.
All connections are going thru this component and will be routed to the target Domino server directly.

You can run this component separately from your Domino server and configure all settings in a YAML file.
In this case the YAML configuration contains settings for the TLS certificate/key and also mapping configuration for your Domino servers.
Very similar to what SafeLinx provides with it static configuration.


nnomad.exe

The more convenient way is to use this component directly on a Domino server in combination with a Nomad servertask.
This server task is started on a Domino server running on the same host.
Both components talk to each other using a private TLS connection.

The Nomad servertask provides configuration information to connect to the own Domino server it is running on.
And also provides connectivity information to other Domino servers in the Domain.

This is comparable to what SafeLinx provides using the LDAP lookup.

Both components work hand in hand and glue together. You can even leverage existing TLS Credentials in your Domino Certificate store (certstore.nsf).

The only configuration needed in this case is the hostname for the certificate specified in a notes.ini setting:

NOMAD_WEB_HOST=domino.acme.com

This would also work for wild-card certificates like this:

NOMAD_WEB_HOST=*.acme.com


The TLS Credentials document just must be assigned to the Domino server to have Domino decrypt the private key.
And it needs to be a unique match in your certstore.nsf.


Using a Load Balancer or Secure Reverse Proxy in front of Nomad Web with SNI


Nomad Web Server and SafeLinx work very similar in handling the connection and establishing the session.
Both also handle the mapping to the right Domino server in the same way analyzing the first NRPC package.


But what if you want to put NGINX or another load balancer in front of your server?
Note: I would even advice you to add a robust load balancer like NGINX as a first line of defense in front of any Node.js application like Nomad Server.

Because the websocket protocol is HTTPS based, most modern load-balancers and reverse proxies can handle the HTTPS session and even dispatch traffic over server name indication (SNI).

This means you can run those HTTPS sessions on the same IP and TLS port 443 you are using for other connections.

The only special requirement for WebSockets is a configuration which supports the WebSocket upgrade header.
You find a sample configuration for NGINX in the Nomad Web Server documentation referenced below.


Conclusion and additional tips


This blog post is mainly intended to give you and overview and not an instruction how to setup Nomad Web.
Specially for intranet environments the Nomad Server is an easy to install component, which helps you to deploy Nomad Server quite quickly.

We added the Nomad Server package to the Domino community image as an add-on, which can be automatically build into the Domino server image --> https://opensource.hcltechsw.com/domino-container/.

Still even for intranet deployments I would always add a secure load balancer in front of the Nomad server.
SafeLinx in contrast is already a secure load-balancer written in C on a more robust stack.

But both options provide you with the required WebSocket connectivity for Nomad Web.



References for more details:

Nomad Web Documentation
https://help.hcltechsw.com/nomad/1.0_web/nomad_web.html

Nomad Server Documentation
https://help.hcltechsw.com/nomad/1.0_admin/nomadserver_domino.html

WebSocket Wikipedia
https://en.wikipedia.org/wiki/WebSocket

RFC6455 The WebSocket Protocol
https://datatracker.ietf.org/doc/html/rfc6455

The WebSocket API (WebSockets)
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API

• Comments [5]

Domino pass-through connections - Anyone using it still large scale and why?

Daniel Nashed  25 January 2023 08:32:11

Domino pass-through (PT) connections are coming from another century where Domino servers had one or more modems to let a user or another server to connect to one server instead of having a need to dial into every server separately.

The connection type was never designed for todays large scale deployments with many concurrent users and a lot of traffic.
It's still used by companies having a PT server in their DMZ allowing users to connect to distinct servers in their intranet.


You can tightly control PT connections by specifying who can

• Route through a server via PT
• Access a server via PT

Using this type of connection the user (or server) session is established with the PT server first.
The PT server opens a new NRPC session to the destination server on the user's behalf.

So the PT server is actively involved in the communication not just passing network packages around.

This also means you need an authenticated session first on the PT server before the server establishes a session on the user's behalf on the destination server.

In result the PT server has some extra load and processing of the NRPC transaction data.
A PT server connection will be always slower than a direct connection. And in earlier releases I have seem some scalability limitations due to high load on the PT server.


Unauthenticated light transactions not supported via PT


But there also so called "light transactions" for new mail notifications, getting server statistics, configuring a client and a couple more, which are unauthenticated and not creating a full Notes session.

Those type of transactions are not supported via PT. Luckily some of those transactions like getting server statistics and polling the mail delivery sequence number for new mail notification have a full and light transaction.
This means that once you have a PT session those transactions will work -- But they can't be the initial transaction to a PT server.


Can' setup a new user via PT connection

In turn setting up a new user will not work over a PT connection, because this transaction is not authenticated or a full session yet.
Also downloading a Notes.ID via PT will not work!
So you would need to setup your user while they are in intranet and can connect directly to the server.


Different location and connection profiles needed

To make PT work on the road you also need a different set of connection to your server while you are on the road.
This usually means you need to manage separate connection documents.
Relying on the PT server setting in your location document isn't sufficient. The client would try to connect directly, time out and will eventually use the PT connection.


Known issues with encrypted sessions with Domino 11 and Domino 12 (including 12.0.2)

We had a customer case recently where where customer is using PT servers in their DMZ to let their whole sales team replicate their CRM over PT.

PT in Domino 11/12 can have an issue with encrypted connections replicating larger databases.
HCL has reproduced the case and there isn't a solution yet.

We had to downgrade the PT server to the latest Domino 10.0.1 FP to get the connection stable again.

User sessions dropped during replication of larger databases.

The error shown on the replicator is:  Network error: buffer was too small


Modern way to connect external users to internal servers

PT users used to be a very good way to connect to internal resources just for Notes and Domino.
In today's environments users usually need to have access to more than just Notes resources.

In modern IT environments customers usually leverage VPNs to connect their external users to intranet resources.

HCL SafeLinx for example would be a solution terminating the network connection in your DMZ and allows to route this traffic to the right servers in your intranet.
And it also offers reverse proxy functionality for other applications in parallel where VPN is required.

Of course there a many other VPN solutions on the market, which are all working on the same principle providing a tunneled connection to explicitly allowed internal resources.

Most of the time the connection is authenticated with strong authentication like certificate based and two factor authentication.


PT is still a valid approach for some special requirements

For a smaller environment PT connections could still be a valid approach to just use Notes connections in a secure way.
When you enable password checking on your PT server, the Notes.ID in combination with the password is a kind of two factor authentication as well.


Why are you using PT connections today?

I would be interested to hear from you if you are still using PT connections today.

• Is this for securing your connection to the Domino server via DMZ?
• Or are you a MSP or hosting provider saving public IP addresses and letting customers connect via a single PT to multiple servers?
  
  The latter one isn't what you should do today, based on what I explained in this article.

• Comments [6]

Domino Server controller listening on all IPs

Daniel Nashed  24 January 2023 18:18:39

By default the server controller detects the host name and tries to bind to the host name only.
In most cases there is no need to bind to the host name, if you are just running with one partition (which should be the case for most of use in today's VM deployment scenarios).

Specially inside a Docker/K8s container name resolution can be a challenge.

I have used this parameter before to explicitly bind to the loop back address 127.0.0.1 for security reasons.

But this also works the other way around. The following parameter lets the server controller bind to all IP addresses on your host!


TCPIP_ControllerTcpIpAddress=0.0.0.0:2050


Please note "TCPIP" is the default port name, which might be different in some configurations.

I came up with his in a private discussion where an admin move Domino to a different box, where the server did have a different host-name.
This will be in my deployment best practices from now on and specially interesting for running Windows in side a container.

Special note:

If you specify 0.0.0.0, you can't connect locally any more, because 0.0.0.0 does not match your local IP address you are connecting with.
So if you need a local connection, specify the IP address of the machine that most likely stays permanently.

In case you are using DHCP with changing IPs this is more challenging. You need to make sure the name resolution works with DHCP and use the DNS name instead.

But this tip is still very useful for containers where you have no local connection!

If you only care to have a local console, 127.0.0.1 should be always the safest bet.
It will just always work, because the loopback IP and network interface is always there.
And it is more safe anyhow!


-- Daniel

• Comments [1]

Moving documents to a different folder with one click

Daniel Nashed  24 January 2023 10:36:49

The Notes @Formula language is one of the underestimated gems in Notes in general.
There are so many powerful formulas and commands, which could be even used by a power user without a design client.

I just needed a simple way to move invoices to a dedicated folder to have them available for taxes etc.

The following command will move documents from the existing folder to the "Invoices" folder.
You have to create the folder once. But then it's just a single smart icon click to move one of more documents to a different folder.
This also works from within a document.

You could add followup flags as well if you add the command to the same action button before the command:

@Command( [FolderDocuments] ; "Invoices" ; "1" )

In my case I just want a folder with all my incoming invoices.
Next step is to sort them by date and detach the attachments to disk :-)

There are so many very simple to use helpful @formulas.
Maybe this inspires you to find your own productivity improving commands...

-- Daniel

• Comments [2]

Cleaning up your disks -- WSL and Docker

Daniel Nashed  23 January 2023 11:24:43

I need some space on disk c:\ to install the latest Visual Studio 2022 compiler.
And the challenge was to find out what might not be needed any more and takes most of the space.

Finding large disk allocations

A very convenient way to find large diskspace allocations is "ncdu" in my WSL Ubuntu instance.
I just ran (needs to be installed on Ubuntu / your Linux distribution running in WSL)

ncdu /mnt/c

It turned out that the biggest allocation was my Ubuntu virtual hard disk.
WSL adds space automatically, but does not release storage.
But if you have hyper-v tools installed, you can down

/mnt/c/Users/daniel.nashed/AppData/Local/Packages

46.8 GiB [##########] /CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc


There is a simple Powershell command you can use to optimize the virtual hard disk file.

First shut down WSL:

wsl --shutdown

Then run the following Powershell command (you have to change the user name and see if you have the same file).
Taking a backup before compacting would be also a good idea.

Optimize-VHD -Path  C:\Users\daniel.nashed\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState\ext4.vhdx  -Mode Full


In my case the VHDX file was less than half the size afterwards.


Removing old Docker Desktop files

I had installed Docker Desktop a while ago. The uninstaller does not remove all the data left over.
You have to remove the data manually.
But the permission to the data is even not granted to an administrator account.

I found the following commands, that helped to adjust the permissions and let me delete the files:


takeown.exe /F "C:\ProgramData\Docker" /R /A /D Y
icacls "C:\ProgramData\Docker" /T /C /grant Administrators:F
rmdir /s /q "C:\ProgramData\Docker"


Both operations in combination took me from 28 GB free to 69 GB free on disk C:

Maybe this helps someone else cleaning up disk drive C: ...

• Comments [0]

Windows Sandbox - A feature you should know

Daniel Nashed  23 January 2023 03:09:37

The sandbox can be a very useful tool for many different situations.
I am often using it for Domino server or client install tests.

But there are many other use cases including training environments etc.
It's a full throw away sandbox environment recreated every time you start it.

The only limitation is that you can't reboot the Windows for example after a software update.
But even installing the Windows re-distributable run-time package does not require a boot.
Most applications like Notes/Domino install it on their own.

I needed it to test my own applications. But there is an easy way to download and silent install it:

curl -LO https://aka.ms/vs/17/release/vc_redist.x64.exe
vc_redist.x64.exe /quiet

Also installing software client the AWS CLI can be automated:

msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi /qn

Once you are done just close the Window, the sandbox is reset and you can start from scratch.
I am using the Sandbox all the time, but I got a surprised thank you from others mentioning it.

In case you don't know it, this is definitively a Windows feature to look at.

Update 24.01/2023:

As Lars pointed out a new version of Windows 11 has support for restarts from inside the sandbox.
I read about it in a Windows blog comment without details. Thanks Lars!

Here is the link Lars shared in a comment:
https://learn.microsoft.com/en-us/windows/security/threat-protection/windows-sandbox/windows-sandbox-overview

The functionality is available starting with  Windows 11 Build 22509.
My other notebook is on Win11 22H2 (a later build).
Just tested it by adding some files and restarted from inside the sandbox. Works like a charm.



-- Daniel





       

• Comments [2]

NGINX TCP Stream with SNI support. More than helpful for lab environments

Daniel Nashed  21 January 2023 15:11:41

In production you usually want centralized certificate handling and off-loading TLS termination to a load-balancer.
I posted scripts to have NGINX realod certs automatically from Domino CertMgr via HTTPS to leverage Domino's Let's Encrypt implementation.
But sometimes you really want all your servers directly exposed over TLS.

For example in a lab environment with limited resources and only one IP, you might want to still have each of the hosts expose their services on their own.
I did know Traefik (https://traefik.io/) has a build-in way to dispatch TLS passthru traffic.
But I just discovered end of last year, NIGNX also has a module to pre-read TLS SNI information to dispatch TCP traffic (http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html).
This becomes very convenient, when you want to expose multiple services over the same TCP port.


DNUG Lab as an example (Domino HTTPS and SafeLinx on both on port 443)

In my example we are am running Domino HTTPS and a Safelinx server on port 443.
And we might add more services in future. And could even introduce a Nomad server in parallel with a different host name on port 443.
But they could be also separate containers or a native application running on a Linux host.

In our case we are also running NGINX in a container.
This approach allows you to run the latest NGINX container, even if you Linux distribution has not included it yet.
Like in our case Domino is exposed on port 444 and the SafeLinx server is exposed on port 445 via Docker.


Combinations are just limited by your imaginations

Having NGINX dispatch all the traffic you can use any number of services on port 443.
Other ports could run on the same NGINX instance.
But you could also have a NGINX instance in TCP Stream node also dispatch traffic into another NGINX instance off-loading TLS for other services or redirecting traffic.

-- Daniel

Example start for a NGINX Docker container

docker run -it -d --name nginx --network host -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx


Example configuration for your nginx.conf file

user nginx;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
  worker_connections 768;
}


stream {
  map $ssl_preread_server_name $name {
    nomad.lab.dnug.eu   nomad;
    default             domino;
  }

  upstream domino {
    server 127.0.0.1:444;
  }

  upstream nomad {
    server 127.0.0.1:445;
  }

  server {
    listen      443;
    proxy_pass  $name;
    ssl_preread on;
  }
}

• Comments [0]

How to get the error message for a Notes error code

Daniel Nashed  12 December 2022 12:09:24

Sometimes scripts or Domino server commands only return an error code and you would like to know the error message.

There is an easy way to get the error message back from a server command.

"show message [module]

In most cases you don't need server tasks specific error messages and just use the decimal error code.

Here is an example:

show message 404

[4E10:000A-670C] Entry not found in index

By the way, It's probably not a coincident Lotus at that time did choose the error code "404" for "Entry not found in index".

Another interesting error code is 1. The resource string is used to specify the release and also the build time of the release.
This is the information shown in the server console and other places and a reliable way to identify the Domino version.

show message 1

[4E10:000A-670C] Release 12.0.2|November 03, 2022            

• Comments [1]

HCL Domino Community Image German Language Pack Support

Daniel Nashed  24 November 2022 23:23:44

Personally I am not a big fan of the language packs.
I keep the server in English and just add the templates I need on a single server if needed.

But there are some customers who really want also German system databases.

Each Language Pack for each Domino release will require separate properties, LP ini files and a lot of testing.
Therefore additional language packs are added on request.

I would still encourage everyone to stay with an English container image.

This was still a good step to understand the LP silent installer.

The installation is another install option for the Domino base image.
 You just specify the language "de" on command-line. The installer will take care replacing the files and the templates are packaged into the install data tar automatically.

-- Daniel

./build.sh domino -domlp=de

12.0.2              [OK] Domino_12.0.2_Linux_English.tar
de-12.0.2           [OK] Domino_12.0.2_SLP_German.tar

--------------------------------------------------------------------------------
Installing Language Pack de-12.0.2
--------------------------------------------------------------------------------

Running Domino Language Pack Silent Install -- This takes a while ...

Language Pack installed successfully

• Comments [0]