Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

 
alt

Daniel Nashed

 

Lotus Script Tip: Lists with Tag with large lists and keys

Daniel Nashed  30 September 2018 05:59:34

I have been looking for a fast way to store and lookup values in an array/list to do some statistics counting entries with a certain key.
My first try was to use an array which I redimed once to have a proper size.
Complile time specified limits are not sufficient high for large arrays in my case.

The routine was needed to evaluate data from ma SpamGeek application (SMTP extension manager) used by a customer internally to control their SMTP traffic (allow only defined hosts to send SMTP mails).

It tuned out that using an array was very inefficient from CPU consumption and also run-time.

One of the customers developers came up with the idea using lists with tags. I did not know the List Tags before which turned out to be very effective.
A test routine showed great performance up to at least 500000 entries which is already quite high!

Here are the interesting details of this approach explained with an example. I took the code a wrote for testing and added some additional code an comments for demonstration purposes.

- You first define a List of the data type you need. In my case a long.
- Than you can add your elements with the key your are using.
- To query if an entry already exists there is the Iselement function
- You can only get elements that are in the list. Else you get a run-time error. So you have to test if it is element first!
- You can loop thru the whole list using Forall.
- The Forall variable is your value that your stored in the list
- The Listatag function is used to return the tag that you used

The run-time is really impressive! And the missing part that I did not know about was the "List Tag".

The implementation in the back-end looks highly optimized and there are simple calls to check the list to get elements or to add new elements.

I highlighted the important parts of the code.

Maybe this could help you in one of your next projects.

-- Daniel

        Dim HostList List As Long
        Dim count As Long
        Dim i As Long
        Dim key As String        
       
        '  build a list where the counter is set to the list lable for testing
       
        For i=1 To 100000
                key = Cstr (i)
                HostList (key) = i        
        Next
       
        ' check if key exists in list and get entry and label!
       
        key = "100000"
        If  Iselement(HostList (key)) Then
                count = HostList (key)
                Messagebox "Found Key: " + key + " with value: "+ Cstr (count)
                HostList (key) = count+123
        Else
                Messagebox  "Not found"
        End If
       
       
        ' loop thru list and check if the key is found unsing the Listtag function
       
        Forall h In HostList
                If (key = Listtag(h)) Then
                        Messagebox "Found Key: " + key + " with new value: "+ Cstr (h)
                End If
        End Forall


Comments

1Lars Berntop-Bos  30.09.2018 21:11:14  Lotus Script Tip: Lists with Tag with large lists and keys

Lists are the best feature of LotusScript.

Combining then with custom classes and you can create pretty powerful software.

One thing to avoid is using lists with Notes 'Product objects' (the Notes... classes).

They quickly become too big.

If I use them to cache stuff I always make custom classes contaning base types or further lists of custom classes. that works fine.

Also, if you need different access keys to the same objects, creating a second list resing the same objects costs you only space for the keys and the pointers, very efficient!

2Lars Berntrop-Bos  30.09.2018 21:12:06  Lotus Script Tip: Lists with Tag with large lists and keys

oops typo: resing should be reusing

3Tinus Riyanto  01.10.2018 0:19:21  Lotus Script Tip: Lists with Tag with large lists and keys

Nice to know that it will work for up to 500 thousand elements. I am always wondering if there is any upper bound limit for list since array seemed to be maxed out around 65000

4Lars Berntrop-Bos  01.10.2018 7:58:18  Lotus Script Tip: Lists with Tag with large lists and keys

Bill Buchan used to have a set of tips on his website, but those seem to have lapsed... Wayback machine to the rescue:

https://web.archive.org/web/20090319033610/{ Link }

He documents having used lists with millions of items. Just make sure those contain relatively simple objects. The product objects (NotesItem and the like) are complex little beasties with sometimes easy to overlook interdependencies, which you avoid if you code your own classes.

Further reading:

Object oriented LotusScript

https://www.slideshare.net/billbuchan/lotusphere-2007-bp301-advanced-object-oriented-programming-for-lotusscript

I've used lists of objects to make an agent from taking multiple hours to complete to taking less then 5 minutes. While doing more.

I learned a lot from Bill Buchan, Julian Robichaux, Tim Tripcony, Bob Balaban (his OLD book Programming Domino 4.6 With Java taught me a lot about the foundations of Domino, still useful to know), and others like Jake Howlett, Karl-Henry Martinsson, Devin Olson.

5Lars Berntrop-Bos  01.10.2018 9:58:18  Option Compare NoCase

tip: be careful with Option Compare NoCase

If you use this, the keys (ListTag) for List elements will allways be stored and returned in UPPERCASE. This tripped me up once, where I was expecting that if I used ListTag for a List element I would get back the lowercase key which was used to set the List element.

So now I usually wrap a list inside an access class which makes sure to lookup values in the proper way.

Links

    Archives


    • [HCL Domino]
    • [Domino on Linux]
    • [Nash!Com]
    • [Daniel Nashed]