Domino on Linux/Unix, Troubleshooting, Best Practices, Tips and more ...

Compressed Documents Still not Byte Served in 8.5

Daniel Nashed  6 March 2009 18:50:26

Carl Tyler asked in the previous post if Domino is still not byte-serving attachments which are compressed.
After checking offline with Carl I did some tests with Domino 8.5.

Here is the result. If you have no compression for an attachment it is byte-served. Else there is no byte-serving!

In general on a Domino Server with DAOS all attachments should have the same encoding and compression.
So from storage best practices point of view enabling compression would make a lot of sense.
But this stands in contrast to having the full functionality for iNotes users downloading attachments.

Sounds like the limitation is in the HTTP back-end code that reads the attachment.
And it is not a new limitation but now more visible and important with DAOS and storage optimization.

So if your user community is running iNotes having compression disabled would make sense.
Sadly you cannot disable compression completely because Notes users always have the compress flag enabled when attaching a document...

What do you think? How important is byte-serving for iNotes with compressed data?

Oh one best practice is to attach as ZIP because this is already compressed and will not be stored compressed so byte-serving works.

-- Daniel

HTTP/1.1 200 OK
Server: Lotus-Domino
Date: Tue, 03 Mar 2009 19:41:59 GMT
Last-Modified: Tue, 03 Mar 2009 19:41:57 GMT
Content-Type: application/x-zip
Content-Length: 3344646
Accept-Ranges: bytes


HTTP/1.1 200 OK
Server: Lotus-Domino
Date: Tue, 03 Mar 2009 19:44:35 GMT
Last-Modified: Tue, 03 Mar 2009 19:44:33 GMT
Content-Type: application/pdf
Content-Length: 17984518
Comments

1Bob Congdon  07.03.2009 2:18:47  Compressed Documents Still not Byte Served in 8.5

I wrote the original byte serving support for attachments in Domino R5 (or was it in 4.6?). Wow, that was more than 10 years ago!

As I recall, we were able to stream compressed attachments directly from NSF but there was no easy way to "seek" to a particular offset within the compressed stream. Which meant that there was efficient way to do byte serving of these files. We didn't want to cache the content nor waste processing cycles decompressing content that we weren't going to send down the pipe. The main usage of byte serving at the time was the Acrobat plugin and it would ask for fragments from a PDF file in a completely weird order.

Back then Notes was still using Huffman compression. I think they've added LZ1 since then. Too bad they haven't found a way to fix this.

2Daniel Nashed  08.03.2009 21:53:40  Compressed Documents Still not Byte Served in 8.5

@ Bob, thanks for the feedback! that's what I thought. It would not be impossible to implement it but it is much harder if the object is compressed (same would also be true for encrypted). One way would be to create a temp file but that would generate additional I/O on the server side. Another way would be to use call that detaches in memory with a call-back and and skip the part that is already read. But that might generate also overhead because the server would need to read the beginning of the attachment again. I guess that would be a good balance between not being able to byte-serve and losing the option to compress. Sounds like the database team needs to build some back-end infrastructure for other teams to read. The java issue I posted sound like there are similar limitations in other areas of Domino.

What do you think?

-- Daniel

3Bob Congdon  09.03.2009 4:35:59  Compressed Documents Still not Byte Served in 8.5

Prior to R5 the Domino web server had to detach files before serving them. It had a server-side file cache for this. Also for Notes bitmaps converted to GIF/JPEG. In R5 we got a streaming API to call but no way to seek within the stream. The file cache was removed. I haven't worked on Notes/Domino code since R6 but I'd imagine that things haven't changed since then. LZ1 is stream-oriented as well. Not sure about the Java applet issue.

4Andrew  30.08.2013 16:27:55  Compressed Documents Still not Byte Served in 8.5

Hi!

Thank you for your post! Concerning the statement:

"Here is the result. If you have no compression for an attachment it is byte-served. Else there is no byte-serving!"

I have an issue, that files bigger than ~20Mb anyway became LZ1 compressed, meantime sizes up to ~20 Mb - uploaded not compressed to document.

Any idea where to control it?

In Web Site Document -> Domino Web Engine -> File compression on upload is Disabled.

Thanks in advance! )


  • [IBM Lotus Domino]
  • [Domino on Linux]
  • [Nash!Com]
  • [Daniel Nashed]