Daniel Nashed's Blog

Leveraging caching proxies and repository mirrors for container builds

I looked into Squid proxy support for container image builds earlier to support customers building container images in corporate environments.
But proxies can provide more than just bridging external connections thru a DMZ.

Caching Proxy

A caching proxy can reduce the build time and also reduced internet connection load.
I have setup a Squid proxy with special proxy rules and it is very effective in repeated container builds -- even for a single environment.

Repository Mirror
In parallel I looked into building my own Ubuntu mirror just for the current release and the selected target platform trying to filter out as much as I can (like AWS kernel support).
The main part isn't that big. But in a corporate environment you can't avoid syncing the universe repository -- which is quite big and has a lot of changes.
To be completely independent, building a full mirror might be still needed.

What works best?

But just for optimizing network traffic using Squid might be the better choice.
In my case the proxy even did keep a lot of the packages in RAM and I am using it from multiple build machines locally.
It's more sustainable and also saves build time.

Ubuntu is using HTTP repositories which make caching simple.
Other distributions might be more difficult to cache.

I am still working out some details and will post my configuration on GitHub.
If your provider provides a repository mirror like Hetzner, using the mirror makes a lot of sense.

There is a new repository in the works to provide those type of details
But I am interested to hear how you are using caching proxies and local mirrors.
---

Hetzner information

https://docs.hetzner.com/robot/dedicated-server/operating-systems/hetzner-package-mirror/

Local Ubuntu Mirror working just for what I need for my local container build

--- /local/ubuntu-mirror/apt-mirror/pool ---
                                       /..
  103.4 GiB [########################] /universe
   30.7 GiB [#######                 ] /main
    5.5 GiB [#                       ] /restricted

Squid cache hit example

1763904244.862     11 192.168.96.58 TCP_HIT/200 174863 GET http://archive.ubuntu.com/ubuntu/pool/main/u/unzip/unzip_6.0-28ubuntu4.1_amd64.deb - HIER_NONE/- application/vnd.debian.binary-package
1763904247.543      0 192.168.96.58 TCP_MEM_HIT/200 56928 GET http://archive.ubuntu.com/ubuntu/pool/main/u/ucf/ucf_3.0043%2bnmu1_all.deb - HIER_NONE/- application/vnd.debian.binary-package

• Comments [0]