Roundup Tracker

Images are some of the larger items we have to serve from Roundup. While detectors could be written to convert file formats and offer up a more efficient file, this has a number of drawbacks:

  1. storage space for the new formats. Often you need to retain the original file as well for auditing or other purposes.
  2. wasted work for converting files that might never be viewed
  3. complexity of detecting the right file to serve

using a service that converts image files on demand to the best format and size mitigates many of these issues.

The imgproxy server is written in go and is available as a docker container. By hosting imgproxy so it has access to the file storage tree of a Roundup Tracker you can use it to convert image files on the fly. This can reduce the shipped bytes by up to 90%. It will refuse to serve a image file in a format it doesn't recognize. Also unless the raw processing directive is used, it will refuse to serve any unrecognized file.

Four things need to be in place to implement this:

  1. A running imgproxy server serving files from Roundup's file storage
  2. A Roundup templating extension that adds imgproxy_url() to the utils

    • class. This function will produces a URL pointing to the imgproxy server.
  3. A suitable configuration for the imgproxy_url() function

  4. A change to a template to call utils.imgproxy_url() to replace the usual

    • file/download_url method.

Running imgproxy

You get a copy of the imgproxy docker container using by running docker pull darthsim/imgproxy:latest.

Since it is a 12 factor app, it has an extensive list of environment variables used to configure it. The following script starts it setting a number of configuration variables:

cat > imgproxy.env <<END_ENV
# security key hex-encoded - see security section at the end of this page
IMGPROXY_KEY=736563726574
# salt hex encoded  - see security section at the end of this page
IMGPROXY_SALT=68656C6C6F

# location under the site where proxy is located
IMGPROXY_PATH_PREFIX=/trackers/tracker1/imgproxy

# limit to only serving files from local filesystem
IMGPROXY_ALLOWED_SOURCES=local://

# allow and prefer webp if supported
IMGPROXY_ENABLE_WEBP_DETECTION=true
IMGPROXY_ENFORCE_WEBP=true

# allow and prefer AVIF if supported
IMGPROXY_ENABLE_AVIF_DETECTION=true
IMGPROXY_ENFORCE_AVIF=true

# Try to prevent image caching. Cache headers includes
# public. But we don't want images served to be cached in
# public servers as people can hit the cache and get access
# to images they should not access. We would like that to be
# private so that the browser can cache it, but.....
IMGPROXY_TTL=1

# root directory to serve images from
IMGPROXY_LOCAL_FILESYSTEM_ROOT=/images

# define a directive called mark to watermark images
IMGPROXY_PRESETS=mark=watermark:0.3:sowe

# base64 encoded image to use as watermark
IMGPROXY_WATERMARK_DATA=iVBORw0KGgoAAAANSUhEUgAAAGcAAABCAQMAAABw7lo/AAA[...]
END_ENV

background="-d"
#background=""

docker run \
       ${background} \
       --rm \
       --name imgproxy \
       --env-file imgproxy.env \
       -v /path/to/tracker/home/db/files/file:/images \
       -p 9080:8080 \
       darthsim/imgproxy:latest

rm -f imgproxy.env

This starts the imgproxy web server on port 9080. It is strongly suggested to reverse proxy to imgproxy the same way you would to roundup-server. In this case, it is available at https://.../trackers/tracker1/imgproxy as set by IMGPROXY_PATH_PREFIX.

Adding imgproxy.py to define utils.imgproxy_url()

Place a copy of imgproxy.py (download) in your tracker's extensions directory.

The code looks like:

   1 """A module to redirect image file loading to imgproxy for more
   2 efficient and smaller image formats.
   3 
   4 This module defines a util.imgproxy_url class that can be
   5 used in place of download_url in img tags.
   6 
   7 E.G. replace
   8 
   9     <img tal:attributes="src file/download_url; alt file/name"
  10         loading="lazy">
  11 
  12 with:
  13 
  14     <img tal:attributes="src
  15        python:utils.imgproxy_url(file, ['rs:fit:400:300',
  16                                         'preset:mark' ]);
  17        alt file/name" loading="lazy">
  18 
  19 to load the image at a size of 400x300 and have a watermark
  20 placed on it.
  21 
  22 It is controlled by the following settings in
  23 extensions/config.ini:
  24 
  25 -----
  26 [imgproxy]
  27 
  28 # URL for imgproxy serving up your tracker's images.
  29 # Must have trailing /.
  30 # If not set, regular download_url for the attached file is generated.
  31 url = https://example.net/tracker/imgproxy/
  32 
  33 # Hex encoded random key used for signature hash.
  34 # Can be stored in a file. Use file://filename relative to
  35 # extensions directory.
  36 # Shell command to generate suitable string is:
  37 #   echo $(xxd -g 2 -l 64 -p /dev/random | tr -d '\n')
  38 #
  39 # If not set, unsigned requests are generated for imgproxy.
  40 # if Accepted by imgproxy, this allows attached files to
  41 # be accessed by unauthorized clients.
  42 key = 736563726574
  43 
  44 # Hex encoded random salt used for signature hash.
  45 # See key documentation for details.
  46 salt = file://../secrets/imgproxy.salt
  47 
  48 # Integer number of seconds that the image url is cached/valid
  49 # (default 5 seconds). Must be > 0. If you are lazy loading
  50 # the images on a page with a lot of images, you need to
  51 # increase this time otherwise images will fail to load.
  52 #ttl = 10
  53 
  54 # Add a watermark to images. Assumes IMGPROXY_WATERMARK_DATA
  55 # or other watermark source is defined in imgproxy environment
  56 # (default False)
  57 #watermark = on
  58 ------
  59 
  60 See: https://imgproxy.net/
  61      https://github.com/imgproxy/imgproxy
  62 """
  63 
  64 import binascii
  65 import base64
  66 import hashlib
  67 import hmac
  68 import logging
  69 import time
  70 
  71 from roundup.anypy.strings import b2s, bs2b
  72 from roundup.anypy.urllib_ import quote
  73 from roundup.configuration import BooleanOption, \
  74     IntegerNumberOption, \
  75     InvalidOptionError, \
  76     OptionValueError, \
  77     SecretOption, \
  78     WebUrlOption
  79 
  80 try:
  81     fromhex = bytes.fromhex
  82 except AttributeError:
  83     fromhex = binascii.unhexlify
  84 
  85 logger = logging.getLogger('extension')
  86 
  87 
  88 def make_signature(processing_url, key_hex, salt_hex):
  89     """Generate the signature for the processing_url with key/salt
  90 
  91        Used to limit access to the image url passed to
  92        imgproxy by signing the processing_url.
  93     """
  94     key = fromhex(key_hex)
  95     salt = fromhex(salt_hex)
  96 
  97     path = bs2b(processing_url)
  98     digest = hmac.new(key, msg=salt+path,
  99                       digestmod=hashlib.sha256).digest()
 100     return base64.urlsafe_b64encode(digest).rstrip(b"=")
 101 
 102 
 103 def get_setting(config, setting, default=""):
 104     """Get a setting from a detector or extension url with
 105        default value if not defined.
 106     """
 107     try:
 108         return config[setting]
 109     except InvalidOptionError:
 110         return default
 111 
 112 
 113 def imgproxy_url(file_obj, render_options=None):
 114     """Take a file object (with a download_url method) and an
 115        optional list of imgproxy processing
 116        options. Specified options take precedence over the
 117        ones added by this function (filename, watermark, expires).
 118 
 119        Return a url to the imgproxy image server that has
 120        access to the tracker's db/files/file directory
 121        tree. Processing directives and other doc at:
 122        https://docs.imgproxy.net/.
 123     """
 124 
 125     ext_config = file_obj._db.config.ext
 126 
 127     url_prefix = get_setting(ext_config, 'IMGPROXY_URL', None)
 128 
 129     if url_prefix is None:
 130         # can't use imgproxy without the url.
 131         # this is a safety since it should never happen
 132         # as this function is not called if url not defined.
 133         return file_obj.download_url()
 134 
 135     key = get_setting(ext_config, 'IMGPROXY_KEY', None)
 136     salt = get_setting(ext_config, 'IMGPROXY_SALT', None)
 137     ttl = get_setting(ext_config, 'IMGPROXY_TTL', 5)
 138     watermark = get_setting(ext_config, 'IMGPROXY_WATERMARK', False)
 139 
 140     if not render_options:
 141         # basic resize 400x300 no shrink or expand
 142         render_options = ["rs:fill:400:300:0:0"]
 143 
 144     # Insert default/configured options at start of
 145     # render_options. Options later in the URL take
 146     # precedence. So watermark can be overridden by
 147     # 'watermark:0' set in render_options passed to
 148     # imgproxy_url().
 149 
 150     # Set filename to basename of image file.
 151     # Suffix is added by imgproxy to match image format.
 152     name = file_obj._klass.get(file_obj._nodeid, 'name')
 153     render_options.insert(0, "filename:%s" % quote(name[:name.rindex('.')]))
 154 
 155     # Add the watermark. Assumes IMGPROXY_WATERMARK_DATA or
 156     # other watermark definition is defined in imgproxy
 157     # environment.
 158     if watermark:
 159         render_options.insert(0, "watermark:0.3:sowe")
 160 
 161     if key and salt:
 162         # Set URL expiration time.
 163         # If there is no signature hash, any user can change the
 164         # expire time and resend the URL. So don't bother
 165         # setting it.
 166         render_options.insert(0, "expires:%i" % (time.time() + ttl))
 167 
 168     rendering = "/".join(render_options)
 169     local_url = "local:///%s" % file_obj._db.subdirFilename(
 170         file_obj._klass.classname,
 171         file_obj._nodeid)
 172 
 173     processing_url = "/%(rendering)s/plain/%(local_url)s" % {
 174         "rendering": rendering, "local_url": local_url, }
 175 
 176     if key and salt:
 177         signature = b2s(make_signature(processing_url, key, salt))
 178     else:
 179         signature = "not_signed"
 180 
 181     imgproxy_url = "%(url_prefix)s%(signature)s%(processing_url)s" % {
 182         "url_prefix": url_prefix, "signature": signature,
 183         "processing_url": processing_url}
 184 
 185     return imgproxy_url
 186 
 187 
 188 def init(instance):
 189     # verify options values meet standards.
 190     try:
 191         instance.config.ext.update_option(
 192             'IMGPROXY_URL', WebUrlOption,
 193             description="Url for imgproxy server.")
 194     except InvalidOptionError:
 195         # It's not set, that's ok. Just call download_url()
 196         # and skip everything else in this module.
 197         instance.registerUtil(
 198             'imgproxy_url',
 199             lambda file_obj, _ignore=None: file_obj.download_url())
 200         return
 201 
 202     try:
 203         option_name = 'IMGPROXY_KEY'
 204         instance.config.ext.update_option(
 205             option_name, SecretOption,
 206             description="Random HMAC key, hex encoded.")
 207 
 208         try:
 209             fromhex(instance.config.ext[option_name])
 210         except ValueError:
 211             raise OptionValueError(
 212                 instance.config.ext.options[option_name],
 213                 instance.config.ext[option_name],
 214                 "in extensions/config.ini. Is not a valid hex encoded string."
 215             )
 216     except InvalidOptionError:
 217         # It's not set, that's ok.
 218         pass
 219 
 220     try:
 221         option_name = 'IMGPROXY_SALT'
 222         instance.config.ext.update_option(
 223             option_name, SecretOption,
 224             description="Random salt used with HMAC key, hex encoded.")
 225 
 226         try:
 227             fromhex(instance.config.ext[option_name])
 228         except ValueError:
 229             raise OptionValueError(
 230                 instance.config.ext.options[option_name],
 231                 instance.config.ext[option_name],
 232                 "in extensions/config.ini. Is not a valid hex encoded string."
 233             )
 234     except InvalidOptionError:
 235         # It's not set, that's ok.
 236         pass
 237 
 238     try:
 239         instance.config.ext.update_option(
 240             'IMGPROXY_TTL', IntegerNumberOption, default=5,
 241             description="Lifetime in integer seconds of image URL.")
 242     except InvalidOptionError:
 243         # It's not set, that's ok. Default is 5 seconds.
 244         pass
 245 
 246     try:
 247         instance.config.ext.update_option(
 248             'IMGPROXY_WATERMARK', BooleanOption, default=False,
 249             description="Print watermark on images.")
 250     except InvalidOptionError:
 251         # It's not set, no watermark.
 252         pass
 253 
 254     instance.registerUtil('imgproxy_url', imgproxy_url)
 255 
 256 
 257 if __name__ == '__main__':
 258     """ Test make_signature() using imgproxy example signature
 259         settings.
 260     """
 261     url = ('/rs:fill:300:400:0/g:sm/aHR0cDovL2V4YW1w/bGUuY29tL2l'
 262            'tYWdl/cy9jdXJpb3NpdHku/anBn.png')
 263     print(url)
 264 
 265     s = make_signature(url, '736563726574', '68656C6C6F')
 266 
 267     print(s)
 268 
 269     assert b2s(s) == "oKfUtW34Dvo2BGQehJFR4Nr0_rIjOtdtzJ3QFsUcXH8"
 270 
 271     print("Test passed, no errors")
download imgproxy.py

Append settings to extensions/config.ini

Append the following lines to config.ini located in your extensions directory. Create the file if you don't have one.

[imgproxy]

# URL for imgproxy serving up your tracker's images.
# Must have trailing /.
# If not set, regular download_url for the attached file is generated.
url = https://example.net/tracker/imgproxy/

# Hex encoded random key used for signature hash.
# Can be stored in a file. Use file://filename relative to
# extensions directory.
# Shell command to generate suitable string is:
#   echo $(xxd -g 2 -l 64 -p /dev/random | tr -d '\n')
#
# If not set, unsigned requests are generated for imgproxy.
# if Accepted by imgproxy, this allows attached files to
# be accessed by unauthorized clients.
key = 736563726574

# Hex encoded random salt used for signature hash.
# See key documentation for details.
salt = file://../secrets/imgproxy.salt

# Integer number of seconds that the image url is cached/valid
# (default 5 seconds). Must be > 0. If you are lazy loading
# the images on a page with a lot of images, you need to
# increase this time otherwise images will fail to load.
#ttl = 10
 
# Add a watermark to images. Assumes IMGPROXY_WATERMARK_DATA
# or other watermark source is defined in imgproxy environment
# (default False)
#watermark = on

Change the url to point to your server. If you run multiple trackers, you probably want one imgproxy per tracker. You could modify the Python code to include a tracker indicator in the local:// url. Then set up a directory served by imgproxy that mounts/links all of the file storage for every tracker. However doing this seems fragile.

The key and salt along with the ttl provide some security to your images. This is discussed in the Security section below.

After you have set up extensions/config.ini and added extensions/imgproxy.py, you have to restart your tracker so it will read the new settings and code.

Add image display to your issue page

You can add this section of TAL to your tracker's issue.item.html. It will display the images attached to a message after the content of the message. The key part is to call imgproxy_url and have the browser download the image file within the URL timeout.

The code creates a div and displays the images inside the div.

the commented out line with file/download_url is the original line. You may or may not have it in you tracker depending on the template used to build your tracker. This div is placed after the tag that prints the message content. A line similar to:

<pre tal:content="structure msg/content/hyperlinked">content</pre>

prints the messsage content as preformatted text. The div would be added after </pre>.

The example call to utils.imgproxy_url takes two arguments. The first is the file object. The second is an optional list of imgproxy processing commands. This particular command resizes the image to 400x300 pixels and enables watermarking of all the images using the predefined mark preset.

Security Concerns

If you have set up your tracker with access lists so that some issues and attachments are private, you have some security issues to evaluate.

The Roundup tracker authorizes a user to see a particular image. imgproxy_url generates a URL with a short lifetime (set by the ttl setting). The URL must be used within this lifetime. It will not work (returns a 404) if used after it expires.

Since imgproxy can't implement Roundup's access controls Roundup creates an authentication/authorization token by signing the URL used to download the image. This token can be used by anybody who possesses it. This is similar to a JSON Web Token (JWT). One defense against unauthorized use is to limit the lifetime of the token. That is what is implemented here. Roundup generates a signed URL using the key and salt. Imgproxy knows the key and salt and can validate the signature. If the expiration (expires) times is modified, the signature no longer matches and the URL is ignored. If an attacker gets access to the URL and tries to use it any time after the expiration date, again the URL is ignored.

This is not foolproof. Adding a watermark to all the images can help identify images that have leaked from Roundup. But this is a detection not a prevention method.

It may be possible to have the Roundup server itself proxy the data from imgproxy. So imgproxy is not contacted by the client but by the backend Roundup server. This provides authentication authorization. If you develop this, please announce it on the roundup-users' list and add to the wiki.

Note that setting expires interferes with some operations. For example, images are not cached in the browser. If the user tries to save an image, they will get an error if they try to re-download the image after the URL has expired. It is also possible that images will not load if the browser or network is slow enough that the image can't be downloaded before the URL expires.

Even if you do not have privacy issues, you should still sign your URL's to prevent DOS by bad actors.

See Also

* Documentation for imgproxy

* Discussion with the developer about the security issues with this configuration

* Roundup issue that is addressed by this idea