Contents
Images are some of the larger items we have to serve from Roundup. While detectors could be written to convert file formats and offer up a more efficient file, this has a number of drawbacks:
- storage space for the new formats. Often you need to retain the original file as well for auditing or other purposes.
- wasted work for converting files that might never be viewed
- complexity of detecting the right file to serve
using a service that converts image files on demand to the best format and size mitigates many of these issues.
The imgproxy server is written in go and is available as a docker container. By hosting imgproxy so it has access to the file storage tree of a Roundup Tracker you can use it to convert image files on the fly. This can reduce the shipped bytes by up to 90%. It will refuse to serve a image file in a format it doesn't recognize. Also unless the raw processing directive is used, it will refuse to serve any unrecognized file.
Four things need to be in place to implement this:
- A running imgproxy server serving files from Roundup's file storage
A Roundup templating extension that adds imgproxy_url() to the utils
- class. This function will produces a URL pointing to the imgproxy server.
A suitable configuration for the imgproxy_url() function
A change to a template to call utils.imgproxy_url() to replace the usual
file/download_url method.
Running imgproxy
You get a copy of the imgproxy docker container using by running docker pull darthsim/imgproxy:latest.
Since it is a 12 factor app, it has an extensive list of environment variables used to configure it. The following script starts it setting a number of configuration variables:
cat > imgproxy.env <<END_ENV
# security key hex-encoded - see security section at the end of this page
IMGPROXY_KEY=736563726574
# salt hex encoded - see security section at the end of this page
IMGPROXY_SALT=68656C6C6F
# location under the site where proxy is located
IMGPROXY_PATH_PREFIX=/trackers/tracker1/imgproxy
# limit to only serving files from local filesystem
IMGPROXY_ALLOWED_SOURCES=local://
# allow and prefer webp if supported
IMGPROXY_ENABLE_WEBP_DETECTION=true
IMGPROXY_ENFORCE_WEBP=true
# allow and prefer AVIF if supported
IMGPROXY_ENABLE_AVIF_DETECTION=true
IMGPROXY_ENFORCE_AVIF=true
# Try to prevent image caching. Cache headers includes
# public. But we don't want images served to be cached in
# public servers as people can hit the cache and get access
# to images they should not access. We would like that to be
# private so that the browser can cache it, but.....
IMGPROXY_TTL=1
# root directory to serve images from
IMGPROXY_LOCAL_FILESYSTEM_ROOT=/images
# define a directive called mark to watermark images
IMGPROXY_PRESETS=mark=watermark:0.3:sowe
# base64 encoded image to use as watermark
IMGPROXY_WATERMARK_DATA=iVBORw0KGgoAAAANSUhEUgAAAGcAAABCAQMAAABw7lo/AAA[...]
END_ENV
background="-d"
#background=""
docker run \
${background} \
--rm \
--name imgproxy \
--env-file imgproxy.env \
-v /path/to/tracker/home/db/files/file:/images \
-p 9080:8080 \
darthsim/imgproxy:latest
rm -f imgproxy.envThis starts the imgproxy web server on port 9080. It is strongly suggested to reverse proxy to imgproxy the same way you would to roundup-server. In this case, it is available at https://.../trackers/tracker1/imgproxy as set by IMGPROXY_PATH_PREFIX.
Adding imgproxy.py to define utils.imgproxy_url()
Place a copy of imgproxy.py (download) in your tracker's extensions directory.
The code looks like:
1 """A module to redirect image file loading to imgproxy for more
2 efficient and smaller image formats.
3
4 This module defines a util.imgproxy_url class that can be
5 used in place of download_url in img tags.
6
7 E.G. replace
8
9 <img tal:attributes="src file/download_url; alt file/name"
10 loading="lazy">
11
12 with:
13
14 <img tal:attributes="src
15 python:utils.imgproxy_url(file, ['rs:fit:400:300',
16 'preset:mark' ]);
17 alt file/name" loading="lazy">
18
19 to load the image at a size of 400x300 and have a watermark
20 placed on it.
21
22 It is controlled by the following settings in
23 extensions/config.ini:
24
25 -----
26 [imgproxy]
27
28 # URL for imgproxy serving up your tracker's images.
29 # Must have trailing /.
30 # If not set, regular download_url for the attached file is generated.
31 url = https://example.net/tracker/imgproxy/
32
33 # Hex encoded random key used for signature hash.
34 # Can be stored in a file. Use file://filename relative to
35 # extensions directory.
36 # Shell command to generate suitable string is:
37 # echo $(xxd -g 2 -l 64 -p /dev/random | tr -d '\n')
38 #
39 # If not set, unsigned requests are generated for imgproxy.
40 # if Accepted by imgproxy, this allows attached files to
41 # be accessed by unauthorized clients.
42 key = 736563726574
43
44 # Hex encoded random salt used for signature hash.
45 # See key documentation for details.
46 salt = file://../secrets/imgproxy.salt
47
48 # Integer number of seconds that the image url is cached/valid
49 # (default 5 seconds). Must be > 0. If you are lazy loading
50 # the images on a page with a lot of images, you need to
51 # increase this time otherwise images will fail to load.
52 #ttl = 10
53
54 # Add a watermark to images. Assumes IMGPROXY_WATERMARK_DATA
55 # or other watermark source is defined in imgproxy environment
56 # (default False)
57 #watermark = on
58 ------
59
60 See: https://imgproxy.net/
61 https://github.com/imgproxy/imgproxy
62 """
63
64 import binascii
65 import base64
66 import hashlib
67 import hmac
68 import logging
69 import time
70
71 from roundup.anypy.strings import b2s, bs2b
72 from roundup.anypy.urllib_ import quote
73 from roundup.configuration import BooleanOption, \
74 IntegerNumberOption, \
75 InvalidOptionError, \
76 OptionValueError, \
77 SecretOption, \
78 WebUrlOption
79
80 try:
81 fromhex = bytes.fromhex
82 except AttributeError:
83 fromhex = binascii.unhexlify
84
85 logger = logging.getLogger('extension')
86
87
88 def make_signature(processing_url, key_hex, salt_hex):
89 """Generate the signature for the processing_url with key/salt
90
91 Used to limit access to the image url passed to
92 imgproxy by signing the processing_url.
93 """
94 key = fromhex(key_hex)
95 salt = fromhex(salt_hex)
96
97 path = bs2b(processing_url)
98 digest = hmac.new(key, msg=salt+path,
99 digestmod=hashlib.sha256).digest()
100 return base64.urlsafe_b64encode(digest).rstrip(b"=")
101
102
103 def get_setting(config, setting, default=""):
104 """Get a setting from a detector or extension url with
105 default value if not defined.
106 """
107 try:
108 return config[setting]
109 except InvalidOptionError:
110 return default
111
112
113 def imgproxy_url(file_obj, render_options=None):
114 """Take a file object (with a download_url method) and an
115 optional list of imgproxy processing
116 options. Specified options take precedence over the
117 ones added by this function (filename, watermark, expires).
118
119 Return a url to the imgproxy image server that has
120 access to the tracker's db/files/file directory
121 tree. Processing directives and other doc at:
122 https://docs.imgproxy.net/.
123 """
124
125 ext_config = file_obj._db.config.ext
126
127 url_prefix = get_setting(ext_config, 'IMGPROXY_URL', None)
128
129 if url_prefix is None:
130 # can't use imgproxy without the url.
131 # this is a safety since it should never happen
132 # as this function is not called if url not defined.
133 return file_obj.download_url()
134
135 key = get_setting(ext_config, 'IMGPROXY_KEY', None)
136 salt = get_setting(ext_config, 'IMGPROXY_SALT', None)
137 ttl = get_setting(ext_config, 'IMGPROXY_TTL', 5)
138 watermark = get_setting(ext_config, 'IMGPROXY_WATERMARK', False)
139
140 if not render_options:
141 # basic resize 400x300 no shrink or expand
142 render_options = ["rs:fill:400:300:0:0"]
143
144 # Insert default/configured options at start of
145 # render_options. Options later in the URL take
146 # precedence. So watermark can be overridden by
147 # 'watermark:0' set in render_options passed to
148 # imgproxy_url().
149
150 # Set filename to basename of image file.
151 # Suffix is added by imgproxy to match image format.
152 name = file_obj._klass.get(file_obj._nodeid, 'name')
153 render_options.insert(0, "filename:%s" % quote(name[:name.rindex('.')]))
154
155 # Add the watermark. Assumes IMGPROXY_WATERMARK_DATA or
156 # other watermark definition is defined in imgproxy
157 # environment.
158 if watermark:
159 render_options.insert(0, "watermark:0.3:sowe")
160
161 if key and salt:
162 # Set URL expiration time.
163 # If there is no signature hash, any user can change the
164 # expire time and resend the URL. So don't bother
165 # setting it.
166 render_options.insert(0, "expires:%i" % (time.time() + ttl))
167
168 rendering = "/".join(render_options)
169 local_url = "local:///%s" % file_obj._db.subdirFilename(
170 file_obj._klass.classname,
171 file_obj._nodeid)
172
173 processing_url = "/%(rendering)s/plain/%(local_url)s" % {
174 "rendering": rendering, "local_url": local_url, }
175
176 if key and salt:
177 signature = b2s(make_signature(processing_url, key, salt))
178 else:
179 signature = "not_signed"
180
181 imgproxy_url = "%(url_prefix)s%(signature)s%(processing_url)s" % {
182 "url_prefix": url_prefix, "signature": signature,
183 "processing_url": processing_url}
184
185 return imgproxy_url
186
187
188 def init(instance):
189 # verify options values meet standards.
190 try:
191 instance.config.ext.update_option(
192 'IMGPROXY_URL', WebUrlOption,
193 description="Url for imgproxy server.")
194 except InvalidOptionError:
195 # It's not set, that's ok. Just call download_url()
196 # and skip everything else in this module.
197 instance.registerUtil(
198 'imgproxy_url',
199 lambda file_obj, _ignore=None: file_obj.download_url())
200 return
201
202 try:
203 option_name = 'IMGPROXY_KEY'
204 instance.config.ext.update_option(
205 option_name, SecretOption,
206 description="Random HMAC key, hex encoded.")
207
208 try:
209 fromhex(instance.config.ext[option_name])
210 except ValueError:
211 raise OptionValueError(
212 instance.config.ext.options[option_name],
213 instance.config.ext[option_name],
214 "in extensions/config.ini. Is not a valid hex encoded string."
215 )
216 except InvalidOptionError:
217 # It's not set, that's ok.
218 pass
219
220 try:
221 option_name = 'IMGPROXY_SALT'
222 instance.config.ext.update_option(
223 option_name, SecretOption,
224 description="Random salt used with HMAC key, hex encoded.")
225
226 try:
227 fromhex(instance.config.ext[option_name])
228 except ValueError:
229 raise OptionValueError(
230 instance.config.ext.options[option_name],
231 instance.config.ext[option_name],
232 "in extensions/config.ini. Is not a valid hex encoded string."
233 )
234 except InvalidOptionError:
235 # It's not set, that's ok.
236 pass
237
238 try:
239 instance.config.ext.update_option(
240 'IMGPROXY_TTL', IntegerNumberOption, default=5,
241 description="Lifetime in integer seconds of image URL.")
242 except InvalidOptionError:
243 # It's not set, that's ok. Default is 5 seconds.
244 pass
245
246 try:
247 instance.config.ext.update_option(
248 'IMGPROXY_WATERMARK', BooleanOption, default=False,
249 description="Print watermark on images.")
250 except InvalidOptionError:
251 # It's not set, no watermark.
252 pass
253
254 instance.registerUtil('imgproxy_url', imgproxy_url)
255
256
257 if __name__ == '__main__':
258 """ Test make_signature() using imgproxy example signature
259 settings.
260 """
261 url = ('/rs:fill:300:400:0/g:sm/aHR0cDovL2V4YW1w/bGUuY29tL2l'
262 'tYWdl/cy9jdXJpb3NpdHku/anBn.png')
263 print(url)
264
265 s = make_signature(url, '736563726574', '68656C6C6F')
266
267 print(s)
268
269 assert b2s(s) == "oKfUtW34Dvo2BGQehJFR4Nr0_rIjOtdtzJ3QFsUcXH8"
270
271 print("Test passed, no errors")
Append settings to extensions/config.ini
Append the following lines to config.ini located in your extensions directory. Create the file if you don't have one.
[imgproxy] # URL for imgproxy serving up your tracker's images. # Must have trailing /. # If not set, regular download_url for the attached file is generated. url = https://example.net/tracker/imgproxy/ # Hex encoded random key used for signature hash. # Can be stored in a file. Use file://filename relative to # extensions directory. # Shell command to generate suitable string is: # echo $(xxd -g 2 -l 64 -p /dev/random | tr -d '\n') # # If not set, unsigned requests are generated for imgproxy. # if Accepted by imgproxy, this allows attached files to # be accessed by unauthorized clients. key = 736563726574 # Hex encoded random salt used for signature hash. # See key documentation for details. salt = file://../secrets/imgproxy.salt # Integer number of seconds that the image url is cached/valid # (default 5 seconds). Must be > 0. If you are lazy loading # the images on a page with a lot of images, you need to # increase this time otherwise images will fail to load. #ttl = 10 # Add a watermark to images. Assumes IMGPROXY_WATERMARK_DATA # or other watermark source is defined in imgproxy environment # (default False) #watermark = on
Change the url to point to your server. If you run multiple trackers, you probably want one imgproxy per tracker. You could modify the Python code to include a tracker indicator in the local:// url. Then set up a directory served by imgproxy that mounts/links all of the file storage for every tracker. However doing this seems fragile.
The key and salt along with the ttl provide some security to your images. This is discussed in the Security section below.
After you have set up extensions/config.ini and added extensions/imgproxy.py, you have to restart your tracker so it will read the new settings and code.
Add image display to your issue page
You can add this section of TAL to your tracker's issue.item.html. It will display the images attached to a message after the content of the message. The key part is to call imgproxy_url and have the browser download the image file within the URL timeout.
The code creates a div and displays the images inside the div.
the commented out line with file/download_url is the original line. You may or may not have it in you tracker depending on the template used to build your tracker. This div is placed after the tag that prints the message content. A line similar to:
<pre tal:content="structure msg/content/hyperlinked">content</pre>
prints the messsage content as preformatted text. The div would be added after </pre>.
The example call to utils.imgproxy_url takes two arguments. The first is the file object. The second is an optional list of imgproxy processing commands. This particular command resizes the image to 400x300 pixels and enables watermarking of all the images using the predefined mark preset.
Security Concerns
If you have set up your tracker with access lists so that some issues and attachments are private, you have some security issues to evaluate.
The Roundup tracker authorizes a user to see a particular image. imgproxy_url generates a URL with a short lifetime (set by the ttl setting). The URL must be used within this lifetime. It will not work (returns a 404) if used after it expires.
Since imgproxy can't implement Roundup's access controls Roundup creates an authentication/authorization token by signing the URL used to download the image. This token can be used by anybody who possesses it. This is similar to a JSON Web Token (JWT). One defense against unauthorized use is to limit the lifetime of the token. That is what is implemented here. Roundup generates a signed URL using the key and salt. Imgproxy knows the key and salt and can validate the signature. If the expiration (expires) times is modified, the signature no longer matches and the URL is ignored. If an attacker gets access to the URL and tries to use it any time after the expiration date, again the URL is ignored.
This is not foolproof. Adding a watermark to all the images can help identify images that have leaked from Roundup. But this is a detection not a prevention method.
It may be possible to have the Roundup server itself proxy the data from imgproxy. So imgproxy is not contacted by the client but by the backend Roundup server. This provides authentication authorization. If you develop this, please announce it on the roundup-users' list and add to the wiki.
Note that setting expires interferes with some operations. For example, images are not cached in the browser. If the user tries to save an image, they will get an error if they try to re-download the image after the URL has expired. It is also possible that images will not load if the browser or network is slow enough that the image can't be downloaded before the URL expires.
Even if you do not have privacy issues, you should still sign your URL's to prevent DOS by bad actors.
See Also
* Discussion with the developer about the security issues with this configuration