libvips 8.14 is now done, so here’s a summary of what’s new. Check the ChangeLog if you need more details.
The headline features are the final switch to meson build system, a new
thread pool and thread recycling system, some useful speedups to
arrayjoin and TIFF load, and the usual small improvements to image format
support. Details below!
We need to thank aksdb, dloebl, ewelot, tlsa, remicollet, DarthSim, ejoebstl, lovell, shado23, kleisauke, and others for their great work on this release.
Build system changes
The previous release added a meson build system alongside the autotools build system we’d used for decades. This release removes the autotools system completely, and a lot of other old stuff too, so it’s now meson all the way down.
The README has some notes to help you get started if you need to build from source, but the quick summary is something like:
cd libvips-x.y.x meson setup build --prefix /my/install/prefix cd build meson compile meson test meson install
N-colour profile support
Printers usually work with four colours: cyan, magenta, yellow and black. libvips has supported (via LittleCMS) conversion to and from CMYK images for a long time.
In 8.14 we’ve added support for N-colour profiles. These profiles add extra
colours, usually the complements of CMY, to expand the colour gamut available
and reduce dithering effects. The
icc_transform set of operators now
support rendering to and from N-colour device images in the obvious way.
The TIFF saver also knows how to read and write these CMYRGK (for example) files, and hopefully does it in a way that’s compatible with PhotoShop.
With this sample 6CLR profile I can run:
$ vips icc_export nina.jpg x.tif --output-profile 6clr-test.icc
To make an image split to 6 colour separations:
$ tiffinfo x.tif === TIFF directory 0 === TIFF Directory at offset 0x8b89008 (146313224) Image Width: 6048 Image Length: 4032 Resolution: 300, 300 pixels/inch Bits/Sample: 8 Sample Format: unsigned integer Compression Scheme: None Photometric Interpretation: separated Extra Samples: 2<unassoc-alpha, unassoc-alpha> Orientation: row 0 top, col 0 lhs Samples/Pixel: 6 Rows/Strip: 128 Planar Configuration: single image plane InkSet: 1 ICC Profile: <present>, 13739188 bytes
We’ve also added support for character as well as word wrapping
vips_text() with the
wrap parameter, added an
rgb mode to
vips_smartcrop() now optionally returns the
location of interest in attention-based cropping.
Improvements to the libvips core
libvips uses threadpools to compute pixels – these are groups of worker threads cooperating on a task. For 8.14, we’ve rewritten the threadpool system and added several very useful new features:
Threadpools now resize dynamically. Each threadpool is able to tell how busy the workers are, and is able to either expand or shrink depending on load. The old
vips-concurrencysetting now sets the maximum threadpool size.
The aim is to improve utilisation on machines with many cores. Why create 16 workers, for example, for a pipeline that only has a small amount of parallelism?
Few idle threads means libvips should make better use of hardware resources on large machines with complex mixed workloads. The new threadpool should also be a bit quicker.
You can also set hints for the amount of parallelism you expect in a pipeline. Again, this helps prevent overcomitting of thread resources.
Finally, there’s a new thread recycling system. Some platforms have very slow or tricky thread start and stop, so rather than killing and recreating threads all the time, libvips will make a set of threads and then recycle them between threadpools.
We had a thread recycling system before, but this new one should be noticably simpler and faster.
We’ve used this new threading system to revise
dzsave, and it’s now quite a
bit quicker. Here’s the previous release, libvips 8.13, running on a
46,000 x 32,914 pixel slide image:
$ /usr/bin/time -f %M:%e vips dzsave CMU-1.svs x 881892:36.65
That’s 900mb of peak memory and 37s of run time. Here’s libvips 8.14:
$ /usr/bin/time -f %M:%e vips dzsave CMU-1.svs x 704360:19.50
Almost twice as fast, and noticably lower memory use. This all comes from the new threading system.
This new release has another feature which can improve slide read
rgb flag to openslideload. This drops the redundant alpha
plane earlier, saving time and memory:
$ /usr/bin/time -f %M:%e vips dzsave CMU-1.svs[rgb] x 547832:13.02
Now it’s three times faster than 8.13 and runs in about half the memory.
arrayjoin operator has been reworked and is now faster
for large sets of images. Here’s libvips 8.13 reassembling 10,000 tiles:
$ vips dzsave CMU-1.svs x --depth one --tile-width 400 --tile-height 400 --overlap 0 --suffix .jpg $ cd x_files/0/ $ ls | wc 9545 9545 94715 $ time vips arrayjoin "$(ls *.jpg | sort -t_ -k2g -k1g)" x.jpg --across 115 real 0m17.357s user 0m37.003s sys 0m44.812s
And here’s 8.14:
$ time vips arrayjoin "$(ls *.jpg | sort -t_ -k2g -k1g)" x.jpg --across 115 real 0m11.883s user 0m47.085s sys 0m46.906s
The speedup becomes dramatic with larger tile sets.
This won’t appeal to many people, but libvips now ships with a simple
bash completion script in
libvips-x.y.z/completions, have a look at the
README in there for install notes. It knows how to complete operator names,
filenames, and required arguments, including enum values.
It’s useful now, and we hope to improve it in the next version, perhaps by expanding optional arguments, for example.
Image format improvements
There have been quite a few improvements to file format support.
Previous releases used libtiff to fetch decoded tiles from compressed TIFF files. This ran the decompressor inside the libtiff lock, so it was single threaded.
For libvips 8.14, we’ve moved jpeg2000 and jpeg decompression outside the libtiff lock so they now run multi-threaded. This can give a really nice speedup.
First, make a large, tiled, JPEG-compressed TIFF:
$ vips copy CMU-1.svs[rgb] x.tif[tile,compression=jpeg]
Then read the file and compute the pixel average. Here’s the previous 8.13 release:
$ time vips avg x.tif 226.581443 real 0m42.776s user 0m48.380s sys 0m0.428s
You can see that the total CPU time (the
user line) is almost equal to the
real clock time (the
real line), so there was very little parallelism. It
was able to parallelize the computation of the average value, but the
decompress (which was almost all of the run time) was single threaded.
$ time vips avg x.tif 226.581443 real 0m3.371s user 0m17.413s sys 0m1.573s
Now tiles are decompressed in parallel and on this 16-core PC there’s a huge speedup, more than 10x. Zoom!
This is just accelerating TIFF load. Perhaps TIFF save will get the same treatment in the next version.
GIF save has a new
interlace option, and the default save behaviour has
changed: for safety it now always recomputes the palette. If you want to
reuse the input palette, set the new
GIF load now handles truncated files much more gracefully.
The saver has a new
encoder parameter you can use to select
the exact save codec library you want.
The FITS load and save operations have been rewritten. Many band FITS images should load dramatically faster, and FITS save is much better at handling duplicated header fields.
PNG load and save operations now support EXIF metadata. Animated webp save
has been rewritten and should perform much better. Saving as
now pick the bext
p*m subformat for you automatically. The jp2k saver
now defaults to chroma subsample off for better compatibility, and writes
jp2 images rather than a simple codestream.
Plus the usual range of small improvements and bugfixes. See the ChangeLog.