EUMETSAT recently launched its third generation geostationary weather satellite: Meteosat Third Generation (MTG for the insiders). With a new satellite comes a new file format to distribute the data, including a new compression method. I describe how to setup your HDF5 install to read those files.

First of all, a disclaimer: these are not official installation instructions in any way. This is what I found the most convenient with my knowledge of cmake and of HDF5. Other disclaimer: this is only tested on Linux.

EUMETSAT's solution

EUMETSAT provides a package called fcidecomp that bundles the compiled jpeg HDF5 decompression filter and a Python library that enables it. I downloaded it (see URL below) and tried to use src/fcidecomp/gen/build.sh as instructed in the INSTALL.md file. It generated too many error messages to my taste, so I simply found the location from which to call cmake directly. Also, as I wish to use the files with other codes in C unrelated to any Python installation, this seemed necessary anyway.

What data?

I consider here the gridded "Level 1 c" radiance images from the multispectral imager aboard MTG, the Flexible Combined Imager (FCI). The data is distributed as NetCDF 4 files representing segments of the full disk image on a geostationary projection. For more information, see the product user guide.

The algorithm chosen for the dissemination the FCI L1c data is CharLS, a codec than can be used to compress JPEG images. The compression is applied internally to the image datasets.

HDF5 and NetCDF

The data is stored in NetCDF 4 files, which are HDF5 files internally. HDF5 datasets can be compressed internally, meaning that the file itself is a plain HDF5 file and that the data goes through a coded upon writing or reading.

The software

EUMETSAT provides the decompression filter on their gitlab repository. Beware that you need to download the code as a zip or tarball from the web interface as a git clone will not work.

The software consists in:

• The jpegls module, which wraps the charls module available elsewhere (ideally in your Linux distribution).
• The H5Zjpegls decompression filter for HDF5: a software component that can be dynamically loaded by the HDF5 library.
• A Python module called fcidecomp that helps to load the filter when using h5py (the reference HDF5 Python library).

A few comments are in order:

• Ultimately, what is necessary is the jpegls module and the H5Zjpegls filter.
• EUMETSAT provides build scripts that call cmake. I got some errors when trying to use them and did not bother much as there are suitable CMakeLists.txt files in the repository.
• The Python code is not strictly necessary as HDF5 is designed to be able to load filters dynamically. The benefit of not using it is that you don't need one copy of fcidecomp per Python environment but one per HDF5 install.
• The method that I outline below was the lowest friction for me. Your mileage may vary.

Steps

Here is the overview of the steps necessary to build and use the software:

1. Install charls + dev
2. Build jpegls
3. Build H5Zjpegls
4. Set LD_LIBRARY_PATH if necessary.
5. Set HDF5_PLUGIN_PATH

The result will be an installation of the filter in /opt/fcidecomp-2.0.1 that is linked against your system's HDF5.

Any C/Fortran code built against the same install of HDF5 should be able to load the files without modification, including Python whose h5py library is linked against it.

Charls

CharLS is found in the distributions' repositories. As root:

• On debian/ubuntu: apt install libcharls-dev libcharls2
• On OpenSUSE: zypper install CharLS-devel libcharls2

jpegls and H5Zjpegls

Visit EUMETSAT's software at https://gitlab.eumetsat.int/open-source/data-tailor-plugins/fcidecomp/-/tree/2.0.1?ref_type=heads and dowload the .zip or .tar.gz file (small download arrow between "Find file" and "Clone"). Decompress the file and visit the corresponding directory.

Provided the directory /opt/fcidecomp-2.0.1 is writable by your user, the following does not need root permissions. You can copy-paste the instructions by omitting the leading $ character.

$ cd /tmp
$ tar zxf ~/Downloads/fcidecomp-2.0.1.tar.gz
$ cd fcidecomp-2.0.1/

Now, you have to compile the modules. Here is how I did it.

$ cd src/fcidecomp/fcicomp-jpegls/
$ mkdir build
$ cd build
$ export HDF5_ROOT="/usr/lib/x86_64-linux-gnu/hdf5/serial"
$ cmake .. -DCMAKE_INSTALL_PREFIX=/opt/fcidecomp-2.0.1 -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON
$ make
$ make test
$ make install

Of course, the HDF5 root and the install location depend on your requirements. On openSUSE this was not necessary, the above command is only for Debian/Ubuntu.

To compile H5Zjpegls you will also need to point the software to your new install of jpegls.

$ cd ../../fcicomp-H5Zjpegls
$ mkdir build
$ cd build
$ export CMAKE_PREFIX_PATH=/opt/fcidecomp-2.0.1/share/cmake/fcicomp_jpegls
$ cmake .. -DCMAKE_INSTALL_PREFIX=/opt/fcidecomp-2.0.1 -DCMAKE_BUILD_TYPE=Release
$ make
$ make install

The export CMAKE_PREFIX_PATH=... command is important as it allows cmake to find the install of jpegls. Note: the tests for H5Zjpegls fail when I build them with -DBUILD_TESTING=ON.

What works

For using the plugin, issue the following commande before invoking h5repack or using Python.

export HDF5_PLUGIN_PATH=/opt/fcidecomp-2.0.1/hdf5/lib/plugin
export LD_LIBRARY_PATH=/opt/fcidecomp-2.0.1/lib

1. What should work in any case is h5repack for decompressing the files. This is because HDF5 is designed to load the filter dynamically from those found in HDF5_PLUGIN_PATH. Doing h5repack -i MTI1_202311021200/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-HRFI-FD--CHK-BODY--DIS-NC4E_C_EUMT_20231102121037_IDPFI_OPE_20231102120908_20231102120924_N_JLS_C_0073_0040.nc -o uncompressed/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-HRFI-FD--CHK-BODY--DIS-NC4E_C_EUMT_20231102121037_IDPFI_OPE_20231102120908_20231102120924_N_JLS_C_0073_0040.nc for instance, should simply output the uncompressed version of the file.
2. Even though nccopy should work as is, provided the HDF5_PLUGIN_PATH is defined, this did not work on my debian machine.
3. A pip-installed python should also work. What is important is that h5py links against the same version of HDF5. It might work by chance if you have the same HDF5 version in your system and in a conda environment but I wouldn't count on it.

I had success with

1. Using Debian 12's stock HDF5 install, h5repack Version 1.10.8.
2. Using openSUSE 15.4's stock HDF5 install, h5repack version 1.10.8.
3. loading MTG FCI files using satpy in a pip "venv" with Python 3.11 and satpy version 0.44.0 and h5py version 3.8.0.

Result!

If you have read so far, you certainly deserved a MTG picture!