Datasets
Reference scientific datasets for benchmarking data reduction techniques. Click on command examples to expand.
Reference scientific datasets for benchmarking data reduction techniques. Click on command examples to expand.
FORMAT: Describes the data organization, including the number of fields in each data file and the data dimensions. Dimensions are listed slowest-varying first (C order, as in numpy), e.g. 26x1800x3600 is 26 slices of 1800x3600; the command examples pass them to the tools fastest-varying first (e.g. -3 3600 1800 26).
SIZE: Indicates the file size in bytes. Since most datasets are distributed as compressed .tar.gz archives, SIZE typically refers to the size of the corresponding .tar.gz file.
Python and Hugging Face. All datasets below except NSTX GPI are also mirrored, byte for byte, on
Hugging Face, and can be loaded as numpy arrays with the
sdrbench package. It downloads single files from Hugging Face,
checks them against their sha256, and falls back to the archives listed here:
pip install sdrbench
import sdrbench
x = sdrbench.dataset("cesm-atm")["CLDHGH"] # numpy array, shape (1800, 3600)
Every field comes with its dtype and C-order shape; see szcompressor/sdrbench for the dataset, variant and field names.
Source: Mark Taylor (SNL)
Dataset1: 79 fields, 2D, 1800x3600
(the cleared version has 77 fields: SFCLDICE and SFCLDLIQ are not included)
Dataset2: 33 fields, 3D, 26x1800x3600
Both single precision, binary, little-endian
Dataset1 (raw): 1.47GB
Dataset1 (cleared): 1.47GB
Dataset2: 17GB
(cleared data zeroed all background data)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.5 | 19.8 |
| min | 1.5 | 2.3 |
| max | 7.8 | 22.6 |
SZ (Compress): sz -z -f -i CLDHGH_1_1800_3600.f32 -M REL -R 1E-2 -2 3600 1800
SZ (Decompress): sz -x -f -i CLDHGH_1_1800_3600.f32 -2 3600 1800 -s CLDHGH_1_1800_3600.f32.sz -a
ZFP: zfp -f -i CLDHGH_1_1800_3600.f32 -z CLDHGH_1_1800_3600.f32.zfp -o CLDHGH_1_1800_3600.f32.zfp.out -2 3600 1800 -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i CLDHGH_1_1800_3600.f32 -d 3600 -d 1800 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL CESM-ATM raw-data-dir f32 3600 1800
Source: EXAALT team
This dataset has been approved for unlimited release by Los Alamos National Laboratory and has been assigned LA-UR-18-25670.
6 fields: x, y, z, vx, vy, vz
Each field stored separately
Single precision, binary, little-endian
Dataset1: 60 MB
Dataset2: 973 MB
Dataset3: 2.4 GB
Data1
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.3 | 19.9 |
| min | 7 | 18.4 |
| max | 7.4 | 20.4 |
Data2
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.7 | 15.4 |
| min | 6 | 9.40 |
| max | 7.1 | 20.1 |
Data3
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.9 | 16.3 |
| min | 6.8 | 15.6 |
| max | 7.2 | 17.6 |
SZ (Compress): sz -z -f -i xx.f32 -M REL -R 1E-2 -1 2869440
SZ (Decompress): sz -x -f -i xx.f32 -1 2869440 -s xx.f32.sz -a
ZFP: zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -1 2869440 -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i xx.f32 -d 2869440 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL EXAALT raw-data-dir f32 2869440
Source: IEEE Visualization 2004 Contest (http://vis.computer.org/vis2004contest/data.html)
13 fields, 3D, 100x500x500
Single precision, binary
(cleared dataset by replacing background by 0)
Snapshot48 archive: the 13 fields of time step 48 plus 7 log10-transformed fields (e.g. CLOUDf48.log10.bin.f32)
Per-field archives: 48 time steps each (e.g. Pf01.f32 ... Pf48.f32; log10 archives: CLOUDf01.bin.log10.f32 ...)
The QCLOUDf archive also contains the QGRAUPf01-48 fields
Snapshot48: 1.25GB
CLOUDf: 575MB
Pf: 4GB
PRECIP: 3.9GB
QCLOUD: 3.7GB
QICE: 502MB
QRAIN: 3.5GB
QSNOW: 3.9GB
QVAPOR: 3.9GB
TC: 3.6GB
U: 4.1GB
V: 4.0GB
W: 4.2GB
CLOUD_log10: 533MB
QCLOUD_log10: 193MB
QGRAUP_log10: 3GB
QICE_log10: 464MB
QRAIN_log10: 2.8GB
QSNOW_log10: 3.3GB
Snapshot48(all 13 fields)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 5.9 | 18.9 |
| min | 0.7 | 1.3 |
| max | 7.5 | 24.2 |
CLOUD(Snapshot1-48)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 1.5 | 3.3 |
| min | 1.2 | 2.5 |
| max | 2.5 | 6.2 |
P
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.3 | 24.2 |
| min | 7.3 | 24.1 |
| max | 7.3 | 24.2 |
PRECIP
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.3 | 22.8 |
| min | 6.3 | 17.8 |
| max | 7.5 | 24.1 |
QCLOUD
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 3.6 | 10.6 |
| min | 0.4 | 0.7 |
| max | 7.4 | 23.4 |
QICE
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 1.3 | 2.9 |
| min | 1.1 | 2.2 |
| max | 2.5 | 6.1 |
QRAIN
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.7 | 20.0 |
| min | 2.3 | 5.2 |
| max | 7.4 | 23.1 |
QSNOW
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.2 | 22.2 |
| min | 6.2 | 17.5 |
| max | 7.5 | 23.7 |
QVAPOR
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.4 | 24.2 |
| min | 7.4 | 24.2 |
| max | 7.4 | 24.2 |
TC
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.2 | 23.6 |
| min | 7.2 | 23.5 |
| max | 7.2 | 23.6 |
U
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.4 | 24.2 |
| min | 7.4 | 24.2 |
| max | 7.4 | 24.2 |
V
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.4 | 24.2 |
| min | 7.3 | 24.2 |
| max | 7.4 | 24.2 |
W
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.4 | 24.2 |
| min | 7.4 | 24.2 |
| max | 7.5 | 24.3 |
SZ (Compress): sz -z -f -i Pf48.bin.f32 -M REL -R 1E-2 -3 500 500 100
SZ (Decompress): sz -x -f -i Pf48.bin.f32 -3 500 500 100 -s Pf48.bin.f32.sz -a
ZFP: zfp -f -i Pf48.bin.f32 -3 500 500 100 -z Pf48.bin.f32.zfp -o Pf48.bin.f32.zfp.out -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i Pf48.bin.f32 -d 500 -d 500 -d 100 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL Hurricane raw-data-dir f32 500 500 100
Source: LCLS
Dataset1: 3 files (raw, dark, calib-fde), 10x32x185x388, unsigned 16-bit integer (*.u16)
Dataset2: 1 field, 986x32x185x388, single precision
Dataset3: 1 field, 130x1480x1552, single precision, plus peak positions (130x2048, double) and peak counts (130, int64)
Binary, little-endian (events x panels x rows x columns)
Dataset1: 51 MB
Dataset2: 8.5 GB
Dataset3: 1 GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| value | 7.4 | 25.7 |
SZ (Compress): sz -z -f -i EXAFEL-LCLS-986x32x185x388.f32 -M REL -R 1E-2 -2 388 5837120
SZ (Decompress): sz -x -f -i EXAFEL-LCLS-986x32x185x388.f32 -s EXAFEL-LCLS-986x32x185x388.f32.sz -2 388 5837120 -a
ZFP: zfp -f -a 1E-2 -i EXAFEL-LCLS-986x32x185x388.f32 -z EXAFEL-LCLS-986x32x185x388.f32.zfp -o EXAFEL-LCLS-986x32x185x388.f32.zfp.out -2 388 5837120 -s
LibPressio: pressio -b compressor=$COMP -i EXAFEL-LCLS-986x32x185x388.f32 -d 5837120 -d 388 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL EXAFEL raw-data-dir f32 388 5837120
Source: HACC team (ECP EXASKY): please cite this paper ([Heitmann et al., Astrophys.J.Suppl. 252 (2021) 2, 19](https://inspirehep.net/literature/1799008)) if you are using HACC data. (https://www.alcf.anl.gov/science/case-studies/opencosmo-community-portal-cosmological-simulations)
1 snapshot: 6 fields (x, y, z, vx, vy, vz)
Each field stored separately
Single precision, binary, little-endian
Dataset3: one region of time step 499, coordinates only (x, y, z)
multi-timestep regions: x, y, z of one region over about 100 time steps (the particle count varies per step), plus particle IDs (ID.u64, unsigned 64-bit)
Dataset1: 19 GB
Dataset2: 5 GB
Dataset3: 18.12 GB
multi-timestep-region1: 13 GB
multi-timestep-region2: 14 GB
multi-timestep-region3: 13 GB
multi-timestep-region4: 13 GB
multi-timestep-region5: 13 GB
multi-timestep-region6: 13 GB
Dataset1
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.2 | 25.3 |
| min | 7.1 | 24.7 |
| max | 7.4 | 25.9 |
Dataset3 (potentially super big)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.6 | 20.6 |
| min | 6.5 | 19.0 |
| max | 6.8 | 21.5 |
SZ (Compress): sz -z -f -i xx.f32 -M ABS -A 0.003 -1 1073726487
SZ (Decompress): sz -x -f -i xx.f32 -s xx.f32.sz -1 1073726487 -a
ZFP: zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -a 0.003 -1 1073726487
LibPressio: pressio -b compressor=$COMP -i xx.f32 -d 1073726487 -t float -o abs=3e-3 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f ABS HACC raw-data-dir f32 1073726487
Source: Zarija Lukic (LBL, Berkeley)
6 fields, 3D, 512x512x512
Single precision, binary, little-endian
Dataset1: 2.7 GB
Dataset1(log): 2.7 GB
Dataset1(512x512x512)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.2 | 25.1 |
| min | 7.0 | 24 |
| max | 7.4 | 25.9 |
Dataset1-log(512x512x512)
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.3 | 25.1 |
| min | 7.0 | 24.0 |
| max | 7.4 | 25.9 |
SZ (Compress): sz -z -f -i temperature.f32 -M REL -R 1E-2 -3 512 512 512
SZ (Decompress): sz -x -f -i temperature.f32 -s temperature.f32.sz -3 512 512 512 -a
ZFP: zfp -f -i temperature.f32 -z temperature.f32.zfp -o temperature.f32.zfp.out -3 512 512 512 -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i temperature.f32 -d 512 -d 512 -d 512 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL NYX raw-data-dir f32 512 512 512
Source: Example molecular 2-electron integral values generated by libint, a library developed by the Valeev research group at Virginia Tech. See https://github.com/evaleev/libint. Libint is an integral evaluation option in NWChemEx. (https://github.com/evaleev/libint)
3 fields, 1D
Double precision (*.bin.d64), binary, little-endian
Single-precision copies (*.bin.f32) and a reader (readbin.cpp) are also included
16GB
SZ (Compress): sz -z -d -i 631-tst.bin.d64 -M REL -R 1E-2 -1 102953248
SZ (Decompress): sz -x -d -i 631-tst.bin.d64 -s 631-tst.bin.d64.sz -1 102953248 -a
ZFP: zfp -d -i 631-tst.bin.d64 -z 631-tst.bin.d64.zfp -o 631-tst.bin.d64.zfp.out -a 1E-2 -1 102953248 -s
LibPressio: pressio -b compressor=$COMP -i 631-tst.bin.d64 -d 102953248 -t double -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -d REL NWCHEM raw-data-dir d64 102953248
Source: Simulation data generated by The Local Ensemble Transform Kalman Filter (LETKF) data assimilation package for the SCALE-RM weather model (provided by RIKEN). Contact: Guo-yuan Lien (guoyuan.lien@gmail.com)
13 fields, 3D
Single precision, binary, little-endian
4.9GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 6.7 | 21.7 |
| min | 4.6 | 10.2 |
| max | 7.4 | 26.1 |
SZ (Compress): sz -f -z -i T-98x1200x1200.f32 -M REL -R 1E-2 -3 1200 1200 98
SZ (Decompress): sz -f -x -i T-98x1200x1200.f32 -s T-98x1200x1200.f32.sz -3 1200 1200 98 -a
ZFP: zfp -f -i T-98x1200x1200.f32 -z T-98x1200x1200.f32.zfp -o T-98x1200x1200.f32.zfp.out -a 1E-2 -3 1200 1200 98 -s
LibPressio: pressio -b compressor=$COMP -i T-98x1200x1200.f32 -d 1200 -d 1200 -d 98 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL SCALE-LETKF raw-data-dir f32 1200 1200 98
Source: QMCPACK performance test (contact: Ye Luo, yeluo@anl.gov)
1 field, 288 orbitals, each a 3D 115x69x69 grid
Single precision, binary, little-endian
The archive holds the original layout, einspline_115_69_69_288.f32 (115x69x69x288, orbital index fastest).
The command examples use the preconditioned layout einspline_288_115_69_69.pre.f32 (288x115x69x69), a transpose of the original that is smoother in space; it is not included in the archive.
1 GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| value | 7.6 | 26.1 |
SZ (Compress): sz -z -f -i einspline_288_115_69_69.pre.f32 -M REL -R 1E-2 -3 69 69 33120
SZ (Decompress): sz -x -f -i einspline_288_115_69_69.pre.f32 -s einspline_288_115_69_69.pre.f32.sz -3 69 69 33120 -a
ZFP: zfp -f -i einspline_288_115_69_69.pre.f32 -z einspline_288_115_69_69.pre.f32.zfp -o einspline_288_115_69_69.pre.f32.zfp.out -a 1E-2 -3 69 69 33120
LibPressio: pressio -b compressor=$COMP -i einspline_288_115_69_69.pre.f32 -d 69 -d 69 -d 33120 -t float -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL QMCPack raw-data-dir f32 69 69 33120
Source: Hydrodynamics code for large turbulence simulations (contact: Peter Lindstrom, lindstrom2@llnl.gov)
Small: 3D, 7 fields, each field is 256x384x384, double precision
Big: 3D, 1 field (density), 3072x3072x3072, single precision, stored as 96 slabs density-00.f32 ... density-95.f32 of 32x3072x3072
Binary, little-endian
Small (256x384x384): 1.87GB
Big (3072x3072x3072): 106GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.1 | 22.5 |
| min | 3.62 | 7.2 |
| max | 7.7 | 25.1 |
SZ (Compress): sz -z -d -i density.d64 -M REL -R 1E-2 -3 384 384 256
SZ (Decompress): sz -x -d -i density.d64 -s density.d64.sz -3 384 384 256 -a
ZFP: zfp -d -3 384 384 256 -i density.d64 -z density.d64.zfp -o density.d64.zfp.out -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i density.d64 -d 384 -d 384 -d 256 -t double -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -d REL Miranda raw-data-dir d64 384 384 256
Source: Kolla, Hemanth (hnkolla@sandia.gov)
11 fields, 3D, 500x500x500
Double precision, binary, little-endian
44 GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.6 | 21.9 |
| min | 7.6 | 21.8 |
| max | 7.6 | 22 |
SZ (Compress): sz -d -z -i stat_planar.1.1000E-03.field.d64 -M REL -R 1E-2 -3 500 500 500
SZ (Decompress): sz -d -x -i stat_planar.1.1000E-03.field.d64 -s stat_planar.1.1000E-03.field.d64.sz -3 500 500 500 -a
ZFP: zfp -d -3 500 500 500 -i stat_planar.1.1000E-03.field.d64 -z stat_planar.1.1000E-03.field.d64.zfp -o stat_planar.1.1000E-03.field.d64.zfp.out -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i stat_planar.1.1000E-03.field.d64 -d 500 -d 500 -d 500 -t double -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -f REL S3D raw-data-dir f32 500 500 500
Source: M. Xia, Y. Li, P. Jiao, B. Wang, X. Liang and H. Guo, "Time-Varying Vector Field Compression with Preserved Critical Point Trajectories," 2026 IEEE 42nd International Conference on Data Engineering (ICDE), Montreal, QC, Canada, 2026, pp. 2235-2249, doi: 10.1109/ICDE65706.2026.00168 (http://basilisk.fr/)
4 fields: velocity components u and v of two time-varying 2D fields
1501x1024x1024 & 1001x2048x2048 (time steps x 2D grid)
File names list the dimensions fastest first (WIDTHxHEIGHTxTIMESTEPS):
u_1024x1024x1501Re10k.f32
v_1024x1024x1501Re10k.f32
u_2048x2048x1001Re10k.f32
v_2048x2048x1001Re10k.f32
Single precision, binary, little-endian
40GB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.36 | 26.45 |
| min | 7.35 | 26.39 |
| max | 7.38 | 26.53 |
SZ (Compress): sz -f -z -i u_1024x1024x1501Re10k.f32 -M REL -R 1E-3 -3 1024 1024 1501
SZ (Decompress): sz -f -x -i u_1024x1024x1501Re10k.f32 -s u_1024x1024x1501Re10k.f32.sz -3 1024 1024 1501 -a
ZFP: zfp -f -3 1024 1024 1501 -i u_1024x1024x1501Re10k.f32 -z u_1024x1024x1501Re10k.f32.zfp -o u_1024x1024x1501Re10k.f32.zfp.out -a 1E-3 -s
LibPressio: pressio -i u_1024x1024x1501Re10k.f32 -o rel=1E-3 -d 1024 -d 1024 -d 1501 -b compressor=$COMP
where $COMP can be sz, zfp, sz3, mgard, etc.
Source: Princeton Plasma Physics Laboratory (PPPL) (https://jychoi-hpc.github.io/adios-python-docs/XGC-mesh-and-field-data.html)
9 timesteps, 3D, unstructured mesh
(the mesh data is in the archive)
20694x512
Double precision, binary, little-endian
1.2 GB
The files are stored in .bp format, you need to install ADIOS2 support to read them.
Source: Michael Churchill, Princeton Plasma Physics Laboratory (PPPL). Copyright: Free to use but need to check with Michael Churchill (rchurchi@pppl.gov) before publishing results.
369,357 steps
2D time-series data (movie), 80x64 image
Double precision, binary, little-endian
4.1 GB
The files are stored in .bp format, you need to install ADIOS1 to read them.
Source: Brown University
1D, double precision, binary, little-endian
(3 datasets with 3 different regularity)
256 MB
256 MB
256 MB
| 8-bit Entropy | 32-bit Entropy | |
|---|---|---|
| avg | 7.5 | 23 |
| min | 7.5 | 23 |
| max | 7.6 | 23 |
SZ (Compress): sz -z -d -i sample_r_B_0.5_26.d64 -M REL -R 1E-2 -1 33554433
SZ (Decompress): sz -x -d -i sample_r_B_0.5_26.d64 -s sample_r_B_0.5_26.d64.sz -1 33554433 -a
ZFP: zfp -d -1 33554433 -i sample_r_B_0.5_26.d64 -z sample_r_B_0.5_26.d64.zfp -o sample_r_B_0.5_26.d64.zfp.out -a 1E-2 -s
LibPressio: pressio -b compressor=$COMP -i sample_r_B_0.5_26.d64 -d 33554433 -t double -o rel=1e-2 -m time -m size -M all
where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker: ./runZCCase.sh -d REL Brown-samples raw-data-dir d64 33554433