Datasets

Reference scientific datasets for benchmarking data reduction techniques. Click on command examples to expand.

Note: This table will be augmented with metrics that matter for users of these datasets as well as recommended settings for error control (lossy compression).

CESM-ATM

Climate simulation

Source: Mark Taylor (SNL)

Format

Dataset1: 79 fields, 2D, 1800x3600
Dataset2: 1 field, 3D, 26x1800x3600
Both single precision, binary

Size

Dataset1 (raw): 1.47GB
Dataset1 (cleared): 1.47GB
Dataset2: 17GB
(cleared data zeroed all background data)

Entropy

8-bit Entropy32-bit Entropy
avg6.519.8
min1.52.3
max7.822.6

Download Links

Dataset1 (raw) Dataset1 (cleared) Dataset2 (raw) Metadata Dataset1 Property Dataset2 Property
Command Examples 
SZ (Compress):   sz -z -f -i CLDHGH_1_1800_3600.f32 -M REL -R 1E-2 -2 3600 1800
SZ (Decompress): sz -x -f -i CLDHGH_1_1800_3600.f32 -2 3600 1800 -s CLDHGH_1_1800_3600.f32.sz -a
ZFP:              zfp -f -i CLDHGH_1_1800_3600.f32 -z CLDHGH_1_1800_3600.f32.zfp -o CLDHGH_1_1800_3600.f32.zfp.out -2 3600 1800 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i CLDHGH_1_1800_3600.f32 -d 3600 -d 1800 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL CESM-ATM raw-data-dir f32 3600 1800

EXAALT

Molecular dynamics simulation

Source: EXAALT team

This dataset has been approved for unlimited release by Los Alamos National Laboratory and has been assigned LA-UR-18-25670.

Format

6 fields: x, y, z, vx, vy, vz
Each field stored separately
Single precision, binary, little-endian

Size

Dataset1: 60 MB
Dataset2: 973 MB
Dataset3: 2.4 GB

Entropy

Data1

8-bit Entropy32-bit Entropy
avg7.319.9
min718.4
max7.420.4

Data2

8-bit Entropy32-bit Entropy
avg6.715.4
min69.40
max7.120.1

Data3

8-bit Entropy32-bit Entropy
avg6.916.3
min6.815.6
max7.217.6

Download Links

Dataset1 Metadata1 Property1 Dataset2 Metadata2 Property2 Dataset3 Metadata3 Property3
Command Examples 
SZ (Compress):   sz -z -f -i xx.f32 -M REL -R 1E-2 -1 2869440
SZ (Decompress): sz -x -f -i xx.f32 -1 2869440 -s xx.f32.sz -a
ZFP:              zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -1 2869440 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i xx.f32 -d 2869440 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL EXAALT raw-data-dir f32 2869440

Hurricane ISABEL

Weather simulation

Source: IEEE Visualization 2004 Contest (http://vis.computer.org/vis2004contest/data.html)

Format

13 fields, 3D, 100x500x500
Single precision, binary
(cleared dataset by replacing background by 0)

Size

1.25GB

Entropy

8-bit Entropy32-bit Entropy
avg5.918.9
min0.71.3
max7.524.2

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -f -i Pf48.bin.f32 -M REL -R 1E-2 -3 500 500 100
SZ (Decompress): sz -x -f -i Pf48.bin.f32 -3 500 500 100 -s Pf48.bin.f32.sz -a
ZFP:              zfp -f -i Pf48.bin.f32 -3 500 500 100 -z Pf48.bin.f32.zfp -o Pf48.bin.f32.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i Pf48.bin.f32 -d 500 -d 500 -d 100 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL Hurricane raw-data-dir f32 500 500 100

EXAFEL

Images from the LCLS instrument

Source: LCLS

Format

1 field, 2D
Single precision
HDF5 and binary

Size

Dataset1: 51 MB
Dataset2: 8.5 GB
Dataset3: 1 GB

Entropy

8-bit Entropy32-bit Entropy
value7.425.7

Download Links

Dataset1 Dataset2 Metadata Property Dataset3 Metadata (Dataset3)
Command Examples 
SZ (Compress):   sz -z -f -i EXAFEL-LCLS-986x32x185x388.f32 -M REL -R 1E-2 -2 388 5837120
SZ (Decompress): sz -x -f -i EXAFEL-LCLS-986x32x185x388.f32 -s EXAFEL-LCLS-986x32x185x388.f32.sz -2 388 5837120 -a
ZFP:              zfp -f -a 1E-2 -i EXAFEL-LCLS-986x32x185x388.f32 -z EXAFEL-LCLS-986x32x185x388.f32.zfp -o EXAFEL-LCLS-986x32x185x388.f32.zfp.out -2 388 5837120 -s
LibPressio:       pressio -b compressor=$COMP -i EXAFEL-LCLS-986x32x185x388.f32 -d 5837120 -d 388 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL EXAFEL raw-data-dir f32 388 5837120

HACC

Cosmology: particle simulation

Source: HACC team (ECP EXASKY)

Format

1 snapshot: 6 fields (x, y, z, vx, vy, vz)
Each field stored separately
Single precision, binary, little-endian

Size

Dataset1: 19 GB
Dataset2: 5 GB

Entropy

8-bit Entropy32-bit Entropy
avg7.225.3
min7.124.7
max7.425.9

Download Links

Dataset1 Metadata1 Property1 Dataset2 Metadata2 Property2
Command Examples 
SZ (Compress):   sz -z -f -i xx.f32 -M ABS -A 0.003 -1 1073726487
SZ (Decompress): sz -x -f -i xx.f32 -s xx.f32.sz -1 1073726487 -a
ZFP:              zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -a 0.003 -1 1073726487
LibPressio:       pressio -b compressor=$COMP -i xx.f32 -d 1073726487 -t float -o abs=3e-3 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f ABS HACC raw-data-dir f32 1073726487

NYX

Cosmology: Adaptive mesh hydrodynamics + N-body cosmological simulation

Source: Lukic et al. "methods: numerical, intergalactic medium, quasars: absorption lines, large-scale structure of universe", journal of Monthly Notices of Royal Astronomical Society

Format

6 fields, 3D, 512x512x512
Single precision, binary, little-endian

Size

2.7 GB

Entropy

8-bit Entropy32-bit Entropy
avg7.225.1
min7.024
max7.425.9

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -f -i temperature.f32 -M REL -R 1E-2 -3 512 512 512
SZ (Decompress): sz -x -f -i temperature.f32 -s temperature.f32.sz -3 512 512 512 -a
ZFP:              zfp -f -i temperature.f32 -z temperature.f32.zfp -o temperature.f32.zfp.out -3 512 512 512 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i temperature.f32 -d 512 -d 512 -d 512 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL NYX raw-data-dir f32 512 512 512

NWChem

Two-electron repulsion integrals computed over Gaussian-type orbital basis sets

Source: Example molecular 2-electron integral values generated by libint, a library developed by the Valeev research group at Virginia Tech. See https://github.com/evaleev/libint. Libint is an integral evaluation option in NWChemEx. (https://github.com/evaleev/libint)

Format

3 fields, 1D
Double precision, binary, little-endian

Size

16GB

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -d -i 631-tst.bin.d64 -M REL -R 1E-2 -1 102953248
SZ (Decompress): sz -x -d -i 631-tst.bin.d64 -s 631-tst.bin.d64.sz -1 102953248 -a
ZFP:              zfp -d -i 631-tst.bin.d64 -z 631-tst.bin.d64.zfp -o 631-tst.bin.d64.zfp.out -a 1E-2 -1 102953248 -s
LibPressio:       pressio -b compressor=$COMP -i 631-tst.bin.d64 -d 102953248 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL NWCHEM raw-data-dir d64 102953248

SCALE-LETKF

Climate simulation

Source: Simulation data generated by The Local Ensemble Transform Kalman Filter (LETKF) data assimilation package for the SCALE-RM weather model (provided by RIKEN). Contact: Guo-yuan Lien (guoyuan.lien@gmail.com)

Format

13 fields, 3D
Single precision, binary, little-endian

Size

4.9GB

Entropy

8-bit Entropy32-bit Entropy
avg6.721.7
min4.610.2
max7.426.1

Download Links

Dataset Metadata Property Dataset2 (log-scale)
Command Examples 
SZ (Compress):   sz -f -z -i T-98x1200x1200.f32 -M REL -R 1E-2 -3 1200 1200 98
SZ (Decompress): sz -f -x -i T-98x1200x1200.f32 -s T-98x1200x1200.f32.sz -3 1200 1200 98 -a
ZFP:              zfp -f -i T-98x1200x1200.f32 -z T-98x1200x1200.f32.zfp -o T-98x1200x1200.f32.zfp.out -a 1E-2 -3 1200 1200 98 -s
LibPressio:       pressio -b compressor=$COMP -i T-98x1200x1200.f32 -d 1200 -d 1200 -d 98 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL SCALE-LETKF raw-data-dir f32 1200 1200 98

QMCPACK

Many-body ab initio Quantum Monte Carlo (electronic structure of atoms, molecules, and solids)

Source: QMCPACK performance test (contact: Ye Luo, yeluo@anl.gov)

Format

1 field, 288 orbitals, 3D, 69x69x115
Single precision, binary, little-endian

Size

1 GB

Entropy

8-bit Entropy32-bit Entropy
value7.626.1

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -f -i einspline_288_115_69_69.pre.f32 -M REL -R 1E-2 -3 69 69 33120
SZ (Decompress): sz -x -f -i einspline_288_115_69_69.pre.f32 -s einspline_288_115_69_69.pre.f32.sz -3 69 69 33120 -a
ZFP:              zfp -f -i einspline_288_115_69_69.pre.f32 -z einspline_288_115_69_69.pre.f32.zfp -o einspline_288_115_69_69.pre.f32.zfp.out -a 1E-2 -3 69 69 33120
LibPressio:       pressio -b compressor=$COMP -i einspline_288_115_69_69.pre.f32 -d 69 -d 69 -d 33120 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL QMCPack raw-data-dir f32 69 69 33120

Miranda

Hydrodynamics code for large turbulence simulations

Source: Hydrodynamics code for large turbulence simulations (contact: Peter Lindstrom, lindstrom2@llnl.gov)

Format

3D, 7 fields, each field is 256x384x384
Double precision, binary, little-endian

Size

Small (256x384x384): 1.87GB
Big (3072x3072x3072): 106GB

Entropy

8-bit Entropy32-bit Entropy
avg7.122.5
min3.627.2
max7.725.1

Download Links

Dataset (Small) Metadata (Small) Property (Small) Dataset (Big) Metadata (Big) Property (Big)
Command Examples 
SZ (Compress):   sz -z -d -i density.d64 -M REL -R 1E-2 -3 384 384 256
SZ (Decompress): sz -x -d -i density.d64 -s density.d64.sz -3 384 384 256 -a
ZFP:              zfp -d -3 384 384 256 -i density.d64 -z density.d64.zfp -o density.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i density.d64 -d 384 -d 384 -d 256 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL Miranda raw-data-dir d64 384 384 256

S3D

Combustion simulation

Source: Kolla, Hemanth (hnkolla@sandia.gov)

Format

11 fields, 3D, 500x500x500
Double precision, binary, little-endian

Size

44 GB

Entropy

8-bit Entropy32-bit Entropy
avg7.621.9
min7.621.8
max7.622

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -d -z -i stat_planar.1.1000E-03.field.d64 -M REL -R 1E-2 -3 500 500 500
SZ (Decompress): sz -d -x -i stat_planar.1.1000E-03.field.d64 -s stat_planar.1.1000E-03.field.d64.sz -3 500 500 500 -a
ZFP:              zfp -d -3 500 500 500 -i stat_planar.1.1000E-03.field.d64 -z stat_planar.1.1000E-03.field.d64.zfp -o stat_planar.1.1000E-03.field.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i stat_planar.1.1000E-03.field.d64 -d 500 -d 500 -d 500 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL S3D raw-data-dir f32 500 500 500

Basilisk-Turbulence

2D decaying turbulence simulation

Source: M. Xia, Y. Li, P. Jiao, B. Wang, X. Liang and H. Guo, "Time-Varying Vector Field Compression with Preserved Critical Point Trajectories," 2026 IEEE 42nd International Conference on Data Engineering (ICDE), Montreal, QC, Canada, 2026, pp. 2235-2249, doi: 10.1109/ICDE65706.2026.00168 (http://basilisk.fr/)

Format

4 fields, 3D
1501x1024x1024 & 1001x2048x2048
u_1024x1024x1501Re10k.f32
v_1024x1024x1501Re10k.f32
u_2048x2048x1001Re10k.f32
v_2048x2048x1001Re10k.f32
Single precision, binary, little-endian

Size

40GB

Entropy

8-bit Entropy32-bit Entropy
avg7.3626.45
min7.3526.39
max7.3826.53

Download Links

Dataset Metadata1 (1501x1024x1024) Metadata2 (1001x2048x2048) README Property
Command Examples 
SZ (Compress):   sz -f -z -i u_1024x1024x1501Re10k.f32 -M REL -R 1E-3 -3 1024 1024 1501
SZ (Decompress): sz -f -x -i u_1024x1024x1501Re10k.f32 -s u_1024x1024x1501Re10k.f32.sz -3 1024 1024 1501 -a
ZFP:              zfp -f -3 1024 1024 1501 -i u_1024x1024x1501Re10k.f32 -z u_1024x1024x1501Re10k.f32.zfp -o u_1024x1024x1501Re10k.f32.zfp.out -a 1E-3 -s

XGC

Fusion Simulation

Source: Princeton Plasma Physics Laboratory (PPPL) (https://jychoi-hpc.github.io/adios-python-docs/XGC-mesh-and-field-data.html)

Format

9 timesteps, 3D, unstructured mesh
(the mesh data is in the archive)
20694x512
Double precision, binary, little-endian

Size

1.2 GB

Download Links

Dataset (Adios2) Dataset (binary) Metadata

The files are stored in .bp format, you need to install ADIOS to read them.

NSTX GPI

Fusion Gas Puff Image (GPI) data

Source: Michael Churchill, Princeton Plasma Physics Laboratory (PPPL). Copyright: Free to use but need to check with Michael Churchill (rchurchi@pppl.gov) before publishing results.

Format

369,357 steps
2D time-series data (movie), 80x64 image
Double precision, binary, little-endian

Size

4.1 GB

Download Links

Dataset Metadata

The files are stored in .bp format, you need to install ADIOS to read them.

Brown Samples

Synthetic, generated to specified regularity

Source: Brown University

Format

1D, double precision, binary, little-endian
(3 datasets with 3 different regularity)

Size

256 MB
256 MB
256 MB

Entropy

8-bit Entropy32-bit Entropy
avg7.523
min7.523
max7.623

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -d -i sample_r_B_0.5_26.d64 -M REL -R 1E-2 -1 33554433
SZ (Decompress): sz -x -d -i sample_r_B_0.5_26.d64 -s sample_r_B_0.5_26.d64.sz -1 33554433 -a
ZFP:              zfp -d -1 33554433 -i sample_r_B_0.5_26.d64 -z sample_r_B_0.5_26.d64.zfp -o sample_r_B_0.5_26.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i sample_r_B_0.5_26.d64 -d 33554433 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL Brown-samples raw-data-dir d64 33554433