Datasets

Reference scientific datasets for benchmarking data reduction techniques. Click on command examples to expand.

FORMAT: Describes the data organization, including the number of fields in each data file and the data dimensions.
SIZE: Indicates the file size in bytes. Since most datasets are distributed as compressed .tar.gz archives, SIZE typically refers to the size of the corresponding .tar.gz file.

CESM-ATM

Climate simulation

Source: Mark Taylor (SNL)

Format

Dataset1: 79 fields, 2D, 1800x3600
Dataset2: 1 field, 3D, 26x1800x3600
Both single precision, binary

Size

Dataset1 (raw): 1.47GB
Dataset1 (cleared): 1.47GB
Dataset2: 17GB
(cleared data zeroed all background data)

Entropy

8-bit Entropy32-bit Entropy
avg6.519.8
min1.52.3
max7.822.6

Download Links

Dataset1 (raw) Dataset1 (cleared) Dataset2 (raw) Metadata Dataset1 Property Dataset2 Property
Command Examples 
SZ (Compress):   sz -z -f -i CLDHGH_1_1800_3600.f32 -M REL -R 1E-2 -2 3600 1800
SZ (Decompress): sz -x -f -i CLDHGH_1_1800_3600.f32 -2 3600 1800 -s CLDHGH_1_1800_3600.f32.sz -a
ZFP:              zfp -f -i CLDHGH_1_1800_3600.f32 -z CLDHGH_1_1800_3600.f32.zfp -o CLDHGH_1_1800_3600.f32.zfp.out -2 3600 1800 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i CLDHGH_1_1800_3600.f32 -d 3600 -d 1800 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL CESM-ATM raw-data-dir f32 3600 1800

EXAALT

Molecular dynamics simulation

Source: EXAALT team

This dataset has been approved for unlimited release by Los Alamos National Laboratory and has been assigned LA-UR-18-25670.

Format

6 fields: x, y, z, vx, vy, vz
Each field stored separately
Single precision, binary, little-endian

Size

Dataset1: 60 MB
Dataset2: 973 MB
Dataset3: 2.4 GB

Entropy

Data1

8-bit Entropy32-bit Entropy
avg7.319.9
min718.4
max7.420.4

Data2

8-bit Entropy32-bit Entropy
avg6.715.4
min69.40
max7.120.1

Data3

8-bit Entropy32-bit Entropy
avg6.916.3
min6.815.6
max7.217.6

Download Links

Dataset1 Metadata1 Property1 Dataset2 Metadata2 Property2 Dataset3 Metadata3 Property3
Command Examples 
SZ (Compress):   sz -z -f -i xx.f32 -M REL -R 1E-2 -1 2869440
SZ (Decompress): sz -x -f -i xx.f32 -1 2869440 -s xx.f32.sz -a
ZFP:              zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -1 2869440 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i xx.f32 -d 2869440 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL EXAALT raw-data-dir f32 2869440

Hurricane ISABEL

Weather simulation

Source: IEEE Visualization 2004 Contest (http://vis.computer.org/vis2004contest/data.html)

Format

13 fields, 3D, 100x500x500
Single precision, binary
(cleared dataset by replacing background by 0)

Size

Snapshot48: 1.25GB
CLOUDf: 575MB
Pf: 4GB
PRECIP: 3.9GB
QCLOUD: 3.7GB
QICE: 502MB
QRAIN: 3.5GB
QSNOW: 3.9GB
QVAPOR: 3.9GB
TC: 3.6GB
U: 4.1GB
V: 4.0GB
W: 4.2GB
CLOUD_log10: 533MB
QCLOUD_log10: 193MB
QGRAUP_log10: 3GB
QICE_log10: 464MB
QRAIN_log10: 2.8GB
QSNOW_log10: 3.3GB

Entropy

Snapshot48(all 13 fields)

8-bit Entropy32-bit Entropy
avg5.918.9
min0.71.3
max7.524.2

CLOUD(Snapshot1-48)

8-bit Entropy32-bit Entropy
avg1.53.3
min1.22.5
max2.56.2

P

8-bit Entropy32-bit Entropy
avg7.324.2
min7.324.1
max7.324.2

PRECIP

8-bit Entropy32-bit Entropy
avg7.322.8
min6.317.8
max7.524.1

QCLOUD

8-bit Entropy32-bit Entropy
avg3.610.6
min0.40.7
max7.423.4

QICE

8-bit Entropy32-bit Entropy
avg1.32.9
min1.12.2
max2.56.1

QRAIN

8-bit Entropy32-bit Entropy
avg6.720.0
min2.35.2
max7.423.1

QSNOW

8-bit Entropy32-bit Entropy
avg7.222.2
min6.217.5
max7.523.7

QVAPOR

8-bit Entropy32-bit Entropy
avg7.424.2
min7.424.2
max7.424.2

TC

8-bit Entropy32-bit Entropy
avg7.223.6
min7.223.5
max7.223.6

U

8-bit Entropy32-bit Entropy
avg7.424.2
min7.424.2
max7.424.2

V

8-bit Entropy32-bit Entropy
avg7.424.2
min7.324.2
max7.424.2

W

8-bit Entropy32-bit Entropy
avg7.424.2
min7.424.2
max7.524.3

Download Links

Snapshot48 Snapshot48-Metadata Snapshot48-Property CLOUD(Snapshot1-48) CLOUD-Property P(Snapshot1-48) P-Property PRECIP(Snapshot1-48) PRECIP-Property QCLOUD(Snapshot1-48) QCLOUD-Property QICE(Snapshot1-48) QICE-Property QRAIN(Snapshot1-48) QRAIN-Property QSNOW(Snapshot1-48) QSNOW-Property QVAPOR(Snapshot1-48) QVAPOR-Property TC(Snapshot1-48) TC-Property U(Snapshot1-48) U-Property V(Snapshot1-48) V-Property W(Snapshot1-48) W-Property CLOUD_log10(Snapshot1-48) CLOUD_log10-Property QCLOUD_log10(Snapshot1-48) QCLOUD_log10-Property QGRAUP_log10(Snapshot1-48) QGRAUP_log10-Property QICE_log10(Snapshot1-48) QICE_log10-Property QRAIN_log10(Snapshot1-48) QRAIN_log10-Property QSNOW_log10(Snapshot1-48) WSNOW_log10-Property
Command Examples 
SZ (Compress):   sz -z -f -i Pf48.bin.f32 -M REL -R 1E-2 -3 500 500 100
SZ (Decompress): sz -x -f -i Pf48.bin.f32 -3 500 500 100 -s Pf48.bin.f32.sz -a
ZFP:              zfp -f -i Pf48.bin.f32 -3 500 500 100 -z Pf48.bin.f32.zfp -o Pf48.bin.f32.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i Pf48.bin.f32 -d 500 -d 500 -d 100 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL Hurricane raw-data-dir f32 500 500 100

EXAFEL

Images from the LCLS instrument

Source: LCLS

Format

1 field, 2D
Single precision
HDF5 and binary

Size

Dataset1: 51 MB
Dataset2: 8.5 GB
Dataset3: 1 GB

Entropy

8-bit Entropy32-bit Entropy
value7.425.7

Download Links

Dataset1 Dataset2 Metadata Property Dataset3 Metadata (Dataset3)
Command Examples 
SZ (Compress):   sz -z -f -i EXAFEL-LCLS-986x32x185x388.f32 -M REL -R 1E-2 -2 388 5837120
SZ (Decompress): sz -x -f -i EXAFEL-LCLS-986x32x185x388.f32 -s EXAFEL-LCLS-986x32x185x388.f32.sz -2 388 5837120 -a
ZFP:              zfp -f -a 1E-2 -i EXAFEL-LCLS-986x32x185x388.f32 -z EXAFEL-LCLS-986x32x185x388.f32.zfp -o EXAFEL-LCLS-986x32x185x388.f32.zfp.out -2 388 5837120 -s
LibPressio:       pressio -b compressor=$COMP -i EXAFEL-LCLS-986x32x185x388.f32 -d 5837120 -d 388 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL EXAFEL raw-data-dir f32 388 5837120

HACC

Cosmology: particle simulation

Source: HACC team (ECP EXASKY): please cite this paper ([Heitmann et al., Astrophys.J.Suppl. 252 (2021) 2, 19](https://inspirehep.net/literature/1799008)) if you are using HACC data. (https://www.alcf.anl.gov/science/case-studies/opencosmo-community-portal-cosmological-simulations)

Format

1 snapshot: 6 fields (x, y, z, vx, vy, vz)
Each field stored separately
Single precision, binary, little-endian

Size

Dataset1: 19 GB
Dataset2: 5 GB
Dataset3: 18.12 GB
multi-timestep-region1: 13 GB
multi-timestep-region2: 14 GB
multi-timestep-region3: 13 GB
multi-timestep-region4: 13 GB
multi-timestep-region5: 13 GB
multi-timestep-region6: 13 GB

Entropy

Dataset1

8-bit Entropy32-bit Entropy
avg7.225.3
min7.124.7
max7.425.9

Dataset3 (potentially super big)

8-bit Entropy32-bit Entropy
avg6.620.6
min6.519.0
max6.821.5

Download Links

Dataset1 Metadata1 Property1 Dataset2 Metadata2 Property2 Dataset3 Metadata3 Property3 multi-timestep-region1 multi-timestep-region2 multi-timestep-region3 multi-timestep-region4 multi-timestep-region5 multi-timestep-region6
Command Examples 
SZ (Compress):   sz -z -f -i xx.f32 -M ABS -A 0.003 -1 1073726487
SZ (Decompress): sz -x -f -i xx.f32 -s xx.f32.sz -1 1073726487 -a
ZFP:              zfp -f -i xx.f32 -z xx.f32.zfp -o xx.f32.zfp.out -a 0.003 -1 1073726487
LibPressio:       pressio -b compressor=$COMP -i xx.f32 -d 1073726487 -t float -o abs=3e-3 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f ABS HACC raw-data-dir f32 1073726487

NYX

Cosmology: Adaptive mesh hydrodynamics + N-body cosmological simulation

Source: Zarija Lukic (LBL, Berkeley)

Format

6 fields, 3D, 512x512x512
Single precision, binary, little-endian

Size

Dataset1: 2.7 GB
Dataset1(log): 2.7 GB

Entropy

Dataset1(512x512x512)

8-bit Entropy32-bit Entropy
avg7.225.1
min7.024
max7.425.9

Dataset1-log(512x512x512)

8-bit Entropy32-bit Entropy
avg7.325.1
min7.024.0
max7.425.9

Download Links

Dataset1(512x512x512) Metadata1(512x512x512) Property1(512x512x512) Dataset2(log)(512x512x512) Metadata2(log)(512x512x512) Property2(log)(512x512x512)
Command Examples 
SZ (Compress):   sz -z -f -i temperature.f32 -M REL -R 1E-2 -3 512 512 512
SZ (Decompress): sz -x -f -i temperature.f32 -s temperature.f32.sz -3 512 512 512 -a
ZFP:              zfp -f -i temperature.f32 -z temperature.f32.zfp -o temperature.f32.zfp.out -3 512 512 512 -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i temperature.f32 -d 512 -d 512 -d 512 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL NYX raw-data-dir f32 512 512 512

NWChem

Two-electron repulsion integrals computed over Gaussian-type orbital basis sets

Source: Example molecular 2-electron integral values generated by libint, a library developed by the Valeev research group at Virginia Tech. See https://github.com/evaleev/libint. Libint is an integral evaluation option in NWChemEx. (https://github.com/evaleev/libint)

Format

3 fields, 1D
Double precision, binary, little-endian

Size

16GB

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -d -i 631-tst.bin.d64 -M REL -R 1E-2 -1 102953248
SZ (Decompress): sz -x -d -i 631-tst.bin.d64 -s 631-tst.bin.d64.sz -1 102953248 -a
ZFP:              zfp -d -i 631-tst.bin.d64 -z 631-tst.bin.d64.zfp -o 631-tst.bin.d64.zfp.out -a 1E-2 -1 102953248 -s
LibPressio:       pressio -b compressor=$COMP -i 631-tst.bin.d64 -d 102953248 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL NWCHEM raw-data-dir d64 102953248

SCALE-LETKF

Climate simulation

Source: Simulation data generated by The Local Ensemble Transform Kalman Filter (LETKF) data assimilation package for the SCALE-RM weather model (provided by RIKEN). Contact: Guo-yuan Lien (guoyuan.lien@gmail.com)

Format

13 fields, 3D
Single precision, binary, little-endian

Size

4.9GB

Entropy

8-bit Entropy32-bit Entropy
avg6.721.7
min4.610.2
max7.426.1

Download Links

Dataset Metadata Property Dataset2 (log-scale)
Command Examples 
SZ (Compress):   sz -f -z -i T-98x1200x1200.f32 -M REL -R 1E-2 -3 1200 1200 98
SZ (Decompress): sz -f -x -i T-98x1200x1200.f32 -s T-98x1200x1200.f32.sz -3 1200 1200 98 -a
ZFP:              zfp -f -i T-98x1200x1200.f32 -z T-98x1200x1200.f32.zfp -o T-98x1200x1200.f32.zfp.out -a 1E-2 -3 1200 1200 98 -s
LibPressio:       pressio -b compressor=$COMP -i T-98x1200x1200.f32 -d 1200 -d 1200 -d 98 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL SCALE-LETKF raw-data-dir f32 1200 1200 98

QMCPACK

Many-body ab initio Quantum Monte Carlo (electronic structure of atoms, molecules, and solids)

Source: QMCPACK performance test (contact: Ye Luo, yeluo@anl.gov)

Format

1 field, 288 orbitals, 3D, 69x69x115
Single precision, binary, little-endian

Size

1 GB

Entropy

8-bit Entropy32-bit Entropy
value7.626.1

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -f -i einspline_288_115_69_69.pre.f32 -M REL -R 1E-2 -3 69 69 33120
SZ (Decompress): sz -x -f -i einspline_288_115_69_69.pre.f32 -s einspline_288_115_69_69.pre.f32.sz -3 69 69 33120 -a
ZFP:              zfp -f -i einspline_288_115_69_69.pre.f32 -z einspline_288_115_69_69.pre.f32.zfp -o einspline_288_115_69_69.pre.f32.zfp.out -a 1E-2 -3 69 69 33120
LibPressio:       pressio -b compressor=$COMP -i einspline_288_115_69_69.pre.f32 -d 69 -d 69 -d 33120 -t float -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL QMCPack raw-data-dir f32 69 69 33120

Miranda

Hydrodynamics code for large turbulence simulations

Source: Hydrodynamics code for large turbulence simulations (contact: Peter Lindstrom, lindstrom2@llnl.gov)

Format

3D, 7 fields, each field is 256x384x384
Double precision, binary, little-endian

Size

Small (256x384x384): 1.87GB
Big (3072x3072x3072): 106GB

Entropy

8-bit Entropy32-bit Entropy
avg7.122.5
min3.627.2
max7.725.1

Download Links

Dataset (Small) Metadata (Small) Property (Small) Dataset (Big) Metadata (Big) Property (Big)
Command Examples 
SZ (Compress):   sz -z -d -i density.d64 -M REL -R 1E-2 -3 384 384 256
SZ (Decompress): sz -x -d -i density.d64 -s density.d64.sz -3 384 384 256 -a
ZFP:              zfp -d -3 384 384 256 -i density.d64 -z density.d64.zfp -o density.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i density.d64 -d 384 -d 384 -d 256 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL Miranda raw-data-dir d64 384 384 256

S3D

Combustion simulation

Source: Kolla, Hemanth (hnkolla@sandia.gov)

Format

11 fields, 3D, 500x500x500
Double precision, binary, little-endian

Size

44 GB

Entropy

8-bit Entropy32-bit Entropy
avg7.621.9
min7.621.8
max7.622

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -d -z -i stat_planar.1.1000E-03.field.d64 -M REL -R 1E-2 -3 500 500 500
SZ (Decompress): sz -d -x -i stat_planar.1.1000E-03.field.d64 -s stat_planar.1.1000E-03.field.d64.sz -3 500 500 500 -a
ZFP:              zfp -d -3 500 500 500 -i stat_planar.1.1000E-03.field.d64 -z stat_planar.1.1000E-03.field.d64.zfp -o stat_planar.1.1000E-03.field.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i stat_planar.1.1000E-03.field.d64 -d 500 -d 500 -d 500 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -f REL S3D raw-data-dir f32 500 500 500

Basilisk-Turbulence

2D decaying turbulence simulation

Source: M. Xia, Y. Li, P. Jiao, B. Wang, X. Liang and H. Guo, "Time-Varying Vector Field Compression with Preserved Critical Point Trajectories," 2026 IEEE 42nd International Conference on Data Engineering (ICDE), Montreal, QC, Canada, 2026, pp. 2235-2249, doi: 10.1109/ICDE65706.2026.00168 (http://basilisk.fr/)

Format

4 fields, 3D
1501x1024x1024 & 1001x2048x2048
u_1024x1024x1501Re10k.f32
v_1024x1024x1501Re10k.f32
u_2048x2048x1001Re10k.f32
v_2048x2048x1001Re10k.f32
Single precision, binary, little-endian

Size

40GB

Entropy

8-bit Entropy32-bit Entropy
avg7.3626.45
min7.3526.39
max7.3826.53

Download Links

Dataset Metadata1 (1501x1024x1024) Metadata2 (1001x2048x2048) README Property
Command Examples 
SZ (Compress):   sz -f -z -i u_1024x1024x1501Re10k.f32 -M REL -R 1E-3 -3 1024 1024 1501
SZ (Decompress): sz -f -x -i u_1024x1024x1501Re10k.f32 -s u_1024x1024x1501Re10k.f32.sz -3 1024 1024 1501 -a
ZFP:              zfp -f -3 1024 1024 1501 -i u_1024x1024x1501Re10k.f32 -z u_1024x1024x1501Re10k.f32.zfp -o u_1024x1024x1501Re10k.f32.zfp.out -a 1E-3 -s
LibPressio:       pressio -i u_1024x1024x1501Re10k.f32 -o rel=1E-3 -d 1024 -d 1024 -d 1501 -b compressor=$COMP
                      where $COMP can be sz, zfp, sz3, mgard, etc.

XGC

Fusion Simulation

Source: Princeton Plasma Physics Laboratory (PPPL) (https://jychoi-hpc.github.io/adios-python-docs/XGC-mesh-and-field-data.html)

Format

9 timesteps, 3D, unstructured mesh
(the mesh data is in the archive)
20694x512
Double precision, binary, little-endian

Size

1.2 GB

Download Links

Dataset (Adios2) Dataset (binary) Metadata

The files are stored in .bp format, you need to install ADIOS2 support to read them.

NSTX GPI

Fusion Gas Puff Image (GPI) data

Source: Michael Churchill, Princeton Plasma Physics Laboratory (PPPL). Copyright: Free to use but need to check with Michael Churchill (rchurchi@pppl.gov) before publishing results.

Format

369,357 steps
2D time-series data (movie), 80x64 image
Double precision, binary, little-endian

Size

4.1 GB

Download Links

Dataset Metadata

The files are stored in .bp format, you need to install ADIOS1 to read them.

Brown Samples

Synthetic, generated to specified regularity

Source: Brown University

Format

1D, double precision, binary, little-endian
(3 datasets with 3 different regularity)

Size

256 MB
256 MB
256 MB

Entropy

8-bit Entropy32-bit Entropy
avg7.523
min7.523
max7.623

Download Links

Dataset Metadata Property
Command Examples 
SZ (Compress):   sz -z -d -i sample_r_B_0.5_26.d64 -M REL -R 1E-2 -1 33554433
SZ (Decompress): sz -x -d -i sample_r_B_0.5_26.d64 -s sample_r_B_0.5_26.d64.sz -1 33554433 -a
ZFP:              zfp -d -1 33554433 -i sample_r_B_0.5_26.d64 -z sample_r_B_0.5_26.d64.zfp -o sample_r_B_0.5_26.d64.zfp.out -a 1E-2 -s
LibPressio:       pressio -b compressor=$COMP -i sample_r_B_0.5_26.d64 -d 33554433 -t double -o rel=1e-2 -m time -m size -M all
                      where $COMP can be sz, zfp, sz3, mgard, etc.
Z-checker:        ./runZCCase.sh -d REL Brown-samples raw-data-dir d64 33554433