New AI method shrinks scientific datasets by up to 100-fold
1/29
Researchers at SLAC National Accelerator Laboratory have developed a neural-network-based compression method that cut scientific data volumes by factors of 10 to 100 in tests, depending on the input data and the required level of accuracy. The key idea is not to compress every part of the data equally, but to preserve structures across different spatial scales that could prove critical to scientific analysis. The study was published in Nature Machine Intelligence.
Scientific instruments are producing more data than infrastructure can absorb
The motivation behind the new method comes from a very practical problem in modern experimental science. Sensor resolution and measurement rates are increasing so quickly that next-generation instruments are beginning to generate data streams that are increasingly expensive to store in full, transfer and process later.
Conventional lossy compression can reduce file sizes, but it may also remove low-amplitude signals or fine structures. In an ordinary photograph, that loss may not matter. In a scientific measurement, however, a tiny detail can contain exactly the physical information researchers are looking for.
SLAC points to the small speckles that appear in X-ray measurements as one example. These structures can reveal internal ordering, defects or dynamics within a material. If a compression algorithm treats them as noise, it can reduce not only the file size but also the scientific value of the measurement.
Wavelet analysis separates information by scale
The method published in Nature Machine Intelligence is called WIEN-INR. It combines wavelet analysis with implicit neural representations.
First, wavelet analysis separates the dataset into components at different spatial scales, distinguishing large-scale structures from fine detail. A neural network then learns to represent the information at those different scales compactly.
This architecture helps avoid a common problem with general-purpose compression, where a broad background structure and a tiny but scientifically important feature may be treated much the same way.
According to SLAC researcher Yuan Ni, the method typically reduced file sizes by factors of 10 to 100, depending on the source data and the desired quality. That result should not, however, be interpreted as fully lossless compression. It is a controlled trade-off between data volume and signal fidelity, designed to preserve fine features that matter for analysis better than a generic compression approach.
That is a more accurate description than saying the system “does not lose any data at all”.
The system can decode only the region you need
A second major strength of the method appears when the data is used later. If a researcher needs only a small region from a large compressed dataset, there is no need to reconstruct the entire file.
WIEN-INR can decode only a selected region of interest and do so at different scales and resolutions. SLAC notes that decompressing a complete large file with a conventional codec can take minutes, hours or even days. Selective decoding avoids much of that work.
In large scientific computing environments, this capability may be just as important as the compression ratio itself. The problem is not simply the cost of storage. Moving data between storage systems, compute nodes and research institutions consumes network capacity, energy and time.
The method worked across very different types of data
The researchers did not limit their tests to a single type of X-ray dataset. They applied the method to molecular and materials measurements, solar magnetic-field data and ordinary photographs. The neural network was able to adapt to different data types and learn the features that mattered for each.
The authors do not present WIEN-INR as a universal replacement for existing compression or data-reduction methods. Instead, it can work alongside them as an additional option in cases where recovering fine-scale structure later is particularly important.
Researchers from the University of California, Davis and Carnegie Mellon University also contributed to the study. The team trained its neural networks using the Perlmutter supercomputer at the US Department of Energy’s NERSC facility. The paper, “Multi-resolution enhancement for full-spectrum neural representations”, was published in Nature Machine Intelligence on 24 August 2026.
Europe’s research infrastructure faces the same challenge
The potential of the method extends far beyond any single instrument at SLAC. European infrastructure for particle physics, synchrotron science, astronomy, climate research and satellite observation is also generating rapidly growing volumes of data.
What matters here is the principle rather than any one specific algorithm. When an instrument produces more information than it is practical to store and move in full, the data pipeline has to decide increasingly early which information to preserve and at what level of fidelity.
AI’s role in science is therefore no longer limited to finding patterns after an experiment has taken place. The next important step may be representing scientific data in a form that dramatically reduces storage and computing demands while preserving the fine structures that could turn out to matter for the next discovery.
