Spatiotemporal reservoir resampling

Spatiotemporal reservoir resampling, commonly known as ReSTIR (from "Reservoir-based SpatioTemporal Importance Resampling"), is a collection of computer graphics techniques for generating and reusing samples during rendering. It was developed primarily to allow more realistic lighting in real-time rendering, because relatively few rays can be traced per pixel while maintaining an acceptable frame rate.^{[1]: 2 [2]} It can also be used to speed up off-line path tracing.

A 3D model (figurine of a dancer) rendered using simple Monte Carlo integration, resampled importance sampling (RIS), ReSTIR spatial reuse, and denoised ReSTIR. Only direct light was rendered, using an environment map to simulate a scene with many lights. All images except the denoised image are unbiased. At most 5 rays were traced per pixel in each image.
Top left: (Monte Carlo integration) 5 light samples per pixel, up to 5 rays traced to test shadowing
Top right: (RIS) 32 light samples, 1 ray traced to test shadowing; faster if light samples are cheap and tracing rays is expensive
Bottom left: Two ReSTIR spatial reuse passes (after initial light sampling using RIS), 4 additional rays per pixel
Bottom right: ReSTIR output processed using Intel Open Image Denoise

A zoomed out version of the above image

The first ReSTIR paper, published in 2020, provided algorithms for direct lighting, allowing scenes containing thousands of lights to be rendered in real time on a high-end GPU. Researchers later proposed versions for rendering indirect lighting (and more recently, motion blur and depth of field) and built up a framework of mathematical tools and notation conventions for analyzing how sample reuse may cause bias in the rendered image, along with practical methods for reducing or removing that bias. Versions for path tracing perform transformations of samples that improve sampling efficiency, such as reusing parts of paths closer to the light while modifying the portion closer to the camera (a technique called reconnection).

ReSTIR-related papers and talks have been presented every year at the SIGGRAPH conference since 2020.

One of the first games to incorporate ReSTIR into its rendering was Cyberpunk 2077.^[3]

Overview and motivation

According to Chris Wyman, one of the co-authors of the original paper, although developers commonly thought that bias was acceptable for real-time rendering, end users (e.g. gamers) are well-aware of the artifacts caused by bias and many have a negative opinion of common sample-reuse techniques such as temporal anti-aliasing (TAA), which may cause "ghosting" when the camera moves, and denoising, which causes blurring and other artifacts.^[2]

ReSTIR techniques can reduce or avoid these types of bias by reusing samples of the set of possible paths taken by light to reach the camera, instead of reusing rendered pixel color values (which are typically the average of multiple samples, discarding information such as the direction of the light). While other techniques reuse samples in a generic post-processing step, ReSTIR passes can test for shadowing, and reused samples are converted into pixel color values by rendering code that takes the characteristics of different materials into account (e.g. by implementing BRDFs). However the output of ReSTIR is noisy, and a denoising pass is typically still used.^[4]: 1

Stochastic ray tracing techniques such as path tracing need to average multiple samples (produced by tracing individual rays) in order to render a visually acceptable image. When using a simple unbiased renderer based on Monte Carlo integration, halving the deviation of the result (apparent as "noise" in the image) requires multiplying the number of samples by four, meaning that a rapidly increasingly number of samples is needed to improve quality,^{[5]: 2.1.4 [3]: 198} Standard ways to mitigate this problem include importance sampling (which requires finding improved sampling distributions for specific situations),^[6]: 2.2.2 and quasi-Monte Carlo integration (which usually still requires tracing a large number of rays).^[7] ReSTIR offers a solution that multiplies the effective number of samples while tracing a fixed number of additional rays per frame.^[1]: 1

Temporal reuse multiplies the effective sample count by the number of frames rendered. Spatial reuse multiplies the effective count by the number of neighboring pixels examined. These two types of reuse can be combined, allowing spatial reuse to be applied recursively, which appears to offer an exponentially increasing effective sample count, however this is quickly limited by the size of the neighborhood used for spatial reuse.^{[1]: 6 [8]: 17 [3]: 198} Spatial reuse is also potentially less effective near shadow and object edges, especially for objects with fine geometric detail, and temporal reuse is limited by movement of the camera and scene elements.^{[1]: 14 [4]: 37}

Variations

Many variations of ReSTIR have been proposed that generalize or improve the original technique (which builds on an earlier method called RIS), specialize it for particular types of illumination or other visual effects, or allow incorporation into rendering algorithms other than standard path tracing. Some published versions are listed below.

Name(s)	Paper	Applicability
Resampled Importance Sampling (RIS)	Talbot et al. 2005 Importance Resampling for Global Illumination[9]	light sampling, BRDF sampling
ReSTIR, ReSTIR DI	Bitterli et al. 2020 Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting[1]	direct illumination
ReSTIR GI	Ouyang et al. 2021 ReSTIR GI: Effective Path Resampling for Real-Time Path Tracing[10]	indirect illumination; optimized for diffuse (Lambertian) surfaces
ReGIR	Boksansky et al. 2021 Rendering many lights with grid-based reservoirs[11]	light sampling during path tracing (uses 3D grid, not screen space)
Volumetric ReSTIR	Lin et al. 2021 Fast Volume Rendering with Spatiotemporal Reservoir Resampling[12]	"participating media" such as clouds and smoke
ReSTIR PT, Generalized Resampled Importance Sampling (GRIS)	Lin et al. 2022 Generalized resampled importance sampling: foundations of ReSTIR[13]	general path tracing; reflective (specular) surfaces; some caustics
Suffix ReSTIR	Kettunen et al. 2023 Conditional Resampled Importance Sampling and ReSTIR[14]	general path tracing; light reflecting from specular surfaces; fast camera motion
ReSTIR FG	Kern et al. 2024 ReSTIR FG: Real-Time Reservoir Resampled Photon Final Gathering[15]	caustics, glass-encased objects, general indirect illumination
Area ReSTIR	Zhang et al. 2024 Area ReSTIR: Resampling for Real-Time Defocus and Antialiasing[16]	general path tracing; depth of field (including bokeh) and fine structures such as hair and foliage
Reservoir Splatting	Liu et al. 2025 Reservoir Splatting for Temporal Path Resampling and Motion Blur[17]	general path tracing; moving objects and motion blur

Algorithms

Basic algorithm

ReSTIR uses a combination of resampled importance sampling (RIS) and weighted reservoir sampling (WRS) which the authors call streaming RIS. RIS processes samples from an initial probability distribution (e.g. a probability distribution for which a cheap sampling method exists) and generates samples in a new probability distribution (e.g. a sampling distribution that is optimal for rendering but is impractical to draw samples from directly). WRS allows this to be done while storing only a small number of samples in memory (in a data structure called a reservoir, which contains the samples and their importance sampling weights), which is especially helpful on a GPU. WRS also allows samples from multiple reservoirs to be combined ("merged") into a single reservoir; this is crucial for sample reuse.^{[1]: 3–4 [3]: 195–197}

Each pixel has a reservoir, typically containing only a single sample when ReSTIR is used for real-time rendering (some implementations use a larger number, e.g. four samples). The reservoir is typically initialized to a sample drawn using a simple method and is then updated by RIS steps and by reservoir merging, so that the pixel value produced by shading using the sample(s) currently in the reservoir, times the importance sampling weight for the sample, is always an unbiased estimate of the correct pixel value. If appropriate resampling steps are used, the variance of this estimate (or some function of it, typically the luminance of the RGB color value) decreases with each step.^[1]: 3–4

A possible sequence of steps performed for each frame, suitable for computing unbiased direct illumination (DI) is:^{[1]: 6, 9}

1. Perform reservoir resampling by drawing multiple light samples and using streaming RIS to choose one, using probabilities based on a target function, e.g. the luminance of the sample's contribution to the pixel. A weight is also computed for the sample. Typically, a single visibility check is performed here, after choosing a sample, setting the weight to 0 if the light is shadowed. Resampling (combined with the visibility check) ensures that the expected value of the weight times the sample brightness is the correct (unbiased) value for the pixel.
2. (temporal reuse) For each pixel, merge the sample(s) from the previous frame into the current reservoir. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in the previous frame's reservoirs may have a different target probability distribution if the objects, lights, or camera have moved.
3. (spatial reuse) For each pixel, choose one or more neighboring pixels and merge their samples into the current pixel's reservoir, again using streaming RIS. Multiple importance sampling (MIS) weights are used to avoid bias due to the fact that the samples in each pixel's reservoir have a different target probability distribution. Because computing unbiased MIS weights requires tracing additional rays (along with other work such as evaluating BRDFs), real-time rendering often uses only a single neighboring pixel.
4. Use the sample in each pixel's reservoir, along with its importance sampling weight, to determine the color of the pixel for the current frame. Alternatively, multiple samples examined during the preceding steps may be averaged and used to shade the pixel instead (decoupled shading and sampling).^{[8]: 11, 12, 14}

For direct lighting, the initial samples used in step 1 are typically drawn by importance sampling from the set of lights in a scene. The algorithm above (from the original ReSTIR paper) draws many lower-quality light samples (e.g. 32) using a fast method, without considering visibility, and chooses one using streaming RIS. Visibility is then tested for the final chosen sample.^{[1]: 6, 9} Considering visibility for each sample drawn would require tracing 32 rays, which would make it much more expensive. The intent is to reduce the number of rays traced,^[8]: 9 relying on the sample reuse in steps 2 and 3 to make up for the loss of quality caused by rejecting many of the rays due to shadowing. A large part of the initial efforts to optimize ReSTIR (to make it run in real-time on available hardware) went into reducing the cost of randomly sampling the lights.^{[8]: 8, 17} Glossy surfaces may require a larger number of samples, and combining light sampling with BRDF sampling (using MIS) may increase quality.^{[4]: 20–21}

Step 2 (temporal reuse) is sometimes skipped for off-line rendering, and the output of multiple repetitions of initial sampling and spatial reuse is averaged instead; this helps avoids artifacts due to correlations.^{[13]: 16–17} Step 3 (spatial reuse) may be repeated multiple times in a single frame.^[8]: 8

The neighboring pixels used in step 3 (spatial reuse) are chosen randomly, typically from a circle or square around the pixel, e.g. with a radius of 10–30 pixels.^{[1]: 9 [13]: 15} Heuristics may be used to choose neighboring pixels that are more likely to have similar surface and lighting characteristics to the target pixel, provided that these heuristics do not examine the samples in the reservoirs (which would introduce bias).^{[4]: 37–38}

In some variations, the neighbor samples used in step 3 come from the previous frame (i.e. this step performs spatiotemporal resampling rather than just spatial resampling), which can help improve performance on GPUs.^[8]: 10

Biased versions

The main cost of the above algorithm is usually tracing rays to check for visibility of lights (shadowing). Some of these shadowing checks may be skipped at the cost of introducing bias. Skipping visibility checks prevents correct MIS weights from being calculated, however since the use of correct MIS weights tends to increase the variance of samples in the scene (increasing visual noise) the biased version may be preferable anyway. If visibility checks are skipped during spatial reuse, the biased version is likely to produce darker pixels around the edges of objects, and around shadow edges.^[1]: 2

Multiple approaches for avoiding tracing additional rays have been explored. One possibility, considered in early versions of ReSTIR, is to not consider visibility at all during resampling and reuse, and test visibility only when using samples for rendering. This can be unbiased, but it produces much noisier output when there are shadows because the probability density function (PDF) of the samples is much further from the ideal PDF for importance sampling (lights are often sampled that have no contribution to the output pixel color). Compared to this method, a strategy called visibility reuse (assuming samples from prior frame and neighboring pixels are visible, and delaying proper visibility checks until after a sample has been chosen) provides a relatively cheap way to get a large increase in quality. Using heuristics to select neighboring pixels that are likely to have similar lighting helps mitigate the bias caused by visibility reuse.^{[1]: 6 [8]: 6, 11–13}

Confidence weights

Typically, new samples are introduced in each frame (e.g. by sampling the set of all lights). These new samples are merged with samples temporally reused from the previous frame. However, if the camera and scene objects have not moved (or a particular pixel was not affected by movement), the samples from the previous frame will be higher quality and should be replaced by new samples less frequently. Confidence weights are used to track how many effective samples contributed to each reservoir, and these weights are used to favor the sample already in the reservoir during reservoir merging. If temporal reuse is not possible for a pixel (due to "disocclusion" when objects or the camera move), the confidence weight is set to zero and the reservoir is always re-initialized to a new sample. The confidence weight is increased by 1 after merging a new sample into the reservoir.^{[13]: 16–17 [4]: 19–20}

During temporal reuse, samples from different reservoirs are merged, and their confidence weights are typically added to get the confidence weight for the chosen sample. However this will cause exponentially increasing confidence weights, which would soon prevent any new samples from being used, and so the value is usually capped at a value such as 20.^{[13]: 15–17 [4]: 19}

Backprojection

As in temporal anti-aliasing (TAA), temporal reuse is improved for moving images by reusing samples from the pixel in the previous frame that most closely matches the current pixel (known as backprojection or temporal reprojection). Motion vectors are typically used to determine this pixel.^{[1]: 9 [10]: 8} Unlike in TAA, an individual sample must be selected, and interpolation is not used. Area ReSTIR and Reservoir Splatting enhance this approach, allowing ReSTIR to perform antialiasing and motion blur.^[16][17]

Generalized (path tracing) algorithm

A limitation of the efficiency of the basic ReSTIR algorithm is that the probability density function (PDF) used for samples in the previous frame, or for neighboring pixels, may be different than the desired PDF for the pixel in the current frame. These different contexts, where the PDF may be different, are referred to as domains. In some cases, it is possible to increase the benefit of sample reuse by adjusting (shifting) the samples during reuse, so that the resulting PDF is likely to be a better fit for the current pixel. This is especially valuable when rendering specular reflection, where a small change in position or surface normal can cause a large change in the distribution of reflected rays (and thus a large change in the optimal PDF for sampling the reflected light). Any deterministic, invertible transformations of samples are allowed provided the probability values can be correctly adjusted when performing the transformation.^{[4]: 22, 28–29}

Shifting of samples can be handled by finding an appropriate transformation (with known inverse) between the samples used in different domains (different pixels or frames), and multiplying probabilities by the Jacobian determinant of the transformation function (or its inverse) when moving between domains. Probability value adjustments are required when computing multiple importance sampling (MIS) weights to remove bias during spatial and temporal reuse.^{[4]: 22–23}

To handle cases such as when a ray is specularly reflected or refracted multiple times before hitting the point being rendered, a variety of transformations of paths (rather than just ray directions) are used. These transformations are called shift mappings. The mappings most commonly used (or studied) for ReSTIR are listed below:

• reconnection shift (vertex copy) - produces a path that has a different ray from camera to first intersection point, and from the first intersection to the second intersection, but keeps remaining path segments leading to the light source unchanged (the point where the two portions of the paths meet is called the reconnection vertex). The modified segments of the path are deterministic functions of the view ray for the pixel, and the reconnection vertex. ReSTIR DI uses a simple case of the reconnection shift where the reconnection vertex is a point on a light source. The reconnection shift typically works well for diffuse surfaces.^{[4]: 22 [13]: 13, 17}
• half-vector copy - uses a deterministic way to modify segments close to the camera, allowing the reconnection shift to be used more effectively with surfaces that are highly reflective ("specular").^{[4]: 29 [13]: 13}
• random replay - uses the same random numbers as were used when generating the original path, but uses them to sample from different PDFs (e.g. when importance sampling the BRDF for each reflection). A single "seed" value for the random number generator determines the entire path, reducing storage requirements for the sample, however paths tend to diverge further away from the camera which can make the technique less effective.^[13]: 13
• hybrid shift - uses random replay for one or more segments close to the camera, along with a deterministic segment to connect to the reconnection vertex. The reconnection vertex is usually the first diffuse (or rough) reflection point found along the path. The remainder of the path is unchanged. This typically improves upon the reconnection shift for glossy surfaces. It tends to produce results similar to the half-vector copy shift, but is more widely applicable.^{[4]: 30 [13]: 13, 17}

For ReSTIR it is important to avoid needing to explicitly store a path (which may have arbitrary length) as a sample in a ReSTIR reservoir. For the random replay shift, only the seed for the random number generator needs to be stored. For other shifts, the location of the reconnection vertex (usually a point on a surface) is stored, and perhaps the amount and direction of incoming light (to reduce the amount of work needed when computing the light contribution of a shifted sample).^{[4]: 30 [13]: 15}

Challenges and limitations

Computational cost

Various implementation issues may limit the applicability of ReSTIR for real-time rendering except on high-end GPUs. These problems may be addressed by GPU performance improvements (for example as ray tracing acceleration improves and GPU cache sizes increase).

• For unbiased rendering, at least two additional rays must be traced for each pixel during each spatial and each temporal reuse step, and one ray must be traced to check visibility (shadowing) when introducing new samples. These figures are for direct illumination only (many more rays are needed when using ReSTIR for full path tracing). At the time ReSTIR was introduced (in 2019), this made unbiased rendering at full resolution infeasible for real-time rendering in games, because typically only a single ray (or less) could be traced per pixel while maintaining an acceptable frame rate (GPU ray tracing performance has since increased; as of 2025, 10–40 rays can be traced per pixel, but often only a single path can be traced per pixel on high-end GPUs).^{[8]: 9 [2]}

• Unbiased temporal reuse requires tracing rays using scene data from the previous frame, which may increase memory use, and may make ReSTIR more difficult to integrate into game engines.^{[4]: 19 [2]}

• Use of ReSTIR for direct illumination (ReSTIR DI) requires generating many independent samples from the full set of lights (emissive surfaces). If the number of lights is very large (or many triangles are needed to represent their geometry), the data for the lights requires a large amount of memory, and random sampling will require slow (uncacheable) memory accesses. This has been addressed by pre-generating sets of light samples shared by tiles of adjacent pixels in a single frame.^{[4]: 33–34 [8]: 8, 17}

Noise and correlation

Because ReSTIR is a stochastic technique (taking random samples) its output is noisy, and it is typically used with a denoising pass. Spatial reuse can cause unwanted correlations in the samples for neighboring pixels, visible as blotchiness, usually because relatively few "good" samples are found and then reused many times in a neighborhood.^[8]: 18 Denoisers may have difficultly dealing with these correlations, e.g. may treat them as surface detail, producing artifacts in the output.^[4]: 43 Some denoisers also expect the noise in consecutive frames to be independent, and cannot handle the temporal correlations introduced by ReSTIR.^[10]: 10

In some situations, unbiased ReSTIR can increase variance (squared magnitude of noise) compared to simpler techniques, for example around shadow edges when using ReSTIR for direct illumination (ReSTIR DI). Partly for this reason (in addition to performance considerations) biased rendering, which causes darkening around edges but has less visible noise, may be preferred.^[1]: 10

Color noise

An extreme example of color noise in a scene with three colors of light. In the top image one thousand samples were averaged. In the bottom image a single, weighted sample was used for rendering (selected using streaming RIS from one thousand samples); each pixel uses the color of a single light source, which produces very visible noise even though the pixels have approximately the correct brightness.

ReSTIR's streaming RIS technique retains a small number of samples in each reservoir, typically only a single sample, regardless of how many samples have been examined. This single sample (along with an importance sampling weight) is used to determine the final pixel color. If luminance is used as the target function during importance sampling, the rendered pixel value will converge to the correct brightness, however it will have a color determined by a single sample, which introduces "color noise" noticeable in scenes where light from different directions has different colors, and the pixel color will not converge to the correct value. This problem does not occur with standard Monte Carlo integration because multiple samples are averaged. Correlations between adjacent pixels may make this color noise more noticeable by producing "clumps" where the same sample was used.^{[12]: 15–16 [13]: 12,19 [16]: 11}

For real-time rendering, "decoupling" sample reuse and rendering, so up to three samples are used for determining pixel color, can reduce color noise, but sometimes increases noise because lower quality samples are used in addition to the sample chosen by reservoir merging.^{[8]: 11, 12, 14} For off-line rendering, color noise is usually reduced or avoided by using the average of the samples produced by many repetitions of ReSTIR, instead of using a single sample for rendering.^{[12]: 15–16 [13]: 12}

Relationship to other techniques

Denoising

The output of the Open Image Denoise NN-based denoiser, applied to the image at the beginning of the article. Top shows denoising of 5 sample Monte Carlo integration. Middle shows denoising of ReSTIR (RIS + spatial reuse) output. The bottom is a reference image using 4096 samples of Monte Carlo integration. The improved samples produced by ReSTIR allow the denoiser to reconstruct an image that appears more detailed and more similar to the reference image.

ReSTIR was developed while exploring ways to add ray tracing to real-time rendering (e.g. for games). Due to hardware limitations, only a relatively small number of rays can be traced per pixel per frame, even on high-end GPUs. This means that stochastic ray tracing techniques (for indirect lighting and realistic direct lighting) can only use a small number of samples, producing output that is very noisy, and a denoising pass is essential for acceptable output. One of the goals of the work that led to ReSTIR was to provide better input samples to denoisers, and early ReSTIR demos (created by a team at Nvidia) used denoisers.^[2]

Denoisers are implemented using techniques such as bilateral filters, or (increasingly) neural networks. Simpler denoisers work with pixel color values only, while more advanced implementations use additional data as input, including surface normal and information about the specular reflections and surface material. Denoisers are limited to working with information that is in the input values (they are not renderers, and cannot trace new rays), and samples generated by ReSTIR can provide more information.^{[1]: 2, 14}

Light sampling

Traditional path tracing uses data structures such as light trees to allow importance sampling of lights, favoring lights that are likely to appear brighter (contributing more light) at the point in the scene being rendered. The disadvantages of such techniques are that the data structures may be expensive to update if the lights move or change,^[1]: 2 may be expensive to query in GPU shader code, and generally do not capture shadowing information (at different points in the scene, different lights will be visible, and this can usually only be determined by tracing a "shadow ray" between the surface point and the point on the light).^{[1]: 10–12}

ReSTIR can be used as an alternative to these data structures, or as a way to augment them with shadowing information. Because the RIS portion of ReSTIR permits using lower-quality initial samples (which are then refined by resampling) ReSTIR may allow using light sampling data structures that are easier to update dynamically when lights move, and cheaper to query, without compromising quality. However, in scenes where there are many lights but few of them are visible at any given point, using a cheap initial sampling method with ReSTIR may not be an effective approach.^[8]: 18 Better light sampling techniques can be used to generate the initial samples, if the quality increase justifies the cost, but this may be counterproductive in practice (e.g. if many of the samples are shadowed or the amount of light reflected towards the camera is small).^{[1]: 9 [3]: 199}

Path guiding

Path guiding techniques for path tracing use data structures, built during an initial pass (or "warm up" phase) or dynamically during rendering, that help make sampling more efficient by preferentially sampling path directions that are likely to contribute more light. These data structures (which may contain a representation of approximate probability distributions of paths) are usually in world space (data is associated with 3D locations or volumes in the scene) rather than screen space (data is associated with pixels or blocks of pixels), making them complex to update and query.^[1]: 12

Chris Wyman (co-author of the original ReSTIR paper) has said that ReSTIR can be viewed as a kind of path guiding, working in screen space. Sharing of statistical information between frames and neighboring pixels allows the distribution of samples in each reservoir to converge to an optimal sampling distribution.^{[2][4]: 8, 15}

A combination of ReSTIR and path guiding, called ReSTIR-PG, uses samples produced by ReSTIR to determine distributions for path guiding in the following frame.^[18]

Use in games

The developers of Cyberpunk 2077 adopted ReSTIR DI in order to support rendering night-time city scenes with a large number of lights, including moving lights from vehicles, which was not possible with the shadow maps typically used for real-time rendering. ReSTIR GI is used selectively for indirect lighting on rough surfaces, in combination with a radiance cache.^[3]

One of the developers of Tiny Glade, Tomasz Stachowiak (who previously wrote an open source renderer called Kajiya that uses ReSTIR) has said that they tried ReSTIR for global illumination and it produced high quality results, but they were able to switch to simpler Monte Carlo integration (with a spatiotemporal filter) because sampling variance is low for the indirect light in their game. However they were experimenting with using ReSTIR DI to support a large number of direct lights.^[19]

See also

• Temporal anti-aliasing
• Ray tracing hardware