Measuring and characterizing the surface topography can be valuable when documenting a painting. The data gathered aids in monitoring change and evaluating condition before, during, and after treatment. In the documentation preceding Sue Ann Chui’s treatment of Rembrandt van Rijn’s Man with a Sheet of Music, 1633 (fig. 1), we pursued different imaging methods, and here we present a comparison of the methodologies. In this study, we chose to evaluate both reflectance transformation imaging (RTI) and photogrammetry methods. Our goal was to compare the advantages and tradeoffs of each approach with respect to accessibility and the value of the data generated.

The earliest studies in documenting the surface topography of paintings took place in the 1990s using different techniques such as moiré fringe analysis, RTI, and related photometric stereo techniques such as polynomial texture mapping (PTM). ^[1] These are computational photography methods used to document and analyze the surface textures and topography of objects. These imaging processes excel in documenting low-relief textures and are well suited to showing the surfaces of paintings to reveal impasto, cracks, or other surface deformations.

RTI is desirable because while it is based on quantitative measurements such as surface normal angles, RTI viewers enable the user to relight the surface virtually. This allows for an intuitive understanding of the nature of the surface as if one were inspecting it with a directional light source (e.g., a flashlight). There are a number of recontruction methods that use different coefficients. PTM and hemispherical harmonics (HSH) are among the more commonly used, ^[2] and there have been comparisons of these methods. ^[3] Recently there have been proposals for newer methods such as radial base function interpolation combined with principal component analysis compression (RBF+PCA) that may offer improvements. ^[4]

Because RTI only computes values required to relight the surface, such as surface normal angles and reconstruction coefficients (e.g., HSH), it allows for visualization of the surface texture without a complete reconstruction of the geometry. This results in much lower processing requirements compared to building and rendering a full three-dimensional representation of the surface. The Highlight RTI capture method requires only a camera, a flash, a string to keep a consistent illumination distance, and a simple glossy sphere to identify the angles of illumination. ^[5] The resulting images can be processed and viewed using free and open-source software packages. ^[6] The lack of the need for specialized equipment, along with free and simple processing and viewing software, greatly reduces the barrier to entry. Combined with the lower computational needs for rendering high-resolution images, this has made RTI a popular technique for documenting surface details for over a decade. ^[7]

It is important to note that RTI is not a 3D model, in that it is always viewed straight-on from the same singular position where the camera was located when taking the imaging set; there is no ability to rotate and view the surface from an angle. While one can infer information about the deformation of the surface from relighting or evaluating the surface normal data, the process does not allow for measurements of height and depth, and viewing can be prone to illusions that make it unclear if an area is receding or proud.

As computing power has substantially increased, photogrammetry, a means of rendering the geometry of surfaces as a 3D model from photographic images and computing the depth based on parallax between images taken from different positions, has gained popularity for cultural heritage documentation. Starting in the 1990s, photogrammetry was used to study the movement of paintings on wood in relation to changes in relative humidity. ^[8] The resolution has exponentially increased, from measuring 25 points over a 600 × 600 mm surface with a precision of 0.01 mm while relying on retro-reflective targets to be placed on the surface to being able to measure millions of points over a similar area with comparable precision and no need for targets placed on the surface. ^[9] In this study we were able to sample over 314 million points over the 660 × 480 mm surface of the painting. Software was used to compute the depth of features through computational analysis of the parallax. This can be highly computationally intensive but can result in quantifiable distance measurements of the surface.

There are a few modalities of 3D data that can be generated from photogrammetry. The first is a point cloud made of precise measurements of the surface in 3D space. ^[10] While having finite measurements at points is valuable, as the term “cloud” implies, it can be difficult to quantify a surface or volume. A mesh is made of polygonal (often triangular) “faces” to approximate a solid surface; ^[11] this can be useful in different types of comparative analysis, as well as allow other visualization techniques such as applying image-based textures or relighting the surface better than a point cloud. There are additional representations such as digital elevation models (DEM) and depth maps that represent 3D height as values from a 2D plane or from a given field of view. ^[12] These values can be visually represented as colors or grayscale tones.

To capture the RTI data set, a PhaseOne iXH 150 MP camera was used with a Canon EX-600 strobe light employing the Highlight RTI capture method in which multiple images are taken with the camera at a fixed position and the strobe light is placed at varying angles while a manual flash exposure at a set distance ensures consistent exposure across the set. A 64-inch-long string was used during the placement of the strobe to maintain consistent distance from the light source to the object while varying the angle of illumination. A pair of glossy black spheres were included in the capture area, from which the RTI building software was able to determine the angle of illumination in each image (fig. 2). In this set, 160 images were captured; however, this number could likely have been reduced to 60 to 70 images with minimal impact on the quality of the resulting render. ^[13] The higher number of images we utilized provides a worst-case example for processing times by using a number of images comparable to those needed for the photogrammetry tests. Files were exported from Capture One Pro Cultural Heritage software using linear profiles and saved as JPGs. An RTI file was generated using the open source Relight Builder and the HSH method with 27 planes (9 per color channel). ^[14]

To create the 3D model using photogrammetry, the panel was placed on a computer-controlled motorized easel that allowed for convenient repositioning of the panel relative to the camera. This enabled us to capture images of the surface from several angles easily and safely. We used a Phase One iXH 150MP designed for cultural heritage use (the same camera system employed to capture the RTI images) with a 53.7 × 40.4 mm (3.76 µm pixel pitch) sensor and a Phase One 72mm Mk II lens (with a diagonal field of view comparable to a 46mm lens on a 135 format “full-frame” sensor) to capture a series of 166 images (fig. 3).

The camera was kept with the sensor in plane to the panel while moving the easel in the x- and y-axes and making captures with at least 67% overlap of the field of view in both directions, ensuring every point on the surface appeared in different locations in at least 9 images. Redundant sets were included with the camera rotated at -45º and +90º around the optical axis to have more points of the painting captured through different areas of the lens, ensuring at least 27 images for nearly every point on the surface to better characterize and calibrate for any lens distortion using a method based on one developed by the US Geological Survey (USGS). ^[15] Exposures were made using LED lighting with correlated color temperature of 5000 K, providing an illumination with a color similar to natural daylight. To ensure color accuracy, the LEDs used have a color rendering index (CRI) of over 98. An aperture of f/9 previously has been determined to have the highest spatial frequency response (SFR) on this lens and sensor system. The peak SFR at f/9 indicates it will provide the best possible sharpness for the system, and this setting was used for all exposures. Calibrated scale bars were included in the capture area to set and validate the scale to submillimeter precision. Images were captured at a distance to create a magnification of approximately 815 pixels per inch (ppi) at the object or approximately 31.1 µm/pixel. Focus distance was maintained throughout exposures to avoid any change in the optical system. RAW images from the camera were processed into TIFFs from Capture One Pro Cultural Heritage software where linear curves with color matching input profiles were applied, with no additional tonal adjustments or sharpening. Images were aligned in Metashape Professional Edition, a commercial photogrammetry software package. ^[16] Through iterative refinement for lens correction using a modified version of the USGS method, root-mean-square (RMS) reprojection error was reduced to 0.199 pixel with a maximum reprojection error of 0.588 pixel. ^[17] Low RMS error is used as an indicator of reduced uncertainty in the camera location alignment and to ensure that the reconstruction is based on reliable data. Using a graphics workstation computer with two NVIDIA RTX 4090 GPUs and 512 GB of RAM, the initial camera alignment was completed in approximately 32 minutes. After alignment, optimization for lens corrections and definition of a region of interest required about an hour of user time. Building a high-resolution point cloud was more time-intensive, requiring approximately 18.5 hours of processing time on the PC workstation to complete a cloud made up of over 314 million unique measurement points across the surface in 3D space. A mesh of 26 million faces required nearly 16 hours of processing time. Refinement of the mesh is possible to generate more faces but requires substantially more processing time. Even with a very powerful computer and commercial software, the photogrammetry processing required multiple days to complete and substantial intervention by a trained technician during that time.

Viewing the RTI file in the RTIViewer software allows the user to move the lighting angle and simulate raking lighting from various directions, which helps illuminate peaks and valleys in the surface (fig. 4a). ^[18] In the RTIViewer desktop software and the OpenLIME web viewer, ^[19] there are settings that allow one to apply specular surface reflection properties to exaggerate details and make them more visible (fig. 4b). One can also view a surface normal map that encodes the angles of the surface by color. However, while the surface normals provide quantitative data, it is often considered more difficult to interpret than the more natural approach of lighting angles.

The RTI Builder provided by Cultural Heritage Imaging, commonly used by the cultural heritage community to create RTI files, was last updated in 2012 and no longer functions on modern operating systems. In early 2023, the Visual Computing Group of CNR-ISTI released a new Relight RTI Builder (see note 18) along with the web-based RTI viewer OpenLIME. ^[20] Using this new builder, we are able to generate an RTI file from the 160 images from the iXH camera, which produces very large 150 megapixel images in approximately 30 minutes. The results can be saved as .RTI files, which can be opened in the relatively easy to use Windows- or MacOS-based RTIViewer. The new Relight Builder can be used to create an OpenLIME viewer file set, effectively creating a website that can be used to post the results online. While the new web browser viewer has the potential for easy sharing, it requires technical expertise to run a web server to host the viewer and files online.

Three-dimensional data, such as that produced by the photogrammetry process in our experiments, has more variations and can be viewed in a multitude of ways. A 3D point cloud can be computed to provide precise measurements in space, the surface of the object can be represented as a mesh that can be shown at various angles or relit, or 3D data can be converted to a 2D representation such as a DEM or surface normal angles of the mesh. Unlike most RTI methods, the surface can be represented with the diffuse albedo (base color of the surface) removed, which is helpful if the painting has lines or colors that create illusions of depth when none exists.

The 314 million point cloud and 26 million face mesh generated are very large files and can cause even powerful computers to struggle. In addition, full-featured 3D viewers are often not intuitive or user-friendly. To visualize the data more easily without computational constraints, the 26 million face mesh was decimated to a lower-resolution mesh with 250,000 faces, drastically reducing the size of the file. While the mesh retained enough data to understand the basic shape of the surface, finer textures such as areas of impasto lost all fine detail. To include fine detail, surface normals were calculated from the high-resolution mesh and saved as a texture map to wrap the low-resolution mesh. Many 3D viewers use these normal texture maps to add lighting to surfaces at lower computational costs than the full-resolution mesh. These surface normal textures allow the model to be relit, showing fine texture and details, and can be shown in a web viewer that works on most computers or even mobile devices. The relighting capability may seem similar to the RTI approach, but it is improved by the fact that a low-resolution 3D model still underlies the normal map. Unlike in RTI, we can rotate the low-resolution 3D model and remove the diffuse albedo photographic textures, enabling us to view the surface of the geometry only and relight the computed surface based on the normal maps. The downside of using the lower-resolution mesh is that the viewing experience is more qualitative, as we do not have the high precision geometry needed to make accurate spatial measurements. But for cases in which a more qualitative or intuitive understanding of the surface that those using RTI have come to expect is desired, this approach allows for high-resolution 3D models to be presented in a more approachable interface. The model was uploaded to the online 3D platform, SketchFab, ^[21] which allows for viewing through any web browser, with no additional software required. In the SketchFab viewer, one can disable the relighting based on the normal texture, revealing the loss of visual detail that resulted from reducing the number of faces, or render in “mat-cap + surface” mode and relight the surface using the surface normal to reveal the lost texture (fig. 5a, b). The level of detail in decimated mesh with surface normal lighting is comparable to the detail seen in the original mesh (fig. 6).

Desktop 3D software is often much more complicated. While this software has far more options, those options increase the hurdles to sharing and opening 3D files locally. The complexities of the software require substantial experience, increasing the barriers to utilization. However, there are online viewers in which once the files are properly processed, the 3D assets can be uploaded to an online service and shared via the web with a much simpler interface. With the increasing popularity of 3D formats outside of the cultural heritage sector, we expect there will be even more options for sharing and viewing 3D models online in the future.

Our evaluation revealed that both RTI and photogrammetry have advantages and disadvantages. Reflectance transformation imaging is more approachable in terms of capture and processing. Processing for the 3D models generated by photogrammetry is substantially more time-consuming and requires more skill. More information can be captured in 3D, but 3D software is much less intuitive than the RTIViewer. For viewing on the web, 3D requires additional processing, but the files can leverage online 3D platforms that enable easier sharing. For many uses, RTI provides sufficient detail to understand the surface of the object and is more easily accessible, especially with the new Relight Builder and OpenLIME web viewer.

When the goal is to reveal surface textures, both methods can be used in many cases, but there are differences in what can be done. While RTI viewers often have specular enhancement viewing modes that help accentuate the shape while relighting, the brightness or darkness of the painted surface will still affect the final render. In contrast, the 3D model from photogrammetry can be rendered as if the surface is all the same material color (see fig. 6). In addition, within the 3D space, models can be rotated to mitigate “hollow-face illusions” and better illustrate if a feature is proud or recessed or the data can be processed into other modalities such as digital elevation maps as a different way to present topography, if applicable. While 3D data provides many more options, those options make the process far more complex. Capturing 3D data at the high resolution needed to reveal such textures comes with costs in the processing time and computing power needed to render the views.

While training is recommended to capture and process both RTI and photogrammetry with repeatability and precision, photogrammetry processing has a higher level of complexity and reviewing the 3D data often requires more expertise. The biggest limitation of photogrammetry is the computational requirements (table 1). While RTI processing can be performed on a reasonably powerful laptop, photogrammetry processing at the level of detail desired required 95.59 GB of RAM to build the mesh model and over 170 GB of RAM to build the point cloud (see notes 21, 22) The monetary cost of the computer and commercial photogrammetry software along with potentially days-long processing times make this process prohibitive at present for all but the most high-value projects. It is expected that software and computational power will continue to improve, which may make these methods viable for more projects in the future.

Table notes 
a. 60–70 images would have been sufficient for RTI captures.
b. Number increases exponentially as resolution requirements increase.

While RTI was the initial imaging technique proposed, the 3D model generated by photogrammetry allowed for a more detailed understanding of the deformation of the surface. A DEM (fig. 7) produced from the initial photogrammetry capture provided a simple but quantitative visual to understand the topography of the surface that did not require the use of specialized software to interpret. This was valuable for defining the initial condition of the panel as a basis of quantitative comparison through the treatment and into the future.

Since the initial evaluation, we have repeated the photogrammetry process to create 3D models of the surface of the Rembrandt panel during stages of treatment, before and after partial removal of the auxiliary support, where the hardboard backing and roughly half the thickness of the balsa-block backing was removed. With 3D geometry, we can create a direct comparison that depicts how the panel changes from its initial shape as the backing is reduced. To accomplish this, two photogrammetry sets were made before and after the first stage of removal of the panel’s backing. The two meshes were aligned using GOM Inspect software with a localized best-fit alignment on a selection at the center of the panel. ^[22] A surface comparison was then computed to determine the distances between the two meshes. We were able to create a color map comparing the pretreatment to the removal of the hardboard backing, showing how the surface has flattened; it revealed that the surface receded as much as 4 mm in the extreme corner after partial removal of the auxiliary support (fig. 8). This change between these two photogrammetry measurements appears to correlate with the deformation documented in DEM made from a different measurement taken before treatment began (see fig. 7).

We plan to repeat the photogrammetry process through the final stages of treatment to monitor the surface as treatment progresses. While RTI is not as quantitative as photogrammetry, we plan to create an after-treatment RTI to have more complete documentation. At that time we hope to have a further evaluation of the newer RBF+PCA RTI method compared to HSH in the context of this object. We are also monitoring new methods of improving the material appearance of photogrammetry captures that promise to improve the ability to faithfully relight surfaces by better accounting for metallic and gloss properties on a surface by measuring the Spatial Varying Bidirectional Reflectance Distribution Function. ^[23] In addition, we are in the process of evaluating open-source comparison software such as MeshLab and CloudCompare as alternatives for comparing 3D data without commercial software such as GOM Inspect. ^[24] We continue to monitor the development of new web-based 3D viewer and annotation tools for collaborating with 3D data, such as those being led by the International Image Interoperability Framework’s (IIIF’s) 3D Technical Specification Group. ^[25] At this time the computational needs of photogrammetry are a significant barrier, particularly for smaller institutions. The Agisoft Metashape software we used in this study for photogrammetry has begun to offer fee-based cloud-based processing, and there are other third-party high performance computing platforms that may be worth investigating as a way to either reduce processing time or provide options to those who cannot justify purchasing a very expensive computing workstation for a small number of projects.

After the treatment is completed, the 3D model can be used as a baseline to monitor the panel for any changes in shape that may occur in future. Having the record of the previous deformation, we have an idea of the range of out-of-plane movement that is possible. Beyond these measurements, materials modeling experiments could one day lead to the creation of tools that can predict the safe limits of movement for paintings on wood. ^[26]