MMS-DATA:
A Multimodal Multi-view Dataset

Federico Lincetto1, Gianluca Agresti2, Mattia Rossi2, Pietro Zanuttigh1
1Media Lab - University of Padova, 2Sony Europe B.V.

CVPR 2025

Overview

MMS-DATA is a multimodal multi-view calibrated dataset published along with MultimodalStudio.
It is captured with 5 different imaging modalities:

  • RGB
  • Monochrome (Mono)
  • Near-infrared (NIR)
  • Polarization (Pol)
  • Multispectral (MS)

MMS-DATA includes 32 object-centric scenes, with 50 viewpoints per scene. The sensor poses are accurately geometrically calibrated. The subjects are common objects made from diffusive, glossy, reflective, and transparent materials, such as plastic, metal, wood, organic, cloth, paper, and glass. Refer to the paper and to the supplementary material for more details about the dataset and the calibration procedure.

Releases

v1.1 - 16/09/2025
New version of MMS-DATA released. In MMS-DATA v1.0, an error was detected in the calibration process of the multispectral camera. The estimated principal point coordinates had an unwanted shift. This issue has been corrected by recalibrating the sensor intrinsics and poses. For this reason, all the MMS-DATA archives offered here were re-processed and updated, including the foreground masks. We warmly suggest to re-download them and rely only on this new version. This issue did not affect the experiments and results included in the paper because it emerged when the data were prepared for release.

v1.0 - 15/05/2025
MMS-DATA released.


Source Data

(35 GB)

Download

This archive of MMS-DATA includes:

  • the original RAW frames for every modality
  • the geometrical camera calibration of all sensors
  • the frames used for the camera calibration
  • few frames of a color checker

All the frames are RAW, thus they are mosaicked and they did not receive any processing or color correction. This archive does NOT include the camera poses of every frame.

Processed Raw Data

(34 GB)

Download

Each scene of this archive of MMS-DATA includes:

  • the processed mosaicked frames
  • the metadata of each scene, including camera poses
  • a sparse point cloud of the scene
  • a point cloud of all the camera positions

All the frames are mosaicked, distorted, and they did not receive any color correction. This archive contains the camera pose for all frames of every scene.
The scenes in this archive are ready-to-use to run MultimodalStudio.

Processed Undistorted Data

(128 GB)

Download

Each scene of this archive of MMS-DATA includes:

  • the demosaicked and undistorted frames
  • the metadata of each scene, including camera poses
  • a sparse point cloud of the scene
  • a point cloud of all the camera positions

All the frames are demosaicked, undistorted, and they did not receive any color correction. This archive contains the camera pose for all frames of every scene.
The scenes in this archive are ready-to-use to run MultimodalStudio.

Sample Scene - Bird House

(6 GB)

Download

This sample scene of MMS-DATA includes:

  • the original RAW frames of Bird House
  • the geometrical camera calibration of all sensors
  • the processed mosaicked frames of Bird House
  • the demosaicked and undistorted frames of Bird House
  • the metadata of Bird House, including camera poses
  • a sparse point cloud of Bird House
  • a point cloud of all the camera positions

All the frames did not receive any color correction. The Bird House scene in this archive is ready-to-use to run MultimodalStudio.

Foreground Masks

(57 MB)

Download

This archive includes:

  • the foreground masks for each frame of every modality of every scene

These masks are used to compute the metrics for the tests involving distorted frames.

Multi-sensor Acquisition Setup

Multi-sensor acquisition setup.

The sensors where mounted on a custom-built rig.
We employed 5 different imaging sensors:

  • RGB: Basler acA2500-14g
  • Monochrome (Mono): Basler acA2500-14gm
  • Near-infrared (NIR): Basler acA1300-60gmNIR
  • Polarization (Pol): FLIR BFS-U3-51S5P-C
  • Multispectral (MS): Silios CMS-C1
All the sensors are stereo calibrated with respect to the RGB camera, considered as reference camera.

Calibration Results

Calibration RMSE.

The calibration process involves two steps:

  • the intrinsics calibration,
  • the joint pose calibration of the different sensors

We calibrated the camera extrinsics assuming a star topology. The RGB sensor is selected as the reference camera, and all the other sensors are stereo calibrated with respect to it.

Acquisition Procedure

Acquisition procedure.

The acquisitions were performed by moving the rig all around the target object placed on the table and by capturing data from all modalities at each viewpoint. The rig was moved manually, therefore the sensor positions are different from scene to scene.
The pictures have been acquired by moving the camera rig around the object in 2 circular patterns, a lower one and an upper one.

Materials per scene

Table of materials per scene.

Table of materials per scene. Each scene contains objects made from materials with different properties.

BibTeX

@inproceedings{lincetto2025multimodalstudio,
  author    = {Lincetto, Federico and Agresti, Gianluca and Rossi, Mattia and Zanuttigh, Pietro},
  title     = {MultimodalStudio: A Heterogeneous Sensor Dataset and Framework for Neural Rendering across Multiple Imaging Modalities},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year      = {2025},
}