Codex 4D: journey in four dimensions into the manuscript

In 2022-2023, the project “Codex4D: four-dimensional journey to the centre of the manuscript” was developed by the Institute of Heritage Science at CNR and the Department of Industrial Engineering at the University of Rome, Tor Vergata, thanks to funding from the Lazio Region ((POR FESR Lazio 2014–2020). The project is concerned with the definition and testing of a methodological pipeline for the digital documentation and 3D exploration of information about visible content and elements that are no longer visible or hidden in the structure of ancient codices, e.g., text parts buried in the binding and stratigraphies of the illuminations, as well as characterisations of the chemical–physical–biological nature of the materials. A 3D model was realised, to which the fourth dimension of depth was added.
The approach is innovative, aimed at creating a multidisciplinary experience of the manuscript inside virtual and mixed reality environments, taking into consideration tangible and intangible values: form and structure; content and meaning; materials, execution techniques, and state of conservation.
Superficial information is associated with the visible spectrum of light, RGB (the additive model of colours red, green, and blue), captured via standard photo cameras. RGB images were also used to produce a photogrammetric model of the object, via structures from motion techniques. The invisible content associated with the sub-superficial layers was acquired via a thermal camera, and consisted of images in the medium infrared range.The relevant information has been mapped as “annotations” on the virtual multidi- mensional model, and as informative/semantic spots.
The interpretation of so much information can reconstruct the complex story of the artefact, including the significance of written and pictorial content, craftsmanship, conservation history, and cultural relations.
The project was developed on three main case studies at the Angelica Library in Rome:
(1) Ms 1474, De Balneis Puteolanis [20], a poem by Pietro da Eboli, dated 1258/1266, of which three poses were acquired in 4D (cc.9v-10r, cc.12v-13r, cc.19v-20r);
(2) Ms 1102, Divina Commedia by Dante Alighieri [21], dated 1351/1400, of which two poses were acquired in 4D (cc.4v-5r, cc.56v-57r);
(3) Ms 459, Libro d’Ore [22], a prayer book for secular use, dated at the beginning of the XV century AD, of which three poses were acquired in 4D (closed book, cc.21v-22r, cc.67v-68r).

The results obtained help to broaden the accessibility of these exceptionally valuable assets, often difficult to enjoy and to understand. This achievement was made possible by the integration of many different competences into the team: palaeographers, codicolo- gists, art historians, conservation scientists, physicians, chemists, biologists, 3D modellers, experts in communication and user studies, computer scientists.
Several outputs were developed in the Codex 4D project, that differ in (a) type of users, (b) context of use, and (c) type of experience, and, therefore, adopting different communication styles:
• A multimedia web site (https://codex4d.it/, accessed on 11 June 2024) designed for in-depth educational and scientific purposes, in which the Codex 4D project, the methodologies and technologies employed, and the results of the research are recounted;
• An online virtual-reality environment (Web App), designed for the scientific visualisation and analytical exploration of the manuscript 4D model, intended for more expert users;
• A holographic showcase designed as a mixed-reality installation for museum and 
library audiences, in which research data are translated into an emotional narrative through a dramatic and engaging style.

The Web Site
From a communicative point of view, the project purpose was to shape a direct multidisciplinary experience with manuscripts.
The Codex 4D website (https://codex4d.it, accessed on May 11, 2024) has educational as well as scholarly value. It offers a wide variety of sections that allow users to embark on a narrative and multimedia journey, characterised by simple interactivity and aimed at a broad audience interested in learning about the context of ancient manuscripts.
The main sections of the site are as follows:
(1)     Project—The objectives and, from an educational perspective, all the survey and documentation methodologies and technologies used in the project are explained with a special focus on integration methods. The composition of the working group is also presented;
(2)     Collection—The codices are introduced and contextualised, with the possibility, for each one, to directly access the complete description, and the photo-gallery of all the sheets. For each manuscript, it is also possible to access the Web App and start the 4D exploration of the codex model, in its various configurations;
(3)     Narrated Glossary—The basic knowledge is provided to embark on the journey through the materials and stages of manuscript making (Figure 21);
Results—The outcomes of the projects, such as digital applications, deliverables, and scientific publications, are presented.

The Web App
An advanced Web3D environment is dedicated to the interactive visualisation and scientific analysis of manuscripts and the questioning of contextualised information (annotations or semantic descriptors) in four dimensions. The main target is an audience predisposed to thorough study and discussion.
The environment is based on the ATON open-source framework developed by the CNR ISPC.
Tools have been implemented allowing the user to choose to explore specific configurations and sheets of the manuscript, explore stratigraphic layers through a detection lens switching from superficial RGB to IR images, take measurements, move the lighting in the scene to better highlight specific details, and open or create annotations related to mapped information on specific points of interest. A video demo of the Web App can be seen here: https://tube.rsi.cnr.it/w/jZi6XWdHcqMBeYZo52SKTf.
The Web App can be accessed through a double profile:
-        public, allowing only the exploration of existing contents;
-        editor (through authentication), which can modify and create new contents.
To manage the large amount and variety of processed content along orderly editorial lines, it was necessary to organise it into categories and subcategories. Such a structure is useful for editors creating new content, but also for end users who can easily focus on what they are interested in, filtering information related to the 4D model at will.
As a result of the editor profile, the virtual-reality environment is being used not only as the final output of a research work but as a real tool for investigation, as a shared laboratory for the analysis, interpretation, and integration of data, updatable over time by the scientific community.

The Holographic Showcase
The 4D models and content that emerged from the surveys are also used inside a holographic museum showcase, based on the Pepper’s Ghost technique. The holographic showcase is designed as a mixed-reality environment; it works as a small theatre, equipped with lights and directing software that offers the possibility of activating multimedia events within it, interactively. The showcase introduces a dramaturgical storytelling: a narrator character, played by an actress filmed in green screen, lives in the illuminations, and tells that world from the inside. She is as small as the illuminated figures and performs actions or activates tools to enhance the reading and understanding of the codex. In this way, the Codex 4D project aims not only to increase the scientific knowledge of the manuscript, but also to push the languages of scientific communication into new experimental terrain to attract and intrigue new audiences.
In the holographic showcase, the 4D model is again enriched with annotations dealing with the iconography, iconology, materials and techniques, and state of conservation, which the user can explore by just moving his/her hand on the model, through simple gestures. Motion capture is managed by the Leap Motion sensor and contributes to raising the user’s curiosity and engagement.
The Codex 4D holographic showcase was presented to the public for the first time at the exhibition “Languages of Heritage Science: from Micro to Macro,” at the Genoa Science Festival (Villa Principe, 20 October–1 November 2022).
On that occasion, a survey was conducted to evaluate the experience and impact of the showcase on the public, represented mainly by high school and university students, families, experts, and individual and group visitors.
Therefore, the contents have been enriched for the new exhibition set up at the Angelica Library in Rome, from 10 November 2023 to 8 February 2024.