The Flying Letters Whispering From The Canvas

An immersive interactive installation combining postcard-based content selection, computer vision, LiDAR spatial sensing, and real-time media systems to create a personalized responsive environment derived from visitors’ selected artwork themes and spatial interaction.

The Flying Letters Whispering From The Canvas is an immersive interactive installation developed as part of Gate of Dream at Fah Lanna Art Museum. The experience connects two forms of interaction: visitor input before entering the immersive environment and spatial interaction inside the projected space. Before entering the immersive room, visitors interact with a postcard-based scanning system. Computer vision is used to identify the scanned postcard and associate visitor input with one of four artwork themes. This information determines which immersive content variation is presented during the experience. Each artwork theme is translated into an approximately one-minute immersive content sequence. In addition to the four artwork-based variations, the experience includes an interactive letter-loop state in which visitors can encounter and interact with their own names within the projected environment. Inside the immersive space, four LiDAR sensors provide spatial sensing across the interactive environment. Rather than processing each LiDAR directly within the main media application, the sensors are connected through the local PoE network to a dedicated Raspberry Pi that operates as a LiDAR sensor-processing server. The Raspberry Pi consolidates LiDAR data and provides configurable area-detection logic. Spatial regions can be defined at the sensor-processing layer, allowing the system to determine whether an object or visitor is present within a particular interaction area and return the resulting state to TouchDesigner through OSC. TouchDesigner uses this processed spatial information to drive real-time interaction and visual behavior, while Resolume operates as part of the media presentation and projection workflow. The resulting system connects visitor identification, content selection, distributed spatial sensing, real-time interaction, and immersive media presentation within a single visitor experience.

Development of the technical workflow connecting postcard-based visitor input, computer-vision content selection, distributed LiDAR sensing, real-time interaction, and immersive media presentation. The work includes developing the workflow used to identify scanned postcards and associate visitor input with different artwork-based content variations, allowing the immersive experience to change according to the selected theme. For spatial interaction, a dedicated Raspberry Pi is used as a LiDAR sensor server for four networked LiDAR sensors. The server consolidates sensor processing and provides configurable area-detection logic independently from the main visual-processing application. Defined spatial regions can be evaluated directly by the sensor server and returned as interaction states through OSC. This allows TouchDesigner to consume higher-level spatial information, such as whether a visitor or object is present within a particular interaction area, without requiring the media application to independently process every LiDAR sensor. The architecture separates sensing, interaction data, real-time media processing, and presentation into distinct system layers, creating a reusable technical foundation that can also support future research instrumentation and spatial interaction studies.

The installation is being developed as a temporary immersive experience that combines personalized content selection with real-time spatial interaction. Its technical architecture allows postcard-based visitor input, multiple LiDAR sensors, configurable interaction areas, real-time media processing, and immersive presentation to operate as connected but independently manageable system components. Beyond the exhibition itself, the deployment is intended to provide a practical foundation for investigating how spatial interaction data can be captured and structured for subsequent research into visitor behavior and responsive immersive environments.