Modular Sensor Server Architecture
Interactive environments often begin with a simple technical relationship: a sensor detects activity and a real-time application responds.
For small installations, connecting a sensor directly to TouchDesigner, Unity, or another experience runtime can be sufficient. As systems become larger, however, this direct relationship can create dependencies between hardware-specific processing and experience-specific logic.
A modular sensor server architecture separates these responsibilities.
Instead of treating sensors as peripherals belonging to a particular visual application, sensing and spatial processing become a reusable infrastructure layer capable of serving different experience engines, installations, and research systems.
The central principle is to separate what happens in physical space from how an experience responds to it.
1. From Direct Integration to Modular Infrastructure
A simple interactive installation may use a direct architecture:
System Architecture
Left → Right
This approach has several practical advantages.
- minimal infrastructure
- fewer machines
- simple deployment
- direct access to raw sensor information
- fast prototyping
For installations with a small number of interaction areas, this may be entirely appropriate.
The limitation appears when the same system begins to require:
- multiple sensors
- multiple experience applications
- distributed computers
- reusable interaction logic
- centralized logging
- research data collection
- replacement of individual technologies
- operation across multiple deployments
At that point, sensor processing becomes infrastructure rather than merely an input to a visual application.
2. The Architectural Separation
The modular architecture separates an interactive environment into several functional layers.
System Architecture
Left → Right
Each layer has a different responsibility.
| Layer | Responsibility | Examples |
|---|---|---|
| Sensor | Observe physical activity | LiDAR, depth camera |
| Edge | Acquire and process sensor data | Raspberry Pi |
| Logic | Interpret spatial behaviour | area detection, tracking |
| Protocol | Exchange normalized information | OSC, REST |
| Runtime | Execute experience logic | TouchDesigner, Unity, Unreal |
| Output | Produce spatial response | projection, LED, audio |
This separation is conceptual rather than dependent on a particular hardware configuration.
A small deployment may combine several layers on one computer.
A larger deployment may distribute them across multiple machines.
The important distinction is the responsibility of each layer.
3. Why Introduce a Sensor Server?
Consider a LiDAR connected directly to TouchDesigner.
System Architecture
Left → Right
The TouchDesigner project may need to:
- communicate with the sensor
- decode sensor measurements
- transform coordinates
- filter unwanted measurements
- detect interaction areas
- maintain interaction state
- generate visual behaviour
- perform projection mapping
The same project is therefore responsible for both understanding the physical environment and rendering the experience.
Introducing an edge-processing layer changes this relationship.
System Architecture
Left → Right
The sensor server becomes responsible for understanding sensor-specific information.
The experience runtime receives information that is already meaningful to the interaction.
4. Raw Sensor Data and Interaction Data
An important distinction is the difference between raw sensor data and interaction data.
Raw sensor data may describe measurements such as:
angle
distance
point position
depth
sensor frame
An experience rarely needs all of this information.
It may only need to know:
visitor present
position x/y
area active
interaction started
interaction ended
The sensor-processing layer can therefore transform hardware-specific information into a normalized representation.
For example:
{
"sensor": "lidar_left",
"x": 0.42,
"y": 0.71,
"area": 2,
"active": true
}
or through OSC:
/tracking/position 0.42 0.71
/interaction/area 2
/interaction/active 1
The experience application no longer needs to know how the original measurement was produced.
5. Interaction Semantics
The sensor server should not always transmit the smallest possible amount of data.
The correct level of abstraction depends on the experience.
For a simple area-triggered installation, the runtime may only require:
/area/1 1
/area/2 0
/area/3 0
For continuous spatial interaction, it may require:
/tracking/x 0.42
/tracking/y 0.71
For a research environment, additional information may be preserved:
/session/id S001
/sensor/id lidar_left
/tracking/x 0.42
/tracking/y 0.71
/interaction/area 2
This suggests that the sensor server should provide interaction semantics appropriate to the downstream system, rather than forcing every application to consume the same representation.
6. The Sensor Layer
The architecture can support different sensing technologies.
Potential inputs include:
- 2D LiDAR
- depth cameras
- computer vision
- RFID
- IMU
- physical controllers
- environmental sensors
A sensor is responsible for observation.
It should not determine the final visual response.
For example, replacing a LiDAR with a depth camera should not necessarily require the entire experience architecture to change.
System Architecture
Left → Right
can become:
System Architecture
Left → Right
The sensing technology changes.
The downstream interface can remain similar.
7. Edge Processing
A Raspberry Pi can operate as a lightweight sensor server for spatial interaction systems.
Its responsibilities may include:
- sensor communication
- coordinate transformation
- filtering
- area detection
- presence detection
- interaction-state management
- network transmission
- session management
- selected data logging
This allows the main rendering workstation to concentrate on real-time media.
System Architecture
Left → Right
The same edge node can potentially serve another runtime:
System Architecture
Left → Right
This is one of the main benefits of separating sensor infrastructure from experience software.
8. Multi-Sensor Systems
A larger responsive environment may require several sensors.
This introduces additional architectural responsibilities.
System Architecture
Left → Right
The system must now consider:
- sensor position
- coordinate transformation
- overlapping sensing areas
- occlusion
- duplicate observations
- shared spatial coordinates
The problem is no longer simply reading four sensors.
The problem becomes creating a coherent representation of one physical environment from several observations.
9. Shared Coordinate Space
Multi-sensor systems benefit from a common coordinate system.
Each sensor initially observes the environment relative to its own position.
Conceptually:
Sensor A coordinates
↓
Transformation A
↓
Shared Spatial Coordinates
Sensor B coordinates
↓
Transformation B
↓
Once observations exist in a shared coordinate space, downstream systems can reason about the environment independently of the physical location of each sensor.
This is especially important for:
- trajectory reconstruction
- heatmaps
- interaction zones
- dwell-time analysis
- cross-sensor tracking
The shared coordinate model therefore becomes useful both for interaction and research.
10. Communication Layer
After processing, interaction information must be transmitted to other systems.
Different communication methods serve different purposes.
OSC
OSC is useful for lightweight real-time control data.
Examples include:
/tracking/position
/interaction/area
/interaction/active
/session/id
REST
REST is useful for discrete system-level events and state changes.
Examples include:
room state
experience finished
door state
system mode
Media Transport
Protocols such as NDI solve a different problem.
They transport media rather than interaction semantics.
This distinction is important.
System Architecture
Left → Right
Interaction data and media transport may operate on the same network while serving different architectural roles.
11. Experience Runtime
Once interaction data has been normalized, different real-time applications can consume it.
Potential runtimes include:
- TouchDesigner
- Unity
- Unreal Engine
- custom applications
The runtime should primarily answer:
Given this interaction information, what should the experience do?
rather than:
How do I decode this particular sensor?
For example:
System Architecture
Left → Right
or:
System Architecture
Left → Right
The same sensor infrastructure can therefore support different creative systems.
12. Distributed Experience Systems
Large installations may contain more than one processing machine.
A broader architecture may resemble:
System Architecture
Left → Right
The processing workstation can generate real-time content while a dedicated media server handles final output and mapping.
This becomes useful when an environment contains:
- multiple projectors
- multiple LED surfaces
- several rooms
- synchronized content
- distributed interactive zones
The sensor server remains independent of the final display topology.
13. Evidence from Practice
The architecture developed incrementally through interactive installation practice rather than appearing as a complete system from the beginning.
Direct Sensor Integration
Earlier interactive systems demonstrated that direct integration can be effective for constrained installations.
A LiDAR can provide area detection directly to TouchDesigner when:
- the interaction model is simple
- only one runtime requires the data
- research logging is not required
- the system does not need to be reused elsewhere
This represents the simplest architecture:
System Architecture
Left → Right
Edge-Processed Interaction
Later interactive systems introduce Raspberry Pi as a sensor-processing layer.
System Architecture
Left → Right
The sensor and game become less tightly coupled.
Distributed Responsive Environments
Larger installations extend this pattern through multiple sensors, processing workstations, media servers, and centralized control.
System Architecture
Left → Right
Across these implementations, a recurring pattern becomes visible:
Sensor infrastructure becomes increasingly valuable when it can survive beyond the application for which it was first created.
14. Example — Mobile Interactive Game
A mobile interactive game provides a useful small-scale example.
The physical system may contain:
- projector
- LiDAR
- Raspberry Pi
- PoE networking
- mini computer
The interaction architecture can be simplified as:
System Architecture
Left → Right
The LiDAR observes player movement.
The Raspberry Pi interprets that movement.
The game receives interaction information.
The projector presents the result.
The advantage is that the game does not need to own the LiDAR-processing implementation.
A different game could potentially consume the same interaction interface.
15. Example — Multi-Sensor Research Installation
A research-oriented responsive environment requires additional infrastructure.
System Architecture
Left → Right
This architecture extends the same sensor-server principle into research instrumentation.
Spatial processing now serves two consumers:
- the interactive experience
- the research dataset
16. Experience and Research as Parallel Paths
The most important extension is the separation between experience response and research observation.
System Architecture
Left → Right
The experience path prioritizes real-time response.
The research path prioritizes structured observation.
Neither needs to control the other.
This reduces the risk that research logging becomes deeply embedded inside creative media logic.
17. Session Identity
Distributed systems need a way to associate observations generated by different components.
A shared session_id provides this connection.
For example:
{
"session_id": "S001",
"timestamp": 18.24,
"source": "lidar_left",
"x": 0.42,
"y": 0.71,
"area": 2,
"event": "interaction"
}
Another machine can record:
{
"session_id": "S001",
"timestamp": 18.31,
"source": "experience_runtime",
"event": "content_trigger",
"content": "layer_02"
}
The common identifier makes later comparison possible.
This becomes increasingly important as the architecture becomes distributed across several computers.
18. Research Instrumentation
Once sensor infrastructure can produce structured observations, the system can support research measures such as:
Spatial Measures
- position
- trajectory
- movement direction
- interaction zones
Temporal Measures
- session duration
- dwell time
- interaction duration
- completion time
Interaction Measures
- interaction count
- trigger frequency
- repeated activation
- content selection
Derived Measures
- heatmaps
- trajectory visualization
- interaction density
- spatial preference
- experience progression
The same sensing infrastructure that drives the experience can therefore contribute to research without requiring the experience runtime to become the primary data-collection system.
19. Failure Isolation
Modularity also changes how technical failures are diagnosed.
In a tightly coupled system:
Sensor
+
Processing
+
Interaction
+
Rendering
=
One Application
a failure can be difficult to isolate.
In a layered architecture:
System Architecture
Left → Right
each boundary can be tested independently.
For example:
- Is the sensor producing data?
- Is the edge computer detecting interaction?
- Is OSC being transmitted?
- Is the runtime receiving the message?
- Is the experience responding?
- Is the output system displaying correctly?
This is particularly useful for permanent installations and systems maintained by people other than the original developer.
20. Reusability
The main value of the architecture appears when components are replaced.
Configuration A
System Architecture
Left → Right
Configuration B
System Architecture
Left → Right
Configuration C
System Architecture
Left → Right
The technologies differ.
The architectural pattern remains recognizable.
21. Design Lessons
Several reusable lessons emerge from this architecture.
Separate sensor-specific processing when reuse matters
Direct integration is not inherently wrong.
It becomes limiting when the same sensing infrastructure must support several applications or deployments.
Send meaningful interaction information
Raw sensor data should only be forwarded when downstream systems actually need it.
Otherwise, normalized position, presence, area, or event information can create a cleaner interface.
Keep media transport conceptually separate
OSC, REST, and NDI may coexist on the same network, but they solve different problems.
Design for replacement
A sensor, runtime, or display technology will eventually change.
Stable boundaries reduce the cost of that change.
Treat research logging as infrastructure
If behavioural data may become important later, logging should be considered during architecture design rather than added only after an installation is complete.
Preserve raw information when necessary
Abstraction should not destroy information that future research may require.
A research system may therefore retain both normalized interaction events and selected lower-level sensor measurements.
22. Limitations
A modular architecture introduces its own costs.
Additional components increase:
- network dependencies
- configuration requirements
- synchronization concerns
- deployment complexity
- debugging surfaces
Edge processing is also not appropriate for every computational task.
A Raspberry Pi may be suitable for LiDAR processing and lightweight interaction logic while being inappropriate for demanding computer-vision or machine-learning workloads.
Normalization also requires deliberate design.
Too little abstraction preserves unnecessary hardware dependencies.
Too much abstraction may remove information required by later applications or research.
The architecture should therefore be scaled according to the complexity and expected lifetime of the installation.
23. When Direct Integration Is Better
Not every project needs a sensor server.
Direct integration may be preferable when:
- the installation is temporary
- there is one sensor
- there is one runtime
- interaction logic is simple
- reuse is unlikely
- no research logging is required
- setup time is more important than modularity
A useful architecture is not necessarily the architecture with the largest number of components.
It is the architecture whose complexity matches the problem.
24. When Modular Infrastructure Becomes Valuable
A sensor server becomes increasingly useful when several of the following conditions appear:
- multiple sensors
- multiple runtimes
- permanent deployment
- distributed machines
- remote maintenance
- reusable installations
- multi-room environments
- research logging
- longitudinal data collection
- future sensor replacement
These conditions shift sensing from a project-specific implementation detail toward infrastructure.
25. From Practice to Reusable Knowledge
The architectural pattern can be summarized as a progression.
System Architecture
Left → Right
The important step is not merely building increasingly complicated installations.
It is identifying which technical decisions remain useful across different installations.
Those recurring decisions become reusable knowledge.
26. Research Direction
The sensor-server architecture provides a foundation for a broader research question:
How can responsive environments maintain a reusable sensing and interaction infrastructure while supporting different experiences, deployments, and forms of spatial research?
Future development can investigate:
- multi-sensor coordinate fusion
- sensor interchangeability
- standardized interaction schemas
- synchronization across distributed systems
- session management
- long-term logging
- deployment comparison
- privacy-aware spatial sensing
- integration of LiDAR, depth, IMU, and RFID
- reusable evaluation infrastructure
The objective is not to produce one universal sensor server.
The objective is to identify stable architectural principles that allow responsive environments to evolve without rebuilding their entire sensing infrastructure.
27. Key Takeaways
A modular sensor server architecture separates:
Sensing → Processing → Interaction → Communication → Experience → Output
rather than allowing every responsibility to accumulate inside a single real-time application.
For research-oriented environments, the architecture can extend into a parallel path:
Spatial Processing → Session Logging → Dataset → Analysis
The approach is most valuable when systems become multi-sensor, distributed, reusable, permanent, or research-oriented.
Direct sensor integration remains appropriate for simpler installations.
The architectural decision should therefore be based on system requirements rather than complexity for its own sake.
The broader principle is:
Build sensor infrastructure around reusable representations of physical interaction, not around the assumptions of a single experience application.