Overview
Spatial sensors do not directly produce interactions.
A LiDAR scanner may produce distance measurements across a scanning plane. A depth camera may produce a depth image, point cloud, body estimate, or other spatial representation. These measurements describe aspects of physical space, but they do not yet describe what the experience should understand.
An interactive system usually needs higher-level information such as:
- whether a visitor is present
- where the visitor is located
- which interaction area is occupied
- whether the visitor entered or left an area
- how long the visitor remained there
- how the visitor moved through the environment
- whether an interaction condition has been satisfied
The transformation from sensor measurement to interaction meaning is therefore a critical layer of responsive-environment architecture.
Raw sensor data describes measurement. Interaction data describes meaning within the experience.
1. The Transformation Pipeline
A reusable spatial interaction system can separate the transformation into several stages:
System Architecture
Left → Right
Each stage reduces hardware-specific information and moves toward a representation that the experience can understand.
A more complete pipeline is:
Physical Space
↓
Sensor Acquisition
↓
Filtering
↓
Coordinate Transformation
↓
Spatial Fusion
↓
Tracking
↓
Region / Area Detection
↓
State Detection
↓
Interaction Events
↓
Experience Runtime
Not every installation requires every stage.
The important principle is that these responsibilities can remain conceptually separate.
2. Raw Sensor Measurements
Different sensors describe space differently.
2D LiDAR
A planar LiDAR may provide a sequence of range measurements:
angle_0 → distance_0
angle_1 → distance_1
angle_2 → distance_2
...
angle_n → distance_n
These measurements can be converted into Cartesian coordinates:
x = r × cos(θ)
y = r × sin(θ)
The resulting points represent detected surfaces or objects within the scanning plane.
Conceptually:
System Architecture
Left → Right
Depth Camera
A depth camera commonly provides depth values across an image:
pixel (u,v) → depth
Depending on the device and processing pipeline, these values may be transformed into:
X
Y
Z
coordinates in camera space.
The underlying data structures differ from LiDAR, but both sensors eventually provide observations about where physical geometry exists.
3. Raw Data Is Not Interaction Data
Consider a LiDAR point:
x = 1.42
y = 2.18
This point alone does not tell the experience whether:
- it belongs to a visitor
- it belongs to a wall
- the visitor is entering an area
- the visitor has remained there long enough
- the interaction has already been triggered
Similarly, a depth pixel containing:
z = 2.1 m
does not directly represent an interaction.
Additional interpretation is required.
This distinction prevents experience logic from becoming dependent on low-level sensor measurements.
4. Filtering
Raw spatial measurements may contain:
- environmental geometry
- measurement noise
- temporary reflections
- invalid measurements
- isolated points
- rapidly fluctuating values
Filtering reduces this information before interaction analysis.
A conceptual pipeline is:
System Architecture
Left → Right
Range Filtering
Measurements outside the useful interaction distance can be removed.
For example:
minimum distance < point < maximum distance
Region of Interest
Only measurements inside the installation's usable spatial region may be retained.
xmin < x < xmax
ymin < y < ymax
Temporal Filtering
Measurements can also be compared over time to reduce rapid instability.
The exact filtering method depends on the sensor, environment, and interaction requirements.
5. Coordinate Systems
Every sensor initially observes the environment from its own coordinate system.
For example:
LiDAR A
origin = left side of room
LiDAR B
origin = right side of room
The same visitor may therefore appear at different coordinates to each sensor.
Before combining these observations, they need to be expressed in a common coordinate system.
System Architecture
Left → Right
A simple 2D transformation can include:
rotation
translation X
translation Y
Conceptually:
sensor coordinates
↓
calibration transform
↓
room coordinates
Once transformed, multiple sensors can describe the same physical environment using the same spatial reference.
6. Shared Room Coordinates
A shared coordinate system can define an installation independently from sensor placement.
For example:
(0,0)
┌──────────────────────────┐
│ │
│ interaction │
│ space │
│ │
└──────────────────────────┘
(6,4)
Sensor positions may change while the logical interaction space remains:
room width = 6 m
room depth = 4 m
This makes area definitions reusable.
Instead of defining an interaction region relative to a specific LiDAR:
LiDAR A x = ...
LiDAR A y = ...
the system can define:
room x = ...
room y = ...
This is particularly useful when an installation uses multiple sensors.
7. Multiple Sensor Fusion
A single sensor may lose visibility because of:
- visitor occlusion
- physical obstacles
- limited field of view
- sensor placement
- other visitors
Multiple sensors can observe overlapping parts of the environment.
System Architecture
Left → Right
After coordinate transformation, measurements from different devices can be combined into a common spatial representation.
The objective is not necessarily to merge every raw point.
Depending on the application, fusion may occur at different levels:
raw point level
cluster level
tracked-object level
interaction-state level
Choosing the appropriate fusion level depends on what the experience actually needs.
8. Clustering Spatial Measurements
A person may generate many sensor points.
The experience usually does not need each point independently.
Instead:
point
point
point
point
point
can be interpreted as:
cluster
and then:
tracked visitor
Conceptually:
System Architecture
Left → Right
A representative position might be derived from:
- cluster centroid
- nearest point
- bounding region
- tracked center
- another application-specific estimate
The method should match the interaction rather than maximizing geometric complexity unnecessarily.
9. Presence Detection
One of the simplest useful abstractions is presence.
Instead of sending thousands of measurements to the experience, the processing system can output:
presence = 0
or:
presence = 1
The transformation becomes:
System Architecture
Left → Right
This is sufficient for interactions such as:
- starting an experience
- switching from idle content
- activating lighting
- beginning a research session
- detecting room vacancy
A simple binary state can therefore be more useful than the original high-resolution sensor stream.
10. Position Data
Other interactions require continuous position.
A tracked visitor may be represented as:
visitor_id = 1
x = 2.31
y = 1.72
The experience can then use position for:
- particle interaction
- sound positioning
- visual deformation
- proximity effects
- movement analysis
- trajectory logging
The processing pipeline becomes:
System Architecture
Left → Right
This provides a reusable interface independent of the original sensor format.
11. Normalized Coordinates
Physical coordinates can be normalized for easier use in real-time applications.
For example:
room x = 0 → 6 m
room y = 0 → 4 m
can become:
normalized x = 0 → 1
normalized y = 0 → 1
The transformation is conceptually:
nx = (x - xmin) / (xmax - xmin)
ny = (y - ymin) / (ymax - ymin)
This creates an interface such as:
visitor_x = 0.38
visitor_y = 0.64
A TouchDesigner or Unity application can then map these values to its own visual coordinate system.
System Architecture
Left → Right
The sensing system no longer needs to know the rendering resolution.
12. Area Detection
Continuous position can be converted into discrete interaction areas.
For example:
AREA 1
AREA 2
AREA 3
AREA 4
AREA 5
Each area can be defined as a spatial region.
A simple rectangular region may use:
xmin
xmax
ymin
ymax
A visitor is inside the area when:
xmin <= x <= xmax
and
ymin <= y <= ymax
The pipeline becomes:
System Architecture
Left → Right
The output may be:
area_1 = 0
area_2 = 1
area_3 = 0
area_4 = 0
area_5 = 0
This is much easier for an experience runtime to consume than raw LiDAR points.
13. From Area State to Events
Area detection provides a state.
For example:
area_2 = 1
But experience logic often needs to know when the state changed.
Consider:
frame 1: area_2 = 0
frame 2: area_2 = 0
frame 3: area_2 = 1
frame 4: area_2 = 1
frame 5: area_2 = 1
frame 6: area_2 = 0
This can be interpreted as:
frame 3 → area_2_enter
frame 6 → area_2_exit
Conceptually:
System Architecture
Left → Right
This transformation is fundamental for stable interaction.
14. State Versus Event
A state describes an ongoing condition:
area_2_active = true
An event describes a transition:
area_2_enter
These should not be treated as interchangeable.
A continuous media effect may use the state:
while area_2_active
→ increase visual response
A one-time animation may use the event:
area_2_enter
→ trigger animation once
Maintaining both representations gives the experience more control.
15. Dwell Time
Interaction meaning can also depend on duration.
A visitor entering an area does not necessarily mean the interaction should activate immediately.
The system can track:
entry time
current time
duration
and calculate:
dwell_time = current_time - entry_time
A condition may then be:
if dwell_time >= threshold
→ interaction_confirmed
The pipeline becomes:
System Architecture
Left → Right
Dwell time can serve both experience design and research analysis.
16. Debouncing and Stability
Spatial sensors may fluctuate near boundaries.
A visitor standing close to an area edge could produce:
inside
outside
inside
outside
inside
within a short period.
If every change becomes an interaction event, the experience may flicker or repeatedly trigger.
Stability mechanisms can include:
- minimum activation time
- minimum exit time
- hysteresis
- cooldown
- temporal smoothing
For example:
detected inside
↓
remain inside for threshold
↓
confirmed inside
This separates sensor fluctuation from meaningful interaction.
17. Interaction Data Interface
Once processed, the sensor system can expose a compact interaction interface.
For example:
presence
visitor_count
visitor_1_x
visitor_1_y
area_1
area_2
area_3
area_4
area_5
Events may include:
visitor_enter
visitor_exit
area_1_enter
area_1_exit
interaction_confirmed
This interface becomes the contract between the sensing infrastructure and the experience runtime.
System Architecture
Left → Right
The runtime does not need to understand the original sensor protocol.
18. OSC as an Interaction Transport
Interaction data can be transmitted through a protocol such as OSC.
Instead of transmitting hardware-specific data, the system may expose addresses such as:
/presence
/visitor/1/x
/visitor/1/y
/area/1
/area/2
/area/3
or events:
/event/visitor_enter
/event/area_enter
The architectural distinction is important:
sensor protocol
≠
interaction protocol
The sensor layer may communicate with hardware using one protocol while the experience layer receives a simpler application-facing representation.
19. Edge Processing
Sensor processing can run on a separate edge computer such as a Raspberry Pi.
Conceptually:
System Architecture
Left → Right
The edge system can perform tasks such as:
- sensor acquisition
- filtering
- coordinate transformation
- area detection
- event generation
- basic logging
The rendering workstation can then focus on:
- real-time graphics
- audio
- scene logic
- media playback
This separation supports the modular architecture introduced in K01.
20. Interaction Data and Scene Logic
The processed data becomes input to the scene architecture introduced in K02.
For example:
raw LiDAR points
↓
visitor position
↓
area detection
↓
area_enter
↓
interaction state
↓
scene response
The complete relationship becomes:
System Architecture
Left → Right
K03 therefore provides the transformation layer between sensing infrastructure and experience logic.
21. Research Data
The same processing pipeline can produce research data.
Useful records may include:
session_id
timestamp
visitor_id
x
y
area
event
For example:
S001,0.00,1,1.82,2.10,,session_start
S001,1.24,1,1.94,2.18,1,area_enter
S001,2.31,1,2.08,2.24,1,
S001,5.88,1,2.52,2.40,1,area_exit
This creates two related forms of data:
Continuous data
x
y
timestamp
Useful for:
- trajectories
- movement patterns
- heatmaps
- spatial distribution
Event data
area_enter
area_exit
interaction
scene_change
Useful for:
- interaction count
- dwell time
- sequence analysis
- experience progression
22. Session Identity
Research data should be associated with a session rather than stored as an undifferentiated stream.
For example:
session_id = S001
A session may begin when:
visitor enters
and end when:
experience completes
or:
room becomes empty
All systems involved in logging should use the same session identifier where possible.
System Architecture
Left → Right
This allows spatial data and experience events to be aligned later.
23. Trajectory Data
A trajectory can be represented as a sequence:
(x1, y1, t1)
(x2, y2, t2)
(x3, y3, t3)
...
This allows analysis of:
- movement paths
- speed
- direction
- spatial preference
- transitions between regions
- return behaviour
The important architectural point is that trajectory logging should use the shared room coordinate system.
Otherwise, data from different sensor configurations may not be directly comparable.
24. Heatmaps
Trajectory samples can be aggregated spatially.
Conceptually:
trajectory points
↓
spatial bins
↓
frequency / duration
↓
heatmap
A heatmap can describe where visitors:
- move frequently
- remain for longer periods
- rarely enter
- cluster around interactions
However, a heatmap alone does not explain why behaviour occurred.
Combining it with scene and interaction events provides stronger contextual interpretation.
25. Dwell Data
Dwell can be measured at several levels.
Area Dwell
time spent inside Area 1
Interaction Dwell
time engaged with a specific interaction
Scene Dwell
time spent in ACTIVE scene
Session Duration
session_start → session_end
These measurements should remain conceptually distinct even when derived from the same timestamps.
26. Interaction Count
Interaction count should also have an explicit definition.
For example, counting every sensor frame while a visitor remains inside an area would produce a misleading result.
Instead:
outside → inside
may count as:
1 interaction entry
or an interaction may only count after:
entry
+
minimum dwell
+
confirmation
The research definition should therefore match the experience definition.
This is another reason to derive research events from interaction states rather than directly from raw measurements.
27. Multi-Sensor Research Logging
When multiple sensors contribute to one tracked visitor, research data should ideally represent the shared spatial estimate rather than duplicate hardware observations.
Instead of:
LiDAR_A point
LiDAR_B point
the research layer may record:
visitor_1
x
y
timestamp
Sensor-specific diagnostic data can still be stored separately when needed.
This creates a distinction between:
system diagnostics
and:
research observations
Both are useful, but they answer different questions.
28. Diagnostic Data
A robust sensing system may also expose diagnostic information such as:
sensor_connected
sensor_last_update
tracking_confidence
active_sensor_count
processing_rate
This information should generally remain separate from experience-facing interaction data.
Conceptually:
System Architecture
Left → Right
One processing system can therefore support three different consumers.
29. Data Layer Separation
The overall architecture can be understood as three data layers.
Sensor Data
Hardware-oriented measurements:
range
angle
depth
point
Interaction Data
Experience-oriented interpretation:
presence
position
area
enter
exit
dwell
Research Data
Analysis-oriented records:
session
timestamp
trajectory
interaction count
dwell time
scene event
The relationship is:
System Architecture
Left → Right
This separation prevents one representation from being forced to serve every purpose.
30. Why the Separation Matters
Without a transformation layer, a TouchDesigner or Unity project may become responsible for:
sensor connection
+
sensor filtering
+
coordinate conversion
+
tracking
+
area detection
+
interaction logic
+
rendering
+
research logging
This creates strong coupling.
A modular pipeline instead separates:
SENSING
↓
SPATIAL PROCESSING
↓
INTERACTION DATA
↓
EXPERIENCE LOGIC
↓
MEDIA RESPONSE
The installation can then replace individual components more easily.
For example:
LiDAR
↓
Depth Camera
does not necessarily require rewriting scene logic if both systems expose compatible interaction data.
31. Sensor Independence
Consider two implementations.
LiDAR Implementation
System Architecture
Left → Right
Depth Camera Implementation
System Architecture
Left → Right
The processing methods differ.
The experience-facing interface can remain similar.
This creates sensor independence at the interaction layer.
32. Runtime Independence
The same principle can apply in the opposite direction.
A spatial processing server may output:
presence
x
y
area
events
These values can be consumed by:
TouchDesigner
Unity
Unreal Engine
lighting control
audio systems
research logger
The sensor infrastructure therefore becomes reusable across multiple real-time runtimes.
33. Calibration as Configuration
Sensor placement and coordinate transformation should ideally be treated as configuration rather than hard-coded experience logic.
For example:
sensor_id
position_x
position_y
rotation
room_width
room_depth
This allows the same processing software to adapt to different installation spaces.
The architecture becomes:
System Architecture
Left → Right
This is particularly useful for systems intended to move between installations or research sites.
34. Research Infrastructure
Once sensor processing, interaction interpretation, and logging use stable interfaces, the system becomes more than an installation-specific solution.
It becomes research infrastructure.
The same architecture can support repeated studies while allowing individual projects to change:
- sensor configuration
- interaction design
- media content
- room geometry
- research questions
The common data model provides continuity across these changes.
35. Design Principles
Several principles emerge from this architecture.
Process measurements before exposing them to experience logic
Experience applications should consume meaningful spatial information whenever possible.
Use a shared coordinate system
Interaction regions and research data should not depend unnecessarily on individual sensor coordinates.
Separate state from events
Continuous conditions and transitions serve different purposes.
Separate interaction data from research data
The experience may need immediate values while research requires persistent, timestamped records.
Define interaction metrics explicitly
Dwell time and interaction count should have clear operational definitions.
Keep hardware details below the interaction interface
Replacing a sensor should not automatically require redesigning the entire experience.
Preserve diagnostic visibility
Abstraction should not make sensor failures impossible to inspect.
36. Limitations
Additional processing layers introduce complexity.
Possible challenges include:
- calibration errors
- synchronization between sensors
- network latency
- tracking ambiguity
- multi-user identity
- sensor occlusion
- processing delay
- inconsistent sampling rates
A modular architecture does not eliminate these problems.
It makes their location in the system clearer.
For simple installations, direct sensor-to-runtime integration may still be sufficient.
37. Relationship to K01 and K02
The first three Knowledge articles describe different architectural layers.
K01 — Modular Sensor Server Architecture
Focus:
sensor infrastructure
edge processing
communication
runtime separation
K02 — Scene-Based Interaction
Focus:
interaction states
experience scenes
transitions
responsive behaviour
K03 — Raw Sensor Data to Interaction Data
Focus:
measurement
filtering
coordinates
tracking
area detection
events
research data
Together:
System Architecture
Left → Right
The three articles form a layered model rather than independent technical notes.
38. Research Direction
The transformation from sensor measurements to interaction data also creates research questions.
Examples include:
- How much spatial precision is necessary for meaningful interaction?
- Which spatial abstractions remain reusable across installations?
- How should multiple sensors represent one visitor?
- How do filtering and dwell thresholds affect perceived responsiveness?
- How can trajectory and event data be synchronized across distributed systems?
- Which interaction representations are most useful for longitudinal comparison?
- How can the same sensing infrastructure support both experience control and research observation?
A broader question is:
How can spatial sensing systems transform uncertain physical measurements into stable interaction representations without removing the behavioural information needed for research?
This positions sensor processing as both a technical and methodological concern.
39. Key Takeaways
Raw sensor data should not be treated as interaction data.
A useful transformation pipeline is:
Measurement → Filtering → Coordinate Transformation → Tracking → Detection → State → Event
From this pipeline, the system can produce reusable information such as:
presence
position
area
enter
exit
dwell
trajectory
These representations can serve two different destinations:
System Architecture
Left → Right
This makes spatial sensing useful not only for controlling an installation, but also for observing and evaluating how people behave within it.
The broader principle is:
Convert hardware-specific measurements into stable spatial meaning before allowing them to define experience behaviour or research evidence.