Overview
Responsive environments rarely respond to sensor data in isolation.
A person entering a room, stepping into an interaction zone, remaining in one location, selecting an object, or leaving the environment can each represent a meaningful change in the experience.
Rather than connecting every sensor event directly to individual media effects, these observations can be interpreted through interaction states and scenes.
A sensor describes what is happening in space. A scene describes what the experience should become because of it.
Scene-based interaction introduces an intermediate experience layer between spatial sensing and media output.
1. From Sensor Events to Experience Behaviour
A direct interaction may be represented as:
System Architecture
Left → Right
This works well for a simple interaction.
As the experience becomes more complex, however, one sensor event may affect several systems simultaneously.
Entering an interaction area might:
- change visual content
- activate an audio layer
- disable another interaction
- start a timer
- update lighting
- change the next available interaction
- create a research event
The relationship is therefore no longer simply:
Input → Effect
A more useful model is:
System Architecture
Left → Right
The interaction state becomes an abstraction between sensing and presentation.
2. What Is a Scene?
A scene represents a coherent condition of the experience.
For example, an immersive installation may contain:
IDLE
INTRO
ACTIVE
INTERACTION
COMPLETION
RESET
Each scene can define multiple behaviours simultaneously.
IDLE
The environment waits for a visitor.
Possible behaviour:
- ambient visual loop
- background audio
- interaction disabled
- entrance available
INTRO
A visitor has entered and the experience begins.
Possible behaviour:
- introductory content
- lighting transition
- interaction preparation
- session initialization
ACTIVE
The main interactive environment becomes available.
Possible behaviour:
- spatial tracking enabled
- interaction zones active
- dynamic visuals running
- visitor movement recorded
COMPLETION
The experience has reached its ending condition.
Possible behaviour:
- final content
- interaction disabled
- exit control activated
- session data finalized
RESET
The system prepares for the next visitor.
Possible behaviour:
- clear temporary state
- reset media
- reset interaction counters
- return to idle
The scene therefore coordinates several systems through one experience-level state.
3. Scene-Based Architecture
A basic scene architecture can be represented as:
System Architecture
Left → Right
The Scene Controller does not need to know how every sensor works.
It receives interpreted events such as:
visitor_entered
area_2_active
interaction_completed
visitor_left
It then determines the appropriate experience state.
4. Spatial State and Experience State
It is useful to distinguish two different forms of state.
Spatial State
Spatial state describes the physical situation.
Examples:
room_empty
visitor_present
visitor_in_area_1
visitor_in_area_2
multiple_visitors
Experience State
Experience state describes the logical condition of the installation.
Examples:
idle
intro
active
waiting
completed
resetting
The two are related but should not always be identical.
For example:
System Architecture
Left → Right
Later, the visitor may remain present while the experience changes:
System Architecture
Left → Right
The physical condition remains similar.
The experience state changes.
This separation prevents spatial sensing from becoming the entire experience logic.
5. Interaction States
Scenes can contain smaller interaction states.
Consider an environment with five interaction areas.
The main scene may remain:
ACTIVE
while interaction states change continuously:
AREA_1_ACTIVE
AREA_2_ACTIVE
AREA_3_ACTIVE
AREA_4_ACTIVE
AREA_5_ACTIVE
This creates a hierarchy:
EXPERIENCE
│
├── IDLE
├── INTRO
├── ACTIVE
│ ├── AREA 1
│ ├── AREA 2
│ ├── AREA 3
│ ├── AREA 4
│ └── AREA 5
├── COMPLETION
└── RESET
The main scene controls the overall experience.
Interaction states control local responses within that scene.
6. Trigger-Based Interaction
Some interactions are event-like.
For example:
visitor enters area
↓
trigger event
↓
start animation
The important information is that a boundary was crossed.
The system may generate:
area_enter
and later:
area_exit
A trigger-based interaction can be represented as:
System Architecture
Left → Right
This model is useful for:
- starting content
- selecting scenes
- activating sound
- opening doors
- counting interactions
- starting timers
7. Continuous Interaction
Other interactions depend on continuously changing spatial information.
For example:
visitor position
↓
normalized coordinates
↓
visual parameter
Instead of producing one trigger, the system continuously updates values.
System Architecture
Left → Right
This may control:
- particle attraction
- visual deformation
- sound position
- light intensity
- object movement
- camera effects
Trigger-based and continuous interaction can exist within the same scene.
8. Combining Trigger and Continuous Interaction
Consider an interactive floor.
Entering an area may first activate an interaction:
AREA ENTER
→ interaction enabled
Once active, position within that area may continuously control the media:
POSITION
→ visual parameter
Leaving the area may stop the interaction:
AREA EXIT
→ interaction disabled
The complete model becomes:
System Architecture
Left → Right
This avoids treating every sensor measurement as an independent event.
9. Scene Transitions
A scene-based system requires explicit transition conditions.
For example:
IDLE
↓ visitor detected
INTRO
↓ intro finished
ACTIVE
↓ completion condition
COMPLETION
↓ visitor exits
RESET
↓ reset finished
IDLE
A transition should answer two questions:
- What condition causes the transition?
- What happens when the transition occurs?
For example:
| Current Scene | Condition | Next Scene |
|---|---|---|
| Idle | visitor detected | Intro |
| Intro | intro finished | Active |
| Active | experience completed | Completion |
| Completion | visitor leaves | Reset |
| Reset | reset complete | Idle |
Explicit transitions make complex installations easier to reason about.
10. Scene Entry and Exit Actions
A scene can define actions that occur when entering or leaving it.
For example:
Enter ACTIVE
enable tracking
enable interaction zones
start ambient media
start session timer
Exit ACTIVE
disable interaction zones
stop temporary effects
store interaction count
This reduces duplicated logic.
Instead of several components independently deciding what to do, scene transitions provide a predictable lifecycle.
11. Example — Layered Interactive Floor
An interactive floor can map different areas to different media layers.
For example:
| Area | Interaction Response |
|---|---|
| Area 1 | Rhythm |
| Area 2 | Bass |
| Area 3 | Synth |
| Area 4 | Vocal |
| Area 5 | FX |
The system does not need to change the entire scene whenever a visitor changes areas.
The overall state may remain:
ACTIVE
while local interaction states determine which media layers respond.
System Architecture
Left → Right
This is an example of scene-level stability with interaction-level variation.
12. Example — Room Experience
A room-based experience may use a larger scene structure.
ROOM EMPTY
↓
IDLE LOOP
visitor enters
↓
INTRO
intro finishes
↓
INTERACTIVE CONTENT
experience finishes
↓
EXIT STATE
visitor leaves
↓
RESET
↓
IDLE LOOP
The architecture can be represented as:
System Architecture
Left → Right
The same scene state can coordinate:
- video
- interactive content
- lighting
- doors
- audio
- system control
This becomes especially useful in multi-room installations.
13. Distributed Scene Control
A scene does not need to exist entirely inside one application.
For example:
System Architecture
Left → Right
The room controller may know:
room_empty
room_active
experience_finished
Unity may know:
intro
interactive
completion
The media server may know:
loop_input
interactive_input
These systems can coordinate through shared state transitions without becoming one monolithic application.
14. Scene State as a Shared Contract
In distributed installations, state can function as a contract between systems.
For example:
{
"zone": 1,
"state": "active"
}
or:
{
"zone": 1,
"event": "experience_finished"
}
The receiving system does not need to understand the entire experience.
It only needs to understand the agreed state or event.
This makes distributed control easier to maintain.
15. Scene-Based Media Routing
Scene state can also determine which media source should be presented.
For example:
IDLE
→ Loop Content
ACTIVE
→ Real-Time Interactive Content
COMPLETION
→ Ending Content
The routing architecture may become:
System Architecture
Left → Right
The experience runtime and media server can therefore remain separate while sharing a common scene model.
16. Reset as a First-Class Scene
Reset behaviour is frequently treated as an afterthought.
For installations, it should be considered part of the experience architecture.
A reset may need to:
- clear interaction variables
- stop timers
- reset media layers
- clear temporary visitor information
- close the previous research session
- prepare physical controls
- restore the idle state
Therefore:
ACTIVE
→ COMPLETION
→ RESET
→ IDLE
is usually safer than:
ACTIVE
→ IDLE
The explicit RESET scene provides a controlled boundary between visitors or sessions.
17. Preventing Repeated Triggers
Continuous sensor measurements can accidentally trigger the same event repeatedly.
For example:
visitor remains in Area 1
frame 1 → Area 1
frame 2 → Area 1
frame 3 → Area 1
frame 4 → Area 1
If every frame is interpreted as a trigger, the experience may repeatedly restart the same behaviour.
State allows the system to distinguish:
OUTSIDE → INSIDE
from:
INSIDE → INSIDE
Only the first transition needs to generate:
area_enter
This is a fundamental reason to introduce state between sensing and media behaviour.
18. Temporal Conditions
Spatial interaction often depends on time as well as position.
For example:
visitor enters area
↓
remains for 2 seconds
↓
activate interaction
This can reduce accidental activation.
Another example:
interaction triggered
↓
cooldown 3 seconds
↓
interaction available again
Scene-based logic can incorporate:
- dwell thresholds
- cooldowns
- timeouts
- delays
- minimum interaction duration
Spatial and temporal conditions together create more stable interaction behaviour.
19. Multi-User Considerations
Scene logic becomes more complex when several visitors are present.
A system must decide whether state belongs to:
- the room
- an interaction area
- an individual visitor
- the entire installation
For example:
ROOM STATE
active
AREA 1 STATE
occupied
AREA 2 STATE
occupied
VISITOR A
area_1
VISITOR B
area_2
These are different levels of state.
A single global variable such as:
current_area = 1
may be insufficient for multi-user environments.
Scene architecture should therefore reflect the scale at which interaction occurs.
20. Experience State and Research Logging
Scene transitions can also provide useful research events.
For example:
session_start
intro_start
active_start
area_2_enter
area_2_exit
completion
session_end
A log might contain:
session_id,timestamp,event,value
S001,0.00,session_start,
S001,4.21,active_start,
S001,7.84,area_enter,2
S001,12.42,area_exit,2
S001,58.30,completion,
S001,62.11,session_end,
This allows interaction logic and research instrumentation to share meaningful event boundaries.
21. Scene State and Continuous Data
Research logging does not need to choose between events and continuous tracking.
Both can be recorded.
System Architecture
Left → Right
The trajectory describes movement.
The event log describes experience structure.
Together they provide richer interpretation than either dataset alone.
22. Why Scene-Based Interaction Improves Reuse
Suppose two installations use different sensors.
Installation A:
System Architecture
Left → Right
Installation B:
System Architecture
Left → Right
The sensor and runtime differ.
But both can share concepts such as:
IDLE
ACTIVE
COMPLETION
RESET
Scene logic therefore provides another reusable layer above sensor abstraction.
K01 separates sensor infrastructure from experience runtime.
K02 extends that idea by separating interaction observations from experience behaviour.
23. Failure Recovery
Explicit scene state also helps systems recover from unexpected conditions.
For example:
ACTIVE
visitor disappears unexpectedly
↓
timeout
↓
RESET
↓
IDLE
Without a clear state model, temporary sensor loss may leave the experience in an undefined condition.
Recovery rules can be defined for each state.
Examples include:
- no visitor detected for a timeout period
- runtime disconnected
- content playback failed
- interaction exceeded maximum duration
This is particularly important for unattended installations.
24. Scene-Based Interaction as Experience Infrastructure
Scene-based interaction should not be understood only as a programming technique.
It becomes part of the experience infrastructure.
The architecture can be summarized as:
System Architecture
Left → Right
Each layer answers a different question.
Human Activity
What is the visitor doing?
Spatial Observation
What can the sensing system observe?
Interaction State
What does that observation mean for interaction?
Experience State
What condition is the experience currently in?
Responsive Behaviour
What should the environment do?
Keeping these questions separate makes complex responsive environments easier to design and maintain.
25. Design Lessons
Several design principles emerge from scene-based interaction.
Do not make every sensor value a media command
Interpret spatial information before deciding how the experience responds.
Distinguish events from states
area_enter is an event.
area_active is a state.
They serve different purposes.
Separate spatial state from experience state
A visitor being present does not uniquely determine what scene should be active.
Make transitions explicit
A clear transition model reduces hidden dependencies.
Treat reset as part of the architecture
Every repeatable installation needs a predictable path back to its initial condition.
Design state at the correct scale
Room state, area state, and visitor state should not automatically be represented by the same variable.
Log meaningful transitions
Scene changes can provide valuable structure for later research analysis.
26. Limitations
Scene-based architecture can become unnecessarily complicated if every minor interaction is represented as a separate scene.
For example:
AREA_1
AREA_2
AREA_3
may be better represented as interaction states inside one ACTIVE scene rather than three complete scenes.
Too many states can create:
- difficult transition graphs
- duplicated behaviour
- hidden dependencies
- difficult debugging
The purpose of scene architecture is to simplify experience logic, not merely move complexity into a state machine.
27. When Scene-Based Interaction Is Useful
The approach becomes particularly valuable when an environment contains:
- multiple stages of experience
- several interaction zones
- repeated visitor sessions
- physical and digital responses
- distributed computers
- timed interaction
- multi-room control
- research logging
- automatic reset requirements
Simpler installations may still use direct interaction logic.
As with modular sensor infrastructure, architecture should match the scale of the system.
28. Relationship to Modular Sensor Architecture
Scene-based interaction sits above the sensor infrastructure described in Modular Sensor Server Architecture.
Together, the two patterns form:
System Architecture
Left → Right
K01 focuses primarily on:
Sensor → Processing → Communication
K02 focuses primarily on:
Interaction State → Scene → Experience Behaviour
The separation allows each part of the system to evolve independently.
29. Research Direction
Scene-based interaction creates several research opportunities.
Future investigation can examine:
- how visitors transition between spatial states
- how dwell time affects scene activation
- how multiple visitors influence shared experience state
- how scene transitions affect perceived continuity
- how physical and digital states remain synchronized
- how interaction state can support longitudinal studies
- how scene models transfer between installations
A broader question emerges:
How can responsive environments transform continuous human activity into coherent experience states without reducing spatial behaviour to a collection of isolated triggers?
This question connects technical architecture with interaction design and spatial experience research.
30. Key Takeaways
Scene-based interaction introduces a structured layer between sensing and media behaviour.
The basic progression is:
Human Activity → Spatial Observation → Interaction State → Experience Scene → Responsive Behaviour
It distinguishes:
- sensor data from interaction meaning
- events from states
- spatial state from experience state
- local interaction from global scene control
- experience behaviour from research observation
Combined with modular sensor infrastructure, scene-based interaction provides a foundation for responsive environments that are easier to reuse, evaluate, maintain, and extend.
The broader principle is:
Design the environment around meaningful changes in experience state, rather than connecting every sensor event directly to a media effect.