Multi-Sensor Spatial Fusion & Coordinate Calibration

Overview

A single spatial sensor observes the environment from one physical position.

This creates limitations.

A visitor may become partially or completely invisible because of:

• another visitor
• installation structures
• limited field of view
• sensor range
• room geometry
• physical occlusion

Adding more sensors can increase spatial coverage, but multiple sensors create a new problem:

each sensor observes the environment from its own coordinate system.

Before their measurements can describe one shared environment, those observations must be transformed into a common spatial reference.

The basic architecture is:

System Architecture

Left → Right

Multi-sensor fusion is not simply the process of connecting more sensors. It is the process of making independent observations describe the same physical space.

---

1. The Single-Sensor Problem

Consider one LiDAR installed on one side of a room.

┌─────────────────────────────┐
│                             │
│                             │
│          VISITOR            │
│                             │
│                             │
●                             │
Sensor A                      │
└─────────────────────────────┘

The sensor can observe objects visible from its position.

However, another object may block the line of sight.

Sensor
  ↓

● ───── Visitor A ───── Visitor B

Visitor B may become partially invisible.

This is an occlusion problem.

A second sensor observing from another direction can provide additional information.

---

2. Multiple Viewpoints

Consider two sensors on opposite sides of the environment.

Sensor A ● ───────── SPACE ───────── ● Sensor B

An object hidden from Sensor A may remain visible to Sensor B.

Conceptually:

System Architecture

Left → Right

The additional viewpoint can improve spatial coverage.

However, Observation A and Observation B cannot necessarily be combined directly.

They first need to describe the same coordinate system.

---

3. Sensor-Local Coordinates

Every spatial sensor begins with its own origin.

For Sensor A:

Sensor A origin

(0,0)
  ●────────────→ +X
  |
  |
  ↓
 +Y

Sensor B may have:

Sensor B origin

                  (0,0)
                    ●
                 ←──┘

A visitor standing at one physical location may therefore be measured as:

Sensor A
x = 2.4
y = 1.3

while Sensor B reports:

Sensor B
x = 1.8
y = 2.7

Both observations may be correct relative to their own origins.

They are not yet directly comparable.

---

4. Shared Room Coordinates

A common coordinate system can represent the physical environment independently from individual sensors.

For example:

Room Coordinates

(0,0)
┌──────────────────────────────┐
│                              │
│                              │
│                              │
│                              │
└──────────────────────────────┘
                         (W,D)

where:

W = room width
D = room depth

Every sensor observation is transformed into this coordinate system.

The architecture becomes:

System Architecture

Left → Right

Once transformed, the same visitor should appear at approximately the same room coordinate regardless of which sensor observed them.

---

5. Coordinate Transformation

A basic 2D transformation usually requires:

translation X
translation Y
rotation

A local point:

(x_local, y_local)

is transformed into:

(x_room, y_room)

using the sensor's calibration parameters.

A conceptual rotation is:

x' = x cos(θ) - y sin(θ)

y' = x sin(θ) + y cos(θ)

followed by translation:

x_room = x' + tx

y_room = y' + ty

where:

θ  = sensor rotation
tx = sensor X position
ty = sensor Y position

This creates the relationship:

LOCAL SENSOR SPACE
        ↓
rotation
        ↓
translation
        ↓
SHARED ROOM SPACE

---

6. Sensor Pose

The position and orientation of a sensor can be represented as its pose.

For a 2D sensor:

sensor_id
position_x
position_y
rotation

Example:

These values describe how each sensor coordinate system relates to the room.

The exact values depend on the physical installation.

---

7. Calibration

Calibration is the process of determining the transformation between sensor space and shared space.

Conceptually:

System Architecture

Left → Right

The goal is:

known physical point
≈
transformed sensor point

If a reference point is physically located at:

room coordinate
(2.0, 1.0)

the transformed sensor observation should appear close to:

(2.0, 1.0)

The difference between them represents calibration error.

---

8. Calibration References

Calibration requires known spatial references.

Possible references include:

• room corners
• wall intersections
• measured floor markers
• temporary calibration objects
• known sensor positions
• known interaction-zone boundaries

For example:

Reference A = (0,0)
Reference B = (6,0)
Reference C = (6,4)
Reference D = (0,4)

These points provide a physical reference against which transformed measurements can be checked.

---

9. Calibration Procedure

A conceptual calibration workflow is:

1. Measure room geometry
2. Define room origin
3. Record sensor position
4. Record sensor orientation
5. Observe known reference points
6. Transform measurements
7. Compare with expected coordinates
8. Adjust calibration
9. Validate across the usable interaction area
10. Save configuration

The important point is that calibration should be treated as a repeatable procedure rather than a collection of manually tuned values hidden inside the runtime.

---

10. Calibration Configuration

Calibration parameters should ideally be stored as configuration.

For example:

sensor_id = lidar_left
x = 0.00
y = 2.00
rotation = 0.0

Another sensor may use:

sensor_id = lidar_right
x = 6.00
y = 2.00
rotation = 180.0

Conceptually:

System Architecture

Left → Right

This allows sensor placement to change without rewriting interaction logic.

---

11. Calibration Versioning

Physical installations can change.

A sensor may be:

• moved
• replaced
• rotated
• remounted
• recalibrated

Research data should therefore be associated with the calibration used during collection.

For example:

calibration_id = FL-FAM-CAL-01

or:

calibration_version = 1.2

This provides traceability when spatial data from different collection periods is compared.

---

12. Transformation Pipeline

Each sensor can independently transform its observations.

System Architecture

Left → Right

After this stage:

Room Points A
Room Points B

share the same spatial reference.

Only then should fusion occur.

---

13. Why Transform Before Fusion?

Consider:

Sensor A point
(2.0,1.0)

and:

Sensor B point
(2.1,1.1)

Without knowing the sensor coordinate systems, these values cannot safely be interpreted as nearby physical points.

After transformation:

Sensor A → Room (3.20,1.70)
Sensor B → Room (3.24,1.68)

the system can reasonably infer that both sensors may be observing the same physical object.

Coordinate normalization therefore precedes spatial association.

---

14. Overlapping Sensor Coverage

Sensors may observe overlapping regions.

Sensor A coverage
██████████████
       ██████████████
       Sensor B coverage

The overlapping region can be valuable because multiple sensors provide observations of the same space.

However, it can also create duplicate detections.

For example:

Sensor A
→ Visitor 1

Sensor B
→ Visitor 1

should not automatically become:

Visitor 1
Visitor 2

The fusion layer must determine whether observations belong to the same tracked entity.

---

15. Spatial Association

A basic association rule can use spatial distance.

Suppose:

Observation A
(2.40,1.50)

Observation B
(2.48,1.46)

The Euclidean distance is:

d = √((x2-x1)² + (y2-y1)²)

If:

d < association_threshold

the observations may be treated as candidates for the same physical object.

Conceptually:

System Architecture

Left → Right

Distance alone is not always sufficient, but it provides a useful conceptual starting point.

---

16. Fused Position

If two observations are associated with the same tracked object, the system can produce one fused position.

For example:

Sensor A
(2.40,1.50)

Sensor B
(2.48,1.46)

may become:

Fused Visitor
(2.44,1.48)

A simple implementation may use an average.

More advanced systems may use:

• weighted averaging
• confidence weighting
• temporal tracking
• probabilistic filtering
• sensor-specific reliability

The fusion method should match the precision and complexity required by the application.

---

17. Fusion Does Not Always Mean Averaging

Multiple observations do not always need to be mathematically averaged.

Fusion may occur at several levels.

Point-Level Fusion

Sensor A points
+
Sensor B points
→ combined point representation

Cluster-Level Fusion

Cluster A
+
Cluster B
→ shared object estimate

Track-Level Fusion

Track A
+
Track B
→ visitor track

State-Level Fusion

Sensor A says Area 1 occupied
+
Sensor B says Area 1 occupied
→ Area 1 occupied

The appropriate fusion level depends on what the experience needs.

---

18. Choosing the Fusion Level

If an installation only needs:

presence = true / false

point-level fusion may be unnecessary.

If it requires:

precise visitor trajectory

track-level or spatial fusion may be more appropriate.

The architecture should therefore begin with:

What spatial information does the experience or research question actually require?

rather than:

How can every sensor point be merged?

---

19. Occlusion

Occlusion occurs when a physical object blocks the sensor's view of another object.

For example:

Sensor A

● ─── Visitor 1 ─── Visitor 2

Sensor A may observe Visitor 1 while losing Visitor 2.

Sensor B may observe the same space from another direction:

Visitor 1 ─── Visitor 2 ─── ●

                           Sensor B

The second viewpoint can preserve information that the first sensor cannot observe.

This is one of the main reasons to use multi-sensor coverage.

---

20. Occlusion Is Not Eliminated

Adding sensors reduces some forms of occlusion but does not guarantee complete visibility.

Possible remaining problems include:

• several visitors blocking each other
• furniture
• projection structures
• sensor blind spots
• visitors standing very close together
• insufficient sensor height or placement

The objective is therefore not:

zero occlusion

but rather:

sufficient spatial observability
for the intended interaction

---

21. Coverage Planning

Sensor placement should be considered before calibration.

A conceptual planning process is:

room geometry
      ↓
interaction zones
      ↓
expected visitor movement
      ↓
possible occlusion
      ↓
sensor placement

Sensors should be placed according to the interaction problem rather than simply distributed symmetrically.

For example, an interaction area with high expected visitor density may benefit from overlapping coverage.

---

22. Side and Floor Sensing

Different sensor placements can observe different aspects of the environment.

A side-mounted planar sensor may be useful for:

presence
body intersection
horizontal position

A floor-oriented sensing configuration may provide another spatial perspective.

The important architectural point is not the mounting direction itself.

It is that every sensing plane must eventually map its useful observations into the same interaction-space representation.

System Architecture

Left → Right

The transformation model may differ depending on sensor orientation and geometry.

---

23. 2D and 3D Sensors

A responsive environment may combine different sensing technologies.

For example:

2D LiDAR
+
Depth Camera

The LiDAR may provide:

x
y

while the depth camera provides:

x
y
z

A shared interaction layer does not necessarily need to preserve all dimensions from every sensor.

For floor-based spatial interaction, the system may project relevant observations into:

room X
room Y

Conceptually:

System Architecture

Left → Right

This allows heterogeneous sensors to contribute to the same interaction model.

---

24. Sensor Roles

Not every sensor needs to perform the same role.

A system may assign:

Sensor A
→ primary position tracking

Sensor B
→ occlusion support

Depth Camera
→ body / gesture information

This is different from assuming every sensor contributes equally to every measurement.

A role-based architecture can simplify fusion.

System Architecture

Left → Right

---

25. Confidence

Some fusion systems may associate confidence with observations.

Conceptually:

Sensor A
position = (2.4,1.5)
confidence = high

Sensor B
position = (2.6,1.4)
confidence = low

The fused estimate may give greater influence to Sensor A.

A generic weighted position can be represented as:

x_fused =
Σ(x_i × w_i) / Σ(w_i)

y_fused =
Σ(y_i × w_i) / Σ(w_i)

where:

w_i = observation weight

The exact confidence model depends on the sensing technology and tracking implementation.

---

26. Temporal Continuity

Spatial proximity alone can become ambiguous when several visitors are close together.

Tracking over time adds another source of information.

For example:

t0 → visitor at (1.0,1.0)
t1 → visitor at (1.1,1.0)
t2 → visitor at (1.2,1.1)

A new observation near the predicted movement path may be associated with the existing track.

Conceptually:

System Architecture

Left → Right

This helps maintain continuity across frames.

---

27. Track Identity

A fused tracking system may assign temporary identifiers:

V01
V02
V03

These identifiers describe tracked entities within a session.

They should not automatically be interpreted as personal identity.

For example:

session_id = S0042
visitor_id = V01

means:

tracked entity V01 within session S0042

not:

known person V01

This distinction is important for both architecture and research data.

---

28. Track Loss

A visitor may temporarily disappear from all sensors.

Possible reasons include:

• occlusion
• leaving the sensing region
• sensor dropout
• clustering failure

The system should distinguish between:

temporarily lost

and:

visitor exited

A short timeout may preserve the track:

tracked
↓
temporarily lost
↓
reacquired

rather than immediately creating a new visitor ID.

---

29. Track Lifecycle

A simplified track lifecycle can be represented as:

This provides more stable interaction behaviour than treating every frame independently.

---

30. Fusion and Area Detection

Once the system produces a fused visitor position:

visitor_id = V01
x = 2.44
y = 1.48

area detection can operate on the fused result.

System Architecture

Left → Right

This is preferable to independently triggering the same interaction from several sensors when the intended unit is one visitor.

---

31. Fusion and Interaction Logic

The experience runtime should ideally receive:

V01
x
y
area

rather than:

lidar_A_point_42
lidar_B_cluster_3
camera_object_7

This preserves the abstraction introduced in K03.

The experience consumes interaction meaning rather than sensor implementation details.

---

32. Fusion and Research Logging

K04 established a trajectory schema such as:

In a multi-sensor system, these coordinates should normally represent the shared spatial estimate.

The primary research trajectory should not automatically duplicate the visitor once for every sensor.

---

33. Sensor-Level Diagnostic Data

Sensor-specific observations can still be useful for diagnostics.

For example:

After transformation:

The research-level estimate may then be:

This creates a useful distinction:

SENSOR OBSERVATION
↓
TRANSFORMED OBSERVATION
↓
FUSED RESEARCH OBSERVATION

---

34. Calibration Data Schema

Calibration itself should also be documented.

A reusable configuration schema may contain:

For 3D systems, additional fields may be required:

position_z
rotation_x
rotation_y
rotation_z

The schema should match the dimensionality of the sensing system.

---

35. Calibration Validation Schema

Calibration quality can also be documented.

For example:

The error for a point can be calculated as:

error =
√((x_observed - x_expected)²
+
(y_observed - y_expected)²)

This provides a quantitative record of calibration quality rather than relying only on visual inspection.

---

36. Calibration Across the Room

Testing only one reference point is insufficient.

A sensor can appear correctly aligned near one location while producing larger error elsewhere.

Validation points should therefore cover the usable interaction region.

Conceptually:

P1 ───────── P2 ───────── P3

│             P4             │

P5 ───────── P6 ───────── P7

The goal is to understand spatial error across the region where visitors actually interact.

---

37. Calibration Error Metrics

Possible summary metrics include:

mean error
maximum error
median error

For example:

These values can be useful when documenting research instrumentation.

However, acceptable error depends on the interaction.

A large interaction zone may tolerate more spatial error than a precise gesture boundary.

---

38. Interaction Boundary Tolerance

Calibration error and tracking noise become especially important near interaction boundaries.

Suppose:

Area 1 ends at x = 2.00

and the visitor is measured around:

1.98
2.02
1.99
2.03

The system may repeatedly alternate between areas.

Possible strategies include:

• hysteresis
• boundary margins
• dwell thresholds
• temporal smoothing

Fusion therefore does not replace the stability mechanisms introduced in K03.

It provides a better spatial estimate that those mechanisms can operate on.

---

39. Sensor Failure

A multi-sensor system should define behaviour when one sensor fails.

For example:

4 sensors active
↓
1 sensor disconnected
↓
3 sensors remain

Possible system responses include:

continue with reduced coverage
raise diagnostic warning
disable affected interaction region
stop session

The correct behaviour depends on whether the remaining sensors provide sufficient observability.

---

40. Graceful Degradation

One advantage of multiple sensors is the possibility of graceful degradation.

System Architecture

Left → Right

If one sensor becomes unavailable, the fusion layer can potentially continue using remaining observations.

The runtime does not necessarily need to change its interface.

It may still receive:

visitor_id
x
y
area

with reduced spatial confidence.

---

41. Diagnostic State

The fusion system can expose diagnostic information separately from interaction data.

For example:

active_sensor_count = 3
expected_sensor_count = 4
fusion_state = degraded

Possible states might include:

healthy
degraded
unavailable

These states can be recorded in the system log defined in K04.

---

42. Multi-Sensor Processing Architecture

A modular implementation can separate each sensor adapter from the fusion layer.

System Architecture

Left → Right

This allows different sensor technologies to coexist.

Each adapter handles hardware-specific communication.

The fusion engine works with normalized spatial observations.

---

43. Edge Processing

The fusion system can run on an edge computer when appropriate.

For example:

System Architecture

Left → Right

The edge layer may perform:

sensor acquisition
filtering
coordinate transformation
fusion
tracking
area detection
logging

The rendering workstation can remain focused on experience execution.

---

44. Distributed Fusion

Not every sensor must connect to the same computer.

A larger architecture may use:

Edge Node A
→ Sensor A + Sensor B

Edge Node B
→ Sensor C + Sensor D

Both nodes can transform observations into shared room coordinates before sending them to a central fusion process.

System Architecture

Left → Right

This becomes useful when sensors are physically distributed across a large environment.

---

45. Coordinate Contracts

Distributed spatial systems need agreement about what coordinates mean.

A coordinate contract should define:

origin
axis directions
units
room dimensions
coordinate dimensionality

For example:

origin = front-left floor corner

+X = room right

+Y = room depth

unit = metres

Without this agreement, two systems may transmit numerically valid coordinates that describe different physical meanings.

---

46. Normalization After Fusion

The fused room coordinates may later be normalized for experience applications.

For example:

Room coordinates

x = 0 → 6 m
y = 0 → 4 m

become:

Normalized

x = 0 → 1
y = 0 → 1

The preferred order is:

sensor coordinates
↓
room coordinates
↓
fusion
↓
shared spatial estimate
↓
normalized application coordinates

This preserves physical coordinates for research while allowing applications to use convenient normalized values.

---

47. Preserve Physical Coordinates for Research

Normalized coordinates are useful for rendering.

However, research logging should generally preserve physical room coordinates where possible.

For example:

x = 2.44 m
y = 1.48 m

is easier to interpret spatially than:

x = 0.4067
y = 0.3700

Normalized values can always be derived later if room geometry is known.

Physical coordinates also make measurements such as trajectory distance directly interpretable.

---

48. Room Configuration

The dataset should preserve the spatial configuration associated with the coordinates.

A room configuration may contain:

This provides context for interpreting trajectories later.

---

49. Interaction Area Configuration

Interaction regions should also be associated with the room coordinate system.

For a rectangular region:

Then:

visitor position
+
area configuration
→ area state

This makes interaction areas configurable rather than hidden inside application code.

---

50. Recalibration

Recalibration should occur when relevant physical conditions change.

Examples include:

sensor moved
sensor replaced
mount adjusted
room geometry changed
interaction floor changed
persistent tracking misalignment detected

The new calibration should receive a new version or identifier.

Previous research data should continue referencing the calibration under which it was collected.

---

51. Pre-Collection Validation

Before research data collection begins, a repeatable validation procedure can verify:

all sensors connected
↓
coordinate transforms loaded
↓
reference points checked
↓
overlap alignment checked
↓
interaction areas tested
↓
logging verified

This can become a deployment checklist.

It reduces the risk of discovering calibration problems after data has already been collected.

---

52. Daily Validation

For temporary research deployments, a lightweight daily check may be useful.

For example:

START OF DAY

1. verify sensor connectivity
2. check fixed reference points
3. inspect shared-space alignment
4. test interaction areas
5. verify logging
6. begin collection

The full calibration procedure does not necessarily need to be repeated every day if the installation has not moved.

The purpose is to detect unexpected physical or technical changes.

---

53. Fusion Validation

Calibration quality and fusion quality are related but different.

Calibration asks:

Does each sensor map correctly into room coordinates?

Fusion asks:

Do observations from multiple sensors become one stable spatial estimate?

A system may be well calibrated but still have poor association logic.

Therefore both should be tested separately.

---

54. Example Fusion Validation

A validation procedure may place one person at known locations.

For each location, record:

Sensor A estimate
Sensor B estimate
Fused estimate
Expected position

Example:

This can reveal whether fusion improves or degrades the spatial estimate.

---

55. Multi-Visitor Validation

Single-person calibration is not sufficient for installations expected to contain multiple visitors.

Testing should also include:

two visitors separated
two visitors crossing
one visitor occluding another
visitors near area boundaries
visitor entering / leaving coverage

These scenarios test tracking continuity and association rather than only geometric calibration.

---

56. Research Traceability

K04 introduced system versioning.

Multi-sensor systems add further configuration dependencies.

A research session may therefore reference:

session_id
system_version
room_configuration_id
calibration_id
interaction_configuration_id

Conceptually:

System Architecture

Left → Right

This makes the spatial measurement process traceable.

---

57. Why Traceability Matters

Suppose trajectories collected in two periods appear different.

Possible explanations include:

visitor behaviour changed

but also:

sensor placement changed
calibration changed
interaction zones changed
tracking algorithm changed

Without configuration records, these possibilities become difficult to distinguish.

Research infrastructure should therefore preserve enough technical context to understand how observations were produced.

---

58. Relationship to K03

K03 described:

Raw Sensor Data
↓
Interaction Data

K06 expands the spatial transformation stage.

Instead of:

one sensor
↓
coordinate transform
↓
position

K06 provides:

multiple sensors
↓
individual transforms
↓
shared room coordinates
↓
association
↓
fusion
↓
shared tracking
↓
interaction data

Conceptually:

System Architecture

Left → Right

---

59. Relationship to K04

K04 defines research records such as:

session_id
timestamp
visitor_id
x
y
area

K06 defines where the spatial fields should come from in a multi-sensor system.

x
y

should represent a documented shared coordinate system rather than arbitrary sensor-local coordinates.

This makes trajectory data more interpretable and reusable.

---

60. Relationship to the CSWORKS Architecture

The Knowledge architecture now becomes:

K01
Sensor Infrastructure
        ↓
K06
Calibration + Multi-Sensor Fusion
        ↓
K03
Interaction Data
        ↓
K02
Experience Logic
        ↓
Responsive Experience

In parallel:

K03
Interaction Data
        ↓
K04
Research Logging

and:

K05
Experience Profile
        ↓
K02
Experience Logic

The complete relationship is:

System Architecture

Left → Right

---

61. Design Principles

Several principles emerge from multi-sensor spatial architecture.

Transform before fusion

Observations should share a coordinate system before spatial association.

Define one room coordinate contract

All participating systems should agree on origin, axes, and units.

Treat calibration as configuration

Avoid hiding calibration values inside interaction logic.

Version calibration

Research data should remain associated with the spatial configuration that produced it.

Fuse according to need

Presence detection does not require the same fusion complexity as precise multi-person tracking.

Separate sensor observations from fused observations

Diagnostic data and research trajectories serve different purposes.

Plan for occlusion

Multiple sensors improve observability but do not guarantee perfect tracking.

Validate across the usable space

One calibration point is not sufficient.

Preserve physical coordinates

Physical room coordinates support interpretation and research analysis.

Design for degradation

A sensor failure should have an explicit system response.

---

62. Limitations

Multi-sensor fusion introduces additional complexity.

Challenges may include:

• calibration error
• sensor synchronization
• overlapping detections
• track identity switching
• network delay
• different sampling rates
• heterogeneous sensor precision
• occlusion
• sensor failure
• physical movement after calibration

More sensors do not automatically produce better tracking.

Poorly calibrated sensors can introduce conflicting observations and potentially reduce stability.

The value of additional sensors therefore depends on placement, calibration, fusion strategy, and the spatial requirements of the experience.

---

63. Research Direction

Multi-sensor responsive environments raise several research questions.

Examples include:

• How much calibration precision is required for different forms of spatial interaction?
• How does sensor placement influence observable visitor behaviour?
• Which fusion level provides sufficient accuracy without unnecessary complexity?
• How should tracking uncertainty be represented in research datasets?
• How robust are spatial metrics across recalibration?
• How can heterogeneous sensors contribute to one shared spatial model?
• How should multi-user identity continuity be maintained during occlusion?
• How can sensor configuration changes be accounted for in longitudinal studies?

A broader question is:

How can distributed spatial sensors produce a stable and research-traceable representation of human movement within responsive environments?

This connects sensor engineering with spatial interaction and HCI research methodology.

---

64. Key Takeaways

Multiple sensors do not automatically create one spatial model.

Each sensor begins with its own:

origin
orientation
coordinate system
observations

A reusable multi-sensor pipeline is:

Sensor Measurement → Local Filtering → Coordinate Transformation → Shared Room Coordinates → Spatial Association → Fusion → Tracking → Interaction Data

The shared room coordinate system is the central contract.

Calibration defines how each sensor enters that space.

Fusion determines how overlapping observations become shared estimates.

Tracking provides temporal continuity.

Interaction processing converts those estimates into experience meaning.

Research logging preserves the resulting spatial observations together with the configuration that produced them.

The broader principle is:

Establish a shared physical coordinate system first, then fuse observations into interaction meaning rather than allowing each sensor to define its own version of the environment.