Commonwealth-Online-Public/docs/animation-architecture-alignment.md

# Animation Architecture Alignment: TiltedEvolution → F4T

## Executive Summary

This document maps TiltedEvolution's proven multiplayer animation and state synchronization architecture onto Fallout 4 Together's codebase. It identifies which concepts are applicable to FO4, what needs adaptation, and FO4-specific constraints.

**Key Finding:** TiltedEvolution's architecture is fundamentally compatible with FO4, but FO4 has stricter constraints around AI, velocity, and action replay mechanics.

---

## High-Level Architecture Comparison

### TiltedEvolution (Skyrim SE) - The Gold Standard

```
Local Player                Remote Server              Other Clients
     |                            |                           |
[Capture] - Save transform, - relay - [Receive] 
 + graph variables               |        + parse JSON
 + actor state flags             |        + update state
 + action events           [Broadcast]
     |                            |
[Send] -------- JSON line ------> |
            ~100ms intervals       |
                                   |
                    [Remote Actor Control]
                          |
                    [InterpolationSystem]
                   - Position lerp
                   - Direction lerp
                   - Graph variable load
                          |
                    [AnimationSystem]
                   - Action replay queue
                   - Actor state apply
                   - Discrete animation trigger
                          |
                   [Result: Smooth movement +
                    playing correct animations]
```

### F4T Current (Pre-Refactor) - SetPosition Only

```
Local Player                Remote Server              Other Clients
     |                            |                           |
[Capture] - movement state  relay [Receive]
 + position              |        + parse JSON
 + flags (move/sprint)         [Broadcast]        + update RemotePlayerState
 + NO actor state        |
 + NO actions                    |
     |                    [Proxy Actor Control]
[Send] -------- JSON ---------> |
                                  |
                         [ProxyActorController]
                        - GetOrCreateSlot()
                        - ApplyRuntimeProxyTransform()
                           |
                           v
                    [SetPosition() only]
                    [+ Move(0, {0,0,0}, false)]
                    [+ SetLinearVelocityImpl()]
                           |
                    [Result: Smooth movement
                    but NO ANIMATIONS
                    (velocity doesn't trigger anim system)]
```

---

## Component Mapping

### 1. Capture & Serialization

| Concept | TiltedEvolution | F4T Current | F4T After Phase 2 |
|---------|-----------------|-------------|-------------------|
| **Local state capture** | `CharacterService::RunLocalUpdates()` | `main.cpp:PollLocalPlayerTransform()` | Same + add `PollLocalPlayerActionState()` |
| **What's captured** | Position, rotation, direction, **animation variables snapshot**, **actor state flags**, **action events** | Position, rotation, direction, movement flags, speed | Same as Tilted (extended) |
| **Capture interval** | 100 ms | ~50-100 ms | ~100 ms |
| **Storage** | `Movement { position, Variables, Direction, ... }` | `RemotePlayerState { x, y, z, angleZ, movementType, cellId, ... }` | Extend with `actorStateFlags1/2`, `ActionEvent[]` |

**Action:** Phase 2.3 adds action capture to `main.cpp`; Phase 2.1 extends struct.

### 2. Network Transmission

| Concept | TiltedEvolution | F4T Current | F4T After Phase 2 |
|---------|-----------------|-------------|-------------------|
| **Packet type** | `ClientReferencesMoveRequest` (C++ struct) | JSON: `{"type":"transform", "x",...}` | Extend JSON with `"actorStateFlags1"`, `"actionEvents":[...]` |
| **Serialization** | Binary (optimized for network) | JSON (text over TCP) | JSON (text over TCP, extended) |
| **Frequency** | ~10 Hz (adaptive) | ~10 Hz (throttled) | ~10 Hz (throttled) |
| **Thread** | Game thread sends | Game thread sends | Game thread sends |

**Action:** Phase 2.2 updates JSON parsing in `F4TNetworking.cpp`.

### 3. Remote State Storage

| Concept | TiltedEvolution | F4T Current | F4T After Phase 2 |
|---------|-----------------|-------------|-------------------|
| **Store** | `std::unordered_map<uint32 playerId, Character>` in CharacterService | `g_remotePlayers` map + mutex | Same + extend struct |
| **Thread** | Network thread writes, game thread reads (snapshot copy) | Network thread writes, game thread reads (snapshot copy) | Same (thread-safe boundary maintained) |
| **Per-player data** | Multiple Character instances (one per remote) | One RemotePlayerState per remote | Extend with flags + action queue |

**Action:** Phase 2.1 modifies struct; thread safety unchanged.

### 4. Proxy Actor Spawning & Lifecycle

| Concept | TiltedEvolution | F4T Current | F4T After Phase 4 |
|---------|-----------------|-------------|-------------------|
| **Spawn method** | `Actor::PlaceAtMe()` on demand | Pre-placed pool in CK | Dynamic `PlaceAtMe()` (like Tilted) |
| **Lifecycle** | Create on connect, destroy on disconnect | Pool reuse; hold out of range when inactive | Pool reuse (same as current) |
| **Count limit** | ~256 (depends on memory) | 4 (pre-placed limit) | 4-8 (dynamic pool size) |
| **AI** | Suppressed (`SetRemote(true)`) | Suppressed (neutralize) | Suppressed (neutralize) |
| **Puppet mode** | Yes (movement external) | Yes (external via SetPosition) | Yes (external via SetPosition + velocity) |

**Action:** Phase 4.1 replaces pool init with dynamic spawn logic.

### 5. Movement & Interpolation

| Concept | TiltedEvolution | F4T Current | F4T After Phase 4 |
|---------|-----------------|-------------|-------------------|
| **Position sync** | `ForcePosition()` (Skyrim API) | `SetPosition()` (FO4 API) | Same (SetPosition works) |
| **Rotation sync** | Lerp + direct set | `SetHeading()` (snap) | `SetHeading()` + lerp (Phase 4.3 later) |
| **Velocity** | Natural from AI controller + character controller velocity | Manual injection via `Move()` + `SetLinearVelocityImpl()` | Same (proven approach) |
| **Interpolation** | `InterpolationSystem` lerps transform + graph vars | Lerp position, snap rotation | Enhance rotation lerp (Phase 7+) |

**Action:** Phase 4.2 verifies velocity injection works on dynamic proxies.

### 6. Animation Variable Sync

| Concept | TiltedEvolution | F4T Current | F4T After Phase 5 |
|---------|-----------------|-------------|-------------------|
| **Method** | Descriptor-based bulk write (by index) | String-based per-variable write | Descriptor-based bulk write (Phase 3 + 5) |
| **Variables** | `Speed`, `Direction`, `IsSprinting`, `IsSneaking`, + others | `Speed` (string fallback), `IsSprinting` | Same minimal set + indexed |
| **Frequency** | Once per action + continuous during interpolation | Every frame | Every frame (indexed, faster) |
| **Thread safety** | `BSScopedLock` on graph manager | No explicit locking | Add locking (Phase 5) |
| **Variable persistence** | Snapshot approach (graph state saved/loaded) | Continuous writes | Hybrid: snapshot for actions, continuous for locomotion |

**Action:** Phase 3 builds descriptor class; Phase 5 refactors sync to use it.

### 7. Action Replay System

| Concept | TiltedEvolution | F4T Current | F4T After Phase 5 |
|---------|-----------------|-------------|-------------------|
| **Capture** | Hook `ActorMediator::PerformAction()` → queue `ActionEvent` | **Not implemented** | Add `PerformAction` hook (Phase 2.3) |
| **Storage** | `RemoteAnimationComponent::TimePoints` (event queue per remote) | **No action queue** | `RemoteActionComponent` per proxy (Phase 5.1) |
| **Replay** | `AnimationSystem::Update()` pops + replays actions in order | **No action replay** | `ApplyRemoteAction()` call in slot update (Phase 5.2) |
| **Content** | `{ Type, EventName, Variables, State1, State2, IdleForm, Target, ... }` | **N/A** | Minimal set: `{ Type, EventName, Variables, State1, State2 }` |
| **Effect** | Discrete animations (combat idles, emotes, equipping) play deterministically | **Proxies stand still** | Combat idles + transitions animate |

**Action:** Phases 2.3, 5.1, 5.2 implement action pipeline.

### 8. Actor State Flags

| Concept | TiltedEvolution | F4T Current | F4T After Phase 2 |
|---------|-----------------|-------------|-------------------|
| **Capture** | `actor→actorState.flags1/flags2` on local player | **Not captured** | Add to state capture (Phase 2.3) |
| **Network** | Packed in `ActionEvent.State1/State2` | **No** | Add to transform packet JSON |
| **Apply** | Before action replay: `actor→actorState.flags1 = remote.State1` | **No** | Apply before animation sync (Phase 4.3) |
| **Purpose** | Signal animation graph FSM (combat mode, sneaking, etc.) | **No effect** | Enable graph-driven state transitions |

**Action:** Phase 2.1 adds fields; Phase 2.3 captures; Phase 4.3 applies.

---

## Data Flow Diagrams

### TiltedEvolution Animation Sync (Reference)

```
PerformAction hook (local)
    ↓
Capture: {Type, EventName, Variables, State1, State2, Idle}
    ↓
Network: Send ActionEvent + Movement (animation vars)
    ↓
Remote receives (network thread)
    ↓
AnimationSystem::Update() (game thread)
    ↓
For each action in queue:
  - actor→actorState.flags1/2 = State1/State2
  - actor→LoadAnimationVariables(Variables)
  - ActorMediator::ForceAction(ActionData)
    ↓
Result: Correct animation plays
```

### F4T Current Animation Sync (Broken)

```
PollLocalPlayerTransform (local)
    ↓
Capture: {position, angle, moving, sprinting}
    ↓
Network: Send transform (JSON)
    ↓
Remote receives (network thread)
    ↓
ProxyActorController::Update() (game thread)
    ↓
ApplyRuntimeProxyTransform()
  - SetPosition()
  - Move(0, {0,0,0}, false) → get controller
  - SetLinearVelocityImpl(calc_velocity)
    ↓
ApplyProxyAnimationFromRemoteState()
  - TrySetGraphFloat("Speed", value)  ← writes by name, slow
  - TrySetGraphBool("IsSprinting", value)
    ↓
Result: Movement works, but animations NEVER play
         (velocity is set, but graph doesn't react)
```

### F4T After Phase 5 (Proposed)

```
Local Player
  ├─ PollLocalPlayerTransform()
  │  └─ Capture: {position, angle, velocity, movementFlags}
  │
  ├─ PollLocalPlayerActionState() [NEW - Phase 2.3]
  │  └─ Capture: {actor→actorState, PerformAction events}
  │
  └─ SendTransformPacket() [EXTENDED - Phase 2.2]
     └─ JSON: {position, angle, actorStateFlags1/2, actionEvents}

Network Thread
  └─ HandleTransformPacket()
     └─ UpdateRemotePlayer() [EXTENDED - Phase 2.1]
        └─ RemotePlayerState: {position, flags, action queue}

Game Thread (ProxyActorController::Update)
  ├─ GetRemotePlayerSnapshot()
  │
  ├─ For each remote:
  │  ├─ GetOrCreateSlot()
  │  ├─ RestoreSlotProxyForRemotePlayer() [First frame]
  │  │  └─ Snap transform
  │  │
  │  └─ UpdateRuntimeSlotForRemotePlayer() [Ongoing]
  │     ├─ ApplyRuntimeProxyTransform() [Phase 4]
  │     │  ├─ SetPosition()
  │     │  ├─ Move(0, {0,0,0}, false)
  │     │  └─ SetLinearVelocityImpl()
  │     │
  │     ├─ ApplyRemoteActionState() [NEW - Phase 4.3]
  │     │  └─ actor→actorState.flags1/2 = remote.flags
  │     │
  │     ├─ ReplayRemoteActions() [NEW - Phase 5.2]
  │     │  └─ For each action:
  │     │     ├─ actor→LoadAnimationVariablesFromDescriptor()
  │     │     └─ ForceProxyAction()
  │     │
  │     └─ ApplyProxyAnimationFromRemoteState() [REFACTORED - Phase 5.3]
  │        └─ LoadAnimationVariablesFromDescriptor() [indexed, bulk]

Result
  ├─ ✅ Smooth movement
  ├─ ✅ Character controller velocity set
  ├─ ✅ Locomotion animations (idle/walk/run/sprint)
  ├─ ✅ Action animations (combat, equips)
  ├─ ✅ Actor state affects graph FSM
  └─ ✅ Performance: O(1) indexed access, not O(n) string lookup
```

---

## FO4-Specific Adaptations & Constraints

### 1. Character Controller Velocity

| Aspect | Skyrim SE | FO4 | F4T Approach |
|--------|-----------|-----|------------|
| **API exposure** | High (character controller accessible) | Lower (not directly exposed) | Use `Actor::Move()` return value to get controller |
| **Velocity injection** | Direct via API | Via Move() + SetLinearVelocityImpl() | Same as current proven approach |
| **Animation system watching velocity** | Yes | Yes (verified in dev-log) | Confirmed working; use same method |

**Note:** F4T already verified this works (see dev-log 2026-06-03). Use same velocity injection.

### 2. Actor State Flags

| Aspect | Skyrim SE | FO4 | F4T Approach |
|--------|-----------|-----|------------|
| **Flags struct** | `ActorState` | `ActorState` | Same struct + same flags |
| **Graph dependency** | Yes (FSM gated by flags) | Yes (FSM gated by flags) | Apply before animation sync |
| **Combat flag** | `inCombat` | `inCombat` | Same |
| **Sneak flag** | `sneaking` | `sneaking` | Same |

**Note:** F4T tried `SetSneaking()` API directly (disabled in code). Better approach: sync `actor→actorState.flags1/2` bitfields. More reliable than API calls.

### 3. Action Replay

| Aspect | Skyrim SE | FO4 | F4T Approach |
|--------|-----------|-----|------------|
| **API** | `ActorMediator::ForceAction()` | `ActorMediator::ForceAction()` | Same |
| **Availability** | Well-exposed in common headers | Exposed in CommonLibF4 | Use same |
| **Idle forms** | Rich idle system with conditions | Rich idle system with conditions | Capture + replay |
| **Combat actions** | Power attacks, spells, shouts | Power attacks, guns, grenades | Defer to Phase 2+ |

**Note:** CommonLibF4 likely has equivalent `ActorMediator` and `ForceAction()` (verify in Phase 1.2 testing).

### 4. Animation Graph Manager

| Aspect | Skyrim SE | FO4 | F4T Approach |
|--------|-----------|-----|------------|
| **Manager lifecycle** | Stable per actor | Stable per actor | Same assumptions |
| **Variable cache** | Accessible via `GetBSAnimationGraph()` | Accessible via `GetAnimationGraphManagerImpl()` | Use CommonLibF4 API (current code works) |
| **Thread safety** | Requires `BSScopedLock` | Requires `BSScopedLock` | Add explicit locking (Phase 5) |
| **Descriptor key** | Hash of graph type name | Hash of graph type name | Build descriptor once, cache |

**Note:** F4T already uses the CommonLibF4 API successfully. No changes needed, just add locking.

### 5. Proxy Puppet Suppression

| Aspect | Skyrim SE | FO4 | F4T Approach |
|--------|-----------|-----|------------|
| **AI blocking** | `SetRemote(true)` | No direct equivalent | Current: `NeutralizeProxyActor()` + flags (kept for Phase 4) |
| **Movement blocking** | External `ForcePosition()` | External `SetPosition()` | Same |
| **Action suppression** | Hook `PerformAction` return 0 | Need to verify hook exists | Assume possible; test in Phase 1 |

**Note:** F4T currently suppresses AI via actor flags/hostile reset. Tilted uses a `Remote` flag. Both approaches work; F4T's is compatible.

---

## Phase-by-Phase Integration Points

### Phase 1: Research
- Verify FO4 variable indices match assumptions
- Confirm `ActorMediator::ForceAction()` exists in CommonLibF4
- Document any differences from Skyrim model

### Phase 2: Protocol & State
- Extend JSON packets with `actorStateFlags1/2`, `actionEvents`
- Add action capture in `PerformAction` hook (or equivalent)
- Maintain backward compatibility

### Phase 3: Descriptors
- Build `F4AnimationDescriptor` using discovered indices
- Implement bulk read/write methods
- Verify thread safety with `BSScopedLock`

### Phase 4: Spawn & Velocity
- Switch to dynamic spawn (proof of concept: single proxy first)
- Verify velocity injection works on dynamic actors
- Apply actor state flags before animation sync

### Phase 5: Action Replay
- Implement `RemoteActionComponent` action queue
- Refactor animation sync to use descriptors
- Ensure action replay doesn't conflict with continuous locomotion

### Phase 6-7: Testing & Polish
- Verify animations play correctly
- Performance benchmark descriptor vs string-based
- Document FO4-specific adaptations

---

## Success Criteria for Phase 1

- [ ] Document all float/bool/int variables for humanoid graph (variable names, likely indices)
- [ ] Confirm TiltedEvolution concepts apply to FO4 (actor state, action replay, descriptors)
- [ ] Verify `Actor::Move()` + `SetLinearVelocityImpl()` works on dynamic proxies
- [ ] No major architectural blockers identified
- [ ] Produce `animation-architecture-alignment.md` (this document)

---

## References

### TiltedEvolution Source

- [AnimationSystem.cpp](https://github.com/tiltedphoques/TiltedEvolution/blob/dev/Code/client/Systems/AnimationSystem.cpp) - Action replay
- [CharacterService.cpp](https://github.com/tiltedphoques/TiltedEvolution/blob/dev/Code/client/Services/Generic/CharacterService.cpp) - State capture
- [AnimationGraphDescriptor_Master_Behavior.cpp](https://github.com/tiltedphoques/TiltedEvolution/blob/dev/Code/encoding/Structs/Skyrim/AnimationGraphDescriptor_Master_Behavior.cpp) - Variable indices

### F4T Source

- `plugin/src/main.cpp` - Local transform capture
- `plugin/src/F4TNetworking.cpp` - JSON serialization
- `plugin/src/F4TProxyAnimationSync.cpp` - Current animation sync (to be refactored)
- `plugin/src/F4TProxyActorController.cpp` - Proxy lifecycle + movement

### CommonLibF4 Headers

- `RE/H/ActorState.h` - Actor state flags
- `RE/H/BSAnimationGraphManager.h` - Graph manager
- `RE/H/IAnimationGraphManagerHolder.h` - Graph holder interface
- `RE/H/Actor.h` - Actor class (Move, SetPosition, etc.)