Perception and brains

A creature acts on what it has perceived, chooses what to do with cheap and legible reasoning, carries the choice out in small reusable steps, and costs nothing when nobody is near. Content declares brains, state trees, memory kinds and profiles in RON; the runtime gives the same decisions, bit for bit, on native and browser servers. The design and its rationale are in Talos D (docs/superpowers/specs/2026-09-27-talos-d-perception-brains-design.md); this page is the current contract for what phases D0 and D1 built (the “hunt and fetch” slice: the boar’s archetype brain and the pet’s state tree), what D2 added (utility selection: the full input catalogue, impulses, behaviour tags, styles, utility child selection and the thrash ledger), what D3 added (awareness and hearing: the full awareness profile, noise, ally calls, second-hand knowledge and sight’s light factor; see Hearing) and what D4 added (the Reduced tier, the execution divisor, the degradation ladder, scheduled wakes and signals; see Level of detail).

Source: manifold-ai (crates/manifold-ai/src/) (wasm-safe logic crate: api always; the brain runtime behind brains; the registries and components behind the default runtime feature) and the server wiring in manifold_server_state::ai (crates/manifold-server-state/src/ai/). Tests: crates/manifold-ai/tests/ (brains.rs, selection.rs and hearing.rs against a scripted world, data.rs, determinism_guard.rs, perf.rs), the ai::tests, ai::selection_tests and ai::hearing_tests modules of manifold-server-state and game/flagship/tests/creature_content.rs.

Data

Seven server-side registries in the data stage. Brains compile when the stage resolves them, so every load error names its file and line.

// data/brains/creatures.ron (the flagship's boar, abridged)
Brain(
    id: "flagship-game:boar",
    parent: Some("flagship-game:template_predator"),
    params: { "sight_radius": Float(20.0), "charge_ability": Ability("flagship-game:boar_charge") },
    presentation_activities: ["idle", "graze", "alert", "charge", "flee"],
    kind: Some(Archetype(evaluate_every: 10, behaviours: [
        (name: "charge", priority: 1, controls: [Move, Look, Attack], requires: [Reachable],
         presents: Some("charge"),
         behaviour: ChargeAttack(ability: "$charge_ability", min_range: 4.0, max_range: 12.0, recovery: 1.2)),
        (name: "glare", priority: 3, controls: [Look], presents: Some("alert"),
         behaviour: LookAtNoticed(min_state: Engaged, max_distance: 32.0, glance: (2.0, 4.0))),
        (name: "graze", priority: 5, controls: [Move, Look], presents: Some("graze"),
         behaviour: Graze(duration: (3.0, 8.0), chance: 0.1)),
        (name: "wander", priority: 6, controls: [Move],
         behaviour: Wander(radius: 12.0, around: Home, pause: (2.0, 6.0))),
    ])),
)

Inheritance. A derive step merges each brain over its parent, root first: params merge by key; senses (by variant), memories, and an archetype’s behaviours (by name) or a full brain’s activities (by id) merge as keyed lists, where a record patches the parent’s field by field, "-<key>" removes and new keys append; other fields replace. abstract is not inherited. "$name" substitutes a typed parameter.

Validation (§4.5): undeclared or mistyped memory references, states no selection or transition reaches, transitions to missing states and Next on a last child, curves leaving [0, 1], target inputs on target-less activities, sensor intervals outside 8–400 ticks, candidate and consideration caps, duplicate behaviour priorities, presents names outside presentation_activities, awareness thresholds that do not increase, inheritance cycles and missing parents.

Components

manifold-ai:brain brings Brain, Knowledge, Memories, AgentPath, AgentEye and A’s Targets. AiPresentation { awareness, activity, look_yaw } is what binding tables read: awareness is the AwarenessState ordinal (0–3), activity the index into the brain’s presentation_activities, and look_yaw the head yaw offset C’s controller writes (manifold_anim::binding::HEAD_YAW_SOURCE, which agents’ replicated views read). Knowledge, memories, scores and traces never leave the server.

Carrying. The server’s AiCarry resource is the authority for what an agent holds (PickUp to PutDown); the carried drop’s position follows the carrier each tick. At the end of AiThinkSystem the carry is published for clients: the carrier’s Carrying { item, model, socket } names the drop’s item kind, the drop’s own manifold-model:model and the socket (carry::CARRY_SOCKET, the pet’s mouth), and the drop’s Carried { carried } is set. Neither names the other entity (entity references in replicated components are not built, Talos F §8.1). Each is inserted the first time it is needed and afterwards only cleared, because the generic replication path sends no removals; neither is saved. Clients draw the item at the carrier’s socket and hide the drop on the ground (manifold_client_state::drops, see gameplay).

The AI block

The server registers the block in engine:tick after ResolveActionHandlesSystem and before C’s navigation and AnimBind (§3.4). The systems share one access set and run in order; agents are processed in (world space, durable id) order.

Perception. Sight tests distance (with lose_radius for a target seen within lose_grace), the cone for the observer’s awareness and the relation from B’s factions, then reads the line-of-sight cache (8 ticks, 16 entries per observer) or queues a ray. A sensor with uses_light: true scales its gain by the light at the target’s eye (0–15, as a fraction, through its light curve; by default a quarter of the gain in darkness, three quarters at half light): the brighter of the cell’s block light and its sky light times daylight (full from 0.3 to 0.7 of the day, a tenth at night), read from the server’s cached light maps. Where no light map is loaded the target counts as fully lit. Rays walk manifold_world::walk_cells from eye to eye, accumulating each cell’s sight occlusion (its light attenuation, where at most 1 occludes nothing) until 15. Eye heights come from the rig’s rest eyes socket, else head. Damage from a known attacker engages at once and writes manifold:last_hurt_by. The Owner sensor reports the OwnedBy target (omniscient by design); the Items sensor writes the nearest acceptable drop. Nothing downstream reads unperceived truth: a known target not perceived in the last 20 ticks resolves to its last-known position.

Knowledge holds per-target entries sorted by durable key with a Q16 meter, promotion after a jittered reaction delay (from DerivedRandom; a higher threshold crossed while a lower promotion waits replaces it, so a fast climb goes straight to the highest state), hold then decay, one-level demotion, falling confidence and C’s give-up (unreachable_until and the nav epoch it was judged at, re-asked when the epoch advances). Memories are one typed slot per declared kind with lazy expiry; every change sets a dirty bit that selection wake masks, transition events and behaviour interrupts test.

Archetype brains evaluate every evaluate_every ticks or on an interrupt bit: running behaviours whose keep or requirement fails stop (losing Reachable records a give-up), then behaviours start in priority order when their flags are free or held by lower priorities, which are interrupted. Built-ins: Wander, Graze, LookAtNoticed, ChargeAttack (its ability, like UseAbility’s, names a declared ability, checked when the brain compiles in the data stage; close to range with line of sight and a straight lane C’s line_walkable accepts, face, request the ability, recover; with no lane it follows the path towards the target until one opens, and a target out of sight is searched for at its last-known position), FleeWhen, ReturnHome, and D3’s Investigate (walk to a noticed target’s last-known position, re-aiming when a new noise moves it, then look around there; a target engaged in plain sight is left to the combat behaviours) and CallAllies (an ally call about the best target at or above min_state, then a cooldown).

The dead (B’s manifold:state.dead) do not think: they leave the awake set, their MoveIntent is cleared so the body lies where it fell, and dead players and creatures are not perceivable until they are revived.

Full brains select at the heartbeat or on a wake bit: candidates from None, Owner, Memory or knowledge; behaviour-tag locks, the style, invariants, the thrash ledger and the reachability gate; compensated products of curved considerations in Q16, times the activity’s weight and the style’s scale; the top rank tier; keep_ratio; one derived weighted draw; commitment. Selection is described under Selection below. The chosen activity runs a state tree under rules R1–R14 (§8.3), with the task library MoveTo, Follow, Face, Wait, Wander, UseAbility, Claim, Release, PickUp, PutDown, PlayAnimation, Emit(Cue), Emit(Call(tag, radius)) (an ally call carrying the activity’s target), Emit(Noise(tag, loudness)), SetMemory and ClearMemory. Claims belong to the claiming state’s parent and are released when it exits.

Seams. Tasks act through AgentEnv, which the server implements over C’s NavWorld and MoveIntent, B’s stats, tags, cues and factions, the stimulus bus (AgentEnv::emit: what an agent emits joins the next tick’s bus), Pygmalion’s ServerAnim::issue and playing (plays and STOPs), the AiAbilities port, whose default GameplayAbilities queues B1’s AbilityRequests and answers from the caster’s AbilityState and the previous tick’s AbilityEnded and AbilityFailed records (an ability the agent is not granted is Unavailable, and the task skips it), and the AiCoordination port, whose default is E’s reservation table and Home.

Budget. Items (execution, due sensors, due selection) carry work-unit estimates from counts (§12.2) and priorities from covered state only; the plan admits forced items (deferred to their class’s limit), then each class within its floor, then the shared pool by priority. AiBudgetReport holds the accounting. The budget is 6,000 WU native and 4,000 browser.

Level of detail

AiLodSystem evaluates each awake agent every 10 ticks (staggered):

Promotion is immediate; demotion needs its condition (radii 16 blocks larger) for 5 s. A watched agent is never dormant. Off its divisor’s ticks an agent is not planned, but a wake event (any dirty bit, a fired impulse) executes it anyway. The Execution class’s items may wait 2 ticks at Full and 8 at Reduced; Selection’s 4 and 16.

Degradation ladder (§12.6, AiScheduler::degradation). When the plan deferred work in 20 of the last 40 ticks, the level steps down one per 40 ticks; after 80 ticks without deferral it steps back up one. Level 1 doubles Reduced agents’ selection cadence (× 8), level 2 demotes Full agents beyond 32 blocks that are neither in combat nor pinned, level 3 doubles Reduced sensor intervals (× 8), level 4 would skip guest hooks for Reduced agents (there are none yet). Changes are logged and reported (AiBudgetReport::degradation, /ai budget, Framescope’s ai_budget).

Waking. Dormant agents wake when a player comes within 128 blocks, on damage, when their owner is online, at their 10-tick wake check when a stimulus of the last 10 ticks reached their hearing, on a signal, or on a scheduled wake; 64 per tick at most, overdue first, then by cause (damage, owner, approach, stimulus, signal, schedule). A woken agent is Full, senses next tick and selects at once.

• ai::schedule_wake(world, agent, tick) puts the agent on the wake wheel (manifold_ai::lod::WakeWheel, covered state): when due, a dormant agent wakes and an awake one gets the SIGNAL dirty bit.
• ai::send_signal(world, AiSignal { name, target }) (after Unreal Mass’s signals, for E’s directors, encounter volumes and I/O links) targets SignalTarget::Agents(entities) or SignalTarget::Area { space, center, radius }; the next LOD pass delivers it in key order: dormant targets wake, awake ones get the SIGNAL bit (§6.5’s bit 56, in selection’s default wake mask).

Selection

A full brain’s activities are scored (activity, target) pairs (§7). Each consideration maps one input through a response curve; inputs normalise by (v − min) / (max − min), clamped and quantised to 1/256 of the range.

Tags match themselves and their descendants. The target’s own stats and tags are read only while it is perceived now or known outright (the owner, a memory’s entity); a remembered target out of sight reads 0. CooldownReady is 1 when the ability is granted, idle and its cooldown effect is absent (AiAbilityPort::ready). GroupKnows, Threat, TokenAvailable and AdScore (D6), BondStrength (D7) and Custom (D5) parse and are refused at load naming their phase.

Impulses (§7.11) inject an activity outside the heartbeat. An activity’s impulses inject that activity; a full brain’s impulses name theirs with activity: Some(..):

impulses: [(on: Damaged, when: Some(StatBelow("manifold:health", 0.2)), rank: 8)]

on is Damaged, Event([...]) (the change events transitions use) or Stimulus(tag) with an optional within (blocks): a stimulus the brain’s hearing reaches, tagged tag or a descendant, within within, whoever made it (a sourceless noise too). Each distinct (tag, within) pair takes one of the dirty bits 57–63 (at most 7), and a Stimulus impulse needs a Hearing sensor. An impulse fires when its bits were dirty at the step’s start and when holds; it waits in Brain.impulses until selection runs (triggers join the wake mask, so the plan admits it), and that pass raises the activity’s candidates to the impulse’s rank (the trace’s SelectReason::Impulse).

Behaviour tags. A state’s locks: ["ai.activity.reposition"] filter matching activities (Rejection::Lock) while it is active; the running activity is never locked by its own states. Locks compile to activity masks.

Styles are Style(id, allow, deny, scale, grants) records. deny filters matching activities (Rejection::Style); a non-empty allow permits only matching ones; scale multiplies matching activities’ utility (0–16; several matches multiply). Each style is resolved against every brain when the library builds (CompiledBrain.styles, by the style registry’s dense id). On archetype brains a denied behaviour does not start and a running one stops. E sets a style with manifold_server_state::ai::set_style(world, agent, Some("mod:style")) (None clears it); the change wakes selection. grants are carried for E and later phases; D acts on none. A style is not saved.

Utility child selection (§8.11). A state with select: Utility(keep_ratio) scores the children whose enter holds with their utility considerations (no rank; the active child keeps a × 1.25 bonus), drops those below keep_ratio × best, and draws among the rest at the state’s derived site; the pick is tried first, then every other child best to worst.

Thrash ledger (§8.10). A failed run is recorded; three failures of one (activity, target) within 200 ticks block it for 400 (Rejection::Thrash, a Thrash trace record, and AiBudgetReport.thrash, which counts the tick’s blocks). A pair that failed on the step it was entered (its tree selects nothing, or it fails on its first step) is filtered until the next heartbeat.

Hearing

Sound is a stimulus at a position (§5.5). AiStimulusCollectSystem builds the tick’s bus (AiStimuli.bus, manifold_ai::hearing::StimulusBus) in an order that depends only on covered state, from:

A source may put at most 4 stimuli on the bus per tick. Tags match themselves and their descendants.

Hearing is event-driven: an agent’s Hearing sensor reads every tick’s bus while it is awake. A stimulus is audible within loudness × sensitivity blocks of the agent’s eye, with no line of sight. For a noise whose maker passes the sensor’s targets filter, knowledge gains gain × falloff(distance / range) (default gain 0.3, falloff linear to 0) at a position blurred on each horizontal axis by up to blur × distance (default 0.15, a derived random draw); a maker seen now keeps its seen position. Heard entries cap at Alert: engagement needs a sense in the profile’s engage_requires.

senses: [Hearing(sensitivity: 1.0, gain: 0.3, targets: HostileTo, calls: ["call.help"], blur: 0.15)]

Calls (§5.7). An allied caller (the listener’s relation to it is Friendly) whose tag the sensor lists in calls sets the CallHeard bit and gives second-hand knowledge of the caller’s target: an Alert entry (the highest state the profile defines at or below it, at once) at the caller’s last-known position with the caller’s confidence × 0.7 and source: Call. A target the listener perceives now keeps its first-hand entry; any first-hand sense later makes the entry Direct again.

Determinism and traces

Scores, curves and meters are Q16 integers; curve tables are baked through manifold-det; randomness is DerivedRandom at sites hashed from registry keys; nothing iterates a hash map. tests/determinism_guard.rs scans every source. scripts/test-text-wasm.sh runs the headless brain tests under wasm32-wasip1, including a 4,000-tick scene (the boar, the pet and two watchers that hear the player’s footsteps and call each other) whose trace digest is pinned.

DecisionTraces keeps a ring per traced agent (512 records; 64 for a brain declaring trace: FlightRecorder) of TraceFrame { header, records } postcard-encoded from manifold_ai::api::trace; G transports them. G’s tools reach D through one seam, talos_observe::agent_view::AgentDebugView, which ai::DAgentView (crates/manifold-server-state/src/ai/debug_view.rs) implements: /ai trace watches an agent (its ring’s history first, then each flush) on the console and the debug stream, the inspector’s brain panel and the sight, hearing and activity overlays read the agent’s brain, and /ai why answers from brain::why_not (utility brains: selection as a dry run with the agent’s locks, style and pending impulses) or exec::archetype::explain (archetype brains, read-only). See console commands. select::why_not names the stage that rejects an activity.

Limits

• Touch, a second ray to the centre, occlusion for hearing, guest hooks, bonds, smart objects and E’s coordination inputs are later phases (D5–D7). There is no WakeIndex (D issue 27); a signal’s name is not yet readable by conditions (D issue 56).
• Block-edit noise attributes a changed cell to the nearest player within 8 blocks, because the world change log carries no actor (D issue 44); a noise scale for sneaking waits for B (D issue 48).
• Light comes from the server’s cached light maps, which the server does not relight after edits (D issue 46).
• Conditions are D’s own typed set; registering them as B’s Condition::Ext atoms waits for an extension slot in B’s evaluation context (D issue 37).
• The flagship boar keeps its archetype brain: the post-slice gore (Chase with flagship-game:boar_gore) and a utility brain for it are not built.
• PlayAnimation names are checked when played (NoAnimation), not at load.
• Only manifold-ai:brain is saved; persisted memory slots and the activity hint are not.
• Picked-up drops ride at the carrier until PutDown; B’s pickup into an inventory arrives with item kinds. The carry socket is one engine constant (mouth), not per brain or archetype, and a carrier without that socket shows nothing in its mouth.