Skip to content

Specification v0.0.1

Echo-D is a library for distributing Entity-Component state in real-time.

Echo-D is network transport and storage agnostic. For example WebRTC and it can integrate into existing ECS libraries or databases.

Echo-D is designed for cloud-based game engines. It is ideal for session-based multiplayer games with server-side physics or rendering.

This document describes the Echo-D protocol, as well as the main Echo-D APIs and data structures.


Table of contents


Options

Related: Context, Action Handler, and Actions

Represents the options for a Echo-D node. It provides a way to configure various settings and behaviors for the node.

There are many possible configurations for an Echo-D node. All of the parameters are encapsulated by a Options. There should be a static extend method to make it easy to create a new instance of an existing set of options, while applying new custom options.

Main functionalities

  • Allows configuring various options for a node
  • Provides default values for the options
  • Ensures that the provided options are an instance of Options
  • Clones and extends the options object

Option Definitions

actions : Object

Related: Action Handler, and Actions

A map of actions that is used by an Action Handler in order to repond to messages that are recieved. They can be overriden in this reference.

{
exampleAction: (payload, context, options) => { /* ... */ },
// ...others...
}

batchActionPayloadSizes : Object

Related: Batch Message Structure

In order to serialize and deserialize batched messages, the size of the message payload needs to be known. However there is a necessary complexity that requires there be a way to indicate a different size if certain flags in options are true:

{
actorInput: { default: 1, rollback: 2 }, // if isRollback 2, otherwise 1
changeComponent: { default: 3, ordered: 4 }, // if isOrdered 4, else 3
mergeSymbols: 2, // always 2
removeComponent: 2,
upsertComponent: { default: 3, ordered: 4 }
}

compressStringsAsInts : Boolean

Related: Symbols, Numeric Reference Table, and Updater

The protocol uses Symbols which are strings that have been mapped to a unique integer. This allows them to be referenced by the associated integer from an index and compresses the size of strings being sent and recieved.

defaultSymbols : Array[String]

Related: Numeric Reference Table

An array that maps the default set of symbols by number. The values of the array are strings that represent symbols and the indexes of the values are the unique number IDs for the symbol.

[ 'symbol1', 'symbol2' ]

enumDefaultSymbols : Object

Related: Numeric Reference Table

An object that maps the default symbols by string. The keys of the object are strings that represent symbols and the values are the unique number IDs for each key (symbol).

{ symbol1: 0, symbol2: 1 }

enableQuerying : Boolean

Related: Queries, Indexes, and Storage

A flag to indicate if querying is enabled. When enabled the payload of actors, entities, and components can be a query object that will filter out unwanted items.

enableRollback : Boolean

Related: Ticked Actions

A flag to indicate if time rollback is enabled on inputs. This expects an extra tick value in the payload of actorInput. The tick can then be used to implment a rollback strategy on the client nodes.

getActorId : Function

A function to get the actor ID. It is passed an ID string, the default behavior is to return it.

getGroupedValue : Function

Related: Grouped Component Updates

A function to get the value from a group. By default it slices a section of an array that matches the defined type size. However it might be necessary for some Storage adapters to convert the values.

// Example
(value, iAt, types, key) => { /* return value[i] */ }

indexes : Object

Related: Indexes, and Storage

An object containing a mapping indexes. You can specify either sorted or spatial indexes. A sorted index will organize values into a sorted array for indexing. A spatial index will use a spatial hashing function to index nearby elements and allow effecient nearest neighbor queries.

{
collider: 'sorted',
hidden: 'sorted',
color: 'spatial',
position: 'spatial',
}

isAuthority : Boolean

Related: Action Handler

A flag to indicate if the node is an authority. An authorative node can is authorized to send updates.

isAsyncStorage : Boolean

Related: Storage, and Async

A flag to indicate if the node uses async storage. This is useful if the Storage adapter is custom and uses an asynchronous API to read and write data. Note that actions that write to the store do not wait for the promise to finish, therefore they do not handle errors.

isComponentRelay : Boolean

Related: Action Handler, and Updater

A flag to indicate if the node is a component relay. Disabling this means component updates will not be sent.

isDiffed : Boolean

Related: Changes

A flag to indicate if the node is diffed. This uses Changes in order to track updates as differences. For example setting a position component to [0, 0, 0] that was previously set to [1, 1, 1] will result in [-1, -1, -1] being sent as an update to changeComponent instead of upsertComponent which will apply the change elsewhere. This allows changes to happen in any order without changing the final value.

isGroupedComponents : Boolean

Related: Grouped Component Updates

A flag to indicate if the node uses grouped components. This allows for structure of arrays (SoA) to be used to speed up grouped component updates.

isOrdered : Boolean

Related: Ordered

A flag to indicate if the node is ticked. This adds a tick value to the end of the payloads for changeComponent and upsertComponent. The tick is used to filter outdated values from being applied as an update.

It is possible to enable isDiffed when this is also enabled, in order to differentiate from other node configurations the tick value will be inverted in updates. It is not recommended to use both these settings at once.

isReadOnly : Boolean

Related: Updater and Pending

A flag to indicate if the node is read only. A node that is read-only will not be capable of sending updates to other nodes.

isSymbolLeader : Boolean

Related: Symbols

A flag to indicate if the node is a symbol leader. Only a symbol leader can declare symbols in the network. There should ideally only be one source of truth. This prevents inconsistencies. In the future a CRDT G-Set maybe used instead, but that would add complexity.

isSymbolRelay : Boolean

Related: Symbols

A flag to indicate if the node is a symbol relay. If disabled then the node will not forward symbol changes to other nodes. A symbol relay sends symbols that are recieved from a leader to other nodes.

onUpdate : Function

Related: Action Handler

A function to call when the node is updated. It will be call once per update, it is recommended that this function be manually throttled if it is being used in the client.

pageSize : Number

Related: Storage

The page size for list actions: actors, components, entities, symbols. This keeps very large lists from being sent by one responder call. The pages are emitted until all items are sent by the responder in multiple calls.

responder : Function

Related: Updater

A function that is called by the Updater to send update actions to other nodes in the network. This allows for any network transport, inter-process communication, or a Pub/Sub library to be used for responses.

skipPending : Boolean

Related: Pending

A flag to skip pending actions. This is useful when component updates need to happen without causing a pending change that the Updater will then use to update the network.

types : Object

Related: Component Value Types, isGroupedComponents, and Storage

An object that defines the component types. A component type can be a class, or a string that maps to a valid type. If an array is provided as the component type, then the first value will be used as type, and the remaining values will provide meta information. The second value will be the size of each section in a group.

{
collider: String,
hidden: Boolean,
color: ['ui8', 4], // [ Type, size ]
position: ['f32', 3],
rotation: ['f32', 3],
velocity: ['f32', 3],
spin: ['f32', 3],
}

setGroupedValue : Function

Related: Grouped Component Updates

A function to set the value in a group. By default this returns the value from the component value that is passed. If the original structure is an object then it may be necessary to map values to an array here.

// Example
(value, types, key) => { /* return [value.x, value.y, value.z] */ }

updateOptions : Object

Related: Batch Message Structure

An object containing options for the updater. See below for more details on each update option:

{
mask: null,
type: true,
batched: true,
batchSize: 100,
validKeys: null,
}
mask : Object

Specify which kind of actions should be ignored by the updater.

{
actors: false, // if true, prevent updates for `actors`
entities: false, inputs: false,
components: false, symbols: false
}
type : Boolean|String

Secondary type argument for responder.

batched : Boolean

Indicate whether or not to batch update messages together

batchSize : Number

Target number of message to send with each batch

validKeys : Object

Selects which component keys are valid for sending updates

{
collider: true, // if true, allow `collider` component
color: true, rotation: true,
hidden: true, velocity: true,
position: true, spin: true
}

storageOptions : Object

Related: Storage

An object containing the storage options, this is added to make it easier to implement custom Storage adapters that work with other ECS libraries.


Component Value Types

Related: types

Echo-D uses a types that are based on common JavaScript/TypeScript object types. Implementations must have access to basic JSON like data structures such as String, Number, and Array.

A list of valid types for use in types:

  • ‘str’ : String — UTF-8 string
  • ‘bool’ : Boolean — true or false
  • ‘num’ : Number — 64 bit float
  • ‘map’ : Map — key-value pairs of objects
  • ‘set’ : Set — unique sequence of objects
  • ‘arr’ : Array — list of objects

Typed Arrays

In order to better support web based platforms Echo-D only uses arrays of non-Number numbers. These are used to group component updates into structure of array (SoA) buffers.

A list of valid typed arrays for effiently storing vectors, and matricies:

  • ‘i8’ : Int8Array — array of 8 bit integers
  • ‘ui8’ : Uint8Array — array of 8 bit unsigned integers
  • ‘ui8c’ : Uint8ClampedArray — array of 8 bit unsigned integers clamped to (0-255)
  • ‘i16’ : Int16Array — array of 16 bit integers
  • ‘ui16’ : Uint16Array — array of 16 bit unsigned integers
  • ‘i32’ : Int32Array — array of 32 bit integers
  • ‘ui32’ : Uint32Array — array of 32 bit unsigned integers
  • ‘f32’ : Float32Array — array of 32 bit floats
  • ‘f64’ : Float64Array — array of 64 bit floats

Context

Related: Options, and Action Handler

Passes a context around to different objects that is used internally to handle logic between actions. Such as Storage, and the Updater.

Main functionalities

  • Managing actors, entities, components, inputs, and symbols.
  • Creating and removing actors, entities, and components.
  • Changing and upserting components.
  • Handling actor inputs.
  • Managing symbols.

Symbols

Related: compressStringsAsInts, isSymbolLeader, and isSymbolRelay

Represents a collection of symbols and provides methods to manipulate and access these symbols.

Echo-D manages a list of symbols that are created from strings. Wherever a string is used in an action message, except in symbol actions, it is replaced by a unique number.

This mapping between strings, and numbers is considered a Symbol.

This significantly reduces the number of strings that will be sent over the network. This behavior is controlled by the compressStringsAsInts flag.

Main functionalities

  • Adding symbols to the collection
  • Finding the index of a symbol
  • Getting the symbol at a specific index
  • Getting the list of symbols
  • Getting the enum of symbols
  • Resetting the collection with a new array of symbols

Message Structure

Related: One Message Handler, and Actions

Echo-D sends and recieves messages as either a tuple or an object. Each with two values, first the action, then the payload which is optional.

// Message Array
[ 'upsertComponent',
[ 'entity', 'position', [0, 0, 0] ] ]
// Message Object
{ action: 'upsertComponent',
payload: ['entity', 'position', [0, 0, 0] ] }

Batch Message Structure

Related: batchActionPayloadSizes, updateOptions, batch, and Many Message Handler

Messages can be batched in Echo-D, the given action is being called multiple times but it is only provided once first, then the remaining elements of the payload which is a flat array contains all batched payloads for the provided action.

// Message Array
[ [ 'spawnEntity', 'entity1', 'entity2' ],
[ 'upsertComponent',
'entity1', 'position', [0, 0, 0],
'entity2', 'position', [1, 1, 1] ] ]
// Message Object
{ action: 'upsertComponent',
payload: [ 'entity1', 'position', [0, 0, 0 ],
'entity2', 'position', [1, 1, 1] ] }

Objects can also be used in array-object hybrid messages.

// Message Object Array
[ { action: 'spawnEntity', payload: ['entity1', 'entity2'] },
{ action: 'upsertComponent',
payload: [ 'entity1', 'position', [0, 0, 0 ],
'entity2', 'position', [1, 1, 1] ] } ]

Grouped Component Updates

Related: isGroupedComponent, getGroupedValue, setGroupedValue, changeComponent, and upsertComponent

Component updates can be made in groups, which basically means they are grouped by an array of IDs that correlates to a flattened array of values.

// Message
[ 'upsertComponent',
[ ['entity1', 'entity2'], 'position', [0, 0, 0, 1, 1, 1] ] ]

Ticked Actions

Related: enableRollback, and Ordered

The last value of actorInput, changeComponent, and upsertComponent is reserved for a tick which is a timestamp that represents the elapsed time from the begining of the world.

Numeric Reference Table

Related: compressStringsAsInts, defaultSymbols, enumDefaultSymbols,
One Message Handler, and Many Message Handler

To avoid strings being repeated for actions, by default, Echo-D should use a hardcoded index map in order to lookup actions by the numberic reference (index).

// Message
[ 21, [ 'entity', 'position', [0, 0, 0] ] ]

Reference table that shows the default indexes for actions:

  • action nameindex
    actorInput0
    actors1
    addSymbol2
    batch3
    changeComponent4
    components5
    createEntity6
    entities7
    fetchSymbol8
    getSymbol9
    mergeActors10
  • action nameindex
    mergeComponents11
    mergeEntities12
    mergeSymbols13
    mergeSymbol14
    removeActor15
    removeComponent16
    removeEntity17
    spawnActor18
    symbol19
    symbols20
    upsertComponent21

Action Handler

Related: Options, Context, Action Handler, actions, isAuthoriy, isComponentRelay, and onUpdate

Responsible for handling messages and performing various actions on the context. It provides methods for handling single and multiple messages, getting the action handler, updating other nodes in the network, spawning and removing actors, updating actors with input, creating and removing entities, and manipulating components of entities.

Handler helpers are needed to make it easier to call actions by their numeric reference. This is done by using defaultSymbols which should contain an entry for every action.

Main functionalities

  • Handles messages and performs actions on the context.
  • Provides methods for handling single and multiple messages.
  • Provides methods for updating other nodes in the network.
  • Provides methods for spawning and removing actors.
  • Provides methods for updating actors with input.
  • Provides methods for creating and removing entities.
  • Provides methods for manipulating components of entities.

One Message Handler

Related: Message Structure, and Numeric Reference Table

Calls one action based on a single message and payload.

Many Message Handler

Related: Batch Message Structure, and Numeric Reference Table

Calls many actions based on a possible batch of payloads from a single message.


Actions

Related: Options, Action Handler, Message Structure, and actions

An Actions Object contains all the handler functions for each action that can be called by a action handler.

Main functionalities

  • Combining actions from different modules into a single object.
  • Providing a convenient way to access and perform actions on actors, components, entities, and symbols in the current context.

Actor Actions

actorInput

Handles input for a specific actor in the current context.

// Payload
[ 0, [ { id: 'actor', /* ...input data... */ }, tick ] ]

actors

Retrieves actors from the current context.

// Payload
[ 1, { /* ...query actors... */ } ]

mergeActors

Merges actors into the current context.

// Payload
[ 10, [ 'actor1', 'actor2' ] ]

removeActor

Removes an actor from the current context.

// Payload
[ 15, 'actor' ]

spawnActor

Spawns a new actor in the current context.

// Payload
[ 18, 'actor' ]

Component Actions

changeComponent

Changes a component in the current context.

// Payload
[ 4, [ 'actor|entity', 'position', [0, 0, 0] ] ]

components

Retrieves components from the current context.

// Payload
[ 5 ]

mergeComponents

Merges components into the current context.

// Payload
[ 18, { actor: { position: [0, 0, 0] }, entity: { position: [0, 0, 0] } } ]

If isGroupedComponent is true then the following payload is also supported.

TODO: Support using grouped upsertComponent when isGroupedComponent

// Payload
[ 18, [ [ ['actor', 'entity'], 'position', [0, 0, 0, 0, 0, 0] ] ] ]

removeComponent

Removes a component from the current context.

// Payload
[ 16, [ 'actor|entity', 'position' ] ]

upsertComponent

Updates an existing component or inserts a new one if it doesn’t exist in the current context.

// Payload
[ 21, [ 'actor|entity', 'position', [0, 0, 0] ] ]

Core Actions

batch

Processes a batch of payloads in the current context.

// Payload
[ 3, [ /* ...batch messages ...*/ ] ]

Entity Actions

createEntity

Creates a new entity in the current context.

// Payload
[ 6, 'entity' ]

entities

Retrieves entities from the current context.

// Payload
[ 7 ]

mergeEntities

Merges entities into the current context.

// Payload
[ 12, ['entity1', 'entity2'] ]

removeEntity

Removes an entity from the current context.

// Payload
[ 17, 'entity' ]

Symbol Actions

addSymbol

Adds a symbol to the current context.

// Payload
[ 2, 'symbol' ]

fetchSymbol

Fetches a symbol from the current context.

// Payload
[ 8, symbol ]

getSymbol

Retrieves a symbol from the current context by its index.

// Payload
[ 9, 0 ]

mergeSymbol

Merges a symbol into the current context.

// Payload
[ 14, ['symbol', 0] ]

mergeSymbols

Merges multiple symbols into the current context.

// Payload
[ 13, [0, 'symbol0', 'symbol1'] ]

symbol

Retrieves a symbol from the current context.

// Payload
[ 19, symbol ]

symbols

Retrieves all symbols from the current context.

// Payload
[ 20 ]

Storage

Related: isAsyncStorage, enableQuerying, pageSize, indexes, and types

Represents a store with actors, entities, components, and inputs. It provides methods to manipulate and retrieve data from the store.

An adapter that is used to provide a backend storage layer for actors, entities, components, inputs, and symbols. Storage by default uses an array of structures (AoS) that is kept in memory.

It is possible to implement custom storage adapters that use other ECS libraries or integrate with other software such as databases.

Main functionalities

  • Store and retrieve actors, entities, components, and inputs in the store.
  • Query the store for entities by component.
  • Remove actors, entities, and components from the store.

Async

Related: isAsyncStorage

Represents an asynchronous store with actors, entities, components, and inputs. It provides methods for manipulating and retrieving data from the store.

Main functionalities

  • Store and retrieve actors, entities, components, and inputs asynchronously.
  • Query the store for entities by component.
  • Update and remove components in the components index.

Queries

Related: enableQuerying

Represents a query that is used to filter actors or entities by component. For example, it can be used to find all entities that have a position component. Furthermore, it can be used to find nearby elements to a specific position.

Indexes

Related: enableQuerying, and indexes

represents an index and provides methods for manipulating and querying the index.

Indexes are used by Storage to allow querying of actors, components, and entities.

Main functionalities

  • Storing values in the index with corresponding IDs
  • Checking if a value is in the index
  • Getting a value from the index
  • Removing a value from the index
  • Querying the index

Components Index

Represents an index that stores values and their corresponding IDs. It provides methods for adding, removing, and querying values in the index.

An basic index that is used to find a list of components that belong to a given entity

Main functionalities
  • Stores values and their corresponding IDs in an index
  • Provides methods for adding, removing, and querying values in the index
  • Supports operations like union, difference, and intersection of indexes

Sorted Index

Represents a sorted index. It provides methods to add, remove, and query values in the index.

An sorted index that is used to find a list of actors or entities by a specific component and value range.

Main functionalities
  • Maintains a sorted index of values.
  • It maintains a sorted list of values and their associated IDs.
  • Values can be added, removed, and queried in the index.

Spatial Index

Represents a spatial index. It provides methods for storing, retrieving, and querying values based on their spatial coordinates.

A spatial index that is used to find a list of actors or entities by a specific component and value area.

Main functionalities
  • Store values in the index based on their spatial coordinates
  • Retrieve values from the index based on their spatial coordinates
  • Check if a value is present in the index based on its spatial coordinates
  • Remove values from the index based on their spatial coordinates
  • Query the index to retrieve values within a specified range of spatial coordinates

Updater

Related: compressStringsAsInts, isComponentRelay isReadOnly, and responder

Echo-D has an update cycle that can be inserted at the end of a run loop, or just used to dispatch pending updates to a responder. It relies on the Pending object in Context to determine what updates to call the responder with.

The updater function updates the context based on the provided options. It processes various updates such as creating entities, spawning actors, removing entities and components, and updating components and inputs. It supports batching of updates and can handle different options such as compressing strings as integers, enabling rollback, and diffing changes.

Main functionalities

  • Updating other Echo-D nodes based on the provided options.
  • Processing various updates such as:
    • creating entities
    • spawning actors
    • removing entities and components
    • updating components
    • appending actor inputs

Pending

Related: skipPending, and isReadOnly

Represents a pending state with removed, updated, and created states. It keeps track of changes made to actors, entities, and components.

A list of pending updates needs to be managed so the updater can efficiently send updates to other nodes in the network. The pending object is used to flag elements that need to be updated. When the updater reponds with an update the pending flag for that element is cleared.

Main functionalities

  • Keeping track of changes made to actors, entities, and components
  • Adding and removing actors, entities, and components
  • Marking entities as created or removed
  • Resetting the state of the Pending object
  • Adding and replacing symbol tuples in the symbols array

Ordered

Related: Ticked Actions, and isOrdered

Represents a collection of tick values and provides methods to change, reset, and insert/update tick values for components.

Component updates should not be applied unless they have the latest tick. In order to determine if the change should be applied a history of previous tick values for each component is needed. Then the last latest tick can be compared to new ticks to see which is greater. If a new tick is greater than the last then the update will be applied.

Main functionalities

  • Storing and managing tick values for components.
  • Changing the tick value of a component.
  • Resetting the tick values.
  • Inserting or updating the tick value of a component.

Changes

Related: isDiffed

Responsible for managing changes in a Context. It provides methods for changing components, retrieving values, resetting changes, and updating or inserting components.

Tracks differences between the last outgoing update for every component change that is made and accumulates the difference into temporary cache. When the Updater is called then it will send the difference to changeComponent which will apply the difference elsewhere.

Main functionalities

  • Changing a component in the context
  • Retrieving the changes of a value
  • Resetting the changes to a new set of changes or an empty object
  • Updating or inserting a component in the context