Specification v0.0.1
Echo-D is a library for distributing Entity-Component state in real-time.
Echo-D is network transport and storage agnostic. For example WebRTC and it can integrate into existing ECS libraries or databases.
Echo-D is designed for cloud-based game engines. It is ideal for session-based multiplayer games with server-side physics or rendering.
This document describes the Echo-D protocol, as well as the main Echo-D APIs and data structures.
Table of contents
- Echo-D Specification
- Options
- actions
- batchActionPayloadSizes
- compressStringsAsInts
- defaultSymbols
- enableRollback
- enableQuerying
- enumDefaultSymbols
- getActorId
- getGroupedValue
- indexes
- isAuthority
- isAsyncStorage
- isComponentRelay
- isDiffed
- isGroupedComponents
- isOrdered
- isReadOnly
- isSymbolLeader
- isSymbolRelay
- onUpdate
- pageSize
- responder
- skipPending
- types
- setGroupedValue
- updateOptions
- Context
- Component Value Types
- Symbols
- Message Structure
- Action Handler
- Actions
- Storage
- Updater
- Options
Options
Related: Context, Action Handler, and Actions
Represents the options for a Echo-D node. It provides a way to configure various settings and behaviors for the node.
There are many possible configurations for an Echo-D node.
All of the parameters are encapsulated by a Options
.
There should be a static extend
method to make it easy to
create a new instance of an existing set of options, while
applying new custom options.
Main functionalities
- Allows configuring various options for a node
- Provides default values for the options
- Ensures that the provided options are an instance of Options
- Clones and extends the options object
Option Definitions
actions :
Object
Related: Action Handler, and Actions
A map of actions that is used by an
Action Handler
in order to repond to messages that are recieved. They can be overriden in this reference.
batchActionPayloadSizes :
Object
Related: Batch Message Structure
In order to serialize and deserialize batched messages, the size of the message payload needs to be known. However there is a necessary complexity that requires there be a way to indicate a different size if certain flags in options are true:
compressStringsAsInts :
Boolean
Related: Symbols, Numeric Reference Table, and Updater
The protocol uses
Symbols
which are strings that have been mapped to a unique integer. This allows them to be referenced by the associated integer from an index and compresses the size of strings being sent and recieved.
defaultSymbols :
Array[String]
Related: Numeric Reference Table
An array that maps the default set of symbols by number. The values of the array are strings that represent symbols and the indexes of the values are the unique number IDs for the symbol.
enumDefaultSymbols :
Object
Related: Numeric Reference Table
An object that maps the default symbols by string. The keys of the object are strings that represent symbols and the values are the unique number IDs for each key (symbol).
enableQuerying :
Boolean
Related: Queries, Indexes, and Storage
A flag to indicate if querying is enabled. When enabled the payload of
actors
,entities
, andcomponents
can be a query object that will filter out unwanted items.
enableRollback :
Boolean
Related: Ticked Actions
A flag to indicate if time rollback is enabled on inputs. This expects an extra
tick
value in the payload ofactorInput
. Thetick
can then be used to implment a rollback strategy on the client nodes.
getActorId :
Function
A function to get the actor ID. It is passed an ID string, the default behavior is to return it.
getGroupedValue :
Function
Related: Grouped Component Updates
A function to get the value from a group. By default it slices a section of an array that matches the defined type size. However it might be necessary for some Storage adapters to convert the values.
indexes :
Object
An object containing a mapping indexes. You can specify either
sorted
orspatial
indexes. A sorted index will organize values into a sorted array for indexing. A spatial index will use a spatial hashing function to index nearby elements and allow effecient nearest neighbor queries.
isAuthority :
Boolean
Related: Action Handler
A flag to indicate if the node is an authority. An authorative node can is authorized to send updates.
isAsyncStorage :
Boolean
A flag to indicate if the node uses async storage. This is useful if the
Storage
adapter is custom and uses an asynchronous API to read and write data. Note that actions that write to the store do not wait for the promise to finish, therefore they do not handle errors.
isComponentRelay :
Boolean
Related: Action Handler, and Updater
A flag to indicate if the node is a component relay. Disabling this means component updates will not be sent.
isDiffed :
Boolean
Related: Changes
A flag to indicate if the node is diffed. This uses
Changes
in order to track updates as differences. For example setting aposition
component to[0, 0, 0]
that was previously set to[1, 1, 1]
will result in[-1, -1, -1]
being sent as an update tochangeComponent
instead ofupsertComponent
which will apply the change elsewhere. This allows changes to happen in any order without changing the final value.
isGroupedComponents :
Boolean
Related: Grouped Component Updates
A flag to indicate if the node uses grouped components. This allows for structure of arrays (SoA) to be used to speed up grouped component updates.
isOrdered :
Boolean
Related: Ordered
A flag to indicate if the node is ticked. This adds a
tick
value to the end of the payloads forchangeComponent
andupsertComponent
. Thetick
is used to filter outdated values from being applied as an update.It is possible to enable
isDiffed
when this is also enabled, in order to differentiate from other node configurations thetick
value will be inverted in updates. It is not recommended to use both these settings at once.
isReadOnly :
Boolean
A flag to indicate if the node is read only. A node that is read-only will not be capable of sending updates to other nodes.
isSymbolLeader :
Boolean
Related: Symbols
A flag to indicate if the node is a symbol leader. Only a symbol leader can declare symbols in the network. There should ideally only be one source of truth. This prevents inconsistencies. In the future a CRDT G-Set maybe used instead, but that would add complexity.
isSymbolRelay :
Boolean
Related: Symbols
A flag to indicate if the node is a symbol relay. If disabled then the node will not forward symbol changes to other nodes. A symbol relay sends symbols that are recieved from a leader to other nodes.
onUpdate :
Function
Related: Action Handler
A function to call when the node is updated. It will be call once per update, it is recommended that this function be manually throttled if it is being used in the client.
pageSize :
Number
Related: Storage
The page size for list actions:
actors
,components
,entities
,symbols
. This keeps very large lists from being sent by oneresponder
call. The pages are emitted until all items are sent by theresponder
in multiple calls.
responder :
Function
Related: Updater
A function that is called by the
Updater
to send update actions to other nodes in the network. This allows for any network transport, inter-process communication, or a Pub/Sub library to be used for responses.
skipPending :
Boolean
Related: Pending
A flag to skip pending actions. This is useful when component updates need to happen without causing a pending change that the
Updater
will then use to update the network.
types :
Object
Related: Component Value Types, isGroupedComponents, and Storage
An object that defines the component types. A component type can be a class, or a string that maps to a valid type. If an array is provided as the component type, then the first value will be used as type, and the remaining values will provide meta information. The second value will be the size of each section in a group.
setGroupedValue :
Function
Related: Grouped Component Updates
A function to set the value in a group. By default this returns the value from the component value that is passed. If the original structure is an object then it may be necessary to map values to an array here.
updateOptions :
Object
Related: Batch Message Structure
An object containing options for the updater. See below for more details on each update option:
mask :
Object
Specify which kind of actions should be ignored by the updater.
type :
Boolean|String
Secondary type argument for responder.
batched :
Boolean
Indicate whether or not to batch update messages together
batchSize :
Number
Target number of message to send with each batch
validKeys :
Object
Selects which component keys are valid for sending updates
storageOptions :
Object
Related: Storage
An object containing the storage options, this is added to make it easier to implement custom
Storage
adapters that work with other ECS libraries.
Component Value Types
Related: types
Echo-D uses a types that are based on common JavaScript/TypeScript
object types.
Implementations must have access to basic JSON
like data structures such as
String
, Number
, and Array
.
A list of valid types for use in types
:
- ‘str’ :
String
— UTF-8 string - ‘bool’ :
Boolean
— true or false - ‘num’ :
Number
— 64 bit float - ‘map’ :
Map
— key-value pairs of objects - ‘set’ :
Set
— unique sequence of objects - ‘arr’ :
Array
— list of objects
Typed Arrays
In order to better support web based platforms Echo-D only uses arrays
of non-Number
numbers.
These are used to group component updates into structure of
array (SoA) buffers.
A list of valid typed arrays for effiently storing vectors, and matricies:
- ‘i8’ :
Int8Array
— array of 8 bit integers - ‘ui8’ :
Uint8Array
— array of 8 bit unsigned integers - ‘ui8c’ :
Uint8ClampedArray
— array of 8 bit unsigned integers clamped to (0-255) - ‘i16’ :
Int16Array
— array of 16 bit integers - ‘ui16’ :
Uint16Array
— array of 16 bit unsigned integers - ‘i32’ :
Int32Array
— array of 32 bit integers - ‘ui32’ :
Uint32Array
— array of 32 bit unsigned integers - ‘f32’ :
Float32Array
— array of 32 bit floats - ‘f64’ :
Float64Array
— array of 64 bit floats
Context
Related: Options, and Action Handler
Passes a context around to different objects that is used internally to handle
logic between actions. Such as Storage
, and the Updater
.
Main functionalities
- Managing actors, entities, components, inputs, and symbols.
- Creating and removing actors, entities, and components.
- Changing and upserting components.
- Handling actor inputs.
- Managing symbols.
Symbols
Related: compressStringsAsInts, isSymbolLeader, and isSymbolRelay
Represents a collection of symbols and provides methods to manipulate and access these symbols.
Echo-D manages a list of symbols that are created from strings. Wherever a string is used in an action message, except in symbol actions, it is replaced by a unique number.
This mapping between strings, and numbers is considered a Symbol
.
This significantly reduces the number of strings that will be sent
over the network.
This behavior is controlled by the compressStringsAsInts
flag.
Main functionalities
- Adding symbols to the collection
- Finding the index of a symbol
- Getting the symbol at a specific index
- Getting the list of symbols
- Getting the enum of symbols
- Resetting the collection with a new array of symbols
Message Structure
Related: One Message Handler, and Actions
Echo-D sends and recieves messages as either a tuple or an object. Each with two values, first the action, then the payload which is optional.
Batch Message Structure
Related: batchActionPayloadSizes, updateOptions, batch, and Many Message Handler
Messages can be batched in Echo-D, the given action is being called multiple times but it is only provided once first, then the remaining elements of the payload which is a flat array contains all batched payloads for the provided action.
Objects can also be used in array-object hybrid messages.
Grouped Component Updates
Related: isGroupedComponent, getGroupedValue, setGroupedValue, changeComponent, and upsertComponent
Component updates can be made in groups, which basically means they are grouped by an array of IDs that correlates to a flattened array of values.
Ticked Actions
Related: enableRollback, and Ordered
The last value of actorInput
, changeComponent
, and upsertComponent
is reserved for a tick
which is a timestamp that represents the
elapsed time from the begining of the world.
Numeric Reference Table
Related: compressStringsAsInts,
defaultSymbols,
enumDefaultSymbols,
One Message Handler,
and Many Message Handler
To avoid strings being repeated for actions, by default, Echo-D should use a hardcoded index map in order to lookup actions by the numberic reference (index).
Reference table that shows the default indexes for actions:
-
action name index actorInput 0 actors 1 addSymbol 2 batch 3 changeComponent 4 components 5 createEntity 6 entities 7 fetchSymbol 8 getSymbol 9 mergeActors 10 -
action name index mergeComponents 11 mergeEntities 12 mergeSymbols 13 mergeSymbol 14 removeActor 15 removeComponent 16 removeEntity 17 spawnActor 18 symbol 19 symbols 20 upsertComponent 21
Action Handler
Related: Options, Context, Action Handler, actions, isAuthoriy, isComponentRelay, and onUpdate
Responsible for handling messages and performing various actions on the context. It provides methods for handling single and multiple messages, getting the action handler, updating other nodes in the network, spawning and removing actors, updating actors with input, creating and removing entities, and manipulating components of entities.
Handler helpers are needed to make it easier to call actions by their numeric
reference. This is done by using defaultSymbols
which should contain an
entry for every action.
Main functionalities
- Handles messages and performs actions on the context.
- Provides methods for handling single and multiple messages.
- Provides methods for updating other nodes in the network.
- Provides methods for spawning and removing actors.
- Provides methods for updating actors with input.
- Provides methods for creating and removing entities.
- Provides methods for manipulating components of entities.
One Message Handler
Related: Message Structure, and Numeric Reference Table
Calls one action based on a single message and payload.
Many Message Handler
Related: Batch Message Structure, and Numeric Reference Table
Calls many actions based on a possible batch of payloads from a single message.
Actions
Related: Options, Action Handler, Message Structure, and actions
An Actions Object
contains all the handler functions for each action
that can be called by a action handler.
Main functionalities
- Combining actions from different modules into a single object.
- Providing a convenient way to access and perform actions on actors, components, entities, and symbols in the current context.
Actor Actions
actorInput
Handles input for a specific actor in the current context.
actors
Retrieves actors from the current context.
mergeActors
Merges actors into the current context.
removeActor
Removes an actor from the current context.
spawnActor
Spawns a new actor in the current context.
Component Actions
changeComponent
Changes a component in the current context.
components
Retrieves components from the current context.
mergeComponents
Merges components into the current context.
If
isGroupedComponent
is true then the following payload is also supported.TODO: Support using grouped upsertComponent when isGroupedComponent
removeComponent
Removes a component from the current context.
upsertComponent
Updates an existing component or inserts a new one if it doesn’t exist in the current context.
Core Actions
batch
Processes a batch of payloads in the current context.
Entity Actions
createEntity
Creates a new entity in the current context.
entities
Retrieves entities from the current context.
mergeEntities
Merges entities into the current context.
removeEntity
Removes an entity from the current context.
Symbol Actions
addSymbol
Adds a symbol to the current context.
fetchSymbol
Fetches a symbol from the current context.
getSymbol
Retrieves a symbol from the current context by its index.
mergeSymbol
Merges a symbol into the current context.
mergeSymbols
Merges multiple symbols into the current context.
symbol
Retrieves a symbol from the current context.
symbols
Retrieves all symbols from the current context.
Storage
Related: isAsyncStorage, enableQuerying, pageSize, indexes, and types
Represents a store with actors, entities, components, and inputs. It provides methods to manipulate and retrieve data from the store.
An adapter that is used to provide a backend storage layer for actors, entities,
components, inputs, and symbols.
Storage
by default uses an array of structures (AoS) that is kept
in memory.
It is possible to implement custom storage adapters that use other
ECS
libraries or integrate with other software such as databases.
Main functionalities
- Store and retrieve actors, entities, components, and inputs in the store.
- Query the store for entities by component.
- Remove actors, entities, and components from the store.
Async
Related: isAsyncStorage
Represents an asynchronous store with actors, entities, components, and inputs. It provides methods for manipulating and retrieving data from the store.
Main functionalities
- Store and retrieve actors, entities, components, and inputs asynchronously.
- Query the store for entities by component.
- Update and remove components in the components index.
Queries
Related: enableQuerying
Represents a query that is used to filter actors or entities by component.
For example, it can be used to find all entities that have a position
component.
Furthermore, it can be used to find nearby elements to a specific position.
Indexes
Related: enableQuerying, and indexes
represents an index and provides methods for manipulating and querying the index.
Indexes are used by Storage
to allow querying of actors,
components, and entities.
Main functionalities
- Storing values in the index with corresponding IDs
- Checking if a value is in the index
- Getting a value from the index
- Removing a value from the index
- Querying the index
Components Index
Represents an index that stores values and their corresponding IDs. It provides methods for adding, removing, and querying values in the index.
An basic index that is used to find a list of components that belong to a given entity
Main functionalities
- Stores values and their corresponding IDs in an index
- Provides methods for adding, removing, and querying values in the index
- Supports operations like union, difference, and intersection of indexes
Sorted Index
Represents a sorted index. It provides methods to add, remove, and query values in the index.
An sorted index that is used to find a list of actors or entities by a specific component and value range.
Main functionalities
- Maintains a sorted index of values.
- It maintains a sorted list of values and their associated IDs.
- Values can be added, removed, and queried in the index.
Spatial Index
Represents a spatial index. It provides methods for storing, retrieving, and querying values based on their spatial coordinates.
A spatial index that is used to find a list of actors or entities by a specific component and value area.
Main functionalities
- Store values in the index based on their spatial coordinates
- Retrieve values from the index based on their spatial coordinates
- Check if a value is present in the index based on its spatial coordinates
- Remove values from the index based on their spatial coordinates
- Query the index to retrieve values within a specified range of spatial coordinates
Updater
Related: compressStringsAsInts, isComponentRelay isReadOnly, and responder
Echo-D has an update cycle that can be inserted at the end of a run loop,
or just used to dispatch pending updates to a responder.
It relies on the Pending
object in Context
to determine what
updates to call the responder with.
The updater
function updates the context based on the provided options.
It processes various updates such as creating entities, spawning actors,
removing entities and components, and updating components and inputs.
It supports batching of updates and can handle different options such as
compressing strings as integers, enabling rollback, and diffing changes.
Main functionalities
- Updating other Echo-D nodes based on the provided options.
- Processing various updates such as:
- creating entities
- spawning actors
- removing entities and components
- updating components
- appending actor inputs
Pending
Related: skipPending, and isReadOnly
Represents a pending state with removed, updated, and created states. It keeps track of changes made to actors, entities, and components.
A list of pending updates needs to be managed so the updater can efficiently send updates to other nodes in the network. The pending object is used to flag elements that need to be updated. When the updater reponds with an update the pending flag for that element is cleared.
Main functionalities
- Keeping track of changes made to actors, entities, and components
- Adding and removing actors, entities, and components
- Marking entities as created or removed
- Resetting the state of the
Pending
object - Adding and replacing symbol tuples in the symbols array
Ordered
Related: Ticked Actions, and isOrdered
Represents a collection of tick values and provides methods to change, reset, and insert/update tick values for components.
Component updates should not be applied unless they have the latest tick
.
In order to determine if the change should be applied a history of
previous tick
values for each component is needed.
Then the last latest tick
can be compared to new ticks
to see which is
greater.
If a new tick
is greater than the last then the update will be applied.
Main functionalities
- Storing and managing tick values for components.
- Changing the tick value of a component.
- Resetting the tick values.
- Inserting or updating the tick value of a component.
Changes
Related: isDiffed
Responsible for managing changes in a Context
.
It provides methods for changing components,
retrieving values, resetting changes, and updating or inserting components.
Tracks differences between the last outgoing update for every component
change that is made and accumulates the difference into temporary cache.
When the Updater
is called then it will send the difference to
changeComponent
which will apply the difference elsewhere.
Main functionalities
- Changing a component in the context
- Retrieving the changes of a value
- Resetting the changes to a new set of changes or an empty object
- Updating or inserting a component in the context