Question Regarding Server State Restoration in Ecspanse

[mackdunkan]

Hello,

I am using your Ecspanse framework for a project, and I have a question about restoring the server state after a restart, redeployment, or migration to a different server.

I noticed that the Ecspanse.Server module has a debug/0 function that returns the current server state as a Ecspanse.Server.State.t struct. I assume that I can save this state and then use it to restore the server. However, I am not entirely sure about the correct process for doing this to avoid errors and data loss.

Could you please clarify the following points:

Correct way to save the state: Is it sufficient to just call debug/0 and save the result, or are there additional steps that need to be taken?

Restoring the state: If I load the saved state and pass it during server startup, will all components and resources be restored correctly? Is any additional initialization required for this?

Data compatibility: How can I ensure that the restored state will be compatible with the current version of Ecspanse and my project’s modules? Are there any particular considerations to keep in mind when migrating state between different versions of the application or the framework?

Best practices: Could you recommend any best practices for ensuring reliability and resilience when restoring the server?

Any information and advice on these matters would be greatly appreciated. Thank you in advance for your help and for the great work you’ve done on this framework!

Best regards,
mackdunkan

[iacobson]

hi @mackdunkan

First of all, thanks for using Ecspanse. Also, Great questions! and I must confess I faced most of them while developing the library.
Second, it will be a long reply 😃 ...

As I don't have a game development background, but a backed developer background, I initially thought that state persistence would be a central part of any game engine.

Well, surprise, it isn't! No game engine I've studied while building Ecspanse had one standard way of persisting the current state, and it looks like each developer chooses some different strategy for their game save/load. Why is that? I suppose because of all the possible problems you are highlighting in your issue above. And it is the reason I left it out of Ecspanse as well. At least until now.

Let's say, you find a way to serialize every component (or object) in your game, and then deserialize back when the game is initializing. But what about the code changes? What about the changes in the structure of a component, what about new components added to some entities?

I think of a game save and load process, as I think of DB migrations, but much, much harder to reason about.

Now I will try to punctually answer your questions, and then discuss a bit about what can be next.

Correct way to save the state: Is it sufficient to just call debug/0 and save the result, or are there additional steps that need to be taken?

No, that will not work. The server state has more to do with scheduling systems and events. The components themselves and resources are persisted in :ets tables.

So, if you want to dump raw data from those tables, you can do so:

:ets.tab2list(:ets_ecspanse_components_state)
:ets.tab2list(:ets_ecspanse_resources_state)

Restoring the state: If I load the saved state and pass it during server startup, will all components and resources be restored correctly? Is any additional initialization required for this?

Theoretically, if the state of those 2 ets tables is restored during startup, you would load the saved state. However, this does not solve the problem of code changes.

Data compatibility: How can I ensure that the restored state will be compatible with the current version of Ecspanse and my project’s modules? Are there any particular considerations to keep in mind when migrating state between different versions of the application or the framework?

This is the most complicated part.
The project where I use Ecspanse, has a simplistic approach, due to the specifics of the project itself. It is a multiplayer game, with short lived sessions. The player cannot pause the game, and if they lose the session they will lose all their state.
The only thing I had to persist between sessions, were global data such as accumulated points, or some meta progression.

My approach for this was to persist just the required info in a database. Just as data. Nothing ECS related.
Then, when the player would re-join the game after a while, on the player creation system, I would pull that info from the DB and create the required components for the player, or some additional entities.

That was also my plan for extending this system for example if the player would carry different items between game sessions. But I'm totally aware that this does not work for all cases. And it is especially difficult, if you have a game client, separated from the server. In my case it is just liveview, so no special concerns to handle code changes, components changes, etc.

Best practices: Could you recommend any best practices for ensuring reliability and resilience when restoring the server?

My main sources of inspiration when developing Ecspanse were Bevy Engine for every ECS-related aspects and Godot for best practices, reads about game engines, etc.

Gotot is not ECS, it's "classic" OOP. So it won't be directly helpful. But here are 2 resources I checked when considering saving and loading state:

• https://docs.godotengine.org/en/stable/tutorials/io/saving_games.html
• https://www.youtube.com/watch?v=43BZsLZheA4&t=1s
• https://www.youtube.com/watch?v=j7p7cGj20jU

What's next?

Now that this subject is brought up again, I'm actually thinking to do something for Ecspanse.
Here's the top level idea:

• have a serialize function that can be run in any sync system and would encode to JSON all components, grouped by entity, and all resources
• extend the serialize to work just for one entity, an entity and their descendants, or on a custom query that outputs entities only - in case partial state needs to be saved
• implement an ECS version (integer) that the dev would bump every time they do breaking updates to the code. The version will be also serialized in the JSON
• implement a deserialize/restore function that would take the JSON, iterate by entity and force-insert it. Meaning that it would overwrite any existing component for that entity, if existing
• those functions serialize/deserialize functions can be implemented by the dev in any sync system, so they can arrange the moment the state is saved and load
• have an on_restore callback (optional) that would receive every entity data (all components specs) before being restored
  • this is useful to ensure data compatibility. Eg. did the name of a field of a component change? intercept it and fix it in the callback, before being restored
  • this is where things begin to get very complicated, especially in the parent-child relations. And I can already see many edge cases, without even thinking too much

Pls note that those are just initial thoughts. Not sure if they would work as such. Even if they do, it will take some good time to develop it, as I cannot allocate more than a few hours per week to this.

What do you think? Any suggestions?

Thanks,
Dorian

[iacobson]

I've been thinking a bit about it, and the solution described above, won't work.
The component state is very flexible, it can store any type of term. So even if I parse the top level structure in a format that is JSON friendly, the component state can contain for example a tuple, that cannot be serialized to JSON. Also, all the atoms are lost when serializing to JSON, and the components are too flexible for this approach.

I thought then to go for :erlang.term2binary and then use Plug.Crypto.non_executable_binary_to_term(binary, [:safe]) to decode. But there are edge cases here as well:

• some encoded atoms may not exist since the last save
• some structs may not exist also, so that they will be decoded as maps...

So I thought about some more flexible approach.

• have an API for the dev to export the current entities and resources, in a standard format (some struct). Then the dev can change them and persist them any way they would like.
• than have another function to import entities and resources, as long as they keep the format (struct) of the export.

For the most simple case of saving and loading the full state, without any changes, the dev would not even care about the content of the export. It can be just fed back in the importer and restore the state.

[mackdunkan]

Hi Dorian,

Thank you for the detailed response and for sharing your thoughts on the challenges of state persistence in Ecspanse.

The more flexible approach you suggested, involving an API for exporting and importing entities and resources in a standardized format, seems like a promising direction. This would indeed allow developers to handle state persistence in a way that best suits their needs, whether that be simple save/load functionality or more complex data migrations. It also provides the flexibility to adapt to code changes without losing data integrity.

Once again, thanks for your continued support and work on Ecspanse. The direction you're considering for state management sounds very promising, and I'm excited to see where it goes.

Have you looked at the Waterpark system? Although it is quite specific, it may prompt some ideas for developing similar functions in Ecspanse, especially in handling distributed state and saving data or the entire project. Just thoughts)) But it would be interesting

Best regards, MackDunkan

[iacobson]

Hi MackDunkan,

I have just released the v0.10.0, introducing the Ecspanse.Snapshot module. Also a new guide with some ideas about saving and loading the ECS state.

If you plan to test it, I am very interested in any feedback.