Significantly Slash Savefile Sizes

[Web-eWorks]

Our savefiles are truly too large. Not only are we storing completely redundant data about MatrixTransform nodes that will immediately be overwritten by the animation system, but we're also storing every last detail about every market in the system regardless of whether we've been there or not.

On top of removing redundant save data from the C++ side, we now only create a station's market when we actually land there, which removes one of the biggest hogs of save data. How big, you ask? Uncompressed, our savefiles dropped from 2MB to ~500KB, and compressed they dropped from 350KB to 75KB. I've tested on a Mars start after taking off and landing at Mars High to generate a representative sample of the average in-system activity a player might have.

This has a nice side effect of making autosaves almost entirely imperceptible to gameplay; it's not quite as smooth as an asynchronous save might be, but we practically fly through the save process now.

@impaktor for your reference, this is following on to your(?) comment in the code about only creating adverts when docked... I just extended that to the entire station market instead of just the BBS.

I'd appreciate some thorough testing - I've put it through a minimal test case, but I'd appreciate it if this could be tested against someone's in-progress campaign just to be sure there aren't any edge cases or catastrophic oversights I've missed!

EDIT: I know I said we'd be freezing for an RC build soon, but while we're bumping the save version I wanted to fix as many "bugs" as possible. 😛

[Gliese852]

That's wonderful!

[Gliese852]

that is, until you land on the station, prices for goods do not exist now?

[Web-eWorks]

@Gliese852 yes, this defers creating a station's market, stocking it with goods, and making ships and advertisements for the BBS until a player physically lands at a station.

Right now a station's prices for goods does not differ from the overall system, so this poses no problem for existing code. In the future, I'd recommend treading lightly with what you serialize for "remote" stations that the player hasn't landed at - a table of string keys and string/number values is the most optimal and lightweight serialization mechanism when we're talking about filesizes and work needed to save/load. Storing subtables (e.g. equipment items) should ideally be avoided if you can, as they represent additional work (and a significantly larger portion of filesize) compared to basic "scalar" values like strings, numbers, and booleans.

Edit: a subtable, at the very lightest, involves an extra 20 bytes of overhead for every single reference to it stored in the save, as well as a decently expensive de-duplication check. Strings and numbers can be simply pushed directly to lua and don't have to go through the same mechanism.

[impaktor]

this defers creating a station's market, stocking it with goods, and making ships and advertisements for the BBS until a player physically lands at a station.

Is there a way to trigger spawning fully populated stations from lua? I've relied on being able to query all stations for stock or adverts or ship market, to gather statistics on how many ships for sale, how many adverts of some flavour, or as you just saw, how many of some stock. It helps to balance things out.

For testers: make sure to check if BBS adverts are duplicated or lost between saves, as this has been recurrent theme.

[Web-eWorks]

@impaktor I definitely had that in mind when writing this; it's not currently supported to generate everything in the system, but adding that when testing is as simple as looping through all stations in an onEnterSystem event and initializing a station market.

I'll look into adding that in a commented out function for your convenience when I get a chance!

[impaktor]

well, if it's as easy as looping through the stations, then it sounds trivial, as only 2-3 lines of code, so no action needed on your part, thanks!

[fluffyfreak]

This makes me wonder what's in that remaining 500KB

[Web-eWorks]

This makes me wonder what's in that remaining 500KB

@fluffyfreak mostly it's space Frames, bodies serialized from C++, modules' copies of advertisements that are also stored on space stations... I've done the 10% to get the 90% benefit, but there's certainly a few more tens of KB we could shave off. Although at this point I think that compression renders it negligible. We're almost fitting our savefile into a single LZ4 frame, which means that we're getting near-optimal compression of repeated data.

[fluffyfreak]

Code looks good as always!
I haven't had chance to test it yet so maybe let it bake whilst others give that a go for a few days?

[impaktor]

I haven't had chance to test it yet so maybe let it bake whilst others give that a go for a few days?

I wonder if the best thing in these cases is to do a windows build and put on our discord channel and ask for players to throw themselves on the blade of the bleeding edge release

[Web-eWorks]

@impaktor tada... https://github.com/pioneerspacesim/pioneer/suites/1669330574/artifacts/31118667 - there's your windows build, now all that's required is to put it on discord! 😄

[Web-eWorks]

Because we're coming very close to cutting a release candidate build for large-scale testing, I'm going to go ahead and merge this in; I haven't seen any issues be reported on IRC and we now have people playing nightly master builds as primary bugtesting.