player-specific key value store #3722

New Issue

AliasAlreadyTaken · 2023-02-04T12:36:13Z

AliasAlreadyTaken commented

2023-02-04 12:36:13 +00:00

We need one (or more than one) key-value stores per player to go around the shortcomings of Minetest.

What's the purpose?

journal (personal log)
yl_settings
yl_quests
factions

Why can't we use player meta?

Because player meta cannot be accessed when the player is offline.

Why can't we use modstorage then?

Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up. To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor.

Storing playerspecific values in modstorage also creates a huge amount of dead entries, see the party mod (which stores each and everyone who ever created an account in a huge file, of which only very few entries are actually needed)

modstorage isone of my least favourite minetest-isms, I'd rather only use it for very simple mod-specific values, not values tied to blocks or players.

Is there an example that already does it?

Yes, the mail mod stores one file per player and yl_speak_up stores one file per NPC

This issue is meant to discuss our options.

So far, my preferred method is to have one JSON file per player per mod and only load that JSON file when necessary. I chose JSON because it can be easily migrated should we feel the need to leave MT behind and edited should something go wrong.
We could ask the core devs to pick up
https://github.com/minetest/minetest/pull/9177
or
https://github.com/minetest/minetest/pull/9164
again.
We could pull in a way to access postgresql from the mod sandbox. This would create a way to not only acces playermeta, but also privs and invs and everything, even blocksdata. That's at least a double edged sword IMO.

We need one (or more than one) key-value stores per player to go around the shortcomings of Minetest. What's the purpose? - journal (personal log) - yl_settings - yl_quests - factions Why can't we use player meta? Because player meta cannot be accessed when the player is offline. Why can't we use modstorage then? Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up. To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor. Storing playerspecific values in modstorage also creates a huge amount of dead entries, see the party mod (which stores each and everyone who ever created an account in a huge file, of which only very few entries are actually needed) modstorage isone of my least favourite minetest-isms, I'd rather only use it for very simple mod-specific values, not values tied to blocks or players. Is there an example that already does it? Yes, the mail mod stores one file per player and yl_speak_up stores one file per NPC This issue is meant to discuss our options. 1. So far, my preferred method is to have one JSON file per player per mod and only load that JSON file when necessary. I chose JSON because it can be easily migrated should we feel the need to leave MT behind and edited should something go wrong. 2. We could ask the core devs to pick up https://github.com/minetest/minetest/pull/9177 or https://github.com/minetest/minetest/pull/9164 again. 3. We could pull in a way to access postgresql from the mod sandbox. This would create a way to not only acces playermeta, but also privs and invs and everything, even blocksdata. That's at least a double edged sword IMO.

AliasAlreadyTaken added the

4. step/discussion

2. prio/controversial

labels 2023-02-04 12:36:22 +00:00

AliasAlreadyTaken referenced this issue

2023-02-04 12:36:40 +00:00

daydream reports: I can't get into my personal l ... #3694

Bla commented

2023-02-04 15:59:15 +00:00

Not sure but I thought postgresql is now supported for all backends which should solve a couple things.

Storing playerspecific values in modstorage also creates a huge amount of dead entries, see the party mod (which stores each and everyone who ever created an account in a huge file, of which only very few entries are actually needed)

That is just bad design decision from the party mod just like personal_log creating one entry per saved location, but JSON wouldn´t solve that they would just create unnecessary files.

Not sure but I thought postgresql is now supported for all backends which should solve a couple things. > Storing playerspecific values in modstorage also creates a huge amount of dead entries, see the party mod (which stores each and everyone who ever created an account in a huge file, of which only very few entries are actually needed) That is just bad design decision from the party mod just like personal_log creating one entry per saved location, but JSON wouldn´t solve that they would just create unnecessary files.

flux commented

2023-02-04 18:27:56 +00:00

Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up.

you can absolutely have read-only access to the sqlite database while the server is up, and that restriction applies to json files as well. if you edit them while the server is running, the changes will probably get clobbered.

To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor.

that's a fair point. i suppose i shouldn't assume that everyone is fluent in SQL :)

Storing playerspecific values in modstorage also creates a huge amount of dead entries

i'm not sure how storing the values in json files would be any better?

one JSON file per player per mod

the number of players we've currently got is over 69000. while file systems can handle that number of files in a directory, you better not try to use "ls" or such.

at the end of the day, though, if we write the API well, we can switch out the backend for something else if it creates performance problems.

We could pull in a way to access postgresql from the mod sandbox. This would create a way to not only acces playermeta, but also privs and invs and everything, even blocksdata...

i started work on an API for allowing access to a postgres database w/out having to explicitly request an insecure environment, but i did not mean for it to be a way to modify the values of engine-controlled data while the server is running. since the server caches most or all of those things and assumes nothing else is touching them, modifications are likely to be clobbered. (aside: i've successfully modified the inventories of players who were not logged in while blocky survival was running, via command-line scripts).

i'm not sure why postgres would be better than sqlite (still can't edit the data w/ vim/emacs/nano/whatever), but i could finish up that mod if we're interested.

> Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up. you can absolutely have read-only access to the sqlite database while the server is up, and that restriction applies to json files as well. if you edit them while the server is running, the changes will probably get clobbered. > To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor. that's a fair point. i suppose i shouldn't assume that everyone is fluent in SQL :) > Storing playerspecific values in modstorage also creates a huge amount of dead entries i'm not sure how storing the values in json files would be any better? > one JSON file per player per mod the number of players we've currently got is over 69000. while file systems can handle that number of files in a directory, you better not try to use "ls" or such. at the end of the day, though, if we write the API well, we can switch out the backend for something else if it creates performance problems. > We could pull in a way to access postgresql from the mod sandbox. This would create a way to not only acces playermeta, but also privs and invs and everything, even blocksdata... i started work on an API for allowing access to a postgres database w/out having to explicitly request an insecure environment, but i did *not* mean for it to be a way to modify the values of engine-controlled data while the server is running. since the server caches most or all of those things and assumes nothing else is touching them, modifications are likely to be clobbered. (aside: i've successfully modified the inventories of players who were not logged in while blocky survival was running, via command-line scripts). i'm not sure why postgres would be better than sqlite (still can't edit the data w/ vim/emacs/nano/whatever), but i could finish up that mod if we're interested.

Bla commented

2023-02-04 20:10:22 +00:00

Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up.

you can absolutely have read-only access to the sqlite database while the server is up, and that restriction applies to json files as well. if you edit them while the server is running, the changes will probably get clobbered.

i'm not sure why postgres would be better than sqlite (still can't edit the data w/ vim/emacs/nano/whatever), but i could finish up that mod if we're interested.

postgres has the advantage that it isn´t restricted to read only while the server is running.

To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor.

that's a fair point. i suppose i shouldn't assume that everyone is fluent in SQL :)

basic SQL isn´t that hard and minetest´s database scheme isn´t that complicated which doesn´t make it really harder than spreadsheet editing.
Also if you are that far down the rabbit hole that you have to edit those entries I think its fair to assume that you can install and start the necessary tools 😉
And for JSON I would still prefer a good editor than any simple texteditor, just to easy to introduce typos or somehow mess up syntax.

> > Because the only efficient way to save modstorage is in a sqlite database, which is not accessible from the outside while the server is up. > > you can absolutely have read-only access to the sqlite database while the server is up, and that restriction applies to json files as well. if you edit them while the server is running, the changes will probably get clobbered. > i'm not sure why postgres would be better than sqlite (still can't edit the data w/ vim/emacs/nano/whatever), but i could finish up that mod if we're interested. postgres has the advantage that it isn´t restricted to read only while the server is running. > > To read or write sqlite while the server is down, we need an sqlite reader, instead of a simple texteditor. > > that's a fair point. i suppose i shouldn't assume that everyone is fluent in SQL :) basic SQL isn´t that hard and minetest´s database scheme isn´t that complicated which doesn´t make it really harder than spreadsheet editing. Also if you are that far down the rabbit hole that you have to edit those entries I think its fair to assume that you can install and start the necessary tools 😉 And for JSON I would still prefer a good editor than any simple texteditor, just to easy to introduce typos or somehow mess up syntax.

Sign in to join this conversation.