Split log in debug and privacy #3135

New Issue

AliasAlreadyTaken · 2022-11-25T17:37:23Z

AliasAlreadyTaken commented

2022-11-25 17:37:23 +00:00

Sometimes staff needs access to the log to look at errors. Unfortunately in the log there's not only errors, but also chat, messages, books, anything auser has a right to be kept private. This information must not be looked at by staff when trying to find an error.

We need a way to either split the logs in a debug and a privacy part or any other method to supply staff with debug, but not pirvacy related parts of the log.

May also happen on the system side of things, like a shellscript that greps ERROR or removes CHAT and DM and similar lines with certain privacy-related identifiers

Sometimes staff needs access to the log to look at errors. Unfortunately in the log there's not only errors, but also chat, messages, books, anything auser has a right to be kept private. This information must not be looked at by staff when trying to find an error. We need a way to either split the logs in a debug and a privacy part or any other method to supply staff with debug, but not pirvacy related parts of the log. May also happen on the system side of things, like a shellscript that greps ERROR or removes CHAT and DM and similar lines with certain privacy-related identifiers

AliasAlreadyTaken added the

4. step/discussion

2. prio/controversial

1. kind/enhancement

labels 2022-11-25 17:37:52 +00:00

flux commented

2022-11-26 00:07:07 +00:00

Sometimes staff needs access to the log to look at errors. Unfortunately in the log there's not only errors, but also chat, messages, books, anything auser has a right to be kept private. This information must not be looked at by staff when trying to find an error.

We need a way to either split the logs in a debug and a privacy part or any other method to supply staff with debug, but not pirvacy related parts of the log.

May also happen on the system side of things, like a shellscript that greps ERROR or removes CHAT and DM and similar lines with certain privacy-related identifiers

i wish there was an easy way to do this, but the engine generates both logs that need to be kept private, and ones that shouldn't be. providing a solution at that level is likely to be contentious, and i'm not sure what a good solution even looks like myself.

probably the best ad-hoc way to handle this is to whitelist log lines with patterns instead of relying only on a necessarily incomplete blacklist.

someone must have a good framework for dealing w/ this issue for other games/contexts.

> Sometimes staff needs access to the log to look at errors. Unfortunately in the log there's not only errors, but also chat, messages, books, anything auser has a right to be kept private. This information must not be looked at by staff when trying to find an error. > > We need a way to either split the logs in a debug and a privacy part or any other method to supply staff with debug, but not pirvacy related parts of the log. > > May also happen on the system side of things, like a shellscript that greps ERROR or removes CHAT and DM and similar lines with certain privacy-related identifiers i wish there was an easy way to do this, but the engine generates both logs that need to be kept private, and ones that shouldn't be. providing a solution at that level is likely to be contentious, and i'm not sure what a good solution even looks like myself. probably the best ad-hoc way to handle this is to *whitelist* log lines with patterns instead of relying only on a necessarily incomplete blacklist. someone must have a good framework for dealing w/ this issue for other games/contexts.

Ravise commented

2022-12-01 15:49:39 +00:00

someone must have a good framework for dealing w/ this issue for other games/contexts.

Usual solution for privacy-related issues in logging is to have dedicated loglevel that is allowed to log secret data. When a log is to be handed over while protecting privacy, according log levels are dropped. When a rich logging toolchain is used, it even allows to set certain logger's level above privacy level and at all dropping troublesome lines in some area of the program while allowing others to log trace.

logger.info("Simon joins the game.")
logger.debug("Simon talks.")
logger.trace("Simon says: ...")

I'm not aware about any rich logging toolchains for minetest mods, and I'm not sure if minetest core's logger is rich as well.

Minetest's lua currently provides only minetest.log([level,] text) call, which is half-rich in my eyes: it lacks dedicated logging functions (+ current log levels are a bit confusing to me) and there is no built-in logger discrimination (other than prefixing text with mod's name).

I found lua library https://lunarmodules.github.io/lualogging/, which seems promissing, but using that package would only be solution A) for mods, B) that will integrate the library.

I believe that a proper clean solution would be to include rich logging toolchain into minetest core, and it's lua, so it'll be offered to mods by default. That will, of course, not solve the problem in hand any time soon.

Can you take a guess on how split is the problem between mods and core?

> someone must have a good framework for dealing w/ this issue for other games/contexts. Usual solution for privacy-related issues in logging is to have dedicated loglevel that is allowed to log secret data. When a log is to be handed over while protecting privacy, according log levels are dropped. When a rich logging toolchain is used, it even allows to set certain logger's level above privacy level and at all dropping troublesome lines in some area of the program while allowing others to log trace. ``` logger.info("Simon joins the game.") logger.debug("Simon talks.") logger.trace("Simon says: ...") ``` I'm not aware about any rich logging toolchains for minetest mods, and I'm not sure if minetest core's logger is rich as well. Minetest's lua currently provides only `minetest.log([level,] text)` call, which is half-rich in my eyes: it lacks dedicated logging functions (+ current log levels are a bit confusing to me) and there is no built-in logger discrimination (other than prefixing text with mod's name). I found lua library https://lunarmodules.github.io/lualogging/, which seems promissing, but using that package would only be solution A) for mods, B) that will integrate the library. I believe that a proper clean solution would be to include rich logging toolchain into minetest core, and it's lua, so it'll be offered to mods by default. That will, of course, not solve the problem in hand any time soon. Can you take a guess on how split is the problem between mods and core?

flux commented

2022-12-01 17:00:44 +00:00

current log levels are a bit confusing to me

the current log levels are a simple hierarchy. NONE > ERROR > WARNING > ACTION > INFO > VERBOSE > TRACE > MAX. NONE is always logged, MAX isn't a proper log level, but conveys the idea that "everything should be logged".

what we need are some additional tags that don't entirely fit into the above hierarchy in a meaningful way:

"personal", which would be used to log what players say in chat/dms, write in books, etc.
"pii" - "personally identifiable information" - which would include IP #s.

> current log levels are a bit confusing to me the current log levels are a simple hierarchy. NONE > ERROR > WARNING > ACTION > INFO > VERBOSE > TRACE > MAX. NONE is always logged, MAX isn't a proper log level, but conveys the idea that "everything should be logged". what we need are some additional tags that don't entirely fit into the above hierarchy in a meaningful way: * "personal", which would be used to log what players say in chat/dms, write in books, etc. * "pii" - "[personally identifiable information](https://en.wikipedia.org/wiki/Personal_data)" - which would include IP #s.

Sign in to join this conversation.