Administrator reports: Add a LBM to detect nether blo ... #5507

New Issue

yourland-report · 2023-11-17T18:30:32Z

yourland-report commented

2023-11-17 18:30:32 +00:00

Administrator reports a bug:

Add a LBM to detect nether blocks in the deep. Let the LBM run once which creates a log entry of type warning with mapblock position and detected itemstring(s) if any nether-y blocks are detected in the deep, read below 0.

Player position:

{
	x = 2846.125,
	y = -25739.845703125,
	z = 388.33099365234
}

Player look:

{
	x = -0.06808003038168,
	y = -0.99539619684219,
	z = 0.0674649477005
}

Player information:

{
	min_rtt = 0.014000000432134,
	avg_rtt = 0.014999999664724,
	max_jitter = 0.1089999973774,
	avg_jitter = 0.0010000010952353,
	connection_uptime = 1913,
	serialization_version = 29,
	patch = 0,
	formspec_version = 7,
	minor = 8,
	state = "Active",
	version_string = "5.8.0-dev-aa912e90a",
	min_jitter = 0,
	max_rtt = 0.20700000226498,
	major = 5,
	protocol_version = 42,
	lang_code = "de",
	ip_version = 6
}

Player meta:

{
	fields = {
		digged_nodes = "75472",
		placed_nodes = "125708",
		died = "185",
		crafted = "175",
		["3d_armor_inventory"] = "return {\"3d_armor:boots_crystal\", \"\", \"\", \"\", \"\", \"\"}",
		yl_church = "return {[\"last_death\"] = {[\"y\"] = 54, [\"x\"] = -3844, [\"z\"] = -5201}}",
		hud_state = "on",
		repellant = "0",
		yl_commons_player_joined = "1700243923",
		inflicted_damage = "16440",
		["ethereal:fly_timer"] = "-99",
		["signslib:pos"] = "(2098,15,1304)",
		lagometer = "1",
		["unified_inventory:bags"] = "return {\"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\"}",
		bitten = "0",
		["stamina:level"] = "0",
		["stamina:poisoned"] = "no",
		["sethome:home"] = "(2218.833984375,40.9880027771,981.63397216797)",
		["stamina:exhaustion"] = "78",
		arenalib_infobox_arenaID = "0",
		xp = "0",
		punch_count = "596",
		yl_commons_thankyou = "14",
		played_time = "3528383"
	}
}

Log identifier


[MOD] yl_report log identifier = 1LyuBpmtXDxfEm8rKaa6ua4j5X3LJXGL

Profiler save:

profile-20231117T183032.json_prettyEE

Status:

# Server: version: 5.7.0-yl-thx-tmm | game: Minetest Game | uptime: 4d 8h 6min 18s | max lag: 0.337s | clients (25/52): Administrator, ait0r7, AliasAlreadyTaken, betzi, Bishiro, bizon, Boot, Cezar, daydream, DragonWrangler1, ElManu, ffelder, flux, formations52, Heron, jackofthebean000, juli10var, Keya, Lichberry, pavlo98, Pif, poppyasdan, Reinsch, rheo, Service

Teleport command:

/teleport xyz 2846 -25740 388

Compass command:

/give_compass Construction 1LyuBpmtXDxfEm8rKaa6ua4j5X3LJXGL D2691E 2846 -25740 388

Administrator reports a bug: > Add a LBM to detect nether blocks in the deep. Let the LBM run once which creates a log entry of type warning with mapblock position and detected itemstring(s) if any nether-y blocks are detected in the deep, read below 0. Player position: ``` { x = 2846.125, y = -25739.845703125, z = 388.33099365234 } ``` Player look: ``` { x = -0.06808003038168, y = -0.99539619684219, z = 0.0674649477005 } ``` Player information: ``` { min_rtt = 0.014000000432134, avg_rtt = 0.014999999664724, max_jitter = 0.1089999973774, avg_jitter = 0.0010000010952353, connection_uptime = 1913, serialization_version = 29, patch = 0, formspec_version = 7, minor = 8, state = "Active", version_string = "5.8.0-dev-aa912e90a", min_jitter = 0, max_rtt = 0.20700000226498, major = 5, protocol_version = 42, lang_code = "de", ip_version = 6 } ``` Player meta: ``` { fields = { digged_nodes = "75472", placed_nodes = "125708", died = "185", crafted = "175", ["3d_armor_inventory"] = "return {\"3d_armor:boots_crystal\", \"\", \"\", \"\", \"\", \"\"}", yl_church = "return {[\"last_death\"] = {[\"y\"] = 54, [\"x\"] = -3844, [\"z\"] = -5201}}", hud_state = "on", repellant = "0", yl_commons_player_joined = "1700243923", inflicted_damage = "16440", ["ethereal:fly_timer"] = "-99", ["signslib:pos"] = "(2098,15,1304)", lagometer = "1", ["unified_inventory:bags"] = "return {\"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\"}", bitten = "0", ["stamina:level"] = "0", ["stamina:poisoned"] = "no", ["sethome:home"] = "(2218.833984375,40.9880027771,981.63397216797)", ["stamina:exhaustion"] = "78", arenalib_infobox_arenaID = "0", xp = "0", punch_count = "596", yl_commons_thankyou = "14", played_time = "3528383" } } ``` Log identifier ``` [MOD] yl_report log identifier = 1LyuBpmtXDxfEm8rKaa6ua4j5X3LJXGL ``` Profiler save: ``` profile-20231117T183032.json_prettyEE ``` Status: ``` # Server: version: 5.7.0-yl-thx-tmm | game: Minetest Game | uptime: 4d 8h 6min 18s | max lag: 0.337s | clients (25/52): Administrator, ait0r7, AliasAlreadyTaken, betzi, Bishiro, bizon, Boot, Cezar, daydream, DragonWrangler1, ElManu, ffelder, flux, formations52, Heron, jackofthebean000, juli10var, Keya, Lichberry, pavlo98, Pif, poppyasdan, Reinsch, rheo, Service ``` Teleport command: ``` /teleport xyz 2846 -25740 388 ``` Compass command: ``` /give_compass Construction 1LyuBpmtXDxfEm8rKaa6ua4j5X3LJXGL D2691E 2846 -25740 388 ```

AliasAlreadyTaken was assigned by yourland-report

2023-11-17 18:30:32 +00:00

AliasAlreadyTaken added the

1. kind/enhancement

2. prio/good first issue

labels 2023-11-17 20:29:01 +00:00

sixer commented

2023-11-18 14:22:52 +00:00

This may be prone to lot of false alarms if someone uses nether blocks in construction of underground base (like nether basalt, favored for its explosion-proof qualities) and then visits it often.

If it is supposed to catch existing blocks, wouldn't a one-off script/program to scan current blocks in a database perform a better job?

This may be prone to lot of false alarms if someone uses nether blocks in construction of underground base (like nether basalt, favored for its explosion-proof qualities) and then visits it often. If it is supposed to catch existing blocks, wouldn't a one-off script/program to scan current blocks in a database perform a better job?

AliasAlreadyTaken commented

2023-11-19 00:52:31 +00:00

This LBM is meant to catch wrongly generated nether only and is meant to run on each mapblock only once.

There may be false positives, but those are easy to detect.

I wish we had a way to look into the database direct and perform operations in the background or even on the testserver, where the generated area is present as well, but currently there is no such thing. If you know how to make such a script, go ahead, we'd be pretty happy :)

Especailly if it can also detect entities as well.

The database is on psotgres and around 175Gb in a backup state, so it will take a LOT of tiem to run through the whole of it.

This LBM is meant to catch wrongly generated nether only and is meant to run on each mapblock only once. There may be false positives, but those are easy to detect. I wish we had a way to look into the database direct and perform operations in the background or even on the testserver, where the generated area is present as well, but currently there is no such thing. If you know how to make such a script, go ahead, we'd be pretty happy :) Especailly if it can also detect entities as well. The database is on psotgres and around 175Gb in a backup state, so it will take a LOT of tiem to run through the whole of it.

flux commented

2023-11-19 20:56:40 +00:00

i've got an old (1 year?) copy of the database that's more than new enough to have all the relevant blocks in it, and a script to process the database and extract the nodenames, which i could adjust to identify mapblocks with nether nodes in them below -25000. however, if i remember right, it takes over a week to run, and my laptop doesn't understand batteries anymore, so that's a problem...

AliasAlreadyTaken commented

2023-11-19 21:11:06 +00:00

I have a dev system on a nice and fast machine. Could you teach me to use the script or is that rocket science?

sixer commented

2023-11-19 21:44:13 +00:00

True, you can have once-per-each-block ABM, but areas far away may still elude detection for long time.

If you know how to make such a script

I can use my java program as a base and rip some unnecessary functionality from it (like counting inventory in drawers within given areas, etc ...) and add some mode that will hunt for blocks with defined nodes. Connection is now hardcoded to sqlite file, but adding Postgresql drivers is fairly simple.

Processing 4.2M blocks (2.3 GB of raw compressed blocks) took it 95 seconds .... so I'm guessing the 175G world should be scannable within approx 2 hours (assuming fast enough network between the machine running the tool and the database server). Possibly faster if I add some multi-threading support. Some optimizations (like not examining blocks above ground) may make it even faster. If 2 hours is less than LOT, then that could be viable.

At the moment it does not support entities, but I may need to add that in the future to deal with remnants of petz infestation on my private server ... so in future maybe :)

True, you can have once-per-each-block ABM, but areas far away may still elude detection for long time. > If you know how to make such a script I can use my java program as a base and rip some unnecessary functionality from it (like counting inventory in drawers within given areas, etc ...) and add some mode that will hunt for blocks with defined nodes. Connection is now hardcoded to sqlite file, but adding Postgresql drivers is fairly simple. Processing 4.2M blocks (2.3 GB of raw compressed blocks) took it 95 seconds .... so I'm guessing the 175G world should be scannable within approx 2 hours (assuming fast enough network between the machine running the tool and the database server). Possibly faster if I add some multi-threading support. Some optimizations (like not examining blocks above ground) may make it even faster. If 2 hours is less than LOT, then that could be viable. At the moment it does not support entities, but I may need to add that in the future to deal with remnants of petz infestation on my private server ... so in future maybe :)

flux commented

2023-11-19 21:44:55 +00:00

I have a dev system on a nice and fast machine. Could you teach me to use the script or is that rocket science?

let me adapt it to find nether nodes and then send it your way. it's not too hard to use.

> I have a dev system on a nice and fast machine. Could you teach me to use the script or is that rocket science? let me adapt it to find nether nodes and then send it your way. it's not too hard to use.

flux commented

2023-11-19 23:18:48 +00:00

this should work. tested against a small sqlite database locally.

usage:

first, make sure you're running this against a database that's been fully updated to the 5.7.0 format - the code doesn't know how to read the older format.

then, make sure you've got progressbar2, psycopg2, and pyzstd in your python environment (use pip or whatever your favorite tool is).

then run:

python3 find_blocks_with_nether_nodes.py --pg_connection="host=127.0.0.1 user=minetest password=mypassword dbname=minetest"

this will produce a file called "output" which contains a list of pairs of coordinates - the first are the "block position", and the second are the coordinates of the "small" corner of the block.

this should work. tested against a small sqlite database locally. usage: first, make sure you're running this against a database that's been fully updated to the 5.7.0 format - the code doesn't know how to read the older format. then, make sure you've got progressbar2, psycopg2, and pyzstd in your python environment (use pip or whatever your favorite tool is). then run: ```bash python3 find_blocks_with_nether_nodes.py --pg_connection="host=127.0.0.1 user=minetest password=mypassword dbname=minetest" ``` this will produce a file called "output" which contains a list of pairs of coordinates - the first are the "block position", and the second are the coordinates of the "small" corner of the block.

find_blocks_with_nether_nodes.py.txt

7.9 KiB

flux commented

2023-11-19 23:47:15 +00:00

Processing 4.2M blocks (2.3 GB of raw compressed blocks) took it 95 seconds .... so I'm guessing the 175G world should be scannable within approx 2 hours (assuming fast enough network between the machine running the tool and the database server).

yeah, i think i was remembering the wrong step in the process taking over a week. that was updating the whole database to the 5.7.0 map format. whatever minetest is doing seems fairly inefficient.

> Processing 4.2M blocks (2.3 GB of raw compressed blocks) took it 95 seconds .... so I'm guessing the 175G world should be scannable within approx 2 hours (assuming fast enough network between the machine running the tool and the database server). yeah, i think i was remembering the wrong step in the process taking over a week. that was updating the whole database to the 5.7.0 map format. whatever minetest is doing seems fairly inefficient.

AliasAlreadyTaken commented

2023-11-20 00:07:31 +00:00

I will probably also have to update the MT database first. I can't do that on a live and running server, right?

Since this is a tool every MT admin of a major server may need at one time or another, maybe make it a repo and publish it?

I will probably also have to update the MT database first. I can't do that on a live and running server, right? Since this is a tool every MT admin of a major server may need at one time or another, maybe make it a repo and publish it?

sixer commented

2023-11-20 07:30:26 +00:00

yeah, i think i was remembering the wrong step in the process taking over a week. that was updating the whole database to the 5.7.0 map format. whatever minetest is doing seems fairly inefficient.

Due to the map size (175GB of blocks) and usual block compression ratio, converting from old format to new one means decompressing approximately 5 terabytes of data (that is fast) and re-compressing it (that is not that fast and is probably why it has taken a week to do). The compression cannot be made significantly faster, although the block converting can be easily parallelized. More computers/cpu cores -> less time.

> yeah, i think i was remembering the wrong step in the process taking over a week. that was updating the whole database to the 5.7.0 map format. whatever minetest is doing seems fairly inefficient. Due to the map size (175GB of blocks) and usual block compression ratio, converting from old format to new one means decompressing approximately 5 terabytes of data (that is fast) and re-compressing it (that is not that fast and is probably why it has taken a week to do). The compression cannot be made significantly faster, although the block converting can be easily parallelized. More computers/cpu cores -> less time.

AliasAlreadyTaken commented

2023-11-21 01:36:46 +00:00

During "recompresssing" I have a lot of those:

2023-11-21 02:26:51: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(12,12,9): Ignoring.
2023-11-21 02:26:51: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(9,10,14): Ignoring.
2023-11-21 02:26:56: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(14,5,11): Ignoring.
2023-11-21 02:27:08: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(4,6,11): Ignoring.

During "recompresssing" I have a lot of those: ``` 2023-11-21 02:26:51: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(12,12,9): Ignoring. 2023-11-21 02:26:51: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(9,10,14): Ignoring. 2023-11-21 02:26:56: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(14,5,11): Ignoring. 2023-11-21 02:27:08: WARNING[Main]: NodeTimerList::deSerialize(): invalid data at position(4,6,11): Ignoring. ```

flux commented

2023-11-21 02:16:27 +00:00

I will probably also have to update the MT database first. I can't do that on a live and running server, right?

Since this is a tool every MT admin of a major server may need at one time or another, maybe make it a repo and publish it?

the script i sent you is based off the script that i publish w/ my moreblocks fork, to generate a whitelist of used nodes in order to reduce node count.

but i agree that some admin's needs might be served by the script. but i'm not sure how to publish it. maybe the minetest forums? i hate the forums, every page there takes 30 seconds to load for me, and the search results are miserable. but some people do use it...

> I will probably also have to update the MT database first. I can't do that on a live and running server, right? > > Since this is a tool every MT admin of a major server may need at one time or another, maybe make it a repo and publish it? the script i sent you is based off the script that i publish w/ my moreblocks fork, to generate a whitelist of used nodes in order to reduce node count. but i agree that some admin's needs might be served by the script. but i'm not sure how to publish it. maybe the minetest forums? i hate the forums, every page there takes 30 seconds to load for me, and the search results are miserable. but some people do use it...

AliasAlreadyTaken commented

2023-11-21 03:40:12 +00:00

This is the recompress feature: 61db32beee/src/main.cpp (L1234)

Even on my numbercruncher and an older database of most likely 160GB, it would run ~30 hours and it needs the server stopped at that time.

Suggestion by two smart people on discord: Make a mechanic to run through all emerged mapblocks, drag them into memory and store them again.
Other option is to cut the database in pieces, migrate them separately and then restore them back
We could also pester MT devs about it, but I doubt there's much chance

The issue is not timecritical per se, but the longer we wait, the larger the database grows. Currently we're at 185GB each export and 414 856 973 mapblocks stored.

This is the recompress feature: https://github.com/minetest/minetest/blob/61db32beee936a5264ef17785d8efaa7cc14bda4/src/main.cpp#L1234 Even on my numbercruncher and an older database of most likely 160GB, it would run ~30 hours and it needs the server stopped at that time. 1. Suggestion by two smart people on discord: Make a mechanic to run through all emerged mapblocks, drag them into memory and store them again. 2. Other option is to cut the database in pieces, migrate them separately and then restore them back 3. We could also pester MT devs about it, but I doubt there's much chance The issue is not timecritical per se, but the longer we wait, the larger the database grows. Currently we're at 185GB each export and 414 856 973 mapblocks stored.

sixer commented

2023-11-21 18:50:02 +00:00

but i'm not sure how to publish it. maybe the minetest forums? i hate the forums, every page there takes 30 seconds to load for me, and the search results are miserable. but some people do use it...

github/gitlab or something like that?

could be some lua code running on server adding some garbage entity to a block and immediately removing it again. This will trigger writing "modified" blocks to disk. As server is already running 5.7, part of the blocks would be already converted, I guess?
Block version is first byte of the block, so should be fairly easy and fast to extract from the running server and see what block has what version.
Would be hard to do with regard to synchronization (block modified by both people on server and the recompress)
Make a change in MT branch and if it will not get adopted in mainline in the end ... well, not your problem :)

> but i'm not sure how to publish it. maybe the minetest forums? i hate the forums, every page there takes 30 seconds to load for me, and the search results are miserable. but some people do use it... github/gitlab or something like that? 1. could be some lua code running on server adding some garbage entity to a block and immediately removing it again. This will trigger writing "modified" blocks to disk. As server is already running 5.7, part of the blocks would be already converted, I guess? Block version is first byte of the block, so should be fairly easy and fast to extract from the running server and see what block has what version. 2. Would be hard to do with regard to synchronization (block modified by both people on server and the recompress) 3. Make a change in MT branch and if it will not get adopted in mainline in the end ... well, not your problem :)

AliasAlreadyTaken commented

2023-11-22 00:53:08 +00:00

More info on the recompress problem:

Amanda (MT discord): looking at the code for recompress_map_database it looks like it just indiscriminately iterates over every single block in the database, deserialises it and then serialises it again using the latest version. if you'd want to modify it to only deserialise and then serialise blocks with an older block version you could put if (ver == 29) continue; above the deserialisation line which should skip any blocks that are on the latest version and hopefully speed up the process (though I haven't tested if this actually works)

{
	MapBlock mb(nullptr, v3s16(0,0,0), &server);
	u8 ver = readU8(iss);

	// if (ver == 29) continue;

	mb.deSerialize(iss, ver, true);

	oss.str("");
	oss.clear();
	writeU8(oss, serialize_as_ver);
	mb.serialize(oss, serialize_as_ver, true, -1);
}

This way it will only touch the "wrong" version ones, not recompress everything.

How do we iterate over the datase? Could we maybe do it in parts? Is the order changed?

Here's the iterator: 61db32beee/src/database/database-postgresql.cpp (L300)

We need to test what exactly happens there and which order they take, then we can maybe cheat it and manually start and stop it for smaller parts.

Yes, the order may change upon restores. That's why we can try on the testserver dump to see how long it may take and then guess towards the live dump after we migrated to the new machine.

More info on the recompress problem: 1. Amanda (MT discord): looking at the code for recompress_map_database it looks like it just indiscriminately iterates over every single block in the database, deserialises it and then serialises it again using the latest version. if you'd want to modify it to only deserialise and then serialise blocks with an older block version you could put if (ver == 29) continue; above the deserialisation line which should skip any blocks that are on the latest version and hopefully speed up the process (though I haven't tested if this actually works) ```cpp { MapBlock mb(nullptr, v3s16(0,0,0), &server); u8 ver = readU8(iss); // if (ver == 29) continue; mb.deSerialize(iss, ver, true); oss.str(""); oss.clear(); writeU8(oss, serialize_as_ver); mb.serialize(oss, serialize_as_ver, true, -1); } ``` This way it will only touch the "wrong" version ones, not recompress everything. 2. How do we iterate over the datase? Could we maybe do it in parts? Is the order changed? Here's the iterator: https://github.com/minetest/minetest/blob/61db32beee936a5264ef17785d8efaa7cc14bda4/src/database/database-postgresql.cpp#L300 We need to test what exactly happens there and which order they take, then we can maybe cheat it and manually start and stop it for smaller parts. Yes, the order may change upon restores. That's why we can try on the testserver dump to see how long it may take and then guess towards the live dump after we migrated to the new machine.

flux commented

2023-11-22 01:33:02 +00:00

i hope to get my server w/ my copy of the map back online soon, i can probably run the script myself then and post the results here.

AliasAlreadyTaken commented

2023-11-22 22:06:00 +00:00

Here's a rust project to read the world: https://github.com/UgnilJoZ/rust-minetestworld

I added it to your-land/administration#140

Here's a rust project to read the world: https://github.com/UgnilJoZ/rust-minetestworld I added it to your-land/administration#140

flux commented

2023-11-25 22:41:46 +00:00

35483 mapblocks matched the criteria. processing took approximately 2 hours 50 minutes.

now someone has to go visit all of those...

probably the next step should be to apply some sort of clustering algorithm

EDIT: oh wait, nether basalt spawns at the bottom of the world now doesn't it. hm.

35483 mapblocks matched the criteria. processing took approximately 2 hours 50 minutes. now someone has to go visit all of those... probably the next step should be to apply some sort of clustering algorithm EDIT: oh wait, nether basalt spawns at the bottom of the world now doesn't it. hm.

output.txt

1.2 MiB

AliasAlreadyTaken commented

2023-11-26 00:32:18 +00:00

The bottom of the world has bedrock, nether basalt is only obtainable from the nether.

35k? Wow, that's a lot.

The bottom of the world has bedrock, nether basalt is only obtainable from the nether. 35k? Wow, that's a lot.

flux commented

2023-11-26 20:15:43 +00:00

The bottom of the world has bedrock, nether basalt is only obtainable from the nether.

whoops, that's what i meant. bedrock. i should probably run the analysis again while ignoring bedrock.

> The bottom of the world has bedrock, nether basalt is only obtainable from the nether. whoops, that's what i meant. bedrock. i should probably run the analysis again while ignoring bedrock.

sixer commented

2023-11-27 00:38:52 +00:00

The bottom of the world has bedrock, nether basalt is only obtainable from the nether.

whoops, that's what i meant. bedrock. i should probably run the analysis again while ignoring bedrock.

Or just exclude the bottom:
grep -v ', -1932, ' <output.txt |wc -l
23962

> > The bottom of the world has bedrock, nether basalt is only obtainable from the nether. > > whoops, that's what i meant. bedrock. i should probably run the analysis again while ignoring bedrock. Or just exclude the bottom: `grep -v ', -1932, ' <output.txt |wc -l` 23962

flux commented

2023-11-27 06:21:03 +00:00

just exclude the bottom

i did that but didn't trust it.

attached is the result of ignoring bedrock nodes explicitly. 24008 matches. probably around 50% of those are false positives. i still think i need to run some clustering algorithms on the results to generate a more analyzable result

> just exclude the bottom i did that but didn't trust it. attached is the result of ignoring bedrock nodes explicitly. 24008 matches. probably around 50% of those are false positives. i still think i need to run some clustering algorithms on the results to generate a more analyzable result

output.txt

845 KiB

Sign in to join this conversation.