mrminer reports: such lag chunk load so slow ... #2653

New Issue

yourland-report · 2022-09-11T20:32:38Z

yourland-report commented

2022-09-11 20:32:38 +00:00

mrminer reports a bug:

such lag chunk load so slow

Player position:

{
	x = 2000.1989746094,
	y = -1488.4990234375,
	z = 1500.3129882813
}

Player look:

{
	x = 0.0049828686751425,
	y = 0.99996191263199,
	z = -0.007164069917053
}

Player information:

{
	connection_uptime = 719,
	serialization_version = 29,
	patch = 0,
	formspec_version = 6,
	lang_code = "",
	state = "Active",
	version_string = "5.6.0",
	ip_version = 6,
	max_jitter = 1.4850000143051,
	min_rtt = 0.1140000000596,
	major = 5,
	minor = 6,
	max_rtt = 1.6180000305176,
	avg_rtt = 0.12099999934435,
	min_jitter = 0,
	protocol_version = 40,
	avg_jitter = 0.0040000006556511
}

Player meta:

{
	fields = {
		["signslib:pos"] = "(2235,-167,1366)",
		["stamina:exhaustion"] = "0",
		yl_commons_player_created = "1646186882",
		yl_commons_player_joined = "1662927668",
		punch_count = "5063",
		digged_nodes = "36803",
		inflicted_damage = "89806",
		died = "291",
		crafted = "6071",
		partychat = "party",
		bitten = "0",
		["stamina:poisoned"] = "no",
		jointime = "1646186882",
		placed_nodes = "12206",
		xp = "11411",
		["3d_armor_inventory"] = "return {\"3d_armor:helmet_crystal 1 17760\", \"3d_armor:chestplate_crystal 1 13260\", \"3d_armor:leggings_crystal 1 17660\", \"shields:shield_crystal 1 14060\", \"3d_armor:boots_crystal 1 16760\", \"\"}",
		hud_state = "on",
		yl_church = "return {[\"last_death_portal\"] = 1662777067, [\"last_heal\"] = 1653095205, [\"last_death\"] = {[\"x\"] = 2062, [\"y\"] = 15, [\"z\"] = 1145}}",
		arenalib_infobox_arenaID = "0",
		repellant = "0",
		yl_commons_thankyou = "14",
		played_time = "1016924",
		["stamina:level"] = "17",
		["unified_inventory:bags"] = "return {\"unified_inventory:bag_small\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\"}"
	}
}

Log identifier


[MOD] yl_report log identifier = AQkSF5NTEzDv6vjqMJ0SDNsV8yZqdvzg

Profiler save:

profile-20220911T203238.json_prettyEE

Status:

# Server: version: 5.5.1-yl | game: Minetest Game | uptime: 7d 3h 37min 46s | max lag: 3.3s | clients: mrminer, DreamNotFound, Livinstone, rabenkind, FailFactory, Davedave000, Service, DefensaSiciliana, AspireMint, polaris_, hatti, Mianmano, Empempires, Therottenpotato, LeetPeet, AliasAlreadyTaken, daniel9, ShelLuke, cypis, The4spaceconstants, drgn, Oakenshield, jackofthebean000, daydream, Mielle, Bla, pitman, flux, Gooey_Honey, Bailiff, Boot

Teleport command:

/teleport xyz 2000 -1488 1500

Compass command:

/give_compass Construction AQkSF5NTEzDv6vjqMJ0SDNsV8yZqdvzg D2691E 2000 -1488 1500

mrminer reports a bug: > such lag chunk load so slow Player position: ``` { x = 2000.1989746094, y = -1488.4990234375, z = 1500.3129882813 } ``` Player look: ``` { x = 0.0049828686751425, y = 0.99996191263199, z = -0.007164069917053 } ``` Player information: ``` { connection_uptime = 719, serialization_version = 29, patch = 0, formspec_version = 6, lang_code = "", state = "Active", version_string = "5.6.0", ip_version = 6, max_jitter = 1.4850000143051, min_rtt = 0.1140000000596, major = 5, minor = 6, max_rtt = 1.6180000305176, avg_rtt = 0.12099999934435, min_jitter = 0, protocol_version = 40, avg_jitter = 0.0040000006556511 } ``` Player meta: ``` { fields = { ["signslib:pos"] = "(2235,-167,1366)", ["stamina:exhaustion"] = "0", yl_commons_player_created = "1646186882", yl_commons_player_joined = "1662927668", punch_count = "5063", digged_nodes = "36803", inflicted_damage = "89806", died = "291", crafted = "6071", partychat = "party", bitten = "0", ["stamina:poisoned"] = "no", jointime = "1646186882", placed_nodes = "12206", xp = "11411", ["3d_armor_inventory"] = "return {\"3d_armor:helmet_crystal 1 17760\", \"3d_armor:chestplate_crystal 1 13260\", \"3d_armor:leggings_crystal 1 17660\", \"shields:shield_crystal 1 14060\", \"3d_armor:boots_crystal 1 16760\", \"\"}", hud_state = "on", yl_church = "return {[\"last_death_portal\"] = 1662777067, [\"last_heal\"] = 1653095205, [\"last_death\"] = {[\"x\"] = 2062, [\"y\"] = 15, [\"z\"] = 1145}}", arenalib_infobox_arenaID = "0", repellant = "0", yl_commons_thankyou = "14", played_time = "1016924", ["stamina:level"] = "17", ["unified_inventory:bags"] = "return {\"unified_inventory:bag_small\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\", \"unified_inventory:bag_large\"}" } } ``` Log identifier ``` [MOD] yl_report log identifier = AQkSF5NTEzDv6vjqMJ0SDNsV8yZqdvzg ``` Profiler save: ``` profile-20220911T203238.json_prettyEE ``` Status: ``` # Server: version: 5.5.1-yl | game: Minetest Game | uptime: 7d 3h 37min 46s | max lag: 3.3s | clients: mrminer, DreamNotFound, Livinstone, rabenkind, FailFactory, Davedave000, Service, DefensaSiciliana, AspireMint, polaris_, hatti, Mianmano, Empempires, Therottenpotato, LeetPeet, AliasAlreadyTaken, daniel9, ShelLuke, cypis, The4spaceconstants, drgn, Oakenshield, jackofthebean000, daydream, Mielle, Bla, pitman, flux, Gooey_Honey, Bailiff, Boot ``` Teleport command: ``` /teleport xyz 2000 -1488 1500 ``` Compass command: ``` /give_compass Construction AQkSF5NTEzDv6vjqMJ0SDNsV8yZqdvzg D2691E 2000 -1488 1500 ```

AliasAlreadyTaken was assigned by yourland-report

2022-09-11 20:32:38 +00:00

flux added the

1. kind/other

3. source/lag

labels 2022-09-11 22:51:33 +00:00

flux commented

2022-09-11 22:56:32 +00:00

from the data above,

max_lag = 3.3
max_rtt = 1.62
# players = 31

also, there were some players in the nether.

from the data above, * max_lag = 3.3 * max_rtt = 1.62 * \# players = 31 also, there were some players in the nether.

AliasAlreadyTaken commented

2022-09-12 07:45:57 +00:00

Angra suggested we use unix sockets instead of TCP sockets.

The testserver already runs on unix sockets, as far as I can tell there's no drawback. We'll add this to the main server upon next update.

#2637 (comment)

Angra suggested we use unix sockets instead of TCP sockets. The testserver already runs on unix sockets, as far as I can tell there's no drawback. We'll add this to the main server upon next update. https://gitea.your-land.de/your-land/bugtracker/issues/2637#issuecomment-26209

AliasAlreadyTaken added the

4. step/ready to QA test

label 2022-09-12 07:47:41 +00:00

AliasAlreadyTaken added this to the 1.1.116 milestone 2022-11-02 13:07:05 +00:00

flux commented

2022-11-18 21:16:33 +00:00

we're on unix sockets, but it remains to be seen whether that helps w/ the lag issue.

flux added

4. step/question

and removed

4. step/ready to QA test

labels 2022-11-18 21:16:55 +00:00

AliasAlreadyTaken commented

2022-11-23 06:49:56 +00:00

Maybe we should move the setting

server_map_save_interval = 30

back to its default 5.3 seconds. Unix sockets fares better at smaller commits.

https://github.com/minetest/minetest/blob/master/minetest.conf.example#L2884

Maybe we should move the setting ``` server_map_save_interval = 30 ``` back to its default 5.3 seconds. Unix sockets fares better at smaller commits. https://github.com/minetest/minetest/blob/master/minetest.conf.example#L2884

flux commented

2022-11-24 03:04:24 +00:00

Maybe we should move the setting
server_map_save_interval = 30
back to its default 5.3 seconds. Unix sockets fares better at smaller commits.

https://github.com/minetest/minetest/blob/master/minetest.conf.example#L2884

we can try that, but the biggest issue we want to solve viz. lag and sockets is read issues, right? when there's more than 30 players on the server, it can take ages to get the server to send new mapblocks to my client, even though chat, inventory interaction, and entity interaction work well enough.

> Maybe we should move the setting > > ``` > server_map_save_interval = 30 > ``` > > back to its default 5.3 seconds. Unix sockets fares better at smaller commits. > > https://github.com/minetest/minetest/blob/master/minetest.conf.example#L2884 we can try that, but the biggest issue we want to solve viz. lag and sockets is *read* issues, right? when there's more than 30 players on the server, it can take ages to get the server to send new mapblocks to my client, even though chat, inventory interaction, and entity interaction work well enough.

flux commented

2022-11-24 03:05:21 +00:00

when there's more than 30 players on the server

it seems to be better than before, but i think we've only pushed it out to perhaps 34 players instead of 30?

> when there's more than 30 players on the server it seems to be better than before, but i think we've only pushed it out to perhaps 34 players instead of 30?

AliasAlreadyTaken commented

2022-11-24 16:55:02 +00:00

The setting is currently on 5.3 and we have the same lags again we had on the old server. There is something fundamentally wrong with my setup, other servers certainly do not feel this lag.

I will set it back to 30 seconds on next restart.

Next candidate is change

server_unload_unused_data_timeout = 1800

to

server_unload_unused_data_timeout = 900

although I don't see how that would help read faster from database. It only means "less stuff in memory". Less stuff in memory may speed up things, but also cause the database forget stuff faster, which is bad.

Unless MT makes a full dump of all what is has in memory to the DB, this shouldn't change much on the write side of things and nothing at all on the read side of things.

I will also move

biome_lib_queue_ratio = -3

to

biome_lib_queue_ratio = 100

to speed up biome lib processing and maybe cause it not to have so much of a backlog. This however may cause lag again, when someone goes exploring and biome lib wants to process everything at once.

The ultimate goal is to either get rid of biome lib or move it to the async env or a new regrow mechanic.

The setting is currently on 5.3 and we have the same lags again we had on the old server. There is something fundamentally wrong with my setup, other servers certainly do not feel this lag. I will set it back to 30 seconds on next restart. Next candidate is change ``` server_unload_unused_data_timeout = 1800 ``` to ``` server_unload_unused_data_timeout = 900 ``` although I don't see how that would help read faster from database. It only means "less stuff in memory". Less stuff in memory may speed up things, but also cause the database forget stuff faster, which is bad. Unless MT makes a full dump of all what is has in memory to the DB, this shouldn't change much on the write side of things and nothing at all on the read side of things. I will also move ``` biome_lib_queue_ratio = -3 ``` to ``` biome_lib_queue_ratio = 100 ``` to speed up biome lib processing and maybe cause it not to have so much of a backlog. This however may cause lag again, when someone goes exploring and biome lib wants to process everything at once. The ultimate goal is to either get rid of biome lib or move it to the async env or a new regrow mechanic.

flux commented

2022-11-24 23:13:38 +00:00

There is something fundamentally wrong with my setup, other servers certainly do not feel this lag.

there are passingly few servers which regularly have over 30 connected people simultaneously, and probably none that also have hundreds of active entities and machines on top of that.

there was another issue where you were able to post a record of the sizes of various network queues, and we were able to figure out that a major issue was smartshops creating temporary detached inventories. possilby that data would be helpful again?

> There is something fundamentally wrong with my setup, other servers certainly do not feel this lag. there are passingly few servers which regularly have over 30 connected people simultaneously, and probably none that also have hundreds of active entities and machines on top of that. there was another issue where you were able to post a record of the sizes of various network queues, and we were able to figure out that a major issue was smartshops creating temporary detached inventories. possilby that data would be helpful again?

AliasAlreadyTaken removed this from the 1.1.116 milestone 2023-01-13 00:06:31 +00:00

AliasAlreadyTaken commented

2023-01-13 00:07:05 +00:00

None of my changes so far had any significant impact on lag.

flux commented

2023-01-13 23:45:09 +00:00

we should try #3299 and see if it helps w/ the map load lag when there's 30-ish or more players.

for the server step lag, the biggest problem, so far as i can tell, is mobs in the nether.

we should try #3299 and see if it helps w/ the map load lag when there's 30-ish or more players. for the server step lag, the biggest problem, so far as i can tell, is mobs in the nether.

AliasAlreadyTaken commented

2024-02-29 22:24:45 +00:00

I believe this was solved by your-land/administration#183

AliasAlreadyTaken closed this issue

2024-02-29 22:24:45 +00:00

AliasAlreadyTaken added

5. result/fixed

and removed

4. step/question

labels 2024-02-29 22:24:56 +00:00

Sign in to join this conversation.