Unnecessary network traffic as potential source of lag in Haven #5994
Labels
No Label
1. kind/balancing
1. kind/breaking
1. kind/bug
1. kind/construction
1. kind/documentation
1. kind/enhancement
1. kind/griefing
1. kind/invalid
1. kind/meme
1. kind/node limit
1. kind/other
1. kind/protocol
2. prio/controversial
2. prio/critical
2. prio/elevated
2. prio/good first issue
2. prio/interesting
2. prio/low
3. source/art
3. source/client
3. source/engine
3. source/ingame
3. source/integration
3. source/lag
3. source/license
3. source/mod upstream
3. source/unknown
3. source/website
4. step/approved
4. step/at work
4. step/blocked
4. step/discussion
4. step/help wanted
4. step/needs confirmation
4. step/partially fixed
4. step/question
4. step/ready to deploy
4. step/ready to QA test
4. step/want approval
5. result/cannot reproduce
5. result/duplicate
5. result/fixed
5. result/maybe
5. result/wontfix
ugh/petz
ugh/QA main
ugh/QA NOK
ugh/QA OK
No Milestone
No project
No Assignees
6 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: your-land/bugtracker#5994
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Many people are having problems when spawning at Haven. Some mobile devices timeout. Some player's desktops "can't handle it" too.
Sometimes I have similar problems while on a laptop + wifi connection: Haven mapblocks load slowly, opening chests on my plot takes 20+ seconds, harvesting crops in my basement with a scythe makes them stick around for a while, etc.
One strange thing I noticed while comparing lag on my wired PC VS a laptop on wifi: chat messages sent from the laptop would instantly appear on the desktop, but echo message (basically showing what I sent in chat) - would appear many seconds later.
Maybe this suggests that router (or something?) gets overloaded with incoming packets?
I filtered minetest traffic with wireshark and here's the plot:
Seems like there may be some spammy mods working at spam. Trying to look into dumps, I often see strings:
!abritorch_spark.png^[multiply:red(torches near arena?)yl_speak_up_mask_cape.png^[combine:32x64:56,20=blank.png... there's a giant stringpetz_sleep_particle.pngnether_particle.png^[colorize:#800080:alphaThe giant spikes are "signs updates". They happen all at once and send massive texture combinations for each letter of the text. They come all at once in groups of 500-600 packets (see numbers "200/675" etc) :
To try this out yourself, install wireshark, then get minetest.lua from minetest sources, put it into
C:\Program Files\Wireshark\plugins, and pressctrl+shift+Lto reload the plugins.You can see your graph in
statistics/IO Graphs. To filter minetest-specific packets enter(minetest) && !(minetest.control.type == 0)into the filter field.I won't go into further detail how to use wireshark, but if anyone needs more instructions, I may expand this.
Also note that the graph is
bytes/second, but if I set it topackets/secondthe shape stays the same - basically, packets are generally small, but the quantity differs.Lag in Haven has been one of the biggest issues I've had while playing on the server, and I swear there was already a bug report about this very issue.
A lot of people I remember chalked it up to the plots in Haven having tons and tons of signs.
i've run similar analysis before. yes. signs, itemframes, and smartshops are the biggest contributors to this. i'm pretty sure we've already got an open issue.
Sorry, I was sure there must be an open issue, but decided to dump this here before I forgot :p
Seems like
abritorch_spark.pngspam comes from doingminetest.add_particlespawnerinside of ABM with 1 second intervals...This is probably bad? Spamming particle generators for every torch every second?
(just go look at how many torches there are at the arena)
Also my attempt at reducing signs load on the client (cpu/network vs memory tradeoff):
https://gitea.your-land.de/whosit/waypoint_compass/wiki/Haven-texture-caching
Found the original issue:
#5234
In your next wireshark session keep an eye on the "leads" mod 😉
Has been a while and I hope they reduced it but was also pretty spammy in my last session 😆
I wonder how MT decides to send updates about entities?
Why NPCs update textures constantly? Ok, NPCs are "animated" - they rotate and stuff, maybe bones can rotate. But why texture names get sent constantly?
Constant updates would need to be sent for leads attached to mobs (this is where SSCSM would save the day), but do static leads also spam?
Just looked it up that was back in November.
Sure if they are attached to something moving they need to update but back then I got 13 ACTIVE_OBJECT_MESSAGES per second for one lead between 2 fence posts in a simple MTG+leads test world.
After a quick glance through code, seems like leads just don't care if both attached objects are static and always update their pos?
89322baf5c/leads.lua (L91)between the two issues, do we use this issue and close the other one, keep both open, or close this issue and move to the other one?
I was thinking maybe this one could be purely about network issues and other one could be general (since there's a different issue of FPS lag too).
But I'm ok with whatever other staff members decide.
I would say keep both open but rename this one.
#-5234 is more specific about lag in just Haven
#-this one is more about general unnecessary stuff mods do which results in network traffic, Haven is just the best testing ground because basically any node/entity available (within city rules) is used there at the same time and rarely unloaded which results in constantly used server ressources.
I guess you are talking about on_step (codeberg cause YL repo´s are private) then yesh, it never checks if there even was a change
Haven lagto Unnecessary network traffic as potential source of lag in HavenI can't contribute any new insights yet.
Haven used to be terribly slow for me on my old machine - the old CPU was too slow to decode incomming packages in time. Now it seems the line gets overwhelmed by packages and my network connection is too slow to handle Haven - while the newer CPU can handle it easily. I constantly run around not-loaded blocks when trying to walk from spawn to the sailship.
Some of the packet senders whosit identified are particles. I'd expect them to just send the information that there are to be particles? That alone shouldn't be too bad. Perhaps too many of those sums up.
That the NPC also contribute worries me. I'll have to look into that and see if I can do anything on the NPC side (they're mobs_redo based). An NPC standing in a torch would constantly try to jump to get out of it, start sweating and eventually die (thus at least eliminating the lag problem). Perhaps there are other mechanisms I'm currently unaware of.
Signs and other things - do they really need to update so often?
Torches near arena send 202 packets per second, 2 packets per torch (554 and 245 bytes per packet = 80k bytes per second).
Ideally they should set up particle generator with the right parameters only once, and client should generate particles without any traffic from the server.
I would just comment out this ABM that creates new particle spawners each second:
aca469cc46/torches.lua (L137)Not sure if the particle spawners persist in mapblocks or how to create them "on load" if they do not persist...
They don't do it often, but they do it after a long time all at the same time - those spikes are massive compared to background traffic (comparable to loading a bunch of new mapblocks, I guess).
Regarding the NPC and all mobs_redo based mobs: If I see that correctly, they do each call do_env_damage() once per second - and that calls update_tag(). Perhaps that is where some of the packages from the NPC come from.
I've disabled this mechanism for the NPC (not for mobs in general; the Miocene are still affected). We don't want to have the NPC burn in torches or die trough flowers or crystals anyway.
Would be great if that could be tested. The yl_speak_up repository is updated.
I did more running around Haven with new filters, to library, to arena, to post office, then to the harbor and turned to look back. Here are the results:
Basically, every time you turn to look towards signs, MT re-sends you all the visible entities and texture modifier strings for them...
(magenta and red are signs - mostly what peaks are made of)
yl_speak_up just makes constant background level, not even visible here.
so, i've mentioned this somewhere but i can't dig up the reference, but i think the solution might be to make signs (and node-bound-entities more generally) only appear when a player is close to them (this is non-trivial but i have ideas). but this will require heavy ABMs and a lot of
get_objects_inside_radiuscalls, which could end up being significantly worse for the server performance. but that might be something we do to help out clients.I have a radical suggestion of converting all the never-changing signs in Haven (that show the way to airport, etc) to nodes with pre-rendered textures. It will eat up some node IDs, but maybe some day, a better way to write text on nodes will be available.... This will get rid of some entities at least - which also would make
get_objects_inside_radius()calls faster :p#5234 (comment) ?
I've made this PR for torches: https://github.com/mt-mods/abritorch/pull/6
It's not the main problem, but it annoys me :)
Seems like I misunderstood how particle spawners work and had to close that PR.
My simplest suggestion for now is to just disable particle spawners for abriflame torches.
Or spawn a particle spawner lasting 10 secords, every 10 seconds and also somehow stagger them.
turning this change into a server setting might better provide us with some relief, but i don't think particles are YL's biggest issue. the biggest issue is signs - textures built out of a composite of hundreds of individual textures, on the fly, and (mostly?) all done on a CPU and not a GPU.
NPC and miocene once per second ticking (re-sending of properties via
AO_CMD_SET_PROPERTIESpackets) is caused by this code:https://gitea.your-land.de/your-land/mobs_redo/src/branch/master/api.lua#L3480
Which seems to call
update_tag()even if nothing is changed...Should be possible to get rid of that tick...
whosit wrote:
That function seems to be pretty new. When I found out about it (an NPC died due to torches), I investigated, found that function - and overloaded it with a dummy function for the NPC. No idea if the version on YL is current enough for that.
This too frequent call to update_tag() may also worsen the situation for me in Voice attacks. So far I had no way to measure it. When there are hundreds of Scouts out there, each packet not sent counts...but that doesn't affect Haven directly.
flux wrote:
Sounds indeed like a lot of network traffic :-/
How can we reduce the usage of signs in Haven? As for storage systems, they might build up a formspec and show players what's in which chest when clicked on a node that creates a suitable formspec on the fly. Kind of a storage overview.
books and posters are much better than the signs that let you put in hundreds of characters. but then again, they're not a panacea - the post office is a prime example of there just being far too many signs in one place. perhaps we can make a "sign" that only has an infotext instead of an entity? the minetest_game signs originally only did that, but signs_lib overrides them. on blocky we ended up creating dupes of the original mtg signs. but they're impossible to read from a distance, and finding an individual mailbox by ID or player name would become quite difficult without them.
Stupid question: Do more characters in signs cause more issues or is it the number of signposts, regardless the content?
This is how texture combinators for signs look:
where
:126,0=signs_lib_font_16px_64.pngis image containing one letter.Unicode fonts looks like this (middle of it):
For unicode
^[sheet\:16x16\:9,6:117,34=signs_lib_uni00.pngis one letter - they use single texture with many characters and cut it in parts withsheet.This both inflates paket size (see first post in this issue, where AO update is split into 675 packets, each packet 554 bytes) and needs to be processed to generate actual texture GPU can draw. This processing causes significant stuttering on my PC (but thankfully only once, at least they are cached).
So, yes, number of characters makes it worse.
Hopefully this will reduce at least some of the mobs_redo entities spam:
#6150
That's what my client logs says:
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (238,34) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (225,51) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (217,68) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (251,85) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (7,118) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (135,118) for [combine
2024-02-29 15:51:55: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (151,0) for [combine
2024-02-29 15:53:50: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (499,34) for [combine
2024-02-29 15:53:50: WARNING[Main]: collisionMoveSimple: maximum step interval exceeded, lost movement details!
2024-02-29 15:53:50: WARNING[Main]: generateImagePart(): Skipping "signs_lib_color_16px_n.png" as it's out-of-bounds (256,17) for [combine
There are about 400 of those if I enter Merchant's Quarter. There seems to be an attempt to fix this: https://github.com/mt-mods/signs_lib/pull/25