Client lag #5005
Labels
No Label
1. kind/balancing
1. kind/breaking
1. kind/bug
1. kind/construction
1. kind/documentation
1. kind/enhancement
1. kind/griefing
1. kind/invalid
1. kind/meme
1. kind/node limit
1. kind/other
1. kind/protocol
2. prio/controversial
2. prio/critical
2. prio/elevated
2. prio/good first issue
2. prio/interesting
2. prio/low
3. source/art
3. source/client
3. source/engine
3. source/ingame
3. source/integration
3. source/lag
3. source/license
3. source/mod upstream
3. source/unknown
3. source/website
4. step/approved
4. step/at work
4. step/blocked
4. step/discussion
4. step/help wanted
4. step/needs confirmation
4. step/partially fixed
4. step/question
4. step/ready to deploy
4. step/ready to QA test
4. step/want approval
5. result/cannot reproduce
5. result/duplicate
5. result/fixed
5. result/maybe
5. result/wontfix
ugh/petz
ugh/QA main
ugh/QA NOK
ugh/QA OK
No Milestone
No project
No Assignees
8 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: your-land/bugtracker#5005
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
This issue is meant to gather all information we have on client lag, most notably Haven, plot area.
This is the pure storage COMPARISON footprint. This does (most likely) not reflect the real amount of data that is transmitted to the client, but it shows which plot are heavier on data and which are not. At the bottom - also only for comparison - there is a "default" plot, without anything on it. Please note, at the time of this snapshot, B7 is very close to that default plot and only has a poster and a couple of books on it.
387K Jul 25 15:22 plot_a1.we
362K Jul 25 15:22 plot_a2.we
438K Jul 25 15:23 plot_a3.we
274K Jul 25 15:23 plot_a4.we
882K Jul 25 15:24 plot_a5.we
399K Jul 25 15:26 plot_a6.we
261K Jul 25 15:26 plot_a7.we
405K Jul 25 15:26 plot_a8.we
479K Jul 25 15:27 plot_b1.we
429K Jul 25 15:27 plot_b2.we
609K Jul 25 15:27 plot_b3.we
475K Jul 25 15:28 plot_b4.we
558K Jul 25 15:28 plot_b5.we
341K Jul 25 15:28 plot_b6.we
253K Jul 25 15:29 plot_b7.we
960K Jul 25 15:29 plot_b8.we
553K Jul 25 15:30 plot_c1.we
1.1M Jul 25 15:30 plot_c2.we
671K Jul 25 15:30 plot_c3.we
1.7M Jul 25 15:31 plot_c4.we
354K Jul 25 15:31 plot_c5.we
445K Jul 25 15:31 plot_c6.we
447K Jul 25 15:31 plot_c7.we
435K Jul 25 15:32 plot_c8.we
182K Jul 25 15:32 plot_d1.we
477K Jul 25 15:32 plot_d2.we
302K Jul 25 15:32 plot_d3.we
328K Jul 25 15:33 plot_d4.we
343K Jul 25 15:33 plot_d5.we
424K Jul 25 15:33 plot_d6.we
424K Jul 25 15:33 plot_d7.we
348K Jul 25 15:33 plot_d8.we
239K Jul 25 15:43 plot_default.we
Theory: The more data, the worse the lag.
How to test this theory? Remove n of the heaviest plots on the testserver and see whether it improves. Cross-check: Copy the most heavy plot in all plot position and see whether it gets worse.
Ultimately we need to find what makes the heavy plot so heavy data-wise and stack that up in a smol area and see what the lag does
In the F6 debug menu there are a couple of values displayed, among them the number of vertices drawn. The more vertices, the longer the drawtime (approximately), coz more to do by the GPU. The same place and look direction and distance (and other grafics settings) demand the same amount of vertices drawn.
Theory: The more vertices, the worse the lag
How to test? Walk a high-lag are three times. If it lags the first time, that might be some transmission lag + drawtime. If it lags the second time less than the first time, that might be because the blocks are already cached and it's only drawtime lag left. If that is true and it's only drawtime lag left, the third time should lag just as much as the second. Keep an eye on F6, page3, vertices drawn. There is a place looking at the adventurer guild where my PC needs to draw 1.5 Million vertices. (2019 60 1147 yaw 119 pitch -40)
Plots with large
we
schematics most likely have more meta and inventories in them. This large meta is:My guess that all of this has minimal impact on fps. And zero after mapblock is finished loading/deserialization.
There may be more factors at work.
The size of the WorldEdit savefile may be an indicator but is not very preceise. It will for example be larger if the names of the nodes used are longer. The internal mapblock format is quite diffrent. Though it still needs to store metadata.
With my old computer, decompressing the mapblocks in time didn't work. It was too slow to handle that. Thus, long freezes when walking from spawn to the sailship.
With the new computer (way faster), decompressing the mapblocks is no longer a problem. But they seem to be too many and too big - 1 MBit downstream just doesn't seem to suffice. So instead of complete freezes, I now run into unloaded mapblocks and have to wait long for them to make it to my machine.
I suspect that something causes mapblocks to be sent over and over again to my client, thus overwhelming either machine or internet connection. A mapblock is usually re-sent to the client when something changes in it. On Tunnelers' Abyss, for reasons unknown technic gold chests caused this effect. We never really found out why. If YL has something similar...
This "sending" happens to whoever one client is logged in. Any way we could visualize the sent mapblocks?
At least one core dev has/had a patched client or knew how to do it. I tried it as well but they arrived too fast.
That could be some mese machinery (although that is forbidden in Haven, so there likely will be close to zero of those) or some different mechanism that causes changes in blocks (growing plants?)
Patching client to log coords of received blocks somewhere (and then visualize that data later) would be fairly simple :)
However I suspect different cause of lag - lot of items does things like
minetest.get_objects_inside_radius(pos, 0.5)
around them - like armor stand (for shown armor pieces), item frame (for the object inside), enchanting table (the floating book), digilines LCD (the text), smartshop (shown goods) and I think some sign libs (not sure if the one on YL too) does this for their text.The more such objects, the more there is chance that some of them will call get_objects_inside_radius and as
get_objects_inside_radius
looks through list of all objects ... we have essentially O(n^2) complexity and that could mean lot of lag on server.Ok, when I saw #5008 I think signs also have their own entities here.
Armor stand, item frame, enchanting table, shop, signs ... I think those objects would be very rare in wilderness and very common in haven. I suspect the lag would be worst when many people spread around different parts of haven (so every player has own "island" of active objects around them) - those parts that contain lot of signs, shops and similar infrastructure.
The lag caused by get_objects_inside_radius would be server-side and the same for all players. The lag that's beeing hunted here is the lag that made my old computer freeze for half a minute - and that makes me run against unloaded chunks on view range 20 in haven while elsewhere view range 400 is no problem. By the time I've thus reached the sailship any focus is long defeated :-/
Lots of moving entities - like in a big voice attack - cause similar trouble. But in Haven the entities don't move - thus they ought not to send any movement packages?
Still, players run around, turn their heads - perhaps that is too much? Haven usually has more players around, and players often tend to at least look around...
Once we know whether it is connection lag, procession lag or fps lag, we can start fighting it.
If it's connection lag, we can look at how much is going on: https://www.tecmint.com/iptraf-ng-linux-network-monitoring/
If its procession lag, please take a look at htop and iotop.
If it's fpslag, we need to try to force your client to render less vertices.
Want to note that there are other places which show such an fps dropdown are (at least on my client) maravillerosa, Puerto del Sol or Tenebris.Searching in MT issues I found https://github.com/minetest/minetest/issues/12900 with a linked PR to get a repro. (and a possible dublication https://github.com/minetest/minetest/issues/13463)After a quick search I would suspect these nodes coming originally from the homedecor_lighting mod for they show the mentioned overlay_tiles + "blend" definition.(https://github.com/minetest/minetest/issues/13463 screenshots show these nodes, they are also in a lot of well-decorated areas where these lags mainly occur)outdated, look below...
items w/ metadata in chests are certainly a problem, but these are generally not the biggest issues for the merchants quarter.
i should record a video at some point about what i can see w/ my hacked client, w.r.t. FPS and general responsiveness. lag is almost entirely due to entities. signs are the biggest culprit, followed closely by smartshops, and then itemframes, and then similar but less common nodes like item shelves, enchantment tables, and anvils.
heck, even nodebox/mesh drawtypes are almost as bad as entities, and a lot of builds in haven use a whole lot of these.
After beeing so tired that I could not even write a good comment yesterday, I'll try to theory and findings with some additional informations from tests here :-)
connection lag: I would expect to run into unloaded chunks and receive late updates on map and objects. But with the current state of the map you should be able to interact without freezes. (I can still run around during a timeout, dig and place stuff, even if they get lost once I reconnect again. But no fps issues)
progression lag: not sure if that would limit rendering (because CPU is busy with something else). A way to test delay caused by (map)decompression could be
map_compression_level_net
in the minetest.conffps lag IMO the lag which most players experience
I would not blame static entities or certain drawtypes in general. In my theory there are certain nodes / entities which have very expensive graphics definitions.
I can WE giant boxes with mesh drawtype nodes in front of me and won't notice fps lag. Meanwhile small amounts of certain nodes are enough to create a lot of it.
Tests I made today show: (21x1x21) nodes, can provide screenshots if needed
So: larger amounts of entities like itemframes, signs, smartshops and certain node drawtypes will reduces fps for sure. But they cannot produce such a high lag, such as in Haven. Meanwhile small amounts of certain nodes can be blamed for much more lag. (I might have max_fps = 60 somewhere, I am never much beyond this value)
why homedecor:oil_lamp
I am sure Haven is not the only place where this lag happens. So I searched in MT issues for FPS and found https://github.com/minetest/minetest/issues/12900 It describes a way to create nodes which will cause a lot of fps lag. (overlay_tiles + use_texture_alpha = "blend") Effect is what we observe in Haven.
I guess the same issue (but less detailed) is described here: https://github.com/minetest/minetest/issues/13463 There I saw the homedecor lamps (which got extracted for YL afaik). I installed the mod and did a grep for overlay_tiles and "blend" on my mt mod folder (which contains a lot of public mods used on yl) and checked for collisions in the output. none except homedecor...
These show the bad definition mentioned in the issue (the linked pr also has a bug confirmation by core dev). I could reproduce too (see above).
Note that it looks like the amount of nodes in the mapchunk and the position of the player is more relevant than the actual position of the problematic node.
How to test: WE these nodes from Haven away and test if fps improves.
Further ideas: would it be possible to do some stuff/script like
iterate through all nodes -> WE a box of those nodes in front of a testacc -> log fps/drawtime and nodename -> next node
afterwards searching the log for nodes which cause much fps lag
if https://github.com/minetest/minetest/pull/13987 gets accepted, we it'd be great to make node-bound entities for signs, smartshops, etc. use that feature.
now that #5476 is live, we can test whether playeranim causes client lag
Oh, who ist this red bad guy?
Maybe a laghunter (Alias, flux/rheo, whosit/whostand accepted) should stop by soon? I have an idea what it could be. Otherwise, I am of course open to change. But not in the sense of Soko, who wanted to confuse all the boxes very quickly yesterday shortly before midnight.
IIRC on C4 there were even some loose meta blocks full of petz (ex-chests that became stone walls)...
Keep in mind that books are stored in node meta too - one letter one byte basically...
And also C4 has lots of signs/posters with text - same problem 1 character = 1 byte.
Since this data is out of date, I think C2 was Sense's shop at that time.
This was the my idea. Just two chests left. I had kept them as a curiosity until now.
I know I´m not on your acceptable laghunter list and I wouldn´t claim to be one but might offer some insight about those data.
the base of your analysis are not measured lag but worldedit schematics filesizes which include everything that small region contains, even air.
Those data might(like Alias stated) have an influence on lag but more at the time people go to Haven and the client has to load everything the first time. After that it gets stored clientside in RAM till the client doesnt need them anymore, like leaving Haven and its out of view range etc.
They are not that relevant if a client still has lag after everything has been loaded, happy witchhunting
Blah, you're always on my list, of course! Since I hardly understand what you are saying about the lag, I have to trust you. But I reject actionism. This should be systematic. By the way, I had already moved 150 books with the Bailiff applications to the new Bailiff office. I'm going to chop off the two ominous stone boxes now. There are only turtles in there anyway and I don't know what to do with them.
Books with long text are quite heavy on the storage and also petz (they contain some fair amount of metadata) ... so those "only turtles" would probably take up some significant storage...
I imagine some people could get their connection swamped with heavy mapblocks as they walk down that road from spawn to sailship... Here are some measurements with crystal boots and minimal view distance for walking east from spawn:
Same, but without crystal boots, but higher view range (50):
So, yes, those mapblocks are 4 times heavier than just jungle or random stuff on the beach.
For me plot loading peaks at 400kbytes/sec which is is 3.2mbit/s... which may be too much for some slow/bad connections I guess?
Also, red/magenta lines are signs/posters and stuff, so actually should be subtracted from green (total), to give pure mapblock size...