MC-33134Mineshaft.dat uses too much CPU
After pruning files trying to figure out why server usage is so high even during idle and no players on, I discovered that the Mineshaft.dat file was the source of it.
Before deletion the file was 839KB vs 14KB for the next largest file, Strongholds.dat.
As for process usage before deletion the server will run at 60% usage on idle, after 20%. Running on a dual core@3GHZ that's a whopping 2.4GHZ of process generated permanently from the single file.
I've tested the server file and pruned the .dat file on another of my computers (x6core) and it showed the same results.
I may be wrong but I don't think the Mineshaft structure data file shouldn't be generating 2/3rds of a server's processes.
Attached Image is a before and after of resources while on idle.
Not sure what causes this (Process loop?) but I hope this report is helpful.
Environment
Microsoft Windows 7 Ultimate 64-bit
Java 7 64-bit
Intel Core 2 Duo e8400@3GHZ
Linked Issues
Attachments1
Created Issue:
Mineshaft.dat uses too much CPU
After pruning files trying to figure out why server usage is so high even during idle and no players on, I discovered that the Mineshaft.dat file was the source of it.
Before deletion the file was 839KB vs 14KB for the next largest file, Strongholds.dat.
As for process usage before deletion the server will run at 60% usage on idle, after 20%. Running on a dual core@3GHZ that's a whopping 2.4GHZ of process generated permanently from the single file.
I've tested the server file and pruned the .dat file on another of my computers (x6core) and it showed the same results.
I may be wrong but I don't think the Mineshaft structure data file shouldn't be generating 2/3rds of a server's processes.
Attached Image is a before and after of resources while on idle.
Not sure what causes this (Process loop?) but I hope this report is helpful.Environment
Microsoft Windows 7 Ultimate 64-bit
Java 7 64-bit
Intel Core 2 Duo e8400@3GHZ
Attachments1
- Unresolved
- Open
- Unconfirmed
- Survival
- Bug Data Mineshaft Process Structure
- 13w39a
I found the cause of this issue, and the solution for it is pretty simple.
First of all, what exactly are these .dat structure files:
These files are a gzip compressed NBTTagCompound, which itself is build out of many other NBTTagCompounds. In case of Mineshaft.dat, these can easily be several thousands.What is wrong with them:
The procedure of saving these files is currently very inefficient.How are these files saved:
Via OutputStreams, they are layered in this fashion:
DataOutputStream(GZipOutputStream(FileOutputStream(File)))
Saving is initiated at the root NBTTagCompound and from there each child-NBTTag is recursively visited. Each time a NBTTag is visited, it is written to the top output stream (the DataOutputStream object).
Here is the culprit: For each write operation on the DataOutputStream, the GZipOutputStream is instantly compressing the new data. The problem is, that we dont have a single write operation with lots of data, but thousands of write operations with very litte data. The overhead of the GZipOutputStream for each compression is building up here.The solution:
Instead of writing each NBTTag directly to a compressed stream, gather the data in an uncompressed one first, and
then write it all at once to the compressed stream.
Gathering can be done with a ByteArrayOutputStream:DataOutputStream(ByteArrayOutputStream())
When done, get the data with a call to .toByteArray() on the ByteArrayOutputStream object and write the byte array to the previous compressed stream.
==========================================
You can even take it one step further:1) Create a copy of the data with .clone() on the root NBTTag, which takes only a fraction of what compressing and writing takes
2) Let a seperate thread write the copy to the compressed stream
Just noticed something strange... above I mentioned a 2.3MB mineshaft.dat... after a month of users exploring new terrain and whatnot... the file is still the same size!
Seems fixed to me in 1.7.5. right?
![[Mod] Ezekiel (ezfe)](../../../data/assets/gravatar/ezekielelin.jpeg){kind=small}
is duplicated by
You've got to be joking. This has gone ignored for literal years, and you just mark it as resolved, for no reason? Yes, it's still an issue- it's not going to stop being an issue until someone fixes it!
If anything, it's even worse than before. Last year, when I got you guys to reopen this, I only noticed it after I generated ~100k chunks, but a few months ago, I found that I couldn't even generate ~4k chunks without the server lagging all to hell.
And 100% of the lag goes away when you delete mineshaft.dat.
Please stop ignoring this, and then closing it for no reason, and actually fix it already.
![[Bot] Arisa](../../../data/assets/gravatar/mustek2.png){kind=small}
![[Mod] Pokechu22](../../../data/avatars/users/92931.png){kind=small}
The 1.7 versions of Minecraft have resulted in the most baby sitting I've ever encountered since 1.8 Beta, before I realized I had to delete the Mineshaft.dat file with every restart. After it got to 11MB, it grows to 3MB in less than 24 hours. Not even the worst snapshot needed this much attention.
Before I determined the issue, I tuned the garbage collector, installed several watchdog scripts, did hourly restarts, did metrics based restarts. I troubleshot everything I could think of. Then, one of my regular players suggested deleting a random file. I couldn't believe this was the solution. But it was. The difference is night and day.
I believe any active world over 8GB will be crippled unless they have the resources to just overpower the problem. All I know is, I went from having to restart 15 times a day, down to only one or two restarts required.
MC-33134 appears related, and I mentioned it has a stack trace. My gut feeling, looking at the stack trace is that there is some kind of race condition. MC-34947, which is currently ranked as #47 in popularity, is also related, but the comments are all over the place. Obviously, the netty timeout is attracting every kind of connectivity failure, not just ones related to this. But I believe the most common bug-related reason people see a client timeout is related to this .dat growth problem.
Ok, hyperbole has no place in a publicly facing bug tracking system with thousands of reports. But I think this might be a near-universal problem.
I agree 100% My server was always on 100% usage no matter what plugin or slots filled I had. I did some searching with my host's live support(getnodecraft.net) they were a great help.With them, I found out Mineshaft.dat was the problem. Now I have to delete it every once in a while to make it less laggy.
I have an issue with mineshaft.dat as well, It's not using all my CPU, but is eating my RAM. Deleting the file will help for a while, but it builds up again.
I have searched for a solution to my high RAM issue for a while. I started to believe that 1.6.4 contained a memory leak from how often I've seen it said. But after I deleted the mineshaft.dat file, the high RAM usage issue went away. My mineshaft.dat file was 18 MB in size due to the great size of the map I host. Additionally, doing a /save-all does not lock the server up for 20 seconds anymore. That issue started with 1.6.4 as well and was resolved with this fix. I thought I'd share my experience with this bug. Thanks.
Problem observed in 1.7.2.
Ok, I've broken down and decided to investigate if I should delete my dat files because Mojang has apparently abandoned us. I know, that's hyperbole, but it sure feels that way.
I have a 11MB mineshaft.dat file. It's been growing for weeks. The /save-all is taking longer and longer every day. Yesterday, the /save-all took 45 seconds. Today, it's up to 50 seconds. I've seen it spike up to 80 seconds.
So I'm going to bite the bullet and delete the mineshaft.dat file. I feel very strongly that this is not something I should have to do. I don't know what the file is for. I have a feeling it isn't important since there are no specific mobs that spawn in mineshafts. But in the future, if mineshaft mobs ever come along, these old mineshafts won't have them.
With no patch on the horizon, it's the only option left. I hope the save spikes at least subside.
It worked.
Ah yes, Mojang has "abandoned you" because we haven't dropped everything to work on this one specific bug out of thousands...
Other files: 6kt. Mineshaft.dat: 9mt.
Thats why it lags when looking direction of mineshaft!
The 1.7 versions of Minecraft have resulted in the most baby sitting I've ever encountered since 1.8 Beta, before I realized I had to delete the Mineshaft.dat file with every restart. After it got to 11MB, it grows to 3MB in less than 24 hours. Not even the worst snapshot needed this much attention.
Before I determined the issue, I tuned the garbage collector, installed several watchdog scripts, did hourly restarts, did metrics based restarts. I troubleshot everything I could think of. Then, one of my regular players suggested deleting a random file. I couldn't believe this was the solution. But it was. The difference is night and day.
I believe any active world over 8GB will be crippled unless they have the resources to just overpower the problem. All I know is, I went from having to restart 15 times a day, down to only one or two restarts required.
MC-33134appears related, and I mentioned it has a stack trace. My gut feeling, looking at the stack trace is that there is some kind of race condition. MC-34947, which is currently ranked as #47 in popularity, is also related, but the comments are all over the place. Obviously, the netty timeout is attracting every kind of connectivity failure, not just ones related to this. But I believe the most common bug-related reason people see a client timeout is related to this .dat growth problem.Ok, hyperbole has no place in a publicly facing bug tracking system with thousands of reports. But I think this might be a near-universal problem.
Please don't be mad at us, Mog
It's just that after finally figuring out the cause of the issue, we tried to get Mojangs attention to it, since we were sure that it a) is the root cause of the problem and b) could be the main cause for at least 100 other bug reports where ppl complain about serious lag and/or high ram/cpu usage and - most importantly - Read Timeouts (wich ppl still falsely blame netty for). Seeing that Mindcrack has the same lag-spike issues, I even tried to contact some of the Mindcrackers, since they seem to have a more direct link to Mojang, but as a single person that task seems even more difficult than contacting one of the Mojangsters directly...
So after trying to get attention to this report for weeks and beeing sure that it is a severe, universial problem, still seeing it as 'unconfirmed' is a frustrating feeling...
Though, nobody can blame you for that either, after all, reaching Mojang and having an influence on the development on Minecraft (in certain borders) is possible and happening all the time. I think this is something that makes Minecraft very unique.
It's just that in cases like this where there are hundreds of bug reports showing the effects but only one or two actually showing the cause of the problem (or at least giving a very good hint on where to look) the ods are ppl upvote the reports they find first - wich are the ones showing the effects of the bug and so you can easily feel left alone...
So please don't take it personally, we love Minecraft and we love Mojang, if it wasn't that way we wouldn't even commit to this bugtracker
Don't worry, I'm not upset, it's just that while this issue with mineshaft.dat may be causing these other issues, I still need to track down exactly why the file is becoming bloated in the first place. Unless there are comments that I'm not seeing, that question is still very much up in the air and will be a bit of a bear to track down.
Well, thanks for looking at it.
I've been having the same issue and my Mineshaft.dat file is about 100k. I've been noticing the process has been using up all of the RAM allotted to it on the command line and basically appearing to GC in a nice loop every 15 seconds. This was definitely not an issue in 1.6.x.
Though I understand server needs grow as more stuff is added over time, e.g. taller worlds, it seems like people with hardware an order of magnitude greater than mine are also seeing the issue.
I have a 32-bit dual core atom, 1.6ghz, 1.5GB max memory. I used the DEFAULT world type for this server, fresh for 1.7.x. Just restarting the server clears up the issue for a while, until enough stuff gets loaded by the server, e.g. entities because someone goes near my animal farm, then it's game over. I've run two or three instances on the same box in the past with no issue. And I can do so now, except after a while it goes into GC looping nightmare mode.
8.2% of 4.5GB (w/ swap) when I started, slowly climbing up to 8.7% now after 10 min, CPU is 47/400%. Eventually that will go up to 33% mem at which point it will GC, but be at 47% CPU while it's not GCing, which I have suggested to people is a good time to work with lava.
[in a flyish voice] "Help me!!!"
@Jim
You have a 32bit dual core Atom, I think that's your issue. :/
@Jim Well that would explain why I have better performance on my server with GC turned off, and why my server spikes to full proc usage (2.3MB mineshaft.dat) when I turn GC back on.
@TonyCubed Don't be so sure. I run my MC server off a core 2 duo and it outperforms some i3 due to the faster individual cores. All depends on how the server is configured (OS), how much overhead, etc.
@Talven81
A Core 2 Duo/i3 would smash the pants off an Atom, even a Celeron based CPU has a good chance depending on which version you get.
I've recently experienced high CPU and RAM usage in one of my worlds leading to stop and go mobs/player behavior due to this. This was in 1.6.4 vanilla.
http://www.minecraftforum.net/topic/2210743-solved-164-playermob-movement-lag/
I fixed it by narrowing it down to Mineshafts.dat through process of elimination, and deleting said file fixed my issues. That's when I Googled the file name and found this bug report on it. I guess this will continue to grow and need constant deletion from what other comments said, but it's at least made said world playable again.
Still having this issue, but deleting the Mineshaft.dat isn't helping as much as it used to because other .dat files are now getting too large as well. Specifically, the Fortress.dat file. It's now 7.5MB. Before, when the Mineshaft.dat file was the issue, it was 11MB. Deleting it was an acceptable workaround because it currently does nothing (that I'm aware of).
My server's total map size is 43GB. I delete the Mineshaft.dat every hour or so, when it gets to about 1MB. But deleting Fortress.dat is completely out of the question. I'd rather restart every 15 minutes.
This is such a known issue, modded server implementations have solutions, like disabling structures:
http://www.spigotmc.org/threads/disabling-structure-saving.8555/
... though this seems like a pretty lousy workaround. In order to find wither skeletons, one must generate new map terrain. Every time these structures are found, once they are unloaded from memory, players have to start over and find new structures. Screw that.
Confirmed on 1.7.4
Having High CPU usage and disconnects when exploring new terrain (Vanilla 1.7.4 created map on 1.7.2) Exploring new terrain is impossible now. Standing in the same spot for 15 minutes is not a problem, server load goes down. When i move a few blocks into unexplorered territory: server load goes to 100% and almost always disconnects me.
@Anthony While deleting .dat files works as a temporary workaround I still don't recommend it. While the mineshaft.dat is not directly used at this time, it could easily be used in the upcoming API for mineshaft specific mods and I would hate to see someone destroy functionality for the majority of their map (though I think I saw a rebuild option for that in MCEditor). Similar to what you said about deleting the fortress.dat... disabling structures would have the same issue.
@Talven81, I agree with you, but if the choice is destroy structure data or restart literally every single minute ... I do not exaggerate. The players on my server decided to travel by nether roof to 300,000 blocks. Vanilla has no way of preventing this kind of thing.
I hate deleting these files. I saw a comment on
MC-33086that claims this is an easy fix. I sure hope so, but the damage to my world has been done. Even if MCEdit can rebuild structures, I'm not sure if that's practical for a world over a certain size.Hmm while no specifics were given the latest snapshot mentions "Overworld, Nether and End are stored differently, increasing performance" will be interesting to see if this effects this ticket.
@Anthony - Yeah I understand what you are running up against on your server, you don't have many options. As far as preventing players from going too far you can set up a command block to testfor any player outside of a certain range and TP them back to a safe spot. This would essentially create a TP wall not allowing them to go outside a specific range. Again not a fix for this problem but just an idea for you.
Very good suggestion. Basically a snowglobe. But prior to 1.6.4, it was never necessary to confine anyone, except to save hard drive space. And it becomes more complicated when we consider Nether travel. Nonetheless, it is nice that there is an option for that if it comes down to it.
Regarding the performance changes, I think having each dimension on its own thread has greatly helped overall performance, even in this situation. Garbage collection is very much improved. Server uptime is very much improved.
As result, I purposefully stopped deleting the Mineshaft.dat files (I did this automatically when the server would start, but I disabled it). However, when I stopped deleting Mineshaft.dat, performance slowly dropped over the days until I had to resume deleting the file once again.
So, the performance changes actually mask part of the issue, perhaps to the point that only very large worlds experience this problem to a crippling degree.
Can this be confirmed?
Again, I think
MC-33086is the root cause for this:No it cannot be confirmed at this time, while it seems to be a known issue the exact root cause is not known.
I will mark it as community consensus though as I'm on board with you guys on several of the major server-related issues.
If you do a memory dump *after the server has been running a while to allow places to be explored*, you will notice the object representing the mineshaft.dat file can use several gigabytes of RAM.
16 Megabyte mineshaft.dat file (gzip compressed as most minecraft files are)
= 100 megabyte uncompressed mineshaft.dat file
= over 2 gigabytes in-memory object
What this means for me, with a server with -Xmx2G set, is that the CPU is spiking and the server is lagging because of continual full GC runs because the object is taking up space. This would seem to be the root cause of this bug, as well as a major contributing factor to
MC-33086(though I notice someone is also providing evidence that changing how the file is saved can help for that)To be clear, I'm not suggesting a memory leak per se. If I load the save 16 meg data file using the python NBT library, it consumes over 2 gigs of RAM. Ditto NBT explorer, based on .NET. There's something common to how all three implementations store the object in memory that's causing this bloat.
I think it qualifies as a memory leak. Not just your garden variety memory leak. It's a memory leak that is persisted to disc.
The NBT format definitely got several issues. A few days ago i found another one, which I document here:
https://github.com/Poweruser/MinetickMod-1.7.2/wiki/Heap-dump-analysis:-Redundant-String-objects
Two more memory related issues i can think of right now:
This file has 708 recorded mineshafts. Each mineshaft has somewhere between 50 and 220 structure pieces (just a quick check in the NBTExplorer. Lets take 100 here, which is a optimistic average. Most are above 100, only a few are below. This makes a estimate of 70800 HashMaps. Additionally to that you have the HashMap-Entry objects which are about 6 times the number of HashMaps: ~425000 (nodes of a structure piece: 4 - 8)
Edit: The NBT format works kinda well for loading/saving files and after that immediately discarding it (like its done with region files and chunks [yes, they are nbt files as well]), while praying that the object is still in the young generation space of the heap, so the frequent garbage collection there can pick it up. If the object made it to the old generation space in the mean time you're screwed.
But it's not suitable for storing something permenantly in memory.
Anthony, your information is incorrect. A memory leak by its very definition is a program retaining allocated memory that is no longer referenced, such that the memory is never deallocated across the runtime of the program. That is most assuredly not what is happening here. I suggest reading Poweruser's description of the issue, as he is 100% correct.
It is by definition not a memory leak, as all of the memory in question is in active use by the NBT system. Mineshaft.dat effectively contains a massive NBT structure containing data on every structure piece of every mineshaft in the entire world, which is highly inefficient in terms of memory usage, but is not a memory leak by any definition currently in use in the programming field.
Poweruser, thank you for the analysis. While we currently have no ETA regarding a resolution for this bug, your information is very helpful in explaining the root cause of this issue.
Poweruser, I've been testing with the following Java options, which give me the best possible up-time, even with theses non-memory-leak issues:
The thing is, *because* this is extreme bloat and not a memory leak, you can just power through by just ensuring you give it enough memory. If you know your mineshaft object is going to take 2 gigs, allocate 2 gigs more heap. CPU problem goes away (because it doesn't have to run the GC constantly to keep the server running). The long save times would remain, though. Thanks Mog for acknowledging Poweruser's analysis, it's reassuring to know you're aware of the root cause of these symptoms, even if the solving it is still ongoing.
1.7.5 still uses a lot of CPU. I was hoping for improvements but i'll stay with spigot for the moment. Vanilla still uses 4x more CPU than spigot...
using the "nogui" option fixed the Cpu usage in 1.8 for me, but this must be relatet to another case.
.the exe Version for Windows didnt work fine(to much cpu usage), but the Java Version runs better, even with this "bug".
Still a problem in 1.8.3.
Why was this marked as cannot reproduce? Here's the steps to reproduce:
1. Start a new Minecraft server
2. Run, fly or teleport around to generate a lot of chunks
3. Restart the server
4. Note the CPU and RAM usage
5. Shut down the server
6. Delete Mineshaft.dat
7. Start the server
8. Note the CPU and RAM usage
9. Compare the before and after CPU and RAM usage
The more chunks you generate, the greater the difference will be, and the weaker the hardware it's running on, the fewer chunks you will need to generate before you start noticing the lag.
Is this still an issue in the most recent versions (currently that is 1.10.2, or 16w42a) of Minecraft? If so, please update the affected versions and help us keeping this ticket updated from time to time.
Yes, this is still an issue; in fact it is even worse than it was before. Please don't resolve first and ask questions later. There's definitely no such sense of urgency here. Did you not notice that this issue tends to get ignored for chunks of time lasting longer than an entire year?
Is this still an issue in Minecraft 1.13.1?
I don't know. I haven't played in years, because you didn't respond for years. Maybe test it before marking it as resolved then asking if it's resolved?
@Asteraoth, yes, still an issue.
[~imp.games.contato@gmail.com], are you sure? I don't think mineshaft.dat is still created in 1.13; I vaguely recall structures being stored in individual chunks instead of all of the separate structure files which avoids having that whole thing loaded... but I'm not 100% sure.
That's correct, pokechu.