TerrorBite
- terrorbite
- terrorbite
- Australia/Sydney
- Yes
- No
MC-72390
Use of Rcon to execute commands that modify server state causes a race condition that may result in a java.util.ConcurrentModificationException in the main server thread if a command modifies data during a server tick.
Conversely, executing commands via Rcon that iterate over internal data structures (such as /save-all) may cause a java.util.ConcurrentModificationException in the Rcon thread if the main thread modifies those structures (see
MC-68374).In the following example, an external script was being used to run commands. A ConcurrentModificationException was thrown when the Rcon thread /summon ed a Creeper and added it to the world's entities list, while the main thread was simultaneously iterating over the same list due to a command block executing /kill @e[type=MinecartTNT] . (Full crash report is attached)
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: effect RobotoRaccoon 11 60 100 true [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Given Resistance (ID 11) * 100 to RobotoRaccoon for 60 seconds] [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: playerinfo RobotoRaccoon [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 86 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned] [2014-09-26 21:14:21] [Server thread/ERROR]: Encountered an unexpected exception u: Executing command block at net.minecraft.server.MinecraftServer.z(SourceFile:603) ~[minecraft.jar:?] at po.z(po.java:228) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.y(SourceFile:531) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.run(SourceFile:447) [minecraft.jar:?] at java.lang.Thread.run(Unknown Source) [?:1.7.0_55] Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.7.0_55] at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.7.0_55] at aqu.a(SourceFile:2230) ~[minecraft.jar:?] at ah.a(SourceFile:433) ~[minecraft.jar:?] at ah.b(SourceFile:136) ~[minecraft.jar:?] at ab.a(ab.java:71) ~[minecraft.jar:?] at aqf.a(SourceFile:97) ~[minecraft.jar:?] at auo.b(SourceFile:53) ~[minecraft.jar:?] at qt.a(SourceFile:527) ~[minecraft.jar:?] at qt.c(SourceFile:199) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.z(SourceFile:599) ~[minecraft.jar:?] ... 4 more [2014-09-26 21:14:21] [Server thread/ERROR]: This crash report has been saved to: /data/minecraft/18-temp/./crash-reports/crash-2014-09-26_21.14.21-server.txt [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 87 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]Reproduction
- Create a new Minecraft server instance.
- Enable Rcon on the default port (25575)
- Set Rcon password to "test"
- Log in to the server and travel to x=0, z=0
- Use Python to run the attached rcon.py script, which will send commands to the server.
- Continue until the server generates a crash report. (I have attached one such crash report below)
I have successfully reproduced a ConcurrentModificatonException crash using this method. Due to the nature of this bug (race condition), it may take some time before the crash occurs.
rcon.py
Use of Rcon to execute commands that modify server state causes a race condition that may result in a java.util.ConcurrentModificationException in the main server thread if a command modifies data during a server tick.
Conversely, executing commands via Rcon that iterate over internal data structures (such as /save-all) may cause a java.util.ConcurrentModificationException in the Rcon thread if the main thread modifies those structures (see
MC-68374).In the following example, an external script was being used to run commands. A ConcurrentModificationException was thrown when the Rcon thread /summon ed a Creeper and added it to the world's entities list, while the main thread was simultaneously iterating over the same list due to a command block executing /kill @e[type=MinecartTNT] . (Full crash report is attached)
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: effect RobotoRaccoon 11 60 100 true [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Given Resistance (ID 11) * 100 to RobotoRaccoon for 60 seconds] [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: playerinfo RobotoRaccoon [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 86 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned] [2014-09-26 21:14:21] [Server thread/ERROR]: Encountered an unexpected exception u: Executing command block at net.minecraft.server.MinecraftServer.z(SourceFile:603) ~[minecraft.jar:?] at po.z(po.java:228) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.y(SourceFile:531) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.run(SourceFile:447) [minecraft.jar:?] at java.lang.Thread.run(Unknown Source) [?:1.7.0_55] Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.7.0_55] at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.7.0_55] at aqu.a(SourceFile:2230) ~[minecraft.jar:?] at ah.a(SourceFile:433) ~[minecraft.jar:?] at ah.b(SourceFile:136) ~[minecraft.jar:?] at ab.a(ab.java:71) ~[minecraft.jar:?] at aqf.a(SourceFile:97) ~[minecraft.jar:?] at auo.b(SourceFile:53) ~[minecraft.jar:?] at qt.a(SourceFile:527) ~[minecraft.jar:?] at qt.c(SourceFile:199) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.z(SourceFile:599) ~[minecraft.jar:?] ... 4 more [2014-09-26 21:14:21] [Server thread/ERROR]: This crash report has been saved to: /data/minecraft/18-temp/./crash-reports/crash-2014-09-26_21.14.21-server.txt [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 87 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]Reproduction
- Create a new Minecraft server instance.
- Enable Rcon on the default port (25575)
- Set Rcon password to "test"
Log in to the server and travel to x=0, z=0- Use Python to run the attached rcon.py script, which will send commands to the server
.Continue until the server generates a crash report. (I have attached one such crash report below)I have successfully reproduced a ConcurrentModificatonException crash using this method. Due to the nature of this bug (race condition), it may take some time before the crash occurs.
Use of Rcon to execute commands that modify server state causes a race condition that may result in a java.util.ConcurrentModificationException in the main server thread if a command modifies data during a server tick.
Conversely, executing commands via Rcon that iterate over internal data structures (such as /save-all) may cause a java.util.ConcurrentModificationException in the Rcon thread if the main thread modifies those structures (see
MC-68374).In the following example, an external script was being used to run commands. A ConcurrentModificationException was thrown when the Rcon thread /summon ed a Creeper and added it to the world's entities list, while the main thread was simultaneously iterating over the same list due to a command block executing /kill @e[type=MinecartTNT] . (Full crash report is attached)
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: effect RobotoRaccoon 11 60 100 true [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Given Resistance (ID 11) * 100 to RobotoRaccoon for 60 seconds] [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: playerinfo RobotoRaccoon [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 86 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned] [2014-09-26 21:14:21] [Server thread/ERROR]: Encountered an unexpected exception u: Executing command block at net.minecraft.server.MinecraftServer.z(SourceFile:603) ~[minecraft.jar:?] at po.z(po.java:228) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.y(SourceFile:531) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.run(SourceFile:447) [minecraft.jar:?] at java.lang.Thread.run(Unknown Source) [?:1.7.0_55] Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.7.0_55] at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.7.0_55] at aqu.a(SourceFile:2230) ~[minecraft.jar:?] at ah.a(SourceFile:433) ~[minecraft.jar:?] at ah.b(SourceFile:136) ~[minecraft.jar:?] at ab.a(ab.java:71) ~[minecraft.jar:?] at aqf.a(SourceFile:97) ~[minecraft.jar:?] at auo.b(SourceFile:53) ~[minecraft.jar:?] at qt.a(SourceFile:527) ~[minecraft.jar:?] at qt.c(SourceFile:199) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.z(SourceFile:599) ~[minecraft.jar:?] ... 4 more [2014-09-26 21:14:21] [Server thread/ERROR]: This crash report has been saved to: /data/minecraft/18-temp/./crash-reports/crash-2014-09-26_21.14.21-server.txt [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 87 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]Reproduction
- Create a new Minecraft server instance.
- Enable Rcon on the default port (25575)
- Set Rcon password to "test"
- Connect a Minecraft client to the server.
- Move the player to x=0, z=0 (or use /spreadplayers 0 0 1 1 false @a) to ensure this location is loaded.
- Use Python to run the attached rcon.py script, which will send commands to the server (spawning entities at x=0 z=0)
- The server should crash in under 60 seconds (usually only a few seconds).
I have successfully reproduced a ConcurrentModificatonException crash using this method. Due to the nature of this bug (race condition), it may take some time before the crash occurs.
Use of Rcon to execute commands that modify server state causes a race condition that may result in a java.util.ConcurrentModificationException in the main server thread if a command modifies data during a server tick.
Conversely, executing commands via Rcon that iterate over internal data structures (such as /save-all) may cause a java.util.ConcurrentModificationException in the Rcon thread if the main thread modifies those structures (see
MC-68374).Update 2016-11-08
During reproduction of this bug, it was observed that introducing a small delay (as low as 0.003 seconds) between each command sent via rcon appears to greatly reduce the chance-per-command of the bug being triggered. However, with zero delay between commands being sent, the chance of the bug being triggered is significant, with usually less than 100 commands being enough to trigger the bug.
End update
In the following example, an external script was being used to run commands. A ConcurrentModificationException was thrown when the Rcon thread /summon ed a Creeper and added it to the world's entities list, while the main thread was simultaneously iterating over the same list due to a command block executing /kill @e[type=MinecartTNT] . (Full crash report is attached)
[2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: effect RobotoRaccoon 11 60 100 true [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Given Resistance (ID 11) * 100 to RobotoRaccoon for 60 seconds] [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: playerinfo RobotoRaccoon [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 86 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned] [2014-09-26 21:14:21] [Server thread/ERROR]: Encountered an unexpected exception u: Executing command block at net.minecraft.server.MinecraftServer.z(SourceFile:603) ~[minecraft.jar:?] at po.z(po.java:228) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.y(SourceFile:531) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.run(SourceFile:447) [minecraft.jar:?] at java.lang.Thread.run(Unknown Source) [?:1.7.0_55] Caused by: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(Unknown Source) ~[?:1.7.0_55] at java.util.ArrayList$Itr.next(Unknown Source) ~[?:1.7.0_55] at aqu.a(SourceFile:2230) ~[minecraft.jar:?] at ah.a(SourceFile:433) ~[minecraft.jar:?] at ah.b(SourceFile:136) ~[minecraft.jar:?] at ab.a(ab.java:71) ~[minecraft.jar:?] at aqf.a(SourceFile:97) ~[minecraft.jar:?] at auo.b(SourceFile:53) ~[minecraft.jar:?] at qt.a(SourceFile:527) ~[minecraft.jar:?] at qt.c(SourceFile:199) ~[minecraft.jar:?] at net.minecraft.server.MinecraftServer.z(SourceFile:599) ~[minecraft.jar:?] ... 4 more [2014-09-26 21:14:21] [Server thread/ERROR]: This crash report has been saved to: /data/minecraft/18-temp/./crash-reports/crash-2014-09-26_21.14.21-server.txt [2014-09-26 21:14:21] [RCON Client #3/INFO]: $SERVCMD Rcon: summon Creeper 2209 87 501 {Fuse:0} [2014-09-26 21:14:21] [RCON Client #3/INFO]: [Rcon: Object successfully summoned]Reproduction
- Create a new Minecraft server instance.
- Enable Rcon on the default port (25575)
- Set Rcon password to "test"
- Connect a Minecraft client to the server.
- Move the player to x=0, z=0 (or use /spreadplayers 0 0 1 1 false @a) to ensure this location is loaded.
- Use Python to run the attached rcon.py script, which will send commands to the server (spawning entities at x=0 z=0)
- The server should crash in under 60 seconds (usually only a few seconds).
I have successfully reproduced a ConcurrentModificatonException crash using this method. Due to the nature of this bug (race condition), it may take some time before the crash occurs.
@[Mojang] Grum (Erik Broes): I managed to make a quick hack in Java (which I don't normally use a lot, so I'm not going to make a full-fledged client). I'm attaching it. Be sure to set the static variables in the source.
@TerrorBite: This is unrelated to MC-72390. They're just both more problematic the more you send.
@Grum
Was this fixed for more than just player nameplates? I notice that related issues are also fixed.
For example, if color codes are used in a kick message, the text is now properly center-justified, whereas it would previously be incorrectly offset. Colored text on signs is also now centered correctly.
Either way, thanks for the fix!
Confirming this is still an issue in Minecraft 1.7.2 and with 1.7.4 client on 1.7.2 server. Have not tested with 1.7.4 server.
Can confirm this issue exists in Minecraft 1.8 running on Java 8. I can also confirm that this is a long-standing issue: I originally reported this bug for Minecraft 1.3.2, see BUKKIT-2546.
Related:
MC-46850It appears that qt.a(boolean) is not threadsafe, and modifies world data structures while the Rcon thread is attempting to read the same structures to save the world.
Looks like these are only warnings, not errors. While it is unusual for these messages to show up, it appears that the game is detecting a potential issue (buffer too small) and resolving it (growing the buffer), so overall there is no problem.
Do you have your view distance set to a very high value, such as 15 or higher? Do the messages stop if you lower your view distance?
Attached Python script used to reproduce the bug. Connects to Rcon on port 25575 with password "test" and issues a rapid series of /summon Fireball commands (server despawns the fireballs immediately).
Attached crash report generated by reproducing the bug using rcon.py
@Amn3zik Out of interest, what were you attempting to do via Rcon that triggered that crash?
Thanks for the info!
This is emergent behaviour due to the way that lava sets flammable blocks on fire.
There is a specific area of blocks above a lava source, namely the 3x3 directly above the lava block, and 5x5 on the next layer up: http://i.stack.imgur.com/1vNGB.png (Fire blocks have a similar area, but it is a different shape)
In technical terms, any air block in this area, which has a flammable block beside it, will be converted to a fire block when it receives a random tick (note that the flammable block does not have to be within this area).
Now, looking at your screenshot:
Comparing this to the valid set of flammable blocks, you will note that none of the wood blocks in your screenshot are next to a block that is valid to become fire. Also note that the top surface of the logs are covered by flowing lava, so they are not valid to become fire blocks.
I am not certain on the rules here, but I believe flowing lava will not set flammable blocks on fire, except for falling lava (created when flowing lava drops down one block - this is why breaking a wood block and letting the lava flow down will set the wood on fire).
In conclusion, I'm going to say that this behaviour, while counter-intuitive, is by design.
It's been over a year since this bug was reported, and the issue persists. It's perhaps worth noting that a popular third-party server project fixed this same issue over three years ago.
This appears to be a duplicate of
MC-72390This is a duplicate of
MC-72390This should not duplicate
MC-77255, since 77255 is caused by the Watchdog, not by Rcon.This should now be considered a child bug of
MC-72390Related to
MC-72390@[Mojang] Grum (Erik Broes): Under
MC-72390, I have a PoC script that is tested and, via Rcon, will crash a server with a ConcurrentModificationException; though I'm not sure if that bug is caused by this one. They're almost certainly related, however.@jonathan2520: do you have further details about how this issue relates to
MC-72390?It is unlikely this bug will be fixed without special attention, since the observed behaviour is an artifact of the way that Rcon has been implemented (as a separate listener thread). The only way to resolve the issue is to implement special handling of Rcon commands by queueing them for the main thread to handle (and likely blocking the Rcon thread until the result of the command is available), rather than naively executing them directly on the Rcon thread where they can interfere with concurrent operations on the main server thread.
Updated rcon.py test script that uses snowballs instead of fireballs (due to some changes to entity handling that cause summoned fireballs to vanish before they spawn)
This issue is confirmed to affect 1.10.2. Crash report attached. Server log attached.
During reproduction of this bug using the attached rcon.py script, it was observed that introducing a small delay (as low as 0.003 seconds) between commands appears to greatly reduce the chance-per-summon of the bug being triggered. However, with zero delay between commands being sent, the chance of the bug being triggered is significant, with usually less than 100 entities being successfully summoned before the crash occurs.
The current test script summons 20 snowballs every 2 seconds, meaning the server usually crashes within 10 seconds of the script being started.
Yes, the specific race condition used as an example in this bug report occurs in the entity processing code. Rcon commands that manipulate entities will cause a crash when the server is also processing entities at the same time, since Rcon and the server operate on different threads. If there are no players on the server and thus few or no entities are loaded, the server will be spending little or no time processing entities, avoiding the race condition.
However, entity processing is only one area that is affected by the root cause of the issue, that cause being the Rcon thread modifying server data in a non-thread-safe manner. As @jonathan2520 has discovered, a crash will also occur if the server thread is handling block updates for water flows while Rcon commands are also trying to modify blocks. This has nothing to do with entities, but the cause of the crash is ultimately the same.
Ultimately, the fix for this bug must involve one of two solutions: Either