Philipp Provenzano
- PhiPro
- phipro
- Europe/Stockholm
- Yes
- No
MC-148580![[Mojang] Georgii Gavrichev](../../../data/avatars/users/22428.png){kind=small}
Normal
Low![[Mojang] Matthew Gatland](../../../data/avatars/users/43046.png){kind=small}
screenshot-3.png
Albeit being used concurrently by multiple threads, PalettedContainer isn't threadsafe. This can cause read
eing threads to see non-meaningful values, in particular Chunk.getBlockState(...) can return block states that never existed at the respective block pos (in the current session of the game).
As an example, this can cause issues with the new lighting engine which runs concurrently to the rest of the game logic. As multithreading issues are hardly reliably reproducible, the following steps use breakpoints to pause threads in order to simulate some bad luck. However, I was also able to reproduce the bugs even without breakpoints, just not reliably.
- Create a new redstone-ready world
- Set a breakpoint somewhere in the lighting code to prevent the next step from finishing/starting the light propagation. You should configure the breakpoint to only pause the thread triggering it, instead of pausing all threads. For example, you can set a breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected int getPropagatedLevel(long long_1, long long_2, int int_1) { if (long_2 == Long.MAX_VALUE) { return 15; } else if (long_1 == Long.MAX_VALUE) { return int_1 + 15 - this.getLightSourceLuminance(long_2); } else if (int_1 >= 15) { return int_1; } else { int int_2 = Integer.signum(BlockPos.unpackLongX(long_2) - BlockPos.unpackLongX(long_1)); int int_3 = Integer.signum(BlockPos.unpackLongY(long_2) - BlockPos.unpackLongY(long_1)); int int_4 = Integer.signum(BlockPos.unpackLongZ(long_2) - BlockPos.unpackLongZ(long_1)); Direction direction_1 = Direction.fromVector(int_2, int_3, int_4); if (direction_1 == null) { return 15; } else { AtomicInteger atomicInteger_1 = new AtomicInteger(); BlockState blockState_1 = this.getStateForLighting(long_2, atomicInteger_1); if (atomicInteger_1.get() >= 15) { return 15; } else { BlockState blockState_2 = this.getStateForLighting(long_1, (AtomicInteger)null); VoxelShape voxelShape_1 = this.getOpaqueShape(blockState_2, long_1, direction_1); VoxelShape voxelShape_2 = this.getOpaqueShape(blockState_1, long_2, direction_1.getOpposite()); return VoxelShapes.unionCoversFullCube(voxelShape_1, voxelShape_2) ? 15 : int_1 + Math.max(1, atomicInteger_1.get()); } } } }
- /setblock 7 60 7 minecraft:sea_lantern
- Some worker thread (and the rendering thread) should now hit the breakpoint. You can now disable the breakpoint and resume the rendering thread, but not the worker thread.
- Set a breakpoint at the last instruction of the following method, again configuring it to only pause the triggering thread
net.minecraft.world.chunk.PalettedContainer.javaprivate void setPaletteSize(int int_1) { if (int_1 != this.paletteSize) { this.paletteSize = int_1; if (this.paletteSize <= 4) { this.paletteSize = 4; this.palette = new ArrayPalette(this.idList, this.paletteSize, this, this.elementDeserializer); } else if (this.paletteSize < 9) { this.palette = new BiMapPalette(this.idList, this.paletteSize, this, this.elementDeserializer, this.elementSerializer); } else { this.palette = this.fallbackPalette; this.paletteSize = MathHelper.log2DeBrujin(this.idList.size()); } this.palette.getIndex(this.field_12935); this.data = new PackedIntegerArray(this.paletteSize, 4096); } }
- Place 14 different blocks (other than stone, sandstone and sea lanterns) in the 0 3 0 subchunk (where the sea lantern was placed)
- This should trigger the second breakpoint from the server thread (and also the rendering thread). Now resume the lighting worker thread. Afterwards, you can disable the second breakpoint and resume all threads.
- Restart the world
- Observe that the sea lantern does not emit light
Similarly, you can make blocks transparent
- Create a new redstone-ready world
- /fill 6 63 6 8 66 8 minecraft:stone hollow
- Procede as above, but instead set the sea lantern at /setblock 7 65 7 minecraft:sea_lantern and set the first breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected void propagateLevel(long long_1, int int_1, boolean boolean_1) { long long_2 = ChunkSectionPos.fromGlobalPos(long_1); Direction[] var7 = DIRECTIONS; int var8 = var7.length; for(int var9 = 0; var9 < var8; ++var9) { Direction direction_1 = var7[var9]; long long_3 = BlockPos.offset(long_1, direction_1); long long_4 = ChunkSectionPos.fromGlobalPos(long_3); if (long_2 == long_4 || ((BlockLightStorage)this.lightStorage).hasLight(long_4)) { this.propagateLevel(long_1, long_3, int_1, boolean_1); } } }
- Observe that the light shines through the stone blocks
The two main issues with PalettedContainer are that
- the underlying data storage is not atomic since block ids can be spread over multiple integer entries, hence non-meaningful block ids can be returned
- resizing the block state palette is not atomic. During resizing, the data stroage and palette field get replaced by new instances and afterwards the data is copied over (and the block ids are even shuffled). This opens the possibility for a reading thread to observe not yet copied entries or an inconsistent storage-palette pair, meaning that the block id is looked up using the wrong palette.
In the examples above, the second problem caused the lighting thread to observe only air blocks since we timed the threads so that the palette was being resized while the lighting thread was reading. Hence the sea lantern didn't emit light (as it was seen as an air block) and the stone blocks didn't block the light.
While this might not be the most game breaking bug, the fact that it's hardly reproducible makes it nevertheless quite annoying, in my opinion. That is because you won't really be able to track some randomly appearing issue down to this bug for the lack of reproducibility.
I have assembled a more detailed description and an implementation for an efficient lock-free solution at https://github.com/OverengineeredCodingDuo/mcoptimizations/tree/blockstatecontainer (see BlockStateContainer.java.patch as a starting point). The patches were written for 1.13.2, but the relevant code hasn't changed since.In my opinion, the solution is quite short and simple and definitely worth the effort, given that it can save some headaches in the future, especially in case you plan to multithread other parts of the engine.
Please don't hesitate to ask if you have any questions
Best,
PhiPro
Albeit being used concurrently by multiple threads, PalettedContainer isn't threadsafe. This can cause reading threads to see non-meaningful values, in particular Chunk.getBlockState(...) can return block states that never existed at the respective block pos (in the current session of the game).
As an example, this can cause issues with the new lighting engine which runs concurrently to the rest of the game logic. As multithreading issues are hardly reliably reproducible, the following steps use breakpoints to pause threads in order to simulate some bad luck. However, I was also able to reproduce the bugs even without breakpoints, just not reliably.
- Create a new redstone-ready world
- Set a breakpoint somewhere in the lighting code to prevent the next step from finishing/starting the light propagation. You should configure the breakpoint to only pause the thread triggering it, instead of pausing all threads. For example, you can set a breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected int getPropagatedLevel(long long_1, long long_2, int int_1) { if (long_2 == Long.MAX_VALUE) { return 15; } else if (long_1 == Long.MAX_VALUE) { return int_1 + 15 - this.getLightSourceLuminance(long_2); } else if (int_1 >= 15) { return int_1; } else { int int_2 = Integer.signum(BlockPos.unpackLongX(long_2) - BlockPos.unpackLongX(long_1)); int int_3 = Integer.signum(BlockPos.unpackLongY(long_2) - BlockPos.unpackLongY(long_1)); int int_4 = Integer.signum(BlockPos.unpackLongZ(long_2) - BlockPos.unpackLongZ(long_1)); Direction direction_1 = Direction.fromVector(int_2, int_3, int_4); if (direction_1 == null) { return 15; } else { AtomicInteger atomicInteger_1 = new AtomicInteger(); BlockState blockState_1 = this.getStateForLighting(long_2, atomicInteger_1); if (atomicInteger_1.get() >= 15) { return 15; } else { BlockState blockState_2 = this.getStateForLighting(long_1, (AtomicInteger)null); VoxelShape voxelShape_1 = this.getOpaqueShape(blockState_2, long_1, direction_1); VoxelShape voxelShape_2 = this.getOpaqueShape(blockState_1, long_2, direction_1.getOpposite()); return VoxelShapes.unionCoversFullCube(voxelShape_1, voxelShape_2) ? 15 : int_1 + Math.max(1, atomicInteger_1.get()); } } } }
- /setblock 7 60 7 minecraft:sea_lantern
- Some worker thread (and the rendering thread) should now hit the breakpoint. You can now disable the breakpoint and resume the rendering thread, but not the worker thread.
- Set a breakpoint at the last instruction of the following method, again configuring it to only pause the triggering thread
net.minecraft.world.chunk.PalettedContainer.javaprivate void setPaletteSize(int int_1) { if (int_1 != this.paletteSize) { this.paletteSize = int_1; if (this.paletteSize <= 4) { this.paletteSize = 4; this.palette = new ArrayPalette(this.idList, this.paletteSize, this, this.elementDeserializer); } else if (this.paletteSize < 9) { this.palette = new BiMapPalette(this.idList, this.paletteSize, this, this.elementDeserializer, this.elementSerializer); } else { this.palette = this.fallbackPalette; this.paletteSize = MathHelper.log2DeBrujin(this.idList.size()); } this.palette.getIndex(this.field_12935); this.data = new PackedIntegerArray(this.paletteSize, 4096); } }
- Place 14 different blocks (other than stone, sandstone and sea lanterns) in the 0 3 0 subchunk (where the sea lantern was placed)
- This should trigger the second breakpoint from the server thread (and also the rendering thread). Now resume the lighting worker thread. Afterwards, you can disable the second breakpoint and resume all threads.
- Restart the world
- Observe that the sea lantern does not emit light
Similarly, you can make blocks transparent
- Create a new redstone-ready world
- /fill 6 63 6 8 66 8 minecraft:stone hollow
- Procede as above, but instead set the sea lantern at /setblock 7 65 7 minecraft:sea_lantern and set the first breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected void propagateLevel(long long_1, int int_1, boolean boolean_1) { long long_2 = ChunkSectionPos.fromGlobalPos(long_1); Direction[] var7 = DIRECTIONS; int var8 = var7.length; for(int var9 = 0; var9 < var8; ++var9) { Direction direction_1 = var7[var9]; long long_3 = BlockPos.offset(long_1, direction_1); long long_4 = ChunkSectionPos.fromGlobalPos(long_3); if (long_2 == long_4 || ((BlockLightStorage)this.lightStorage).hasLight(long_4)) { this.propagateLevel(long_1, long_3, int_1, boolean_1); } } }
- Observe that the light shines through the stone blocks
The two main issues with PalettedContainer are that
- the underlying data storage is not atomic since block ids can be spread over multiple integer entries, hence non-meaningful block ids can be returned
- resizing the block state palette is not atomic. During resizing, the data stroage and palette field get replaced by new instances and afterwards the data is copied over (and the block ids are even shuffled). This opens the possibility for a reading thread to observe not yet copied entries or an inconsistent storage-palette pair, meaning that the block id is looked up using the wrong palette.
In the examples above, the second problem caused the lighting thread to observe only air blocks since we timed the threads so that the palette was being resized while the lighting thread was reading. Hence the sea lantern didn't emit light (as it was seen as an air block) and the stone blocks didn't block the light.
While this might not be the most game breaking bug, the fact that it's hardly reproducible makes it nevertheless quite annoying, in my opinion. That is because you won't really be able to track some randomly appearing issue down to this bug for the lack of reproducibility.
I have assembled a more detailed description and an implementation for an efficient lock-free solution at https://github.com/OverengineeredCodingDuo/mcoptimizations/tree/blockstatecontainer (see BlockStateContainer.java.patch as a starting point). The patches were written for 1.13.2, but the relevant code hasn't changed since.
In my opinion, the solution is quite short and simple and definitely worth the effort, given that it can save some headaches in the future, especially in case you plan to multithread other parts of the engine.Please don't hesitate to ask if you have any questions
Best,
PhiProAlbeit being used concurrently by multiple threads, PalettedContainer isn't threadsafe. This can cause reading threads to see non-meaningful values, in particular Chunk.getBlockState(...) can return block states that never existed at the respective block pos (in the current session of the game).
As an example, this can cause issues with the new lighting engine which runs concurrently to the rest of the game logic. As multithreading issues are hardly reliably reproducible, the following steps use breakpoints to pause threads in order to simulate some bad luck. However, I was also able to reproduce the bugs even without breakpoints, just not reliably.
- Create a new redstone-ready world
- Set a breakpoint somewhere in the lighting code to prevent the next step from finishing/starting the light propagation. You should configure the breakpoint to only pause the thread triggering it, instead of pausing all threads. For example, you can set a breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected int getPropagatedLevel(long long_1, long long_2, int int_1) { if (long_2 == Long.MAX_VALUE) { return 15; } else if (long_1 == Long.MAX_VALUE) { return int_1 + 15 - this.getLightSourceLuminance(long_2); } else if (int_1 >= 15) { return int_1; } else { int int_2 = Integer.signum(BlockPos.unpackLongX(long_2) - BlockPos.unpackLongX(long_1)); int int_3 = Integer.signum(BlockPos.unpackLongY(long_2) - BlockPos.unpackLongY(long_1)); int int_4 = Integer.signum(BlockPos.unpackLongZ(long_2) - BlockPos.unpackLongZ(long_1)); Direction direction_1 = Direction.fromVector(int_2, int_3, int_4); if (direction_1 == null) { return 15; } else { AtomicInteger atomicInteger_1 = new AtomicInteger(); BlockState blockState_1 = this.getStateForLighting(long_2, atomicInteger_1); if (atomicInteger_1.get() >= 15) { return 15; } else { BlockState blockState_2 = this.getStateForLighting(long_1, (AtomicInteger)null); VoxelShape voxelShape_1 = this.getOpaqueShape(blockState_2, long_1, direction_1); VoxelShape voxelShape_2 = this.getOpaqueShape(blockState_1, long_2, direction_1.getOpposite()); return VoxelShapes.unionCoversFullCube(voxelShape_1, voxelShape_2) ? 15 : int_1 + Math.max(1, atomicInteger_1.get()); } } } }
- /setblock 7 60 7 minecraft:sea_lantern
- Some worker thread (and the rendering thread) should now hit the breakpoint. You can now disable the breakpoint and resume the rendering thread, but not the light worker thread.
- Set a breakpoint at the last instruction of the following method, again configuring it to only pause the triggering thread
net.minecraft.world.chunk.PalettedContainer.javaprivate void setPaletteSize(int int_1) { if (int_1 != this.paletteSize) { this.paletteSize = int_1; if (this.paletteSize <= 4) { this.paletteSize = 4; this.palette = new ArrayPalette(this.idList, this.paletteSize, this, this.elementDeserializer); } else if (this.paletteSize < 9) { this.palette = new BiMapPalette(this.idList, this.paletteSize, this, this.elementDeserializer, this.elementSerializer); } else { this.palette = this.fallbackPalette; this.paletteSize = MathHelper.log2DeBrujin(this.idList.size()); } this.palette.getIndex(this.field_12935); this.data = new PackedIntegerArray(this.paletteSize, 4096); } }
- Place 14 different blocks (other than stone, sandstone and sea lanterns) in the 0 3 0 subchunk (where the sea lantern was placed)
- This should trigger the second breakpoint from the server thread (and also the rendering thread). Now resume the lighting worker thread. Afterwards, you can disable the second breakpoint and resume all threads.
- Restart the world
- Observe that the sea lantern does not emit light
Similarly, you can make blocks transparent
- Create a new redstone-ready world
- /fill 6 63 6 8 66 8 minecraft:stone hollow
- Procede as above, but instead set the sea lantern at /setblock 7 65 7 minecraft:sea_lantern and set the first breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected void propagateLevel(long long_1, int int_1, boolean boolean_1) { long long_2 = ChunkSectionPos.fromGlobalPos(long_1); Direction[] var7 = DIRECTIONS; int var8 = var7.length; for(int var9 = 0; var9 < var8; ++var9) { Direction direction_1 = var7[var9]; long long_3 = BlockPos.offset(long_1, direction_1); long long_4 = ChunkSectionPos.fromGlobalPos(long_3); if (long_2 == long_4 || ((BlockLightStorage)this.lightStorage).hasLight(long_4)) { this.propagateLevel(long_1, long_3, int_1, boolean_1); } } }
- Observe that the light shines through the stone blocks
The two main issues with PalettedContainer are that
- the underlying data storage is not atomic since block ids can be spread over multiple integer entries, hence non-meaningful block ids can be returned
- resizing the block state palette is not atomic. During resizing, the data stroage and palette field get replaced by new instances and afterwards the data is copied over (and the block ids are even shuffled). This opens the possibility for a reading thread to observe not yet copied entries or an inconsistent storage-palette pair, meaning that the block id is looked up using the wrong palette.
In the examples above, the second problem caused the lighting thread to observe only air blocks since we timed the threads so that the palette was being resized while the lighting thread was reading. Hence the sea lantern didn't emit light (as it was seen as an air block) and the stone blocks didn't block the light.
While this might not be the most game breaking bug, the fact that it's hardly reproducible makes it nevertheless quite annoying, in my opinion. That is because you won't really be able to track some randomly appearing issue down to this bug for the lack of reproducibility.
I have assembled a more detailed description and an implementation for an efficient lock-free solution at https://github.com/OverengineeredCodingDuo/mcoptimizations/tree/blockstatecontainer (see BlockStateContainer.java.patch as a starting point). The patches were written for 1.13.2, but the relevant code hasn't changed since.In my opinion, the solution is quite short and simple and definitely worth the effort, given that it can save some headaches in the future, especially in case you plan to multithread other parts of the engine.
Please don't hesitate to ask if you have any questions
Best,
PhiPro
Albeit being used concurrently by multiple threads, PalettedContainer isn't threadsafe. This can cause reading threads to see non-meaningful values, in particular Chunk.getBlockState(...) can return block states that never existed at the respective block pos (in the current session of the game).
As an example, this can cause issues with the new lighting engine which runs concurrently to the rest of the game logic. As multithreading issues are hardly reliably reproducible, the following steps use breakpoints to pause threads in order to simulate some bad luck. However, I was also able to reproduce the bugs even without breakpoints, just not reliably.
- Create a new redstone-ready world
- Set a breakpoint somewhere in the lighting code to prevent the next step from finishing/starting the light propagation. You should configure the breakpoint to only pause the thread triggering it, instead of pausing all threads. For example, you can set a breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected int getPropagatedLevel(long long_1, long long_2, int int_1) { if (long_2 == Long.MAX_VALUE) { return 15; } else if (long_1 == Long.MAX_VALUE) { return int_1 + 15 - this.getLightSourceLuminance(long_2); } else if (int_1 >= 15) { return int_1; } else { int int_2 = Integer.signum(BlockPos.unpackLongX(long_2) - BlockPos.unpackLongX(long_1)); int int_3 = Integer.signum(BlockPos.unpackLongY(long_2) - BlockPos.unpackLongY(long_1)); int int_4 = Integer.signum(BlockPos.unpackLongZ(long_2) - BlockPos.unpackLongZ(long_1)); Direction direction_1 = Direction.fromVector(int_2, int_3, int_4); if (direction_1 == null) { return 15; } else { AtomicInteger atomicInteger_1 = new AtomicInteger(); BlockState blockState_1 = this.getStateForLighting(long_2, atomicInteger_1); if (atomicInteger_1.get() >= 15) { return 15; } else { BlockState blockState_2 = this.getStateForLighting(long_1, (AtomicInteger)null); VoxelShape voxelShape_1 = this.getOpaqueShape(blockState_2, long_1, direction_1); VoxelShape voxelShape_2 = this.getOpaqueShape(blockState_1, long_2, direction_1.getOpposite()); return VoxelShapes.unionCoversFullCube(voxelShape_1, voxelShape_2) ? 15 : int_1 + Math.max(1, atomicInteger_1.get()); } } } }
- /setblock 7 60 7 minecraft:sea_lantern
- Some worker thread (and the rendering thread) should now hit the breakpoint. You can now disable the breakpoint and resume the rendering thread, but not the light worker thread.
- Set a breakpoint at the last instruction of the following method, again configuring it to only pause the triggering thread
net.minecraft.world.chunk.PalettedContainer.javaprivate void setPaletteSize(int int_1) { if (int_1 != this.paletteSize) { this.paletteSize = int_1; if (this.paletteSize <= 4) { this.paletteSize = 4; this.palette = new ArrayPalette(this.idList, this.paletteSize, this, this.elementDeserializer); } else if (this.paletteSize < 9) { this.palette = new BiMapPalette(this.idList, this.paletteSize, this, this.elementDeserializer, this.elementSerializer); } else { this.palette = this.fallbackPalette; this.paletteSize = MathHelper.log2DeBrujin(this.idList.size()); } this.palette.getIndex(this.field_12935); this.data = new PackedIntegerArray(this.paletteSize, 4096); } }
- Place 14 different blocks (other than stone, sandstone and sea lanterns) in the 0 3 0 subchunk (where the sea lantern was placed)
- This should trigger the second breakpoint from the server thread (and also the rendering thread). Now resume the lighting worker thread. Afterwards, you can disable the second breakpoint and resume all threads.
- Restart the world
- Observe that the sea lantern does not emit light
Similarly, you can make blocks transparent
- Create a new redstone-ready world
- /fill 6 63 6 8 66 8 minecraft:stone hollow
- Procede as above, but instead set the sea lantern at /setblock 7 65 7 minecraft:sea_lantern and set the first breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected void propagateLevel(long long_1, int int_1, boolean boolean_1) { long long_2 = ChunkSectionPos.fromGlobalPos(long_1); Direction[] var7 = DIRECTIONS; int var8 = var7.length; for(int var9 = 0; var9 < var8; ++var9) { Direction direction_1 = var7[var9]; long long_3 = BlockPos.offset(long_1, direction_1); long long_4 = ChunkSectionPos.fromGlobalPos(long_3); if (long_2 == long_4 || ((BlockLightStorage)this.lightStorage).hasLight(long_4)) { this.propagateLevel(long_1, long_3, int_1, boolean_1); } } }
- Observe that the light shines through the stone blocks
The two main issues with PalettedContainer are that
- the underlying data storage is not atomic since block ids can be spread over multiple integer entries, hence non-meaningful block ids can be returned
- resizing the block state palette is not atomic. During resizing, the data st
roage and palette field get replaced by new instances and afterwards the data is copied over (and the block ids are even shuffled). This opens the possibility for a reading thread to observe not yet copied entries or an inconsistent storage-palette pair, meaning that the block id is looked up using the wrong palette.In the examples above, the second problem caused the lighting thread to observe only air blocks since we timed the threads so that the palette was being resized while the lighting thread was reading. Hence the sea lantern didn't emit light (as it was seen as an air block) and the stone blocks didn't block the light.
While this might not be the most game breaking bug, the fact that it's hardly reproducible makes it nevertheless quite annoying, in my opinion. That is because you won't really be able to track some randomly appearing issue down to this bug for the lack of reproducibility.
I have assembled a more detailed description and an implementation for an efficient lock-free solution at https://github.com/OverengineeredCodingDuo/mcoptimizations/tree/blockstatecontainer (see BlockStateContainer.java.patch as a starting point). The patches were written for 1.13.2, but the relevant code hasn't changed since.In my opinion, the solution is quite short and simple and definitely worth the effort, given that it can save some headaches in the future, especially in case you plan to multithread other parts of the engine.
Please don't hesitate to ask if you have any questions
Best,
PhiProAlbeit being used concurrently by multiple threads, PalettedContainer isn't threadsafe. This can cause reading threads to see non-meaningful values, in particular Chunk.getBlockState(...) can return block states that never existed at the respective block pos (in the current session of the game).
As an example, this can cause issues with the new lighting engine which runs concurrently to the rest of the game logic. As multithreading issues are hardly reliably reproducible, the following steps use breakpoints to pause threads in order to simulate some bad luck. However, I was also able to reproduce the bugs even without breakpoints, just not reliably.
- Create a new redstone-ready world
- Set a breakpoint somewhere in the lighting code to prevent the next step from finishing/starting the light propagation. You should configure the breakpoint to only pause the thread triggering it, instead of pausing all threads. For example, you can set a breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected int getPropagatedLevel(long long_1, long long_2, int int_1) { if (long_2 == Long.MAX_VALUE) { return 15; } else if (long_1 == Long.MAX_VALUE) { return int_1 + 15 - this.getLightSourceLuminance(long_2); } else if (int_1 >= 15) { return int_1; } else { int int_2 = Integer.signum(BlockPos.unpackLongX(long_2) - BlockPos.unpackLongX(long_1)); int int_3 = Integer.signum(BlockPos.unpackLongY(long_2) - BlockPos.unpackLongY(long_1)); int int_4 = Integer.signum(BlockPos.unpackLongZ(long_2) - BlockPos.unpackLongZ(long_1)); Direction direction_1 = Direction.fromVector(int_2, int_3, int_4); if (direction_1 == null) { return 15; } else { AtomicInteger atomicInteger_1 = new AtomicInteger(); BlockState blockState_1 = this.getStateForLighting(long_2, atomicInteger_1); if (atomicInteger_1.get() >= 15) { return 15; } else { BlockState blockState_2 = this.getStateForLighting(long_1, (AtomicInteger)null); VoxelShape voxelShape_1 = this.getOpaqueShape(blockState_2, long_1, direction_1); VoxelShape voxelShape_2 = this.getOpaqueShape(blockState_1, long_2, direction_1.getOpposite()); return VoxelShapes.unionCoversFullCube(voxelShape_1, voxelShape_2) ? 15 : int_1 + Math.max(1, atomicInteger_1.get()); } } } }
- /setblock 7 60 7 minecraft:sea_lantern
- Some worker thread (and the rendering thread) should now hit the breakpoint. You can now disable the breakpoint and resume the rendering thread, but not the light worker thread.
- Set a breakpoint at the last instruction of the following method, again configuring it to only pause the triggering thread
net.minecraft.world.chunk.PalettedContainer.javaprivate void setPaletteSize(int int_1) { if (int_1 != this.paletteSize) { this.paletteSize = int_1; if (this.paletteSize <= 4) { this.paletteSize = 4; this.palette = new ArrayPalette(this.idList, this.paletteSize, this, this.elementDeserializer); } else if (this.paletteSize < 9) { this.palette = new BiMapPalette(this.idList, this.paletteSize, this, this.elementDeserializer, this.elementSerializer); } else { this.palette = this.fallbackPalette; this.paletteSize = MathHelper.log2DeBrujin(this.idList.size()); } this.palette.getIndex(this.field_12935); this.data = new PackedIntegerArray(this.paletteSize, 4096); } }
- Place 14 different blocks (other than stone, sandstone and sea lanterns) in the 0 3 0 subchunk (where the sea lantern was placed)
- This should trigger the second breakpoint from the server thread (and also the rendering thread). Now resume the lighting worker thread. Afterwards, you can disable the second breakpoint and resume all threads.
- Restart the world
- Observe that the sea lantern does not emit light
Similarly, you can make blocks transparent
- Create a new redstone-ready world
- /fill 6 63 6 8 66 8 minecraft:stone hollow
- Procede as above, but instead set the sea lantern at /setblock 7 65 7 minecraft:sea_lantern and set the first breakpoint at the first line of
net.minecraft.world.chunk.light.ChunkBlockLightProvider.javaprotected void propagateLevel(long long_1, int int_1, boolean boolean_1) { long long_2 = ChunkSectionPos.fromGlobalPos(long_1); Direction[] var7 = DIRECTIONS; int var8 = var7.length; for(int var9 = 0; var9 < var8; ++var9) { Direction direction_1 = var7[var9]; long long_3 = BlockPos.offset(long_1, direction_1); long long_4 = ChunkSectionPos.fromGlobalPos(long_3); if (long_2 == long_4 || ((BlockLightStorage)this.lightStorage).hasLight(long_4)) { this.propagateLevel(long_1, long_3, int_1, boolean_1); } } }
- Observe that the light shines through the stone blocks
The two main issues with PalettedContainer are that
- the underlying data storage is not atomic since block ids can be spread over multiple integer entries, hence non-meaningful block ids can be returned
- resizing the block state palette is not atomic. During resizing, the data storage and palette field get replaced by new instances and afterwards the data is copied over (and the block ids are even shuffled). This opens the possibility for a reading thread to observe not yet copied entries or an inconsistent storage-palette pair, meaning that the block id is looked up using the wrong palette.
In the examples above, the second problem caused the lighting thread to observe only air blocks since we timed the threads so that the palette was being resized while the lighting thread was reading. Hence the sea lantern didn't emit light (as it was seen as an air block) and the stone blocks didn't block the light.
While this might not be the most game breaking bug, the fact that it's hardly reproducible makes it nevertheless quite annoying, in my opinion. That is because you won't really be able to track some randomly appearing issue down to this bug for the lack of reproducibility.
I have assembled a more detailed description and an implementation for an efficient lock-free solution at https://github.com/OverengineeredCodingDuo/mcoptimizations/tree/blockstatecontainer (see BlockStateContainer.java.patch as a starting point). The patches were written for 1.13.2, but the relevant code hasn't changed since.In my opinion, the solution is quite short and simple and definitely worth the effort, given that it can save some headaches in the future, especially in case you plan to multithread other parts of the engine.
Please don't hesitate to ask if you have any questions
Best,
PhiPro
The overall issue dealt with in this report is that block changes are not properly synchronized with the lighting engine; in particular they are not properly synchronized with the lightmap handling which determines the region the lightign engine can operate on and t
eh region where skylight optimizations are applicable. As shown below, this can lead to persistent lighting glitches.
In the following we'll look into 3 variants of this general issue that are relevant for my proposed solution. I attached a world download containing a simple setup for all 3 problems with the labeling as below. Due to the multithreading nature of the problem, the issues are not nocessarily reliably reproducable, although they were on my system; it would be nice if someone else could confirm that the setup works on their system. Below I'll try to extract the relevant points.A boring solution to these issues would be to employ some copy-on-write data structure to synchronize block changes to the lighting thread in a more controlled fashion. However, that might add quite a lot of overhead in order to fix some bug that might not occur that often in practice. The solution developed below should be much more efficient and in fact not very complicated.
While the first issue is not directly a multithreading issue, it is still relevant for the solution of the general bug.
Problem 1: Lightmaps get removed too early
When the last block of a subchunk gets removed, the lighting engine is told to remove the corresponding lightmap (if there are no nearby blocks). Only afterwards it is told to reevaluate the removed block. However, at this point the lightmap was already removed, so the lighting engine simply ignores the light check; even if it wouldn't be ignored, all the relevant information would be erased.
This can be visualized as follows:
- Create a new redstone-ready world
- /setblock 7 82 7 minecraft:sea_lantern
- destroy the sea lantern
- Observe that the block light value is still 15. This is actually not directly the bug we are after. This part of the issue will disappear upon reloading the world or triggering some block light update. We will come back to this later.
- Reload the world to fix the previous point.
- Note that the light is still broken in subchunk 0 4 0
As soon as we break the sea lantern, the lightmap of that subchunk gets removed, so the lighting engine cannot process the block update to reduce the blocklight level. Hence the light stays in subchunk 0 4 0 which doesn't have its lightmap removed because of nearby blocks. The blocklight level for subchunk 0 5 0 is set to 0 by removing the lightmap, as we see after relaoding the world.
The relevant code pieces are
net.minecraft.world.chunk.WorldChunk.java@Nullable public BlockState setBlockState(BlockPos pos, BlockState state, boolean bl) { ... boolean bl2 = chunkSection.isEmpty(); BlockState blockState = chunkSection.setBlockState(i, j & 15, k, state); ... boolean bl3 = chunkSection.isEmpty(); if (bl2 != bl3) { this.world.getChunkManager().getLightingProvider().updateSectionStatus(pos, bl3); } ... }net.minecraft.world.World.javapublic boolean setBlockState(BlockPos pos, BlockState state, int flags) { .... BlockState blockState = worldChunk.setBlockState(pos, state, (flags & 64) != 0); ... this.profiler.push("queueCheckLight"); this.getChunkManager().getLightingProvider().checkBlock(pos); this.profiler.pop(); } ...Chunk.setBlockState(...) calls updateSectionStatus(...) to issue the creation and removal of lightmaps to the lighting engine. World.setBlockState(...) first calls Chunk.setBlockState(...) and only afterwards checkBlock(pos) which processes the block update for the lighting engine.
To fix this issue, we should first issue the lighting check and only afterwards the removal of the lightmap. By a similar argument, we should first issue the creation of a lightmap and then the lighting check for a newly added block. As we will see later, we must in fact issue the creation of a lightmap before changing the block via chunkSection.setBlockState(...).
Hence I propose to move the lightmap handling (updateSectionStatus(...)) to World.setBlockState(...) and change the order to
- issue lightmap creations if necessary
- call Chunk.setBlockState(...)
- issue lig
th checks- issue lightmap removals if necessary
Also, the lighting engine should process the issued operations in this order. Keeping only a single call to updateSectionStatus(...) as in the current code and adding some priority system to achieve the right order might work in the single threaded case, but not in the multithreaded case, as one operation might be processed before the others get issued.There is a similar version for skylight included in the attached world, however it requires some more operations as there is some special handling for adding and removing skylight maps.
For completeness, let's discuss why in the example above the light stayed in subchunk 0 5 0 until reloading the world. The main thread does not read light values directly from the internal lightmaps, but gets its own immutable copies that are updated inside
net.minecraft.world.chunk.light.LightStorage.javaprotected void notifyChunkProvider() { if (!this.field_15802.isEmpty()) { M chunkToNibbleArrayMap = this.lightArrays.copy(); chunkToNibbleArrayMap.disableCache(); this.uncachedLightArrays = chunkToNibbleArrayMap; ... }field_15802 gets populated whenever a light value changes or a new lightmap is added (which can contain precalculated light values and hence cause a change of light values as well), but not when a lightmap is removed. Hence the immutable copy of the lightmaps does not get updated until reloading the world or triggering some blocklight update, as observed above.
By itself this isn't really a bug. If everything else worked correctly, lightmap removal shouldn't change light values, hence there shouldn't be a reason to create a new copy of the lightmaps, other than freeing some memory.Images produced by the attached world:
Problem 2: Blocks can be seen too early
Since the lighting engine reads blocks directly from the main thread world, it can see block changes "immediately". In particular, it can see blocks before all relevant lightmaps have been created; even though lightmap creation should be issued before the actual block change is carried out, as noted above, the lighting engine might encounter a new block while processing other light updates. Although the issued lightmap creation will be visible to the lighting thread at that point (by looking at the queued operations), there is no reason why this operation should have been processed. Even if we processed all pending operations just before the block lookup, it could happen that the block gets placed and the lightmap creation gets issued inbetween those two operations.
This can lead to the problem that the lighting engine can see and take into account newly placed blocks although the relevant adjacent lightmaps, which that block can affect, have not yet been created, so the light updates will get stuck at those neighbor subchunks. Later on, the relevant lightmaps will be added and a light check for the newly added block will be carried out. However, the light is already correct around this block; the lighting errors occur at the boundary to the neighbor subchunk. Hence, the light check at the newly added block will succeed and no further updates will be carried out, keeping the errors at the chunk boundary.
For skylight this also causes the skylight optimization to be applied wrongly; the optimization only works 16 tiles away from non-air blocks, so is not applicable around those newly placed blocks. But since the lighting engine has not been told yet about those new non-air blocks, it might apply the optimization nevertheless, causing errors. This is what we will demonstrate in the following.
The setup is as follows:
We have some large stone platform high up in the sky (at y=255 in order to cause long light propagation times). Subchunk 1 6 1 has a lightmap that is kept alive by nearby blocks, but subchunk 1 5 1 doesn't. Inside a single tick
- punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of other lightmap creations and light updates, so the priority system doesn't accidentally move the next step before processing the updates of the previous one. Don't cause too many updates, as otherwise the light propagation of the previous step will be finished before we can do cool stuff.
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next step until done
- Place a platform of leave blocks in subchunk 1 6 1 below the hole
- On its way down, the light will now see the leave blocks and correctly incorporate them into the result. As subchunk {{ 1 5 1}} doesn't have a lightmap, the skylight optimization will kick in and simply copy down the lig
th values from subchunk 1 6 1.
After passing the leave blocks, the light should decay to 0 after 15 blocks. But because of the wrongly applied optimization, it travels a whole 16 blocks (or arbitrarily many) without any decay.- Now the lighting engine adds the lightmap for subchunk 1 5 1 which is initialized by copying down the values from subchunk 1 6 1, ie. the values as produced by the skylight optimization. Also, light checks are carried out for the leave blocks. But as the light was already correct around those, nothing changes and the light in subchunk 1 5 1 stays broken.
- Add some blocks in subchunk 1 5 1 to better visualize the issue (F3 would work as well)
What we learn from this is that we should make sure that the lighting engine doesn't see any non-air blocks for subchunks that it has been told to be empty, as this can conflict with the skylight optimization.
The lighting engine already keeps track of which subchunks should be empty according to the main thrad and spawns lightmaps accordingly. I propose to employ this information inside the block lookup used by the lighting engine: First test whether the queried subchunk should be empty and return air in that case. Otherwise, proceed as normal. This prevents the described issue that blocks get seen before all relevant lightmaps have been added.Unfortunately, things can get a bit more delicate as Problem 3 shows.
Problem 3: Lightmap removal must be cancellable
The basic issue encountered here is that lightmap removal should be canceled if a subchunk gets repopulated before the lightmap is actually removed, rather than removing the lightmap and then recreating it, as this would erase information.
In order to get a better understanding, let's look at the setup in the attached world. The situation is mainly the same as in Problem 2: we have a stone platform high up in the sky and subchunk 1 6 4 has a light map. This time subchunk 1 5 4 has a ligth map that is kept alive by some nearby block, as well. Similarly as above, inside a single tick
- punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of lightmap creations and light updates to prevent priority reordering
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next steps until done
- As in Problem 2, we will spawn a leave platform while the lighting engine is propagating the light downwards in order to cause some issues. However, we want the lighting engine to remove the lightmap for subchunk 1 5 4 before it gets kept alive by the leave platform, but after propagating the light downwards, in order to produce the issue outlined above. Hence we need some other steps before spawning the leave platform.
Now, issue removal for the lightmap of subchunk 1 5 4 by deleting the blocks that keep it alive, but in such a way that the lightmap for subchunk 1 6 4 does not get removed.- Again cause some more lightmap creations and lig
th updates to prevent priority reordering- Now spawn the leave platform in subchunk 1 6 4 below the hole
- On its way down, the lighting engine will now see and incorporate the leaves blocks, as in Problem 2. This time, however, subchunk 1 5 4 does have a lightmap, so we don't wrongly apply the skylight optimization, but correctly propagate the skylight to subchunk 1 5 4. Subchunk 1 4 4 now correctly stays dark.
- Now the lighting engine processes the issued removal of the lightmap at 1 5 4. The issued recreation of the lightmap through the leave platform is not yet processed, as we issued a bunch of other lightmap creations, so it is still stuck in the queue.
- Now we finally process the issued recreation of the lightmap at 1 5 4. However, at this point the lighting information has already been erased and the new lightmap for 1 5 4 will be initialized by copying down the values from 1 6 4, hence the light below the leave platform does not decay, looking as in Problem 2. However, subchunk 1 4 4 will stay dark, in contrast to Problem 2, showing that we indeed observe a different issue here.
This issue is a mixture of Problem 1 and 2. In some sense, the issue is that the lighting engine sees the leave blocks before it is told that subchunk 1 5 4 is not empty (ie. before it processes the lightmap creations). However, the solution for Problem 2 of simply treating that block as air does not apply here, because at the time the leave block is seen, that subchunk is not marked empty, but gets only marked empty later on. In some sense this is also related to Problem 1, as lightmaps that are relevant to the leave blocks get removed after the lighting engine sees the leave blocks but before it processes the removal of the leave blocks.
The general lesson learned here is that the lightmap creation issued by spawning the leave platform should cancel the previous removals of lightmaps.The precise order of operations needed is summarized in the following section.
Proposed Solution
As noted above, the main thread should carry out operations in the following order:
- issue an operation to mark the subchunk as non-empty, as necessary
- call Chunk.setBlockState(...)
- issue lig
th checks- issue an operation to mark the subchunk as empty, as necessary
It will become clear in a moment why the subchunk must be marked non-empty not only before the light check, but also before the block gets written to the chunk.
For block lookups, the lighting engine should first test whether the queried subchunk should be empty, according to its current knowledge, and in this case return air.
The lighting engine should carry out operations in the following order:
- When starting an update cycle, take a snapshot of the operation queue and don't process any later updates. This makes sure that we don't process lig
th checks without processing the corresponding lightmap creation or lightmap removals without the corresponding light checks.- Process all operations that mark subchunks as non-empty
- Process all light checks and propagate all light updates
- In case that only a limited number of light updates is carried out at a time, finish the current set of updates before processing any new operation
- After all pending updates have been processed, process the operations that mark subchunks as empty. In this step, we need to take another snapshot of the operation queue and search it for operations that mark subchunks as non-empty, and cancel operations accordingly. This is needed as shown by Problem 3.
The correctness of this last step requires some small argument, as follows. Suppose some subchunk has been marked as empty. We consider two cases:
- The lighting engine has not seen any block that was added to the subchunk after the main thread issued the operation to mark it as empty. In this case, we can safely mark this subchunk as empty, since we processed all pending lig
th updates that were caused by the removal of blocks and no new blocks have been observed.- The lighting engine does see some block that was added to the subchunk after the main thread issued the operation to mark it as empty. Now we are in the situation of Problem 3. The important point is now that we can use acquire-release semantics on reading this newly added block and the corresponding write on the main thread. Because the operation to mark subchunks as non-empty is issued before writing the block
, acquire-release semantics make sure that we will see this operation in the operation queue after observing the block. Hence we will find the operation in the second snapshot of the operation queue and correctly cancel the operation.
In fact, acquire-release semantics are not precisely guaranteed as reading blocks isn't even threadsafe (MC-162080), however in practice they will hold on x86 architectures as long asMC-162080doesn't occur.- Leave all remaining operations to mark subchunks as non-empty for the next cycle or process them right away.
Unfortunately, as far as I can tell, this is not so easily achievable with the current design of the multithreading code. That is because all operations are wrapped inside some generic runnable tasks and put into a single task queue, so it is not straightforward to scan this task queue without actually processing its elements.
I propose to introduce more specialzed "task queues" for different types of tasks which then can be scanned more easily. More concretely, I propose to add 3 sets (and possibly more for other stuff not covered here) lightChecks, markEmpty, markNonEmpty. When the lighting engine starts its update cycles, it takes ownership over all 3 sets and gives the main thread new ones to populate. Now we can easily process markNonEmpty and lightChecks. When processing markEmpty, we again take ownership over the new markNonEmpty and cancel any subchunks occuring in both. Now we can either merge the remaining operations of markNonEmpty back into the current set for the main thread to populate or simply keep it and merge it with the next update cycle.
The main thread then populates the sets as follows:
- use some lock, so that we don't write to the sets after the lighting engine has taken ownership
- add light checks to lightChecks (obviously
)
- when issuing a new empty subchunk, add it to markEmpty
- when issuing a new non-empty subchunk, first try to cancel a corresponding entry in markEmpty and otherwise add it to markNonEmpty.
- issuing an empty subchunk must not cancel any operation in markNonEmpty, as this operation might be needed to cancel a removal in the currently ongoing cycle. Canceling entries of markEmpty by non-empty subchunk, on the other hand, is unproblematic.
I hope this report was more or less understandable. If anything is unclear, please dont hesitate to ask
Best,
PhiProThe overall issue dealt with in this report is that block changes are not properly synchronized with the lighting engine; in particular they are not properly synchronized with the lightmap handling which determines the region the lightign engine can operate on and the region where skylight optimizations are applicable. As shown below, this can lead to persistent lighting glitches.
In the following we'll look into 3 variants of this general issue that are relevant for my proposed solution. I attached a world download containing a simple setup for all 3 problems with the labeling as below. Due to the multithreading nature of the problem, the issues are not necessarily reliably reproducible, although they were on my system; it would be nice if someone else could confirm that the setup works on their system. Below I'll try to extract the relevant points.A boring solution to these issues would be to employ some copy-on-write data structure to synchronize block changes to the lighting thread in a more controlled fashion. However, that might add quite a lot of overhead in order to fix some bug that might not occur that often in practice. The solution developed below should be much more efficient and in fact not very complicated.
While the first issue is not directly a multithreading issue, it is still relevant for the solution of the general bug.
Problem 1: Lightmaps get removed too early
When the last block of a subchunk gets removed, the lighting engine is told to remove the corresponding lightmap (if there are no nearby blocks). Only afterwards it is told to reevaluate the removed block. However, at this point the lightmap was already removed, so the lighting engine simply ignores the light check; even if it wouldn't be ignored, all the relevant information would be erased.
This can be visualized as follows:
- Create a new redstone-ready world
- /setblock 7 82 7 minecraft:sea_lantern
- destroy the sea lantern
- Observe that the block light value is still 15. This is actually not directly the bug we are after. This part of the issue will disappear upon reloading the world or triggering some block light update. We will come back to this later.
- Reload the world to fix the previous point.
- Note that the light is still broken in subchunk 0 4 0
As soon as we break the sea lantern, the lightmap of that subchunk gets removed, so the lighting engine cannot process the block update to reduce the blocklight level. Hence the light stays in subchunk 0 4 0 which doesn't have its lightmap removed because of nearby blocks. The blocklight level for subchunk 0 5 0 is set to 0 by removing the lightmap, as we see after relaoding the world.
The relevant code pieces are
net.minecraft.world.chunk.WorldChunk.java@Nullable public BlockState setBlockState(BlockPos pos, BlockState state, boolean bl) { ... boolean bl2 = chunkSection.isEmpty(); BlockState blockState = chunkSection.setBlockState(i, j & 15, k, state); ... boolean bl3 = chunkSection.isEmpty(); if (bl2 != bl3) { this.world.getChunkManager().getLightingProvider().updateSectionStatus(pos, bl3); } ... }net.minecraft.world.World.javapublic boolean setBlockState(BlockPos pos, BlockState state, int flags) { .... BlockState blockState = worldChunk.setBlockState(pos, state, (flags & 64) != 0); ... this.profiler.push("queueCheckLight"); this.getChunkManager().getLightingProvider().checkBlock(pos); this.profiler.pop(); } ...Chunk.setBlockState(...) calls updateSectionStatus(...) to issue the creation and removal of lightmaps to the lighting engine. World.setBlockState(...) first calls Chunk.setBlockState(...) and only afterwards checkBlock(pos) which processes the block update for the lighting engine.
To fix this issue, we should first issue the lighting check and only afterwards the removal of the lightmap. By a similar argument, we should first issue the creation of a lightmap and then the lighting check for a newly added block. As we will see later, we must in fact issue the creation of a lightmap before changing the block via chunkSection.setBlockState(...).
Hence I propose to move the lightmap handling (updateSectionStatus(...)) to World.setBlockState(...) and change the order to
- issue lightmap creations if necessary
- call Chunk.setBlockState(...)
- issue light checks
- issue lightmap removals if necessary
Also, the lighting engine should process the issued operations in this order. Keeping only a single call to updateSectionStatus(...) as in the current code and adding some priority system to achieve the right order might work in the single threaded case, but not in the multithreaded case, as one operation might be processed before the others get issued.There is a similar version for skylight included in the attached world, however it requires some more operations as there is some special handling for adding and removing skylight maps.
For completeness, let's discuss why in the example above the light stayed in subchunk 0 5 0 until reloading the world. The main thread does not read light values directly from the internal lightmaps, but gets its own immutable copies that are updated inside
net.minecraft.world.chunk.light.LightStorage.javaprotected void notifyChunkProvider() { if (!this.field_15802.isEmpty()) { M chunkToNibbleArrayMap = this.lightArrays.copy(); chunkToNibbleArrayMap.disableCache(); this.uncachedLightArrays = chunkToNibbleArrayMap; ... }field_15802 gets populated whenever a light value changes or a new lightmap is added (which can contain precalculated light values and hence cause a change of light values as well), but not when a lightmap is removed. Hence the immutable copy of the lightmaps does not get updated until reloading the world or triggering some blocklight update, as observed above.
By itself this isn't really a bug. If everything else worked correctly, lightmap removal shouldn't change light values, hence there shouldn't be a reason to create a new copy of the lightmaps, other than freeing some memory.Images produced by the attached world:
Problem 2: Blocks can be seen too early
Since the lighting engine reads blocks directly from the main thread world, it can see block changes "immediately". In particular, it can see blocks before all relevant lightmaps have been created; even though lightmap creation should be issued before the actual block change is carried out, as noted above, the lighting engine might encounter a new block while processing other light updates. Although the issued lightmap creation will be visible to the lighting thread at that point (by looking at the queued operations), there is no reason why this operation should have been processed. Even if we processed all pending operations just before the block lookup, it could happen that the block gets placed and the lightmap creation gets issued inbetween those two operations.
This can lead to the problem that the lighting engine can see and take into account newly placed blocks although the relevant adjacent lightmaps, which that block can affect, have not yet been created, so the light updates will get stuck at those neighbor subchunks. Later on, the relevant lightmaps will be added and a light check for the newly added block will be carried out. However, the light is already correct around this block; the lighting errors occur at the boundary to the neighbor subchunk. Hence, the light check at the newly added block will succeed and no further updates will be carried out, keeping the errors at the chunk boundary.
For skylight this also causes the skylight optimization to be applied wrongly; the optimization only works 16 tiles away from non-air blocks, so is not applicable around those newly placed blocks. But since the lighting engine has not been told yet about those new non-air blocks, it might apply the optimization nevertheless, causing errors. This is what we will demonstrate in the following.
The setup is as follows:
We have some large stone platform high up in the sky (at y=255 in order to cause long light propagation times). Subchunk 1 6 1 has a lightmap that is kept alive by nearby blocks, but subchunk 1 5 1 doesn't. Inside a single tick
- punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of other lightmap creations and light updates, so the priority system doesn't accidentally move the next step before processing the updates of the previous one. Don't cause too many updates, as otherwise the light propagation of the previous step will be finished before we can do cool stuff.
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next step until done
- Place a platform of leave blocks in subchunk 1 6 1 below the hole
- On its way down, the light will now see the leave blocks and correctly incorporate them into the result. As subchunk {{ 1 5 1}} doesn't have a lightmap, the skylight optimization will kick in and simply copy down the light values from subchunk 1 6 1.
After passing the leave blocks, the light should decay to 0 after 15 blocks. But because of the wrongly applied optimization, it travels a whole 16 blocks (or arbitrarily many) without any decay.- Now the lighting engine adds the lightmap for subchunk 1 5 1 which is initialized by copying down the values from subchunk 1 6 1, ie. the values as produced by the skylight optimization. Also, light checks are carried out for the leave blocks. But as the light was already correct around those, nothing changes and the light in subchunk 1 5 1 stays broken.
- Add some blocks in subchunk 1 5 1 to better visualize the issue (F3 would work as well)
What we learn from this is that we should make sure that the lighting engine doesn't see any non-air blocks for subchunks that it has been told to be empty, as this can conflict with the skylight optimization.
The lighting engine already keeps track of which subchunks should be empty according to the main thrad and spawns lightmaps accordingly. I propose to employ this information inside the block lookup used by the lighting engine: First test whether the queried subchunk should be empty and return air in that case. Otherwise, proceed as normal. This prevents the described issue that blocks get seen before all relevant lightmaps have been added.Unfortunately, things can get a bit more delicate as Problem 3 shows.
Problem 3: Lightmap removal must be cancellable
The basic issue encountered here is that lightmap removal should be canceled if a subchunk gets repopulated before the lightmap is actually removed, rather than removing the lightmap and then recreating it, as this would erase information.
In order to get a better understanding, let's look at the setup in the attached world. The situation is mainly the same as in Problem 2: we have a stone platform high up in the sky and subchunk 1 6 4 has a light map. This time subchunk 1 5 4 has a light map that is kept alive by some nearby block, as well. Similarly as above, inside a single tick
- punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of lightmap creations and light updates to prevent priority reordering
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next steps until done
- As in Problem 2, we will spawn a leave platform while the lighting engine is propagating the light downwards in order to cause some issues. However, we want the lighting engine to remove the lightmap for subchunk 1 5 4 before it gets kept alive by the leave platform, but after propagating the light downwards, in order to produce the issue outlined above. Hence we need some other steps before spawning the leave platform.
Now, issue removal for the lightmap of subchunk 1 5 4 by deleting the blocks that keep it alive, but in such a way that the lightmap for subchunk 1 6 4 does not get removed.- Again cause some more lightmap creations and light updates to prevent priority reordering
- Now spawn the leave platform in subchunk 1 6 4 below the hole
- On its way down, the lighting engine will now see and incorporate the leaves blocks, as in Problem 2. This time, however, subchunk 1 5 4 does have a lightmap, so we don't wrongly apply the skylight optimization, but correctly propagate the skylight to subchunk 1 5 4. Subchunk 1 4 4 now correctly stays dark.
- Now the lighting engine processes the issued removal of the lightmap at 1 5 4. The issued recreation of the lightmap through the leave platform is not yet processed, as we issued a bunch of other lightmap creations, so it is still stuck in the queue.
- Now we finally process the issued recreation of the lightmap at 1 5 4. However, at this point the lighting information has already been erased and the new lightmap for 1 5 4 will be initialized by copying down the values from 1 6 4, hence the light below the leave platform does not decay, looking as in Problem 2. However, subchunk 1 4 4 will stay dark, in contrast to Problem 2, showing that we indeed observe a different issue here.
This issue is a mixture of Problem 1 and 2. In some sense, the issue is that the lighting engine sees the leave blocks before it is told that subchunk 1 5 4 is not empty (ie. before it processes the lightmap creations). However, the solution for Problem 2 of simply treating that block as air does not apply here, because at the time the leave block is seen, that subchunk is not marked empty, but gets only marked empty later on. In some sense this is also related to Problem 1, as lightmaps that are relevant to the leave blocks get removed after the lighting engine sees the leave blocks but before it processes the removal of the leave blocks.
The general lesson learned here is that the lightmap creation issued by spawning the leave platform should cancel the previous removals of lightmaps.The precise order of operations needed is summarized in the following section.
Proposed Solution
As noted above, the main thread should carry out operations in the following order:
- issue an operation to mark the subchunk as non-empty, as necessary
- call Chunk.setBlockState(...)
- issue light checks
- issue an operation to mark the subchunk as empty, as necessary
It will become clear in a moment why the subchunk must be marked non-empty not only before the light check, but also before the block gets written to the chunk.
For block lookups, the lighting engine should first test whether the queried subchunk should be empty, according to its current knowledge, and in this case return air.
The lighting engine should carry out operations in the following order:
- When starting an update cycle, take a snapshot of the operation queue and don't process any later updates. This makes sure that we don't process light checks without processing the corresponding lightmap creation or lightmap removals without the corresponding light checks.
- Process all operations that mark subchunks as non-empty
- Process all light checks and propagate all light updates
- In case that only a limited number of light updates is carried out at a time, finish the current set of updates before processing any new operation
- After all pending updates have been processed, process the operations that mark subchunks as empty. In this step, we need to take another snapshot of the operation queue and search it for operations that mark subchunks as non-empty, and cancel operations accordingly. This is needed as shown by Problem 3.
The correctness of this last step requires some small argument, as follows. Suppose some subchunk has been marked as empty. We consider two cases:
- The lighting engine has not seen any block that was added to the subchunk after the main thread issued the operation to mark it as empty. In this case, we can safely mark this subchunk as empty, since we processed all pending light updates that were caused by the removal of blocks and no new blocks have been observed.
- The lighting engine does see some block that was added to the subchunk after the main thread issued the operation to mark it as empty. Now we are in the situation of Problem 3. The important point is now that we can use acquire-release semantics on reading this newly added block and the corresponding write on the main thread. Because the operation to mark subchunks as non-empty is issued before writing the block
In fact, acquire-release semantics are not precisely guaranteed as reading blocks isn't even threadsafe (MC-162080), however in practice they will hold on x86 architectures as long asMC-162080doesn't occur.- Leave all remaining operations to mark subchunks as non-empty for the next cycle or process them right away.
Unfortunately, as far as I can tell, this is not so easily achievable with the current design of the multithreading code. That is because all operations are wrapped inside some generic runnable tasks and put into a single task queue, so it is not straightforward to scan this task queue without actually processing its elements.
I propose to introduce more specialzed "task queues" for different types of tasks which then can be scanned more easily. More concretely, I propose to add 3 sets (and possibly more for other stuff not covered here) lightChecks, markEmpty, markNonEmpty. When the lighting engine starts its update cycles, it takes ownership over all 3 sets and gives the main thread new ones to populate. Now we can easily process markNonEmpty and lightChecks. When processing markEmpty, we again take ownership over the new markNonEmpty and cancel any subchunks occuring in both. Now we can either merge the remaining operations of markNonEmpty back into the current set for the main thread to populate or simply keep it and merge it with the next update cycle.
The main thread then populates the sets as follows:
- use some lock, so that we don't write to the sets after the lighting engine has taken ownership
- add light checks to lightChecks (obviously
- when issuing a new empty subchunk, add it to markEmpty
- when issuing a new non-empty subchunk, first try to cancel a corresponding entry in markEmpty and otherwise add it to markNonEmpty.
- issuing an empty subchunk must not cancel any operation in markNonEmpty, as this operation might be needed to cancel a removal in the currently ongoing cycle. Canceling entries of markEmpty by non-empty subchunk, on the other hand, is unproblematic.
I hope this report was more or less understandable. If anything is unclear, please dont hesitate to ask
Best,
PhiPro
The overall issue dealt with in this report is that block changes are not properly synchronized with the lighting engine; in particular they are not properly synchronized with the lightmap handling which determines the region the lightign engine can operate on and the region where skylight optimizations are applicable. As shown below, this can lead to persistent lighting glitches.
In the following we'll look into 3 variants of this general issue that are relevant for my proposed solution. I attached a world download containing a simple setup for all 3 problems with the labeling as below. Due to the multithreading nature of the problem, the issues are not necessarily reliably reproducible, although they were on my system; it would be nice if someone else could confirm that the setup works on their system. Below I'll try to extract the relevant points.A boring solution to these issues would be to employ some copy-on-write data structure to synchronize block changes to the lighting thread in a more controlled fashion. However, that might add quite a lot of overhead in order to fix some bug that might not occur that often in practice. The solution developed below should be much more efficient and in fact not very complicated.
While the first issue is not directly a multithreading issue, it is still relevant for the solution of the general bug.
Problem 1: Lightmaps get removed too early
When the last block of a subchunk gets removed, the lighting engine is told to remove the corresponding lightmap (if there are no nearby blocks). Only afterwards it is told to reevaluate the removed block. However, at this point the lightmap was already removed, so the lighting engine simply ignores the light check; even if it wouldn't be ignored, all the relevant information would be erased.
This can be visualized as follows:
- Create a new redstone-ready world
- /setblock 7 82 7 minecraft:sea_lantern
destroy the sea lantern- Observe that the block light value is still 15. This is actually not directly the bug we are after. This part of the issue will disappear upon reloading the world or triggering some block light update. We will come back to this later.
- Reload the world to fix the previous point.
- Note that the light is still broken in subchunk 0 4 0
As soon as we break the sea lantern, the lightmap of that subchunk gets removed, so the lighting engine cannot process the block update to reduce the blocklight level. Hence the light stays in subchunk 0 4 0 which doesn't have its lightmap removed because of nearby blocks. The blocklight level for subchunk 0 5 0 is set to 0 by removing the lightmap, as we see after relaoding the world.
The relevant code pieces are
net.minecraft.world.chunk.WorldChunk.java@Nullable public BlockState setBlockState(BlockPos pos, BlockState state, boolean bl) { ... boolean bl2 = chunkSection.isEmpty(); BlockState blockState = chunkSection.setBlockState(i, j & 15, k, state); ... boolean bl3 = chunkSection.isEmpty(); if (bl2 != bl3) { this.world.getChunkManager().getLightingProvider().updateSectionStatus(pos, bl3); } ... }net.minecraft.world.World.javapublic boolean setBlockState(BlockPos pos, BlockState state, int flags) { .... BlockState blockState = worldChunk.setBlockState(pos, state, (flags & 64) != 0); ... this.profiler.push("queueCheckLight"); this.getChunkManager().getLightingProvider().checkBlock(pos); this.profiler.pop(); } ...Chunk.setBlockState(...) calls updateSectionStatus(...) to issue the creation and removal of lightmaps to the lighting engine. World.setBlockState(...) first calls Chunk.setBlockState(...) and only afterwards checkBlock(pos) which processes the block update for the lighting engine.
To fix this issue, we should first issue the lighting check and only afterwards the removal of the lightmap. By a similar argument, we should first issue the creation of a lightmap and then the lighting check for a newly added block. As we will see later, we must in fact issue the creation of a lightmap before changing the block via chunkSection.setBlockState(...).
Hence I propose to move the lightmap handling (updateSectionStatus(...)) to World.setBlockState(...) and change the order to
issue lightmap creations if necessarycall Chunk.setBlockState(...)issue light checksissue lightmap removals if necessary
Also, the lighting engine should process the issued operations in this order. Keeping only a single call to updateSectionStatus(...) as in the current code and adding some priority system to achieve the right order might work in the single threaded case, but not in the multithreaded case, as one operation might be processed before the others get issued.There is a similar version for skylight included in the attached world, however it requires some more operations as there is some special handling for adding and removing skylight maps.
For completeness, let's discuss why in the example above the light stayed in subchunk 0 5 0 until reloading the world. The main thread does not read light values directly from the internal lightmaps, but gets its own immutable copies that are updated inside
net.minecraft.world.chunk.light.LightStorage.javaprotected void notifyChunkProvider() { if (!this.field_15802.isEmpty()) { M chunkToNibbleArrayMap = this.lightArrays.copy(); chunkToNibbleArrayMap.disableCache(); this.uncachedLightArrays = chunkToNibbleArrayMap; ... }field_15802 gets populated whenever a light value changes or a new lightmap is added (which can contain precalculated light values and hence cause a change of light values as well), but not when a lightmap is removed. Hence the immutable copy of the lightmaps does not get updated until reloading the world or triggering some blocklight update, as observed above.
By itself this isn't really a bug. If everything else worked correctly, lightmap removal shouldn't change light values, hence there shouldn't be a reason to create a new copy of the lightmaps, other than freeing some memory.Images produced by the attached world:
Problem 2: Blocks can be seen too early
Since the lighting engine reads blocks directly from the main thread world, it can see block changes "immediately". In particular, it can see blocks before all relevant lightmaps have been created; even though lightmap creation should be issued before the actual block change is carried out, as noted above, the lighting engine might encounter a new block while processing other light updates. Although the issued lightmap creation will be visible to the lighting thread at that point (by looking at the queued operations), there is no reason why this operation should have been processed. Even if we processed all pending operations just before the block lookup, it could happen that the block gets placed and the lightmap creation gets issued inbetween those two operations.
This can lead to the problem that the lighting engine can see and take into account newly placed blocks although the relevant adjacent lightmaps, which that block can affect, have not yet been created, so the light updates will get stuck at those neighbor subchunks. Later on, the relevant lightmaps will be added and a light check for the newly added block will be carried out. However, the light is already correct around this block; the lighting errors occur at the boundary to the neighbor subchunk. Hence, the light check at the newly added block will succeed and no further updates will be carried out, keeping the errors at the chunk boundary.
For skylight this also causes the skylight optimization to be applied wrongly; the optimization only works 16 tiles away from non-air blocks, so is not applicable around those newly placed blocks. But since the lighting engine has not been told yet about those new non-air blocks, it might apply the optimization nevertheless, causing errors. This is what we will demonstrate in the following.
The setup is as follows:
We have some large stone platform high up in the sky (at y=255 in order to cause long light propagation times). Subchunk 1 6 1 has a lightmap that is kept alive by nearby blocks, but subchunk 1 5 1 doesn't. Inside a single tick
punch a small hole in the stone platform- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of other lightmap creations and light updates, so the priority system doesn't accidentally move the next step before processing the updates of the previous one. Don't cause too many updates, as otherwise the light propagation of the previous step will be finished before we can do cool stuff.
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next step until done
- Place a platform of leave blocks in subchunk 1 6 1 below the hole
- On its way down, the light will now see the leave blocks and correctly incorporate them into the result. As subchunk {{ 1 5 1}} doesn't have a lightmap, the skylight optimization will kick in and simply copy down the light values from subchunk 1 6 1.
After passing the leave blocks, the light should decay to 0 after 15 blocks. But because of the wrongly applied optimization, it travels a whole 16 blocks (or arbitrarily many) without any decay.- Now the lighting engine adds the lightmap for subchunk 1 5 1 which is initialized by copying down the values from subchunk 1 6 1, ie. the values as produced by the skylight optimization. Also, light checks are carried out for the leave blocks. But as the light was already correct around those, nothing changes and the light in subchunk 1 5 1 stays broken.
- Add some blocks in subchunk 1 5 1 to better visualize the issue (F3 would work as well)
What we learn from this is that we should make sure that the lighting engine doesn't see any non-air blocks for subchunks that it has been told to be empty, as this can conflict with the skylight optimization.
The lighting engine already keeps track of which subchunks should be empty according to the main thrad and spawns lightmaps accordingly. I propose to employ this information inside the block lookup used by the lighting engine: First test whether the queried subchunk should be empty and return air in that case. Otherwise, proceed as normal. This prevents the described issue that blocks get seen before all relevant lightmaps have been added.Unfortunately, things can get a bit more delicate as Problem 3 shows.
Problem 3: Lightmap removal must be cancellable
The basic issue encountered here is that lightmap removal should be canceled if a subchunk gets repopulated before the lightmap is actually removed, rather than removing the lightmap and then recreating it, as this would erase information.
In order to get a better understanding, let's look at the setup in the attached world. The situation is mainly the same as in Problem 2: we have a stone platform high up in the sky and subchunk 1 6 4 has a light map. This time subchunk 1 5 4 has a light map that is kept alive by some nearby block, as well. Similarly as above, inside a single tick
punch a small hole in the stone platform- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of lightmap creations and light updates to prevent priority reordering
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next steps until done
- As in Problem 2, we will spawn a leave platform while the lighting engine is propagating the light downwards in order to cause some issues. However, we want the lighting engine to remove the lightmap for subchunk 1 5 4 before it gets kept alive by the leave platform, but after propagating the light downwards, in order to produce the issue outlined above. Hence we need some other steps before spawning the leave platform.
Now, issue removal for the lightmap of subchunk 1 5 4 by deleting the blocks that keep it alive, but in such a way that the lightmap for subchunk 1 6 4 does not get removed.- Again cause some more lightmap creations and light updates to prevent priority reordering
- Now spawn the leave platform in subchunk 1 6 4 below the hole
- On its way down, the lighting engine will now see and incorporate the leaves blocks, as in Problem 2. This time, however, subchunk 1 5 4 does have a lightmap, so we don't wrongly apply the skylight optimization, but correctly propagate the skylight to subchunk 1 5 4. Subchunk 1 4 4 now correctly stays dark.
- Now the lighting engine processes the issued removal of the lightmap at 1 5 4. The issued recreation of the lightmap through the leave platform is not yet processed, as we issued a bunch of other lightmap creations, so it is still stuck in the queue.
- Now we finally process the issued recreation of the lightmap at 1 5 4. However, at this point the lighting information has already been erased and the new lightmap for 1 5 4 will be initialized by copying down the values from 1 6 4, hence the light below the leave platform does not decay, looking as in Problem 2. However, subchunk 1 4 4 will stay dark, in contrast to Problem 2, showing that we indeed observe a different issue here.
This issue is a mixture of Problem 1 and 2. In some sense, the issue is that the lighting engine sees the leave blocks before it is told that subchunk 1 5 4 is not empty (ie. before it processes the lightmap creations). However, the solution for Problem 2 of simply treating that block as air does not apply here, because at the time the leave block is seen, that subchunk is not marked empty, but gets only marked empty later on. In some sense this is also related to Problem 1, as lightmaps that are relevant to the leave blocks get removed after the lighting engine sees the leave blocks but before it processes the removal of the leave blocks.
The general lesson learned here is that the lightmap creation issued by spawning the leave platform should cancel the previous removals of lightmaps.The precise order of operations needed is summarized in the following section.
Proposed Solution
As noted above, the main thread should carry out operations in the following order:
issue an operation to mark the subchunk as non-empty, as necessarycall Chunk.setBlockState(...)issue light checksissue an operation to mark the subchunk as empty, as necessaryIt will become clear in a moment why the subchunk must be marked non-empty not only before the light check, but also before the block gets written to the chunk.
For block lookups, the lighting engine should first test whether the queried subchunk should be empty, according to its current knowledge, and in this case return air.
The lighting engine should carry out operations in the following order:
- When starting an update cycle, take a snapshot of the operation queue and don't process any later updates. This makes sure that we don't process light checks without processing the corresponding lightmap creation or lightmap removals without the corresponding light checks.
- Process all operations that mark subchunks as non-empty
- Process all light checks and propagate all light updates
- In case that only a limited number of light updates is carried out at a time, finish the current set of updates before processing any new operation
- After all pending updates have been processed, process the operations that mark subchunks as empty. In this step, we need to take another snapshot of the operation queue and search it for operations that mark subchunks as non-empty, and cancel operations accordingly. This is needed as shown by Problem 3.
The correctness of this last step requires some small argument, as follows. Suppose some subchunk has been marked as empty. We consider two cases:
- The lighting engine has not seen any block that was added to the subchunk after the main thread issued the operation to mark it as empty. In this case, we can safely mark this subchunk as empty, since we processed all pending light updates that were caused by the removal of blocks and no new blocks have been observed.
- The lighting engine does see some block that was added to the subchunk after the main thread issued the operation to mark it as empty. Now we are in the situation of Problem 3. The important point is now that we can use acquire-release semantics on reading this newly added block and the corresponding write on the main thread. Because the operation to mark subchunks as non-empty is issued before writing the block
In fact, acquire-release semantics are not precisely guaranteed as reading blocks isn't even threadsafe (MC-162080), however in practice they will hold on x86 architectures as long asMC-162080doesn't occur.- Leave all remaining operations to mark subchunks as non-empty for the next cycle or process them right away.
Unfortunately, as far as I can tell, this is not so easily achievable with the current design of the multithreading code. That is because all operations are wrapped inside some generic runnable tasks and put into a single task queue, so it is not straightforward to scan this task queue without actually processing its elements.
I propose to introduce more specialzed "task queues" for different types of tasks which then can be scanned more easily. More concretely, I propose to add 3 sets (and possibly more for other stuff not covered here) lightChecks, markEmpty, markNonEmpty. When the lighting engine starts its update cycles, it takes ownership over all 3 sets and gives the main thread new ones to populate. Now we can easily process markNonEmpty and lightChecks. When processing markEmpty, we again take ownership over the new markNonEmpty and cancel any subchunks occuring in both. Now we can either merge the remaining operations of markNonEmpty back into the current set for the main thread to populate or simply keep it and merge it with the next update cycle.
The main thread then populates the sets as follows:
use some lock, so that we don't write to the sets after the lighting engine has taken ownershipadd light checks to lightChecks (obviouslywhen issuing a new empty subchunk, add it to markEmptywhen issuing a new non-empty subchunk, first try to cancel a corresponding entry in markEmpty and otherwise add it to markNonEmpty.issuing an empty subchunk must not cancel any operation in markNonEmpty, as this operation might be needed to cancel a removal in the currently ongoing cycle. Canceling entries of markEmpty by non-empty subchunk, on the other hand, is unproblematic.I hope this report was more or less understandable. If anything is unclear, please dont hesitate to ask
Best,
PhiProThe overall issue dealt with in this report is that block changes are not properly synchronized with the lighting engine; in particular they are not properly synchronized with the lightmap handling which determines the region the lightign engine can operate on and the region where skylight optimizations are applicable. As shown below, this can lead to persistent lighting glitches.
In the following we'll look into 3 variants of this general issue that are relevant for my proposed solution. I attached a world download containing a simple setup for all 3 problems with the labeling as below. Due to the multithreading nature of the problem, the issues are not necessarily reliably reproducible, although they were on my system; it would be nice if someone else could confirm that the setup works on their system. Below I'll try to extract the relevant points.A boring solution to these issues would be to employ some copy-on-write data structure to synchronize block changes to the lighting thread in a more controlled fashion. However, that might add quite a lot of overhead in order to fix some bug that might not occur that often in practice. The solution developed below should be much more efficient and in fact not very complicated.
While the first issue is not directly a multithreading issue, it is still relevant for the solution of the general bug.
Problem 1: Lightmaps get removed too early
When the last block of a subchunk gets removed, the lighting engine is told to remove the corresponding lightmap (if there are no nearby blocks). Only afterwards it is told to reevaluate the removed block. However, at this point the lightmap was already removed, so the lighting engine simply ignores the light check; even if it wouldn't be ignored, all the relevant information would be erased.
This can be visualized as follows:
- Create a new redstone-ready world
- /setblock 7 82 7 minecraft:sea_lantern
- Destroy the sea lantern
- Observe that the block light value is still 15. This is actually not directly the bug we are after. This part of the issue will disappear upon reloading the world or triggering some block light update. We will come back to this later.
- Reload the world to fix the previous point.
- Note that the light is still broken in subchunk 0 4 0
As soon as we break the sea lantern, the lightmap of that subchunk gets removed, so the lighting engine cannot process the block update to reduce the blocklight level. Hence the light stays in subchunk 0 4 0 which doesn't have its lightmap removed because of nearby blocks. The blocklight level for subchunk 0 5 0 is set to 0 by removing the lightmap, as we see after relaoding the world.
The relevant code pieces are
net.minecraft.world.chunk.WorldChunk.java@Nullable public BlockState setBlockState(BlockPos pos, BlockState state, boolean bl) { ... boolean bl2 = chunkSection.isEmpty(); BlockState blockState = chunkSection.setBlockState(i, j & 15, k, state); ... boolean bl3 = chunkSection.isEmpty(); if (bl2 != bl3) { this.world.getChunkManager().getLightingProvider().updateSectionStatus(pos, bl3); } ... }net.minecraft.world.World.javapublic boolean setBlockState(BlockPos pos, BlockState state, int flags) { .... BlockState blockState = worldChunk.setBlockState(pos, state, (flags & 64) != 0); ... this.profiler.push("queueCheckLight"); this.getChunkManager().getLightingProvider().checkBlock(pos); this.profiler.pop(); } ...Chunk.setBlockState(...) calls updateSectionStatus(...) to issue the creation and removal of lightmaps to the lighting engine. World.setBlockState(...) first calls Chunk.setBlockState(...) and only afterwards checkBlock(pos) which processes the block update for the lighting engine.
To fix this issue, we should first issue the lighting check and only afterwards the removal of the lightmap. By a similar argument, we should first issue the creation of a lightmap and then the lighting check for a newly added block. As we will see later, we must in fact issue the creation of a lightmap before changing the block via chunkSection.setBlockState(...).
Hence I propose to move the lightmap handling (updateSectionStatus(...)) to World.setBlockState(...) and change the order to
- Issue lightmap creations if necessary
- Call Chunk.setBlockState(...)
- Issue light checks
- Issue lightmap removals if necessary
Also, the lighting engine should process the issued operations in this order. Keeping only a single call to updateSectionStatus(...) as in the current code and adding some priority system to achieve the right order might work in the single threaded case, but not in the multithreaded case, as one operation might be processed before the others get issued.There is a similar version for skylight included in the attached world, however it requires some more operations as there is some special handling for adding and removing skylight maps.
For completeness, let's discuss why in the example above the light stayed in subchunk 0 5 0 until reloading the world. The main thread does not read light values directly from the internal lightmaps, but gets its own immutable copies that are updated inside
net.minecraft.world.chunk.light.LightStorage.javaprotected void notifyChunkProvider() { if (!this.field_15802.isEmpty()) { M chunkToNibbleArrayMap = this.lightArrays.copy(); chunkToNibbleArrayMap.disableCache(); this.uncachedLightArrays = chunkToNibbleArrayMap; ... }field_15802 gets populated whenever a light value changes or a new lightmap is added (which can contain precalculated light values and hence cause a change of light values as well), but not when a lightmap is removed. Hence the immutable copy of the lightmaps does not get updated until reloading the world or triggering some blocklight update, as observed above.
By itself this isn't really a bug. If everything else worked correctly, lightmap removal shouldn't change light values, hence there shouldn't be a reason to create a new copy of the lightmaps, other than freeing some memory.Images produced by the attached world:
Problem 2: Blocks can be seen too early
Since the lighting engine reads blocks directly from the main thread world, it can see block changes "immediately". In particular, it can see blocks before all relevant lightmaps have been created; even though lightmap creation should be issued before the actual block change is carried out, as noted above, the lighting engine might encounter a new block while processing other light updates. Although the issued lightmap creation will be visible to the lighting thread at that point (by looking at the queued operations), there is no reason why this operation should have been processed. Even if we processed all pending operations just before the block lookup, it could happen that the block gets placed and the lightmap creation gets issued inbetween those two operations.
This can lead to the problem that the lighting engine can see and take into account newly placed blocks although the relevant adjacent lightmaps, which that block can affect, have not yet been created, so the light updates will get stuck at those neighbor subchunks. Later on, the relevant lightmaps will be added and a light check for the newly added block will be carried out. However, the light is already correct around this block; the lighting errors occur at the boundary to the neighbor subchunk. Hence, the light check at the newly added block will succeed and no further updates will be carried out, keeping the errors at the chunk boundary.
For skylight this also causes the skylight optimization to be applied wrongly; the optimization only works 16 tiles away from non-air blocks, so is not applicable around those newly placed blocks. But since the lighting engine has not been told yet about those new non-air blocks, it might apply the optimization nevertheless, causing errors. This is what we will demonstrate in the following.
The setup is as follows:
We have some large stone platform high up in the sky (at y=255 in order to cause long light propagation times). Subchunk 1 6 1 has a lightmap that is kept alive by nearby blocks, but subchunk 1 5 1 doesn't. Inside a single tick
- Punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of other lightmap creations and light updates, so the priority system doesn't accidentally move the next step before processing the updates of the previous one. Don't cause too many updates, as otherwise the light propagation of the previous step will be finished before we can do cool stuff.
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next step until done
- Place a platform of leave blocks in subchunk 1 6 1 below the hole
- On its way down, the light will now see the leave blocks and correctly incorporate them into the result. As subchunk {{ 1 5 1}} doesn't have a lightmap, the skylight optimization will kick in and simply copy down the light values from subchunk 1 6 1.
After passing the leave blocks, the light should decay to 0 after 15 blocks. But because of the wrongly applied optimization, it travels a whole 16 blocks (or arbitrarily many) without any decay.- Now the lighting engine adds the lightmap for subchunk 1 5 1 which is initialized by copying down the values from subchunk 1 6 1, ie. the values as produced by the skylight optimization. Also, light checks are carried out for the leave blocks. But as the light was already correct around those, nothing changes and the light in subchunk 1 5 1 stays broken.
- Add some blocks in subchunk 1 5 1 to better visualize the issue (F3 would work as well)
What we learn from this is that we should make sure that the lighting engine doesn't see any non-air blocks for subchunks that it has been told to be empty, as this can conflict with the skylight optimization.
The lighting engine already keeps track of which subchunks should be empty according to the main thrad and spawns lightmaps accordingly. I propose to employ this information inside the block lookup used by the lighting engine: First test whether the queried subchunk should be empty and return air in that case. Otherwise, proceed as normal. This prevents the described issue that blocks get seen before all relevant lightmaps have been added.Unfortunately, things can get a bit more delicate as Problem 3 shows.
Problem 3: Lightmap removal must be cancellable
The basic issue encountered here is that lightmap removal should be canceled if a subchunk gets repopulated before the lightmap is actually removed, rather than removing the lightmap and then recreating it, as this would erase information.
In order to get a better understanding, let's look at the setup in the attached world. The situation is mainly the same as in Problem 2: we have a stone platform high up in the sky and subchunk 1 6 4 has a light map. This time subchunk 1 5 4 has a light map that is kept alive by some nearby block, as well. Similarly as above, inside a single tick
- Punch a small hole in the stone platform
- Trigger the lighting engine to propagate the light downwards from this hole by causing some more light checks
- Cause a bunch of lightmap creations and light updates to prevent priority reordering
- The lighting thread will now propagate the light downwards from the hole and hence not process any operation caused by the next steps until done
- As in Problem 2, we will spawn a leave platform while the lighting engine is propagating the light downwards in order to cause some issues. However, we want the lighting engine to remove the lightmap for subchunk 1 5 4 before it gets kept alive by the leave platform, but after propagating the light downwards, in order to produce the issue outlined above. Hence we need some other steps before spawning the leave platform.
Now, issue removal for the lightmap of subchunk 1 5 4 by deleting the blocks that keep it alive, but in such a way that the lightmap for subchunk 1 6 4 does not get removed.- Again cause some more lightmap creations and light updates to prevent priority reordering
- Now spawn the leave platform in subchunk 1 6 4 below the hole
- On its way down, the lighting engine will now see and incorporate the leaves blocks, as in Problem 2. This time, however, subchunk 1 5 4 does have a lightmap, so we don't wrongly apply the skylight optimization, but correctly propagate the skylight to subchunk 1 5 4. Subchunk 1 4 4 now correctly stays dark.
- Now the lighting engine processes the issued removal of the lightmap at 1 5 4. The issued recreation of the lightmap through the leave platform is not yet processed, as we issued a bunch of other lightmap creations, so it is still stuck in the queue.
- Now we finally process the issued recreation of the lightmap at 1 5 4. However, at this point the lighting information has already been erased and the new lightmap for 1 5 4 will be initialized by copying down the values from 1 6 4, hence the light below the leave platform does not decay, looking as in Problem 2. However, subchunk 1 4 4 will stay dark, in contrast to Problem 2, showing that we indeed observe a different issue here.
This issue is a mixture of Problem 1 and 2. In some sense, the issue is that the lighting engine sees the leave blocks before it is told that subchunk 1 5 4 is not empty (ie. before it processes the lightmap creations). However, the solution for Problem 2 of simply treating that block as air does not apply here, because at the time the leave block is seen, that subchunk is not marked empty, but gets only marked empty later on. In some sense this is also related to Problem 1, as lightmaps that are relevant to the leave blocks get removed after the lighting engine sees the leave blocks but before it processes the removal of the leave blocks.
The general lesson learned here is that the lightmap creation issued by spawning the leave platform should cancel the previous removals of lightmaps.The precise order of operations needed is summarized in the following section.
Proposed Solution
As noted above, the main thread should carry out operations in the following order:
- Issue an operation to mark the subchunk as non-empty, as necessary
- Call Chunk.setBlockState(...)
- Issue light checks
- Issue an operation to mark the subchunk as empty, as necessary
It will become clear in a moment why the subchunk must be marked non-empty not only before the light check, but also before the block gets written to the chunk.
For block lookups, the lighting engine should first test whether the queried subchunk should be empty, according to its current knowledge, and in this case return air.
The lighting engine should carry out operations in the following order:
- When starting an update cycle, take a snapshot of the operation queue and don't process any later updates. This makes sure that we don't process light checks without processing the corresponding lightmap creation or lightmap removals without the corresponding light checks.
- Process all operations that mark subchunks as non-empty
- Process all light checks and propagate all light updates
- In case that only a limited number of light updates is carried out at a time, finish the current set of updates before processing any new operation
- After all pending updates have been processed, process the operations that mark subchunks as empty. In this step, we need to take another snapshot of the operation queue and search it for operations that mark subchunks as non-empty, and cancel operations accordingly. This is needed as shown by Problem 3.
The correctness of this last step requires some small argument, as follows. Suppose some subchunk has been marked as empty. We consider two cases:
- The lighting engine has not seen any block that was added to the subchunk after the main thread issued the operation to mark it as empty. In this case, we can safely mark this subchunk as empty, since we processed all pending light updates that were caused by the removal of blocks and no new blocks have been observed.
- The lighting engine does see some block that was added to the subchunk after the main thread issued the operation to mark it as empty. Now we are in the situation of Problem 3. The important point is now that we can use acquire-release semantics on reading this newly added block and the corresponding write on the main thread. Because the operation to mark subchunks as non-empty is issued before writing the block
In fact, acquire-release semantics are not precisely guaranteed as reading blocks isn't even threadsafe (MC-162080), however in practice they will hold on x86 architectures as long asMC-162080doesn't occur.- Leave all remaining operations to mark subchunks as non-empty for the next cycle or process them right away.
Unfortunately, as far as I can tell, this is not so easily achievable with the current design of the multithreading code. That is because all operations are wrapped inside some generic runnable tasks and put into a single task queue, so it is not straightforward to scan this task queue without actually processing its elements.
I propose to introduce more specialzed "task queues" for different types of tasks which then can be scanned more easily. More concretely, I propose to add 3 sets (and possibly more for other stuff not covered here) lightChecks, markEmpty, markNonEmpty. When the lighting engine starts its update cycles, it takes ownership over all 3 sets and gives the main thread new ones to populate. Now we can easily process markNonEmpty and lightChecks. When processing markEmpty, we again take ownership over the new markNonEmpty and cancel any subchunks occuring in both. Now we can either merge the remaining operations of markNonEmpty back into the current set for the main thread to populate or simply keep it and merge it with the next update cycle.
The main thread then populates the sets as follows:
- Use some lock, so that we don't write to the sets after the lighting engine has taken ownership
- Add light checks to lightChecks (obviously
- When issuing a new empty subchunk, add it to markEmpty
- When issuing a new non-empty subchunk, first try to cancel a corresponding entry in markEmpty and otherwise add it to markNonEmpty.
- Issuing an empty subchunk must not cancel any operation in markNonEmpty, as this operation might be needed to cancel a removal in the currently ongoing cycle. Canceling entries of markEmpty by non-empty subchunk, on the other hand, is unproblematic.
I hope this report was more or less understandable. If anything is unclear, please dont hesitate to ask
Best,
PhiPro
This is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(.... There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing. This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.
- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as earlier: Don't change light values by creating lightmaps, but initialize them to the values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule additional light and darkness propagations in order to compensate for the changed values.
Concretely, this means to initialize topmost lightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.As far as I understand it, the current initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume MC-? to be fixed.
Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, wheres the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap get scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }This basically pushes skylight to the topmost lightmap from above and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming MC-? has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.
- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independently of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in {{pre_light}}ed but not yet {{light}}ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
Now, I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previously topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }Also, there is no need to relight the new topmost lightmap after removing lightmaps (in fact, that doesn't even seem to be necessary with teh current coe).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend to the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks.
Also note, that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means rhat we can skip propagation from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiProThis is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as earlier: Don't change light values by creating lightmaps, but initialize them to the values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule additional light and darkness propagations in order to compensate for the changed values.
Concretely, this means to initialize topmost lightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.As far as I understand it, the current initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume MC-? to be fixed.
Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, wheres the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap get scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }This basically pushes skylight to the topmost lightmap from above and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming MC-? has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.
- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independently of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in {{pre_light}}ed but not yet {{light}}ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
Now, I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previously topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }Also, there is no need to relight the new topmost lightmap after removing lightmaps (in fact, that doesn't even seem to be necessary with teh current coe).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend to the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks.
Also note, that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means rhat we can skip propagation from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiPro
This is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as earlier: Don't change light values by creating lightmaps, but initialize them to the values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule additional light and darkness propagations in order to compensate forthechanged values.
Concretely, this means to initialize topmost lightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.As far as I understand it, the current initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume MC-? to be fixed.
Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, wheres the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap get scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }Th
isbasically pushes skylight to the topmost lightmapfrom aboveand then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming MC-? has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.
- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ
entlyof my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.Fixing this is rather easy: When a subchunk in {{pre_light}}ed but not yet {{light}}ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
Now,I hope that my suggested fix is now sufficiently convincingFinally, let's discuss some simplifications and otimizations that are now possible:
Since the initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previouslytopmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }Also, there is no need to relight the new topmost lightmap after removing lightmap
s(in fact, that doesn't even seem to be necessary with teh current coe).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend
tothe above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks.
Also note,that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this meansrhat we can skip propagation from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiProThis is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as for
MC-148580: Don't change light values by creating lightmaps, but initialize them to the exact values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule any additional light and darkness propagations in order to compensate for changed values.
Concretely, this means to initialize topmost skylightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.
As far as I understand it, the current lightmap initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume MC-? to be fixed.
Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps above (or directly associated to) them.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, whereas the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap gets scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }The initialization then basically pushes skylight from above to the topmost lightmap and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming MC-? has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.
- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in a pre_light ed but not yet light ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the (current) lightmap initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previous topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }
Also, there is no need to relight the new topmost lightmap after removing a lightmap (in fact, that doesn't even seem to be necessary with the current code).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks. This optimization will not interfere with our invariant above, as it says that the pseudo-empty region of not yet lighted chunks can be opaque.
Also, note that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means that we can skip propagations from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps of already lighted chunks to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate for these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the current lightmap initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiPro
This is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as for
MC-148580: Don't change light values by creating lightmaps, but initialize them to the exact values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule any additional light and darkness propagations in order to compensate for changed values.
Concretely, this means to initialize topmost skylightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.
As far as I understand it, the current lightmap initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume MC-? to be fixed.
Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps above (or directly associated to) them.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, whereas the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap gets scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }The initialization then basically pushes skylight from above to the topmost lightmap and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming MC-? has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.
- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in a pre_light ed but not yet light ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the (current) lightmap initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previous topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }
Also, there is no need to relight the new topmost lightmap after removing a lightmap (in fact, that doesn't even seem to be necessary with the current code).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks. This optimization will not interfere with our invariant above, as it says that the pseudo-empty region of not yet lighted chunks can be opaque.
Also, note that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means that we can skip propagations from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps of already lighted chunks to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate for these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the current lightmap initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiProThis is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as for
MC-148580: Don't change light values by creating lightmaps, but initialize them to the exact values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule any additional light and darkness propagations in order to compensate for changed values.
Concretely, this means to initialize topmost skylightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.
As far as I understand it, the current lightmap initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume
MC-170012to be fixed.Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps above (or directly associated to) them.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, whereas the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap gets scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }The initialization then basically pushes skylight from above to the topmost lightmap and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming
MC-170012has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in a pre_light ed but not yet light ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the (current) lightmap initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previous topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }
Also, there is no need to relight the new topmost lightmap after removing a lightmap (in fact, that doesn't even seem to be necessary with the current code).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks. This optimization will not interfere with our invariant above, as it says that the pseudo-empty region of not yet lighted chunks can be opaque.
Also, note that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means that we can skip propagations from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps of already lighted chunks to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate for these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the current lightmap initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiPro
This is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15.As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as for
MC-148580: Don't change light values by creating lightmaps, but initialize them to the exact values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule any additional light and darkness propagations in order to compensate for changed values.
Concretely, this means to initialize topmost skylightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.
As far as I understand it, the current lightmap initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume
MC-170012to be fixed.Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps above (or directly associated to) them.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, whereas the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap gets scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }The initialization then basically pushes skylight from above to the topmost lightmap and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming
MC-170012has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in a pre_light ed but not yet light ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the (current) lightmap initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previous topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }
Also, there is no need to relight the new topmost lightmap after removing a lightmap (in fact, that doesn't even seem to be necessary with the current code).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks. This optimization will not interfere with our invariant above, as it says that the pseudo-empty region of not yet lighted chunks can be opaque.
Also, note that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means that we can skip propagations from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps of already lighted chunks to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate for these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the current lightmap initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiProThis is a direct continuation of
MC-148580. I post it as a new report since the old one already contains several similar bugs and I don't want it to digress too much, given that it's already closed for quite a while now.The issue remains the same, namely that the topmost lightmap is not correctly initialized to a light value of 15, but rather to a value of 0.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createLightArray(long pos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.lightArraysToAdd.get(pos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(pos, Direction.UP); int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(ChunkSectionPos.withZeroZ(pos)); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY && ChunkSectionPos.getY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightArray(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }The lightmap will then be registered for relighting inside onLightArrayCreated(...) and finally be relighted in updateLightArrays(...). There are however some problems with this initialization code:
- If the subchunk is non-empty, light is only propagated from above, but not from the sides. There are cases where this is not sufficient. However, this issue is completely shadowed by the next one.
- The initialization is completely skipped if another lightmap is added above it.
I attached a world producing wrong lighting using this second problem. Since I lag the lighting engine to spawn mutible blocks at once before the lighting engine can react, the provided world might not reliably work on some systems, but at least it does reliably work on mine. Would be nice if someone can confirm this on their system.
The setup is as follows:
- We have some subchunk that does not have a lightmap and no lightmap above, but all its 4 neighbors do have one.
- Schedule a bunch of lightmap creations and light updates to lag the lighting engine, so we can spawn several blocks at once before the lighting engine can react to it. (This is similar to
MC-169913, but a fix for it will not help here.)- Spawn a stone platform in the beforementioned subchunk.
- Now this subchunk will get a lightmap, which gets scheduled for initialization. However, afterwards the subchunk above it will get a lightmap as well, canceling the initialization again.
In the end, the lightmap of the subchunk containing the stone platform will remain uninitialized.- Since all its neighbors already had lightmaps, the initialization code does not spread light form there to the subchunk containing the stone platform, so light contributions from the side are missing.
This is also the reason for the first problem I mentioned above regarding the initialization code. wouldn't it be shadowed by the second one.- Since block updates schedule light checks not only for the affected blocks but also for all 6 neighbors, we do get a contribution from the blocks directly below the stone platform, but not from any other blocks.
- This produces the following image
There is another bug caused by the initialization code, namely
MC-169498. The problem here is that upon removing a lightmap, the new topmost lightmap is scheduled for reinitialization. In case the corresponding subchunk is empty, initialization then sets the whole lightmap to 15 and spreads ligth to its neighbors. In the setup described in the bug, as soon as the stone block is broken, this will cause the subchunk it was in to be set to a lightvalue of 15. Note that alsoMC-169913plays a role here as it causes lightmaps to be removed before the block removal is properly processed; however, that doesn't really matter for the solution discussed below.
As always, the problem with this is that this initialization changes light values without informing everyone about it; in this case, the subchunk containing the sandstone blocks is not marked dirty and hence not rerendered. As can be seen, the lighting is actually correctly at a value of 15, but the subchunk is not rerendered and hence still shows the value of 14.
My suggested solution is the same as for
MC-148580: Don't change light values by creating lightmaps, but initialize them to the exact values that would have been returned before. As no light values change, you don't have to inform anyone about any changes and you don't have to schedule any additional light and darkness propagations in order to compensate for changed values.
Concretely, this means to initialize topmost skylightmaps of already lighted
This will get rid off both issues and in fact allows to remove some now obsolete code, reducing code complexity by quite a bit.
As far as I understand it, the current lightmap initialization code basically handles the initial skylight. So, let's look into that in more detail and see that my suggested solution doesn't interfere with this. In fact, it will allow to simplify and optimize the code a bit. In the following, I will assume
MC-170012to be fixed.Starting with some definition, let's call all subchunks of a given chunk that do not have a lightmap above them (or associated directly to them) the empty region, and all subchunks that do not have a block above (or inside) them the pseudo-empty region; so the empty-region is that part of the pseudo-empty region that does not have associated lightmaps above (or directly associated to) them.
Before a chunk is lighted, it does not receive any direct skylight, ie. the empty region is completely dark.
The invariant we want to prove is:
- Before a chunk is lighted, it properly receives and propagates all skylight contributions from its neighbors, but its pseudo-empty region might be opaque to skylight. More precisely, the interior of every subchunk in the pseudo-empty region is transparent to skylight, but its faces (except the top face) might be opaque, whereas the top face is also transparent.
- Once a chunk is lighted, its pseudo-empty region is now properly transparent and has a skylight value of 15. All skylight is properly propagated to neighbors (except to their pseudo-empty regions which might be opaque).
During initial lighting, the topmost lightmap gets scheduled for initialization inside
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected void setLightEnabled(long l, boolean bl) { this.updateAll(); if (bl && this.lightEnabled.add(l)) { int i = ((SkyLightStorage.Data)this.lightArrays).topArraySectionY.get(l); if (i != ((SkyLightStorage.Data)this.lightArrays).defaultTopArraySectionY) { long m = ChunkSectionPos.asLong(ChunkSectionPos.getX(l), i - 1, ChunkSectionPos.getZ(l)); this.method_20810(m); this.checkForUpdates(); } } else if (!bl) { this.lightEnabled.remove(l); } }The initialization then basically pushes skylight from above to the topmost lightmap and then propagates it, using some optimization in case the subchunk is empty as mentioned before (but that's not relevant here).
Now let's prove the invariant inductively.
- Since not yet lighted chunks do not receive direct skylight, skylight can only exist in a 3x3 neighborhood around lighted chunks. Assuming
MC-170012has been fixed, all relevant lightmaps have been created in that region. Hence, all skylight updates will be properly propagated, except for the case when they get stuck at the boundary to the empty region because of missing lightmaps. This case is however in accordance with our inductive hypothesis, so this is not a problem.- When a chunk gets lighted, it already receives all contributions from neigbor chunks. The only remaining defects are that it does not yet receive direct skylight and the pseudo-empty region might be opaque. The beforementioned initialization code will now propagate direct skylight to the topmost blocks of the topmost lightmap, which will then propagate and light up the whole pseudo-empty region, setting it to a value of 15 and hence making it transparent to skylight.
Hence, the chunk itself receives all skylight contributions after it has been lighted.- The initialization code also pushes skylight from lightmaps in the pseudo-empty region, and more generally from everywhere except the empty region, to all neighbor chunks. We need to show that all lightmaps in the non-pseudo-empty region of a neighbor chunk properly receive their contributions, ie. that no lightmap in the non-pseudo-empty-region of a neighbor chunk is adjacent to the empty region of the chunk that is currently being lighted. However, this is clear, since a subchunk in the non-pseudo-empty region has a block above it by definition. This causes a lightmap to be created in the chunk that is currently being lighted that is above the lightmap in question, ie. the lightmap in question might be adjacent to the pseudo-empty region of the chunk that is currently being lighted, but not to the empty region, which is what we wanted to show.
Note that this argument never used the complicated initialization logic for lightmaps of already lighted chunks. All the relevant logic is carried out by the initialization code as scheduled by setLightEnabled(...). Hence, it is safe to simply initialize topmost lightmaps in already lighted chunks with a skylight value of 15, without causing any interference with the initial lighting code.
There might in fact be a missing point in the above argument, namely in case the pseudo-empty region of not yet lighted chunks can change over time. This would cause bugs already in the current version of the code and would need a fix independ of my suggested fix for the issue described here. I'm not familiar enough with the worldgen code to say whether or not this event is possible. For completeness, I will describe the issue more percisely and show that the current code would not deal with it.
Suppose we have a chunk that has a lighted neighbor, so it is sufficiently far through worldgen to have reached the pre_ligth stage (and hence probably shouldn't receive any worldgen block updates; but as I said, I am not sure about this) but is not yet lighted itself (and hence probably shouldn't receive any block changes caused by actual game ticks). Now suppose it is possible for this chunk to receive block updates, in particular, we may spawn a stone platform in the empty region. Now, the subchunk containing the stone platform and all subchunks below are pushed to the non-pseudo-empty region. This breaks our invariant because they must no longer be opaque.
The current initialization code manages to push in light from the side in case a new lightmap is spawned in the respective neighbor chunks. However, if the neighbor chunk already had a lightmap, the initialization code won't do anything and we are missing contributions from all sides. As the stone platform blocks light from above and there is no mechanism that will eventually push in light from the sides, this will indeed result in a presistent lighting error.
Fixing this is rather easy: When a subchunk in a pre_light ed but not yet light ed chunk turns non-empty, we need to pull in skylight from all sides for this subchunk and all subchunks below that were previously in the pseudo-empty region. Note that we actually don't need to pull in light from the top faces, as these were already transparent before, but we do need to pull in light from all bottom faces, which might have been opaque before.
As you can see, this fix is independent of lightmap initialization for already lighted chunks.As I said, I can't say whether this is relevant or not.
I hope that my suggested fix is now sufficiently convincing
Finally, let's discuss some simplifications and otimizations that are now possible:
Since the (current) lightmap initialization code now solely deals with initial skylight, we can now properly move it to the rest of the initialization code, where it belongs, instead of being scattered around.
As lightmap initialization now indeed no longer changes any light values, we can remove those parts of the code that compensated for changes in light values. Concretely, that is the last part of updateLightArrays(...) that currently spreads darkness to the previous topmost lightmapnet.minecraft.world.chunk.light.SkyLightStorage.javathis.pendingSkylightUpdates.clear(); if (!this.field_15816.isEmpty()) { var4 = this.field_15816.iterator(); label90: while(true) { do { do { if (!var4.hasNext()) { break label90; } l = (Long)var4.next(); } while(!this.field_15820.remove(l)); } while(!this.hasLight(l)); for(ag = 0; ag < 16; ++ag) { for(j = 0; j < 16; ++j) { long ai = BlockPos.asLong(ChunkSectionPos.getWorldCoord(ChunkSectionPos.getX(l)) + ag, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getY(l)) + 16 - 1, ChunkSectionPos.getWorldCoord(ChunkSectionPos.getZ(l)) + j); lightProvider.updateLevel(Long.MAX_VALUE, ai, 15, false); } } } }
Also, there is no need to relight the new topmost lightmap after removing a lightmap (in fact, that doesn't even seem to be necessary with the current code).
Removing those two pieces of code should already simplify the code by quite a bit and hence improve maintainability.Now, diving into some small optimizations that may squeeze out a bit of performance:
- The optimization that empty subchunks are directly set to 15 and light is then manually spread to all sides, instead of letting the lighting engine propagate the light from the top face, can be applied to all lightmaps in the pseudo-empty region upon initial lighting, not only to the topmost lightmap.
- Currently, the code pushes light from the pseudo-empty region from the chunk that is being lighted to the pseudo-empty region of all neighbors. In particular, in practice the topmost lightmaps are often on the same y-level. This causes light to be spread to a lightmap that will later be overridden with a value of 15 anyway. While this is not drastically bad, it is easy to optimize. Simply extend the above optimization to skip propagations from lightmaps in the pseudo-empty region to the pseudo-empty region of (not yet lighted) neighbor chunks. This optimization will not interfere with our invariant above, as it says that the pseudo-empty region of not yet lighted chunks can be opaque.
Also, note that we can safely skip propagations to the pseudo-empty region of lighted neighbors, as they already have a skylight value of 15. In conclusion this means that we can skip propagations from the pseudo-empty region to the pseudo-empty region of all neighbors.Summary
- Both described bugs can simply be solved by initializing topmost skylight maps of already lighted chunks to a value of 15 and not doing anything else. This doesn't interfere with the initial lighting code.
- Lightmap initialization not changing values allows to get rid off some code that is currently needed to compensate for these changes. Hence the proposed solution will reduce code complexity.
- There might be some bug with the current initial lighting code, depending on whether or not there can be block changes for chunks between pre_light and light stage. You can probably judge better than me whether this is a problem or not
- Restricting the current lightmap initialization code solely to initial lighting allows moving it to a more fitting place and allows for some easy optimizations.
Hope this is helpful
Best,
PhiPro
This is again rather a code discussion than an actual bug report.
But my proposed solution is easy, so I think it is worth mentioning.I do not have any specific numbers or examples of this happening, but it can clearly have a negative impact on chunk loading performance, which can be easily avoided.
Basically, the issue is that worldgen tasks cannot be canceled once scheduled to the worldgen task queue, which can cause clogging if players move around too fast.
As mentioned in MC-177685, the ChunkTicketManager does some artificial throttling to limit the number of chunk tickets for which the respective chunk is not fully loaded yet. If the player moves away again before the loading is complete, the chunk ticket and hence the throttling is removedagain. However, all the already scheduled worldgen tasks (for which there can be many per single chunk ticket) are not canceled and will still be executed. So, if a player moves around too fast and loads and unloads many chunks (especially in multiplayer environments where there are many players), this can lead to the worldgen task queue becoming clogged by tasks that are not needed anymore. Due to MC-177685, this queue is linear and not prioritized, so the old pending tasks indeed take precedence over new importanttasks.
Currently, when a chunkHolderlevel is demoted, all the pending futures above the new level will be completed with Unloaded, which does however not abort the underlying tasks associated to them directly.Actually, this does abort all but the lowest level pending task per chunk, since all higher level tasks will receive an Unloaded chunk, so they will abort once they get executed. The lowest level taskscan however not be canceledcurrently, so in particular chunk loading from diskcannot be canceled.I propose that worldgen tasks (and chunk loading from disk) should as first operation check if the
providedchunkHolder still has a sufficiently high level to execute the task and otherwise abort early.
It is important that this operation is not (only) performed when creating the futures, but also when actually processing the tasks associated to them which do the actual work.
In particular, it is not so important to add such a check innet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> createChunkFuture(ChunkHolder chunkHolder, ChunkStatus chunkStatus) { ChunkPos chunkPos = chunkHolder.getPos(); if (chunkStatus == ChunkStatus.EMPTY) { return this.loadChunk(chunkPos); } else { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = chunkHolder.createFuture(chunkStatus.getPrevious(), this); return completableFuture.thenComposeAsync((either) -> { ... if (chunk.getStatus().isAtLeast(chunkStatus)) { CompletableFuture completableFuture2; if (chunkStatus == ChunkStatus.LIGHT) { completableFuture2 = this.generateChunk(chunkHolder, chunkStatus); } else { completableFuture2 = chunkStatus.runNoGenTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(chunkHolder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, chunkStatus); return completableFuture2; } else { return this.generateChunk(chunkHolder, chunkStatus); } }, this.mainThreadExecutor); } }which does not do much work on its own
. But rather add the check inside the futures created bythis.loadChunk(...) andthis.generateChunk(...) which do the actual work.Note that the task associated to the future created in this.generateChunk(...) does not run on the main thread, but on the worldgen thread (the task associated to this.loadChunk(...) currently does run on the main thread due to MC-177729), so this proposed check accesses
chunkHolder concurrently.
Hence, we should not usechunkHolder.level which might sporadically change as it is used as the underlying storage for tracking the distance to the next chunk ticket. Instead, we should use a version that is better controlled, i.e. only changed in ChunkHolder.tick(...) as is already provided bychunkHolder.lastLevel.
Note that this level should be updated before scheduling the worldgen tasks, so they don't get immediately aborted.chunkHolder.lastLevel getscurrentlyupdated at the end of ChunkHolder.tick(...).
Also note that thischunkHolder.lastLevel need not be volatile. It suffices to be atomic with weak memory ordering, as we only require that it is indeed sufficiently large if since scheduling the task the level never droppedbelow the required level for the task. This is automatically provided by java forint, but of course it does not cause harm to declare the field{{ volatile}}(except of negligible performance concerns).
Best,
PhiProThis is again rather a code discussion than an actual bug report. I do not have any specific numbers or examples of this happening, but it can clearly have a negative impact on chunk loading performance, which can be easily avoided. My proposed solution is easy, so I think it is worth mentioning this issue.
Basically, the issue is that worldgen tasks cannot be canceled once scheduled to the worldgen task queue, which can cause clogging if players move around too fast.
As mentioned in MC-177685, the ChunkTicketManager does some artificial throttling to limit the number of chunk tickets for which the respective chunk is not fully loaded yet. If the player moves away again (before the loading is complete), the chunk ticket and hence the throttling is removed; however, all the already scheduled worldgen tasks (for which there can be many per single chunk ticket) are not canceled and will still be executed. So, if a player moves around too fast and loads and unloads many chunks (especially in multiplayer environments where there are many players), this can lead to the worldgen task queue becoming clogged by tasks that are not needed anymore. Due to MC-177685, this queue is linear and not prioritized, so the old pending tasks indeed take precedence over new important ones.Currently, when a ChunkHolder level is demoted, all the pending futures above the new level will be completed with Unloaded, which does however not abort the underlying tasks associated to them directly. In fact, this does abort all but the lowest level pending task per chunk, since all higher level tasks will receive an Unloaded chunk, so they will abort once they get executed. The lowest level task can however not be canceled. So in particular chunk loading from disk will not be canceled, which is always present even if the chunks are already generated but not loaded.
I propose that worldgen tasks (and chunk loading from disk) should as first operation check if the associated ChunkHolder still has a sufficiently high level to execute the task and otherwise abort early.
It is important that this operation is not (only) performed when creating the futures, but also when actually processing the tasks associated to them which do the actual work.
In particular, it is not so important to add such a check innet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> createChunkFuture(ChunkHolder chunkHolder, ChunkStatus chunkStatus) { ChunkPos chunkPos = chunkHolder.getPos(); if (chunkStatus == ChunkStatus.EMPTY) { return this.loadChunk(chunkPos); } else { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = chunkHolder.createFuture(chunkStatus.getPrevious(), this); return completableFuture.thenComposeAsync((either) -> { ... if (chunk.getStatus().isAtLeast(chunkStatus)) { CompletableFuture completableFuture2; if (chunkStatus == ChunkStatus.LIGHT) { completableFuture2 = this.generateChunk(chunkHolder, chunkStatus); } else { completableFuture2 = chunkStatus.runNoGenTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(chunkHolder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, chunkStatus); return completableFuture2; } else { return this.generateChunk(chunkHolder, chunkStatus); } }, this.mainThreadExecutor); } }which does not do much work on its own (other than maybe this.convertToFullChunk(...)). But rather add the check inside the futures created by this.loadChunk(...) and this.generateChunk(...) which do the actual work.
Note that the task associated to the future created in this.generateChunk(...) does not run on the main thread, but on the worldgen thread (the task associated to this.loadChunk(...) currently does run on the main thread due to MC-177729), so this proposed check accesses ChunkHolder concurrently.
Hence, we should not use ChunkHolder.level which might sporadically change as it is used as the underlying storage for tracking the distance to the next chunk ticket. Instead, we should use a version that is better controlled, i.e. only changed in ChunkHolder.tick(...) as is already provided by ChunkHolder.lastLevel.
Note that this level should be updated before scheduling the worldgen tasks, so they don't get immediately aborted. Currently, ChunkHolder.lastLevel gets updated at the end of ChunkHolder.tick(...).
Also note that this ChunkHolder.lastLevel need not be volatile. It suffices to be atomic with weak memory ordering, as we only require that it is indeed sufficiently large if since scheduling the task the level never dropped too low. This is automatically provided by java for int, but of course it does not cause harm to declare the field volatile (except of negligible performance concerns).
Best,
PhiPro
This is again rather a code discussion than an actual bug report. I do not have any specific numbers or examples of this happening, but it can clearly have a negative impact on chunk loading performance, which can be easily avoided. My proposed solution is easy, so I think it is worth mentioning this issue.
Basically, the issue is that worldgen tasks cannot be canceled once scheduled to the worldgen task queue, which can cause clogging if players move around too fast.
As mentioned in MC-177685, the ChunkTicketManager does some artificial throttling to limit the number of chunk tickets for which the respective chunk is not fully loaded yet. If the player moves away again (before the loading is complete), the chunk ticket and hence the throttling is removed; however, all the already scheduled worldgen tasks (for which there can be many per single chunk ticket) are not canceled and will still be executed. So, if a player moves around too fast and loads and unloads many chunks (especially in multiplayer environments where there are many players), this can lead to the worldgen task queue becoming clogged by tasks that are not needed anymore. Due to MC-177685, this queue is linear and not prioritized, so the old pending tasks indeed take precedence over new important ones.Currently, when a ChunkHolder level is demoted, all the pending futures above the new level will be completed with Unloaded, which does however not abort the underlying tasks associated to them directly. In fact, this does abort all but the lowest level pending task per chunk, since all higher level tasks will receive an Unloaded chunk, so they will abort once they get executed. The lowest level task can however not be canceled. So in particular chunk loading from disk will not be canceled, which is always present even if the chunks are already generated but not loaded.
I propose that worldgen tasks (and chunk loading from disk) should as first operation check if the associated ChunkHolder still has a sufficiently high level to execute the task and otherwise abort early.
It is important that this operation is not (only) performed when creating the futures, but also when actually processing the tasks associated to them which do the actual work.
In particular, it is not so important to add such a check innet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> createChunkFuture(ChunkHolder chunkHolder, ChunkStatus chunkStatus) { ChunkPos chunkPos = chunkHolder.getPos(); if (chunkStatus == ChunkStatus.EMPTY) { return this.loadChunk(chunkPos); } else { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = chunkHolder.createFuture(chunkStatus.getPrevious(), this); return completableFuture.thenComposeAsync((either) -> { ... if (chunk.getStatus().isAtLeast(chunkStatus)) { CompletableFuture completableFuture2; if (chunkStatus == ChunkStatus.LIGHT) { completableFuture2 = this.generateChunk(chunkHolder, chunkStatus); } else { completableFuture2 = chunkStatus.runNoGenTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(chunkHolder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, chunkStatus); return completableFuture2; } else { return this.generateChunk(chunkHolder, chunkStatus); } }, this.mainThreadExecutor); } }which does not do much work on its own (other than maybe this.convertToFullChunk(...)). But rather add the check inside the futures created by this.loadChunk(...) and this.generateChunk(...) which do the actual work.
Note that the task associated to the future created in this.generateChunk(...) does not run on the main thread, but on the worldgen thread (the task associated to this.loadChunk(...) currently does run on the main thread due to MC-177729), so this proposed check accesses ChunkHolder concurrently.
Hence, we should not use ChunkHolder.level which might sporadically change as it is used as the underlying storage for tracking the distance to the next chunk ticket. Instead, we should use a version that is better controlled, i.e. only changed in ChunkHolder.tick(...) as is already provided by ChunkHolder.lastLevel.
Note that this level should be updated before scheduling the worldgen tasks, so they don't get immediately aborted. Currently, ChunkHolder.lastLevel gets updated at the end of ChunkHolder.tick(...).
Also note that this ChunkHolder.lastLevel need not be volatile. It suffices to be atomic with weak memory ordering, as we only require thatit is indeed sufficiently large if since scheduling the task the level never dropped too low. This is automatically provided by java for int, but of course it does not cause harm to declare the field volatile (except of negligible performance concerns).
Best,
PhiProThis is again rather a code discussion than an actual bug report. I do not have any specific numbers or examples of this happening, but it can clearly have a negative impact on chunk loading performance, which can be easily avoided. My proposed solution is easy, so I think it is worth mentioning this issue.
Basically, the issue is that worldgen tasks cannot be canceled once scheduled to the worldgen task queue, which can cause clogging if players move around too fast.
As mentioned in MC-177685, the ChunkTicketManager does some artificial throttling to limit the number of chunk tickets for which the respective chunk is not fully loaded yet. If the player moves away again (before the loading is complete), the chunk ticket and hence the throttling is removed; however, all the already scheduled worldgen tasks (for which there can be many per single chunk ticket) are not canceled and will still be executed. So, if a player moves around too fast and loads and unloads many chunks (especially in multiplayer environments where there are many players), this can lead to the worldgen task queue becoming clogged by tasks that are not needed anymore. Due to MC-177685, this queue is linear and not prioritized, so the old pending tasks indeed take precedence over new important ones.Currently, when a ChunkHolder level is demoted, all the pending futures above the new level will be completed with Unloaded, which does however not abort the underlying tasks associated to them directly. In fact, this does abort all but the lowest level pending task per chunk, since all higher level tasks will receive an Unloaded chunk, so they will abort once they get executed. The lowest level task can however not be canceled. So in particular chunk loading from disk will not be canceled, which is always present even if the chunks are already generated but not loaded.
I propose that worldgen tasks (and chunk loading from disk) should as first operation check if the associated ChunkHolder still has a sufficiently high level to execute the task and otherwise abort early.
It is important that this operation is not (only) performed when creating the futures, but also when actually processing the tasks associated to them which do the actual work.
In particular, it is not so important to add such a check innet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> createChunkFuture(ChunkHolder chunkHolder, ChunkStatus chunkStatus) { ChunkPos chunkPos = chunkHolder.getPos(); if (chunkStatus == ChunkStatus.EMPTY) { return this.loadChunk(chunkPos); } else { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = chunkHolder.createFuture(chunkStatus.getPrevious(), this); return completableFuture.thenComposeAsync((either) -> { ... if (chunk.getStatus().isAtLeast(chunkStatus)) { CompletableFuture completableFuture2; if (chunkStatus == ChunkStatus.LIGHT) { completableFuture2 = this.generateChunk(chunkHolder, chunkStatus); } else { completableFuture2 = chunkStatus.runNoGenTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(chunkHolder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, chunkStatus); return completableFuture2; } else { return this.generateChunk(chunkHolder, chunkStatus); } }, this.mainThreadExecutor); } }which does not do much work on its own (other than maybe this.convertToFullChunk(...)). But rather add the check inside the futures created by this.loadChunk(...) and this.generateChunk(...) which do the actual work.
Note that the task associated to the future created in this.generateChunk(...) does not run on the main thread, but on the worldgen thread (the task associated to this.loadChunk(...) currently does run on the main thread due to MC-177729), so this proposed check accesses ChunkHolder concurrently.
Hence, we should not use ChunkHolder.level which might sporadically change as it is used as the underlying storage for tracking the distance to the next chunk ticket. Instead, we should use a version that is better controlled, i.e. only changed in ChunkHolder.tick(...) as is already provided by ChunkHolder.lastLevel.
Note that this level should be updated before scheduling the worldgen tasks, so they don't get immediately aborted. Currently, ChunkHolder.lastLevel gets updated at the end of ChunkHolder.tick(...).
Also note that this ChunkHolder.lastLevel need not be volatile. It suffices to be atomic with weak memory ordering, as we only require that the observed value from the worldgen thread is indeed sufficiently large if on the main thread since scheduling the task the level never dropped too low. This is automatically provided by java for int, but of course it does not cause harm to declare the field volatile (except of negligible performance concerns).
Best,
PhiPro
Note: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...),
ensuring thatgetPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently neeed is due to the convention about src and dst position discussed above, so the code wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actuallymore meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiProNote: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently neeed is due to the convention about src and dst position discussed above, so the code wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actuallymore meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiPro
Note: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently neeed is due to the convention about src and dst position discussed above, so the code wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actuallymore meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiProNote: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently needed is due to the convention about src and dst position discussed above, so the code currently wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actually more meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source skylight contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiPro
Note: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently needed is due to the convention about src and dst position discussed above, so the code currently wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actually more meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source skylight contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiProNote: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently needed is due to the convention about src and dst position discussed above, so the code currently wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actually more meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for MC-?, since the code handling this source skylight contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution of MC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiPro
Note: I will fill in the linked reports once they are created.
Just some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010and MC-?, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently needed is due to the convention about src and dst position discussed above, so the code currently wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actually more meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for
MC-?, since the code handling this source skylight contribution ingetPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution ofMC-?. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiProJust some small suggestions to improve readability and reduce complexity of the skylight propagation code.
While not being bugs on their own, some of these vanilla code pieces cause issues with my other bugfixes, especiallyMC-170010andMC-196725, so it is very convenient to clean these code pieces up before fixing the other bugs.
I collect these cleanups here separately as I don't want to cluttter the other reports any further.First of all, quite a bit of complexity in ChunkSkyLightProvider.getPropagatedLevel(long src, long dst, int level) comes from the fact that the skylight optimization code calls it with non-adjacent src and dst positions. (Concretely, propagateLevel(long src, int level, boolean brighten) skips downwards propagations into empty sections and instead directly propagates its values to the next non-trivial section below and all neighbors on the way, using the original src position for the call to getPropagatedLevel(...), or rather to LevelPropagator.propagateLevel(...) which in turn calls getPropagatedLevel(...).)
This three-way propagation then causes the following code inside getPropagatedLevel(...)
net.minecraft.world.chunk.light.ChunkSkyLightProvider.javaif (direction2 != null) { ... } else { voxelShape = this.getOpaqueShape(blockState2, sourceId, Direction.DOWN); if (VoxelShapes.unionCoversFullCube(voxelShape, VoxelShapes.empty())) { return 15; } int r = bl ? -1 : 0; Direction direction3 = Direction.fromVector(o, r, q); if (direction3 == null) { return 15; } VoxelShape voxelShape4 = this.getOpaqueShape(blockState, targetId, direction3.getOpposite()); if (VoxelShapes.unionCoversFullCube(VoxelShapes.empty(), voxelShape4)) { return 15; } }
I propose to change this convention and pass the block below src at the appropriate y-level to LevelPropagator.propagateLevel(...), i.e., the neighbor of dst that is below the original src, so that getPropagatedLevel(...) receives adjacent src and dst positions. This will completely eliminate all the code associated to the three-way propagation.
This convention actually looks semantically more meaningful to me, but I guess that's not really a valid reason to change thingsNote that this convention is in fact slightly more optimal for ChunkSkyLightProvider.recalculateLevel(long pos, long neighbor, int neighborLevel) (which gets passed the contribution of one neighbor). Currently, in case this neighbor is not directly adjacent, when processing the contribution from the corresponding direction the code first has to search for the next non-empty section above, just to come to the conclusion that this contribution is already provided. With the proposed change the code could skip this neighbor directly.
This change needs to be implemented both in propagateLevel(...) and recalculateLevel(...).
Next, recalculateLevel(...) manually implements the light lookup for neighbors without an associated lightmap. This code already exists in getLight(...), which currently only handles the main-thread light lookup, using the uncached lightmaps.
I propose to move this code to a common place, so both main thread and lighting engine can use it with the appropriate set of lightmaps. recalculateLevel(...) can then employ this instead of implementing everything on its own.
This whole codeblockfor(m = BlockPos.removeChunkSectionLocalY(m); !((SkyLightStorage)this.lightStorage).hasSection(n) && !((SkyLightStorage)this.lightStorage).isAtOrAboveTopmostSection(n); m = BlockPos.add(m, 0, 16, 0)) { n = ChunkSectionPos.offset(n, Direction.UP); } ChunkNibbleArray chunkNibbleArray4 = ((SkyLightStorage)this.lightStorage).getLightSection(n, true); if (m != excludedId) { int p; if (chunkNibbleArray4 != null) { p = this.getPropagatedLevel(m, id, this.getCurrentLevelFromSection(chunkNibbleArray4, m)); } else { p = ((SkyLightStorage)this.lightStorage).isSectionEnabled(n) ? 0 : 15; }then simply becomes
if (m != excludedId) p = this.getPropagatedLevel(m, id, this.getLight(m));
Note that this uniform treatment also fixes the bug mentioned in this comment.
I think one reason why this manual lookup is currently needed is due to the convention about src and dst position discussed above, so the code currently wants to know where the ligth data comes from. This is no longer the case with the convention proposed above.
Related to the previous point is the fact that recalculateLevel(...) excludes contributions from above if there is no lightmap directly associated to the neighbor. Instead, it invokes getPropagatedLevel(...) with a dummy position Long.MAX_VALUE, which in turn contains some additional logic to handle this source skylight.
I propose to simply include contributions from above in the uniform treatment discussed in the previous point and remove the logic handling this special value Long.MAX_VALUE from getPropagatedLevel(...), simplifying the code again.In my opinion it is actually more meaningful to think of skylight as always coming from above, with the source infinitely high up in the sky. Rather than the source sitting at the topmost lightmap which changes over time.
Also note that the current code is actually slightly wrong. The source skylight contribution only is 15 for the topmost block of the topmost lightmap, otherwise it is 0. That means that a block with no lightmap directly above will not get any source skylight contribution, as soon as there is some lightmap above, even if the block is directly exposed to skylight, i.e., has no obstacle above. Since recalculateLevel(...) in this case excludes the contribution from above, the resulting value will actually be wrong. This issue doesn't really show up, though, since recaclculateLevel(...) is invoked in sufficiently rare situations that manage to spread the correct value through some other means.
I also want to mention that this code causes some slight issue with my proposed solution for
MC-196725, since the code handling this source skylight contribution in getPropagatedLevel(...) assumes that the topmost block of the topmost lightmap is air. This is true for the current vanilla code, but not for my proposed solution ofMC-196725. The uniform treatment discussed above avoids this issue.
Finally, some small cleanup slightly unrelated to the rest.
ChunkLightProvider.resetLevel(long pos) is used to recalculate the light value at a position, e.g. after a block change. In case there is no associated lightmap at the specified position, ChunkSkyLightProvider.resetLevel(long pos) will delegate this to the next position above that does have an associated lightmap.
This does not really make any sense to me. If there is no lightmap, simply skip the action, as is already done by ChunkLightProvider.resetLevel(...). If there was any need to recalculate the light level at the delegated position, it would have received a request on its own.
One reason I could think of for this code to exist is to force the block above to re-emit its light, hiding some effects ofMC-169913orMC-170010. But that's just some wild speculation. In that case, this code can be removed after fixing the other bugs.
Best,
PhiPro
Note: I will fill in the linked reports once they are created
While the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips
uninitialized lightmaps, i.e., those that were created via the last path of the above codereturn new ChunkNibbleArray(); and never received any light change.
Thiscauseslightmaps that need to be initialized without queued data upon reloading a chunk. If this initialization does not happenfromtoptobottom, then this search for a lightmap to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottomtotop, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization orderreallyundefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for
allunderground subchunks, which causes them to become fully bright with my fix forMC-170010.Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone- Fly to chunk -14 0
- Fly back to chunk 0 0 and observe that subchunk 0 0 0 is fully bright
The explanation here is that upon reloading chunk 0 0 when flying back, the lightmap at 0 2 0 is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk 0 0 the lightmap at 0 0 0 gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at 0 2 0 instead of the correct 0 1 0 which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk 3 5 1 and observe that it is too bright
The issue here is that after reloading the world the lightmap at 3 6 1 is not loaded, as it gets loaded by chunk 4 1 which is too far away, so that upon placing the stone block the lightmap at 3 5 1 gets initialized and inherits its values from 3 7 1 instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmap without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. And again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk 3 5 1 and observe that it is too dark
Again the lightmap at 3 5 1 is not loaded after reloading the world. When clearing the stone platform at 2 7 1 the skylight optimization then pierces through 3 5 1 without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of {{createSection(...)}, and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for MC-? which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way a
nd bring some further optimizations on top of that.Best,
PhiProNote: I will fill in the linked reports once they are created
While the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.
The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }
This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated yet, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips uninitialized lightmaps, i.e., those that were created via the last path of the above code return new ChunkNibbleArray(); and never received any light change.
This leads to lightmaps that need to be initialized without having queued data upon reloading a chunk. If this initialization does not happen top-to-bottom, then this search for a lightmap above to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottom-to-top, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization order undefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for most underground subchunks, which causes them to become fully bright with my fix for
MC-170010.
Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 0)
- Fly back to chunk (0 0) and observe that subchunk (0 0 0) is fully bright
The explanation here is that upon reloading chunk (0 0) when flying back, the lightmap at (0 2 0) is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk (0 0) the lightmap at (0 0 0) gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at (0 2 0) instead of the correct (0 1 0) which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone
- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk (3 5 1) and observe that it is too bright
The issue here is that after reloading the world, the lightmap at (3 6 1) is not loaded, as it gets loaded by chunk (4 1) which is too far away, so that upon placing the stone block the lightmap at (3 5 1) gets initialized and inherits its values from (3 7 1) instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.
The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmaps that are not actually added to the world without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. Again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone
- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk (3 5 1) and observe that it is too dark
Again the lightmap at (3 5 1) is not loaded after reloading the world. When clearing the stone platform at (2 7 1) the skylight optimization then pierces through (3 5 1) without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of createSection(...), and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2 kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for MC-? which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way, as lightmaps will only be created when really needed, and will also clean up empty lightmaps more aggressively.
Best,
PhiPro
Note: I will fill in the linked reports once they are created
While the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.
The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }
This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated yet, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips uninitialized lightmaps, i.e., those that were created via the last path of the above code return new ChunkNibbleArray(); and never received any light change.
This leads to lightmaps that need to be initialized without having queued data upon reloading a chunk. If this initialization does not happen top-to-bottom, then this search for a lightmap above to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottom-to-top, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization order undefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for most underground subchunks, which causes them to become fully bright with my fix for
MC-170010.
Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 0)
- Fly back to chunk (0 0) and observe that subchunk (0 0 0) is fully bright
The explanation here is that upon reloading chunk (0 0) when flying back, the lightmap at (0 2 0) is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk (0 0) the lightmap at (0 0 0) gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at (0 2 0) instead of the correct (0 1 0) which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone
- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk (3 5 1) and observe that it is too bright
The issue here is that after reloading the world, the lightmap at (3 6 1) is not loaded, as it gets loaded by chunk (4 1) which is too far away, so that upon placing the stone block the lightmap at (3 5 1) gets initialized and inherits its values from (3 7 1) instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.
The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmaps that are not actually added to the world without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. Again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone
- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk (3 5 1) and observe that it is too dark
Again the lightmap at (3 5 1) is not loaded after reloading the world. When clearing the stone platform at (2 7 1) the skylight optimization then pierces through (3 5 1) without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of createSection(...), and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2 kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for MC-? which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way, as lightmaps will only be created when really needed, and will also clean up empty lightmaps more aggressively.
Best,
PhiProWhile the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.
The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }
This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated yet, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips uninitialized lightmaps, i.e., those that were created via the last path of the above code return new ChunkNibbleArray(); and never received any light change.
This leads to lightmaps that need to be initialized without having queued data upon reloading a chunk. If this initialization does not happen top-to-bottom, then this search for a lightmap above to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottom-to-top, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization order undefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for most underground subchunks, which causes them to become fully bright with my fix for
MC-170010.
Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 0)
- Fly back to chunk (0 0) and observe that subchunk (0 0 0) is fully bright
The explanation here is that upon reloading chunk (0 0) when flying back, the lightmap at (0 2 0) is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk (0 0) the lightmap at (0 0 0) gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at (0 2 0) instead of the correct (0 1 0) which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone
- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk (3 5 1) and observe that it is too bright
The issue here is that after reloading the world, the lightmap at (3 6 1) is not loaded, as it gets loaded by chunk (4 1) which is too far away, so that upon placing the stone block the lightmap at (3 5 1) gets initialized and inherits its values from (3 7 1) instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.
The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmaps that are not actually added to the world without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. Again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone
- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk (3 5 1) and observe that it is too dark
Again the lightmap at (3 5 1) is not loaded after reloading the world. When clearing the stone platform at (2 7 1) the skylight optimization then pierces through (3 5 1) without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of createSection(...), and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2 kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for
MC-196725which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way, as lightmaps will only be created when really needed, and will also clean up empty lightmaps more aggressively.Best,
PhiPro
While the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.
The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }
This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated yet, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips uninitialized lightmaps, i.e., those that were created via the last path of the above code return new ChunkNibbleArray(); and never received any light change.
This leads to lightmaps that need to be initialized without having queued data upon reloading a chunk. If this initialization does not happen top-to-bottom, then this search for a lightmap above to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottom-to-top, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization order undefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for most underground subchunks, which causes them to become fully bright with my fix for
MC-170010.
Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14
0)- Fly back to chunk (
0 0) and observe that subchunk (000) is fully bright
The explanation here is that upon reloading chunk (
0 0) when flying back, the lightmap at (020) is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk (0 0) the lightmap at (000) gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at (020) instead of the correct (010) which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone
- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk (3 5 1) and observe that it is too bright
The issue here is that after reloading the world, the lightmap at (3 6 1) is not loaded, as it gets loaded by chunk (4 1) which is too far away, so that upon placing the stone block the lightmap at (3 5 1) gets initialized and inherits its values from (3 7 1) instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.
The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmaps that are not actually added to the world without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. Again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone
- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk (3 5 1) and observe that it is too dark
Again the lightmap at (3 5 1) is not loaded after reloading the world. When clearing the stone platform at (2 7 1) the skylight optimization then pierces through (3 5 1) without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of createSection(...), and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2 kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for
MC-196725which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way, as lightmaps will only be created when really needed, and will also clean up empty lightmaps more aggressively.Best,
PhiProWhile the following issue might sound rather theoretical and not really practically relevant, it has some rather bad interaction with my proposed fix for
MC-170010. FixingMC-170010without fixing this issue first basically causes all underground subchunks to become fully bright.
The first part of the issue comes down to the fact that upon initialization of skylight-maps the search for a lightmap above the specified position (which will be used to inherit its values in case no lightmap was queued up for the specified position directly) only takes into account lightmaps already added to the world but ignores lightmaps that are queued up to be added at some later time.
net.minecraft.world.chunk.light.SkyLightStorage.javaprotected ChunkNibbleArray createSection(long sectionPos) { ChunkNibbleArray chunkNibbleArray = (ChunkNibbleArray)this.queuedSections.get(sectionPos); if (chunkNibbleArray != null) { return chunkNibbleArray; } else { long l = ChunkSectionPos.offset(sectionPos, Direction.UP); int i = ((SkyLightStorage.Data)this.storage).columnToTopSection.get(ChunkSectionPos.withZeroY(sectionPos)); if (i != ((SkyLightStorage.Data)this.storage).minSectionY && ChunkSectionPos.unpackY(l) < i) { ChunkNibbleArray chunkNibbleArray2; while((chunkNibbleArray2 = this.getLightSection(l, true)) == null) { l = ChunkSectionPos.offset(l, Direction.UP); } return new ChunkNibbleArray((new ColumnChunkNibbleArray(chunkNibbleArray2, 0)).asByteArray()); } else { return new ChunkNibbleArray(); } } }
This would be fine if there are either no queued lightmaps at all, i.e., the chunk has not been generated yet, or if every lightmap that needs to be initialized does have queued up data. However, saving chunks strips uninitialized lightmaps, i.e., those that were created via the last path of the above code return new ChunkNibbleArray(); and never received any light change.
This leads to lightmaps that need to be initialized without having queued data upon reloading a chunk. If this initialization does not happen top-to-bottom, then this search for a lightmap above to inherit from can return the wrong lightmap, causing lighting glitches.
And indeed this initialization order is unspecified. Whenever a chunk is promoted to the light stage, it will initialize all non-initialized lightmaps of itself and its neighbors that are near a non-empty subchunk. This initialization seems to happen bottom-to-top, but this is rather implementation dependent. This will however not necessarily load all lightmaps of the chunk or its neighbors, since some lightmaps might be loaded through a non-empty subchunk of a chunk not yet in the lighting stage, hence making the initialization order undefined.This primary bottom-to-top initialization then causes the lightmap creation code to hit the last codepath return new ChunkNibbleArray(); for most underground subchunks, which causes them to become fully bright with my fix for
MC-170010.
Let's now give some steps to visualize this issue in Vanilla.
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 48 0 47 48 47 minecraft:sandstone /fill 16 31 16 31 31 31 minecraft:sandstone /fill 16 48 16 31 48 31 minecraft:air /fill 16 15 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 1)
- Fly back to chunk (1 1) and observe that subchunk (1 0 1) is fully bright
The explanation here is that upon reloading chunk (1 1) when flying back, the lightmap at (1 2 1) is already loaded by the neighbors and is completely bright, but no lightmap below is loaded yet. When loading chunk (1 1) the lightmap at (1 0 1) gets initialized without queued data, as the lightmap was uninitialized before saving, and inherits its values from the already loaded lightmap at (1 2 1) instead of the correct (1 1 1) which is not yet added to the world.
A similar situation can be created when dealing with block changes near the edge of the loaded world.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 48 128 0 63 128 47 minecraft:stone /fill 64 112 0 79 112 47 minecraft:stone
- Set the render distance to 2
- Reload the world
- /setblock 40 88 24 minecraft:stone
- Fly to subchunk (3 5 1) and observe that it is too bright
The issue here is that after reloading the world, the lightmap at (3 6 1) is not loaded, as it gets loaded by chunk (4 1) which is too far away, so that upon placing the stone block the lightmap at (3 5 1) gets initialized and inherits its values from (3 7 1) instead.
While this might sound similar to
MC-170012, note that fixingMC-170012will only move the issue by one chunk, but moving every step by 1 chunk in +x direction would still work as all block operations would still be in the accessible region. The fundamental issue of not taking into account queued up lightmaps is not solved byMC-170012.
The second related part of the issue was already mentioned in the discussion of
MC-170010in regards to the vanilla data retainment mechanism. The issue mentioned there is that the skylight optimization simply pierces through such queued lightmaps that are not actually added to the world without changing their light values. When then later on readding the lightmap to the world these missing light changes will result in a lighting glitch. While the discussion ofMC-170010was mostly concerned with this effect for initial lighting, we can again create a similar situation when dealing with block changes near the edge of the loaded world. Again this scenario is unaffected byMC-170012.
- Create a redstone-ready world
- Set the render distance to 10 (or anything > 3)
- Run the following commands
/setworldspawn 1000 56 1000 /fill 32 112 0 63 112 47 minecraft:stone /setblock 72 88 24 minecraft:stone
- Set the render distance to 2
- Reload the world
- /fill 32 112 16 47 112 31 minecraft:air
- Fly to subchunk (3 5 1) and observe that it is too dark
Again the lightmap at (3 5 1) is not loaded after reloading the world. When clearing the stone platform at (2 7 1) the skylight optimization then pierces through (3 5 1) without changing its values and then later on the old unchanged lightmap is added.
The solution to this problem sounds rather straightforward: Just take these queued lightmaps into account for lightmap creation and skylight optimization.
There is however an issue. As mentioned above, the code currently (wrongly) creates most lightmaps unitialized, i.e., using the last codepath of createSection(...), and hence they take up no memory (or at least far less than the usual 2 kB for initialized lightmaps). After solving this bug, these lightmaps will then be created by inheriting the correct values from the lightmap above (which will be mostly 0). This will cause them to take up the full 2 kB of memory albeit still being mostly 0. One solution would then be to check for this case and leave the lightmaps unitiailized if this occurs.
I propose to instead implement my suggested solution for
MC-196725which automatically contains a fix for the bug discussed here. This will decouple the lightmap handling from the current skylight-optimization distance tracking, and hence naturally places all lightmaps into a single datastructure rather than splitting them into two. Furthermore, this will solve this last concern about memory consumption in a more general way, as lightmaps will only be created when really needed, and will also clean up empty lightmaps more aggressively.Best,
PhiPro
Upon chunk loading, promotion of a chunk to the light stage does require neighbors to be loaded in the features stage (which is the stage at which the lighting engine is able to interact with a chunk), so that border chunks, i.e., the last accessible chunks, directly after they are loaded do have their neighbors loaded in the correct state for the lighting engine to interact with them. However, if those neighbors get unloaded later on, the chunk is not demoted from the light stage again, so you can have chunks in light (and full) stage without all their neighbors being loaded. In particular, this can happen for border chunks which can then lead to light updates getting stuck at the chunk boundary to the unloaded neighbor.
This can be visualized using the following steps:
- Create a redstone-ready world
- Set the render distance to 2
- /setworldspawn 1000 56 1000
- Fly to chunk (-14 0) (This will unload chunk (1 0))
- Fly to chunk (-3 0)
- /setblock 14 56 8 minecraft:sea_lantern
- Fly back to chunk (0 0) and observe that the light got stuck at the boundary to (1 0)
Currently, promotion of a chunk to the BORDER state only requires the chunk itself to be loaded as full chunk. I propose to add here the requirement that also the neighbors should be loaded in the features stage. In contrast to the worldgen stages, e.g., light or full, the accessibility status, i.e., INACCESSIBLE, BORDER, TICKING and ENTITY_TICKING does get demoted if the chunk ticket level drops too low and hence the condition is reevaluated each time the chunk gets promoted again. Hence with this change promotion to the BORDER status would re-enforce neighbors to be loaded in the features stage.
This simply requires to changenet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return holder.getChunkAt(ChunkStatus.FULL, this).thenApplyAsync((either) -> { ... }to
public CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return this.getRegion(holder.getPos(), 1,(d) ->ChunkStatus.byDistanceFromFull(ChunkStatus.getDistanceFromFull(centerChunkTargetStatus) + d)).thenApplyAsync((either) -> { ... }
Also note that the light stage is currently special cased to enforce the neighbor-loading even if the chunk was already generated into the light stage before.
net.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunk(ChunkHolder holder, ChunkStatus requiredStatus) { ... if (chunk.getStatus().isAtLeast(requiredStatus)) { CompletableFuture completableFuture2; if (requiredStatus == ChunkStatus.LIGHT) { completableFuture2 = this.upgradeChunk(holder, requiredStatus); } else { completableFuture2 = requiredStatus.runLoadTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(holder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, requiredStatus); return completableFuture2; } else { return this.upgradeChunk(holder, requiredStatus); } ... }
With my proposed change this special handling is no longer needed. The light stage will still require neighbors in the features stage for the initial lighting, but once the chunk is generated into the light stage it should no longer enforce neighbors to be loaded and leave that to the BORDER status, which is exactly how all other worldgen stages work.
On a further note, there seems to be a clash between the two concepts of worldgen stages (which are in the code represented as the ChunkStatus) and between the accessibility status. The World exposes a method Chunk getChunk(int chunkX, int chunkZ, ChunkStatus leastStatus, boolean create) to load a chunk into the specified worldgen stage, and in fact WorldChunk getChunk(int i, int j) just calls this with ChunkStatus.FULL. However, as this problem here shows, the ChunkStatus only contains the persistent information about the worldgen stage the chunk has passed and does not correspond to the ticket level that keeps the chunk loaded. In most cases this is not the desired information and it would be more appropriate to provide the required accessibility status or something similar instead.
On the other hand, digging into the code shows that all instances where this actually matters call the method with the create flag set to true which then issues a chunk ticket with the correct level for a border chunk. So in all these cases the chunk is indeed implicitly loaded in the BORDER state as usually required. Nevertheless, the Future returned by the method only depends on the full stage, so the promotion to the BORDER status usually happens slightly after the future is resolved, which can again cause problems.In conclusion I would suggest to think about a more appropriate public interface for getChunk(...) that conveys the correct concept, which in my opinion the ChunkStatus does not.
Best,
PhiProUpon chunk loading, promotion of a chunk to the light stage does require neighbors to be loaded in the features stage (which is the stage at which the lighting engine is able to interact with a chunk), so that border chunks, i.e., the last accessible chunks, directly after they are loaded do have their neighbors loaded in the correct state for the lighting engine to interact with them. However, if those neighbors get unloaded later on, the chunk is not demoted from the light stage again, so you can have chunks in light (and full) stage without all their neighbors being loaded. In particular, this can happen for border chunks which can then lead to light updates getting stuck at the chunk boundary to the unloaded neighbor.
This can be visualized using the following steps:
- Create a redstone-ready world
- Set the render distance to 2
- /setworldspawn 1000 56 1000
- Fly to chunk (-14 0) (This will unload chunk (1 0))
- Fly to chunk (-3 0)
- /setblock 14 56 8 minecraft:sea_lantern
- Fly back to chunk (0 0) and observe that the light got stuck at the boundary to (1 0)
Currently, promotion of a chunk to the BORDER state only requires the chunk itself to be loaded as full chunk. I propose to add here the requirement that also the neighbors should be loaded in the features stage. In contrast to the worldgen stages, e.g., light or full, the accessibility status, i.e., INACCESSIBLE, BORDER, TICKING and ENTITY_TICKING does get demoted if the chunk ticket level drops too low and hence the condition is reevaluated each time the chunk gets promoted again. Hence with this change promotion to the BORDER status would re-enforce neighbors to be loaded in the features stage.
This simply requires to changenet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return holder.getChunkAt(ChunkStatus.FULL, this).thenApplyAsync((either) -> { ... }to
public CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return this.getRegion(holder.getPos(), 1, ChunkStatus::byDistanceFromFull).thenApplyAsync((either) -> { ... }
Also note that the light stage is currently special cased to enforce the neighbor-loading even if the chunk was already generated into the light stage before.
net.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunk(ChunkHolder holder, ChunkStatus requiredStatus) { ... if (chunk.getStatus().isAtLeast(requiredStatus)) { CompletableFuture completableFuture2; if (requiredStatus == ChunkStatus.LIGHT) { completableFuture2 = this.upgradeChunk(holder, requiredStatus); } else { completableFuture2 = requiredStatus.runLoadTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(holder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, requiredStatus); return completableFuture2; } else { return this.upgradeChunk(holder, requiredStatus); } ... }
With my proposed change this special handling is no longer needed. The light stage will still require neighbors in the features stage for the initial lighting, but once the chunk is generated into the light stage it should no longer enforce neighbors to be loaded and leave that to the BORDER status, which is exactly how all other worldgen stages work.
On a further note, there seems to be a clash between the two concepts of worldgen stages (which are in the code represented as the ChunkStatus) and between the accessibility status. The World exposes a method Chunk getChunk(int chunkX, int chunkZ, ChunkStatus leastStatus, boolean create) to load a chunk into the specified worldgen stage, and in fact WorldChunk getChunk(int i, int j) just calls this with ChunkStatus.FULL. However, as this problem here shows, the ChunkStatus only contains the persistent information about the worldgen stage the chunk has passed and does not correspond to the ticket level that keeps the chunk loaded. In most cases this is not the desired information and it would be more appropriate to provide the required accessibility status or something similar instead.
On the other hand, digging into the code shows that all instances where this actually matters call the method with the create flag set to true which then issues a chunk ticket with the correct level for a border chunk. So in all these cases the chunk is indeed implicitly loaded in the BORDER state as usually required. Nevertheless, the Future returned by the method only depends on the full stage, so the promotion to the BORDER status usually happens slightly after the future is resolved, which can again cause problems.In conclusion I would suggest to think about a more appropriate public interface for getChunk(...) that conveys the correct concept, which in my opinion the ChunkStatus does not.
Best,
PhiPro
Upon chunk loading, promotion of a chunk to the light stage does require neighbors to be loaded in the features stage (which is the stage at which the lighting engine is able to interact with a chunk), so that border chunks, i.e., the last accessible chunks, directly after they are loaded do have their neighbors loaded in the correct state for the lighting engine to interact with them. However, if those neighbors get unloaded later on, the chunk is not demoted from the light stage again, so you can have chunks in light (and full) stage without all their neighbors being loaded. In particular, this can happen for border chunks which can then lead to light updates getting stuck at the chunk boundary to the unloaded neighbor.
This can be visualized using the following steps:
- Create a redstone-ready world
- Set the render distance to 2
- /setworldspawn 1000 56 1000
- Fly to chunk (-14 0) (This will unload chunk (1 0))
- Fly to chunk (-3 0)
- /setblock 14 56 8 minecraft:sea_lantern
- Fly back to chunk (0 0) and observe that the light got stuck at the boundary to (1 0)
Currently, promotion of a chunk to the BORDER state only requires the chunk itself to be loaded as full chunk. I propose to add here the requirement that also the neighbors should be loaded in the features stage. In contrast to the worldgen stages, e.g., light or full, the accessibility status, i.e., INACCESSIBLE, BORDER, TICKING and ENTITY_TICKING does get demoted if the chunk ticket level drops too low and hence the condition is reevaluated each time the chunk gets promoted again. Hence with this change promotion to the BORDER status would re-enforce neighbors to be loaded in the features stage.
This simply requires to changenet.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return holder.getChunkAt(ChunkStatus.FULL, this).thenApplyAsync((either) -> { ... }to
public CompletableFuture<Either<WorldChunk, ChunkHolder.Unloaded>> makeChunkAccessible(ChunkHolder holder) { return this.getRegion(holder.getPos(), 1, ChunkStatus::byDistanceFromFull).thenApply(either -> either.mapLeft(list -> list.get(list.size() / 2))).thenApplyAsync((either) -> { ... }
Also note that the light stage is currently special cased to enforce the neighbor-loading even if the chunk was already generated into the light stage before.
net.minecraft.server.world.ThreadedAnvilChunkStorage.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunk(ChunkHolder holder, ChunkStatus requiredStatus) { ... if (chunk.getStatus().isAtLeast(requiredStatus)) { CompletableFuture completableFuture2; if (requiredStatus == ChunkStatus.LIGHT) { completableFuture2 = this.upgradeChunk(holder, requiredStatus); } else { completableFuture2 = requiredStatus.runLoadTask(this.world, this.structureManager, this.serverLightingProvider, (chunkx) -> { return this.convertToFullChunk(holder); }, chunk); } this.worldGenerationProgressListener.setChunkStatus(chunkPos, requiredStatus); return completableFuture2; } else { return this.upgradeChunk(holder, requiredStatus); } ... }
With my proposed change this special handling is no longer needed. The light stage will still require neighbors in the features stage for the initial lighting, but once the chunk is generated into the light stage it should no longer enforce neighbors to be loaded and leave that to the BORDER status, which is exactly how all other worldgen stages work.
On a further note, there seems to be a clash between the two concepts of worldgen stages (which are in the code represented as the ChunkStatus) and between the accessibility status. The World exposes a method Chunk getChunk(int chunkX, int chunkZ, ChunkStatus leastStatus, boolean create) to load a chunk into the specified worldgen stage, and in fact WorldChunk getChunk(int i, int j) just calls this with ChunkStatus.FULL. However, as this problem here shows, the ChunkStatus only contains the persistent information about the worldgen stage the chunk has passed and does not correspond to the ticket level that keeps the chunk loaded. In most cases this is not the desired information and it would be more appropriate to provide the required accessibility status or something similar instead.
On the other hand, digging into the code shows that all instances where this actually matters call the method with the create flag set to true which then issues a chunk ticket with the correct level for a border chunk. So in all these cases the chunk is indeed implicitly loaded in the BORDER state as usually required. Nevertheless, the Future returned by the method only depends on the full stage, so the promotion to the BORDER status usually happens slightly after the future is resolved, which can again cause problems.In conclusion I would suggest to think about a more appropriate public interface for getChunk(...) that conveys the correct concept, which in my opinion the ChunkStatus does not.
Best,
PhiPro
This is related to the discussion of
MC-196614. I want to point out here that the stripping of uninitialized lightmaps, as disussed inMC-196614, already causes issues on its own without relying onMC-196614, i.e., queued lightmaps being ignored for lightmap initialization.
I put this here mainly for reference. Similarly toMC-196614this is automatically fixed by my proposed solution toMC-196725.
Consider a similar setup as in the other report:
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill -15 32 -15 63 32 63 minecraft:sandstone /fill 16 32 16 31 32 31 minecraft:air /fill 16 1 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air replace minecraft:sandstone /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 1)
- Fly back to chunk (1 1) and observe that subchunk (1 0 1) is fully bright (The other broken lightmaps are instances of
MC-196614and can be ignored for this discussion)
Upon saving chunk (1 1) the lightmap at (1 0 1) gets stripped is at was uninitialized. Upon reloading it gets "correctly" (in comparison to
MC-196614) initialized by inheriting the values from the lightmap directly above it.
The issue in this case is that the lightmap should not have been initialized by inheriting values from above in the first place, but it should have been left uninitialized.
One could now patch this specific case by not initializing lightmaps for non-empty subchunks, at least during loading.
However, there are more subtle cases that show that there is in general no easy way to hackfix this issue:
- Create a redstone-ready world with generator settings 33*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 0 0 31 15 47 minecraft:air /fill 0 16 0 31 31 47 minecraft:air /fill 32 16 0 47 31 47 minecraft:air /fill 45 32 16 45 32 31 minecraft:air
- Fly to chunk (-14 1) and back to chunk (-2 1)
- /setblock 8 8 8 minecraft:sandstone
- Fly back to subchunk (1 0 1) and observe that the blocks at the boundary to subchunk (2 0 1) have a light value of 1 instead of 0.
Again, upon saving the uninitialized lightmap at (1 0 1) is stripped. When placing the block at (8 8 8) it is regenerated and "correctly" (in comparison to
MC-196614) inherits its value from the lightmap above. And again the lightmap should have been left uninitialized instead of inheriting from the lightmap above.
A conflict now arises with the following variation:
- In the initial block of commands, additionally /fill 32 0 0 47 15 47 minecraft:air
This additionally clears the remaining subchunks{({2 0 0) .. (2 1 2)}}.- This time the lightmap at (1 0 1) gets removed because there are no nearby non-empty subchunks
- When placing the block at (8 8 8) the lightmap at (1 0 1) needs to be regenerated again, but this time the correct value is indeed by inheriting from above.
- The only difference to the previous example lies in the chunks (2 0) .. (2 2) which are not loaded at the time the lightmap is generated. But the correct result is different from the prevvious example.
This shows that stripping uninitialized lightmaps really incurs a data loss that cannot be repaired easily.
Similar to the discussion of
MC-196614, you should not simply remove the stripping of uninitialized lightmaps, as this will lead to a "massive" increase in memory usage for lightmaps. Instead this issue should be fixed via my proposed solution forMC-196725.Also, when implementing my proposed fix for
MC-196725, old worlds should be forced to relight once in order to account for the missing data discussed here, rather than trying to somehow partially recover the missing data from broken old saves.
Best,
PhiProThis is related to the discussion of
MC-196614. I want to point out here that the stripping of uninitialized lightmaps, as disussed inMC-196614, already causes issues on its own without relying onMC-196614, i.e., queued lightmaps being ignored for lightmap initialization.
I put this here mainly for reference. Similarly toMC-196614this is automatically fixed by my proposed solution toMC-196725.
Consider a similar setup as in the other report:
- Create a redstone-ready world with generator settings 16*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill -15 32 -15 63 32 63 minecraft:sandstone /fill 16 32 16 31 32 31 minecraft:air /fill 16 1 16 31 15 31 minecraft:stone /fill 0 0 0 47 7 47 minecraft:air replace minecraft:sandstone /fill 0 8 0 47 15 47 minecraft:air replace minecraft:sandstone
- Fly to chunk (-14 1)
- Fly back to chunk (1 1) and observe that subchunk (1 0 1) is fully bright (The other broken lightmaps are instances of
MC-196614and can be ignored for this discussion)
Upon saving chunk (1 1) the lightmap at (1 0 1) gets stripped is at was uninitialized. Upon reloading it gets "correctly" (in comparison to
MC-196614) initialized by inheriting the values from the lightmap directly above it.
The issue in this case is that the lightmap should not have been initialized by inheriting values from above in the first place, but it should have been left uninitialized.
One could now patch this specific case by not initializing lightmaps for non-empty subchunks, at least during loading.
However, there are more subtle cases that show that there is in general no easy way to hackfix this issue:
- Create a redstone-ready world with generator settings 33*minecraft:sandstone;minecraft:desert
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 16 1000 /fill 0 0 0 31 15 47 minecraft:air /fill 0 16 0 31 31 47 minecraft:air /fill 32 16 0 47 31 47 minecraft:air /fill 45 32 16 45 32 31 minecraft:air
- Fly to chunk (-14 1) and back to chunk (-2 1)
- /setblock 8 8 8 minecraft:sandstone
- Fly back to subchunk (1 0 1) and observe that the blocks at the boundary to subchunk (2 0 1) have a light value of 1 instead of 0.
Again, upon saving the uninitialized lightmap at (1 0 1) is stripped. When placing the block at (8 8 8) it is regenerated and "correctly" (in comparison to
MC-196614) inherits its value from the lightmap above. And again the lightmap should have been left uninitialized instead of inheriting from the lightmap above.
A conflict now arises with the following variation:
- In the initial block of commands, additionally /fill 32 0 0 47 15 47 minecraft:air
This additionally clears the remaining subchunks (2 0 0) .. (2 1 2).- This time the lightmap at (1 0 1) gets removed because there are no nearby non-empty subchunks
- When placing the block at (8 8 8) the lightmap at (1 0 1) needs to be regenerated again, but this time the correct value is indeed by inheriting from above.
- The only difference to the previous example lies in the chunks (2 0) .. (2 2) which are not loaded at the time the lightmap is generated. But the correct result is different from the prevvious example.
This shows that stripping uninitialized lightmaps really incurs a data loss that cannot be repaired easily.
Similar to the discussion of
MC-196614, you should not simply remove the stripping of uninitialized lightmaps, as this will lead to a "massive" increase in memory usage for lightmaps. Instead this issue should be fixed via my proposed solution forMC-196725.Also, when implementing my proposed fix for
MC-196725, old worlds should be forced to relight once in order to account for the missing data discussed here, rather than trying to somehow partially recover the missing data from broken old saves.
Best,
PhiPro
The lighting engine creates lightmaps for exactly those chunks that have any nearby (i.e., in a radius of 1 subchunk) non-air blocks and deletes them once the last nearby subchunk becomes empty. As a consequence, unloading a chunk, which marks all its subchunks as empty, will delete ligtmaps of still loaded neighbors if there are no other non-empty subchunks around to keep the lightmaps alive.
This can be visualized via the following steps:
- Create a redstone-ready world
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 56 1000 /fill 16 80 0 16 95 15 minecraft:stone /setblock 16 88 8 minecraft:sea_lantern
- Fly to chunk (-14 0)
- Fly back to chunk (0 0) and observe that the lightmap for subchunk (0 5 0) got erased
There is also a variation for skylight
- Create a redstone-ready world
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 56 1000 /fill 0 128 0 47 128 47 minecraft:stone /fill 32 80 16 32 127 31 minecraft:stone /fill 33 128 16 47 128 31 minecraft:air /fill 32 96 16 32 96 31 minecraft:air
- Fly to chunk (-13 0)
- Fly back to chunk (1 0) and observe that the lightmap for subchunk (1 5 1) and (1 6 1) got erased
The vanilla code contains some data retainment mechanism that puts lightmaps back to the queued lightmaps, rather then deleting them. However, this mechanism is disabled upon promoting a chunk to the light stage.
One possible solution for this issue would be to reenable this retainment mechanism once a neighbor chunk gets unloaded.
I propose an alternative to this data retainment mechanism. This will decouple the lightmap handling from the skylight-optimization distance tracking that currently controls the lightmaps. Hence this approach will naturally avoid the bug discussed here and make the vanilla data retainment mechanism obsolete.
Furthermore, this approach will naturally contain a complete fix forMC-196614and provide a more aggressive cleanup for trivial lightmaps, compared to the current vanilla code.
- The main idea is to create lightmaps on demand, i.e., when the lighting engine wants to set a light value, and delete them once they become trivial in the appropriate sense. When there is no lightmap associated to a subchunk, then skylight is inherited from the next lightmap above or direct skylight if there is none, whereas blocklight is simply 0. In order to check when a lightmap is trivial we need to track some complexity measure that can be easily updated upon chaning light values and is 0 iff the lightmap is trivial, i.e., if it coincides with the values that would be produced with no lightmap present.
- For example, we can simply take the sum of all blocklight values and for skylight we can for each position in the subchunk take the absolute value of the difference between the light value at that position and the light value at the position above (where in case of the topmost layer the light value at the position above needs to be taken from the lightmap above or direct skylight) and then sum up all these values. Note that in case of skylight this complexity measure not only depends on the lightmap but also on its position in the world.
- One important point is now that creating and removing trivial lightmaps does not change any light values, making the lightmap handling completely transparent to the lighting propagator. This also means that newly created lightmaps will always have a complexity of 0. This does however require
MC-170010to be fixed first, as otherwise some lightmaps will not be properly initialized upon creation, causing a change of light values and hence messing around with the light propagator. This causes a bit of a cyclic dependency between the two bug reports, so they should ideally be fixed simultaneously.
- The skylight optimization is then applicable to subchunks that are not near any non-air blocks, as determined by the current distance tracking, and that don't have an associated lightmap. This second condition that having a non-trivial lightmap disables the skylight optimization basically takes care of the second part of
MC-196614.- Note that changing a light value at the bottom of a lightmap will cause subchunks below without an associated lightmap to automatically change their light value as well, since that is how missing lightmaps are handled in the light lookup. This means that before changing a light value at the bottom of a lightmap, we first have to find the next subchunk below for which skylight optimization is not applicable and create a lightmap for it, in case it does not already have one, fixing the old light value so that changes are then properly propagated through the usual skylight optimization.
- It is convenient to forcefully disable any light propagations to chunks before the pre_light stage (using terminology of
MC-170012), allowing to defer the complexity initialization until the promotion to the pre_light stage. When unloading chunks, propagations should then be forcefully disabled again, so that the unloaded region is very well controlled. Ideally, this manual mechanism should not be necessary as any violation of it is always accompanied by some lighting glitch. However, in view ofMC-164281it might be a good idea to take some precautions and avoid further bugs caused by screwed complexity trackings.- Furthermore, for the unloaded region lightmaps can be directly added to the world as nothing will be interacting with them, given that we have just forcefully disabled all such interactions. Lightmaps getting queued for already loaded chunks, i.e., on the client, might be better placed in some queue first, so they can be added at a more controlled point in time. Furthermore, when adding such a lightmap to an already loaded chunk its complexity tracking has to be reevaluated and if any value at the bottom changes, the necessary steps need to be taken for skylight, as explained above.
- Removal of trivial lightmaps should only be done once every update cycle or upon saving or something similar, in order to avoid rapidly removing and reallocating lightmaps
One concern that might come up is regarding the interaction between skylight optimization and unloaded chunks. More concretely, note that the last accessible chunk is in the full state but its neighbors are only guaranteed to be in pre_light state. Hence there can be light updates into chunks that are only in pre_light state and their neighbors are not guaranteed to be loaded. So the issue one might see here is that there are light updates into chunks that do not have complete information about their neighbors, so the skylight optimization might be applied in situations where it shouldn't. We will in the following sketch a proof that the extra condition that the skylight optimization is disabled for subchunks with an associated lightmap is in fact sufficient to ensure correct results. Note that some similar discussion could have been already placed under
MC-170012orMC-196614, but I think it fits here nicely as well. Of course one could simply avoid this whole discussion by just increasing the chunk loading distance by 1, so that light updates only take place in chunks that are in full state; however that would be rather inefficient given that this is actually not an issue.For the following discussion we will assume
MC-170012to be fixed and use its terminology. We will assume from the chunk loading mechanism that whenever a block change in a chunk c1 could potentially cause a light update into chunk c2 (or some boundary touching it) then c2 should be loaded in pre_light stage. Note that any violation of this will result in a lighting glitch simply because light updates would get stuck at chunk boundaries, so this would be a completely unrelated issue on its own. More technically, we can state this condition as follows: If there occurs a block update in some chunk c1 and we are given two neighbors c and c2 of it (where we also count diagonal neighbors and also say that a chunk is a neighbor of itself) such that they are also neighbors to one another and such that c has its initial lighting done, i.e., was generated into the light stage but is not necessarily loaded currently, then c2 should be loaded in pre_light stage.
Note that this assumption might not be true on the client. However there are other mechanisms taking care of such effects for the client light syncing, so we don't want to worry about this here.We want to show that the end result of our lighting model is correct, for which it suffices to show local correctness, i.e., each block has a light value that is precisely the maximum over all contributions of its neighbors, where the contributions are specified in the ideal lighting model where no skylight optimization takes place and all chunks are loaded. It is always true that subchunks for which no skylight optimization is applied are locally correct and also unloaded chunks retain their local correctness since by the assumption above no block directly adjacent to an unloaded chunk changes its light value. Hence it is enough to show local correctness for subchunks for which skylight optimization is applied
So, for sake of concreteness assume that skylight optimization is applied to subchunk (0 0 0) for which we want to show local correctness. We may assume inductively that all subchunks at y-level >= 1 are locally-correct.
Consider some horizontal chunk border of it, e.g., the border to chunk (-1 0), consisting of the blocks (0 0 0)..(0 15 15). If no light value for a block adjacent to this border changed, i.e., for no block in the region (-1 -1 -1)..(1 16 16) then it retains its local correctness and there is nothing to check for this border. Otherwise, such a light change must come from a block change in a chunk from the region (-1 -1)..(0 1) and the skylight source propagated through this block change must lie in the same chunk region and furthermore this skylight source and the block change must lie in chunks that are neighbors of one another. Hence we can conclude by the assumption about chunk loading that some quadrant containing the border is loaded in pre_light stage, i.e., the quadrant (-1 -1)..(0 0) or (-1 0)..(0 1). By the specification for the skylight optimization we then know that the subchunks of this quadrant that lie in the region (-10-1)..(1 1 1) contain only air blocks.
Consider now the region consisting of all such quadrants intersected with (-10-1)..(1 1 1). This region then has the following properties which allow to deduce quite a lot of information about the lighting model:
- It is star-shaped with respect to (0
00)- It contains only air blocks
- Its boundary can be decomposed into 3 parts:
- The top faces
- The faces that are one subchunk away from (0 0 0) (excluding the top faces)
- The faces touching (0 0 0)
- The faces of (0 0 0) that also belong to the boundary of this region retain local correctness, so there is nothing to check for those
We need to show local correctness for those blocks in (0 0 0) that are strictly inside this region, i.e., excluding those borders for which we already know it. We now consider two lighting models: The
ideal model not applying any skylight optimization and thereal model applying the skylight optimization according to our specification, where we replace propagations at y-levels >= 16 with ideal ones, as we already know local correctness in that area by induction.We will show that for any propagation path from the boundary of the constructed region into(0 0 0)in the real model there exists a corresponding propagation path from the boundary to the same block in the ideal model such that both give rise to the same contribution, and conversely for every propagation path in the ideal model there is a corresponding path in the real model. This suffices since this implies that both models lead to the same result and theideal model is locally correct.
Note that the distinction between changed and unchanged borders was necessary as the unchanged borders are now treated as sources in this model, hence proving local correctness in this model does not provide useful information about them (which is no problem as we already know that these are fine).
We consider the following cases according to the decomposition of the boundary as above:
- Both in the real and ideal model there are no propagation paths from the faces that are one subchunks away from (0 0 0), excluding the top faces, so this case is easy
- Both in the real and ideal model the only propagation paths starting from the top faces start with direct skylight access, i.e., with a skylight value of 15. An optimal propagation path in the ideal model then first goes straight down to the correct y-value and then propagates in the xz-plane. An optimal path in the real model first goes straight down until y=16, then propagates in the xz-plane and then propagates for the remaining y-level using the skylight optimization, hence not reducing the light value. This gives a 1:1 correspondence between optimal ideal and optimal real paths producing the same light contributions, as desired.
- For sake of illustration, suppose that the constructed region consists of the chunks (0
-1)..(1 1), so that the remaining faces have x-value 0. Both foroptimalreal and ideal propagation paths we can find an alternative onethat first travels along these faces, i.e., stays at x=0, and then propagates in the xy-plane. Note that we can simply leave outthisfirstpart as the new starting point will then still lieon thefaces under consideration, so we can restrict our attention to those optimal paths onlytravelling in the xy-plane, so in particular the starting point lies in the chunk(0 0) and we can restrict the y-level to y=0..16.
In the real model an optimal path now starts at y=16 then travels to the correct x-value and then travels down using skylight optimization, so that the value doesn't change. An optimal ideal path can be chosen to first travel down to the correct y-value and then travel in x-direction. The important observation is now that sincesubchunk (0 0 0)does not have an associated lightmap, its light values are constant along columns, so travelling down along this face does not change the light value. Hence an optimal ideal path will directly startatthecorrect y-level and then onlytravelinx- direction. And it has thesamestartingvalue as the optimal real path that starts at y=16, again producing a 1:1 correspondence between optimal real and ideal propagation paths giving rise to the same contributions.This finishes this rather lengthy proof
(without giving full details of the optimal propagation paths).
Best,
PhiProThe lighting engine creates lightmaps for exactly those chunks that have any nearby (i.e., in a radius of 1 subchunk) non-air blocks and deletes them once the last nearby subchunk becomes empty. As a consequence, unloading a chunk, which marks all its subchunks as empty, will delete ligtmaps of still loaded neighbors if there are no other non-empty subchunks around to keep the lightmaps alive.
This can be visualized via the following steps:
- Create a redstone-ready world
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 56 1000 /fill 16 80 0 16 95 15 minecraft:stone /setblock 16 88 8 minecraft:sea_lantern
- Fly to chunk (-14 0)
- Fly back to chunk (0 0) and observe that the lightmap for subchunk (0 5 0) got erased
There is also a variation for skylight
- Create a redstone-ready world
- Set the render distance to 2
- Run the following commands
/setworldspawn 1000 56 1000 /fill 0 128 0 47 128 47 minecraft:stone /fill 32 80 16 32 127 31 minecraft:stone /fill 33 128 16 47 128 31 minecraft:air /fill 32 96 16 32 96 31 minecraft:air
- Fly to chunk (-13 0)
- Fly back to chunk (1 0) and observe that the lightmap for subchunk (1 5 1) and (1 6 1) got erased
The vanilla code contains some data retainment mechanism that puts lightmaps back to the queued lightmaps, rather then deleting them. However, this mechanism is disabled upon promoting a chunk to the light stage.
One possible solution for this issue would be to reenable this retainment mechanism once a neighbor chunk gets unloaded.I propose an alternative to this data retainment mechanism. This will decouple the lightmap handling from the skylight-optimization distance tracking that currently controls the lightmaps. Hence this approach will naturally avoid the bug discussed here and make the vanilla data retainment mechanism obsolete.
Furthermore, this approach will naturally contain a complete fix forMC-196614and provide a more aggressive cleanup for trivial lightmaps, compared to the current vanilla code.
- The main idea is to create lightmaps on demand, i.e., when the lighting engine wants to set a light value, and delete them once they become trivial in the appropriate sense. When there is no lightmap associated to a subchunk, then skylight is inherited from the next lightmap above or direct skylight if there is none, whereas blocklight is simply 0. In order to check when a lightmap is trivial we need to track some complexity measure that can be easily updated upon chaning light values and is 0 iff the lightmap is trivial, i.e., if it coincides with the values that would be produced with no lightmap present.
- For example, we can simply take the sum of all blocklight values and for skylight we can for each position in the subchunk take the absolute value of the difference between the light value at that position and the light value at the position above (where in case of the topmost layer the light value at the position above needs to be taken from the lightmap above or direct skylight) and then sum up all these values. Note that in case of skylight this complexity measure not only depends on the lightmap but also on its position in the world.
- One important point is now that creating and removing trivial lightmaps does not change any light values, making the lightmap handling completely transparent to the lighting propagator. This also means that newly created lightmaps will always have a complexity of 0. This does however require
MC-170010to be fixed first, as otherwise some lightmaps will not be properly initialized upon creation, causing a change of light values and hence messing around with the light propagator. This causes a bit of a cyclic dependency between the two bug reports, so they should ideally be fixed simultaneously.
- The skylight optimization is then applicable to subchunks that are not near any non-air blocks, as determined by the current distance tracking, and that don't have an associated lightmap. This second condition that having a non-trivial lightmap disables the skylight optimization basically takes care of the second part of
MC-196614.- Note that changing a light value at the bottom of a lightmap will cause subchunks below without an associated lightmap to automatically change their light value as well, since that is how missing lightmaps are handled in the light lookup. This means that before changing a light value at the bottom of a lightmap, we first have to find the next subchunk below for which skylight optimization is not applicable and create a lightmap for it, in case it does not already have one, fixing the old light value so that changes are then properly propagated through the usual skylight optimization.
- It is convenient to forcefully disable any light propagations to chunks before the pre_light stage (using terminology of
MC-170012), allowing to defer the complexity initialization until the promotion to the pre_light stage. When unloading chunks, propagations should then be forcefully disabled again, so that the unloaded region is very well controlled. Ideally, this manual mechanism should not be necessary as any violation of it is always accompanied by some lighting glitch. However, in view ofMC-164281it might be a good idea to take some precautions and avoid further bugs caused by screwed complexity trackings.- Furthermore, for the unloaded region lightmaps can be directly added to the world as nothing will be interacting with them, given that we have just forcefully disabled all such interactions. Lightmaps getting queued for already loaded chunks, i.e., on the client, might be better placed in some queue first, so they can be added at a more controlled point in time. Furthermore, when adding such a lightmap to an already loaded chunk its complexity tracking has to be reevaluated and if any value at the bottom changes, the necessary steps need to be taken for skylight, as explained above.
- Removal of trivial lightmaps should only be done once every update cycle or upon saving or something similar, in order to avoid rapidly removing and reallocating lightmaps
One concern that might come up is regarding the interaction between skylight optimization and unloaded chunks. More concretely, note that the last accessible chunk is in the full state but its neighbors are only guaranteed to be in pre_light state. Hence there can be light updates into chunks that are only in pre_light state and their neighbors are not guaranteed to be loaded at all. So the issue one might see here is that there are light updates into chunks that do not have complete information about their neighbors, so the skylight optimization might be applied in situations where it shouldn't. We will in the following sketch a proof that the extra condition that the skylight optimization is disabled for subchunks with an associated lightmap is in fact sufficient to ensure correct results. Note that some similar discussion could have been already placed under
MC-170012orMC-196614, but I think it fits here nicely as well. Of course one could simply avoid this whole discussion by just increasing the chunk loading distance by 1, so that light updates only take place in chunks that are in full state; however that would be rather inefficient given that this is actually not an issue.For the following discussion we will assume
MC-170012to be fixed and use its terminology. We will assume from the chunk loading mechanism that whenever a block change in a chunk c1 could potentially cause a light update into chunk c2 (or some boundary touching it) then c2 should be loaded in pre_light stage. Note that any violation of this will result in a lighting glitch simply because light updates would get stuck at chunk boundaries, so this would be a completely unrelated issue on its own. More technically, we can state this condition as follows: If there occurs a block update in some chunk c1 and we are given two neighbors c and c2 of it (where we also count diagonal neighbors and also say that a chunk is a neighbor of itself) such that they are also neighbors to one another and such that c has its initial lighting done, i.e., was generated into the light stage but is not necessarily loaded currently, then c2 should be loaded in pre_light stage.
Note that this assumption might not be true on the client. However there are other mechanisms taking care of such effects for the client light syncing, so we don't want to worry about this here.We want to show that the end result of our lighting model is correct, for which it suffices to show local correctness, i.e., each block has a light value that is precisely the maximum over all contributions of its neighbors, where the contributions are specified in the ideal lighting model where no skylight optimization takes place and all chunks are loaded. It is always true that subchunks for which no skylight optimization is applied are locally correct and also unloaded chunks retain their local correctness since by the assumption above no block directly adjacent to an unloaded chunk changes its light value. Hence it is enough to show local correctness for subchunks for which skylight optimization is applied.
So, for sake of concreteness assume that skylight optimization is applied to subchunk (0 0 0) for which we want to show local correctness. We may assume inductively that all subchunks at y-level >= 1 are locally-correct.
Consider some horizontal chunk border of it, e.g., the border to chunk (-1 0), consisting of the blocks (0 0 0)..(0 15 15). If no light value for a block adjacent to this border changed, i.e., for no block in the region (-1 -1 -1)..(1 16 16) then it retains its local correctness and there is nothing to check for this border. Otherwise, such a light change must come from a block change in a chunk from the region (-1 -1)..(0 1) and the skylight source propagated through this block change must lie in the same chunk region and furthermore this skylight source and the block change must lie in chunks that are neighbors of one another. Hence we can conclude by the assumption about chunk loading that some quadrant containing the border is loaded in pre_light stage, i.e., the quadrant (-1 -1)..(0 0) or (-1 0)..(0 1). By the specification for the skylight optimization we then know that the subchunks of this quadrant that lie in the region (-1 -1 -1)..(1 1 1) contain only air blocks. Similarly, we conclude that a corner-column of subchunk (0 0 0) either retains local correctness or the unique quadrant containing the column in its interior is loaded in pre_light stage and the intersection with (-1 -1 -1)..(1 1 1) consists purely of air blocks.
Consider now the region consisting of all such quadrants intersected with (-1 -1 -1)..(1 1 1). This region then has the following properties which allow to deduce quite a lot of information about the lighting model:
- It is star-shaped with respect to chunk (0 0)
- It contains only air blocks
- Its boundary can be decomposed into 3 parts:
- The top faces
- The faces that are one subchunk away from (0 0 0) (excluding the top faces)
- The faces touching (0 0 0)
- The faces and corner-columns of (0 0 0) that also belong to the boundary of this region retain local correctness, so there is nothing to check for those
We need to show local correctness for those blocks in (0 0 0) that are strictly inside this region, i.e., excluding those borders for which we already know it. We now consider two lighting models: The real model applying the skylight optimization according to our specification, where we replace propagations at y-levels >= 16 with ideal ones, as we already know local correctness in that area by induction. And the semi-ideal model that additionally treats subchunk (0 0 0) as non-optimizable, i.e., uses ideal propagations. We will show that both models give the same solution with boundary conditions given by the boundary of the region. This suffices since the solution to the semi-ideal model is locally correct on subchunk (0 0 0) by construction. In order to show this it is in turn sufficient to show that both solutions agree on subchunk (0 0 0) since we can then add it to the boundary conditions and both models are identical outside this subchunk, hence have the same solution.
For this we show that for any propagation path from the boundary of the constructed region into subchunk (0 0 0) in the real model there exists a corresponding propagation path from the boundary to the same block in the semi-ideal model that has at least the same contribution, and conversely for every propagation path in the semi-ideal model there is a corresponding path in the real model that has at least the same contribution.
Note that the distinction between changed and unchanged borders was necessary as the unchanged borders are now treated as sources in this model, hence proving local correctness in this model does not provide useful information about them (which is no problem as we already know that these are fine).
We consider the following cases according to the decomposition of the boundary as above:
- Both in the real and semi-ideal model there are no propagation paths from the faces that are one subchunks away from (0 0 0), excluding the top faces, so this case is easy
- Both in the real and semi-ideal model the only propagation paths starting from the top faces start with direct skylight access, i.e., with a skylight value of 15. An optimal propagation path in the real model first goes straight down until y=16, then propagates in the xz-plane and then propagates for the remaining y-level using the skylight optimization, hence not reducing the light value. An optimal path in the semi-ideal model only does part of its xz-movement at y=16 and does the rest at the final y-level, since subchunk (0 0 0) is treated as non-optimizable. This gives a 1:1 correspondence between optimal real and optimal semi-ideal paths giving the same light contributions, as desired.
- Finally consider propagation paths starting from a face touching (0 0 0). Both for real and semi-ideal propagation paths we can find an alternative one giving at least the same contribution that first travels along the boundary of the region, then travels on the xz-plane inside a single non-optimizable subchunk (including (0 0 0) in case of the semi-ideal model) and then possibly travels downwards through an optimizable subchunk using skylight optimization. To prove this consider such a path appended by a single-step propagation into a non-optimizable subchunk. Using that the region consists purely of air blocks, the propagations can then be reordered in such a way that the subchunk is crossed along the boundary of the region, so that this part can be absorbed into the first part of the path and the resulting path is again of the desired form.
Note that the boundary of the region is "locally correct" with respect to the real model, i.e., each light level on the boundary is at least the maximum of all real contributions from neighbors lying on the boundary, or in other words, the propagated value of a real path along the boundary is at most the given boundary value. This is true simply because the boundary values are chosen from the global solution of the real model. The same is also true for the semi-ideal model, since it only differs from the real model on subchunk (0 0 0) and for blocks of this subchunk that belong to the boundary of the region we know that they are locally correct in the ideal model and hence also in the semi-ideal model. Hence we can leave out this first part of the propagation paths travelling along the boundary as the new starting point will then still lie on the boundary and has at least the same contribution by the argument just given.
So we can restrict our attention to those optimal paths first travelling on the xz-plane of a single non-optimizable subchunk and then possibly travelling downwards through an optimizable subchunk using skylight optimization.
In the real model an optimal path now starts at y=16 (which is treated as a non-optimizable subchunk in both models) then travels on the xz-plane and then travels down into subchunk (0 0 0) using skylight optimization, so that the value doesn't change. An optimal semi-ideal path starts at the correct y-value and then travel on the xz-plane. The important observation is now that since subchunk (0 0 0) is optimizable, it does not have an associated lightmap and hence its light values are constant along columns (including y=16). Hence we can transform an optimal real path into an optimal semi-ideal path by starting at the correct y-value, which then has the same starting value and gives the same contribution. And also conversely an optimal semi-ideal path can be transformed into a real path by starting at y=16 with the same starting value and same contribution. This produces again a 1:1 correspondence between optimal real and semi-ideal propagation paths giving rise to the same contributions, as desired.
This finishes this rather lengthy proof.
Note that this argument requires a slightly inefficient specification for the skylight optimization, namely that non-air blocks also make the neighbor subchunks above non-optimizable. This is the one implemented by Vanilla. (This was used when transforming paths to first travel along the boundary and then only inside a single chunk. Without this assumptions the subchunks at y=-1 might not be empty and there would be more complicated optimal propagation paths).
In the absence of unloaded chunks this is not needed and it would be sufficient that a non-air block only makes neighbor subchunks that have less or equal y-value non-optimizable. In the presence of unloaded chunks the argument given above requires the slightly inefficient version and in fact one can (easily) construct examples showing that the more efficient specification would produce wrong results and is hence not sufficient.
So if one would like to implement this optimization, extra care needs to be taken in the presence of unloaded chunks and the optimization needs to be disabled in those cases or information about unloaded neighbors needs to be provided through other means.
Best,
PhiPro
As requested, this is a copy of this
comment onMC-170012, posted as a separate report.
Upon loading from disk, lightmaps are only loaded for chunks that were already lighted and discarded otherwise, although they are always saved to disk.
net.minecraft.world.ChunkSerializer.javapublic static ProtoChunk deserialize(...) { ... boolean bl = compoundTag.getBoolean("isLightOn"); ... if (bl) { if (compoundTag2.contains("BlockLight", 7)) { lightingProvider.enqueueSectionData(LightType.BLOCK, ChunkSectionPos.from(pos, k), new ChunkNibbleArray(compoundTag2.getByteArray("BlockLight")), true); } if (bl2 && compoundTag2.contains("SkyLight", 7)) { lightingProvider.enqueueSectionData(LightType.SKY, ChunkSectionPos.from(pos, k), new ChunkNibbleArray(compoundTag2.getByteArray("SkyLight")), true); } } ... }
This basically erases any light propagations to chunks in pre_light stage when unloading them before the light stage and hence causes lighting glitches.
The Vanilla code currently uses this mechanism to erase cached data, which only removes the isLightOn field but not the lightmaps themselves.
As a concrete example I use the setup of
MC-199952, which I suppose to be an instance of this:
- Create a new world with seed 1122583309043747515
- Set the render distance to 2
- /tp -616 77 200
- Reload the world
- Fly to -669 80 207 and observe the lighting glitch
Also note that the issue doesn't occur if omitting the reloading step, as expected.
As requested, this is a copy of this comment on
MC-170012, posted as a separate report.
Upon loading from disk, lightmaps are only loaded for chunks that were already lighted and discarded otherwise, although they are always saved to disk.
net.minecraft.world.ChunkSerializer.javapublic static ProtoChunk deserialize(...) { ... boolean bl = compoundTag.getBoolean("isLightOn"); ... if (bl) { if (compoundTag2.contains("BlockLight", 7)) { lightingProvider.enqueueSectionData(LightType.BLOCK, ChunkSectionPos.from(pos, k), new ChunkNibbleArray(compoundTag2.getByteArray("BlockLight")), true); } if (bl2 && compoundTag2.contains("SkyLight", 7)) { lightingProvider.enqueueSectionData(LightType.SKY, ChunkSectionPos.from(pos, k), new ChunkNibbleArray(compoundTag2.getByteArray("SkyLight")), true); } } ... }
This basically erases any light propagations to chunks in pre_light stage when unloading them before the light stage and hence causes lighting glitches.
The Vanilla code currently uses this mechanism to erase cached data, which only removes the isLightOn field but not the lightmaps themselves.
As a concrete example I use the setup of
MC-199952, which I suppose to be an instance of this:
- Create a new world with seed 1122583309043747515
- Set the render distance to 2
- /tp -616 77 200
- Reload the world
- Fly to -669 80 207 and observe the lighting glitch
Also note that the issue doesn't occur if omitting the reloading step, as expected.
The issue described in the following is the root cause of
MC-214. While it has only become visible in practice recently due to793MC-224893, I think this is still an important concurrency issue on its own and hence deserves its own report.
During worldgen, speciallightchunk tickets are created just before thelightgeneration stage, in order to keep the chunk and its neighbors loaded until initial lighting is finished. Or at least this seems to be the idea behind this mechanism. However, the ticket is removed right before startingtheactual initial lighting, rather than waiting until it is finished.net.minecraft.server.world.ServerLightingProvider.javapublic CompletableFuture<Chunk> light(Chunk chunk, boolean excludeBlocks) { ChunkPos chunkPos = chunk.getPos(); chunk.setLightOn(false); this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.PRE_UPDATE, Util.debugRunnable(() -> { ... super.setColumnEnabled(chunkPos, true); ... this.chunkStorage.releaseLightTicket(chunkPos); }, () -> "lightChunk " + chunkPos + " " + excludeBlocks )); return CompletableFuture.supplyAsync(() -> { chunk.setLightOn(true); super.setRetainData(chunkPos, false); return chunk; }, (runnable) -> this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.POST_UPDATE, runnable) ); }This is of course too early and leaves the possibility for the neighbor chunks to unload before the initial lighting has finished, causing lighting glitches at the chunk borders (the chunk itself is actually kept from unloading due to the pending light task). These border glitches can happen even in 1.16.
Note that chunks below the FEATURES stage are considered opaque by the lighting engine. Hence, even though the chunk itself is kept from completely unloading, its ticket level might drop belowFEATURESstagewhich would make the chunk opaque to the lighting engine (during initial lighting), hence causing dark chunks as observed inMC-214793. In 1.16, this issue does actually not occur since chunks won't be demoted once they finish a generation stage (more precisely, therespective Futurefor that generation stage stays valid, which is precisely what the lighting engine checks).However, this is no longer true in 1.17 due toMC-224893and hencetheissue actually becomes visible.In order to provoke this issue, one can traverse the world at high speed, e.g., using spectator mode (with very high speed) or tp commands in quick succession. Only a sparse set of chunks will make it to the LIGHT stage during this traversion, other chunks will already abort generation after the FEATURES stage. Furthermore, at the time these LIGHT stages are processed, the player is already far away, leaving the ticket level below the FEATURES level. Both of these facts together, sparseness of light tickets and chunk ticket levels being too low when the initial lighting is processed, make the above explanation applicable. And indeed one can observe completely dark chunks when visiting the traversed region again. Note however, that this might require some attempts due to the random nature of this bug. (In my test cases these were however quite reliably reproducable).
Note that while
MC-214793mentions corrupted features that often occur in conjunction with dark chunks when flying in spirals, no corrupted features were observed with the above steps. In conlusion, I think that the above steps really isolate the lighting bug itself, whereas the situation described inMC-214793is a convolution of the pure lighting bug and other phenomena ofMC-224893,see the latter for a small discussion.
Finally, let me mention that I did move the releaseLightTicket(...) call to the POST_UPDATE phase and no further dark chunks were observed, as expected. However, I did observe some small lighting glitches at chunk boundaries. This was in fact expected, since the light ticket only loads neighbors into LIQUID_CARVERS stage, whereas they should really be in FEATURES stage for the lighting engine to interact with them. So this is actually yet another issue with the light ticket mechanism (in 1.17).
Remarks
Let me also remark that this issue is actually an instance of a more general concurrency issue present in the chunk loading code. As mentioned in the beginning, it could already happen in 1.16 that neighbor chunks unload before the initial lighting is completed, causing light glitches at the boundaries, due to the light ticket being removed too early. This effect is not new to 1.17.
More generally, the chunk loading/worldgen code does not take any precautions to guarantee that a chunk stays loaded once it is passed to any worldgen step. Once a Future completes, the chunk is passed to the next generation step (on the worldgen thread) which can then hold on to the reference indefinitely. Independently, on the server thread, the chunk then may be unloaded before this next generation step does its work, so any progress after this point would be lost.
Similarly, the light ticket only handles the case of initial lighting. Any further updates after the initial lighting do not take any measures to keep the chunk and its neighbors in a state where the lighting engine can interact with them. So, any light updates after the LIGHT generation stage (but before promotion to FULL stage) might get stuck randomly due to chunks being unloaded or being demoted from the correct state.As a solution, one should consider to implement a mechanism similar to the light tickets for general interactions during worldgen.
Best,
PhiProIt turns out that
MC-214808,MC-216148andMC-214793are all caused by the same code change introduced in the 1.17 snapshots. In the following I will describe this change and explain how it leads to each of these bugs. I post this as a separate report as it affects multiple other reports, so it doesn't really fit anywhere as a comment.World generation/loading stores one Future for each generation stage of a chunk in a corresponding ChunkHolder. These Futures are created once something requests the chunk in the respective stage, under the condition that the chunk ticket level is sufficiently large to issue generation into that stage. What changed in the 1.17 snapshots is the handling of demotion of chunk ticket levels.
In 1.16 demoting the chunk ticket level completed all pending Future s above the new level with a special UNLOADED_CHUNK marker, while leaving already finished Future s untouched.net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { completableFuture.complete(either); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }This has two important consequences:
- Once a generation stage finishes, the Future stays completed with the correct result until the chunk is unloaded
- A Future that is still pending at some point can only be aborted (completed with UNLOADED_CHUNK, which does not abort the actual generation step) if the chunk ticket level drops too low, otherwise it will complete with a valid chunk. This requires some small argument since the Future cannot only be aborted directly by the above code, but also indirectly if any of its dependencies is aborted (completed with UNLOADED_CHUNK). However, the latter cannot happen if the ticket level stays high enough (after observing the pending Future), by construction of the ticket system.
In the 1.17 snapshots, this handling of demotion fundamentally changed. Instead of directly aborting the pending Future s above the new level, all Future s, including already finished ones, are replaced with a new one that completes with an UNLOADED_CHUNK once the original Future completes.
net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { this.futuresByStatus.set(i, completableFuture.thenApply(__ -> either)); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }As a result, both of the above properties are no longer valid:
- An already finished stage might be replaced with an UNLOADED_CHUNK_FUTURE
- If the chunk ticket level drops below, say, t then the corresponding Future is not directly aborted, but instead replaced with a Future that will be lazily aborted. If the ticket level is now raised above t again while the original Future is still running, then no new Future will be issued for this generation stage.
net.minecraft.server.world.ChunkHolder.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunkAt(ChunkStatus targetStatus, ThreadedAnvilChunkStorage chunkStorage) { int i = targetStatus.getIndex(); CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { Either<Chunk, ChunkHolder.Unloaded> either = (Either)completableFuture.getNow((Object)null); if (either == null || either.left().isPresent()) { return completableFuture; } } if (getTargetStatusForLevel(this.level).isAtLeast(targetStatus)) { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture2 = chunkStorage.getChunk(this, targetStatus); this.combineSavingFuture(completableFuture2, "schedule " + targetStatus); this.futuresByStatus.set(i, completableFuture2); return completableFuture2; } else { return completableFuture == null ? UNLOADED_CHUNK_FUTURE : completableFuture; } }This was fine in 1.16 since the Future could only be pending if it was not already (indirectly) aborted (which is exactly point 2 above). However, in 1.17 the Future is already indirectly aborted (it is replaced by a lazily aborted one) so it will always produce an UNLOADED_CHUNK even if the ticket level does not fall again. This breaks property 2.
It turns out that this code was changed in 21w06a, which is precisely the version where all 3 bugs were reported. (I actually checked the precise version retrospectively after figuring out the cause for these issues).
So, how does violation of these 2 properties cause the 3 linked bugs?
1. Ticking-Futures might never complete
Every time the ticket level is raised high enough, a chunk is promoted to BORDER, TICKING and ENTITY_TICKING state respectively. This will create additional Future s that depend on the generation Future s in some small neighborhood of the chunk. Upon completion, these will execute the respective tasks for promoting the chunk to the respective state, like registering tick schedulers, marking the chunk tickable/entity-tickable and sending the chunk data to players.
Assuming property 2 above, these Future s will always complete with a valid chunk at some point, unless the ticket level drops too low. In the latter case, the Future s get recreated once the chunk is promoted again, so this is not an issue.
Now, in the 1.17 snapshots property 2 is violated and hence these special Future s might never complete (with a valid chunk) since they might depend on lazily aborted Future s upon creation. Hence the respective promotion tasks might never execute. In particular, the ticking-Future, which is responsible for sending the chunk data to players, might not execute, hence causingMC-214808.
Note that all 3 Future s can fail independently. For example, I did observe chunks where the ticking-Future completed but the entity-ticking-Future did not, causing the chunk to load but entities to get stuck in these chunks. Also note that this issue can similarly lead to the "Chunk not there when requested: " error when the ServerChunkManager queries a lazily aborted Future for the FULL stage. I actually observed this crash a few time during debugging.In order to provoke this issue, one can try the following steps. Due to the random nature of these bugs, several attempts might be needed. Run
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
- /tp ~1000 ~ ~
in quick succession, e.g., with one tick inbetween, or while pausing the server thread through the debugger. The first teleport will create the ChunkHolder s at the target location and create the generation Future s up to FULL stage. The second tp will then lazily abort these Future s again and the third and final tp will create the ticking-Future s on the lazily aborted but still pending generation Future s, hence causingMC-214808.
I also observed this issue with only a single tp, although this was less reproducable. This might seem strange at first, since the ticket levels should change monotonically in this case and not experience the jitter required for the above explanation. I think this is caused by an interaction of the POST_TELEPORT chunk ticket and the player ticket throttling. The POST_TELEPORT ticket is created right after teleporting, but expires before the player tickets are added because of the throttling, hence causing the required jitter.2. Futures might be erased completely
If the chunk ticket level drops low enough, the corresponding ChunkHolder is scheduled for unloading. It can still be revoked up to the point where it is actually saved to NBT and all the unloading tasks are done. This can be a few ticks from the actual scheduling, depending on server load. By property 1, once the corresponding chunk has passed the first generation/loading stage, all Future s will keep the reference to this chunk, so that it can indeed be revoked from the ChunkHolder.
However, in 1.17 when the ChunkHolder is scheduled for unloading, all generation Future s are replaced with (lazily) aborted ones. If this ChunkHolder gets then later on revoked (before it could properly save and unload) all generation Future s are recreated and the very first stage then reloads the chunk from disk again. Hence, the chunk object gets replaced with a completely new version that is reloaded from disk, erasing any progress since the last save. This is exactlyMC-216148.
Note that the unloading tasks did not run in this scenario, so any externally stored data is still present. In particular, the chunk is still marked as loaded and will hence skip the block-entity loading step.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprivate CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> convertToFullChunk(ChunkHolder chunkHolder) { ... worldChunk2.loadToWorld(); if (this.loadedChunks.add(chunkPos.toLong())) { worldChunk2.setLoadedToWorld(true); worldChunk2.updateAllBlockEntities(); } ... }As a consequence, block-entity tickers will not be recreated and still reference the old copies, hence block-entities will not be ticked. This is indeed a side effect noted in the bug report.
In order to provoke this issue, run for example
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
in quick succession, so that the ChunkHolder s do not save and unload inbetween. This can be achieved by pausing the server thread, for example. The first tp will erase all generation Future s and the second tp will then regenerate them and reload the chunk from disk.
In the report, it is noted that this seems to happen near nether portals. I'm not really sure how this is correlated. Most certainly, the relevant feature of nether portals are the PORTAL tickets created upon using them. However, I don't know how they enter the equation. They might play a similar role to the POST_TELEPORT_TICKET of the previous section or it might be something else.
3. Chunks might drop below FEATURES stage during initial lighting
While
MC-214793is actually caused by another concurrency bug, namelyMC-224894, it might still be worthwhile to explain why this issue only showed up recently, even though the other bug is way older than 1.16.
Chunks below the FEATURES stage, i.e., chunks for which the FEATURES generation Future does return UNLOADED_CHUNK, are considered opaque by the lighting engine. Due toMC-224894, chunks are not kept in this required stage during initial lighting and hence might become opaque to the lighting engine.
However, due to property 1, in 1.16 the chunk always stayed in FEATURES stage once it was scheduled for initial lighting (and even was kept from unloading due to the pending light task). What could have happened even in 1.16 is that neighbor chunks might be unloaded between starting the initial lighting and finishing, which would cause glitches at the border. However, these neighbor chunks were kept loaded by the light ticket that is only removed (wrongly) at the start of the initial lighting, so the time frame was usually way too small, given that the actual unloading usually requires some extra ticks.
On the other hand, in 1.17 the chunk will immediately lose its FEATURES status once the light ticket is removed (and the player is sufficiently far away), hence triggering the issue with a relatively small amount of work required by the server thread which thus gives a large enough time frame for the issue to happen.
Furthermore, note that in 1.16 the light generation stage can be aborted before execution if any pending dependency was aborted due to ticket levels dropping too low. In 1.17 this is no longer the case since generation Future s don't get aborted directly. Hence, 1.17 more often processes initial lighting even if the player is already far away, compared to 1.16, increasing the chance for glitches at he chunk borders.
MC-224894already describes how to trigger this lighting issue alone. However,MC-214793mentions corrupted features in conjuntion with this, I am not sure what is happening here, but I think this is a combination of the pure lighting issue and the previous section (progress being reverted). For example, one might argue that chunks which do make it to the LIGHT stage are kept from unloading a bit longer than their neighbors (which already finish after the FEATURES stage), due to the additional FUTURE that is waited upon by the unloading code. Hence, these ChunkHolder s might still be alive while their neighbors are already saved and unloaded and are hence susceptible to the previous issue which then regenerates these chunks completely, erasing all features that were leaking in from neighbors. However, as the light data is stored externally in the lighting engine, it will stay broken and not be regenerated. The neighbors were already saved to disk and will hence not need to regenerate, so they keep their features, causing corruption at the boundary.
Anyway, this is just a wild guess and probably not the whole story. But I think it's not too important to know precisely what's going on here. Debugging this is a real nightmare due to the rather random nature of everything involved and light tickets interacting with the whole argument.Conclusion
The actual bug should be reasonably easy to understand and fix, e.g., by reverting to the 1.16 behavior. On the other hand, there might be good reasons for this change, so some more sophisticated solution might be necessary.
In any case, I hope these explanations have made clear the connections to the other 3 bugs and improved understanding of the general concurrency issues.
Best,
PhiPro
It turns out that
MC-214,808MC-216148andMC-214793are all caused by the same code change introduced in the 1.17 snapshots. In the following I will describe this change and explain how it leads to each of these bugs. I post this as a separate report as it affects multiple other reports, so it doesn't really fit anywhere as a comment.
Worldgeneration/loading stores oneFuturefor each generation stage of a chunk in a correspondingChunkHolder. These Futuresare created once something requests the chunk in the respective stage, under the condition that the chunk ticket level is sufficiently large to issue generation into that stage. What changedin the1.17 snapshots is the handling of demotion of chunk ticket levels.
In 1.16 demoting the chunk ticket level completed all pending Future s above the new level with a special UNLOADED_CHUNK marker, while leaving already finished Future s untouched.net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { completableFuture.complete(either); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }This has two important consequences:
- Once a generation stage finishes, the Future stays completed with the correct result until the chunk is unloaded
- A Future that is still pending at some point can only be aborted (completed with UNLOADED_CHUNK, which does not abort the actual generation step) if the chunk ticket level drops too low, otherwise it will complete with a valid chunk. This requires some small argument since the Future cannot only be aborted directly by the above code, but also indirectly if any of its dependencies is aborted (completed with UNLOADED_CHUNK). However, the latter cannot happen if the ticket level stays high enough (after observing the pending Future), by construction of the ticket system.
In the 1.17 snapshots, this handling of demotion fundamentally changed. Instead of directly aborting the pending Future s above the new level, all Future s, including already finished ones, are replaced with a new one that completes with an UNLOADED_CHUNK once the original Future completes.
net.minecraft.server.world.ChunkHolder.javaprotectedvoid tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(newChunkHolder.Unloaded() {...}); for(inti = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i);if(completableFuture !=null) { this.futuresByStatus.set(i, completableFuture.thenApply(__ -> either)); }else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }As a result, both of the above properties are no longer valid:
- An already finished stage might be replaced with an UNLOADED_CHUNK_FUTURE
- If the chunk ticket level drops below, say, t then the corresponding Future is not directly aborted, but instead replaced with a Future that will be lazily aborted. If the ticket level is now raised above t again while the original Future is still running, then no new Future will be issued for this generation stage.
net.minecraft.server.world.ChunkHolder.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunkAt(ChunkStatus targetStatus, ThreadedAnvilChunkStorage chunkStorage) { int i = targetStatus.getIndex(); CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { Either<Chunk, ChunkHolder.Unloaded> either = (Either)completableFuture.getNow((Object)null); if (either == null || either.left().isPresent()) { return completableFuture; } } if (getTargetStatusForLevel(this.level).isAtLeast(targetStatus)) { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture2 = chunkStorage.getChunk(this, targetStatus); this.combineSavingFuture(completableFuture2, "schedule " + targetStatus); this.futuresByStatus.set(i, completableFuture2); return completableFuture2; } else { return completableFuture == null ? UNLOADED_CHUNK_FUTURE : completableFuture; } }
This was fine in 1.16 since the Futurecould only be pending if it was not already (indirectly) aborted (which is exactly point 2 above). However,in 1.17theFutureis already indirectly aborted (it is replaced by a lazily aborted one) so it will always produce anUNLOADED_CHUNKeven if the ticket level does not fall again. This breaks property 2.It turns out that this code was changed in 21w06a, which is precisely the version where all 3 bugs were reported. (I actually checked the precise version retrospectively after figuring out the cause for these issues).
So, how does violation of these 2 properties cause the 3 linked bugs?
1. Ticking-Futures might never complete
Every time the ticket level is raised high enough, a chunk is promoted to BORDER, TICKING and ENTITY_TICKING state respectively. This will create additional Future s that depend on the generation Future s in some small neighborhood of the chunk. Upon completion, these will execute the respective tasks for promoting the chunk to the respective state, like registering tick schedulers, marking the chunk tickable/entity-tickable and sending the chunk data to players.
Assuming property 2 above, these Futures will always complete with a valid chunk at some point, unlesstheticket level drops too low. In the latter case, theFuture s get recreated once the chunk is promoted again, so this is not an issue.
Now, in the 1.17 snapshots property 2 is violated and hence these special Future s might never complete (with a valid chunk) since they might depend on lazily aborted Future s upon creation. Hence the respective promotion tasks might never execute. In particular, the ticking-Future, which is responsible for sending the chunk data to players, might not execute, hence causingMC-214.808
Note that all 3 Future scan fail independently. For example, I did observe chunks where theticking-Future completed but the entity-ticking-Futuredid not, causing the chunk to load but entities to get stuck in these chunks. Also note that this issue can similarly lead to the"Chunk not there when requested: "error when the ServerChunkManager queries a lazily aborted Future for the FULL stage. I actually observed this crash a few time during debugging.In order to provoke this issue, one can try the following steps. Due to the random nature of these bugs, several attempts might be needed. Run
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
- /tp ~1000 ~ ~
in quick succession, e.g., with one tick inbetween, or while pausing the server thread through the debugger. The first teleport will create the ChunkHolder s at the target location and create the generation Future s up to FULL stage. The second tp will then lazily abort these Future s again and the third and final tp will create the ticking-Future s on the lazily aborted but still pending generation Future s, hence causingMC-214808.
I also observed this issue with only a single tp, although this was less reproducable. This might seem strange at first, since the ticket levels should change monotonically in this case and not experience the jitter required for the above explanation. I think this is caused by an interaction of the POST_TELEPORT chunk ticket and the player ticket throttling. The POST_TELEPORT ticket is created right after teleporting, but expires before the player tickets are added because of the throttling, hence causing the required jitter.2. Futures might be erased completely
If the chunk ticket level drops low enough, the correspondingChunkHolder is scheduled for unloading. It can still be revoked up to the point where it is actually saved to NBT and all the unloading tasks are done. This can be a few ticks from the actual scheduling, depending on server load. By property 1, once the corresponding chunk has passed the first generation/loading stage, all Future s will keep the reference to this chunk, so that it can indeed be revoked from the ChunkHolder.However, in 1.17 when theChunkHolder is scheduled for unloading, all generation Future s are replaced with (lazily) aborted ones. If this ChunkHolder gets then later on revoked (before it could properly save and unload) all generation Future s are recreated and the very first stage then reloads the chunk from disk again. Hence, the chunk object gets replaced with a completely new version that is reloaded from disk, erasing any progress since the last save. This is exactlyMC-216148.
Note that the unloading tasks did not run in this scenario, so any externally stored data is still present. In particular, the chunk is still marked as loaded and will hence skip the block-entity loading step.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprivate CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> convertToFullChunk(ChunkHolder chunkHolder) { ... worldChunk2.loadToWorld(); if (this.loadedChunks.add(chunkPos.toLong())) { worldChunk2.setLoadedToWorld(true); worldChunk2.updateAllBlockEntities(); } ... }As a consequence, block-entity tickers will not be recreated and still reference the old copies, hence block-entities will not be ticked. This is indeed a side effect noted in the bug report.
In order to provoke this issue, run for example
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
in quick succession, so that the ChunkHolder s do not save and unload inbetween. This can be achieved by pausing the server thread, for example. The first tp will erase all generation Future s and the second tp will then regenerate them and reload the chunk from disk.
In the report, it is noted that this seems to happen near nether portals. I'm not really sure how this is correlated. Most certainly, the relevant feature of nether portals are the PORTAL tickets created upon using them. However, I don't know how they enter the equation. They might play a similar role to the POST_TELEPORT_TICKET of the previous section or it might be something else.
3. Chunks might drop below FEATURES stage during initial lighting
While
MC-214793is actually caused by another concurrency bug, namelyMC-224894, it might still be worthwhile to explain why this issue only showed up recently, even though the other bug is way older than 1.16.
Chunks below the FEATURES stage, i.e., chunks for which the FEATURES generation Future does return UNLOADED_CHUNK, are considered opaque by the lighting engine. Due toMC-224894, chunks are not kept in this required stage during initial lighting and hence might become opaque to the lighting engine.
However, due to property 1, in 1.16 the chunk always stayed in FEATURES stage once it was scheduled for initial lighting (and even was kept from unloading due to the pending light task). What could have happened even in 1.16 is that neighbor chunks might be unloaded between starting the initial lighting and finishing, which would cause glitches at the border. However, these neighbor chunks were kept loaded by the light ticket that is only removed (wrongly) at the start of the initial lighting, so the time frame was usually way too small, given that the actual unloading usually requires some extra ticks.
On the other hand, in 1.17 the chunk will immediately lose its FEATURES status once the light ticket is removed (and the player is sufficiently far away), hence triggering the issue with a relatively small amount of work required by the server thread which thus gives a large enough time frame for the issue to happen.
Furthermore, note that in 1.16 the light generation stage can be aborted before execution if any pending dependency was aborted due to ticket levels dropping too low. In 1.17 this is no longer the case since generation Future s don't get aborted directly. Hence, 1.17 more often processes initial lighting even if the player is already far away, compared to 1.16, increasing the chance for glitches at he chunk borders.
MC-224894already describes how to trigger this lighting issue alone. However,MC-214793mentions corrupted features in conjuntion with this, I am not sure what is happening here, but I think this is a combination of the pure lighting issue and the previous section (progress being reverted). For example, one might argue that chunks which do make it to the LIGHT stage are kept from unloading a bit longer than their neighbors (which already finish after the FEATURES stage), due to the additional FUTURE that is waited upon by the unloading code. Hence, these ChunkHolder s might still be alive while their neighbors are already saved and unloaded and are hence susceptible to the previous issue which then regenerates these chunks completely, erasing all features that were leaking in from neighbors. However, as the light data is stored externally in the lighting engine, it will stay broken and not be regenerated. The neighbors were already saved to disk and will hence not need to regenerate, so they keep their features, causing corruption at the boundary.
Anyway, this is just a wild guess and probably not the whole story. But I think it's not too important to know precisely what's going on here. Debugging this is a real nightmare due to the rather random nature of everything involved and light tickets interacting with the whole argument.Conclusion
The actual bug should be reasonably easy to understand and fix, e.g., by reverting to the 1.16 behavior. On the other hand, there might be good reasons for this change, so some more sophisticated solution might be necessary.
In any case, I hope these explanations have made clear the connections to the other 3 bugs and improved understanding of the general concurrency issues.
Best,
PhiProThe issue described in the following is the root cause of
MC-214793. While it has only become visible in practice recently due to MC-224893, I think this is still an important concurrency issue on its own and hence deserves its own report.During worldgen, special light chunk tickets are created just before the light generation stage, in order to keep the chunk and its neighbors loaded until initial lighting is finished. Or at least this seems to be the idea behind this mechanism. However, the ticket is removed right before starting the actual initial lighting, rather than waiting until it is finished.
net.minecraft.server.world.ServerLightingProvider.javapublic CompletableFuture<Chunk> light(Chunk chunk, boolean excludeBlocks) { ChunkPos chunkPos = chunk.getPos(); chunk.setLightOn(false); this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.PRE_UPDATE, Util.debugRunnable(() -> { ... super.setColumnEnabled(chunkPos, true); ... this.chunkStorage.releaseLightTicket(chunkPos); }, () -> "lightChunk " + chunkPos + " " + excludeBlocks )); return CompletableFuture.supplyAsync(() -> { chunk.setLightOn(true); super.setRetainData(chunkPos, false); return chunk; }, (runnable) -> this.enqueue(chunkPos.x, chunkPos.z, ServerLightingProvider.Stage.POST_UPDATE, runnable) ); }This is of course too early and leaves the possibility for the neighbor chunks to unload before the initial lighting has finished, causing lighting glitches at the chunk borders (the chunk itself is actually kept from unloading due to the pending light task). These border glitches can happen even in 1.16.
Note that chunks below the FEATURES stage are considered opaque by the lighting engine. Hence, even though the chunk itself is kept from completely unloading, its ticket level might drop below FEATURES stage which would make the chunk opaque to the lighting engine (during initial lighting), hence causing dark chunks as observed inMC-214793. In 1.16, this issue does actually not occur since chunks won't be demoted once they finish a generation stage (more precisely, the respective Future for that generation stage stays valid, which is precisely what the lighting engine checks). However, this is no longer true in 1.17 due to MC-224893 and hence the issue actually becomes visible.In order to provoke this issue, one can traverse the world at high speed, e.g., using spectator mode (with very high speed) or tp commands in quick succession. Only a sparse set of chunks will make it to the LIGHT stage during this traversion, other chunks will already abort generation after the FEATURES stage. Furthermore, at the time these LIGHT stages are processed, the player is already far away, leaving the ticket level below the FEATURES level. Both of these facts together, sparseness of light tickets and chunk ticket levels being too low when the initial lighting is processed, make the above explanation applicable. And indeed one can observe completely dark chunks when visiting the traversed region again. Note however, that this might require some attempts due to the random nature of this bug. (In my test cases these were however quite reliably reproducable).
Note that while
MC-214793mentions corrupted features that often occur in conjunction with dark chunks when flying in spirals, no corrupted features were observed with the above steps. In conlusion, I think that the above steps really isolate the lighting bug itself, whereas the situation described inMC-214793is a convolution of the pure lighting bug and other phenomena of MC-224893, see the latter for a small discussion.Finally, let me mention that I did move the releaseLightTicket(...) call to the POST_UPDATE phase and no further dark chunks were observed, as expected. However, I did observe some small lighting glitches at chunk boundaries. This was in fact expected, since the light ticket only loads neighbors into LIQUID_CARVERS stage, whereas they should really be in FEATURES stage for the lighting engine to interact with them. So this is actually yet another issue with the light ticket mechanism (in 1.17).
Remarks
Let me also remark that this issue is actually an instance of a more general concurrency issue present in the chunk loading code. As mentioned in the beginning, it could already happen in 1.16 that neighbor chunks unload before the initial lighting is completed, causing light glitches at the boundaries, due to the light ticket being removed too early. This effect is not new to 1.17.
More generally, the chunk loading/worldgen code does not take any precautions to guarantee that a chunk stays loaded once it is passed to any worldgen step. Once a Future completes, the chunk is passed to the next generation step (on the worldgen thread) which can then hold on to the reference indefinitely. Independently, on the server thread, the chunk then may be unloaded before this next generation step does its work, so any progress after this point would be lost.
Similarly, the light ticket only handles the case of initial lighting. Any further updates after the initial lighting do not take any measures to keep the chunk and its neighbors in a state where the lighting engine can interact with them. So, any light updates after the LIGHT generation stage (but before promotion to FULL stage) might get stuck randomly due to chunks being unloaded or being demoted from the correct state.As a solution, one should consider to implement a mechanism similar to the light tickets for general interactions during worldgen.
Best,
PhiPro
It turns out that
MC-214808,MC-216148andMC-214793are all caused by the same code change introduced in the 1.17 snapshots. In the following I will describe this change and explain how it leads to each of these bugs. I post this as a separate report as it affects multiple other reports, so it doesn't really fit anywhere as a comment.World generation/loading stores one Future for each generation stage of a chunk in a corresponding ChunkHolder. These Futures are created once something requests the chunk in the respective stage, under the condition that the chunk ticket level is sufficiently large to issue generation into that stage. What changed in the 1.17 snapshots is the handling of demotion of chunk ticket levels.
In 1.16 demoting the chunk ticket level completed all pending Future s above the new level with a special UNLOADED_CHUNK marker, while leaving already finished Future s untouched.net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { completableFuture.complete(either); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }This has two important consequences:
- Once a generation stage finishes, the Future stays completed with the correct result until the chunk is unloaded
- A Future that is still pending at some point can only be aborted (completed with UNLOADED_CHUNK, which does not abort the actual generation step) if the chunk ticket level drops too low, otherwise it will complete with a valid chunk. This requires some small argument since the Future cannot only be aborted directly by the above code, but also indirectly if any of its dependencies is aborted (completed with UNLOADED_CHUNK). However, the latter cannot happen if the ticket level stays high enough (after observing the pending Future), by construction of the ticket system.
In the 1.17 snapshots, this handling of demotion fundamentally changed. Instead of directly aborting the pending Future s above the new level, all Future s, including already finished ones, are replaced with a new one that completes with an UNLOADED_CHUNK once the original Future completes.
net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { this.futuresByStatus.set(i, completableFuture.thenApply(__ -> either)); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }As a result, both of the above properties are no longer valid:
- An already finished stage might be replaced with an UNLOADED_CHUNK_FUTURE
- If the chunk ticket level drops below, say, t then the corresponding Future is not directly aborted, but instead replaced with a Future that will be lazily aborted. If the ticket level is now raised above t again while the original Future is still running, then no new Future will be issued for this generation stage.
net.minecraft.server.world.ChunkHolder.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunkAt(ChunkStatus targetStatus, ThreadedAnvilChunkStorage chunkStorage) { int i = targetStatus.getIndex(); CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { Either<Chunk, ChunkHolder.Unloaded> either = (Either)completableFuture.getNow((Object)null); if (either == null || either.left().isPresent()) { return completableFuture; } } if (getTargetStatusForLevel(this.level).isAtLeast(targetStatus)) { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture2 = chunkStorage.getChunk(this, targetStatus); this.combineSavingFuture(completableFuture2, "schedule " + targetStatus); this.futuresByStatus.set(i, completableFuture2); return completableFuture2; } else { return completableFuture == null ? UNLOADED_CHUNK_FUTURE : completableFuture; } }This was fine in 1.16 since the Future could only be pending if it was not already (indirectly) aborted (which is exactly point 2 above). However, in 1.17 the Future is already indirectly aborted (it is replaced by a lazily aborted one) so it will always produce an UNLOADED_CHUNK even if the ticket level does not fall again. This breaks property 2.
It turns out that this code was changed in 21w06a, which is precisely the version where all 3 bugs were reported. (I actually checked the precise version retrospectively after figuring out the cause for these issues).
So, how does violation of these 2 properties cause the 3 linked bugs?
1. Ticking-Futures might never complete
Every time the ticket level is raised high enough, a chunk is promoted to BORDER, TICKING and ENTITY_TICKING state respectively. This will create additional Future s that depend on the generation Future s in some small neighborhood of the chunk. Upon completion, these will execute the respective tasks for promoting the chunk to the respective state, like registering tick schedulers, marking the chunk tickable/entity-tickable and sending the chunk data to players.
Assuming property 2 above, these Future s will always complete with a valid chunk at some point, unless the ticket level drops too low. In the latter case, the Future s get recreated once the chunk is promoted again, so this is not an issue.
Now, in the 1.17 snapshots property 2 is violated and hence these special Future s might never complete (with a valid chunk) since they might depend on lazily aborted Future s upon creation. Hence the respective promotion tasks might never execute. In particular, the ticking-Future, which is responsible for sending the chunk data to players, might not execute, hence causingMC-214808.
Note that all 3 Future s can fail independently. For example, I did observe chunks where the ticking-Future completed but the entity-ticking-Future did not, causing the chunk to load but entities to get stuck in these chunks. Also note that this issue can similarly lead to the "Chunk not there when requested: " error when the ServerChunkManager queries a lazily aborted Future for the FULL stage. I actually observed this crash a few time during debugging.In order to provoke this issue, one can try the following steps. Due to the random nature of these bugs, several attempts might be needed. Run
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
- /tp ~1000 ~ ~
in quick succession, e.g., with one tick inbetween, or while pausing the server thread through the debugger. The first teleport will create the ChunkHolder s at the target location and create the generation Future s up to FULL stage. The second tp will then lazily abort these Future s again and the third and final tp will create the ticking-Future s on the lazily aborted but still pending generation Future s, hence causingMC-214808.
I also observed this issue with only a single tp, although this was less reproducable. This might seem strange at first, since the ticket levels should change monotonically in this case and not experience the jitter required for the above explanation. I think this is caused by an interaction of the POST_TELEPORT chunk ticket and the player ticket throttling. The POST_TELEPORT ticket is created right after teleporting, but expires before the player tickets are added because of the throttling, hence causing the required jitter.2. Futures might be erased completely
If the chunk ticket level drops low enough, the corresponding ChunkHolder is scheduled for unloading. It can still be revoked up to the point where it is actually saved to NBT and all the unloading tasks are done. This can be a few ticks from the actual scheduling, depending on server load. By property 1, once the corresponding chunk has passed the first generation/loading stage, all Future s will keep the reference to this chunk, so that it can indeed be revoked from the ChunkHolder.
However, in 1.17 when the ChunkHolder is scheduled for unloading, all generation Future s are replaced with (lazily) aborted ones. If this ChunkHolder gets then later on revoked (before it could properly save and unload) all generation Future s are recreated and the very first stage then reloads the chunk from disk again. Hence, the chunk object gets replaced with a completely new version that is reloaded from disk, erasing any progress since the last save. This is exactlyMC-216148.
Note that the unloading tasks did not run in this scenario, so any externally stored data is still present. In particular, the chunk is still marked as loaded and will hence skip the block-entity loading step.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprivate CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> convertToFullChunk(ChunkHolder chunkHolder) { ... worldChunk2.loadToWorld(); if (this.loadedChunks.add(chunkPos.toLong())) { worldChunk2.setLoadedToWorld(true); worldChunk2.updateAllBlockEntities(); } ... }As a consequence, block-entity tickers will not be recreated and still reference the old copies, hence block-entities will not be ticked. This is indeed a side effect noted in the bug report.
In order to provoke this issue, run for example
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
in quick succession, so that the ChunkHolder s do not save and unload inbetween. This can be achieved by pausing the server thread, for example. The first tp will erase all generation Future s and the second tp will then regenerate them and reload the chunk from disk.
In the report, it is noted that this seems to happen near nether portals. I'm not really sure how this is correlated. Most certainly, the relevant feature of nether portals are the PORTAL tickets created upon using them. However, I don't know how they enter the equation. They might play a similar role to the POST_TELEPORT_TICKET of the previous section or it might be something else.
3. Chunks might drop below FEATURES stage during initial lighting
While
MC-214793is actually caused by another concurrency bug, namely MC-?, it might still be worthwhile to explain why this issue only showed up recently, even though the other bug is way older than 1.16.
Chunks below the FEATURES stage, i.e., chunks for which the FEATURES generation Future does return UNLOADED_CHUNK, are considered opaque by the lighting engine. Due to MC-?, chunks are not kept in this required stage during initial lighting and hence might become opaque to the lighting engine.
However, due to property 1, in 1.16 the chunk always stayed in FEATURES stage once it was scheduled for initial lighting (and even was kept from unloading due to the pending light task). What could have happened even in 1.16 is that neighbor chunks might be unloaded between starting the initial lighting and finishing, which would cause glitches at the border. However, these neighbor chunks were kept loaded by the light ticket that is only removed (wrongly) at the start of the initial lighting, so the time frame was usually way too small, given that the actual unloading usually requires some extra ticks.
On the other hand, in 1.17 the chunk will immediately lose its FEATURES status once the light ticket is removed (and the player is sufficiently far away), hence triggering the issue with a relatively small amount of work required by the server thread which thus gives a large enough time frame for the issue to happen.
Furthermore, note that in 1.16 the light generation stage can be aborted before execution if any pending dependency was aborted due to ticket levels dropping too low. In 1.17 this is no longer the case since generation Future s don't get aborted directly. Hence, 1.17 more often processes initial lighting even if the player is already far away, compared to 1.16, increasing the chance for glitches at he chunk borders.MC-?already describes how to trigger this lighting issue alone. However,MC-214793mentions corrupted features in conjuntion with this, I am not sure what is happening here, but I think this is a combination of the pure lighting issue and the previous section (progress being reverted). For example, one might argue that chunks which do make it to the LIGHT stage are kept from unloading a bit longer than their neighbors (which already finish after the FEATURES stage), due to the additional FUTURE that is waited upon by the unloading code. Hence, these ChunkHolder s might still be alive while their neighbors are already saved and unloaded and are hence susceptible to the previous issue which then regenerates these chunks completely, erasing all features that were leaking in from neighbors. However, as the light data is stored externally in the lighting engine, it will stay broken and not be regenerated. The neighbors were already saved to disk and will hence not need to regenerate, so they keep their features, causing corruption at the boundary.
Anyway, this is just a wild guess and probably not the whole story. But I think it's not too important to know precisely what's going on here. Debugging this is a real nightmare due to the rather random nature of everything involved and light tickets interacting with the whole argument.Conclusion
The actual bug should be reasonably easy to understand and fix, e.g., by reverting to the 1.16 behavior. On the other hand, there might be good reasons for this change, so some more sophisticated solution might be necessary.
In any case, I hope these explanations have made clear the connections to the other 3 bugs and improved understanding of the general concurrency issues.
Best,
PhiProIt turns out that
MC-214808,MC-216148andMC-214793are all caused by the same code change introduced in the 1.17 snapshots. In the following I will describe this change and explain how it leads to each of these bugs. I post this as a separate report as it affects multiple other reports, so it doesn't really fit anywhere as a comment.World generation/loading stores one Future for each generation stage of a chunk in a corresponding ChunkHolder. These Futures are created once something requests the chunk in the respective stage, under the condition that the chunk ticket level is sufficiently large to issue generation into that stage. What changed in the 1.17 snapshots is the handling of demotion of chunk ticket levels.
In 1.16 demoting the chunk ticket level completed all pending Future s above the new level with a special UNLOADED_CHUNK marker, while leaving already finished Future s untouched.net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { completableFuture.complete(either); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }This has two important consequences:
- Once a generation stage finishes, the Future stays completed with the correct result until the chunk is unloaded
- A Future that is still pending at some point can only be aborted (completed with UNLOADED_CHUNK, which does not abort the actual generation step) if the chunk ticket level drops too low, otherwise it will complete with a valid chunk. This requires some small argument since the Future cannot only be aborted directly by the above code, but also indirectly if any of its dependencies is aborted (completed with UNLOADED_CHUNK). However, the latter cannot happen if the ticket level stays high enough (after observing the pending Future), by construction of the ticket system.
In the 1.17 snapshots, this handling of demotion fundamentally changed. Instead of directly aborting the pending Future s above the new level, all Future s, including already finished ones, are replaced with a new one that completes with an UNLOADED_CHUNK once the original Future completes.
net.minecraft.server.world.ChunkHolder.javaprotected void tick(ThreadedAnvilChunkStorage chunkStorage) { ... Either<Chunk, ChunkHolder.Unloaded> either = Either.right(new ChunkHolder.Unloaded() {...}); for(int i = bl2 ? chunkStatus2.getIndex() + 1 : 0; i <= chunkStatus.getIndex(); ++i) { completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { this.futuresByStatus.set(i, completableFuture.thenApply(__ -> either)); } else { this.futuresByStatus.set(i, CompletableFuture.completedFuture(either)); } } ... }As a result, both of the above properties are no longer valid:
- An already finished stage might be replaced with an UNLOADED_CHUNK_FUTURE
- If the chunk ticket level drops below, say, t then the corresponding Future is not directly aborted, but instead replaced with a Future that will be lazily aborted. If the ticket level is now raised above t again while the original Future is still running, then no new Future will be issued for this generation stage.
net.minecraft.server.world.ChunkHolder.javapublic CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> getChunkAt(ChunkStatus targetStatus, ThreadedAnvilChunkStorage chunkStorage) { int i = targetStatus.getIndex(); CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture = (CompletableFuture)this.futuresByStatus.get(i); if (completableFuture != null) { Either<Chunk, ChunkHolder.Unloaded> either = (Either)completableFuture.getNow((Object)null); if (either == null || either.left().isPresent()) { return completableFuture; } } if (getTargetStatusForLevel(this.level).isAtLeast(targetStatus)) { CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> completableFuture2 = chunkStorage.getChunk(this, targetStatus); this.combineSavingFuture(completableFuture2, "schedule " + targetStatus); this.futuresByStatus.set(i, completableFuture2); return completableFuture2; } else { return completableFuture == null ? UNLOADED_CHUNK_FUTURE : completableFuture; } }This was fine in 1.16 since the Future could only be pending if it was not already (indirectly) aborted (which is exactly point 2 above). However, in 1.17 the Future is already indirectly aborted (it is replaced by a lazily aborted one) so it will always produce an UNLOADED_CHUNK even if the ticket level does not fall again. This breaks property 2.
It turns out that this code was changed in 21w06a, which is precisely the version where all 3 bugs were reported. (I actually checked the precise version retrospectively after figuring out the cause for these issues).
So, how does violation of these 2 properties cause the 3 linked bugs?
1. Ticking-Futures might never complete
Every time the ticket level is raised high enough, a chunk is promoted to BORDER, TICKING and ENTITY_TICKING state respectively. This will create additional Future s that depend on the generation Future s in some small neighborhood of the chunk. Upon completion, these will execute the respective tasks for promoting the chunk to the respective state, like registering tick schedulers, marking the chunk tickable/entity-tickable and sending the chunk data to players.
Assuming property 2 above, these Future s will always complete with a valid chunk at some point, unless the ticket level drops too low. In the latter case, the Future s get recreated once the chunk is promoted again, so this is not an issue.
Now, in the 1.17 snapshots property 2 is violated and hence these special Future s might never complete (with a valid chunk) since they might depend on lazily aborted Future s upon creation. Hence the respective promotion tasks might never execute. In particular, the ticking-Future, which is responsible for sending the chunk data to players, might not execute, hence causingMC-214808.
Note that all 3 Future s can fail independently. For example, I did observe chunks where the ticking-Future completed but the entity-ticking-Future did not, causing the chunk to load but entities to get stuck in these chunks. Also note that this issue can similarly lead to the "Chunk not there when requested: " error when the ServerChunkManager queries a lazily aborted Future for the FULL stage. I actually observed this crash a few time during debugging.In order to provoke this issue, one can try the following steps. Due to the random nature of these bugs, several attempts might be needed. Run
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
- /tp ~1000 ~ ~
in quick succession, e.g., with one tick inbetween, or while pausing the server thread through the debugger. The first teleport will create the ChunkHolder s at the target location and create the generation Future s up to FULL stage. The second tp will then lazily abort these Future s again and the third and final tp will create the ticking-Future s on the lazily aborted but still pending generation Future s, hence causingMC-214808.
I also observed this issue with only a single tp, although this was less reproducable. This might seem strange at first, since the ticket levels should change monotonically in this case and not experience the jitter required for the above explanation. I think this is caused by an interaction of the POST_TELEPORT chunk ticket and the player ticket throttling. The POST_TELEPORT ticket is created right after teleporting, but expires before the player tickets are added because of the throttling, hence causing the required jitter.2. Futures might be erased completely
If the chunk ticket level drops low enough, the corresponding ChunkHolder is scheduled for unloading. It can still be revoked up to the point where it is actually saved to NBT and all the unloading tasks are done. This can be a few ticks from the actual scheduling, depending on server load. By property 1, once the corresponding chunk has passed the first generation/loading stage, all Future s will keep the reference to this chunk, so that it can indeed be revoked from the ChunkHolder.
However, in 1.17 when the ChunkHolder is scheduled for unloading, all generation Future s are replaced with (lazily) aborted ones. If this ChunkHolder gets then later on revoked (before it could properly save and unload) all generation Future s are recreated and the very first stage then reloads the chunk from disk again. Hence, the chunk object gets replaced with a completely new version that is reloaded from disk, erasing any progress since the last save. This is exactlyMC-216148.
Note that the unloading tasks did not run in this scenario, so any externally stored data is still present. In particular, the chunk is still marked as loaded and will hence skip the block-entity loading step.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprivate CompletableFuture<Either<Chunk, ChunkHolder.Unloaded>> convertToFullChunk(ChunkHolder chunkHolder) { ... worldChunk2.loadToWorld(); if (this.loadedChunks.add(chunkPos.toLong())) { worldChunk2.setLoadedToWorld(true); worldChunk2.updateAllBlockEntities(); } ... }As a consequence, block-entity tickers will not be recreated and still reference the old copies, hence block-entities will not be ticked. This is indeed a side effect noted in the bug report.
In order to provoke this issue, run for example
- /tp ~1000 ~ ~
- /tp ~-1000 ~ ~
in quick succession, so that the ChunkHolder s do not save and unload inbetween. This can be achieved by pausing the server thread, for example. The first tp will erase all generation Future s and the second tp will then regenerate them and reload the chunk from disk.
In the report, it is noted that this seems to happen near nether portals. I'm not really sure how this is correlated. Most certainly, the relevant feature of nether portals are the PORTAL tickets created upon using them. However, I don't know how they enter the equation. They might play a similar role to the POST_TELEPORT_TICKET of the previous section or it might be something else.
3. Chunks might drop below FEATURES stage during initial lighting
While
MC-214793is actually caused by another concurrency bug, namelyMC-224894, it might still be worthwhile to explain why this issue only showed up recently, even though the other bug is way older than 1.16.
Chunks below the FEATURES stage, i.e., chunks for which the FEATURES generation Future does return UNLOADED_CHUNK, are considered opaque by the lighting engine. Due toMC-224894, chunks are not kept in this required stage during initial lighting and hence might become opaque to the lighting engine.
However, due to property 1, in 1.16 the chunk always stayed in FEATURES stage once it was scheduled for initial lighting (and even was kept from unloading due to the pending light task). What could have happened even in 1.16 is that neighbor chunks might be unloaded between starting the initial lighting and finishing, which would cause glitches at the border. However, these neighbor chunks were kept loaded by the light ticket that is only removed (wrongly) at the start of the initial lighting, so the time frame was usually way too small, given that the actual unloading usually requires some extra ticks.
On the other hand, in 1.17 the chunk will immediately lose its FEATURES status once the light ticket is removed (and the player is sufficiently far away), hence triggering the issue with a relatively small amount of work required by the server thread which thus gives a large enough time frame for the issue to happen.
Furthermore, note that in 1.16 the light generation stage can be aborted before execution if any pending dependency was aborted due to ticket levels dropping too low. In 1.17 this is no longer the case since generation Future s don't get aborted directly. Hence, 1.17 more often processes initial lighting even if the player is already far away, compared to 1.16, increasing the chance for glitches at he chunk borders.
MC-224894already describes how to trigger this lighting issue alone. However,MC-214793mentions corrupted features in conjuntion with this, I am not sure what is happening here, but I think this is a combination of the pure lighting issue and the previous section (progress being reverted). For example, one might argue that chunks which do make it to the LIGHT stage are kept from unloading a bit longer than their neighbors (which already finish after the FEATURES stage), due to the additional FUTURE that is waited upon by the unloading code. Hence, these ChunkHolder s might still be alive while their neighbors are already saved and unloaded and are hence susceptible to the previous issue which then regenerates these chunks completely, erasing all features that were leaking in from neighbors. However, as the light data is stored externally in the lighting engine, it will stay broken and not be regenerated. The neighbors were already saved to disk and will hence not need to regenerate, so they keep their features, causing corruption at the boundary.
Anyway, this is just a wild guess and probably not the whole story. But I think it's not too important to know precisely what's going on here. Debugging this is a real nightmare due to the rather random nature of everything involved and light tickets interacting with the whole argument.Conclusion
The actual bug should be reasonably easy to understand and fix, e.g., by reverting to the 1.16 behavior. On the other hand, there might be good reasons for this change, so some more sophisticated solution might be necessary.
In any case, I hope these explanations have made clear the connections to the other 3 bugs and improved understanding of the general concurrency issues.
Best,
PhiPro
There are two codepaths that invoke saving of chunks.
1. ThreadedAnvilChunkStorage.tryUnloadChunk(...) which saves chunks that are no longer kept alive by any chunk ticket, before finally unloading them from the world.
2. ThreadedAnvilChunkStorage.save(flush) which saves chunks periodically and upon closing the world.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprotected void save(boolean flush) { if (flush) { ... } else { this.chunkHolders.values().stream().filter(ChunkHolder::isAccessible).forEach((chunkHolder) -> { Chunk chunk = (Chunk)chunkHolder.getSavingFuture().getNow((Object)null); if (chunk instanceof ReadOnlyChunk || chunk instanceof WorldChunk) { this.save(chunk); chunkHolder.updateAccessibleStatus(); } }); } }This code saves all accessible chunks, i.e., chunks that were in the BORDER status at some point since the last save, periodically (when flush is false) and upon closing the world (when flush is true). The second case is similar (and hence not included in the code snippet) but additionally waits for pending futures in the accessible chunks.
The point is now that neither of these cases covers chunks that are still kept alive by chunk tickets when closing the world, but do not have a sufficient ticket level to be BORDER chunks. Those are chunks that are only partially generated, but can already contain features and light propagations from adjacent chunks. Not saving these chunks will hence lead to corrupted features and broken light at chunk borders.
In the following I will present twoconcrete examples where chunk tickets can keep such chunks alive and will hence prevent saving. Some other ticket types can most certainly lead to the same effect.1. Spawn chunks
The first example uses START tickets created on spawn chunks (the same works with FORCED tickets), which are not removed upon unloading the world and hence will keep some chunks alive.
- Create a new world with render distance 2 (e.g., with seed -4530634556500121041)
- Quit and reload directly after generating the world
- One can now turn up the render distance again
- Observe that features between the 11th and 12th chunk away from the spawn point are broken (World spawn loads chunks in a radius of 11 into BORDER status)
Due to the low render distance, chunks in distance 12 from the spawn point are not requested into BORDER status, but still kept from unloading due to the START ticket, hence triggering the described issue.
2. Overload the lighting engine
The second example uses LIGHT tickets created during worldgen to keep chunks alive. While these tickets will eventually be removed when the worldgen is finished, the goal here is to overload the lighting engine so that these generations steps will not be finished (and hence the tickets will not be removed) in time before the flush save completes. Note that the flush save only waits for accessible chunks to finish any remaining futures, but not for partially generated chunks.
Note that this behavior is quite random and hence the following steps might require several attempts for reproduction or may not work at all, depending on the setup. Nevertheless, I think this example is still of practical relevance as described in the final section.
In order to stress the lighting engine we can try the following steps
- Create an AMPLIFIED world with render distance 32
- Fly a few chunks in one direction and then close the world immediately
- Repeat the previous step several times, always flying in the same direction
- After repeating for a few times, revisit the generated chunks again and see if some corrupted features or light glitches occur.
(Note that there are also some lighting glitches on the chunk border)Closing the world immediately after flying a few chunks does not give the lighting engine enough time to finish its jobs before unloading the world (at least sometimes), hence causing the described issue.
Remarks
I think the issue described here is a good candidate for explaining MC-125007 (respectively
MC-188086).
I am not sure, though, to which extent it is related toMC-214793. It again seems to be a good explanation for the corrupted features that seem to occur in conjunction with the dark chunks. However, it fails to explain why these chunks are completely dark; it only explains missing light propagations at the chunk borders, since these are discarded upon saving. So, it looks like some other unknown issue is interfering here aswell.There are two codepaths that invoke saving of chunks.
1. ThreadedAnvilChunkStorage.tryUnloadChunk(...) which saves chunks that are no longer kept alive by any chunk ticket, before finally unloading them from the world.
2. ThreadedAnvilChunkStorage.save(flush) which saves chunks periodically and upon closing the world.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprotected void save(boolean flush) { if (flush) { ... } else { this.chunkHolders.values().stream().filter(ChunkHolder::isAccessible).forEach((chunkHolder) -> { Chunk chunk = (Chunk)chunkHolder.getSavingFuture().getNow((Object)null); if (chunk instanceof ReadOnlyChunk || chunk instanceof WorldChunk) { this.save(chunk); chunkHolder.updateAccessibleStatus(); } }); } }This code saves all accessible chunks, i.e., chunks that were in the BORDER status at some point since the last save, periodically (when flush is false) and upon closing the world (when flush is true). The second case is similar (and hence not included in the code snippet) but additionally waits for pending futures in the accessible chunks.
The point is now that neither of these cases covers chunks that are still kept alive by chunk tickets when closing the world, but do not have a sufficient ticket level to be BORDER chunks. Those are chunks that are only partially generated, but can already contain features and light propagations from adjacent chunks. Not saving these chunks will hence lead to corrupted features and broken light at chunk borders.
In the following I will present three concrete examples where chunk tickets can keep such chunks alive and will hence prevent saving. Some other ticket types can most certainly lead to the same effect.1. Spawn chunks
The first example uses START tickets created on spawn chunks (the same works with FORCED tickets), which are not removed upon unloading the world and hence will keep some chunks alive.
- Create a new world with render distance 2 (e.g., with seed -4530634556500121041)
- Quit and reload directly after generating the world
- One can now turn up the render distance again
- Observe that features between the 11th and 12th chunk away from the spawn point are broken (World spawn loads chunks in a radius of 11 into BORDER status)
Due to the low render distance, chunks in distance 12 from the spawn point are not requested into BORDER status, but still kept from unloading due to the START ticket, hence triggering the described issue.
2. Overload the lighting engine
The second example uses LIGHT tickets created during worldgen to keep chunks alive. While these tickets will eventually be removed when the worldgen is finished, the goal here is to overload the lighting engine so that these generations steps will not be finished (and hence the tickets will not be removed) in time before the flush save completes. Note that the flush save only waits for accessible chunks to finish any remaining futures, but not for partially generated chunks.
Note that this behavior is quite random and hence the following steps might require several attempts for reproduction or may not work at all, depending on the setup. Nevertheless, I think this example is still of practical relevance as described in the final section.
In order to stress the lighting engine we can try the following steps
- Create an AMPLIFIED world with render distance 32
- Fly a few chunks in one direction and then close the world immediately
- Repeat the previous step several times, always flying in the same direction
- After repeating for a few times, revisit the generated chunks again and see if some corrupted features or light glitches occur.
(Note that there are also some lighting glitches on the chunk border)Closing the world immediately after flying a few chunks does not give the lighting engine enough time to finish its jobs before unloading the world (at least sometimes), hence causing the described issue.
3. Interacting with the world
Every world interaction, or more precisely every call to World.getChunk(...), creates an UNKNOWN chunk ticket with the intent to keep the chunk loaded and hence valid until the end of the tick. Every ticking entity and every player creates such a ticket at its location every tick, e.g., due to movement operations querying the chunk. These UNKNOWN tickets are in fact not cleared at the end of the current tick, but at the start of the next one. Consequently, these UNKNOWN tickets from the last tick are not purged upon shutting down the world and are hence still presnt during saving, prohibiting unloading and saving of some chunks (in contrast, PLAYER tickets are explicitly removed upon shutting down the world by disconnecting all players).
This does not seem to be an issue on high render distances, presumably because there are no entities far away from the player, at least not in the very last tick, and hence no such UNKNOWN tickets are created away from the player, whereas the PLAYER ticket itself has a much higher range and hence dominates these UKNOWN tickets. Hencethe UNKNOWN tickets only affect chunks that were already accessible due to the PLAYER ticket and hence do not trigger the saving issue.
However, this argument does not work for small render distances, e.g., render distance 2, where the UNKNOWN tickets do keep chunks alive that were not yet accessible, hence causing the saving issues. Indeed, the issue does show up deterministically when exploring new chunks with a render distance of 2 and reloading the world every few chunks.Remarks
I think the issue described here is a good candidate for explaining MC-125007 (respectively
MC-188086).
I am not sure, though, to which extent it is related toMC-214793. It again seems to be a good explanation for the corrupted features that seem to occur in conjunction with the dark chunks. However, it fails to explain why these chunks are completely dark; it only explains missing light propagations at the chunk borders, since these are discarded upon saving. So, it looks like some other unknown issue is interfering here aswell.
There are two codepaths that invoke saving of chunks.
1. ThreadedAnvilChunkStorage.tryUnloadChunk(...) which saves chunks that are no longer kept alive by any chunk ticket, before finally unloading them from the world.
2. ThreadedAnvilChunkStorage.save(flush) which saves chunks periodically and upon closing the world.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprotected void save(boolean flush) { if (flush) { ... } else { this.chunkHolders.values().stream().filter(ChunkHolder::isAccessible).forEach((chunkHolder) -> { Chunk chunk = (Chunk)chunkHolder.getSavingFuture().getNow((Object)null); if (chunk instanceof ReadOnlyChunk || chunk instanceof WorldChunk) { this.save(chunk); chunkHolder.updateAccessibleStatus(); } }); } }This code saves all accessible chunks, i.e., chunks that were in the BORDER status at some point since the last save, periodically (when flush is false) and upon closing the world (when flush is true). The second case is similar (and hence not included in the code snippet) but additionally waits for pending futures in the accessible chunks.
The point is now that neither of these cases covers chunks that are still kept alive by chunk tickets when closing the world, but do not have a sufficient ticket level to be BORDER chunks. Those are chunks that are only partially generated, but can already contain features and light propagations from adjacent chunks. Not saving these chunks will hence lead to corrupted features and broken light at chunk borders.
In the following I will presentthree concrete examples where chunk tickets can keep such chunks alive and will hence prevent saving. Some other ticket types can most certainly lead to the same effect.1. Spawn chunks
The first example uses START tickets created on spawn chunks (the same works with FORCED tickets), which are not removed upon unloading the world and hence will keep some chunks alive.
- Create a new world with render distance 2 (e.g., with seed -4530634556500121041)
- Quit and reload directly after generating the world
- One can now turn up the render distance again
- Observe that features between the 11th and 12th chunk away from the spawn point are broken (World spawn loads chunks in a radius of 11 into BORDER status)
Due to the low render distance, chunks in distance 12 from the spawn point are not requested into BORDER status, but still kept from unloading due to the START ticket, hence triggering the described issue.
2. Overload the lighting engine
The second example uses LIGHT tickets created during worldgen to keep chunks alive. While these tickets will eventually be removed when the worldgen is finished, the goal here is to overload the lighting engine so that these generations steps will not be finished (and hence the tickets will not be removed) in time before the flush save completes. Note that the flush save only waits for accessible chunks to finish any remaining futures, but not for partially generated chunks.
Note that this behavior is quite random and hence the following steps might require several attempts for reproduction or may not work at all, depending on the setup. Nevertheless, I think this example is still of practical relevance as described in the final section.
In order to stress the lighting engine we can try the following steps
- Create an AMPLIFIED world with render distance 32
- Fly a few chunks in one direction and then close the world immediately
- Repeat the previous step several times, always flying in the same direction
- After repeating for a few times, revisit the generated chunks again and see if some corrupted features or light glitches occur.
(Note that there are also some lighting glitches on the chunk border)Closing the world immediately after flying a few chunks does not give the lighting engine enough time to finish its jobs before unloading the world (at least sometimes), hence causing the described issue.
3.
Interacting with the worldEvery world interaction, or more precisely every call to World.getChunk(...), creates an UNKNOWN chunk ticket with the intent to keep the chunk loaded and hence valid until the end of the tick. Every ticking entity and every player creates such a ticket at its location every tick, e.g., due to movement operations querying the chunk. These UNKNOWN tickets are in fact not cleared at the end of the current tick, but at the start of the next one. Consequently, these UNKNOWN tickets from the last tick are not purged upon shutting down the world and are hence still presnt during saving, prohibiting unloading and saving of some chunks (in contrast, PLAYER tickets are explicitly removed upon shutting down the world by disconnecting all players).
This does not seem to be an issue on high render distances, presumably because there are no entities far away from the player, at least not in the very last tick, and hence no such UNKNOWN tickets are created away from the player, whereas the PLAYER ticket itself has a much higher range and hence dominates these UKNOWN tickets. Hencethe UNKNOWN tickets only affect chunks that were already accessible due to the PLAYER ticket and hence do not trigger the saving issue.
However, this argument does not work for small render distances, e.g., render distance 2, where the UNKNOWN tickets do keep chunks alive that were not yet accessible, hence causing the saving issues. Indeed, the issue does show up deterministically when exploring new chunks with a render distance of 2 and reloading the world every few chunks.Remarks
I think the issue described here is a good candidate for explaining MC-125007 (respectively
MC-188086).
I am not sure, though, to which extent it is related toMC-214793. It again seems to be a good explanation for the corrupted features that seem to occur in conjunction with the dark chunks. However, it fails to explain why these chunks are completely dark; it only explains missing light propagations at the chunk borders, since these are discarded upon saving. So, it looks like some other unknown issue is interfering here aswell.There are two codepaths that invoke saving of chunks.
1. ThreadedAnvilChunkStorage.tryUnloadChunk(...) which saves chunks that are no longer kept alive by any chunk ticket, before finally unloading them from the world.
2. ThreadedAnvilChunkStorage.save(flush) which saves chunks periodically and upon closing the world.net.minecraft.server.world.ThreadedAnvilChunkStorage.javaprotected void save(boolean flush) { if (flush) { ... } else { this.chunkHolders.values().stream().filter(ChunkHolder::isAccessible).forEach((chunkHolder) -> { Chunk chunk = (Chunk)chunkHolder.getSavingFuture().getNow((Object)null); if (chunk instanceof ReadOnlyChunk || chunk instanceof WorldChunk) { this.save(chunk); chunkHolder.updateAccessibleStatus(); } }); } }This code saves all accessible chunks, i.e., chunks that were in the BORDER status at some point since the last save, periodically (when flush is false) and upon closing the world (when flush is true). The second case is similar (and hence not included in the code snippet) but additionally waits for pending futures in the accessible chunks.
The point is now that neither of these cases covers chunks that are still kept alive by chunk tickets when closing the world, but do not have a sufficient ticket level to be BORDER chunks. Those are chunks that are only partially generated, but can already contain features and light propagations from adjacent chunks. Not saving these chunks will hence lead to corrupted features and broken light at chunk borders.
In the following I will present some concrete examples where chunk tickets can keep such chunks alive and will hence prevent saving. Some other ticket types can most certainly lead to the same effect.1. Spawn chunks
The first example uses START tickets created on spawn chunks (the same works with FORCED tickets), which are not removed upon unloading the world and hence will keep some chunks alive.
- Create a new world with render distance 2 (e.g., with seed -4530634556500121041)
- Quit and reload directly after generating the world
- One can now turn up the render distance again
- Observe that features between the 11th and 12th chunk away from the spawn point are broken (World spawn loads chunks in a radius of 11 into BORDER status)
Due to the low render distance, chunks in distance 12 from the spawn point are not requested into BORDER status, but still kept from unloading due to the START ticket, hence triggering the described issue.
2. Overload the lighting engine
The second example uses LIGHT tickets created during worldgen to keep chunks alive. While these tickets will eventually be removed when the worldgen is finished, the goal here is to overload the lighting engine so that these generations steps will not be finished (and hence the tickets will not be removed) in time before the flush save completes. Note that the flush save only waits for accessible chunks to finish any remaining futures, but not for partially generated chunks.
Note that this behavior is quite random and hence the following steps might require several attempts for reproduction or may not work at all, depending on the setup. Nevertheless, I think this example is still of practical relevance as described in the final section.
In order to stress the lighting engine we can try the following steps
- Create an AMPLIFIED world with render distance 32
- Fly a few chunks in one direction and then close the world immediately
- Repeat the previous step several times, always flying in the same direction
- After repeating for a few times, revisit the generated chunks again and see if some corrupted features or light glitches occur.
(Note that there are also some lighting glitches on the chunk border)Closing the world immediately after flying a few chunks does not give the lighting engine enough time to finish its jobs before unloading the world (at least sometimes), hence causing the described issue.
3. Player tickets
As pointed out by Rik Lubking, PLAYER tickets can cause this issue very deterministically (comparable to the START tickets) and are hence probably the most common and severe case. While PLAYER tickets are explicitly removed upon shutting down the world by disconnecting all players before saving, this ticket removal does involve some threading and hence might not be completed before the saving executes, hence keeping all chunks loaded by players alive during saving and consequently triggering the issue.
4. Interacting with the world
Every world interaction, or more precisely every call to World.getChunk(...), creates an UNKNOWN chunk ticket with the intent to keep the chunk loaded and hence valid until the end of the tick. Every ticking entity and every player creates such a ticket at its location every tick, e.g., due to movement operations querying the chunk. These UNKNOWN tickets are in fact not cleared at the end of the current tick, but at the start of the next one. Consequently, these UNKNOWN tickets from the last tick are not purged upon shutting down the world and are hence still present during saving, prohibiting unloading and saving of some chunks (in contrast, PLAYER tickets are explicitly removed upon shutting down the world by disconnecting all players).
This does not seem to be an issue on high render distances, presumably because there are no entities far away from the player, at least not in the very last tick, and hence no such UNKNOWN tickets are created away from the player, whereas the PLAYER ticket itself has a much higher range and hence dominates these UKNOWN tickets. Hencethe UNKNOWN tickets only affect chunks that were already accessible due to the PLAYER ticket and hence do not trigger the saving issue.
However, this argument does not work for small render distances, e.g., render distance 2, where the UNKNOWN tickets do keep chunks alive that were not yet accessible, hence causing the saving issues. Indeed, the issue does show up deterministically when exploring new chunks with a render distance of 2 and reloading the world every few chunks.Remarks
I think the issue described here is a good candidate for explaining MC-125007 (respectively
MC-188086).
I am not sure, though, to which extent it is related toMC-214793. It again seems to be a good explanation for the corrupted features that seem to occur in conjunction with the dark chunks. However, it fails to explain why these chunks are completely dark; it only explains missing light propagations at the chunk borders, since these are discarded upon saving. So, it looks like some other unknown issue is interfering here aswell.
The bug
When a block on fire is broken, the light of the destroyed fire is not updated.
How to reproduce
Put a tree on fire and wait for the destruction of the blocks and you will observe the not updating of the brightness level (also affects the menu F3).
Note
From Philipp Provenzano:
This is partially resolved in 18w44a (iirc already 18w43c).
Lighting is now correct clientside. However according to F3 it's still wrong serverside.
Relogging fixes the serverside part. However, at least for spawn chunks, teleporting away and returning won't fix the issue and the wrong lighting is then actually synced to the client.
The bug
When TNT explodes in around campfires, the smoke particles sometimes still appear when the campfire is blown up already. Reloading the world fixes this issue.
Code analysis by Narcoleptic Frog and Philipp Provenzano can be found in this comment.
The bug
I've encountered some sort of lag spike which I narrowed down to be a floating structure. This structure can be a single block.
The lag spike seems to occur when I'm moving into the spawn chunks while an elevated block is placed on the opposite side of the spawn chunks. It occurs when the block is at Y96 or higher.
To replicate what I did on a 1.14.4 vanilla server:
- Create a default world with seed: -4952361208952771886
- Place a block at x 143 y 63 z -371 and another block at x 144 y 63 z -371. This is the location to test for the lag.
- Place a block at x 307 y 96 z -302. This will be the block to create the lag.
- Set the spawn point to x 227 y 66 z -376
- Walk back and forth over the two blocks you placed earlier.
- If you see a stutter, feel free to remove the block at x 307 y 96 z -302 and check again. Lag will be gone.
I cannot attach the video as the file size is slightly too large but you may watch it here: https://youtu.be/jmtq3jcXujo
I should note that it is not specific to my PC. It occurs on other clients on other devices.
Render distance will affect where the lagspike occurs in vanilla singleplayer, but is relative to the view-distance on servers. I believe it's distance + 1 in chunks, or there abouts.
Code analysis
Code analysis and further explanation by Philipp Provenzano can be found in this comment. There also exists a Fabric mod that fixes this issue: comment.
![[Mod] violine1101](../../../data/avatars/users/92858.png){kind=small}
I've talked a bit with Philipp Provenzano and this seems to be a general issue with light not propagating properly across chunk borders. We're going to collect those issues in MC-172980.
Reopened, I was successfully able to reproduce this in 20w17a, thanks to Philipp Provenzano's assistance over on Discord.
Apparently, the Invulnerable tag is not synced to the client.
For mobs, attackEntityFrom(DamageSource source, float amount) checks whether it is called from the client which is why they aren't killed on the client. However, this check is missing for Items and XPOrbs.
There are some other things going on, which actually cause dealFireDamage() to be called on the client for Items but not mobs.
Anyway, adding the check fixes the problem.
I can't reproduce explosions killing the item. Maybe, they just turned invisible and got pushed away, so you couldn't pick them up?
Edit: Apparently, that's what @Marcono1234 already found out; missed the comment
The relevant code lines that cause the issue are
inside World.getRawLight(...). The opacity for solid light sources is set to 1 (for whatever reason...).
However,
MC-3961is mainly caused by an unrelated issue. SeeMC-117067for a detailed description and solution.The bug is still present in 18w07c.
The heart of the problem is that "rebuildNear" (or whatever it is called in the profiler) became much slower in the 1.13 snapshots. It is the process of collecting data of a section that is then used by the GPU to render this section. This has to be done every time an update occurs in the section. For near sections, this is done on the main thread, sections that are farther away are rebuilt on worker threads. That's why the issue disappears when moving away from the contraption.
Redstone and light updates are just the easiest way to cause a bunch of sections to be rebuilt, but the underlying issue is completely unrelated to redstone and lighting. Sure, they cause a fair amount of lag by themselves, but that's another topic.
To see this, go in creative mode and spam click to destroy blocks (1-2 blocks per second). In the 1.13 snapshots, you will notice some spikes in the FPS graph, however they don't appear in 1.12.
Updates at chunk borders cause all adjacent sections to be rebuilt, so those updates have an even stronger effect on the issue. This is exactly
MC-1692.Your FPS are unlimited. Limiting it to, say, 60 FPS should solve the "issue".
This issue also relates to MC-123584 and
MC-123263which are about the decreased performance of "rebuildNear" in the 1.13 snapshots I described above..Duplicate of
MC-117030This is fixed in 18w43a.
This is fixed in 18w43a.
This is partially resolved in 18w44a (iirc already 18w43c).
Lighting is now correct clientside. However according to F3 it's still wrong serverside.
Relogging fixes the serverside part. However, at least for spawn chunks, teleporting away and returning won't fix the issue and the wrong lighting is then actually synced to the client.
Might be related to
MC-30939Seems to be fixed in 18w46a.
However,
MC-30939still seems to exist.Can anyone confirm this?
Here's a code analysis on this bug:
To take away the punch line: The issue is completely client-side. Chunks are loaded properly on the server and sent to the client. The client, however, discards them when they fall outside its view range which leads to multiple problems.
There are two codepaths on the client that throw away chunks when they fall outside the clients view distance.
(1) A filter that discards chunks right away when the chunk packet arrives.
(2) A clean-up routine that unloads all chunks outside the clients view radius once per tick (or whenever a new chunk packet arrives).
This causes (at least) three slightly different appearances of the bug.
Appearance 1: ALL chunks outside the login area are missing
When joining a multiplayer server and clientViewDistance < serverViewDistance, no chunks outside the login area are visible. This is caused by the filter codepath. The server will send chunks as soon as the player is less than serverViewDistance chunks away. However, the client will directly discard them if they fall outside clientViewDistance. So, when moving continously (ie. not teleporting) all new chunks will be discarded by the client because its view distance is stricly smaller than that of the server.
The server doesn't know about this and assumes the client knows all chunks it sent. As a result, all of those chunks will be permanently missing on the client until relogging (or moving away far enough, so the server sends them again).
This mismatch of view distances between client and server is the most servere of the three appearances.
Workaround:
As others have noticed, increasing the client view distance helps to counteract the problem. From the analysis, it is clear that you need a view distance of at least serverViewDistance.
Appearance 2: Spawn chunks and chunks loaded by other players are missing
This is a slight variation of Appearance 1, where only spawn chunks and chunks loaded by other players are missing.
One detail that I didn't mention in Appearance 1, is that the server view distance isn't actually what you configure, but it is 1 larger.
The server view distance is basically set as follows:
... omitting some call levels ....
Note the additional + 1 in MathHelper.clamp(int_1 + 1, 3, 33). This does not happen for the clientViewDistance.
Hence we should distinguish the chunkTrackingDistance = serverViewDistance + 1 where the latter is what you configure in server.properties.
Because of this, we have Appearance 1 again in disguise. It does not just happen when clientViewDistance < serverViewDistance but actually when clientViewDistance < chunkTrackingDistance.
So, why does clientViewDistance == serverViewDistance help to solve Appearance 1 in some cases?
There's another variable controlling the chunkLoadingDistance. This one is now actually == serverViewDistance (I haven't read that codepath completely to determine the exact value. But from Appearance 1 I conclude that it should be this). Because of this, the effective view distance for not yet loaded chunks is capped by the chunkLoadingDistance, ie. serverViewDistance, and so setting the client view distance to this solves Appearance 1 for all not yet loaded chunks.
However, as soon as you come near a chunk already loaded on the server, e.g. spawn chunks, the effective view distance increases to chunkTrackingDistance == serverViewDistance + 1, because you don't need to load the chunk first, so chunkLoadingDistance is irrelevant. This triggers Appearance 1 again.
Workaround:
As for Appearance 1, increase the clientViewDistance. It now becomes clear that we need to set it to at least serverViewDistance + 1.
Appearance 3: Chunks are missing when "moving too quickly"
This issue is now caused by the clean-up unloading codepath. It can be reliably produced as follows:
Because of the clean-up unloading path, the client unloads the chunks behind you as soon as they get out of the client's view range. When the player is then teleported back by the server, those chunks will be missing. As before, the server doesn't know the client unloaded them and hence won't send them again.
From the explanation, it gets clear why this only happens for chunks behind you but not for chunks in front of you.
Workaround:
Again, increase the client view distance. As it gets larger, the probability of encountering this issue gets smaller, as you need to be teleported back at least chunkTrackingDistance - clientViewDistance chunks. Note that setting the client view distance higher than serverViewDistance + 1 shouldn't cause any harm, because the server won't send more chunks anyway. Hence I recommend setting it to 32.
Proposed Solution
The client should tell the server its view distance. And the server should take that into account for determining which chunks to send to the client. Whenever the server determines that the client should unload a chunk, it should tell it to do so (this mechanism already exists). The client should never automatically unload any chunks. (As mentioned above, allowing the client to automatically unload chunks results in the server thinking the client has a chunk that it does not have, so the server will never send that chunk again, and this is why some chunks appear to never load.)
The suggestion to have unload controlled by the server is optimal. In particular, this will reduce the bandwidth usage compared to the current algorithm, which still sends chunks that the client just discards. And it should be fairly easy to implement as most of the logic already exists. Basically, one only needs to remove the client side filter and automatic clean-up logic and instead execute unload requests sent by the server.
If you wish to allow the client to automatically unload chunks, things get more complicated. Either the client should tell the server whenever it unloads a chunk, and/or the client should be able to request to receive a chunk even if the server thinks the client already has it. Unfortunately, in this scenario, network lag may still cause client/server desync, similar to Appearance 3. For the most seamless experience, with least bandwidth, chunk unloading desired by client and server should not occur asynchronously to each other.
I really don't see any benefit in the client making unloading decisions, especially since that complicates things alot. So, I strongly recommend to just get rid of this and allow the server to decide which chunks the client should unload, based on the minimum of server view distance and client render distance.
Hope this is info is helpful
Thanks to Timothy Miller for helping me with testing and writing the report.
Copied my code analysis from
MC-138114:Only the first part of the issue was fixed in pre-5.
The second part still exists.
I have put them inside a single bug report, because they are thematically related, both having to do with initialization of lightmaps.
However, my reproduction steps above for the second part turned out to be wrong. My comment that "clientside lighting is not affected by the above issue, so fixing it will not fix this new issue" was actually wrong. Because the steps cause large numbers of block updates, light packets get sent to the client, so the client does get affected by the server light. This hided the fact that my reproduction steps were wrong.
The problem with the reproduction steps was that I assumed that a block update only schedules light checks for the block itself. However, it turns out that also light checks are scheduled for the 6 neighbors, which creates some problems.
Fortunately, I could design another list of repdroduction steps (the idea is basically the same as for the old steps, so I won't explain any further. Also, there are some additional steps because I have to circumvent
MC-148723):While the image looks a bit different, it still shows the same basic features: the left subchunk is too dark and the right subchunk too bright.
Admittedly, those reproduction steps are rather artifical. I have designed them to show the case where "darkness is not spread to the neighbors", ie. the lighting engine does not recognize that the subchunk turns dark due to the missing initialization and hence does not remove light propagation to the neighbors. This is only meant for showing that there actually is a problem with the current initialization code, which can be fixed for example by the initialization approach i suggested.
On the other hand, the fact that the left subchunk is too dark can be reproduced more easily as follows
This situation occurs naturally when building under an overhang (although that overhang must be at a considerable height above ground).
Hi Fry, thanks for your answer
If I understand you correctly, I think your first approach was also the first approach I mentioned for handling not yet lighted chunks, ie. "Just use the simple initialization algorithm all the time". That one I didn't like because you have to remove all of the light after worldgen again, which is exactly the problem you had with it.
. How did you handle light updates into unlit chunks?
Your second approach also sounds like my second approach, ie. "Keep everything dark before the chunk is initially lighted, ..."
In case you just discard them, this might be the cause for the black spots you were observing.
You can write me on Discord, if you want to discuss in greater detail (or ofc continue discussion here). But take your time experimenting with it after fixing the other bug.
@Michel That issue doesn't really look related. It is not about lightmaps not being initialized when they are created, but it seems like lightmaps actually get lost during saving. I have a few ideas what could be the cause of it and will look into it during the weekend (hopefully)
Is this still working as intended, given that directional lighting is now possible?
You might want to add this to
MC-139435.As far as I can tell, this is fixed in 1.14.
This is still not completely fixed in 1.14. I'm posting this here since the new issue is again caused by lightmaps not being properly initialized, so it has the same cause as the previous issues.
You can reproduce the new issue by the following steps:
Observe that the ground of the right chunk is completely bright.
Actually, already the subchunk below the platform is bright
Looking at the code, light maps for already lighted chunks are now (most of the time) properly initialized inside
However, the check if (this.m.contains(ChunkSectionPos.asLong(ChunkSectionPos.unpackLongX(long2), integer7 - 1, ChunkSectionPos.unpackLongZ(long2)))) prevents the lightmap to be initialized if it is the new topmost lightmap for this chunk and there was another lightmap added above the old topmost one in the current update run.
So, what is happening in the example above?
First, the lightmap for the left subchunk (in the picture) containing the stone platform is created. This also creates lightmaps for the 26 neighbor subchunks (diagonal neighbors included). Apart from the first chunk, the lightmaps are created and initialized bottom to top. This means, the lowest lightmap is created first and initialized correctly to a skylight value of 15. Then it is passed to the old initialization logic, which adds the subchunk coordinates to the set m. Later, the lightmap above it (corresponding to the height where the stone platform was added) gets created and initialized. This time, though, it gets initialized to a skylight value of 0, because of the additional check. Then, by the old initialization logic, it gets added to the set m and the previous subchunk gets removed and added to the set n instead. Finally, the topmost lightmap will be created and initialized at some later point. And then, analogously, the lightmap for the subchunk containing the stone platform will get removed from m and added to n.
In the end, apart from the first column that was processed, the situation looks as follows:
What does the old initialization logic do then?
As far as I can tell, this.l is a set consisting of the previous topmost lightmaps, ie. the topmost lightmaps before the current update run. In particular, it does not contain the two lower lightmaps which were only temporarily at the top, but it only contains the lightmap slightly above the ground, which was the previous topmost lightmap. Hence, this initialization logic will just be skipped for those two lightmaps.
In summary, this means for the right chunk in the picture
Because the subchunk containing the platform is already dark after the lightmaps are added to the world, the lighting engine does not propagate darkness caused by the added stone platform. And hence it will not notice that the subchunk below is too bright.
So, the problem is exactly the same as in the previous cases: The subchunk containing the stone platform is not initialized properly, and hence the lighting engine doesn't notice that it should propagate updates to the (correctly initialized) subchunk below it.
What is the idea behind this additional check if (this.m.contains(ChunkSectionPos.asLong(ChunkSectionPos.unpackLongX(long2), integer7 - 1, ChunkSectionPos.unpackLongZ(long2))))?
Apart from that check, the new initialization logic looks correct to me. If everything works correctly, the old initialization logic should be obsolete and could be removed.
Duplicate of
MC-142134, you should add relevant information thereWas the light also disappearing for light sources in the same chunk as you WHILE afk fishing? Or did this only happen after walking around?
To those who experienced the bug while just standing afk:
Updated the code in the Github repo.
An analysis and solution is given in
MC-170010.This is simply a rendering issue: the bottom chunk is not marked for rerendering. Causing any block update in that chunk or reloadong the world will fix the issue.
The actual skylight as shown in F3 is correct.
It turns out that the interaction between initial lighting and skylight optimization introduces some subtleties I didn't really deal with; basically "Hence, all skylight updates will be properly propagated" isn't completely true due to the interaction between skylight optimization and the fact that some subchunks might be partially opaque to skylight.
I will update that discussion once I figured things out completely.
Nevertheless, this doesn't really affect the general theme of the solution: Initialize lightmaps to the exact values that would have been returned before adding it and only schedule additional propagations as needed to fullfill the desired invariant for initial lighting. Don't mix lightmap initialization with initial lighting code and compensate the change in values with a bunch of other checks. That will only complicate the code and has a high probability that some check is missing.
For the moment, I would suggest to simply keep that part of the initialization logic that spreads light to neighbors (in fact, independent of whether the subchunk is empty or not), but properly initialize lightmaps inside createLightArray(...) and remove those parts of the logic that reinitalize lightmaps and spread darkness.
In order to figure out the correct way to deal with the interaction between initial lighting and skylight optimization, it would be quite helpful to know whether the event of a block change inside a chunk between pre_ligth and light stage as described above is possible or not
After looking through the code some more, I feel like it is indeed possible that there are block changes inside chunks between the pre_light and light stage. In particular, ProtoChunk.setBlockState(...) contains the clause
suggesting the possibility of block changes in those stages.
For what it's worth, I could only come up with one simple implementation that trivially works correctly (which I will present below). Everything else I tried causes problems in one way or another. The basic problem is that the skylight optimization is only aplpicable away from any non-air blocks, but that the empty region behaves necessarily as if it were opaque. This makes the skylight optimization inapplicable near the empty region of any not yet lighted chunk, as we don't necessarily have enough lightmaps around. Hence many applications of it during initial lighting are in fact not correct; however, simply turning off this optimization would cause problems as well, as we have to deal with missing lightmaps around the empty region in some way.
I can also cook up examples that break the current vanilla code and produce persistent errors, as long as block changes in those chunks are not forbidden. I also suspect some more issues than I currently found...
So, in summary, I think it's not at all straightforward to come up with some solution (other than the one I present below) that works in all edge cases. The current solution definitely does not, although those situations don't arise in practice with high probability.
If block changes were forbidden, the situation would be much simpler. In that case, we wouldn't really care about what happens in the pseudo-empty region, as it will be completely filled with skylight later on, anyway. So in this case we could simply state in the invariant that the pseudo-empty region does whatever it wants (as long as light is correctly propagated to the outside, perhaps). But as soon as the pseudo-empty region is allowed to change and light can decrease, things are really easy to mess up. Keeping any invariant always seems to require some repair mechanism.
On another note, there is currently also a problem when removing the topmost lightmap of the pseudo-empty region: When removing such a lightmap, light values will be overridden with 0. The reason for this change in light values is that we model the empty region as opaque, but do not properly react to this change in opacity. The result from this is that we now produce ghost light, in case there was any outgoing light from this lightmap before its removal. If later on some skylight is decreased, the lighting engine won't be able to properly track this through the missing light map and will produce persistent errors. So, currently we also should counteract this situation by spreading darkness when removing such a lightmap. However, this is not needed in the solution below.
Also, there is currently some mechanism in the code that prevents lightmaps from being completely erased before the chunk finishes its initial lighting. Readding a previously removed lightmap will simply restore its old values. I'm not really sure what the idea behind this is. My guess is that it should prevent lightmaps from being deleted between loading the chunk from disk and registering its non-empty subchunks. The problem with this is that this mechanism is only turned off once the initial lighting is done. So, if we readd some lightmap during initial lighting, some stale values will be restored instead of creating and initializing a new one. It is again easy to cook up examples, where this stale lightvalues will cause persistent lighting errors. So, I would suggest here to disable this mechanism as soon as the non-empty subchunks have been registered and not only after initial lighting.
So, let's now discuss my proposed solution for initial skylight. It is conceptually quite easy and avoids all this discussion about inapplicable skylight optimization and actions required upon adding and removing lightmaps. Currently, the main problem is that the empty region behaves as if it was filled with stone, causing all kinds of problems around it. To overcome this issue, we can simply push up this misbehaving region high up into the sky.
we can then simply remove the forceloaded lightmap.
Basically, I want to model initial skylight simply as if there was a stone platform high up above the world and then use the normal lighting code. This will solve all these nasty issues.
Concretely, this means that we should forceload a lightmap in the void above the world in the pre_light stage together with all the regular lightmaps (as far as I can tell, lightmaps can currently be created 1 subchunk below and above the world bounds already) and make it receive all light contributions from all sides, but no direct skylight from above. This perfectly simulates the situation that there was a stone platform just above this lightmap. Preventing direct skylight contribution is already implemented, so there isn't really much to do here. The chunk now behaves as any other ordinary chunk, simply with a stone platform directly above it. Hence, as for any other chunk, there are no special actions required upon adding or removing lightmaps and skylight optimizations are now perfectly fine.
Initial lighting now takes the form of removing this imaginary stone platform. Concretely, this means that we should set all lightmaps in the pseudo-empty region to a value of 15 and spread light to everything adjacent to the pseudo-empty region, as is currently almost done (currently we only spread light to neighbors below the topmost lightmap, but we should in fact spread light for every y-level in the (pseudo)-empty region). Afterwards
While this approach might be slightly less optimal due to the additional lightmap, I don't think that the difference will be really noticeable. Given the fact that it avoids all kinds of light checks upon adding and removing lightmaps, it might even pay off performance-wise.
Basically, this is the same solution as already presented in the original report, with the only difference being this forceloaded lightmap high up in the sky that solves all the nasty issues. So this keeps all the benefits listed above, primarily it allows to simply solve the bug originally addressed in this report by simply initializing lightmaps with the previous values at the respective positions and not doing any other action upon lightmap addition or removal. It also moves all of the initial lighting code to one place, instead of being scattered around, making the coder easier to understand and maintain.
I would suggest to implement this solution, mainly for its simplicity and correctness.
It would be cool to hear your opinions on this. In particular, I would be interested in what your idea was for the current initial lighting code. Maybe we can come up with some fancy more optimal and actually correct solution together
Best,
PhiPro
The method you describe already exists and is way higher up the chunk loading chain
However, the chunk ticket created in this.getChunkFuture(x, z, leastStatus, create) only lasts for the current tick, hence the chunk will be unloaded again immediately in the next tick. Furthermore, this ticket is only strong enough to load the chunk as border chunk, but not as ticking chunk.
Anyway, this doesn't explain why something is already joined in the very early loading stage. Note further that the main thread can still execute scheduled tasks while waiting in getChunk(...), so it can at least do something useful, as long as there are unfinished tasks, but it is completely frozen while waiting in getNbt(...).
Took me quite a while to fit all the puzzle pieces together, but I finally have a satisfying explanation for this bug with all its weird features.
Basically, it is caused by
MC-170010and my proposed fix for it will fix this issue as well.Let me first explain what is causing the lagspikes. It is basically the same lagspike as experienced in the following issue:
This lagspike occurs whenever a block is placed high above the ground, where "ground" really means the topmost block of all neighbor chunks. As soon as there is only a single block above, the lag spike disappears. For example, if you /setblock 7 255 7 minecraft:glass then the above steps won't cause any lagspike.
The reason for this is
MC-170010. As mentioned there, topmost skylight maps are not properly initialized to their previous values. Instead, they are created completely dark and then get relighted. Placing a block high above ground creates 27 new lightmaps. Due to the order of creating them bottom to top, all of them will be uninitialized and have to be relighted. So, although we place a glass block, which doesn't cause any light change at all, the client has to relight 27 subchunks at once due to this bug, which takes quite some time.As soon as there is any block above, no (or fewer) lightmaps get created, hence avoiding the issue.
So, my proposed fix for this issue is my proposed fix for
MC-170010.While not really relevant for the solution, I think it is still interesting to explain why the bug doesn't occur in many situations and why it is related to spawnchunks and is also stateful. So, let's start by explaining how the previous lag spike relates to the one of this bug report.
The following easy setup is sufficient for triggering it:
When you move back from (4 0) to (3 0), the chunk (0 0) gets loaded on the client. This causes the lightmaps to be created around subchunk (0 6 0) and hence leads to the exact same issue as previously described. Namely, those lightmaps are created uninitialized and have to be relighted all at once, which costs too much time. (Actually, in this case not all of the 27 lightmaps are directly initialized, but only those in the already loaded chunks. Nevertheless, this seems to be enough to lag the client quite bad.)
Why does it happen only for certain chunk borders and not all of the time?
First of all, the issue relies on lightmaps being created upon loading chunks on the client. The lightmaps created for the newly loaded chunk are in fact initialized, namely to the data the client receives from the server. So those don't contribute. (Actually, the topmost lightmap of the newly loaded chunk is still thrown away and relighted in any case. But this single lightmap seems to have sufficiently small impact to not cause lag spikes all of the time. Nevertheless, this unnecessary relighting would be eliminated with my proposed solution as well.) So, a necessary condition for the issue is that the newly loaded chunk is much higher than its already loaded neighbors.
Nevertheless, even in this case, there are still a lot of awkward situations that avoid the bug:
These strange stateful features of the bug were really giving me some headaches while hunting it down
The block isn't loaded on the client while at (4 0), so replacing it by air and then glass again shouldn't have any effect on the client state, yet it turns out that it avoids the lag spike.
The easiest case is what happens if you reload the world at (4 0). In this case, when the server sends the data for chunk (1 0), it also sends the lightmaps for (1 5 0), (1 6 0) and (1 7 0). Similarly for (1 -1) and (1 1). Those lightmaps are not directly added to the world, since there is no nearby block yet. Instead they are kept queued. Once we move to (3 0), the block at (0 6 0) creates all the surrounding lightmaps. Now, in this case, the lightmaps for (1 -1), (1 0) and (1 1) are not uninitialized, but are initialized from the queued up data that the client received from the server. That is why the bug does not occur in this case.
If we now move back to (4 0), the block at (0 6 0) and hence all the surrounding lightmaps get unloaded. Now, the neighbor chunks do not anymore have any queued up data from the server, and the server won't resend the data unless we move even further away to (5 0). So, when now crossing the chunk border to (3 0) once again, the lightmaps for chunk (1 -1), (1 0) and (1 1) will now indeed be uninitialized and have to be relighted, causing the lagspike.
What happens if you replace the block by air and then glass again while at (4 0)?
When replacing the block by air, the server deletes all the lightmaps. When afterwards spawning the glass block again, the server encounters exactly the same issue of
MC-170010and has to relight the uninitialized lightmaps. This will be detected as ligth changes, although the final light values actually haven't changed. Now, some other piece of code comes into play.What it does is to sync all light updates at the edge of the client's view area. Hence, the (wrongly) detected light updates cause the server to sync the lightmaps for (1 -1), (1 0), (1 1) to the client again, which basically puts us in the same starting point as in the previous case. Those synced lightmaps are used for initialization and hence the issue doesn't occur.
Finally, let us understand the role of spawn chunks. As it turns out, this is due to yet another effect, namely that the chunk tracking distance is 1 larger than the chunk loading distance. Concretely, this means the following:
So the difference between spawn chunks and non-spawn chunks is that they are kept loaded on the server and hence shade this discrepancy between tracking and loading distance. This also means that the same rules apply to chunks loaded by any other means, e.g. other players.
Further remarks
As another suggestion, it might be useful to run the client lighting engine offthread as well, as is already done for the server. Of course, you want some way for the client thread to wait for light updates to finish before rendering, as even the slightest latency of light updates near the player usually feels very annoying. Such a way to wait for light updates to finish is needed anyway for
MC-164281. Additionally, you probably also want to implement a fix for MC-177685 such that updates near the player can be prioritized and the client thread only has to wait for those, rather than all pending updates. Nevertheless, even without a fix for MC-177685, putting the clientside lighting engine offthread can help mitigating this issue further.As a final remark, I really strongly recommend to implement my proposed solution for
)
MC-170010instead of putting more and more band-aid on it, as it makes it increasingly difficult to track down issues. This one already gave me a lot of headaches as it had so many strange fatures related to different code paths. (Admittedly, the challenge was also kindof funWorkaround for players
As a temporary workaround for player, you can place a ceiling of minecraft:barrier at y=255 above the whole map. As can be seen from the analysis, the issue arises from a height difference of neighbor chunks. Putting a ceiling above the world eliminates any height difference and reduces the issue to its minimal intensity of flat worlds. Putting the ceiling only above a small part of the world will simply push out the problematic chunk boundaries to the end of the ceiling, so for this workaround to work you need to cover all of the area you visit. While this is definitely not perfect, it should at least help in the case when you are moving in already generated chunks and not visiting new ones.
Best,
PhiPro
As an additional remark, in order to remove the artificial throttling described above, one should first fix MC-183841, so this doesn't result in more tasks getting executed than before.
On the other hand, even without a fix for MC-183841, a fix for this issue will increase chunk loading performance. since the tasks will be properly prioritized. Nevertheless, without a fix for MC-183841 this might have a negative impact on the total CPU and IO load.
While implementing this fix as a fabric mod, i noticed that
MC-170012should ideally be fixed first, as it makes the handling of the forced lightmaps, as mentioned in the previous comment, a bit easier. The problem is that upon initial lighting we need those forced lightmaps to be present for the neighbors, which is currently not so nice because ofMC-170012. Alternatively, instead of fixing that issue first, you can also explicitly force these lightmaps on the neighbors, but I think this will be the more cumbersome way to do things.As a reference, I have implemented my suggested solution as a fabric mod. The code can be found at https://github.com/PhiPro95/mc-fixes/tree/mc-170010. While doing so, I noticed a few things that are worth documenting here. Also note that the code really isn't that long or complicated, although the use of mixins vs. direct source control sometimes makes things look more complicated than they actually are.
Observe that this also solves
MC-162253as explained there.First, as already noted in the last comment, it is convenient to fix
MC-170012first. This is done in https://github.com/PhiPro95/mc-fixes/tree/mc-170012. In my implementation I introduced a new pre_light generation stage and mirrored the state of the current light stage, i.e., it requires chunks in the features stage within a margin of 1. This is due to the fact that ProtoChunk does not properly handle lightmaps, as noted in the code. So I didn't want to introduce problems by promoting chunks too early and hence mirrored the current light stage. Whether or not this margin can be removed or if the missing lightmap handling in ProtoChunk is in fact a bug requires some knowledge about the worldgen code that I don't have. So that decision is up to you.Edit: Looking a bit more into the worldgen code it indeed looks like the missing lightmap handling is a bug. Once the missing lightmap handling is added and the blockligth version of
MC-170012, as noted in the comment there, is fixed, this margin is no longer needed. This is done in the most recent version of my implementation.Second, it turns out that the current data retainment mechanism, as mentioned in the report and comment above, has some more far reaching problems than I thought. It not only overrides lightmaps in certain rare situations, as mentioned above, but it can in fact lead to loss of data in surprisingly common situations. I will link the bug report to that once it is created.
As a consequence, I would ignore this lightmap retainment code for the moment and simply leave it as it is. Simply write the code as if the issue with the lightmap retainment code was solved. This doesn't cause any regression compared to the current vanilla code, any potential problems already exist within the current code.
The retainment code can then be fixed later. In fact, it may be even convenient to solve this issue here first, so that newly created lightmaps are always trivial in the appropriate sense. This would at least slightly help for the solution I have in mind, but more on that in the report for the data retainment issue.
Third, it turns out that the current code hides some problems of
MC-169913in some rare cases. For example, consider the following situation:This will already trigger part 1 of
MC-169913, but this is hidden by the current lightmap initialization code. With my proposed fix, this is no longer the case and these steps will produce some dark spots on the ground. However, the current code only helps in some edge cases and it's easy to find situations where it doesn't help. For example, modify the steps above by placing a block above the platform, e.g., /setblock 7 255 7 minecraft:glass. This will suffice to trigger the issue with the current vanilla code as well. So, I would say that this little regression is a fair compromise, given that it solves the rather severe performance issue ofMC-162253.Nevertheless, I have also implemented a fix for this part of
MC-169913, which can be found at https://github.com/PhiPro95/mc-fixes/tree/mc-169913. While fixing the whole issue described there requires a bit of work, as the multithreading code needs to be rewritten, the first part of the issue, which is what we observe here and which is also the most common and severe part of the issue, can be fixed rather easily. As explained there, we must simply reorder operations on the main thread as follows:and the lighting engine must move the lightmap removal to the POST_UPDATE phase, whereas lightmap creation needs to stay in PRE_UPDATE. However, there is a small problem with this, namely that earlier removals of lightmaps can override later creations, as they are carried out in a later stage.
This is solved by my proposed solution for the complete problem by cancelling operations properly, but this requires the rewrite of the multithreading code. Fortunately, we can easily work around this issue by scheduling an additional lightmap creation into the POST_UPDATE phase. This will eliminate any overrides and make sure that the final configuration of lightmaps is now indeed correct. However, as noted in the bugreport, this will not eliminate the problem of lightmaps being removed and recreated, causing data loss. But this situation shouldn't be that common in practice.
In summary, this easy solution for part 1 of
MC-169913greatly improves the situation over the current vanilla code and eliminates any regression introduced by eliminating code that currently shadows these issues.I hope these inputs assist you in implementing a fix for the issue.
Best,
PhiPro
As a proof of concept and as a temporary workaround for players, I have implemented my proposed solution for
MC-170010as a fabric mod. As explained above, this also solves this performance issue. The code can be found at https://github.com/PhiPro95/mc-fixes/tree/mc-170010. Note that this requires the fabric mod loader.Edit: Due to
MC-196614you also require https://github.com/PhiPro95/mc-fixes/tree/mc-196725 in order to avoid amplified versions of that bug.A complete bundle of all required mods can be downloaded here
Also note that it is important to have this mod installed on the client in order to fix the lag spikes. (However, in order to fix
MC-170010itself, it is mainly important to have it installed on the server).Best,
PhiPro
There is also a related issue for blocklight. As mentioned in the discussion of
MC-170010ProtoChunk.setBlockState(...) schedules light checks for chunks that have passed the features generation stage, suggesting that such block updates are indeed possible. Looking at the worldgen code also suggests that this is possible since the features generation stage has access to an 8 chunk radius.The problem is now that such a block change could place a light source and the scheduled light check would then propagate the light too early. In contrast to the original report about skylight, lightmaps do exist for the neighbors, since they will be created due to the placed blocklight source (well, not quite due to the second issue below). However, the lighting engine treats chunks before the features stage as opaque, so the light propagation can get stuck at the chunk boundary nevertheless. So the end-result is similar to the original report, namely that light propagation can get stuck if carried out too early, i.e., before the neighbors are ready, altough the underlying cause is slightly different in this case.
Note that this version for blocklight cannot be as easily visualised by relighting the world using erase cached data as it relies on blocks being placed between the features and light generation stage.
The fix for this is again quite easy. Just let the lighting engine check if a chunk already has its initial lighting done, and otherwise ignore the luminance of a block for light calculations. This way, blocks don't emit light by themselves before the initial lighting and are properly turned on during initial lighting, similar to my proposed solution for initial skylight in
MC-170010.Note that you cannot simply remove the light checks that cause the early propagation. Initial lighting spreads light too all neighbors, which are only guaranteed to be in pre_light stage at that point. Any block changes thereafter need to schedule light checks as otherwise you can have missing or ghost-contributions to those neighbors. When later on those neighbors are Initially lighted they will not be completely relighted but only add their own blocklight-sources. So, you need any lighting information to stay consistent after the pre_light stage and hence you need these light checks.
Also note that you should not fix this bug by not treating chunks before the features stage as opaque. Leaving in the early propagation of blocklight would then cause light propagations into chunks before the features stage and hence the above discussion would also apply to those, so you would need to schedule light checks before the features stage as well. However, for performance reasons this is not desired.
There is another issue related to block changes after the features stage. ProtoChunk only schedules light checks upon block changes (for chunks after the features stage). but it does not handle lightmap creation and removal. Concretely, it misses the calls to LightingView.updateSectionStatus(BlockPos pos, boolean status) that are present in WorldChunk and World. In view of
MC-169913lightmap creation schould be scheduled before the block change and lightmap removal should be scheduled after the light checks.Let me know if I should post these as separate bug reports.
I found another small bug impacting initial skylighting. I post it here since the report already kindof derailed into a discussion about initial lighting. Let me know if I should open a new bug report.
ChunkSkyLightProvider.recalculateLevel(long id, long excludedId, int maxLevel) contains some logic to calculate contributions from neighbors that don't have an associated lightmap, namely it looks up the light value by inherting it from the next lightmap above, as usual.
The problem now occurs if there is no such lightmap, i.e., the neighbor block is directly exposed to skylight, in which case it uses p = ((SkyLightStorage)this.lightStorage).isLightEnabled( n) ? 0 : 15;.
However, 0 is the skylight value of the neighbor (note that light values are inverted inside the lighting engine) but p should contain the propagated value, as is the case when chunkNibbleArray4 != null. Hence this should not be 0 but 1, as this case only deals with horizontal propagations.
Or even better it should be this.getPropagatedLevel(m, id, ((SkyLightStorage)this.lightStorage).isLightEnabled( n) ? 0 : 15).
This issue basically only affects intial skylight, as it can only occur if the block in question is directly exposed to skylight, which means that it will be completely bright anyways. However, before initial lighting this conclusion does not hold true.
This bug then makes the lighting model depend on the empty region which cuases problems when the empty region changes, similar to issues discussed above.
This is invalid. Light updates are only calculated until y=-16. Blocks below that simply inherit the sky-light values from y=-16.
The reason that the behavior changes below y=-2048 is that the queried BlockPos is converted to a long internally, decreasing the range of valid values. Hence, below y=-2048 the queried value wraps around and internally queries some large y, which is correctly returned as 15.
@Sir Apetus You are experiencing some different issue in this case. Note that the lighting issue discussed here is not the only bug that can cause lag spikes when crossing chunk borders, but it seems to be a very common reason.
One other issue I know of is
MC-65587. I dont know of a good workaround for that, though, other than removing all player skulls and see if that helps (after creating a backup of the world).Just some random thought I had while implementing some other bugfixes. I put this here preemptively in case this point comes up in your discussions as well.
One could say that it sounds rather inefficient to first carry out all light updates before removing lightmaps, just to throw away the results immediately afterwards. Once we know that a subchunk and all its neighbors are empty, we know rather easily what the the lighting should look like in the end (at least for blocklight, so the result will be 0). So one could propose to schedule removal of the lightmap before scheduling the light updates, as is currently done in the vanilla code, hence optimizing away all the propagations for the lightmap that will be removed anyway. This might have even been the original idea behind the current vanilla code.
The main issue with this approach is that it not only eliminates all propagations inside the removed lightmap but also propagations to all neighbors, which might not be removed. Whenever you do such an optimization overriding light values, you need to inform all relevant neighbors about the changes. Concretely, with this approach you would need to propagate the new values to all neighbors (that still have a lightmap) whenever removing a lightmap. Even if this would not cause any actual propagation to take place, as in most cases you can expect that the ligthvalues don't actually change upon removing a lightmap, the lighting engine would still have to check a lot of blocks, just to come to the conclusion that there is nothing to do.
In practice, it should be very likely that the geometry of non-air blocks is connected and changes continously with time (whatever that means exactly). In this scenario, you would indeed only remove lightmaps after all their nearby blocks have been replaced with air some time ago, hence their light would be trivial. So in this case the issue discussed in this bug report does not occur and it does not matter whether we first remove blocks and carry out their light updates or first remove lightmaps. In particular, the hypothetical optimization considered above would not actually optimize anything, as the light was already trivial before the lightmap removal. However, since the lighting engine does not know this, it would still have to propagated potential changes to all neighbors. So this hypothetical optimization would in fact increase the workload in the expected case.
So in conclusion I would expect this hypothetical optimization to be actually more expensive in most practical cases, compared to the solution proposed in the report, due to the compensation needed for manually chaning light values.
One could then try to apply this optimization only to lightmaps that are sufficiently far away that also their neighbors will be removed, i.e., those which are 3 subchunks away from any non-air block. In this case, we would not need to propagate changes to the neighbors, since those would be removed as well (at least for blocklight; for skylight one would still need the far-reaching downwards propagation).
However, I don't think that this would be worth the effort, as it will only be applicable for very discontinous geometry transformations.
Also note that this discussion only concerns the first part of the bug report. The more difficult multithreading part of the issue is unaffected by this approach.
I have implemented my proposed solution at https://github.com/PhiPro95/mc-fixes/tree/mc-196725.
There is one important remark I want to put here. There are two places in the code that manually change light values: Initial lighting which sets the topmost sections up to the first non-empty section to a skylight value of 15 and unloading chunks which sets all light to 0. It is important that one first removes all pending light updates for the affected sections before changing any light value. Otherwise the light propagator will lose track of the pending updates and not remove them properly later on, leaving behind some orphaned updates causing major issues.
For the current vanilla code this simply means to individually for each section remove pending updates before changing the section (which is indeed done in the vanilla code), because light updates can only be present for sections that have an associated lightmap, so changing the lightmap of one section does not interfere with any other section that contains scheduled ligth updates. With my proposed solution this is no longer the case: Light updates can also be scheduled for sections without an associated ligthmap, as long as there are nearby blocks, in case of skylight. Changing light values in one section can now lead to a change of light values in other sections, by the way sections without lightmaps are handled, and these other sections may contain pending light updates.
Because of this one needs to be careful to first remove all affected pending light updates before changing any lightmaps or changing the skylight source values.
Also note that upon removing a trivial lightmap one must not remove pending light updates if there are still nearby blocks, i.e., as long as the skylight optimization is still not applicable for that section, as otherwise of course some light updates get lost. But conversely, if skylight optimization does become applicable after removing a trivial lightmap, i.e., there are no nearby blocks, then one should remove pending updates. Because in this case the code will no longer create a lightmap when the inherited light value would change by a light update above , opposed to the case for non-skylight optimizable sections. Hence we would have changes in light values that are not directlly set by the level propagator, causing aforementioned issues. So one needs to remove all pending updates once a section becomes skylight optimizable and prevent any further propagations into such a section (which is already done in the vanilla code).
Similarly, if the distance tracking from non-air blocks changes and a section gets far enough away from blocks to become eligible for skylight optimization, then pending light updates must be removed if and only if the section does not have an sssociated lightmap, i.e., if it indeed becomes skylight optimizable.
Info to everyone using my mod as temporary workaround: There is a (vanilla) bug, see
MC-196614, that caused basically all underground subchunks to become fully bright up to and including version mc-170010-v1.2.1. This issue is solved in https://github.com/PhiPro95/mc-fixes/tree/mc-196725.A complete bundle of all required mods, including the fix for this issue, can be found here.
In order to repair already affected chunks you need to erase cached data, see
MC-142134for further instructions.Sorry for the inconvenience.
Best,
PhiPro
@CalXee Yes, I am aware of this. I need to get in contact with her and see what we can work out.
On another note, please post any issues regarding my bugfix mod on the Github bugtracker, if possible, in order to keep the discussion here on-topic.
Looks like I somehow managed to overlook some important piece of code... The ChunkTaskPrioritySystem actually does wait for tasks to complete on the dedicated queues before scheduling new ones, instead of scheduling everything at once. So the issue is invalid. Sorry about that.
Nevertheless, I propose the solution explained in the report as a simpler, more efficient and less obscure implementation. So I leave the report open for now.
On a further note, the skylight optimization code currently queries in each step if the currenty processed neighbor position has an associated lightmap.
If the answer is negative, then the result won't even be cached by the currently employed cache in ChunkToNibbleArrayMap, resulting in 16 HashMap lookups per subchunk.
This can be optimized by first iterating over the subchunks and checking only once if the neighbor has an associated lightmap, and then iterating over the 16 blocks of the subchunk.
It turns out that the workaround for the easy part 1 of this issue that I described in this comment does not work. This is because Vanilla schedules light checks to the POST_UPDATE phase, so scheduling lightmap removal to the POST_UPDATE phase aswell does not guarantee that all pending light updates have been carried out before the lightmaps are removed. Also, light checks cannot be moved to the PRE_LIGHT phase as they would otherwise be carried out before the relevant lightmaps are created, which would again cause issues.
Fortunately, this part of the issue is automatically solved by my proposed solution for
MC-196725. Note that the argument given there at the end regarding the interaction of skylight optimization and unloaded chunks also shows that my proposed solution mitigates this easy part of the issue here.One important point in the argument was that non-optimizable subchunks are always locally correct. This does not hold in the current Vanilla code, precisely because of the issue discussed here, i.e., non-trivial lightmaps getting removed, hence changing values without informing neighbors. This is fixed by my proposed solution for
MC-196725which only removes lightmaps once they are trivial.Also note that it is not an issue that the distance tracking of non-empty subchunks gets updated before all pending light updates are carried out. The argument only requires that if in the end state there is a non-air block then the surrounding subchunks must not be optimizable. Hence it is sufficient to mark a subchunk as empty after all blocks have been removed, but possibly before the corresponding light updates have been fully processed. Which is what the Vanilla code currently does.
Hence no extra treatment for the easy part of this issue is required once implementing my solution for
MC-196725.In fact, the argument given in
MC-196725can be greatly simplified for what we need here, since we do not need to deal with unloaded chunks for this matter. Hence the region that was constructed in the argument is simply (-1 -1 -1)..(1 1 1) in this case, so the only interesting parts of the boundary are the top faces. And for those we didn't even use that the subchunks at y=-1 were empty, hence the more efficient specification of the skylight optimization, as briefly mentioned at the end ofMC-196725, would be sufficient here.There is another related issue regarding chunks before the light stage. Upon loading from disk, lightmaps are only loaded for chunks that were already lighted and discarded otherwise, although they are always saved to disk.
This basically erases any light propagations to chunks in pre_light stage when unloading them before the light stage and hence causes lighting glitches.
The Vanilla code currently uses this mechanism to erase cached data, which only removes the isLightOn field but not the lightmaps themselves.
I posted the second comment as MC-199966.
I would prefer to leave the first comment as is. It is more of a theoretical input and the glitches caused by it would be rather sporadic, so a reproducible setup is very hard to find. So I think for the moment it fits best here, as it falls under the general issue that pre_light (or features) and light stage aren't properly distinguished.
In case this report gets marked as fixed without fixing the comment, I will repost it as separate report.
The symptoms are exactly those described by
MC-162253, namely lag spikes when crossing chunk borders in the direction of worldspawn (or chunks loaded by any other means), together with large height differences near those chunks required to trigger the lighting issue. The boundaries where the lag spikes are observed should be render distance away from any chunks in the spawn area (that have sufficient height differences to their neighbors).You can try increasing the render distance and see if the problematic boundaries move further away from the spawn chunks. In that case this would most certainly be a duplicate of
MC-162253.Not sure if this is caused by
MC-224729. Can you please answer the following questions:Also a question from my side: Do you know if these chunks were previously generated and just didn't reload? Or were they freshly explored?
Whoops, ofc 1.16.5 is not affected. Guess all my other reports affected older versions aswell, so I clicked it kinda automatically.
The implemented fix for this issue did not revert to the 1.16 code, but instead just dropped the abortion part alltogether
Unfortunately, this breaks property 2 above. (Guess I was a bit too sloppy when sketching the argument why it holds for 1.16 and I honestly didn't spot the issue either when briefly looking at the implemented solution).
The problem is that there are still some mechanisms that can indirectly abort the Future s, namely ThreadedAnvilChunkStorage.getRegion(...) which returns an UNLOADED_CHUNK if any of the requested chunks has ticket level below EMPTY (note that the chunk does not need to be completely unloaded for the check to fail, as the code only checks the live chunk holder map, but does not attempt to revoke any chunk) or ThreadedAnvilChunkStorage.convertToFullChunk(...) which does abort if the chunk has ticket level below FULL.
This can lead to generation Future s that are still pending (for example when looking at it for creating the ticking-Future) but are already indirectly aborted through their dependencies (because the ticket level dropped too low at some point in the past), and this information might not yet have been propagated, e.g., because it needs to pass through other threads. This can hence break property 2.
Note that in 1.16 this does not happen since Future s are directly aborted when the ticket level drops too low, so one can conclude from observing a still pending generation Future that the ticket level did not drop (below the respective status) since creation of the Future, and then conclude from this that the Future is also not indirectly aborted. In 21w18a this argument does not work since Future s are not directly aborted and hence do not allow any conclusion about past ticket levels.
As a consequence, symptom 1 (Ticking-Future s might never complete) can still occur (whereas the other 2 symptoms were caused by failure of property 1, which is indeed fixed). This was indeed reported in
MC-224986.In the following I will present some very artificial steps to reproduce the issue deterministically. (Unfortunately, these steps do have some chance for failure since there is no way to prohibit the server thread from looking at chunks around the player (even with spectator mode), so that these steps will unintentionally pause the server thread even when only explicitly pausing the worldgen thread. This leads to some timing issue which can cause these steps to fail. Nevertheless, they worked quite reliably for me.)
After generating the world, place a breakpoint in the requiredStatus.runGenerationTask(...) line. This will eventually pause the worldgen thread and prohibit chunks from finishing generation.
This problem shows that it is necessary to keep track of the history of the ticket level, either by directly storing that information in the Future by aborting it like 1.16 does. Or by using some other external tracking which then needs to extend still pending Future s with some retry mechanism upon increasing ticket level (or something in that direction)
Best,
PhiPro
Thanks. It's consistent with my explanation then (which only really works for newly generated chunks).
See this comment on MC-224893.
All these instances seem to be near worldspawn, so this might be caused by
MC-224729(see section "1. Spawn chunks"). Can you please answer the following questions to check this claim:@Rik Lubking I added a third example to the original report which should explain your case. As I said, the list is certianly not exhaustive.
I guess you did your tests with a render distance of 2? Just out of interest, does it also happen on higher render distances, e.g., 20? (away from spawn to exclude problem 1)
Well, removal of the PLAYER tickets involves some threading, so it's likely that they actually don't get removed entirely before saving (so this effect is probably slightly random). So I would guess that this is actually caused by the PLAYER tickets that are not cleared before saving. Anyway, the general issue stays the same: chunks not getting saved due to not being unloaded because of some tickets.
I'll maybe add this case to the original report later (it's probably the most dominant case).
This issue actually still exists in 1.17-pre3 to some extent. It took me quite a while to find out what actually changed in pre3 at all, but as far as I can tell, the relevant change is that the getRegion(...) calls are now done eagerly instead of lazily, or more precisely ThreadedAnvilChunkStorage.getChunk(...) executes immediately instead of waiting for the previous worldgen stage to finish. As a consequence, they should indeed no longer be able to produce UNLOADED_CHUNK results by construction of the ticket system, hence solving the example in my previous comment.
As a side remark: This change might have some negative impact on performance, as it gets increasingly difficult to abort already scheduled worldgen steps. Lazy evaluation of the getRegion(...) calls (or rather of the getChunk(...) code) was at least able to abort all but the lowest pending worldgen stages (which still misses a bunch of steps which could possibly be aborted (MC-183841)), whereas with eager evaluation no stages at all are aborted. I haven't actually looked into actual numbers to tell whether or not this is relevant. Might be a good idea to look investigate.
Anyway, back to the original issue. There is still one place that can produce UNLOADED_CHUNK results when generating new chunks, namely ThreadedAnvilChunkStorage.convertToFullChunk(...). The following steps shows how this can be employed to prevent a single chunk from loading to the client:
This will now trigger the breakpoint on the worldgen thread. The returned completableFuture corresponds to the FULL stage due to the breakpoint condition. Since the server thread was not paused, the corresponding call to convertToFullChunk(...) could already finish and produce an UNLOADED_CHUNK result, as the player was already teleported away, so the ticket level of the chunk is below FULL. This can be verified by inspecting the returned completableFuture. (This step might fail due to the POST_TELEPORT ticket, which can keep the ticket level of some of the chunks at FULL. In this cause simply continue until hitting a breakpoint where the completableFuture contains an UNLOADED_CHUNK).
The UNLOADED_CHUNK result is not propagated since it is still stuck in the paused worldgen thread. Teleporting back to the area creates the ticking-Future which then depends on the still pending Future for the FULL stage, which will hence eventually produce an UNLOADED_CHUNK. As in the previous examples, this will prevent this chunk from loading to the client.
Best,
PhiPro
@Rik Lubking There is already an easy hackfix by SuperCoder79, simply removing the isAccessible filters: https://github.com/SuperCoder7979/chunksavingfix-fabric
Note though that a proper fix should redesign some aspects of the saving code, in particular because there are some other race conditions (caused by removing chunks from the chunkHolder map before saving them, so that the save(...) method does not necessarily wait for those chunks while they are neither in the chunkHolder map nor in the unloadQueue) which aren't covered by this fix.
Nevertheless, such a redesign is probably out of scope for mods and should be done by Mojang. The simple fix is enough for the deterministic part of the issue and should be good enough as hackfix for the moment.