Lee Ward
- leeward
- leeward
- Europe/Stockholm
- Yes
- No
MC-154629
![[Mojang] slicedlime](../../../data/assets/gravatar/slicedlime.jpeg){kind=small}
Lee Ward Being snarky does not help the bug get fixed. If you're not going to contribute any new information, please refrain from commenting.
Lee WardLee Ward Being snarky does not help the bug get fixed. If you're not going to contribute any new information, please refrain from commenting.
Of course it is. Have you heard of some attempt to fix the issue? It's not likely to just get old and die on it's own
To reproduce, just start the server; No extra flags needed. No need to even play the game. Just wait until the world initializes, then check the CPU load. You can aggravate the issue if you choose to reproduce within a virtual machine. If you do that, though, then check the host CPU load, not the guest.
Did you read the, multiple, other bug reports that already give exact reproducers?
Agreed. Without the snark, then...
Yes. The problem is still present. Verified by testing 1.14 pre-release 5.
To reproduce:
1) Install the server.jar into an empty directory
2) Change to the directory referred to in step (1) and start the server, per instructions. I.e. "java -Xmx1024M -Xms1024M -jar server.jar nogui"
3) Allow it to error and exit, due to not having accepted the EULA.
4) Edit the EULA file, setting the accepted parameter to true.
5) Start the server again, just as in step (2), above.
6) Monitor ./logs/latest.log, waiting for the "Done" message to appear.
7) Acquire a list of the relevant running threads. On my machine, this looks like:
ps -T -p 707
PID SPID TTY TIME CMD
707 707 pts/0 00:00:00 java
707 708 pts/0 00:00:03 java
707 709 pts/0 00:00:04 VM Thread
707 710 pts/0 00:00:00 Reference Handl
707 711 pts/0 00:00:00 Finalizer
707 712 pts/0 00:00:00 Signal Dispatch
707 713 pts/0 00:00:21 C2 CompilerThre
707 714 pts/0 00:00:02 C1 CompilerThre
707 715 pts/0 00:00:00 Sweeper thread
707 716 pts/0 00:00:00 Common-Cleaner
707 717 pts/0 00:00:00 Service Thread
707 718 pts/0 00:00:00 RMI TCP Accept-
707 719 pts/0 00:00:00 RMI TCP Accept-
707 720 pts/0 00:00:00 RMI TCP Accept-
707 721 pts/0 00:00:00 VM Periodic Tas
707 723 pts/0 00:00:29 Server-Worker-1
707 724 pts/0 00:00:00 Snooper Timer
707 725 pts/0 00:00:00 Server Infinisl
707 726 pts/0 00:03:21 Server thread
707 727 pts/0 00:00:00 Server console
707 728 pts/0 00:00:00 ObjectCleanerTh
707 729 pts/0 00:00:00 Netty Epoll Ser
707 730 pts/0 00:00:00 Server Watchdog
707 736 pts/0 00:00:00 RMI Scheduler(0
707 745 pts/0 00:00:00 JMX server conn
707 794 pts/0 00:00:02 RMI TCP Connect
707 796 pts/0 00:00:02 RMI TCP Connect
8) Find the thread in that list that is using the most CPU; For instance, that thread has ID 726 in the above listing.
9) Trace that process. You will get something like:
...
{tv_sec=0, tv_nsec=708}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=689}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=687}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=701}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=703}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=707}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=699}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=704}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=697}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
{tv_sec=0, tv_nsec=714}futex(0x7f16b0b7a2e8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0,
) = -1 ETIMEDOUT (Connection timed out)
...
These messages repeat, ad nauseum, and without pause.
I also started the server with profiling enabled and have a screenshot of sampling results. I do not see a way to attach that screen shot to this comment, though? The screenshot indicates that two items are collecting all of the runtime. The first is "Server console handler". Under that, it shows that ug$2.run()/java.io.BufferedReader.ReadLine() is very active. The other is "Netty epoll server IO #0" and under that, the active chain is:
java.lang.Thread.run ()
io.netty.util.concurrent.SingleThreadEventExecutor$5.run ()
io.netty.channel.epoll.EpollEventLoop.run ()
io.netty.channel.epoll.EpollEventLoop.epollWait ()
io.netty.channel.epoll.Native.epollWait ()
io.netty.channel.epoll.Native.epollWait0[native] ()
Finally, /proc/interrupts indicates that timer interrupts are increasing at quite a rate:
root@ontest:/home/lee# while true
> do
> grep -i timer /proc/interrupts
> sleep 1
> done
0: 10 IO-APIC 2-edge timer
LOC: 86102299 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86120146 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86137633 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86155141 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86172589 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86190086 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86207560 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86225069 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86242565 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86260068 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86277560 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86295081 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86312563 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
0: 10 IO-APIC 2-edge timer
LOC: 86330096 Local timer interrupts
HVS: 0 Hyper-V stimer0 interrupts
^C
What more can I add?
I figured out how to attach the screenshot. It is Screenshot (1).png in the attached files section.
I'm sad now
I don't think I'm misinterpreting anything. I provided no interpretation. Only information. After being, rightfully, accused of being "snarky" I deliberately and pedantically shied away from any interpretation, conjecture, supposition, or opinion.
I agree, the profiling information I provided seems virtually useless. It was a piece of information I developed in the course, so I included it. Nothing more, nothing less.
The machine is a server, with an unusual amount of disk. At a high level, though, it is:
CPU: Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
Motherboard: ASRock Rack C226 WS+
32GB of ECC-enabled memory
I think that is what you are really asking for but you might want an exhaustive list. If you truly do want an exhaustive inventory of the hardware, do let me know and I can provide it.
Software, on the VM host, versions:
The basic package is an unmodified Debian 9.9 release with all updates applied. Specifically, the kernel is the as-from-debian Linux. It's not been rebooted in awhile so the kernel may be dated. The uname command reports (and reported when the previous info was developed) that the kernel is "tochtli 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux". The package for that kernel is ii linux-image-4.9.0-9-amd64 4.9.168-1+deb9u3 amd64 Linux 4.9 for 64-bit PCs
I am using qemu to host the guest on this machine. dpkg-query reports that the related qemu packages are:
ii ipxe-qemu 1.0.0+git-20161027.b991c67-1 all PXE boot firmware - ROM images for qemu
ii qemu-kvm 1:2.8+dfsg-6+deb9u7 amd64 QEMU Full virtualization on x86 hardware
ii qemu-system-common 1:2.8+dfsg-6+deb9u7 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 1:2.8+dfsg-6+deb9u7 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.8+dfsg-6+deb9u7 amd64 QEMU utilities
The hypervisor is managed using libvirt. Those packages are:
ii libvirt-clients 3.0.0-4+deb9u4 amd64 Programs for the libvirt library
ii libvirt-daemon 3.0.0-4+deb9u4 amd64 Virtualization daemon
ii libvirt-daemon-system 3.0.0-4+deb9u4 amd64 Libvirt daemon configuration files
ii libvirt-glib-1.0-0:amd64 1.0.0-1 amd64 libvirt GLib and GObject mapping library
ii libvirt0 3.0.0-4+deb9u4 amd64 library for interfacing with different virtualization systems
ii virt-manager 1:1.4.0-5 all desktop application for managing virtual machines
ii virt-viewer 5.0-1 amd64 Displaying the graphical console of a virtual machine
ii virtinst 1:1.4.0-5 all Programs to create and clone virtual machines
The following XML code is the configuration for the test machine:
<domain type='kvm'>
<name>ontest</name>
<uuid>cb423de8-b80d-43b1-b92d-fb7da6f16cc1</uuid>
<title>Lee Test</title>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>10485760</currentMemory>
<vcpu placement='static' current='1'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.8'>hvm</type>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/u1/vm-storage-pools/default/ontest.qcow2'/>
<target dev='vda' bus='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hda' bus='ide'/>
<readonly/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:6e:d6:53'/>
<source bridge='on0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
<image compression='off'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</memballoon>
</devices>
</domain>
That guest VM is as it was when I sent the information you are asking about. It's not been updated so I believe all info below reflects the state at that time.
The reported guest CPU info is:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 60
model name : Intel Core Processor (Haswell, no TSX, IBRS)
stepping : 1
microcode : 0x1
cpu MHz : 3499.996
cache size : 16384 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds
bogomips : 6999.99
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
A "free -h" reports
total used free shared buff/cache available
Mem: 9.8Gi 79Mi 9.6Gi 8.0Mi 113Mi 9.5Gi
Swap: 0B 0B 0B
The kernel reports, using "uname -a", that it is:
Linux ontest 4.19.0-5-amd64 #1 SMP Debian 4.19.37-3 (2019-05-15) x86_64 GNU/Linux
The package for that kernel is: ii linux-image-4.19.0-5-amd64 4.19.37-3 amd64 Linux 4.19 for 64-bit PCs (signed)
I think the only relevant software would be the JVM packages, no? They are:
ii ca-certificates-java 20190405 all Common CA certificates (JKS keystore)
ii default-jre-headless 2:1.11-71 amd64 Standard Java or Java compatible Runtime (headless)
ii java-common 0.71 all Base package for Java runtimes
ii jdk-11.0.3 11.0.3-1 amd64 Java Platform Standard Edition Development Kit
ii openjdk-11-jre-headless:amd64 11.0.3+1-1 amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
I'd really like to help you avoid wasting all your time trying to exactly duplicate my environment. I believe (have verified in fact) that almost all of the above is, just like the profiling information I sent along, virtually irrelevant. Here is a much more easily accomplished reproducer:
1) Install Debian Linux 9.9 on bare metal.
2) Install the default VM packages. No need for xen, just install kvm+libvirt+qemu and gnome.
3) Login as root, using the gnome desktop. Then, use the gnome VM tool to create a VM. Just take the defaults. Maybe increase the default amount of memory, so there is plenty?
4) Start the newly created guest and install Debian 9.9 (or Debian 10, now) on the guest.
5) Install the jvm and jdk.
6) Post-install, bring any 1.14 server.jar to the guest and start it per the instructions on the Mojang download page.
7) Start minecraft and let it generate the initial world. Then, give it an extra minute or so to quiesce.
On the guest, the CPU load will fall to, or near, 0. On the host, the load will be 100%, minimum.
Now, I'll risk a little interpretation....
At least, the 1.14 minecraft server jars employ a timed mutex that is far, far worse than a "bit short". Look at the strace I sent again. Here is one example of the relevant system call, taken on the guest:
futex(0x7f16b0b7a338, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=699}) = -1 ETIMEDOUT (Connection timed out)
Notice the timeout value? That futex call request has a timeout of 699 nanoseconds; Nano, as in billionths of a second. In your comment above, you noted that the minecraft server clock needs a 20Hz clock. This timeout, though, is many orders of magnitude shorter than that; It's asking for service 69.9 million times a second. My math says that is 3,495,000 times more frequent than the "20 ticks per second" you were discussing.
The story gets worse, though, too. The Linux hypervisor is forced to emulate high-precision timer interrupts. These timers are implemented, both on the bare metal and on the guest, by setting an asynchronous timer to interrupt the operating system once the requested period has elapsed. On bare metal, it can probably be done at this rate but it is an outrageous number of interrupts to handle in any case. For a VM, though, it's practically impossible on today's hardware and, by the way, explains the observed behavior. The guest is requesting a virtual interrupt on the order of 50 - 70 million times a second. Linux hypervisors must virtualize these interrupts and, so, the timer request itself is being trapped or faulted in order for the hypervisor to accomplish that. I.e., all the CPU load is attributed to the hypervisor, and, since, the guest isn't even running while that is going on it sees no load at all.
Then, worse yet. The resolution of the high-precision timers under hypervisors on Linux is on the order of 10's of microseconds. However, here, the requested timeout is at least one order of magnitude smaller than that. Put another way, the timeout is attempting to trigger on clock noise, not actual values.
Given the rate at which /proc/interrupts reports the timer interrupts growing on the guest for what I reported above, I would estimate that the hypervisor on my machine only capable of emulating somewhere around 17,847 timer interrupts per second.
Which implies, to me, that a timeout value less than about 56 microseconds is, at best, pointless. Even that, value, though, would obviously only reproduce the 100% host load we are currently observing.
Moving beyond interpretation to complete conjecture and supposition, now. The minecraft server seems to have an asynchronous timer that is egregiously parameterized. I suspect relating to the networking code. Did someone, somewhere, somewhen, perhaps believe that a very short timeout would improve the network latency observed by the client? Good thought. Perhaps, though, the very low latency service desire is better thought of in the milliseconds time-frame as opposed to the sub-microsecond time-frame, though?
More specifically... If the server only updates 20 times per second, in other words a 50 millisecond period, then maybe we could agree that 50ms is the Nyquist frequency for any sampling? If we could agree on that, then, isn't a 25 millisecond timeout fast enough? Anything shorter than that should have no observable effect, right?
Whatever the reason though, clearly that sub-microsecond mutex call needs some love.
By the way, here is a small C program that reproduces the symptoms. It just concretely demonstrates that the Linux host-side of the virtualized machine is in no way capable of handling the requested timeout value. Just compile and run on the guest. Note that the guest load is 0, while one core on the host is completely consumed.
#include <stdint.h>
#include <time.h>
#include <assert.h>
#include <errno.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <linux/futex.h>
int futex(uint32_t *uaddr, int futex_op, int val, const struct timespec *timeout, int *uaddr2, int val3);
int
{ uint32_t u; struct timespec t = \{ 0, 780 }main()
;
int i;
u = 0;
for (;
(void )futex(&u, FUTEX_WAIT|FUTEX_PRIVATE_FLAG, u, &t, &u, 0);
// Not reached.
return 0;
}
int
futex(uint32_t *uaddr, int futex_op, int val, const struct timespec *timeout, int *uaddr2, int val3)
{
return syscall(SYS_futex, uaddr, futex_op, val, timeout, uaddr2, val3);
}
Sorry for the ugly formatting...
Apologies. I just read my latest through one more time. Two errors...
1) When describing the guest software version I forgot to note that the base install was Debian 10. Yes, 9.9 on the host, still, but 10 on the guest.
2) When discussing a more appropriate timeout value, I misspoke. 50 ms is the, assumed, rate and not the Nyquist frequency. I suggested the Nyquist frequency as a minimum, permissible, timeout. I really do know the definition for Nyquist frequency, in spite of my idiotic usage. Sorry.
Well, I don't know of course. I don't have access to the server code. The server does not appear to be stalling at start, though. World generation seems to proceed smoothly in the reproducer, above. Restart of an existing, production, 1.14.3, server also seems to proceed smoothly. For that same, existing, server the symptoms on the virtual guest include:
1) An, infrequent, number of can't-keep-up-messages when the server is completely idle and a veritable barrage of same when there are players present.
2) On the production guest, I configure the CPU as pass-through and include the TSC clock. When that configuration is used the guest kernel reports that the TSC is behaving badly, enough to be unusable, and switches to the HPET timer-counter.
3) Persistent lag and rubber-banding behaviors during game play. Network does not seem to matter. It happens to players on long-haul links and on the (10 Gb/s wired) LAN.
On the host side, when the server is idle CPU load hovers around 100 to 103 percent. It goes up only a small amount when players are present.
Most of those symptoms are displayed by the reproducer I gave. I tried to simplify the reproducer by using the generic qemu CPU (Haswell) and I didn't enable the TSC. Also, I only tried game-play from the wired LAN.
Do those symptoms sound like they might be related to this JVM availableProcessors() bug you are talking about? If so, do you have a workaround you would care to share?
I did that. The production VM is configured with both clocks. The kernel, on the guest and very quickly after the minecraft server starts, reports that the TSC is unusable and switches to HPET. The test VM is configured HPET only. Both configurations reliably reproduce the problems.
No. It is not fixed, nor mitigated at all. Honestly, I just got tired of going around and around with the devs on the issue. If I'm remembering correctly, the final substantial activity from them was when they tried to reproduce and couldn't. I suspect they tried to reproduce using a Linux guest on a Windows host, which will not reproduce the issue.
Really sorry. I just gave up. The overhead I incurred to test plus the lack of meaningful effort on the part of the Mojang developers just left me feeling super frustrated.
As an aside, my efforts lead me to believe either the server is not using the Java Netty package in a nice way or that there is a bug in the Netty package. I checked the Netty bug reports and, yes, others are having the same problems with it. Still... See https://github.com/netty/netty/issues/5896 In that case, the reporting individual ditched it and wrote their own. Doesn't bode well for a fix in Minecraft then
Just to clarify, the host was unchanged, still Ubuntu 18.04?
Krister's workaround, above, verified. Tested with 64 bit Debian 10.8 host and guest running stock kernels.
Guest usage fell from 100% to ~24%
Power usage on the host server decreased from ~113W to ~87W
Thank you very much, Krister!