Assessing Changes to JVM Options on an Existing System
How do you tell if an updated option is better, worse, or neutral to your existing application?
You have a couple options. You can write JMH tests to test the change at the micro-level or you can use metrics to test it at the macro-level.
For large scale changes, like changing your garbage collector, I suggest focusing on the macro-level. After updating the JVM option, run load testing (such as Locust tests) on both two versions of your application: with the change and without the change. Assuming you have existing metrics (such as Prometheus metrics) available on your application, compare metrics to see if you see the desired change.
For smaller scale changes, such as changing the MaxInlineLevel, I suggest starting with a micro-level focus and then assuming you see the desired results, also doing macro-level assertions.
For MaxInlineLevel, I used a JMH test that uses streams and lambdas. The reason for this is due to the level of abstraction that are involved here that affects the compiler’s ability to optimize. See this article by the GraalVM team on their focus on the difference in Stream performance between HotSpot and Graal.
On my system (JDK8, MacOS Mojave, 2.2 GHz Intel Core i7, 32 GB 2400 MHz DDR4), comparing MaxInlineLevel=9 and MaxInlineLevel=15
Benchmark Threads Samples Score Error (99.9%) Unit
mapReduce 1 6 47.178451 3.712375 ops/s mapReduce 1 6 47.267885 3.926508 ops/sThere isn’t a distinct benefit so in real life I would likely stop here.
For the sake of this article, I ran load testing and also compared the metrics.
These comparisons showed the same result. This change was a neutral change to the environment. As a result, I did not move forward with making this change on the system.
