[Action Required] JVM Flags Update for Java 17 Services Running on ECS Fargate

What is it about?

CPU detection by the JVM has changed in newer versions (11.0.17, 17.0.5, 21). This led to the resource-hungry G1 GC (even hungrier for Java 17+) to be used on small Fargate tasks. To respond and prevent these unexpected changes in the future, we will make our JVM flags more explicit.

Who is this for?

Anyone using AWS ECS to run Java 17 services, especially if you recently have issues related to JVM GC.

Why is this needed?

The setting below tells JVM the correct number of CPUs, which it will use to determine the size of internal thread pools (one of the use cases is GC threads) and to return Runtime.getRuntime().availableProcessors().

Without this setting, we’ve seen memory related problems in some Java ECS services, especially those with 1 vCPU, due to the JVM detect 2 CPUs (we suspect it’s the CPUs of the ECS Fargate host). JVM will then uses G1GC which has high CPU & memory usage,

What do you need to do?

For each of your Java services, decide the JVM parameters. Here are our latest standard JVM flags for ECS Fargate.

CPU count & GC algorithm, based on Fargate task size:

≤ 1vCPU: -XX:ActiveProcessorCount=1 -XX:+UseSerialGC

2vCPU ≤ 5GB memory:-XX:ActiveProcessorCount=2 -XX:+UseParallelGC

≥ 4vCPU, or 2vCPU 6+ GB: -XX:ActiveProcessorCount=<number of vCPUs> -XX:+UseG1GC

Heap Size, based on Fargate task memory

1-3GB: -XX:InitialRAMPercentage=66 -XX:MaxRAMPercentage=66
≥ 4GB: reduce by ~1,500 MB, e.g. if task memory is 8GB: -Xms6500m -Xmx6500m

Then, set the flags in the ECS Task Definition:

Before the change

"containerDefinitions": [
        {
            "name": "app",
            "cpu": <C1>,
            "memory": <M1>,
            "environment": [
                ...
                {
                    "name": "JAVA_TOOL_OPTIONS",
                    "value": "<old heap & gc flags> <jmx flags>"
                }
            ]
            ...
        },
        {
            "name": "datadog",
            "cpu": <C2>,
            "memory": <M2>,
            ...
        },
        {
            "name": "other_sidecar_container_if_any",
            "cpu": <C3>,
            "memory": <M3>,
            ...
        }
  ]

After the change

"containerDefinitions": [
        {
            "name": "app",
            
            // remove the "cpu" and "memory" parameters from all containers
            
            "environment": [
                ...
                {
                    "name": "JAVA_TOOL_OPTIONS",
                    "value": "<new cpu, gc, and heap size flags> <jmx flags>"
                }
            ]
            ...
        },
        {
            "name": "datadog",

            // prevent sidecar containers from stopping the application container
            "essential": false,
            ...
        },
        {
            "name": "other_sidecar_container_if_any",
            "essential": false,
            ...
        }
  ]

Update bei gradle java plugin (which will add -XX:+AlwaysPreTouch -Xlog:gc:stdout:tags -XshowSettings:system as default JVM flags)

v8.1.5 for non-springboot 3 Java 17 repositories, v8.2.1 for springboot 3 repositories
In your service’s build.gradle file, remove the now-unused dockerConfig.jvmArgs

Then, verify the used GC and detected CPU & memory from the service’s log, e.g.:

Used GC (when you set -XX:+UseParallelGC)

Detected CPUs (when you set -XX:ActiveProcessorCount=1)

Detected memory (should be the same as Fargate task memory)

Questions/Concerns?

Ask @Sal / @fajrin in #backend slack channel