google-cloud-platform - GCP IAP Authentication won't get service account credentials unless using environment var pointing at local json credentials

Question

I tried Google's Java IAP authentication example for service accounts: https://cloud.google.com/iap/docs/authentication-howto#iap_make_request-java

This works well when there is a local service account json credentials file which is mentioned in the env var GOOGLE_APPLICATION_CREDENTIALS=<path-to-file>.

In another documentation page https://cloud.google.com/docs/authentication/production#auth-cloud-implicit-java it says the following:

If the environment variable isn't set, ADC uses the default service account that Compute Engine, Kubernetes Engine, App Engine, and Cloud Functions provide, for applications that run on those services.

If I don't provide the env var though then the auth library in the example seems to generate a different type of credentials than ServiceAccountCredentials, which leads to an error being thrown. I logged the credentials that are created and they are resolved to ComputeEngineCredentials which leads to this part of Google's example throwing the error:

if (credentials == null || !(credentials instanceof ServiceAccountCredentials)) {
            throw new Exception("Google credentials : service accounts credentials expected");
        }

In the scenarios I am working on I won't always have the option of uploading the json credentials file and running with that env variable and at any rate I would expect that a compute engine can identify as the service account in its configuration. Is there anything that I am missing?