Failing HealthChecks on Regional Google Cloud TCP Load Balancer

Often Google are right, but sometimes not.

Recently I was looking at why GCP TCP Load Balancer healthchecks were failing, despite I have configured strictly my local OS firewall to allow the documented health check IP ranges :

(https://cloud.google.com/compute/docs/load-balancing/network/ – Section Firewall rules and Network load balancing  – 209.85.152.0/22209.85.204.0/22, and 35.191.0.0/16 at the time of writing).

Network Services–>Load Balancing–>Load Balancers tab was stating my instances were unhealthy:

Instance <instance name> is unhealthy for <LBIP>

It turned out Google is sending healthcheck from their metadata IP address instead  – metadata.google.internal (169.254.169.254)

Allowing this in the OS firewall made the HCs succeed!