I came afoul of this problem today, thought i'd quickly write about it before I forget. Maybe it helps someone else out there one day.
In an existing istio installation I was tasked with creating a second ingress gateway that would be using an internal azure load balancer - this bit isn't important, just giving some background.
I prepared my istio operator YAML descriptor by adding a new ingress gateway item under components.ingressGateways and specified a different namespace for it to go in.
When running istioctl install -f myfile.yaml everything appears to go OK at first, but the istio ingress gateway pod in the namespace i had created for it doesn't ever appear to start successfully - health checks fail, istioctl times out and undoes any changes.
The reason for this is that the new istio ingress gateway pod is trying to connect to istiod and it's falling back on the default of istiod.istio-system.svc.cluster.local which is fine if you have not performed any canary upgrades of istio where istiod gets a different name, like istiod-1-15-0 for example. So the correct hostname for istio is istiod-1-15-0.istio-system.svc.cluster.local, and istio-proxy containers get this hostname from a config map that contains a property called discoveryAddress, which in istio-system namespace is present and correct, but in a NEW namespace, is totally missing. Therefore in order for all of this to work, one must create this config map in the new namespace, run istioctl install, wait for it to complete and then re-create the config map again because istioctl will prune anything it doesn't think should be there, thereby deleting the config map and causing problems if ever your ingress controller pods restart for whatever reason - say during a cluster upgrade.
The config map i'm referring to is called istio (in my case istio-1-15-0 because i had installed istio side by side with a previous version of istio using the canary upgrade path)
apiVersion: v1
data:
mesh: |-
defaultConfig:
discoveryAddress: istiod-1-15-0.istio-system.svc:15012
proxyMetadata: {}
tracing:
zipkin:
address: zipkin.istio-system:9411
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
meshNetworks: 'networks: {}'
kind: ConfigMap
metadata:
annotations:
labels:
install.operator.istio.io/owning-resource: unknown
install.operator.istio.io/owning-resource-namespace: istio-system
istio.io/rev: 1-15-0
operator.istio.io/component: Pilot
operator.istio.io/managed: Reconcile
operator.istio.io/version: 1.15.0
release: istio
name: istio-1-15-0
Phew - anyway, them's my notes on this. A bit rushed.
Until next time.
Comments