It is quite common when your project use Kubernetes, your ingress controller of choice is Ingress Nginx. It’s well known, cloud agnostic, huge support from community.

The story begin

Let’s say you have your shiny new project ready to run on top of kubernetes with Ingress Nginx. You push your image to registry, and configure all lovely yaml from Deployment, Service, and Ingress, one kubectl command later, boom it runs and accessible perfectly.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-cloudflare-dns # my go to TLS management tool
  name: echo-server
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: echo.local.dirathea.com
    http:
      paths:
      - backend:
          service:
            name: echo-server
            port:
              number: 80
        path: /awesome/tools
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - echo.local.dirathea.com
    secretName: echo-k8s-orb-local-tls

several features later, you now have another path to expose. Because this new feature going to have additional configuration on top of it, you create different ingress object.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-cloudflare-dns # my go to TLS management tool
  name: echo-server-v1
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: echo.local.dirathea.com
    http:
      paths:
      - backend:
          service:
            name: echo-server
            port:
              number: 80
        path: /new/awesome/feature
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - echo.local.dirathea.com
    secretName: echo-k8s-orb-local-tls

Works fine too, no problem. All path are accessible.

To make it more professional, you also had a plan to add custom 404 page. To do that, you enable the defaultBackend feature from Ingress Nginx Helm Chart, that you will update the image later.

controller:
  allowSnippetAnnotations: true # for new feature ingress config
defaultBackend:
  enabled: true # custom 404

Feels everything goes as expected, next you want to redirect your customer from the old path to the new path. Easy peasy

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: letsencrypt-cloudflare-dns # my go to TLS management tool
+   nginx.ingress.kubernetes.io/permanent-redirect: /new/awesome/feature
  name: awesome-path-1
  namespace: default
spec:
...

Once again, all goes well

Custom 404 backend

Now it’s time to develop the 404 page. You push the image to registry, update the defaultBackend image to yours, and ready to test. When you hit the arbitrary url,

what’s just happened? All arbitrary url go to the new page instead of our awesome 404 page on defaultBackend?

How? Why this is happening?

Yes, I also cant believe this is happening. For the sake of debugging, here are the details of my setup

I try to explore what happened via reviewing my ingresses setup, but all the settings is very simple, without no additional customization on both ingress nginx and ingress object itself.

Then I realized, that ingress nginx is basically controller that watch Kubernetes Ingress object, then construct nginx config. With now long forgotten ingress nginx kubectl plugin I try to get the configuration out of the cluster

To get the plugin works on Apple Silicon, you need to build the plugin by yourself.
this github comment on the issue can help you to build it.
Dont forget to also install Krew then put the binary to ~/.krew/bin directory.

I do with very simple command to list down all ingresses that matches the domain, just in case I missed something

➜ kubectl ingress-nginx ingresses
INGRESS NAME     HOST+PATH                                     ADDRESSES      TLS   SERVICE       SERVICE PORT   ENDPOINTS
echo-server      echo.local.dirathea.com/awesome/tools         198.19.249.2   YES   echo-server   80             1
echo-server-v1   echo.local.dirathea.com/new/awesome/feature   198.19.249.2   YES   echo-server   80             1

nope, nothing strange. All matches with what we’ve done so far.

Go deeper

Because those exploration doesn’t answer my question, let’s go deeper. With the same plugin, we can pull the nginx.conf more easily, filtered by the domain

The following config is truncated for readability.
If you do it too, look up at the very last location that being generated by ingress nginx

	server {
		server_name echo.local.dirathea.com ;
		
		...

		location / {
			
			set $namespace      "default";
			set $ingress_name   "echo-server";
			set $service_name   "";
			set $service_port   "";
			set $location_path  "/";
			set $global_rate_limit_exceeding n;
			
			...
			
			return 301 /new/awesome/feature;
			
			proxy_pass http://upstream_balancer;
			
			proxy_redirect                          off;
			
		}
		
	}

huh 🤔, why this / is there, and has return 301 on it? Why the $ingress_name equal to echo-server ?

One searching later, from google, stack overflow, to the github repository, I found this issue. From the title itself it seems not related, but this comment, and this comment shows exactly what happened here.

As my understanding, Ingress Nginx try to create default path handler, and some of the configuration got overriden by information from the first ingress created. Why? I believe to properly configure the catch all configuration.

Let’s take a look our example above again. When we create the first ingress object, we specify the path as /awesome/tools on host echo.local.dirathea.com. We don’t specify configuration for /, so ingress nginx need to figure out the config to catch all, thus it tries to get the information from the first ingress that we created. Unfortunately, it takes more configuration that it should, in our case the redirect 302.

As this blog published, the issue closed by the maintainer, remaining unresolved / until now. There’s a potential workaround is to create your own ingress with / to override the auto generated config, avoiding unintended behavior happening again.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      return 404;
  name: frontend-workaround
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: localhost
    http:
      paths:
      - backend:
          service:
            name: frontend
            port:
              name: http
        path: /
        pathType: Prefix

Conclusion

There’s a bug on the default behaviour of ingress nginx to handle catch all request. It seems by default the catch all configuration copies the oldest ingress created as the catch all configuration, including some customization, like 302 redirect, resulting unintended behaviour.

The issue on github has been closed, wait for more information. In the mean time, there’s workaround to handle the 404 not found handler by create the ingress manually.