Learn Docker With My Newest Course

Dive into Docker takes you from "What is Docker?" to confidently applying Docker to your own projects. It's packed with best practices and examples. Start Learning Docker →

Updated on August 29, 2023 in #deployment, #docker

Binding to Low Ports as a Non-root User with Docker and Kubernetes

binding-to-low-ports-as-a-non-root-user-with-docker-and-kubernetes.jpg

A low port is anything < 1024. You can use sysctl to fix permission errors to run a containerized web process that listens on port 80.

Quick Jump:

Prefer video? It’s up on YouTube covering everything in this post as well as additional content that demos the solutions.

Depending on what process you’re running (nginx, Apache, etc.), you may have seen an error like listen tcp :80: bind: permission denied. This indicates that you’re trying to run a process on a port that usually requires root privileges. It’s not limited to port 80 either.

Normally you would only encounter this if you’re also running your process within that container as a non-root user. That’s a great idea for security purposes. I’ve written about that topic in the past.

There’s a couple of ways to solve this, all of which do not require modifying any system settings on your Docker host or Kubernetes nodes. Instead we can make changes to either our app’s config or set a Docker / Kubernetes config option that’s isolated to a specific docker-compose.yml or deployment.yaml file.

Here’s 3 ways to fix the error:

  • Use a higher port such as 8080
  • Use sysctl and set net.ipv4.ip_unprivileged_port_start = 0
  • Set the capability CAP_NET_BIND_SERVICE

I’m sure there’s other ways but the above 3 are what we’ll cover in this post and I personally would suggest the 1st or 2nd option depending on your use case.

# Are Low Ports More Secure?

Before we get into any of the solutions, an important question to ask and answer is “is it less secure to use a lower port?”.

The answer is not really. At a fundamental level listening on port 8080 and 80 do the same thing. One isn’t more secure than the other if you’re talking about functionality.

You could make a case that instead on listening on port 22 for SSH, you may want to listen on 23617 or another random high port to avoid low hanging fruit where someone may try to connect to your default SSH port.

Sure, I’ll generally agree with that but in this case the port number alone isn’t adding or removing “security”. Using a custom port here is mainly a filter to prevent the most basic bots doing internet wide scans. Someone can still easily port scan you and find your custom port if they want to.

Why Do We Have Low and High Ports?

Low ports are technically “system ports”. They’re well known ports for common services like SSH, DNS, HTTP, HTTPS, SMTP and so on.

In my opinion you shouldn’t use any of these low ports for anything but their intended use but technically there’s nothing stopping you.

Having these low ports require root access is completely reasonable if you rewind time to decades ago where you might have 1 server with many different user accounts. You surely wouldn’t want an average Joe or plain Jane binding to port 80 on your server with their regular user account to run something custom.

Even today in a single user / single server environment it’s reasonable. It’s a good reminder that you’re interacting with something at a system level or more generally a semi-formally reserved port.

What about Containers?

Now things get a bit more interesting. If everyone has their own isolated network, file system and overall run-time environment should you really need root access to be able to bind to port 80?

What if you factor in wanting to do the right thing and make your application more secure by running it as a non-root user and also disable privilege escalation all together?

For example, you may choose to listen on ports 80 / 443 with nginx or Apache but have them fully configured to run as a non-root user. Since these are web servers and reverse proxies, it’s normal to want to bind to these ports.

You could say, well I’ll bind them to ports 8080 and 4430 and then put a load balancer in front of them. Sure, you can do that. But what if you don’t have a load balancer?

Ok, now let’s look at a few different ways to solve this issue.

# Use a High Port

If you can easily do it, it’s not a bad idea to use a higher port. If you have a dedicated load balancer in front of your reverse proxy (nginx, etc.) then there’s no harm in configuring nginx to use higher ports since the network facing load balancer will be using 80 and 443.

Personally I run nginx outside of Docker for single server deploys so I don’t run into this issue but I have worked on a number of projects using Kubernetes for contract work where someone may have their web app + Apache running together in a pod.

I understand the real world happens for other reasons too. It’s completely fine to run a large classification of apps on 1 server without a load balancer, so maybe you can’t use a high port since nginx or Apache is being directly accessed from the internet.

Or maybe you have a dozen apps with reverse proxies already configured to use 80 / 443 and you moved them into Kubernetes around the time Kubernetes 1.23 came out (late 2021) and now you’re finally upgrading to 1.24+ and you started to get access denied errors – we’ll cover why that is in a bit.

In any case, using a high port is an option but if not…

# Set Kernel Parameters with sysctl

You can modify a Linux kernel parameter for your specific containers that need this access. This can be done in a docker-compose.yml file or a Kubernetes config for your deployment.

Before we get into code, here’s the parameter that will address this issue:
net.ipv4.ip_unprivileged_port_start = 1024 (default)

It’s literally how it reads. It lets you define which port number no longer requires root access to use. For example if you set 42 then ports 1-41 will require root.

If you’re running Linux you can run sysctl net.ipv4.ip_unprivileged_port_start on your system / Docker host to see the current value. It’ll likely be 1024 and for your system that’s a good value to have it set to.

But check it out inside of any container that you’ve run:

Feel start to start up some of my example Dockerized web apps such as the Rails one which has sysctl installed by default (the Flask app doesn’t btw):

$ docker container exec <container_id> sysctl net.ipv4.ip_unprivileged_port_start

Chances are you’ll get 0 back as a value which allows all ports to be bound without root.

The good news is as long as you’re using a version of Docker that’s at least 20.10.0 which was released in December 2020 then Docker and Docker Compose already works this way out of the box. You don’t need to do anything thanks to this PR.

If you have an ancient version of Docker and are using Docker Compose you can set:
services:
  web:
    build: "."
    sysctls:
    - "net.ipv4.ip_unprivileged_port_start=0"

Also, with Kubernetes < 1.24 if you’re using the Docker container runtime this worked out of the box since it was set for you. But if you’re using 1.24+ or any version of Kubernetes with containerd as the container runtime then you’ll get permission denied.

Unlike Docker, Kubernetes doesn’t default to this but there is an open issue for that and even an issue to make it the default for containerd 2.0. As of Kubernetes 1.22 they also classified this parameter as safe which means you don’t need to modify your nodes to start using it and it will only influence workloads running in your pod, not the whole cluster.

With modern versions of Kubernetes you can set the securityContext on the pod:
apiVersion: "apps/v1"
kind: "Deployment"
spec:
  template:
    spec:
      securityContext:
        sysctls:
        - name: "net.ipv4.ip_unprivileged_port_start"
          value: "0"

After applying that, you’ll be good to go.

Is It Safe?

I’m comfortable using it as a solution as needed. Docker sets it and it may become a default for Kubernetes and containerd too.

Even if Kubernetes and containerd don’t make it a default I’d still think the same way. We know low ports aren’t inherently less secure. We also know Kubernetes says this setting is only isolated to the pod with this setting as it was explicitly marked as safe.

# Set Capabilities

Capabilities are a Linux feature where you can give an unprivileged user specific capabilities that normally require root access.

One of those capabilities is CAP_NET_BIND_SERVICE which lets you bind to lower ports. This can be done without modifying the Linux kernel parameter we mentioned in the last section.

This can also be set in Docker Compose and Kubernetes. There’s no need to do this in Docker Compose with modern versions of Docker.

With Kubernetes you can set this on a container:

      containers:
      - name: "demo-app"
        securityContext:
          capabilities:
            drop:
            - "ALL"
            add:
            - "NET_BIND_SERVICE"

There’s currently an open issue in Kubernetes that hints using this along with allowPrivilegeEscalation: false can produce surprising results.

There’s a number of comments that also suggest using kernel parameters as a more reliable way of binding to low ports.

Personally I didn’t explore this option much because I do think the kernel param approach is sound. Plus I don’t know a whole lot about Linux capabilities and quite frankly I didn’t want to use something in production that I didn’t have a sound understanding of.

Based on the research I did, the kernel param approach feels better at least to me so performing a deep dive on capabilities didn’t feel warranted for this use case. That’ll be an adventure for another day.

# What Led Me down This Rabbit Hole?

I didn’t just wake up today eager to write about this topic. I was doing some client work where they have a dozen web apps that are using Apache as a reverse proxy. It’s your typical PHP-FPM / Apache set up.

The application is well over a decade old and its runtime history is:

  • Apache listening on ports 80 / 443 without Docker and no load balancer
  • Apache listening on ports 80 / 443 without Docker and a load balancer
  • Apache listening on ports 80 / 443 with Docker Compose and a load balancer
  • Apache listening on ports 80 / 443 with Kubernetes 1.22 and a load balancer
  • Apache listening on ports 80 / 443 with Kubernetes 1.23 and a load balancer
  • Apache listening on ports 80 / 443 with Kubernetes 1.24 and a load balancer

I came in at step 3 to help them Dockerize their apps and then eventually move into using Kubernetes because it made sense for their use case. It’s all been super smooth, we’re also running 1.27+ nowadays and upgrade every 6 months.

When I attempted to upgrade the cluster from 1.23 to 1.24 I started to get an access denied error on Apache but only in the test cluster, it didn’t happen locally with Docker Compose or in CI with Docker Compose.

I knew 1.23 to 1.24 changed from using Docker to containerd as the container runtime but at the time I had no idea about net.ipv4.ip_unprivileged_port_start and how it related to Docker.

Each app has its own repo and its own copy of Apache, both PHP-FPM and Apache run in 1 pod as 2 separate containers. I didn’t want to make 12 separate updates to all of the repos.

A bit of Googling ultimately led to everything you read in this post and I solved it by modifying our Kustomize base web deployment in 1 spot to set net.ipv4.ip_unprivileged_port_start = 0 and everything immediately worked.

Too often are solutions pretty easy once you know the problem!

The demo video below demos running a few Docker and Kubernetes commands that show the permission error and how to tweak net.ipv4.ip_unprivileged_port_start in Kubernetes.

# Demo Video

Timestamps

  • 0:35 – What is a low port?
  • 1:57 – At least 3 ways to solve this
  • 2:40 – Are low ports more secure?
  • 3:57 – Why do we have low and high ports?
  • 6:25 – Solution 1: Use a high port
  • 8:34 – Solution 2: Tweaking a Linux kernel param
  • 10:00 – Docker 20.10+ sets this by default
  • 12:13 – With K8s 1.24+ you need to tweak it
  • 15:33 – Solution 3: Linux capabilities
  • 18:21 – Demoing the issue in Kubernetes
  • 22:30 – Solving it with solution 2
  • 24:28 – What led me down this rabbit hole?

References

Kubernetes Config

Here’s a demo.yaml that you can kubectl apply -f demo.yaml into your Kubernetes cluster.

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "echo-app"
  labels:
    app: "echo-app"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "echo-app"
  template:
    metadata:
      labels:
        app: "echo-app"
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        sysctls:
        - name: "net.ipv4.ip_unprivileged_port_start"
          value: "0"
      containers:
      - name: "echo-app"
        image: "jmalloc/echo-server"
        env:
        - name: "PORT"
          value: "80"
#        securityContext:
#          capabilities:
#            drop:
#            - "all"
#            add:
#            - "NET_BIND_SERVICE"
---
kind: "Service"
apiVersion: "v1"
metadata:
  name: "echo-service"
spec:
  selector:
    app: "echo-app"
  ports:
  - port: 80
    targetPort: 80
---
apiVersion: "networking.k8s.io/v1"
kind: "Ingress"
metadata:
  name: "example-ingress"
spec:
  rules:
  - http:
      paths:
      - path: "/"
        pathType: "Prefix"
        backend:
          service:
            name: "echo-service"
            port:
              number: 80

Have you used any of these methods to solve permission errors? Let us know below!

Never Miss a Tip, Trick or Tutorial

Like you, I'm super protective of my inbox, so don't worry about getting spammed. You can expect a few emails per year (at most), and you can 1-click unsubscribe at any time. See what else you'll get too.

Comments