Updated on January 23, 2018 in #dev-environment, #docker

Fixing exec format errors with Docker ENTRYPOINT Scripts on Windows

You may have gotten cryptic errors when trying to use ENTRYPOINT scripts in your images while running Windows. Here's how to fix them.

Docker ENTRYPOINT scripts are a wonderful thing, but it’s not so wonderful when they start failing in unexpected ways, especially when you’re very confident that “identical” ENTRYPOINT scripts worked on other Docker hosts.

# Does This Error Look Familiar?

standard_init_linux.go:195: exec user process caused "exec format error"

I’m not someone who hacks on the Docker code base itself but when I encountered this error, the exec caught my eye. I thought “hmm, well I’m using exec "$@" in my entrypoint, maybe something is wrong with that?”.

But I knew there wasn’t much that could go wrong because the script did nothing especial.

My ENTRYPOINT Script Was Super Basic

#!/bin/sh
set -e

exec "$@"

It does nothing except pass control back to whatever process is ran in your CMD instruction in your Dockerfile. That’s what exec "$@" does.

Setting #!/bin/sh is standard for setting up a shell environment. This is exactly what the official PostgreSQL image uses in its ENTRYPOINT script. It also uses set -e which tells your shell to halt on any error, and finally it also ends its script off with exec "$@".

Using that logic, we can conclude without a doubt that there is nothing wrong with the above script in terms of syntax, yet it continues to throw that exec error.

Time to Investigate the Dockerfile

If the script isn’t the issue then it must be the ENTRYPOINT instruction or something to do with permissions on the script itself. Here’s the important bits of the Dockerfile:

RUN chmod +x docker-entrypoint.sh
ENTRYPOINT ["/app/docker-entrypoint.sh"]

If I’ve learned anything from working as a web developer for the last ~20 years, it’s that you should never trust anything from the client.

In this context, that means I shouldn’t trust that the script is already executable when building the Docker image, so even though it adds an extra layer to the image, it’s important to chmod +x the script in the Dockerfile.

I was already doing that because I’ve been bit by that issue in the past.

The entrypoint location is also normal. That’s the absolute path to where it exists inside of the Docker image (it was COPY’d in a previous step not listed above).

Thought Process on Debugging This Further

At this point the Docker image builds but it fails to run due to the script. The next step is simple then, just comment out the ENTRYPOINT instruction and re-build it. Sure enough, the image builds and runs correctly.

With this new information we now have 5 facts:

1. The script itself has no syntax errors
2. The script is most definitely executable
3. The Docker image builds, so the ENTRYPOINT has the correct file location
4. A very popular Docker image (PostgreSQL) is using a nearly identical entrypoint
5. The script is causing the error because it works without the ENTRYPOINT instruction

It’s unlikely there’s something wrong with the Docker daemon since the PostgreSQL image confirms it works, so I’m happy to rule that out considering I’ve ran the PostgreSQL image using the same Docker daemon version as I did with my entrypoint script.

# Resolving the Error on IRC

At this point my eye was starting to twitch, and when that happens, that means it’s time to ask for external help, so I hopped on IRC.

I explained the situation and within a few minutes we uncovered the issue. I’m including the conversation here because I think the process is what’s important, not the solution:

<nickjj> has anyone else ever seen this error when trying to use an entrypoint? standard_init_linux.go:195: exec user process caused "exec format error"
<ada> nickjj: usually see that when there's an architecture mismatch
<ada> nickjj: you may be trying to run an amd64 image/binary on arm, for example
<nickjj> ada, the image works if i comment out the ENTRYPOINT line, it's all running on amd64
<ada> nickjj: what do you mean "works"

Looking back, I like ada’s comment here about questioning me on what I meant by “works”. When it comes to real time communication, sometimes you don’t pick your words wisely. What I should have said was the Docker image builds successfully.

Anyways, let’s continue with the conversation:

<nickjj> it even fails with that error with a completely blank entrypoint script
<programmerq> nickjj▸ then your ENTRYPOINT is likely the thing with the mismatched architecture.
<ada> nickjj: if an image has no entrypoint/cmd, it does nothing when it starts
<programmerq> if it's a script, then the #! line could be the issue
<ada> nickjj: so you've already narrowed down the culprit - something in your entrypoint is wrong
<nickjj> programmerq, the script is literally just this:
<nickjj> #!/bin/sh
<nickjj> exec "$@"

Yep, I used the word “literally” wrong here because it had set -e and empty lines too! I just wanted to keep the script short on IRC (terrible idea btw).

<programmerq> your /bin/sh could be the issue
<nickjj> it's using the jessie base image btw
<programmerq> or your script could have wrong permissions
<nickjj> it's set to chmod +x
<nickjj> programmerq, here's the exact perms from inside the container: -rwxr-xr-x 1 root root docker-entrypoint.sh
<programmerq> if you fire up a shell and then try to call that script directly, does it behave as expected?

Oh, now that’s a good idea. I didn’t even think about executing the script manually inside of the container. Thinking back, that was such an obvious thing to do, but guess what, this is how you learn.

If an entrypoint script ever goes wrong in the future for me, or if one of the people taking my Docker course has a problem with an entrypoint script, you can be sure running it in the container directly will be near the top of my list of things to do / ask.

<nickjj> programmerq, something is seriously crazy here
<nickjj> if i manually run it with `sh docker-entrypoint.sh`, it executes with all sorts of errors
<ada> script full of unprintable unicode characters?
<nickjj> i get a bunch of these exact errors -> : not foundypoint.sh: 3: docker-entrypoint.sh:

Ok, so, what’s line 3 of the script? Ah, it’s an empty new line. Side note, I didn’t include my exact script when asking for help which was a mistake because from their POV, line 3 was something different. Never try to be tricky when asking for help, but that’s a lesson for another day!

<mgolisch> weird encoding? windows linendings?
<nickjj> but line 3 (and all of the problematic lines) are empty new lines
<ada> delete & recreate
<nickjj> and foundypoint.sh has no relevance to anything in terms of file names

I have to admit, seeing foundypoint.sh there threw me off. What the heck does that have to do with anything. Looking at it now, the entire error is : not foundypoint.sh: 3: docker-entrypoint.sh:

That really says “not found” and point.sh is part of the docker-entrypoint.sh file name. Why it came out like that is above my pay grade, but at least it could in theory make some sense now.

<mgolisch> id guess wrong line endings
<mgolisch> that causes all sorts of weird errors in shellscripts
<programmerq> if it was newlines, then your #!/bin/sh was probably more like #!/bin/sh\r\n
<ada> I have found 0-width characters in copy-pasted code before
<programmerq> and that evaluates to an invalid interpretor, which would explain the exec error.

Well, now it’s starting to add up. Guess who moved their code editor to be running in Windows the other month? Also, guess who recently moved to using VSCode? This guy!

Prior to that, I was running Sublime Text in a Linux driven VM for the last ~5 years. CRLF vs LF line endings isn’t something I dealt with in over 5+ years.

Let’s continue with the story because it shows how to detect the issue and resolve it.

<nickjj> my environment is...
<nickjj> 1. docker for windows on win 10 pro , 2. using WSL , 3. code being built into the image lives inside of windows outside of WSL
<nickjj> 4. i'm accessing the code through the WSL mount which does make permissions a bit nutty
<ada> what does "using wsl" mean in this context?
<nickjj> windows subsystem for linux
<ada> right but how does that play into this
<ada> docker-for-windows doesn't use wsl afaik
<jready> I dont even think wsl supports docker
<ada> it doesn't
<mgolisch> no but i think you can use it , think the cli actualy works in it
<nickjj> it doesn't, but i am using WSL and set up the docker client to talk to the daemon running on docker for windows
<jready> That seems like a pretty roundabout way to do things lol

Things are starting to drift a little off topic, but ada guides us back to the main problem.

<ada> still thinking line endings / encoding is the main problem here
<ada> makes sense to me, I do the same
<mgolisch> yeah check the entrypoint script
<ada> when I have to use a windows box
<nickjj> ada, yep, that's possible. i am using vscode installed directly on windows
<ada> nickjj: what does "file docker-entrypoint.sh" say from the terminal
<nickjj> ada, docker-entrypoint.sh: POSIX shell script, ASCII text executable, with CRLF line terminators
<ada> there ya go

Bingo, that confirms the issue. Pesky CRLF line endings (\r\n) are in the file. The PostgreSQL entrypoint script has Unix style LF line endings (\n), but most code editors and GitHub don’t visibly show this, so it was invisible to the naked eye.

<ada> crlf line endings are windows formatted - set all of your editors to use Unix line endings
<ada> nickjj: there's a dos2unix command line utility to let you convert line ending characters
<nickjj> ada, hmm i'll hunt around for a vscode setting to force unix line endings
<nickjj> ada, the bottom right of vscode does say CRLF
<ada> click on it, select LF
<ada> should change your line endings in that file
<nickjj> ada, yep the entrypoint is working now with no additional changes
<nickjj> thanks a lot, it would have taken a long time to draw this conclusion haha

Special thanks to ada, programmerq and mgolisch for helping solve this mystery.

Now we know what we need to do, so let’s do it.

# Configuring VSCode and WSL for LF Line Endings

Forcing Unix line endings in VSCode was very straight forward. Just drop this into your user settings: "files.eol": "\n". You could alternatively place that into a workspace setting for specific projects, in case you have projects that run on both Linux and Windows.

New files that you create should have a LF in the bottom right of the VSCode window instead of CRLF, but existing files that have CRLF line endings will continue to be CRLF.

Installing and Using dos2unix

From within WSL run sudo apt-get install dos2unix to install a utility that will automatically convert CRLF to LF.

Then just use it on a file by running dos2unix docker-entrypoint.sh or whatever file you’re converting. Done deal!

Now if you ran file docker-entrypoint.sh it would look like this instead: docker-entrypoint.sh: POSIX shell script, ASCII text executable. It no longer has with CRLF line terminators as we saw in the IRC chat log.

Of course, if you’ve been setting CRLF line endings for a project, you probably have a ton of files to convert and doing it 1 by 1 would be really tedious.

For that you can use the power of Unix pipes to recursively run dos2unix against a directory path of your choosing.

Recursively run dos2unix against all files in a path:

All you have to do is run find . -type f -print0 | xargs -0 dos2unix.

That will run it against the current directory. Replace . with a different path if you want.

If you’ve made it this far, now you know why the problem happened and how to fix it.

BONUS: Applying This Process to Other Problems

This is why I stand by the statement that breaking down problems is the #1 skill to have as a software developer. The solution was simple once the problem was understood and we got there by breaking things down and using a process of elimination.

And in case you’re wondering, I most definitely used the Rubber Duck debugging technique before I asked for help on IRC.

Have you ever been bit by CRLF vs LF line endings? Let me know below!