Setting Up a Python Development Environment with and without Docker
Part of being a developer includes setting up your computer so that you can develop the applications you want to write.
You might be wondering what “setting up your computer” means exactly. To me, that means coming up with a plan to solve the following problems:
- What code editor should I use?
- How do I install Python?
- How do I install my app’s dependencies?
- How do I install required external services?
- How do I run my application?
Technically these problems are the same for any language / framework, but let’s stick with Python for this article. Specifically web development with Python.
# What Code Editor Should I Use?
This one is easy because whether or not you use Docker, the answer is going to be the same. Just install whatever code editor you prefer.
Personally I’m a fan of using Vim and tmux but I’ve used VSCode and Sublime Text for years in the past. They are all really good, it comes down to preference!
# Getting Your Python App Running without Docker
The first thing you need to do is install Python on your operating system of choice. Are you using Windows, MacOS or Linux? If Linux, what distro are you using?
I’m not going to show you how to do this for each OS choice because I have a low tolerance for self induced pain, but please Google for something like “how to install Python on Windows 10” and you’ll find all sorts of answers.
Now that Python 2.7x is end of life, the decision on which Python version to go with is a bit easier, such as using the latest stable Python 3.x release, but there’s still decisions to be made here and I’m sure it’ll be a fun decision if Python 4.x gets released later.
Next up, you’ll want to solve the problem of environment variables. These are great for dealing with secrets and other values that might change between development and production, such as how many gunicorn workers you want to run.
A popular Python library to use for this task is python-dotenv because it lets you keep all of your variables in a single .env file. The libray will make them available when you start your app. If you decide not to use this then you’ll need to remember to export / source them every time you open your terminal.
After that, you need to deal with your app’s dependencies. There’s a few different types of dependencies here. There’s installing packages that your application needs to run (such as Flask if you’re building a web application, or Django if you prefer that), but there’s also external services too, such as your database (which we’ll go over shortly).
Let’s focus on your app’s packages first. Technically you could just install pip (which is Python’s package manager) and start installing everything directly on your main OS, but that’s going down a dangerous path.
If you install your packages to your system’s version of Python directly then you’re forfeiting your right to be able to work on more than 1 project that requires 2 different versions of the same package.
This comes up all the time in web development. You might be working on a project today but then 6 months from now you start a new project and you want to use the shiny new version of a package you started using 6 months ago.
Well, you can’t just use the new version in your new app without caring because if you upgrade it, it will likely break your old project from 6 months ago due to backwards compatibility issues.
This is a really big problem but fortunately smart people have solved this problem.
You can install a tool called Virtualenv. Virtualenv is a Python tool that lets you create these little virtual environments (not to be confused with virtual machines btw).
Now when you start a new project, you also create a Virtualenv for it too. When you install all of your app’s required packages, they all get stored in their own separate Virtualenv.
You can even tell it to use a specific version of Python. That’s pretty sweet.
The problem is now totally solved. All of your projects have their own package versions, and your system’s version of Python is not polluted by anything. It can simply act as a runtime for your Virtualenv wrapped apps.
But we’ve only begun, what about system dependencies? Often times a specific Python package requires having a C compiler and certain other system level dependencies available because something needs to be compiled.
Others depend on system level packages to be installed before they work. For example a number of popular image manipulation libraries require having imagemagick installed.
This is an interesting problem because the instructions are wildly different depending on what OS you use. Sometimes you’ll find cases where an extension won’t work on anything but Linux due to its dependency requirements.
It gets even more complicated too because sometimes certain Python libraries require specific versions of a system package in order to work but Virtualenv isn’t going to isolate those packages so now you’re stuck in a situation where upgrading project B might conflict with project A’s system dependencies.
But, let’s say you managed to track all of that down and get everything you need installed and working because you don’t care about developing multiple projects for now.
Onwards we go, by installing external services. Real world web applications often consist of at least a database, and could easily require many more services such as Redis, etc..
If you plan to use Postgres or MySQL in production it would be a really bad idea to use SQLite in development because some of your code and queries may depend on using features that simply don’t exist with SQLite.
So the best thing you could do is install Postgres on your computer. That’s going to be very different depending on what OS you use.
You’ll also want to make sure you get the same version you plan to run “for real” on your server later, or whatever version your managed database service uses (such as DigitalOcean’s managed DB or RDS on AWS, etc.).
Technically it will never be the same because the MacOS or Windows version of Postgres is not the same binaries that run on a Linux server, but it’s likely “close enough”.
You’ll likely have to do the same thing for Redis too because most real world web apps will have a background worker (such as Celery) and Redis is great to use as a broker and cache, but now we’re getting pretty deep in the woods on a side topic, so let’s get back to services.
By the way, Redis doesn’t even run on Windows officially. You’ll have to settle for running a custom port from Microsoft, or alternatively you can use WSL 2 or delve into the wonderful world of virtual machines and potentially use tools like Vagrant.
WARNING: You’re about to enter a world of serious pain and confusion with VMs if you’re new to software development. It’s also going to delay writing your app by quite a lot of time while you soak in how to set up all of this stuff. Expect many sleepless nights.
By the way, it’s going to get really tricky to install multiple specific versions of PostgreSQL and Redis because chances are if you have multiple projects you will want to use different versions of these and like system level dependencies, Virtualenv isn’t going to help you isolate them.
It also gets weird when unrelated tools on your system want to install a specific version of PostgreSQL. For example, the very popular DaVinci Resolve video editor will install PostgreSQL since it uses that to manage how it saves projects.
But like before, let’s just blindly ignore the issue of wanting to install and juggle multiple versions of PostgreSQL and Redis.
You’ve made it, now it’s time to run your app. If you made it this far, you’re serious about web development for sure. Too bad the bar of entry was so high before you could even start writing your application.
I mean, we’re still not there yet, because now we have to figure out how to run our app.
We have a few choices, but there is 1 thing to be aware of. In order for your application to work, you’ll need to make sure Postgres and maybe Redis is running first.
You could potentially run them all the time, but that means they will be eating up your system’s memory and CPU cycles even when you’re not developing. Sure, you can start and stop them manually between development sessions, but that’s not fun.
Once you figure out how you want to run Postgres and Redis, you need to worry about starting up your Python app too. I’m a big fan of Flask for web development, which is a micro web framework for Python. I like it so much that I created a whole course on it.
Anyways, you’ll need to start your web app’s server along with your background worker, which means you’ll be juggling multiple terminals. It’s going to get worse too as you introduce more services to your app over time.
However, like Virtualenv, this is a solved problem. There’s a really nice tool
called Honcho. It
lets you create this thing called a Procfile and then define what services you
want when you run it. It runs everything through one terminal window.
You could add your web server, background worker and even Postres / Redis to
this file and be good to go. Now if you want to start everything up, you just
run honcho start and you’re golden. It even color codes the terminal output
for all of your services.
Let’s deploy our application to production. We can’t really do that in this blog post but it is worth bringing up 1 important topic related to deployment.
Up until now, we’ve done everything on our dev box which is likely running Windows, MacOS or some distro of Linux that’s not running on your production server.
That means you’ll need to figure out how to do everything you just did on whatever server operating system distro you picked.
Then you’ll likely want to research various process management tools because you’ll want to make sure that all of your services get restart if they crash or if you reboot your server.
Good times.
To recap, before you could even start writing your app, you had to:
Hunt down how to install Python, figure out a strategy for loading environment variables, get pip installed, install Virtualenv and now deal with using Virtualenv on a very regular basis, otherwise you’re going to enter dependency hell.
Then you had to deal with installing C compilers for certain packages and also track down certain system level dependencies to make a number of popular Python packages work.
Then you had to learn how to install and configure Postgres, Redis and any other services that your application may use. Finally after that, you had to figure out how to run everything in a pleasant way with Honcho.
After that, you had to throw out most of your knowledge and learn how to do everything again from scratch on your production server since the environment there is much different than your dev box. Technically you could have written your app before this, but it’s still something you have to deal with.
By the way, we didn’t even talk about working with a small team of developers or open sourcing your project because now you’re on the hook for writing documentation so that other folks can get everything up and running on their machine.
That’s a long road, and it may have taken weeks or months to get there if you were uncovering things as you went.
# Getting Your Python App Running with Docker
If you’re expecting me to say “but with Docker, it’s so easy, all you have to do is run 1 command and you’re done”, then you’re mistaken!
While, it’s true, once you have things set up and understand Docker, it really will be so easy that you can set up any real world Python web app in 1 command, and it will work just the same on Windows, MacOS and Linux in both development and production.
You don’t need to worry about project isolation because Docker handles that for you. You can have separate projects use whatever Python versions they need, along with whatever pip package versions they require. Virtualenv won’t even be used.
You also don’t need to think about using python-dotenv because Docker has a way for you to load in environment variables through a file with 1 line of code without any extra dependencies. Speaking of dependencies…
System level dependencies are easy mode too because you only need to solve the problem once inside of the Docker image, and then it runs the same no matter what OS is running Docker itself.
Installing external services are a breeze too. You won’t be installing them directly on your main OS. They will run locally on your computer, but they will be ran through Docker. For example, getting Postgres or Redis up and running requires running 1 command and no advanced configuration. Each project can have its own isolated version of each too.
Lastly, you won’t need a tool like Honcho and you don’t need to worry about external services running when you’re not developing your app because Docker has its own tool to spin up and shut down all of your app’s services in a few seconds with 1 command.
Oh yeah, Docker handles process management too. It’s really straight forward to have your services get restart on error or come up automatically after rebooting.
All of this sounds great, but it does come at the cost of a tiny bit of complexity.
You’ll first need to install Docker. That’s the easy part and it only takes about ~7 minutes.
The complexity is figuring out how Docker works. It’s pretty simple once you get it, but to get to the point where everything clicks can be a lot to take in if you’re new to Python development or web development in general.
I have a whole tutorial on running a real world Flask application with Docker if you want to read it. It doesn’t go into the gory details of everything, but you will have a real world Flask app running in Docker in about 15 minutes.
If you want the gory details on how Docker works and learn how you can start using it on your own projects then I recommend checking out my Dive into Docker course. It’s 5 hours of premium video along with 30+ guided labs and challenges to get you going with Docker for Python apps (or any language really).
I know, this is where you’re thinking “omg, this whole thing was a sales pitch for buying a Docker course”, but it’s really not. You can totally do it the long way without Docker using the tools I mentioned above, or you can learn Docker on your own by reading the tutorial I linked and then scouring the internet for dozens of articles to fill in the gaps.
The value of my course is that I’ve done all of the research for you and I’ve been using Docker since 2014. I’ve ran it in development and production for years. Along the way I’ve came up with a bunch of best practices and it’s all in the course.
Plus I offer lifetime updates and support. If you have any questions, I’m only an email away. I’ve answered over 3,500 questions from 20,000+ people since 2015. Support is something I take really seriously! Hop on board and join us in the Dive into Docker course.
Are you using Docker for Python web development already? Let me know below.