Virtual Environments Demystified

The fucking nerd with his Virtual Boy

Here’s a (presumably non-exhaustive) list of programs that do something with virtual environments:

  1. autoenv
  2. pew
  3. pipenv
  4. pyenv
  5. pyenv-virtualenv
  6. pyenv-virtualenvwrapper
  7. pyvenv
  8. rvirtualenv
  9. venv
  10. vex
  11. v
  12. virtualenv-burrito
  13. virtualenv
  14. VirtualEnvManager
  15. virtualenvwrapper
  16. virtualenvwrapper-win
  17. virtual-python
  18. workingenv

Wow. This stuff must be really hard to get right. I also must be a moron, since, after having written some thousand lines of Python, I don’t even know what problem we are trying to solve here. The abundance of relevant programs with subtly different names has deterred me from reading up on it so far.

So what is a virtual environment?

The official docs’ tutorial describes a virtual environment as

a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

So it’s a directory with a Python interpreter? Easy enough.

$ mkdir virtual_env
$ cd virtual_env
$ cp /bin/python3 .

Let’s see. Directory? Check. Contains a Python installation? Check. Contains a number of additional packages? … Zero is a number! (Check.) Particular version? Um…

$ ./python --version
Python 3.6.3

I think that will do. Is it self-contained, though? It doesn’t contain itself…

Another nerd: Bertrand Russell in 1916

Jokes aside, there are only two things missing to actually make our directory a virtual environment as specified by PEP 405, the proposal that integrated a standard mechanism for virtual environments with Python. (Before PEP 405 was accepted, virtual environments were purely the domain of third-party tools with no direct support from the language itself.)

  1. A file called pyvenv.cfg containing the line home = /usr/bin
  2. A lib/python3.6/site-packages subdirectory

(Both paths are subject to the OS and the second one also to the Python version used.)

$ echo 'home = /usr/bin' > pyvenv.cfg
$ mkdir -p lib/python3.6/site-packages

Because of what I assume to be a bug in CPython, we also need to move the Python binary into a bin subdirectory.

$ mkdir bin && mv python3 bin/

Fair. We have a directory that formally qualifies as a virtual environment. This leads us to the next question.

What’s the point?

When we run our copy of the Python binary, the pyvenv.cfg file changes what happens during startup a bit: the presence of the home key tells Python the binary belongs to a virtual environment, the value (/usr/bin) tells it where to find a complete Python installation that includes the standard library.

The bottom line is that ./lib/python3.6/site-packages becomes part of the module search path. The point is that we can now install modules to that location, in particular, specific versions that may conflict with the dependencies of another Python program on the same system.

venv

In practice, one does not simply create virtual environments by hand, which brings us back to the dauntingly long list of tools above. Fortunately, one of them is not like the others. While it’s predated by most of them, this one ships with Python as part of the standard library: venv.1

In it simplest form, venv is used to create a virtual environment like so:

$ python3 -m venv virtual_env

This creates the virtual_env directory and also copies or symlinks the Python interpreter:

$ cd virtual_env
$ find -name python3
./bin/python3

(It also copies a bunch of other stuff. I get 650 files in 89 subdirectories using up about 10 MiB of disk space in total.)

Summary

Appendices

History

I think Ian Bicking’s non_root_python.py qualifies as the first tool for creating virtual environments. Based on that, virtual-python.py was added to EasyInstall in version 0.6a6 in October 2005. Here’s a timeline summarizing some main events.

2005-10-17
virtual-python.py is added to EasyInstall.
2006-03-08
Ian Bicking, the author of non_root_python.py—on which virtual-python.py is is based—publishes a blog post about improving virtual-python.py titled “Working Environment Brainstorm”.
2006-03-15
Ian Bicking announces working-env.py.
2006-04-26
Ian Bicking announces an improved version of working-env.py called workingenv.
2007-09-14
virtualenv’s first commit
2007-10-10
Ian Bicking announces virtualenv: “Workingenv is dead, long live Virtualenv!
2009-10-24
virtual-python.py is removed from EasyInstall.
2011-06-13
PEP 405 is created.
2012-05-25
PEP 405 is accepted for inclusion in Python 3.3.
2012-09-29
Python 3.3 is released. venv and pyvenv become part of the standard library.
2014-03-16
Python 3.4 is released. “[venv] defaults to installing pip into all created virtual environments.”.
2015-09-13
Python 3.5 is released. “The use of venv is now recommended for creating virtual environments.
2016-12-23
Python 3.6 is released; “pyvenv was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is deprecated in Python 3.6.

More definitions of “virtual environment”

A virtual environment is a semi-isolated Python environment that allows packages to be installed for use by a particular application, rather than being installed system wide.
https://docs.python.org/3/installing/

A [virtual environment is a] cooperatively isolated runtime environment that allows Python users and applications to install and upgrade Python distribution packages without interfering with the behaviour of other Python applications running on the same system.
https://docs.python.org/3/glossary.html#term-virtual-environment

Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally.
—“Creating Virtual Environments” subsection of the “Installing Packages” tutorial at packaging.python.org

Potential bug 1

The following doesn’t appear to work when the Python executable is in the same directory as pyvenv.cfg.

[If the Python binary belongs to a virtual environment] sys.prefix is set to the directory containing pyvenv.cfg.
PEP 405

$ pwd
/home/meribold/virtual_env
$ ./python3 --version
Python 3.6.3
$ ./python3
>>> import sys; sys.prefix
'/home/meribold'

Potential bug 2

This doesn’t appear to be the case:

By default, a virtual environment is entirely isolated from the system-level site-packages directories.
PEP 405

If pyvenv.cfg […] contains the key include-system-site-packages set to anything other than false […], the system-level prefixes will still also be searched for site-packages; otherwise they won’t.
Documentation of the site module (emphasis added)

Here’s what actually happens: site.py I.e., when the file doesn’t contain the key include-system-site-packages at all, system_site will still be "true".

What I need to do is explicitly put include-system-site-packages = false into pyvenv.cfg. Otherwise I can still import numpy etc.

Footnotes

  1. Actually, pyvenv also ships with Python, but was deprecated in version 3.6 (only 3 minor versions after its introduction). Both venv and pyvenv were added to Python in version 3.3.