Class 3 - virtual envs, packaging

During class we covered virtual environments, basics of packaging, uploading your packages to the test PyPI server and running your own server with simple-pypiserver. The final version of the package is here.

Virtual environments

Virtual environments provide a specific Python interpreter and software libraries and binaries to support a project. They are isolated from system Python installation and other virtual environments. Specifically, running pip install <package> inside a venv will install the package without making any changes to the system Python. When starting a new project, it is common to start with making a dedicated venv for it.

Python standard library provides the venv module since version 3.3. We create a new project directory, enter it and create a new environment by executing:

mkdir new_project
cd new_project
python -m venv .venv

The new virtual environment is located at .venv directory. We activate it under Mac OS, Linux, Cygwin, Git for Windows shell by sourcing a script:

source .venv/bin/activate

source .venv/Scripts/activate

Under Windows Power Shell, we activate it by executing:

.venv/Scripts/Activate.ps1

You can check with pip list that your Python installation now only contains pip and setuptools. Activating the environment modifies the PATH environmental variable and makes .venv/bin/python your default Python interpreter. It also sets the site directory to appropriate path under .venv/. To check where your current Python interpreter looks for modules you can execute:

python -c 'import sys; import pprint; pprint.pprint(sys.path)'

which should show something similar to:

['',
 '/Users/gkowzan/.pyenv/versions/3.11.3/lib/python311.zip',
 '/Users/gkowzan/.pyenv/versions/3.11.3/lib/python3.11',
 '/Users/gkowzan/.pyenv/versions/3.11.3/lib/python3.11/lib-dynload',
 '/Users/gkowzan/d/praca/DYDAKTYKA/python3/class3/my_project/.venv2/lib/python3.11/site-packages']

Packaging

Packages provide a way to distribute libraries, programs, associated data and documentation. There are two kinds of distributions that are used to share packages: source distributions (.tar.gz) and binary distributions (.whl). Source distributions are simply copies of your package tree and they are the same for all CPU architectures and operating systems. Binary distributions can differ when they contain extension modules.

During classes we prepared a basic package. Please see:

for guides that cover mostly the same material. We used the setuptools build backend because it allows for building extension modules, i.e. Python modules written in a compiled language, whereas more modern build backends often neglect this functionality.

PyPI server

We then used twine tool to upload the package to TestPyPI, see Using Test PyPI. We also used pypiserver to run a local repository for our packages. After installing and creating a directory to store the packages with:

pip install pypiserver
mkdir ~/python-packages

you can run the server with:

pypi-server run -p 4040 -i 127.0.0.1 -P . -a . -o ~/python-packages

This will make the server available only locally (hence the 127.0.0.1 IP address) on port 4040 and allow uploading packages without authentication. You can install packages from the repository by using --index-url option with pip:

pip install --index-url=http://127.0.0.1:4040/simple <packages>

Packages not found in the local repository will be obtained from the global Python Package Index.

To upload packages to the TestPyPI server and our local server, you need to specify them in $HOME/.pypirc config file:

[distutils]
index-servers =
  testpypi
  local

[local]
repository = http://127.0.0.1:4040
username = username
password = password

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = <your-testpypi-token>

The local repository user name and password really are just username and password, but you need to obtain the test PyPI password from you user account. Once configured, you can use -r option of twine to specify package index:

twine upload -r local <package_files>
twine upload -r testpypi <package_files>