When developing Python code we are constantly adding and committing changes. However, nothing stops us from committing low quality code, e.g. code containing unused imports, unformatted code or functions that do not work correctly. Therefore, to make sure the code we commit is of sufficient quality, we use formatting-, linting- and testing tools to check our code. These checks will result in increased code readability, maintainability and quality and will improve collaboration amongst colleagues. We can run these checks manually, but this is time consuming and prone to errors. Therefore, we recommend automating these checks to simplify your development workflow.
In this blog, we are looking into a tool stack that helps us running these checks. We will highlight such a setup with the tools pre-commit and Makefile. When set up right, they complement each other. We will describe why you would want to use them, how they work and what a minimal setup looks like to for your new project.
To ensure our code quality we often make use of formatting-, linting- and testing tools. Each of the three has its own purpose and complement another:
In our setup we use Black as formatting-, Pylint as linting- and Pytest as testing tool. Black is a PEP8 compliant formatter without much to configure. Pylint comes with a code rating and is the most used linting tool for Python. Pytest makes it easy to write unit tests without boilerplate.
Pre-commit is a tool that is used to run scripts (hooks) to automatically identify issues in your code. Configured hooks are triggered when committing your code to Git.
To use pre-commit you need to:
pip install pre-commit
.pre-commit-config.yaml
. This file will contain the hooks you want to run and their settings.pre-commit install
There are many hooks which can be used to check your code, like hooks from pre-commit itself, or third party tools, like Black and Pylint. A minimal example of a .pre-commit-config.yaml
file is shown below:
This setup ensures that for every commit, pre-commit will automatically run the set of specified hooks. First a set of hooks from pre-commit reformats and checks your code. Secondly, Black reformats your Python code. Finally, Pylint scans your code and gives an overall rating. Only if your committed code passes all checks your commit will be completed. Otherwise you (or the hooks) have to change the code accordingly and commit again.
Note that Pytest is not included in our pre-commit configuration as running tests is usually expensive. A list of useful pre-commit hooks can be found here.
Using pre-commit, checks are automatically ran on each commit. However, we might also want to run checks before committing code. Although running checks can be done with separate commands, a more efficient setup can be achieved by using a Makefile. A Makefile makes it possible to create shortcuts that easily run a task (such as your pre-commit) or a set of tasks (such as pre-commit and testing). These shortcuts can then be triggered with the make
command in your terminal. An example of a Makefile
is shown below:
With this setup you can run multiple tasks with simple shortcuts (stated after .PHONY
and opening each task). The make precommit
command runs all your pre-commit hooks on all your Python files. The make black
command runs Black on all Python files in your current working directory. The make lint
command runs Pylint on your src
directory. The make test
command runs Pytest using the tests
directory. Finally, the make ci
command runs previously described tasks in sequence.
Pre-commit and Makefile are great tools to deliver quality Python code in an early stage. They will speed up your development workflow which results in more time for code logic. Both have their own purpose:
As it is easy to integrate pre-commit and Makefile, make sure you try them in your new project! If you need help, please respond in the comments below.
These Stories on Data Engineering
Coltbaan 4C
3439 NG NIEUWEGEIN
+31 30 227 2961
info@vantage-ai.com
Volg ons: