It’s All in the Details
Upgrading Haystack to have a source code formatter and a linter
All professional open source project have a set of standards. It is the standards set by the project leaders that keep everyone’s code similar and free from unnecessary error. It is what makes the code consistent among numerous additions from multiple contributors.
In this weeks goal, I want to set up my own standards for my project, Haystack. To achieve this, I will need to do three main things: create a CONTRIBUTING.md file, add a Python source code formatter, and add a Python linter.
Adding a CONTRIBUTING.md File
If someone wanted to contribute to Haystack, how would they go about it? As it stands now I have a simple step by step guide on how to download Haystack onto one’s local machine using Git, but I have no guide on how to easily add to the source code. This is where the CONTRIBUTING.md file comes in handy; I moved the Haystack installation from my README.md to my newly created contributing file and proceeded to add steps on how to run the code and commit any additions that they may have made. In doing so I realized, “how can I make sure that they truly followed the same coding style as me? How can I be sure that they didn’t make any typos?” The answer, Black and PyLint.
Adding a Source Code Formatter — Black
Black is a Python source code formatter that is ran from the command line — it ensures that the source code follows the same coding style throughout the entire file. Using this allows for consistent code among multiple contributors. Installing and using Black was fairly simple: assuming you have Python installed (along with pip), type “pip install black” in the command line. Let pip do its thing and once its finally downloaded, issue the following command from within the folder containing the code that you want to format: “black <filename>”, in my case it was “black haystack.py”. That’s it… Black took care of the dirty work and I was left with a freshly formatted source file. Using this tool in my Haystack produced interesting finds:
Here you can see that all method calls are instead written over multiple lines, which I think is better than what I was doing previously — cramming it all onto the same line.
Moreover, you can see in my argparse argument initialization method that each of the argument’s parameters are now written and expanded horizontally over multiple lines. Looking at the comparison (via git diff) between the two, I can see how much more readable Black’s styling is.
Furthermore, it changed all the single quotation strings into double quotation. I noticed that in my original source code, I had a mix between the two which can be confusing for new contributors. Needless to say, I was pleased with Black’s coding style changes.
Adding a Python Linter
Now, Black is good at formatting code but how can I be sure that other contributors haven’t made any typos or even perhaps unconventional method names? This is where a linter, PyLint, comes in handy. It essentially performs logical and stylistic checks for errors, along with trying to enforce a coding standard. To start using PyLint (assuming you have Python and pip installed), type “pip install pylint” into the command line. Once that is downloaded, go to the source code folder and type: “pylint <filename>”, in my case it was “pylint haystack.py”. You will see a list of warnings (unless you are a coding messiah) that will need fixing. Here are some of the warnings that I got:
Here you can see that my code is missing a lot of method docstrings. I intentionally didn’t put them at first since most of the methods are short and self explanatory. However, I need my code to be indisputably understood by everyone and I decided that PyLint made the right call here — thus I began documenting.
Interesting side note: PyLint rates your code based on the amount of warnings it has given you (and also shows a change in your score based on your previous code rating), which I guess makes it easier for the developer to see if they are making progress.
By the end, I managed to get it up to a 9.85/10, with the only warning being a too general exception clause. I decided to leave it at that, since the general exception is for the URL requests — it needs to be broad to be able to handle any status code and or error that can come from validating the URLs. Satisfied, I moved on to adding a section on my contributing file on how to download and run Black and Pylint.
Visual Studio Code Setup
To integrate the above tools into my IDE (VS Code), I put the following steps in my contributing. This is for other contributors that want to have an easier time working on and formatting the code in Haystack, and for me who sometimes uses my spare laptop to work from my study room downstairs. Integrating these extensions were fairly simple: one had to essentially check off some parameters in VS Code’s settings (for more detail, see CONTRIBUTING.md).
With everything all set, the contributing file, the source code formatter, and the linter, I committed it all. With a quick rebase and squash, I merged it onto my master and pushed.
So… what did I learn from this process? It’s that the details matter, especially on open source projects. Any file that contains work from multiple programmers needs to be consistent throughout, and having tools like Black and PyLint help you to achieve that level of consistency.
