Ideally you run the tests of a project often (on every commit) in a clean environment. This way you can make sure that not only the product works properly, but also that the buildout of the product works on a clean system. That last part is really helpful during sprints: if someone cannot get the buildout running, it’s most likely the problem of the machine of the developer, not the product itself.
If you haven’t seen the tutorial.todoapp then definitely check it out. It shows some great best practises, including tests and CI.
Travis is free for any public GitHub repository. You can also have paid subscriptions for private builds.
Travis already ran 757k tests for +10k open source projects. They are crowd funded: +120k USD from +700 people. They support a lot of programming languages and preinstalled DBs (you only have to enable them).
Travis is very easy to setup. To get started you:
.travis.yml file..travis.yml file.Example of a very simple .travis.yml file:
language: python
python:
- 2.7
install:
- python bootstrap.py
- bin/buildout
script:
- bin/test
(Note that because the tests are run from a clean Ubuntu install, you have to have a buildout configured for you project.)
With Jenkins it is harder to get started. With Travis the minimal configuration is quite simple. But you can get more complex if you want to. The drawback is that you only get the console output. With Jenkins you can get more information (e.g. coverage reports).
There are a number of services on the default Ubuntu image that is used. You just have to enable them. For instance, the X Virtual Frame Buffer.
You can whitelist or blacklist branches of your repository. This
prevents errors when you are just trying things out. If you want to
skip a specific build from being tested, you can add “[ci skip]” to
the commit message.
Build notifications can be sent. Mail and IRC are the most common ones, but many more options are available.
There is
Travis browser extensions
that will show you the Travis status of a project when you go to the
page. If you want to, you can also create a status image
(https://travis-ci.org/[YOUR_GITHUB_USERNAME]/[YOUR_PROJECT_NAME].png),
e.g for on your PyPI page.
You can also have pull request testing. If someone issues a pull request from a branch, Travis merges the code and you get a notification whether the pull request is good to merge because the tests succeed (or not). You can even see the status per commit (if there is some back-and-forth on an issue for example).
Limitations of Travis: there’s a 15 minute build runtime limit.1 The limit is for the entire build—from start to the end of the test. And since the build starts from scratch every time, this may be an issue. And again, the reporting is still limited.
Tips to speed up the build:
socket-timeout = 3” and
“allow-hosts = ...” to the [buildout] section)Nejc would like to see Travis and Jenkins go hand in hand in Plone.
Advantages of Travis CI:
View the slides or watch the video.
Update (2021-08-02): This information also looks outdated. Please refer to the Travis CI documentation ↩︎
The book is written for Django application developers and requires basic knowledge of Python and Django. And although the code samples are indeed explained, the context is not always expanded on. This makes sense for this book and the author recommends to at least do the Django tutorial. I think you’ll get even more out of the book if you’ve got a bit more experience with building Django applications.
As the title already gives away, the book focusses on two subjects: testing and debugging. These topics are discussed while developing a survey application. Because this application is used consistently in the book, the examples require less explanation which means that the focus in more on the subject at hand (e.g. writing a unit test) instead of developing this specific application.
The first five chapters dive into the world of testing. Karen starts
with explaining how to run the tests and what success and failure
looks like. The second chapter has a focus on
doctests. Not only does
she show examples of tests, she also lists pros and cons of using
doctests. Meanwhile she also discusses whether a test is even useful
(example: you should not test Django itself). The examples in the book
consist of doctests that are placed in the docstring of functions. She
does not show how to run a doctest in a separate file
(e.g. README.txt) which can be more useful for explaining or
demonstrating the API of the application as a whole, or doing
integration tests.
Unit tests are
discusses more extensively in chapters three to five. Karen starts by
reimplementing the doctests as unit tests and continues by showing
things like the setUp and tearDown methods and also
django.test.Client to test views. In the last chapter about testing
she introduces the reader to test runners, test coverage and using
twill instead of
django.test.Client. The only thing I want to nag about is the style
of some of the examples. Occasionally she writes things like:
def testHome(self):
from django.core.urlresolvers import reverse
...
Where I would write:
from django.core.urlresolvers import reverse
def testHome(self):
...
In my opinion every programming book should pay attention to programming style. Although it does not have to be addressed explicitly, the style of the examples should be consistent and follow the appropriate coding standards/guides (like PEP8 for example which states that imports must be at the top of the file).
Anyway, on with the book review… The remainder of the book is about debugging and troubleshooting. Karen first describes the functionality Django offers to debug your application. For instance, the debug error pages are explained in detail. The style Karen chose for this part of the book resembles watching a programmer at work: a mistake is made and results in an error page, the error reported by Django is investigated to discover the cause of the problem and the code is fixed.
Next subject is how to find errors that are less obvious: Django does not show an error page, but the application does not work as it should. Initially some custom code is created to show the SQL queries involved in a view. Although this code is replaced by the Django Debug Toolbar later on, it is also used to show how to package the custom code for reuse. (Personally, I would have kept the custom query display section a bit shorter and skip the packaging altogether. It’s beyond the scope of the book and I think there are other resources that describe developing for reuse much better.) Karen does a great job describing the features offered by the Django Debug Toolbar. She also explains why using print statements is evil and using logging is good.
The next subject (chapter nine) is the Python debugger (pdb). Not only
is this chapter a thorough description of the debugger, there’s also a
number of pages devoted to
race conditions.
Although this is an important subject, I personally sometimes forget about it
when developing an application. So I think it’s great that it the subject is
brought up in this book!
Chapter ten (“When All Else Fails: Getting Outside Help”) is a nice description of what you should do when you encounter a problem in Django. Karen points the reader to the Django documentation (to confirm you are using the API as it was intended) and Django’s bug tracker (where you might already find a fix for your problem). She also gives clear instructions and tips on what to do when the problem isn’t already fixed or perhaps hasn’t even been reported yet.
The last chapter of the book is about moving your code to production and making it accessible for the world. It gives an example for the required WSGI script and the Apache configuration. Furthermore, it also describes debugging the production environment and load testing with Siege.
Finally Karen discusses the use of Apache and mod_wsgi during development. Although she claims that the Apache/mod_wsgi setup can be made nearly as convenient as using the Django development server I think it really depends on the situation. I agree that it may be a good way to tackle some problems and setup issues early on. But if you are working with a number of developers on the project and each developer is working on several different projects (sometimes even switching projects throughout the day), I’m not sure whether it’s worth the effort.
I’ve been developing in Python a couple of years, mainly working on Plone projects since 2007 and doing Django applications since 2009. I was already more than familiar with the topics in the book (doctests, unit test, logging, debugging, et cetera) and have also done a bit of test driven development. As a result, the book did not open a whole new world for me. However, I do think that the book was worth the read. I picked up a few new tricks and gained an even better insight in testing and debugging Django applications.
One last complaint though: the title of the book could have been a bit better: by putting the Django version number in the title, the book seems to target Django 1.1 specifically. Sure, Django 1.1 and 1.1.1 have been used in the book. And sure, some examples might not work in earlier or later versions of Django. However, the principles remain the same! In my opinion the version should have been left out of the title.
Although I may not agree with everything, I think Django 1.1 Testing and Debugging is a good book. So I’d like to thanks Karen and Packt for writing and publishing it.
]]>The first article I read is called It’s OK Not to Write Unit Tests and is written by Chris “cashto” Ashton). Although Chris isn’t saying that all unit tests are worthless, he attributes less value to them than they deserve, at least in my opinion. For the projects I’ve been working on, unit tests can actually be very useful.
For example, I’ve frequently caught bugs after refactoring because a unit test failed. Does this make me a bad programmer? Unit tests can also be used to define the API of a certain piece of code. By writing unit tests in the form of doctests you are forced to think about the way something should work and at the same time document it.
The other article I’ve read is The Problem with Unit Testing. Although I like test driven development, I wholeheartedly agree to the view that 100% unit test coverage is probably overkill in a project. You’ll constantly have to evaluate whether writing a unit test for a piece of code is worth it. For code that is more or less isolated (only used in a tiny part of the project), very straight forward and easy to debug (worst case scenario), I’m likely to skip writing tests for instance.
Note that I’m only talking about unit tests here. There are many more ways to make sure the quality of your code is what it should be.
]]>When developing a Zope based application, the
Products.PrintingMailHost
package can really help you out. By including this package in your
setup, the Zope MailHost class is patched so no actual emails are
sent. Instead the content of email is printed to the standard output.
But when working on a Django application (or any other non Zope
project) there is no MailHost class that can be monkey
patched. Python’s
smtpd module to the
rescue. The first step is to configure the application to use
localhost as the SMTP server on a random port (say: 1025). Next, go to
the command line and type:
python -m smtpd -n -c DebuggingServer localhost:1025
Just like the PrintingMailHost, this SMTP server prints the emails to standard output. For more information see the “Testing e-mail sending” section in the Django documentation.
mail_debug management command. This commands starts the same SMTP
debugging server, but you don’t have to remember the right
incantation.
Somewhat related to this: if you want to test your application against an HTTP server, you can use this command:
python -m SimpleHTTPServer
The SimpleHTTPServer module can be used to get server up-and-running quickly. It can also be a simple way to, for instance, copy files from one machine to another. By running the HTTP server in the directory containing the files, you can access the files via your browser on another machine. Safe? No. Convenient? Yes.
]]>