Paul had a project where PDFs were workflowed to several sites using MultiSite. This meant there was lots of duplication. After migrating to Plone 4 and blob storage, there was still a lot of data.
He created a script (bscompress) for:
In his case, the blob storage went from 14625 Megabytes to 2395 Megabytes.
Things still to do:
ZODB/src/ZODB/blob.py.Dylan Jay talked about attracting more new people to get started in Plone. Dexterity and PloneIDE are great, but we need more. David Glick started to build a theme editor (plone.app.themeeditor). It uses jQuery UI and is a lot better than the Zope Management Interface (ZMI). There is going to be a PLIP about it.
Johannes made a new release of plone.app.event. Please take note of the installation instructions. They include configuring the right time zone.
Johannes did a quick demo of the package. There are still issues, but it’s safe to call it a beta version.
Johannes thanks all the contributors for their work.
Mark: note that the package was moved from the Collective to the Plone organisation on GitHub on October 13th.
All the team leaders (security team, UI team, etc) went on stage so people would know who is who. And whom to talk to if you want to get involved.
His company (4teamwork) created ftw.tabbedview. It can be integrated with collective.js.extjs, collective.quickupload (if installed) and ftw.table.
The package ftw.tabbedview contains filtering, sorting, grouping,
managing columns, batching, flexible sources (catalog, SQLAlchemy,
…), data transformation (icon, authors, dates, …) and is highly
configurable. It can also be used on pretty much any context.
Philip helped to organise the Plone Konferenz München 2012. About 150 people attended and there were lots of new people. The goal of the conference was to attract decision makers. It was a very successful conference.
Tip: include a sprint lounge where people can sprint the whole conference. This way you can attract people that want to code and work together, but do not speak the language the conference is held in.
Such local conferences increase the visibility of the Plone community and can attract new members to join the community.
They were able to donate € 3000 to the Plone Foundation.
Solving problems in plone package at a time:
[Mark: The video of this lightning talk is definitely worth watching.]
If we have to believe the Mayan symbols, this year will be the end of the world. What better way to experience that day than by joining a sprint? So join the “End of the world” sprint in Ushuaia, Provincia de Tierra del Fuego, Argentina on December 21st, 2012.
Whiskers is a Pyramid application that collects all packages used in your buildouts. That is, if you use buildout.sendpickedversions. This way you can quickly see which buildouts use a certain version of a package. You might even be able to use this data to find out which eggs in your buildout cache are not used anymore.
A new solution to change how we collaborate: desk surfing. There is a Plone core developer dedicated desk in São Paulo. (You can also use it if you are not a core developer by the way.) If you are in São Paulo go there and contact Érico.
(Calvin Hendryx-Parker adds that you are also welcome in Fortville.)
Since Calvin missed is own presentation earlier that day, he did the three minute version of his talk “Playing With Blocks: 6 Pro Tips for Building Your Next Super-Charged Plone Site”:
The talk he would have held this morning will be recorded and put online.
STX Next created a small addon for Plone: stxnext.grayscale. This package transforms content of a page to grey scale. It has been developed for Polish user requirements related to national mourning after a tragedy has occurred. Governmental sites and bank sites change their colour scheme to grey scale out of respect. The images are cached on file system and are thus not generated each request.
A talk about amplify.cc, a prototype to replace PostNuke based community portals. It uses only one content type. They use Dexterity behaviours on content instances instead of on a class. Their conclusion is that having one content type for everything is possible and even works really nice.
A lot of countries have their own localised Plone site, e.g. Italy, the Netherlands, Brazil, Spain, France and Poland. They all look different. But by using a bar at the top of the page which links to other Plone sites in the world, we can unite those sites and make it easier to discover them.
]]>A sprint is a great way to discuss a problem with a package. If the maintainer is not at the sprint him-/herself, then usually someone involved in the package is available.
The most important rules:
Basically a sprint is a lot of people who can do anything they want. That could be a personal project, a PLIP, a package, etc. Everything is fine. You can also join a sprint topic of someone else. Just do what you want to do or find interesting. Either way: ask questions.
People always seem very busy. But people prefer helping you fixing stuff than fixing things themselves.
There’s a theory: whenever there are more than ten Plone people in a
room, they start moving the tables. :).
Sprinting doesn’t stop with coding. It’s the other way around: the non-coders might even be more productive. A sprint is also a great opportunity to discuss problems or approaches.
]]>This is the second half of Cris’s previous talk. This part is about using and extending ZopeSkel. By creating your own templates you can start with a customer project really fast.
Do you have code you write over and over again? Is the code almost the same each time and are the places where you change stuff always the same? You can avoid writing all that code by creating a template.
You can write the template the way you want. This is just a system. This means you can version control your own process by documenting it in a template.
The first question you need to ask yourself is “what do I want?” The
next thing you will want to investigate is what you have already got.
You can use “bin/templer --list” and “bin/templer <template_name> --list-variables” to find out more about the templates ZopeSkel has
to offer. Then have a look at the code provided by the template.
To wire everything up, you need to do four basic things:
Cris has already written the documentation for this tutorial. It currently lives on his laptop, but it should be posted online soon. I (Mark) decided to just write down some notes instead of trying to replicate Cris’ documentation.
The related documentation can be found here:
Tips:
python setup.py develop” (so “develop” instead of “install”).templer
namespace; for your own project/infrastructure specific
templates, use your company’s namespace.We used to have ArchGenXML to generate code for us. But it only created (Archetype) content types. This may have been the reason for content type centric development. Later we needed eggs and other scaffolding and thus another tool.
On May 25th 2006, the first commit to ZopeSkel was made by Daniel
Nouri. From there it grew organically. Then local commands happened so
developers could enhance already created packages (for instance to
include portlets, content types). But most templates asked too many
questions. And the descriptions around the question were not always
clear. Many templates looked similar so which one should your choose?
There was way too much (repeated!) code. That code was not updated
consistently so for example one template generated a tests directory
while another created a tests.py file.
To have local commands, your package has to depend on Paste. As a
result, the Paste, PasteDeploy and PasteScript eggs end up
inside your product. Even worse: if you are not careful, you can
accidentally add these packages to your version control system. This
is a big problem.
On the No-Fun ZopeSkel BBQ Sprint in 2009 the question “Who is ZopeSkel for?” was answered. It was decided that is it primarily a tool to help newcomers. For experienced developers it’s easier to start with empty structure and fill the bits and pieces in themselves.
The outcome of the BBQ sprint was the “bin/zopeskel” script which:
paste create -t” commandHowever, the existing documentation was not cleaned up. As a result
people still use the “paste create -t” command and don’t even know
what the “zopeskel” command is for.
Another outcome was that Cris became the official maintainer. And the plan was formed to break up ZopeSkel (see the GitHub repository which still contains a document with the reasoning behind this).
Work to break up started in early 2010. Templer provides templates in packages of related functionality: a buildout package, Zope related things, Plone related things, etc. But the commands for the user should remain the same because there’s so many documentation already available.
Main difference between the old ZopeSkel and Templer: you cannot create a shared structure in ZopeSkel. Templer provides structures, which are similar to templates but they cannot be run individually and only exist to be included in templates. For instance you can pull the structure for the requested licence when creating a package.
The new ZopeSkel version is a Templer application which bundles
stuff so you can use it as you were used to. You can just use
easy_install or pip to install ZopeSkel (your version of pip
needs to be new enough though).
Or put it in your development buildout by including a part which includes PasteScript and ZopeSkel, for example:
parts =
...
zopeskel
[zopeskel]
recipe = zc.recipe.egg
eggs =
PasteScript
ZopeSkel
The next talk teaches how to create templates, in the sprint following the conference you can actually create templates.
Note that Templer currently still depends on Paster and uses Cheeta templates. Crushinator wants to solve these problems. The only dependency would be a package for the templating language you want to use for your templates.
]]>David split up the changes into categories. But first something about the process.
A PLIP (Plone improvement proposal, basically a ticket) first needs to be approved by the Framework Team. Once it has been approved, implementation starts. When the code is ready the Framework Team does a review. If the Framework Team says “yes it’s okay” there is still more work to be done before it is completely integrated in Plone (like final documentation, small changes).
PLIP criteria:
plone.app.collection yet.).zip
file with the theme which contains your rules.xml file. You can
also edit a theme with the ACE editor. And a rule builder to spit
out the Diazo rules file.plone.app.dexterity[grok,relations] in your
eggs. Both the grok and relations parts were not ready for
default inclusion but are available and useful. If you used
Dexterity already, definitely include relations.placeholder attribute).z3c.form support to plone.app.portlets (Plone 4.3).<body>. (Plone 4.3) to make theming
easier, e.g. when using Diazo.on” instead of
using “live” or binding by event name.Tip: upgrade to DateTime version 3.0: it uses far less memory!
plone_deprecated skin is no longer enabled (Plone
4.3). It is still there but it is just not used by default.There are bunch of packages under consideration for Plone 4.4, for instance:
Interesting PLIPs David would like to see:
portal_actionsportal_skinsportal_cssportal_javascriptsz3c.form instead)portal_properties (move them to plone.app.registry)David asks us to get involved in building the next version of Plone:
What will be the next version, 4.4 or 5? If there is backwards incompatible stuff we want to include, it’s probably going to be version 5. Then again, releasing version 5 might also ‘just’ be good for marketing reasons. In short: David also does not know yet.
]]>Ideally you run the tests of a project often (on every commit) in a clean environment. This way you can make sure that not only the product works properly, but also that the buildout of the product works on a clean system. That last part is really helpful during sprints: if someone cannot get the buildout running, it’s most likely the problem of the machine of the developer, not the product itself.
If you haven’t seen the tutorial.todoapp then definitely check it out. It shows some great best practises, including tests and CI.
Travis is free for any public GitHub repository. You can also have paid subscriptions for private builds.
Travis already ran 757k tests for +10k open source projects. They are crowd funded: +120k USD from +700 people. They support a lot of programming languages and preinstalled DBs (you only have to enable them).
Travis is very easy to setup. To get started you:
.travis.yml file..travis.yml file.Example of a very simple .travis.yml file:
language: python
python:
- 2.7
install:
- python bootstrap.py
- bin/buildout
script:
- bin/test
(Note that because the tests are run from a clean Ubuntu install, you have to have a buildout configured for you project.)
With Jenkins it is harder to get started. With Travis the minimal configuration is quite simple. But you can get more complex if you want to. The drawback is that you only get the console output. With Jenkins you can get more information (e.g. coverage reports).
There are a number of services on the default Ubuntu image that is used. You just have to enable them. For instance, the X Virtual Frame Buffer.
You can whitelist or blacklist branches of your repository. This
prevents errors when you are just trying things out. If you want to
skip a specific build from being tested, you can add “[ci skip]” to
the commit message.
Build notifications can be sent. Mail and IRC are the most common ones, but many more options are available.
There is
Travis browser extensions
that will show you the Travis status of a project when you go to the
page. If you want to, you can also create a status image
(https://travis-ci.org/[YOUR_GITHUB_USERNAME]/[YOUR_PROJECT_NAME].png),
e.g for on your PyPI page.
You can also have pull request testing. If someone issues a pull request from a branch, Travis merges the code and you get a notification whether the pull request is good to merge because the tests succeed (or not). You can even see the status per commit (if there is some back-and-forth on an issue for example).
Limitations of Travis: there’s a 15 minute build runtime limit.1 The limit is for the entire build—from start to the end of the test. And since the build starts from scratch every time, this may be an issue. And again, the reporting is still limited.
Tips to speed up the build:
socket-timeout = 3” and
“allow-hosts = ...” to the [buildout] section)Nejc would like to see Travis and Jenkins go hand in hand in Plone.
Advantages of Travis CI:
View the slides or watch the video.
Update (2021-08-02): This information also looks outdated. Please refer to the Travis CI documentation ↩︎
Plone is currently pretty fast, especially for low edit sites. It runs fine on a 256Mb VM. Things that help us: Varnish, plone.app.caching. It’s easy to setup. Portal implementations are different because there are a lot of authenticated users there.
But that’s for integrators and users. What about product developers?
We’ve got a blind spot for performance and security. We don’t deliberately write ‘bad’ code. We do think about our data structures (some even overdo it).
A recap on tips from an earlier presentation Matthew gave:
FieldIndex.OOBTree.Lengths of things you need
instead of iterating over all items just to find out how many
items there are.For loops inside for loops are a recipe for slow code.sets and itertools. They are good.Very few people use already available tools. For instance: a show of
hands proved that cache:ruleset ZCML declaration are not used
much. If you do use this, you can get an enormous performance
boost. You can go from 79 milliseconds for a request without
cache:ruleset to zero milliseconds with the ruleset. You can
directly assign a ruleset to an interface. Integrators can help here
by adding these rulesets to products you use.
Edge side includes (ESIs) are generally used to do package composition. But it’s easy to leak data so be careful. An ESI is a way to say you want to include a certain URL on a certain location. That way you can cache the rest of the page and include e.g. the user name dynamically. Again, make sure you don’t cache secret stuff for everyone. Don’t trust on ETags since they can be faked. You will have to have a custom VCL and code. ESIs can give your site a good boost, but it’s also hard work. It’s a lot easier if addons are written with ESI in mind. So plan your templates for ESIs:
Integrators should not need more than a couple of minutes to add ESI to your templates.
As said, it’s easy to make a mistake regarding testing. So you really
need to test the caching. This goes for integrators, but also for
product developers. Far too few people run automated tests on their
caching. If you already have plone.app.testing based tests, it is
easy to do: create a new layer using a ZServer, start up Varnish and
make some testbrowser calls.
We need helper classes and methods for writing cache tests and actual tests in popular packages. That way they can serve as an example.
The current plone.app.caching rulesets are targeted at
integrators. Product developers need a series of rules which we agree
to use so configuration would be easier.
Want to help put this into practice? Mail to sales at
thedistillery.eu.
SlickGrid Touch: Making a complex Javascript table widget work on mobile devices. Although you’ll want your website to work on mobile, older Javascript widgets don’t work on touch devices. SlickGrid doesn’t support touch devices.
Touch support means that you are able to select/unselect, follow link, select row, show menu, scroll, reorder row, delete row.
Maurizio is a project manager. He needed a project management tool. ScrumDo is a Django tool, but integration with Plone is hard. They created a Dexterity based tool and also want to sprint on it on Saturday. Have a look at the simplemanagement repository.
The Python Software Foundation created Python Brochure. 32 pages A4 full colour. Their aim: 10.000 distributions world wide. Diverse target groups. Check out brochure.getpython.info.
Sometimes you can easily fix your own system by watching your
logs. Plone logs you want to watch are located in var/log. The two
interesting logs are event.log and instance-Z2.log. You can also
find them using e.g. “locate event.log” on *nix based system.
On Linux systems also check out /var/log. Interesting files are
e.g. faillog, messages and secure.log or auth.log.
You can monitor a log file with tail -f.
Need help with your logs? Ask Elizabeth.
Martijn presented Crom. Basically it is zope.component redone: the same API but less cruft. He gave lots of code examples. Unfortunately it’s backwards incompatible so we cannot use it in Plone.
Fuzzy search in Plone. In an intranet the could not use Google but they did want to have similar suggestions. They did not want to use Solr because they found it too hard to install and config.
Their solution uses the Levenshtein distance, is Python only and the Japanese language is supported (using MeCab).
They reimplemented CorkboardMe in Plone for use on an intranet: collective.noticeboard. It is currently in a personal repository but it will be moved to the collective mid November once it is finished.
Lars created a post install hook for virtualenv:
buildout bash completion. With
it you can easily add the bin directory of your buildout to you
path. It also contains some commands that make working with a buildout
and virtualenv easier.
They had 460GB of data in an Oracle database. Everything was stored three times and 90% of the data was not used. After they cleaned up, ‘only’ 130Gb was left. They dumped the binary data on file system and exported the relational data to a new database with a sane structure. They used transmogrifier. For the files (that were already on file system) they created stubs in the ZODB and then moved the files to the right location in the blob storage.
Jamie showed a product he is working on. It shows which Diazo theme rules work and which do not by colouring them green and red. The product is not yet available.
How to make Plone fast? Avoid Plone and serve Javascript and CSS directly from file system. See collective.assets and the webassets.
Eric presented Resurrectio, a Chrome extension. From the documentation: “Resurrectio is a Chrome extension allowing to record a sequence of browser actions and to produce the corresponding CasperJS script.”
His company created a book publishing addon: ftw.book. It uses ftw.pdfgenerator (which is based on LaTeX) to generate PDFs out of content.
They used it for financial reports, court reports, end user product documentation, …
There is an Ubuntu bug in the network manager since 2008. To prevent
issues to remain open for that long, he launched
fundhub.org today. It’s a crowdfunding site to
fix free software one bug at a time. It currently only supports
bitcoins because paying is hard and bitcoins made it easier.
Multiple pledgers per issue are possible.
]]>When you want to get started with Plone development, you need to know where to get help:
#plone on Freenode)To get started with a buildout, theme or product, use ZopeSkel or Templer. Also check out the local commands these provide.
To speed up buildout, you can:
allow-hosts to limit where buildout is looking-t 5.-N so buildout doesn’t look for newer packagesUse mr.developer to check out the code
of eggs in your buildout so you can easily develop on them. (And to update them
all at once when you continue working on a later moment, use bin/develop up.)
The package sauna.reload picks up changes in the files of your packages and restart Zope automatically. But somehow Zope is restarted very quickly. For Plone 3 use plone.reload, which requires manual action to reload Python code or ZCML.
With collective.recipe.omelette it is easier to look into the code of Plone and third party addons.
plone.app.debugtoolbar shows debug information and it allows you to do stuff you normally need to do from the ZMI.
plone.app.themeing allows you to install a zip file with the theme (works with Diazo).
If a site is unresponsive, you can use
mr.freeze to
investigate. You can use it to print a stack trace for all threads so you
can see what’s going on. Kim used to use
Products.signalstack,
but mr.freeze can do more, like drop Zope to a pdb debug prompt.
Use jarn.mkrelease or zest.releaser to release a package. It beats performing all the steps individually by hand.
Continuous integration with Travis CI for open source GitHub projects or Jenkins for internal company projects. Travis can be used for closed source projects but they don’t advertise with it and it’s quite expensive.
plone.api is still under development but already very useful!
products.pdbdebugmode allows for post-mortem debugging, but it can conflict with sauna.reload.
Finally: see collective.exampledevtools for more details.
]]>A few years ago Gocept already did hosting, but it was neither ideal for the customers nor for Gocept itself. They changed that and this talk looks back on that.
Infrastructure basically marries software with entropy. The developer environment is clean and you can always start fresh. Operations can only start clean exactly once. Infrastructure make handling the entropy bearable. For Gocept, infrastructure is building a platform that the application developer can count on.
The first thing to have is a data centre. At Gocept they just buy that; they do not want to manage the building, air conditioning, etc. They take a data centre for granted and plug in their hardware. The are four types of hardware: routers, storage servers, virtualisation servers and backup servers.
Everything is run in a virtualised environment (which is really affordable). The OS is also maintained by Gocept. There are only a few components an application developer or customer can change. They rather run five smaller Zope servers than a single bigger one. They try to make all the pieces as small as possible.
Automation is handled via Puppet. They want the automation to be convergent and idempotent: regardless of the initial state it should converge to a certain state, no matter how many times the scripts run. The scripts should also be versioned and have everything as code so there is not much manual work necessary. Puppet is triggered by cron but they are thinking about moving it to some sort of message queue so machines can figure out if they are affected by the change and can act on it.
Management was binary: either I manage something or you. But there are also tools where you can allow someone to add a snippet. Backups and other day-to-day operations also fall under management. It’s not something a application developer should need to worry about.
For monitoring they use Nagios. It’s good enough.
You need a help desk for support. One reason is that you want to shield your developers from getting interrupted continuously.
There’s still a lot left for developers to do on the application level. For instance deployment. It has to be quick, repeatable, platform-independent and a single click. They currently use Batou which is inspired by Fabric.
Good components are e.g. Nginx, Varnish, HAProxy, PostgreSQL, Memcached, Supervisor and Postfix. Bad components: Apache (because although configtest says the config is okay, Apache might still crash), MySQL (BDB) and OpenLDAP.
They use a “production-ready” checklist. It contains items like log rotation, database maintenance, system startup/shutdown and monitoring (processes and ports).
One config file per environment (ini style) and they AES encrypt it. This way they can check in secret stuff into source control. Using PGP would be better because then they can encrypt it in a way only certain developers can decrypt it.
The staging environment should be identical to production. And the configuration of your development environment should be structurally identical (but you can e.g. leave out Nginx, etc).
Tips for web applications:
Use Supervisor and Superlance (httpok and memmon) to kill of processes. Use the 3.0 release candidate of Supervisor, it’s stable.
Build tips:
-t 3” to timeoutallowed-hosts=” to limit the places where buildout tries to fetch
packagesServers:
Load balancer: use “leastconn” to balance. HAProxy has live statistics UI.
Nagios: model the dependencies so a broken router doesn’t trigger many, many mails.
Use check_webpage.rb.
SLAs set expectations. How much availability do you need? That depends on how much you can afford. Don’t think in “nines”: 99% means 72 hours down/year. Does that mean you can just take down a server for three days straight and still perform according to the SLA? 99.99% is just 54 minutes per year, this is really not much time!
Instead talk about how many hours downtime per day/month are acceptable. How much does that downtime cost for the customer? By having this cost clear, it’s also clear what it’s worth.
Align technical and organisational measures (when will someone be able to handle a problem). Failover is just a temporary measure to bridge the time it takes someone can solve the problem.
You can read the book The practice of system and network administration.
]]>