My takeaways

This was my first AI Engineer conference and also the first AI conference I attended. It was a nice way to gauge where both I and we as a company stand. This is especially true when I realise that mostly (only?) companies at the forefront of these developments were there. AI is hot, but as we have heard, only 12% of the (big) companies are “AI achievers”.

What I think the speakers agreed on:

• The role of the engineer is shifting. Less coding, more writing specifications, planning and reviewing
• Codebases must be designed for humans and agents to read and understand
• Guardrails are essential. If we want to give agents more freedom to solve problems, we need to give them a certain amount of freedom, but we need guardrails first
• We need to have feedback loops. The quality of the feedback loop determines the quality of the output
• The human stays in the loop (at least for now)
• Introducing AI is a process. Start with non-critical, well-scoped tasks and let the system “prove” itself and earn trust.

There are also things the speakers do not agree on:

• Code is free vs Code is not cheap
• Move fast, have the agents do the full job and remove humans from the loop to speed up vs slow down and think

Other takeaways (that were not explicitly mentioned by multiple speakers):

• Agents are bad at self-evaluation
• Only the first ~100K tokens of context are useful. More context will only make the agent dumber
• Agents are consuming APIs, documentation and websites and might even already make up a large portion of their users
• I should probably use more skills

In general, I think we, as an industry, are still figuring out how to work with AI. This is also directly related to the rapid pace at which things are changing. Something may work today, but may be obsolete next month.

The conference

Overall, I liked the atmosphere and energy. The organizers managed to get a great bunch of speakers on stage who delivered insightful content.

I’m a bit on the fence about the workshop day. I really liked that the speakers had more time to go in depth (which is hard in the 20-minute timeslot of the breakout sessions). On the other hand, I was expecting to do more “work” in a workshop. However, most talks I attended were just that: longer talks. And if it were not for taking notes, I would not have needed to bring my laptop.

Having said that, I had fun, learned a lot and would love to go again next year.

The trip

This wasn’t my first trip to London, but it was my first conference there. The conference was held in the Queen Elizabeth II Centre, which is in the center of the city. And especially since we stayed in a hotel close to the venue and could walk over there each day, I was very aware of where we were, which was great. (I like London, can you tell?)

This was also my first time going to a conference with such a big group. Sixteen people from Schuberg Philis were there! This also meant that for most (perhaps even all?) of the sessions I joined, I was not the only one from our company in the room, which means I could discuss what we’ve heard and how it applies to our company and customers. This added a lot of value for me.

And it’s also nice to get to know my colleagues a bit better, especially the people I don’t work with on a day-to-day basis. And while we talked a lot of shop there, there was also time and space to bond on a more personal level.

Conclusion

I think I can say that we, as a company, are in a good place with how we are adopting AI and helping our customers. Having said that: AI engineering is a field that is still very much developing. So we’ll continue to learn and adapt.

As for myself: I’ll work, together with my colleagues, on integrating the takeaways into my everyday work.

Skills

First of all a designer needs to be creative. After all, they are the person that needs to capture the ideas of the client and visualize them. However, it is also very important for the designer to understand the web. Something that works in print, may completely fail in a browser. One of the reasons is that print media is meant to be seen/read, while websites need to be interacted with. Closely related is anticipating user generated content. This means that a design should also look great with less (or more) text than the lorem ipsum in the design.

I think these skills are not disputed. But what about the ability to code? I agree with Lukas Mathis that there are risks when designers are also involved in implementation. In his essay Designers are not Programmers he more or less says that designing and coding are two separate worlds. To be able to be good at designing, you’ll have to ignore everything you know about coding. Otherwise the design is restricted by technical limitations: you know what you can implement and you’ll design within those boundaries. Even worse: by already thinking ahead about the way it’s going to be coded, the focus will be on the code instead of the user experience.

Photoshop or HTML?

A related discussion is what the deliverable of a design should be. In most of the projects I’m involved in, the design results in a Photoshop file. This file is then cut to extract the needed graphics and the HTML/CSS is coded. I guess you know the drill.

However, quite often the role of designer is combined with the role of front-end developer, especially in smaller shops. In these cases it can be easier to do the design in HTML directly, as described by Meagan Fisher in Make Your Mockup in Markup. (The Django package django.contrib.webdesign even helps by generating sample text.¹) One advantage is that some design changes are easier to make in CSS than in Photoshop. Depending on your skills, the whole design process may even be faster. Meagan also demonstrates that CSS3 gives you the ability to create a lot of effects without having to resort to images, which means you’ll have to spend even less time in the graphics editor.

Another advantage of designing in HTML is that the client can see how the design works in the browser. You have to be careful though: since it’s only a design, it’s very likely that the code is not cross browser compatible yet. If the client uses a wrong browser, the design may not come across as intended. (To prevent this, you could export the page as an image e.g. by using the Firefox add-on Screengrab. no longer available)

You’ll also carefully have to manage client expectations. If the design is done in HTML, the client may incorrectly assume the front-end work is done. They may not appreciate the time that is still required to make the site look good in all targeted browsers. Or the time needed to make the static HTML more dynamic by integrating the application or adding AJAX effects.

However…

Perhaps the most important skill of a good designer is being able to communicate. Not just because the design should communicate the right things, but also because communication with the client and development team is, in my opinion, the key to success.

The first article I read is called It’s OK Not to Write Unit Tests and is written by Chris “cashto” Ashton). Although Chris isn’t saying that all unit tests are worthless, he attributes less value to them than they deserve, at least in my opinion. For the projects I’ve been working on, unit tests can actually be very useful.

For example, I’ve frequently caught bugs after refactoring because a unit test failed. Does this make me a bad programmer? Unit tests can also be used to define the API of a certain piece of code. By writing unit tests in the form of doctests you are forced to think about the way something should work and at the same time document it.

The other article I’ve read is The Problem with Unit Testing. Although I like test driven development, I wholeheartedly agree to the view that 100% unit test coverage is probably overkill in a project. You’ll constantly have to evaluate whether writing a unit test for a piece of code is worth it. For code that is more or less isolated (only used in a tiny part of the project), very straight forward and easy to debug (worst case scenario), I’m likely to skip writing tests for instance.

Note that I’m only talking about unit tests here. There are many more ways to make sure the quality of your code is what it should be.

The Joel Test was devised to rate the quality of a software team. In contrast to other methods, it is designed to quickly provide some basic insight in how a team is doing. And although Joel used the words “highly irresponsible” and “sloppy” I though it was fun to see how we at Zest rate.

We mainly use agile development to create buildout based Plone applications. In answering the questions, I’ll focus on those type of projects.

(Disclaimer: obviously this represents my own view on the matter and is not necessarily the opinion of Zest or any of my co-workers.)

1. Do you use source control?

Yes! I cannot even imagine working without it. We mainly use Subversion but some of us are (or have been) experimenting with other source control software, like Bazaar , Git and Mercurial.

2. Can you make a build in one step?

Hmmm… tough one. We don’t really “build” in the traditional way. Since Python is an interpreted language, we don’t need to explicitly compile. For instance: restarting Zope is enough to use the latest code (and with plone.reload restarting isn’t even necessary in many cases).

But okay sometimes you’ll have to rerun the buildout to get/update all the dependencies. Rerunning buildout is a single step so I’ll count this as a yes.

3. Do you make daily builds?

In the way Joel describes this, it basically translates for us to doing a clean checkout, running the buildout and then running the tests. (Assuming the tests include integration tests which need the whole Plone/Zope stack.)

But no, we don’t do this. We did have buildbot running for a couple of projects to run the tests after every commit, but I have to admit we are not doing this often enough.

4. Do you have a bug database?

Yes we do. If we encounter a bug, we store it either in a (user) story or we put it in the issue tracker associated with the project.

5. Do you fix bugs before writing new code?

This depends on the project. Most of the time we try to estimate the time needed to solve the problem and report back to the customer if we expect that the fix will take a significant amount of time. Since in Agile development the customers set the priorities, they can decide that the bug is less important than getting a new feature in the next release.

For our own projects or community projects we do want to make sure we ship code as bug free as possible and in general we give bugs a high priority.

6. Do you have an up-to-date schedule?

For each project we maintain a list of features in our project management tool. Each feature (or story in XP terminology) has an estimate of how much time it will take to implement. So we can determine when the project will be done.

However, at each iteration meeting the customer could decide to add new feature, change the priorities, etcetera. So the schedule is in no way set in stone.

7. Do you have a spec?

Each story in the project represents a single feature. Since the set of features for a project isn’t static (new ones can be added, others can be dropped), we don’t write a lengthy document specifying every detail of a project. We tend to describe the feature in a couple of sentences in the story. By splitting it up in tasks when we start working on it, more details are added.

(By the way, not writing specifications and justifying that by saying “XP doesn’t need specs” is wrong in my opinion. You do need to have other practices in place to compensate the lack of written specs.)

Another characteristic of agile development: we only build what is requested. That is: we don’t try to accommodate for every possible future use case of a feature. We try to find a balance between iterative design and thinking things though enough to limit the refactoring needed when more use cases have to be implemented.

And since we have iteration meetings every (other) week where we demo and discuss the work done in the previous iteration, the customer has the opportunity to refine their ideas.

8. Do programmers have quiet working conditions?

No we don’t. Which is a good thing when sprinting on a project: by having all developers in the same room it is incredibly easy to exchange knowledge. Pair programming also helps here. In my experience if one of the partners gets distracted by the discussion (e.g. to answer a question) the other half of the pair can quickly get him focussed again.

When we are not sprinting though, it can be a bit hard to focus on your task. Especially if co-workers are discussing a project on which you aren’t working right now, but you are familiar with. (Perhaps I’m overly curious and therefore more easily distracted…) However, at the office we do have a place where you are isolated from the other developers and it is also possible to work from home.

9. Do you use the best tools money can buy?

If we need tools and our demands are reasonable, sure. It just happens that many of the tools we think are the best, are open source tools. (Although it also depends on the platform that is being used. I’ve got the feeling that the Mac users at Zest use more proprietary software than the Linux users.)

10. Do you have testers?

No we don’t. This doesn’t mean our software is deployed at random though. First of all we try to write automated tests (unit, functional and integration tests), although I do have to admit that this heavily depends on the quality level the customer asks for. But this is a different discussion.

Furthermore: usually the project manager tests the application to see whether it meets the requirements. Then we demo it to the customer. And finally: most of the times we first deploy the code to a preview server where the customer can play around to test it.

11. Do new candidates write code during their interview?

No, as far as I know we don’t require candidates to write code. (I certainly didn’t have to do it. :) ) But at the same time we also haven’t yet hired developers that didn’t meet up to our expectations. Perhaps we don’t have as high standards as e.g. Joel does, perhaps it has to do with the fact that we are a small company. I don’t know.

12. Do you do hallway usability testing?

It’s no excuse, but this is hard to do this if you’re in a small company. I think most co-workers are too “technology infected” by now to give a completely objective opinion. If you’ve been working with a system long enough (e.g. Plone) you get used to it’s quirks.

However, we do frequently request co-workers to give a second opinion about the (user) interface. And since we work in short iterations, the end-user also gets to work with the software shortly after writing it. So any strange behaviour should be detected quickly.

Conclusion

Perhaps I’m not being objective, but I think that some of the questions from the Joel Test don’t fit in our development process. I therefore don’t want to calculate a score but let you draw your own conclusions. I certainly found a couple of areas where we can improve…