Development infrastructure for small distributed projects

I was recently asked to advise on building infrastructure for an organization that was going to develop and maintain several software projects. Here is a summary.

Let us fix the terminology before we go:

• component is a piece of software, a document, or any other artifact being made;
• milestone is a component release that has a release date: software releases, deliverables, etc;

Pillars

There are two pillars of the infrastructure: tickets and wiki.

Tickets

Ticket is a brief description of something that should be done. It is a single replacement for all TODO lists, TODO emails, verbal instructions, etc. Typically, it has

• problem description
• component that should be worked on
• milestone that sets the deadline
• person set responsible for doing the work.

  People should only work within a ticket, and should not accept any work that has no ticket. Even if the boss wants you do briefly spend an hour on something... The 'What are you doing now?' question should always be answered with a ticket number.

  Are tickets a typical nerdy invention made by developers who dont want to talk to each other? Yes, developers have pioneered this idea, together with many other computer-related ideas. And yes, tickets may reduce the amount of talking, but would not immediately switch talkers into workers.

  There are different common types of tickets: fixing a bug, developing a new feature, documenting, reviewing (documentation or code), even thinking. Any activity can be given a ticket type.

  What is essential, that tickets describe what should be done, maybe with some hints on how. They are not deliverables or reports. They are not program code. They are not thoughts.

  For example, there may be a ticket to arrange a seminar. The ticket would state seminar topic, and the responsible person. Scheduling the seminar should be done via a scheduling system, such as MS Outlook. Announcing should be done via email. Slides can be done in PowerPoint. Seminar notes and reading material can be placed on a Wiki or intranet. But all these are done outside the ticket. The ticket is only to remember about the action to be taken.

  How large should be a piece of work to make a ticket for it? I would say 10 times larger then the effort to make the ticket itself. 30 minutes?

  What a surprise, there was ticket #434 created for me to write this document!

  To technically support it, we have Bigzilla and Trac as the first candidates to look at.

Wiki

Wikis come from organizations that created shared drives full of files with notes made by employees. Many international still use this system as the base for their intranet. Essentially, Wiki is a set of text documents with

• a little bit of markup, such as would be depicted as bold, and
• ability to link pages to each other, e.g. by writing something like

  Both ideas, simple text and easy linking, aim at the same goal - simplify contributions. There is only one problem with any kind of note taking - people dont want to do it. They either dont like writing, or have no time, or have difficulties with all these html, xml, latex. With Wikis the technical overhead is reduced to the minimum.

  We are used to Word, how can people ever use this primitive markup? Yes, Wiki is _not_ a publishing system. It is not meant to be. And we should not be distructed too much by the way our texts look like.

  Wiki seems to be a good choice for documents in progress. But can I freeze, say, a specification document after it is finished? Yes, most Wiki systems allow fine grained control, and also freezing some pages. Especially JspWiki and MediaWiki are good at that, Trac is simpler.

  We have two hundred people in our projects, how can we ensure that they do not destroy our Wiki? There are two assumptions here:

• people are, in general, cooperative and follow the rules. It goes beyond Wikis straight into the basics of society. When these two hundred participants come together on a plenary meeting, we would expect them to follow a number of rules, such as do not talk (loud), do not dance or stay dressed. In this respect Wikis are much less restrictive than a plenary would be.
• people are identified, each change has authors' name attached. Typically this is enough to run a corporate Wiki with hundreds participants and have no misuse at all.

  In fact, we all can delete a page by mistake or do something similar. Wikis are all based on versioning systems (discussed below):

• every change to every document is logged,
• it is easy to see who changed what
• it is easy to roll back unwanted changes.

  Accordingly, if we do something wrong to a Wiki page, or a malicious user does, no data is lost, it can all be rolled back.

  What happen if two persons accidentally update the same page? Nothing terrible - typically, the first one who would press the 'submit' button would get his version in, and the second one would not be allowed to commit. Advanced Wiki systems would also allow a user to see that somebody is working on a page.

  Here we have the general rule for versioned systems (Wiki, or program code) - make frequent small commits. Smaller each change is, in both time and space, less is the chance of a conflict.

Workflow

Ticket-based workflow is easy:

• before planning something, somebody should create a ticket. These should be the masters in case of slavery. These can be ordinary staff members in democratic organizations. It is perfectly fine to create a ticket for yourself, in fact most of the tickets are created for yourself.
• switch the tickets to the 'accepted' state, before actually starting working on a ticket. By doing this you would know what you are working on. And your manager would know it as well.
• reassign a ticket to somebody else if you want him to follow up.
• put those interested in the CC list of the ticket, so that they get emails.
• mark the ticket as resolved when you are done with it.

  Tickets are not meant to provide tight control, and should not be used for that. Instead, their major use should be for people to remember what they were working on, to be printed out and brought to the appraisal meeting in December.

Infrastructure

LDAP

As they say, people are our most valuable asset. Sure. If anything happens they start massively laying this asset off. Setting up a central authentication service is the first thing to do.

Having an LDAP server is no news, and it is quite likely that your institution is already using one. The problem comes from the fact that your institutional LDAP is often limited to staff members, who have a formal relationship with it. It may be impossible to create accounts for external people who want to commit to your projects, but are not related to your institution otherwise.

It make sense to set up your LDAP server in a way that it adds to the existing institutional LDAP, so that local staff does not need to create additional accounts.

Here we have LDAP as the standard, with several good open-source implementations: Open LDAP, Apache Directory Studio. Do not forget to back it up!

Version Control

A version control repository is needed to store program code, and to allow a development team to work on it.

Here we have Subversion is a very good option. It has a number of benefits compared to the legacy CVN. The most important one is that Subversion does transactional commits: a typical commit of a number of files may break in the middle. In this case CVN would leave the commit broken: the files that were already committed would be stored on the server, and the others would stay on developers machine, expecting him to recommit them. Subversion would revert the whole commit, ensuring that only successful commits are stored.