This is the third of a series of posts intended to help scientists take part in the open science revolution by pointing them towards effective tools, approaches, and technologies.

The previous advice is to use free software, and to take the advocacy with a pinch of salt.

3. Find a Strong Community

New projects are often very exciting, and might show amazing promise. And using a brand-new free software project is far less dangerous than taking up a beta version of some proprietary tool. A proprietary tool might never make it to a formal release, and if the vendor decides not to follow through on early development then it might even become impossible to legally use it. That can’t happen to free software: the license guarantees that you will always be able to use, adapt, copy, and distribute it. However, you usually want more than that.

Free software is not exempt from a general rule of thumb in the industry: there is a high “infant mortality rate”, and a majority of projects never make it to “version 2.0″. A shiny new tool is likely to become dull and unloved, and may never get the vital upgrade to a new language version, or the module for compatibility with a new protocol, or the latest operating system. It is paradoxical but true: the useful life expectancy of a software product increases as the product matures.

Growing into maturity, a piece of software acquires a community of developers and users, with mailing lists, IRC channels, blogs, twitter feeds, and meetings – everything from informal online gatherings through pub meets up to international conferences. This community forms a complex ecosystem, and it’s this network of people, systems, companies, and technologies that gives longevity to the software:

• A diverse base of developers makes a project robust in event of loss of a key person, and helps to guard against unwise development choices.
• A critical mass of users ensures a rich flow of new requirements, serves as the source of new developers, and may often provide some financial and material support for development (from server hardware to conference sponsorship).
• An active community builds interfaces, plugins, and modules to connect a good project to a wide diversity of other systems, and these connections in turn allow the user base to expand.

This rich context is the sign of a mature software system, and until it has formed – which may take years from a tool’s first public announcement or release – you are taking a risk in using a project. Sooner or later you will want a new feature, or a bug fix, or compatibility with a new operating system or interface, or simply guidance through some obscure corner of the code, and at those times you will be glad of the community of fellow users. Without it, you will have to find your own way. Choosing free software ensures you are always free to do that: you can write a serial driver controlling your new lab instrument, or a visualisation tool to generate PDF charts directly from your analysis results. But if you are one of hundreds or thousands using the same tool, the chances are that someone will have done that work already. Or at least that you can find fellow-travellers to develop the code with you.

Free software communities are often very strong, and may survive for several decades. But they can atrophy and die. Potentially fatal problems include:

• Stalled development: if the stream of new releases dries up, users will start to drift away to competing tools which provide the new functionality they need. This sort of project stall can be due to various causes: key people moving on, loss of corporate involvement, developmental over-reach or a defect mire, or even office politics and personality clashes.
• Incompatible releases: If new releases break existing applications, the user base will become fragmented (as each user continues to use whichever old release works for them). Potential new users see this fragmentation and are either confused or simply put off.
• Unwise development direction: a core development team focused on their own ideas, to the exclusion of the community’s requirements, will gradually lose users to other projects. This can result from core developers using the development of the system itself for research or experimental purposes. For instance, the developers of a programming language implementation might well be interested in a cool idea about programming languages (e.g. a new concurrency model, or a new syntactic possibility) and use their language to explore that direction. But the users mostly don’t want a Cool New Language; they want the Same Old Language, but with new libraries, a faster compiler, better debugger, etc.

So these are all things to consider when checking for a healthy community: are there frequent new releases of software? Are they backward-compatible, and is that a priority of the developers? Is the process for steering development flexible and responsive to community input?

Open-source software communities are great for productive conversations with colleagues and fellow researchers. Because they bring together users with a shared interest in the software, but who might work in different fields, you may well find grounds, not just for collective software development, but for research directions, and even collaborations. So once you have found a strong and healthy community, be sure to get involved, engage in discussions, and go to meetings. It is often highly rewarding.

This is the second of a series of posts intended to help scientists take part in the open science revolution by pointing them towards effective tools, approaches, and technologies.

Having established in the previous post the importance of using open-source software for open science, I continue to the second in my set of five simple rules for scientists. This is one which most of us know and use instinctively in our daily lives, but somehow when it comes to computers and software tools we can lose sight of this basic idea:

2. Don’t Believe the Hype

Many open-source projects have active communities of users and developers. These people are using and developing the tool because it solves their problems. Unfortunately, this success may blind some of them to difficulties which you may face when you use the tool in your research. They may be so enthused about their project that they become evangelical about its use: it becomes the solution to everyone’s problems. In their minds, they have the World’s Greatest Hammer, so your project looks like a nail.

When you are first researching a tool for potential use, reading documentation and opening conversations with existing users, you will come across some of these project evangelists, and be exposed to some of their hype. They may well be very experienced in their own arena, but are likely to have little or no experience in your research domain, or the class of problem you are trying to solve. Particularly with younger project communities, whose tools have not been applied to a wide variety of problems, and which have grown out of a narrow problem domain, the ignorance can be breathtaking, and the hype can be extreme.

Of course, hype for proprietary software is often far more widespread and more extreme than in the free software world. Some commercial marketing departments may have no compunction about deliberately distorting the truth, or outright lying, to secure a sale, and (due to common stringent licensing conditions which can prohibit users from telling war stories), they often have a great deal of control over the “messaging” about their products. But since we already disposed of proprietary software with rule 1, it’s important that we should be honest with ourselves about some problems in open source communities.

Here’s a list of particularly common subjects of hyperbolic rhetoric. For each subject, I’ve suggested some “antidotes” to the Kool-Aid: questions you should ask yourself or the community, to help you judge the true suitability of the tool for your use.

• Productivity: those jobs which used to take months, or which were especially difficult to get right, are now automatically handled in the background.
  Antidotes:
  • Are those jobs you are familiar with? Are they likely to feature in your project? Yes, it’s cool that the tool makes it really easy to drive a robotic instrument through the USB port. But is that something you need to do on your project?
  • Do you already have a good way of carrying out that job? Is it wise to discard that expertise, or would it be better to continue with your existing technique?
• Performance: “Faster than FORTRAN” is a common cry. This is quite a hard target for algorithms which are very numeric-intensive, and which have been optimized for a particular platform by an experienced developer. It’s hard because compiling FORTRAN for high-performance computation is a problem which has been attacked for six decades by many of the finest minds in computer science, and parts of FORTRAN have been refined to allow the expression of algorithms in a performance-sensitive way (and computers have evolved alongside, to enable FORTRAN programs to run faster). Competing with FORTRAN is not such a big deal in general-purpose programming—algorithms without data parallelism—or for more modern languages in which parallelism, dependency, and aliasing can be fully expressed, and which have mature compilers developed with performance in mind.
  Antidotes:
  • “faster than X” is meaningless. “Faster than X, on this specific computer, solving this problem, with this code and this data” is meaningful, and should be assessed by comparing that problem with your own.
  • How computationally intensive is your problem? Will it take hours of computer time? Of super-computer time? How much of your time is it worth to save this amount of computer time?
  • How much of your computation is going to be performance-critical? It’s often only a very small numeric core, which—once located—can be farmed out to a separate small program written in FORTRAN or some other special-sauce language.
• Installation: you will be told that installation “is a snap”. This is very common, because long-term users of any software have usually not had to install it from scratch for a long time. They may never have installed it at all (it may have come “out-of-the-box”), and if they did then they may well have forgotten any difficulty they had. Furthermore, many users have only ever installed it on a single “platform”: their own computer, running some particular version of that operating system with particular versions of any required packages. Software installation is often tricky, with complex dependencies and many compatibility headaches.
  Antidotes:
  • Read through the installation instructions for your operating system. Do they identify the dependencies, including particular versions? Do they have trouble-shooting advice?
  • Are there existing developers and users across a wide diversity of platforms? Is it easy to find users with your particular mix of operating system and other dependencies?
  • Are there active discussion forums, or a wiki, or a knowledge base? Browse them: do newcomers get real assistance, or a brush-off?
• Compatibility: you will certainly be assured that the system has interfaces to XML, and JSON, and SOAP/RPC, and NetCDF, and SVG, and SQLite, and on and on. But if that functionality is critical to your project, research it further. The interface may well have been developed to support one or two previous projects, and only provide whatever functionality they required. Or it may be antique, not up-to-date with recent changes.
  Antidotes:
  • Look through the interface documentation. Is it complete? Is it up-to-date?
  • Ask questions specific to your use. Instead of asking “does it work with Matlab files?”, ask “can I read MAT-File Level 5 format files, with sparse arrays and byte-swapping?”
  • Find other users with similar compatibility requirements and ask to see their code.
• Your problem doesn’t matter: because the tool is so awesome that this other technique is better. The language doesn’t need namespaces—you can always use a prefix. You don’t need a working C interface—you can always reimplement that library in this other way. You don’t really need PDF charts: SVG is so much more 21st century.
  Antidotes:
  • You are kidding, right?
  • (yes, it’s true about SVG, but many publishers and other tools still want PDF).
• Future development: It’s not faster than FORTRAN yet, but it will be by Christmas because I just read this cool paper about how to do it. Version 3 is going to have support for PDF defrobnification. The iPhone app will be out Real Soon Now.
  Antidotes:
  • As with all software, you should assume that any feature not already functional in a released version is very unlikely to be working by the suggested date, and may well never exist.
  • So: keep one eye on project announcements, and bear the tool in mind for future use, but if your project needs a feature today, then either choose a tool which has that feature now, or plan to develop it yourself.

Hype is at its worst at particular phases in a project’s life-cycle: at the beginning (when the future is bright, and everything is possible) and near the end (when the project is dying, and this is obvious to all but the True Faithful, who are sustained by unshakeable belief). Those are times to avoid a project anyway, and will be addressed in future posts in this series.

This is the first of a series of posts intended to help scientists take part in the open science revolution by pointing them towards effective tools, approaches, and technologies.

When I make presentations to groups of scientists about the Climate Code Foundation and the Science Code Manifesto, I am often asked to recommend particular systems, or to advise on the advantages and disadvantages of some particular combination of tools. Many scientists are keen to adopt free software systems, open-source tools, and modern development methods, to become part of the growing open science revolution, and are looking for guidance.

I’m happy to discuss such questions individually, but there are far too many possible systems, configurations, and sets of requirements for anyone to give a single comprehensive answer. One tool might be excellent for most uses but fall short in some particular aspect which is critical for your project. Another might be generally weak but have a tremendously strong feature which addresses your needs. Over the next few days I will be giving some specific answers to common questions—identifying the strengths and weaknesses of my own favourite systems—but first I will try to help any scientist to pick winners for themselves, by giving some useful rules of thumb.

So I have formulated five general rules for making good choices in the open-source development world, and this is the first in a series of posts laying out those rules.

1. Is it really free? Is it really open?

The first question to ask about any particular tool is: what restrictions are there on using, modifying, or sharing it? Can my audience use it too?

The open science revolution is built on free software, and I strongly encourage all researchers to use free software whenever possible. However, the term “free software” has a very specific technical meaning, which is often misunderstood by those outside the software world. It does not mean “zero-cost software”, it means “software that does not restrict its users from studying, modifying, and sharing it”.

Because of this confusion, I usually favour the term “open-source software”, which isn’t so easily misunderstood and also has two other advantages. First, it has the ring of jargon, so listeners immediately grasp that it has a specific technical meaning. Secondly, it chimes with other pillars of twenty-first century science: “open data” and “open access”, with which researchers are often already familiar. Open-ness—making one’s results available to others to be criticised, praised, and built upon—is a core scientific value, and this term emphasizes that the same principle is at work for software.

The formal definitions of the two terms overlap almost totally—certainly all the software I recommend is both free and open-source—and many people use them synonymously: I mostly use “free software” for audiences of software professionals, and “open-source software” for others. It’s very regrettable that some people, including software community leaders, see conflict between the two. In my experience most people actually writing free software aren’t very interested in the distinction.

In any case, the important thing to remember is that software which is zero-cost but not truly free is a very risky choice which will not provide many of the great advantages of open-source software. Scientists and researchers working in large institutions in rich countries often have a very wide choice of zero-cost software tools:

• Some will be available “for free” for some limited trial period. For example, many new computers come pre-installed with suites of software which become unusable after 30 days, unless a fee is paid. The same is often true of software provided at a departmental or group level, and of many web services (for example, search tools for research literature). Remember, the first hit of any drug is always free.
• Some will have been bought for the institution, the department, or the team. A license may need renewing—often annually—at a price to be determined by the vendor. Next year, or next week, the license may come up for renewal and be cancelled due to budget pressure.
• Some will not have been bought, but will have been copied illegally, and any licensing mechanism will have been subverted. Such software may be suddenly disabled (and the people responsible punished).
• Some may have been bought by a single researcher or small group, and made available to a few people. The fact that it’s on your computer doesn’t guarantee that you have the right to use it.
• Finally, some will be truly free, open-source, software.

Always, ask yourself this: can my audience use this tool? Open science is all about sharing your research, including your code, with others, and if they can’t use the tool then your code is much less useful to them—they won’t read it and they won’t improve it—and to you: they won’t cite it.

Your audience might be your research colleagues, or those elsewhere in your department, or at other institutions around the world (including those with less generous budgets), or independent researchers, or the public. One key audience member, often the most important, is your own future self: you can use the tool right now, but your institution might stop licensing it, or you might move to work somewhere without a license, or the software vendor might go broke (so that suddenly nobody has a license). Do you want to be able to use your own research in five years’ time?

Open science uses open-source software. Make sure you do too.