The April issue of Nature Climate Change features a figure produced by Climate Code Foundation, illustrating this article. This page is the “supplementary information” regarding that figure.

NASA GISTEMP (blue), Clear Climate Code ccc-gistemp (pink,
offset -0.2°C for clarity). Difference ccc-gistemp - GISTEMP (green,
right-hand scale, note x20 scale).

Updated to state these errata: in the print edition of Nature Climate Change, the ’0.0′ label on the y-axis has been misprinted as ’0.6′, and the caption credits ‘Climate Change Foundation’, not ‘Climate Code Foundation’.

The figure compares the global temperature anomaly analysis using our ccc-gistemp software against the analysis published by NASA using their GISTEMP software. To produce it you need to have run ccc-gistemp (to get a valid result directory), and you need to have Inkscape on your PATH. Then go python tool/multi.py nature201002. This will create nature.pdf.

As the figure caption says, the ccc-gistemp result in the figure was produced using “software revision 700“. The version of tool/multi.py (and related files) used to make nature.pdf was revision 727. Both computation and visualisation were run with Python 2.7 (although the Python version should not affect these results).

The input data for the figure is in this zip archive held at our source code repository. These are just copies of publicly available datasets (GHCN, USHCN, and so on), but because the available copies change (typically every month), we need to keep a copy if we’re going to reproduce the figure exactly.

The numbers for the GISTEMP curve come from the NASA GISTEMP website. The published datafiles change every month, but previous versions are not made publicly available. In order to exactly reproduce the Nature figure, we have to archive a copy of the file we used: again at the googlecode repository.

Why are we publishing this blog post? Across science, reproducibility of computer results is seen as increasingly important. We believe it is vital to public understanding of climate science in particular. Every number and every figure in every paper is the result of processing data with a computer program; releasing the programs allows interested readers to better understand the processing, and also to check the programs for errors. It should be possible for an interested reader to reproduce the figures exactly by re-running the programs.

[Updated to add link to Nature Climate Change article]

One of our goals is to see more scientific code published. Nature kindly gave us space to voice this opinion earlier in the year. In the world of software tools (our home planet if you like) we have seen huge strides forward because people published the source code to their software. It’s where the Open Source movement began. We believe that science will similarly be improved by having more scientists publish more of their code.

By publish we don’t necessarily mean polished, documented, formatted, and printed in a glossy peer-reviewed journal. We just mean made available. Whatever you wrote. Just stick it on the web somewhere. A zipfile is fine.

What should this look like? The upcoming issue of Annals of Applied Statistics provides a fine example. McShane and Wyner have published an article in this journal, and their are various discussions of the article in the same issue. One of which is Davis & Liu, and their code is published as supplementary material. It’s a great example of what we mean when we say you should publish your code. The code (it’s a few dozen lines of R) is clearly more or less as Davis & Liu wrote it. It has a comment at the top telling you how to download the inputs and run it, that looks like it might have been added in haste later.

As code goes it’s not great code: it’s poorly documented, and full of magic numbers. But it doesn’t have to be great code. It does the job; no-one is going to be building nuclear power stations or recommending the purchase of a cancer-busting drug using this code. The important thing is that it’s the code used to produce the figures in the paper, and it’s published.

(Davis & Liu are not the only ones to make their code available, there is plenty more in the supplementary materials: McShane and Wyner make available their R code. The rest use Matlab: Smerdon’s; Tingley’s; Holmström’s; Kaplan’s)

Stein’s editorial in the same issue of Annals of Applied Statistics is well worth reading, and he has useful things to say on peer-review, data, statistical testing, uncertainty, and the relationship between code and reproducibility. He notes that (emphasis mine) “There is a movement in various disciplines to make all numerical results reported on in published papers reproducible by providing all of the data and code used to generate the results“, and goes on to say that this reproducibility “should be a requirement for research that has potentially important public policy implications whenever permissible”.

Naturally we agree.

The Climate Code Foundation advisory committee met for the first time yesterday, 2010-11-30. The committee is independent of the Foundation, meets once per quarter, and currently include seven climate scientists, one computer scientist, two specialists in open science, and one writer. Members are distributed around the world, so the meeting was held by Skype conference call, with agenda and minutes in Etherpad. Despite the scheduling challenge, eight committee members were able to attend.

In advance of the meeting, the Foundation produced a status report to the committee, describing activities in the last quarter and plans for the next quarter, and requesting advice on several specific points. The meeting minutes were agreed in the meeting and immediately published on the Foundation website.

The meeting elected Dr Kate Willett of the UK Met Office as the chair of the committee, agreed basic committee rules and procedures, considered the status report, and provided the Foundation with very useful advice and offers of assistance.

The members are not compensated for this committee work. The Foundation would like to thank the members, their employers, and their families for the generous donation of their time. We look forward to the next meeting in February.

I am proud to announce that this week’s Nature features an opinion piece by me, arguing that all science software–from tiny scripts to huge models–should be published.  There is also a related news article–with quotes from many luminaries, distinguished company for yours truly–about the very prevalent use of software in science, and some related problems (primarily a lack of training and openness).

This issue is important for the whole of science, and I was delighted to be approached by Nature to write the article.  However, the word-limit was quite strict and it was not possible to address many questions and concerns.  Over the next few weeks we will be posting a number of blog articles and white papers here on the Foundation website, to help fill in these gaps.  Just as a quick bullet-list of tasters:

• No, publication on its own is not enough.  We want to see open-source publication, so that code can be re-used by other scientists and join the great competitive collaborative enterprise that is peer-reviewed science.
• Yes, training is important and must be funded.  If I had to pick five top software skills which all scientists should learn, they would be source code management, defect tracking, literate programming, unit testing, and evolutionary development.
• Yes, open development is important.  Open source-code management, open defect-tracking.  Sourceforge or Google Code are good models here; there are also science-specific open workflow tools.
• Yes, there are specific requirements and a specific urgency in climate science, because of present and future public policies which are developed and decided based on the science results.  Public support for these policies has been substantially eroded, in part due to doubts about climate science software.  That is why the Foundation exists.
• Yes, I am an outsider, but I have known scientists all my life and have worked with climate scientists for several years on Clear Climate Code.  I do know what I’m talking about.
• Yes, right now the Foundation is just tiny, and unfunded.  But we have advisors, goals, a plan, and are seeking sponsorships and partnerships. We won’t be so small for long.
• Yes, our goals are ambitious, but the longest program starts with a single line of code.

This week I am in Brussels talking to NGOs and European officials, building up the organisational networks which we will need to have any effect on policies of agencies and inter-governmental bodies.  So I haven’t actually been able to pick up a copy of Nature.  I look forward to seeing my name in print when I get home tomorrow.

Welcome to the Climate Code Foundation, a non-profit organisation founded in August 2010 to promote the public understanding of climate science.

The work of the Foundation on climate science software includes continuing the Clear Climate Code project, to clarify the software, and the Open Climate Code project, to encourage its publication. The Foundation provides a unifying framework for these efforts.

Members of the Foundation are also attending the Surface Temperatures workshop at the UK Met Office in September 2010, to promote better and more open software practices within that project.

The Foundation intends to work with climate scientists, funding bodies, national and international organisations, and science publishers, to establish climate science in the forefront of science software quality and transparency.

If you support the Foundation’s goals, there are many ways to contribute to its work.