Experiment Template

This template is intended to be a framework for designing and documenting experiments. Following this template should ensure that the author(s):

• Have suitable clarity of purpose
• Gather results that are useful both now, and in the future
• Document the results in a way that’s useful to themselves and others

After reading an experiment doc, people should be clear on the following:

• When this was attempted
• What the context, constraints, and assumptions were, including:
  • What spurred this experiment
  • How much time was/could be spent on this experiment
  • What options were considered
• What the author intended to try
• What the author actually did
• What outcomes the author hoped for
• What the outcomes were
• What conclusions the author drew from the outcomes

Given this context, a reader should be able to learn from this experiment, whilst understanding whether the approaches and conclusions are accurate for their own needs.

It’s best to write the document in three phases:

1. Research & high level design: establish the Background, Aim and Hypothesis. After completing these, design the Approach.
2. Experiment: record the Results.
3. Analysis & conclusions: record your Analysis, draw your Conclusions, and summarise the experiment in the Abstract.

If changes are required as you proceed, it’s best to record what your original idea was, what you replaced it with, and why you made that change. This is valuable information to readers (including yourself, in future).

Abstract

Provide a brief (1-3 paragraph) summarization of the experiment and outcomes.

Potential useful outcomes of this section:

• Readers quickly understand whether this document is of any relevance to them
• Readers can quickly refresh their memory about the high-level outcomes

Background

Provide context for the reader to understand why this experiment was performed, and why it was performed in this way:

• Describe the rationale for performing this experiment.
• Explain any involved technologies, techniques or practices in sufficient detail.
• Explain any requirements and goals.
• Explain any limitations and constraints.

Potential useful outcomes of this section:

• Readers can understand why various approaches were included or excluded from the experiment
• Readers understand what you’re attempting to achieve, at a broad, contextual level

Aim

Provide a succinct (1-3 sentence) description of what you’re attempting to achieve with this experiment.

Potential useful outcomes of this section:

• Readers understand what you’re attempting to achieve with your approach
• Readers understand what’s in and out of scope in this experiment

Hypothesis

Provide a one or two sentence description of what you expect to demonstrate with this experiment. Whether you’re right or not, in the long run, is irrelevant - surprising outcomes are often more useful.

Potential useful outcomes of this section:

• You better understand what you’re hoping to achieve, and can check that your approach is suitable to the goal
• Readers can confirm the suitability of the approach compared to the stated goals

Approach

Describe how you’ll perform the experiment. Include sufficient detail to allow someone else to attempt the same experiment and confirm your results, within reason. Your goal is to ensure that someone with sufficient knowledge of the tools, techniques etc. used in this approach can run the same experiment without contacting you. Part of this should involve isolating your testing environment as much as feasible (e.g. not relying on peculiarities of a single AWS account)

It’s worth taking a bit of extra time to ensure this experiment can be performed by others. This will particularly help if there was an issue with the experiment’s design, or the results had issues, or someone wants to expand upon this experiment later.

Potential useful outcomes of this section:

• Readers can confirm the approach tests what you intended to test
• Readers can confirm the experiment produces the same results in their own context
• Readers can extend the experiment to include other things that are of concern to them
• Successful patterns can be replicated elsewhere
• Readers learn more about the concepts from the Background section by seeing them used in context

Results

Describe what outcomes occurred, while trying to leave it as “unprocessed” as possible. Your goal is to give future readers enough information to try to reproduce your results, and/or do their own analysis and see if they agree with your conclusions.

Potential useful outcomes of this section:

• Missed measurements can be identified
• Readers can double-check the results that fed into your conclusions
• Readers can come up with alternative conclusions using context that you didn’t have at the time
• Raw results could be partially applied to situations that weren’t considered in this experiment

Analysis

Analyse the results, and describe your reasons for the conclusions you draw. This section is critical for future readers to confirm the conclusions match their own context.

Potential useful outcomes of this section:

• Readers can identify potential flaws in the conclusions
• Readers can confirm whether the conclusions are accurate for their context, and what they may have to reassess for themselves

Conclusions

Draw conclusions based on the hypothesis, results and discussion. Determine whether you met the aim of the experiment, and whether the hypothesis was correct. Detail anything that might be important to a reader, including inconclusive results and any context that significantly affects outcomes.

Potential useful outcomes of this section:

• Readers understand how your conclusions apply to them
• Readers understand what context affects the outcomes most strongly
• Readers understand what future experiments may be worthwhile to them