Research protocol: colour, letters or both? 20 hypotheses and how we will test them

Research questions

1. RQ1 Does colour coding shorten the time to find and identify elements, compared with a single-colour model?
2. RQ2 Do engraved symbols shorten it further, on top of colour?
3. RQ3 Do engraved symbols make atom identification more accurate?
4. RQ4 Does colour plus symbol improve immediate learning about molecular composition?
5. RQ5 Does it improve what students remember a week later?
6. RQ6 Does what students learn carry over to molecules they have never handled?
7. RQ7 Do the benefits differ between students with little and with more prior chemistry knowledge?
8. RQ8 Do symbols on the atoms reduce how often students need a colour key or ask for help?
9. RQ9 Do colour and letters change how quickly students find specific bonds, and do they leave purely geometric judgements untouched?
10. RQ10 Do letters lower or raise the mental effort of working with the model, and which version is the most efficient overall?
11. RQ11 Can the engraved symbols be read reliably on real prints, and does placing them all around the sphere help?
12. RQ12 Does using the model as a self-test (guess first, then turn it to check) improve memory?
13. RQ13 Do engraved symbols open the model to students who cannot rely on colour, or on sight?
14. RQ14 Do ready-to-print models with a lesson plan save teachers preparation time?
15. RQ15 Does holding the model help students connect an infrared spectrum to the bonds that produce it?
16. RQ16 Does one object that joins the molecule with its IR, NMR and mass-spectrum data help students combine evidence from different methods?

Hypotheses

We list every hypothesis we intend to test, including the uncomfortable ones. Each names the comparison, the task and the direction, together with the result that would count against it. They are grouped by the study that tests them (section 8.2) and will be written down before any data are collected.

One more expectation is practical rather than theoretical. Colour + engraved needs a multicolour printer; engraved prints on any machine in one filament. If engraved turns out to give most of what colour + engraved gives, that is excellent news for every school with an ordinary printer.

We plan four steps, because these are different questions and need different designs:

1. “can the letters be read?”
2. “do they speed up seeing?”
3. “do they improve learning?”
4. “how should the model be used?”

Plus smaller secondary studies on access and on teachers’ time. Studies 1 to 3 will be preregistered on the project’s OSF page before recruitment (Nosek, Ebersole, DeHaven & Mellor, 2018) and run with a partner university under its ethics approval.

Study 0 · Can students actually read the letters?

A short technical study on real prints, before any learning experiment. If the letters cannot be read in a real classroom, the engraved versions would fail for a technical reason rather than a cognitive one, and the learning study would measure the wrong thing.

1. Prints: the four caffeine versions from the same files, plus engraving variants on the smallest spheres (hydrogen): depth, letter height and stroke width.
2. Close-up photographs of every symbol type on every print, at a fixed scale, published with the files.
3. Reading test: students name symbols at arm’s length under ordinary classroom light, with the model held still in a fixed orientation; we record the share of atoms named correctly without turning the model.
4. Front-facing versus all-round letters compared on the same measure.
5. Print quality: blurred or broken letters, stringing and surface damage recorded per print; a model that fails is reprinted, not used.

Study 1 · Seeing: a fast, within-person experiment

1. Who: undergraduates in introductory chemistry. Each person sees all four versions, so everyone serves as their own control.
2. What: short timed trials on printed models: FIND (“touch every oxygen”), NAME (“which element is the atom with the sticker?”), BOND (“touch both C=O double bonds”) and a few purely geometric questions, with no colour key on the table.
3. Molecules: several, not only caffeine, so nobody can simply memorise one shape. Each molecule appears in each version across participants, rotated in a Latin square.
4. Measured: accuracy, and response time on correct answers.

Study 2 · Learning: a randomised classroom experiment

1. Before (T0): a short test of element symbols, structural formulas, familiarity with the CPK colours and prior experience with models, plus a spatial-ability test (Purdue Visualization of Rotations; Bodner & Guay, 1997), because spatial ability matters in chemistry (Harle & Towns, 2011).
2. Random assignment to one of the four versions, stratified by prior knowledge so that no group starts ahead. A fifth, drawings-only group, as in the original project plan, can be added as a baseline.
3. Ten minutes with caffeine, the same instructions, the same worksheet and the same instructor script for everyone. Only the model differs.
4. Immediately after (T1), without the model: NAME, FIND on paper, composition (“how many N atoms?”), matching the model to its 2D structure, and a few purely geometric questions as the built-in check (H6).
5. Seven days later (T2), without the model: the delayed test. A further test after about 28 days if the partner can host it.
6. Transfer: theophylline (CAS 58-55-9), caffeine’s close relative with one methyl group fewer, as near transfer; aspirin (CAS 50-78-2), with no nitrogen at all, as far transfer.

The demonstrations in the caffeine lesson kit (smell test, label, infrared, crystals) are not part of the study session. If a participating class also runs them, every group gets exactly the same demonstrations, at the same point, so that they cannot become a hidden difference between the four model versions.

Two design details matter more than they look. First, tests are themselves learning events (Roediger & Karpicke, 2006), so every group takes exactly the same tests at the same times. Second, speed means nothing without accuracy: a model that makes students answer faster but more often wrongly is not better.

Study 3 · Using the model: rotate to reveal

Our measurement in section 6 turned a design limitation into a teaching tool. Because the letters face forward, the back of the model shows no symbols at all, so the same object can work like a flashcard: look at the plain side, name each atom, then turn the model round and check.

Retrieving an answer is known to strengthen memory more than re-reading it (Roediger & Karpicke, 2006). Study 3 tests whether that holds for a model in the hand.

1. Who: students working with an engraved model (engraved or colour + engraved), randomly split into two groups.
2. Rotate to reveal: the model starts with its plain side facing the student; the student names each atom, then turns the model and checks.
3. Look and learn: the same time spent with the labelled side facing the student.
4. Measured: the same seven-day test as Study 2 (H15).

Secondary studies

• Colour-vision deficiency (H16): students who report a colour-vision deficiency take the Study 1 tasks on all four versions; analysed separately because the group will be small.
• Touch (H17): with specialist partners, blind and low-vision students identify engraved symbols by touch on prints at several sizes. Exploratory: its first result may simply be the smallest letter that works.
• Teachers’ time (H18): preparation time and NASA-TLX workload for a lesson built with and without ready-to-print models, as described in the project plan.
• Model and spectrum (H19): two randomised groups receive the same short talk, time, instructions and questions with the same caffeine IR spectrum in the viewer; one group also holds an engraved model. Measured:
  • correct assignment of selected bands to the groups that contribute most,
  • correct identification of those atoms,
  • the quality of the explanation linking spectrum and geometry,
  • and a test without the model after seven days.
  Smell, crystals and colour stay out of this study, so that the model is the only difference.
• Evidence cube (H20): two randomised groups get the same IR, NMR and mass-spectrum data, time and questions; one group has the data on the evidence cube around the molecule, the other on separate slides. Measured:
  • interpretation of each spectrum,
  • what each method can and cannot tell,
  • links between data and structure,
  • retention after seven days and transfer to theophylline.

What we measure

“Learning efficiency”, defined before we measure it. The phrase is easy to use loosely, so we fix its meaning in advance. We will report it in two ways:

• accuracy per minute of learning time, and
• instructional efficiency, which combines test performance with the mental effort students report (Paas & Van Merriënboer, 1993).

A version that gives the same score with less effort is more efficient; a version that is fast to read but leaves nothing behind a week later is not. Faster identification of an atom is not the same as faster understanding of a molecule, and we will report the two separately.

Analysis and sample size

Response times are skewed and come from many items per person, so we will analyse them with mixed-effects models that treat participants and items as random effects (Baayen, Davidson & Bates, 2008), using generalised models suited to response-time distributions instead of forcing them into a normal shape (Lo & Andrews, 2015).

• Accuracy: accuracy will be analysed with logistic mixed models.
• Contrasts: the three planned contrasts are colour versus no colour, letter versus no letter, and the key comparison colour + engraved versus colour.
• Reporting: we will report effect sizes with confidence intervals, exact p-values, missing data and checks of model assumptions, correct for multiple comparisons, and use equivalence tests where the question is “is there no meaningful difference?” (Lakens, 2017).

Sample sizes below were computed for a two-sided test at α = 0.05 and 80% power, the standard approach implemented in G*Power (Faul, Erdfelder, Lang & Buchner, 2007). They are planning figures; the preregistration will fix the final numbers using pilot data.

The table shows why Study 1 comes before Study 2: a within-person design reaches useful precision with a few dozen students, and its results tell us which effect size is realistic before we ask four classroom groups of around a hundred students each to take part.

A small pilot of Study 2, with 20 to 30 students per version, will test the procedure and the materials, not the hypotheses. Before it, the school pilot MolGod Research Partners compares only two versions, the same single-colour model with and without engraved symbols, for the same purpose: to test the procedure and the materials, not the hypotheses.

Planned before preregistration

• Classes as clusters. Students in the same class, and the two students in a pair, are not independent. Analyses will model class (and pair, where models are shared) as random effects, and sample sizes will be inflated for clustering once the intra-class correlation can be estimated from the pilot.
• Power by simulation. The sample sizes on this page are planning figures from standard formulas. Because the planned analyses are mixed models, the final numbers will be set by simulation from pilot data and fixed in the preregistration, with the smallest effect of interest stated and justified there.
• What NAME tasks can show. When a letter is printed on the atom, naming it with the model in hand is expected to be easy. The interesting tests are therefore the ones without the model, a week later and on a new molecule, together with the colour-only control.
• Preregistration first. Hypotheses, measures, exclusion rules and analyses will be registered in a public, time-stamped registry before any study data are collected; this page will then link to that frozen record, and later changes will be listed as deviations.
• Measures. The tests are our own; their reliability will be checked in the pilot, and the delayed test will use parallel items so that students do not simply repeat the same questions.

Deciding in advance what we will do with each possible outcome is the best protection against reading the data the way we would like. Here is our commitment.

• Ethics approval — obtained by the partner institution before any data collection; participation voluntary, with informed consent and the right to withdraw.
• Data — as little personal data as possible, pseudonymised, handled under applicable data-protection law.
• Preregistration — hypotheses, primary outcomes, sample size and analysis fixed on OSF before recruitment; any deviation reported and explained.
• All results published — including null, negative or mixed findings, together with the anonymised data, the test materials and the exact print files with their checksums.
• Conflict of interest — we design and publish these models, and MolGod.org also provides commercial chemical-documentation services.
  • The models are free and openly licensed, and no external funding supports this project.
  • We will not describe our own study as independent validation; scoring will be done blind to version where possible.

When the data are in, we intend to submit the results to a peer-reviewed chemistry education journal such as the Journal of Chemical Education, Chemistry Education Research and Practice or Biochemistry and Molecular Biology Education.

A working title: Do engraved element symbols improve molecular identification and learning? A comparison of single-colour, colour-coded and symbol-integrated 3D-printed molecular models.

• Novelty. A colourful new object is exciting. Excitement is not learning, which is why the delayed test and the transfer molecules carry so much weight.
• Who runs the study. We have an interest in the result. Independent scoring, a partner institution, preregistration and open data are how we try to keep ourselves honest.
• One molecule is not chemistry. Caffeine contains only four elements. Halogens, sulfur and phosphorus, and much larger molecules, may behave differently. Later studies should vary molecular size, geometry and elements.
• Small spheres, small letters. Hydrogen spheres leave little room for a symbol. Study 0 exists to catch this first.
• Prior knowledge of colours. Students who already know the CPK code are, in effect, reading the colour model with an invisible key. That is why familiarity with the colours is measured and why H10 exists.
• Colour on the bonds, too. In our colour models the C=O and C=N double bonds take the colour of their heteroatom. That is the same on colour and colour + engraved, so it does not affect the central comparison between them, but it is part of what “colour” means when colour is compared with one-colour, and H5 measures it directly.
• Visual clutter. Colour plus letters is more information. If it distracts rather than helps, mental effort will rise and accuracy will not; our measures are designed to show that.
• Feel and handling. Engraved and filled surfaces feel slightly different. We will keep material, size and mass the same and note any differences we cannot remove.
• Physical, not digital. Results apply to printed models in the hand. On-screen models may behave differently (Stull & Hegarty, 2016).
• The multicolour print is still a design. The colour + engraved model has not yet been printed. Renders say nothing about real legibility; only real prints go into the study.