2: What makes a Good Procedure?

Here’s the scenario: You have been hired by the new online mega-petstore G.O.P. (Guaranteed
Overnight Pets) to formulate a new fish food. Their current fish food produces rather slow
growth, requiring too many days to rear fish to an acceptable weight for being sent through the mail.

You’ve done the hard work — tested various kinds of insect larvae, nematodes, and algae,
you have created (you believe) the perfect formula, named it “Fish2Whale”, and
sent it off. You are waiting for your paycheck when instead you receive a rather terse note:

Edgar, the mascotSir/Madam:

Please be advised that we do not intend to make any payment for your formula,
“Fish2Whale”. We tried it out on Edgar, the company mascot, and he didn’t
seem to get very big.

Sincerely,

G.O.P.

You might be excused for feeling a bit put out at this point. Management at G.O.P. has not
followed the scientific method when they tested your formula. Below I list several statements
about fishfood. Pick out the one(s) that describe/s a satisfactory test of your fishfood. (This
is not the same as picking the statements that you think are the most likely explanations. All
we care about is that the idea is stated in a way that is testable.)

 

Good scientific procedure?

yes
no
If you feed a group of fish with Fish2Whale and they get to be at least 10 cm long,
that proves that Fish2Whale works.
not a good procedure: there is no group to compare the
Fish2Whale fish with.
yes
no
You should write back and tell the company that Fish2Whale is healthy for fish and
will make their scales shine.
This is not a scientific procedure, this sounds like an advertisement!
Fish2Whale is healthy compared to what? And how will you compare their effects?
yes
no
You should feed one fish Fish2Whale and one fish normal food, and see which fish gets bigger.
it’s good that you are comparing your treatment to a control,
but what if the ‘treatment’ fish (the one you’re feeding Fish2Whale to) happens
to be unwell?
yes
no
You should write back and tell them that its stupid to use the old food when they’ve
already bought the new food.
“Stupid” is a value judgement, not a testable proposition.
This is not even a commercial, it’s just career suicide!
yes
no
You should feed normal fish food to one group of fish (the control group) and
Fish2Whale to another group of fish (the treatment group) and see which group does better.
This is pretty close. You have a comparison and replication, but
you don’t have a way to measure which group is “doing better” (length? weight?
number of surviving offspring? scale shininess?)
yes
no
You should feed normal fish food to one group of fish (the control group) and Fish2Whale
to another group of fish (the treatment group) and see which group grows longer in length.
This is a sound scientific procedure. You have a way to measure
success, something to compare your success to, and replication to ensure that random
effects don’t stuff up your experiment.

 

Answers:
no, no, no, no, no, yes

Once again, make sure that when you design an experiment, you think in terms of
MEASURE-COMPARE-REPLICATE. And when you are analysing a statement above, look to
see how the authors used each of these elements.