Section 15 Designing a Statistical Study Main Ideas

Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Section 15
Designing a Statistical Study
Main Ideas
•Designing an Observational
Study
•Designing a Survey
•Designing an Experiment
•Statistical Problem Solving
•Asking a Question in a Statistical
Way
•Drawing Samples From a
Population
•Collecting and Analyzing Data
•Interpreting the Results
You’re talking with a few of your friends, and someone makes a strange claim. You
don’t believe it, but you’re not quite sure how to disprove the claim. Statistical studies are one way to investigate such claims.
Exploration 15.1 Knowledge Detective
How might you conduct a statistical study to test each of the following claims?
(a) After a good workout, it takes 5 hr for the body to return to its normal
temperature.
(b) Being a statistician is a better job than being either a doctor or a lawyer.
(c) The average IQ of females is 8 points higher than that of males.
(d) The average human head weighs 10 lb.
(e) It is impossible to eat six saltine crackers in 1 min without licking your
lips.
Three types of statistical studies are observational studies, surveys, and ­experiments.
An observational study involves observing and recording data on some variable(s)
of interest. A (sample) survey uses questioning to gather data. Observational
studies and surveys attempt to leave the sample undisturbed. An experiment
(­experimental study) differs in that it attempts to answer a statistical question by
manipulating all or part of the sample.
Designing an Observational Study
After collecting and analyzing data in an observational study, the findings in the
sample are used to draw conclusions about the population. The sample and population do not have to be people.
Example 1 Identifying the Sample and Population
Every Friday, you have a quiz in science class. You wonder whether the amount
of sleep you get during the week is connected to the points you score on these
quizzes. For 20 weeks, you carefully record the number of hours of sleep you
get each night, as well as your Friday quiz score. If you did not manipulate your
sleep patterns or quiz performance, you have conducted an observational study.
What variables did you measure? What is the sample? What is the population?
Solution The variables you are measuring are sleep (in hours per night during the week prior to the quiz) and quiz performance (in points). The sample is
the 20 weeks. The population is all of the weeks in the science class.
Quick Question 15.2 Speedy Shopping
What are the variables, sample, and population in the following observational
study?
In order to see whether males and females differ in the amount of time
they spend in a grocery store, you stand outside a supermarket and
record the time when every tenth person enters the store and the time
when that same person exits.
128 Advanced Quantitative Reasoning
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
The Nielsen Company has specialized in market research since the 1920s. (It is
possible to conduct observational studies without modern technology!) Figure 15.1
presents the results of an observational study about online shopping conducted by
the Nielsen Company.
Quick Question 15.3 Interpreting the Results of an Observational Study
Refer to Figure 15.1 to answer the following questions:
(a) What information is presented in this chart?
(b) What useful information is missing?
(c) How might the Nielsen Company have observed the sample?
(d) What are some conclusions you might make by analyzing the chart?
Figure 15.1 Percentage of males and females over 18 years of age who shopped online
during a 30-day period. [Source: Todd Hale, Nielsen Wire, 2011]
As Figure 15.1 shows, observations are not limited to visible events. Modern
observational studies often take place online in the form of pieces of text called
“cookies.” Internet sites can assign a cookie to each visitor; this cookie includes a
name–value pair that records user data. The data may include time of visit, number
of visits to the site, and ads the visitor clicked on. Until the user deletes the cookies,
the site will accumulate information with each visit. The sites can then use these
data to advertise directly to your interests. Whether such methods are considered
“cool” or “creepy,” this form of observational study is a huge industry.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 129
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Designing a Survey
Numbers Everywhere Gross National Happiness
Surveys can range from a few short
questions to extensive interviews,
even when attempting to measure
the same variable. In contrast with
Hadley Cantril’s two-­question happiness survey, leaders in the country
of Bhutan developed a happiness
survey that took respondents more
than 6 hr to complete. The survey
results were used in calculating
­Bhutan’s “gross national happiness,”
a different way of measuring a country’s progress from the more traditional gross national product.
The idea of measuring a country’s
status through the well-being of its
people quickly spread around the
world, leading to the development
of the Global GNH Survey, which
gave countries a numeric value for
each of the following areas: economics, physical health, environment,
workplace, politics, social health,
and mental health. The units used
to measure happiness are termed
happy life years (HLY). A person’s
HLY is calculated by multiplying
their life expectancy at birth and
their average happiness (a value between 0 and 1, based on a happiness
survey).
(a) Japan has the highest life expectancy for females in the world
at 86 years. Yuki is from Japan,
and has an average happiness of
.80. Pete is from the U.S., whose
life expectancy for males is 76
years. What would Pete’s average happiness have to be for
him to have the same number of
expected HLY as Yuki?
(b) How could you use an equation
to express all the possible combinations of life expectancy and
average happiness that would
result in 60 HLY?
Imagine that you want to measure the level of happiness or suffering of students at
your school. How might you do so?
You could conduct an observational study by choosing a random sample of students and then capturing their facial expressions at several points throughout the
day. But how would you measure their happiness? Someone who appears to be
suffering may be concentrating on a hard problem.
In the mid–20th century, a public opinion researcher named Hadley Cantril was
interested in this same question. He designed a two-question survey known as the
Cantril Self-Anchoring Striving Scale (1965). Picture a ladder with rungs numbered
from 0 (the bottom—the worst life you can imagine for yourself) to 10 (the top—the
best life you can imagine for yourself). Now, answer the following two questions:
1. On which step of the ladder do you feel you are standing at this time?
2. On which step do you think you will be standing in five years?
After testing and refining the survey on a large number of respondents, here’s
how Cantril scored the answers:
• If you scored at least a ‘7’ on the first question, and at least an ‘8’ on the second question, you are “thriving.” You have a positive and strong sense of
well-being. People in this category reportedly worry less, have less stress,
fewer health problems, and are generally happy.
• If you scored a ‘4’ or below on both questions, you are “suffering.” Your wellbeing is at high risk and your future outlook is dismal. People in this category
are twice as likely as thriving respondents to suffer from disease, stress, and
a lack of basic necessities.
• If your answers did not place you in one of the first two categories, you are
“struggling.” Your well-being is moderate and unstable. People in this category tend to eat less healthily, report more daily stress, and take more than
twice as many sick days as those who are “thriving.”
Gallup has used Cantril’s survey to measure suffering around the world. In a
recent survey of countries, Denmark had the lowest percentage of suffering respondents (1%); Zimbabwe had the highest (40%); and the United States was in
between (5%).
Exploration 15.4 Evaluating Happiness
(a) How accurately do you think Cantril’s survey measures happiness?
(b) What are some problems with using the survey to measure happiness and
suffering?
(c) How would you design a study to measure happiness?
Just as with observational studies, surveys attempt to measure samples without
disturbing the samples. Imagine how Cantril’s results might be distorted if the person
conducting the survey gave out $100 bills along with each survey.
Investigation 15.5 Another Way of Measuring Happiness
Research Dr. Lyubomirsky’s Subjective Happiness Scale and then answer the
following questions.
(a) Would this scale provide better results than Cantril’s survey? Why or why
not?
(b) What difficulties might arise with using the Subjective Happiness Scale?
130 Advanced Quantitative Reasoning
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Designing an Experiment
In an experiment, the researcher manipulates the sample to provide answers to
specific questions.
Example 2 Comparing Experimental Methods
History Note Random Assignment
Although random assignment in
human experiments is now commonplace, scientists who use animals for
experiments have only recently come
under fire for not practicing random
assignment.
Suppose that psychologist I. M. Curious wishes to test the belief that eating
­cinnamon improves memory. What are the pros and cons of the following
­methods? Assume that memory can be tested with reasonable accuracy.
• Method 1: Using herself as the subject, Curious eats a spoonful of cinnamon
once every 2 days, and tests her memory every day.
• Method 2: The same as Method 1, but Curious uses 20 persons, excluding
herself.
• Method 3: Curious selects 20 people and divides them into two groups of
10. Group 1 eats a spoonful of cinnamon every day for a month. Group 2
does not eat any cinnamon. Curious tests their memory every day.
Solution Method 1 allows no way of knowing how long cinnamon takes to
impact memory, or how long the impact lasts. If the effect is enduring, there
might be a steady increase in memory, but it would be difficult to know whether
the cinnamon was responsible. Also, each individual is unique, so the effect of
cinnamon on Curious’s own memory may be quite different from the average
effect on people. Method 2 would take care of the individual variability problem, but would still be unable to account for the time issue. Method 3 takes care
of both of these problems, but there are still major potential problems.
Thoughtfully planned sampling is essential in all types of statistical studies: observational studies, surveys, and experiments. Recall that, in a random sample, each
member of the population is equally likely to be included in the sample. A random
sample is unbiased, and thus the results can be extended to the population. The
subsection on Drawing Samples From a Population will address other types of
sampling and whether the results obtained from such samples can be generalized
to the population of interest.
In addition to carefully selecting a sample from the population, there are two
other techniques that can improve the quality of an experimental study, such as the
experiment discussed in Example 2:
1. Random assignment. Random assignment involves fairly dividing the e­ ntire
sample into groups (usually a control group and one or more treatment
groups); each member of the sample is equally likely to be placed in any
particular group.
Quick Question 15.6 Random Sampling Versus Random Assignment
What is the difference between taking a random sample and making a random
assignment?
2. Pretest and posttest. A pretest measures the variable of interest before an experiment begins, and a posttest measures the variable at the end of the experiment. Although random assignment helps to the equalize groups, there is still
the possibility that the sample is divided unfairly in some way. In Example 2,
if subjects with better memories are placed in the treatment group, they are
likely to perform better on a final memory test. This possibility could lead
to a false conclusion that the cinnamon was responsible for improving their
memory. Testing each group on the variable of interest (memory) both before
and after the treatment (cinnamon) helps determine whether the treatment is
responsible for any changes in the groups.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 131
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Investigation 15.7 The Hawthorne Effect
In the late 1920s an experiment was conducted on workers in an electric
company named Hawthorne Works. Before informing workers about the
experiment, researchers secretly observed their productivity for two weeks.
Researchers then increased the brightness of the lights in the factory and
observed the workers’ productivity levels. For a short time, productivity
increased. Productivity also increased when changing the amount of break
time, providing food during breaks, and shortening the working day.
(a) What factors do you think could have caused the increased productivity?
(b) How could you improve the design of such an experiment?
(c) What were the conclusions of the Hawthorne study?
Quick Question 15.8 Identifying Flaws in Experimental Design
What weaknesses can you identify in the following design?
Suppose scientists were interested in whether a medicine called “Revenal”
would lower blood pressure. By advertising in a newspaper, they secured 120
participants. The scientists used a random number generator to split the 120
people into two groups—a control group of 40 people and an experimental
group of 80 people. They then separated the experimental group into a low
blood pressure group and a high blood pressure group, each with 40 people.
They gave the experimental groups a daily dose of 100 mg of Revenal. Each
day, they also gave the control group a placebo (a false treatment such as a
water-filled pill used to prevent participants from knowing whether they are
in the treatment or the control group). After one month, the scientists found
that the low blood pressure group had significantly lower blood pressure
than the control group. They declared Revenal a success.
Statistical Problem Solving
Throughout Part I, you used the four-step problem solving process developed by
George Pólya: understand the problem, devise a plan, carry out the plan, and look
back. The Guidelines for Assessment and Instruction in Statistics Education (GAISE)
Report describes a similar four-step process for statistical problem solving:
1. Formulate Questions: What questions am I interested in answering? What
data can I collect to answer the questions?
2. Collect Data: How can I collect data to answer my questions?
3. Analyze Data: How can I analyze the data to answer my questions?
4. Interpret the Results: What do my data mean? How do my data help me to
answer the questions?
Asking a Question in a Statistical Way
Statistical questions have answers that vary. The mathematical probability question, “What is the most likely sum of two fair dice?” has one answer (even though
different people may give different answers), and is not a statistical question. The
question “What is the most likely sum of these two dice?” would be a statistical
question with two standard six-sided dice if it is unknown whether the dice are
weighted. On the other hand, the same question would not be a statistical question
if each of the dice had a “6” imprinted on every side. There would be no variability
because the sum of the dice would always be 12.
132 Advanced Quantitative Reasoning
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Quick Question 15.9 Identifying Questions With Variability
Which of the following questions have answers that could vary?
(a) What is the hottest place on the earth’s surface?
(b) Who is the cutest boy in the high school?
(c) Who was the wife of the 30th President of the United States?
(d) How tall is the average basketball hoop?
Drawing Samples From a Population
After choosing an appropriate question, the next step is to determine your population of interest and from whom or what you will collect data. There are many ways to
select a group that is representative of a population. You could select every member
of the population, a census, but limitations on time, access, or money often make a
census impractical. It is more common to choose a sample, a subset of the population.
Quick Question 15.10 Sampling Students at Your High School
Suppose that you are convinced that the curfew your parents have set for
you is too strict. You hope to persuade them to relax it by showing them the
curfews of other students at your high school. Because you don’t have time
to take a census, you decide to poll a sample instead. What are some ways to
select a sample of students that represents all of the students at your school?
Stratified Sample Versus Random
Assignment
Do not confuse stratified random
sampling with random assignment.
In random assignment, first you
draw a sample from the population,
then you break the sample randomly
into groups (by randomly assigning
individuals in the sample to a particular group). In stratified sampling,
first you identify the groups (strata)
within the population, then you
select random samples from each
strata of the population.
A simple random sample (SRS) is any sampling method that gives each member of the population an equal chance of being selected for the sample. Although
simple in theory, an SRS often is difficult to obtain in practice. An SRS is unbiased,
and the results can be generalized to the population. In Quick Question 15.10, the
population of interest is the entire student body of your school, so you could obtain
a complete list of the students and then choose a random sample from the list.
Alternatively, you could choose every nth student from the list to create a type of
systematic sample. Although convenient, such samples typically are not random.
If you think that curfews differ by some variable, say gender, you could create
a stratified (random) sample by dividing the student list into males and females
and then taking an SRS from each list. In this case, the strata are the two genders.
Exploration 15.11 Considering Various Sampling Methods
Using the curfew scenario from Quick Question 15.10,
(a) What are some advantages of taking an SRS of the students at your
school?
(b) Why might you want to use a stratified sample of the students instead of
an SRS? Give at least two reasons.
(c) Taking an SRS, a systematic sample, and a stratified sample all require
­access to a population list. How could you take a representative sample of
the students at your school without a population list?
The three sampling methods discussed above involve tracking down individuals and could be quite time-consuming. Often a more efficient sampling method
is a cluster sample, in which you (a) divide the population into groups (clusters),
(b) randomly select a certain number of the clusters, and (c) take a census of each
individual in the selected clusters.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 133
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
A two-stage (cluster) sample involves (a) dividing the population into clusters,
(b) randomly selecting a certain number of the clusters, and (c) taking a random
sample from each of these randomly selected clusters.
Quick Question 15.12 Using Homerooms as Clusters
Using your school’s student body as the population and your school’s homerooms as the clusters,
(a) Explain how you could select a cluster sample.
(b) Explain how you could select a two-stage sample.
(c) Why might you want to use one of these sample methods instead of an
SRS? Give at least two reasons.
Convenience Samples Are Biased
Recall from Errors in S
­ tatistical Studies in Section 14 that a ­convenience
sample involves selecting individuals
because they are easily accessible. If
a statistical study uses convenience
sampling, the results of the study
cannot be generalized to the population of interest.
Finally, the easiest (but least fair and least likely to be representative) method is a
convenience sample, defined in Section 14. For example, you might choose students
who you could easily contact (perhaps your entire AQR class, your lunch table,
or the students around you in each class). Convenience samples are biased. The
results produced cannot be generalized to the population.
Sampling strategies affect the generalizability of results. Even though you may
decide to choose a sample rather than take a census, you are primarily interested in
the population. The more representative your sample, the more valid your sample
results are for the population. The results from a convenience sample of 18-yearolds with part-time jobs will not be as generalizable as those from an SRS of all of
the students at your high school. However, you may be interested in changing your
population to 18-year-olds with a common trait instead of all high school students.
Exploration 15.13 Does Oat Bran Lower Cholesterol?
In the late 1980s, many people tried to lower their cholesterol by eating foods
high in oat bran. In 1990, The New England Journal of Medicine published a
study claiming that oat bran had not significantly lowered the cholesterol
levels of the participants more than wheat flour. Appearing in the New York
Times, these results dealt a fatal blow to the oat bran fad. Unfortunately, the
article did not mention that the participants were 20 healthy, middle-age
women, most of whom were nurses with low cholesterol levels.
(a) Why was it a mistake to generalize the study results to the entire U.S.
population?
(b) What would have been a fair generalization of these results?
Source: Swain, Rouse, Curley, and Sacks (1990).
Collecting and Analyzing Data
Suppose you wanted to make fitted hats for your classmates. A logical first step
would be to collect data. How would you measure your classmates’ hat sizes?
What tool(s) would you use? What units?
Once you have collected the data, what would you do next? Would you group
the measurements into categories like small, medium, and large? Would you average these numbers? Would you try to determine the largest and smallest head
sizes? Would you handle the data differently if you wanted to order hats instead
of making them?
All of these are reasonable questions about how you might collect and analyze
the data. We will explore some of these methods in greater detail later in Part II.
134 Advanced Quantitative Reasoning
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Interpreting the Results
Exploration 15.14 Three Types of Significance
The results of these three studies could all be considered significant in some
sense. In each case, how are the results significant?
(a) Two thousand students agreed to participate in a study on whether ginkgo
biloba (a medicinal plant) quickens toenail growth. Half were given ginkgo
and the other half served as a control. Toenails of the experimental group
grew, on average, 1 mm per month more than those of the control group.
Based on the sample size, this difference was calculated to be significant.
(b) A similar experiment was conducted with garlic instead of ginkgo biloba,
and with only 20 students. Although the difference was determined not
to be significant mathematically, the garlic group’s toenails grew 1 cm per
month more than those of the control group.
(c) A third experiment was conducted to measure the effect of ginkgo biloba
on brain cell growth. It was concluded that ginkgo does stimulate brain
cell growth.
In quantitative studies, the results are often interpreted using three measures:
­statistical significance, effect size, and practical significance. Statistical significance provides a mathematical measure of confidence in the results. Effect size is a measure
of the strength of the treatment. To understand the difference, consider the example
of two weight loss pills that are both tested and found to have a statistically significant effect. Does this mean the pills are equally effective? Not at all! Even though
they are both effective, one may cause people to lose 0.4 lb on average, and the
other may cause people to lose 6.1 lb. For this reason, it is always important to look
at effect size whenever you read about a study that claims something was found
to be statistically significant. The third measure of interpretation, practical significance, is arguably the most important.
• Statistical significance answers the question, “Is there likely an effect?”
• Effect size answers the question, “How strong is the effect?”
• Practical significance answers the question, “Why does the effect matter?”
As you might guess, statistics can only answer the first two questions; you must
use your life experience to answer the third.
Quick Review for Section 15
1.–10. True or False? Decide whether each statement is true or
false. If it is false, give a reason why it is false.
1.A population is a set of individuals or cases.
2.A statistical variable can be quantitative or categorical.
3.A quantitative variable can be discrete or continuous.
4.A categorical variable is a nominal variable.
5.A census collects data from an entire population.
6.A sample is a subset of a population.
7.A convenience sample is random.
8.Undercoverage bias occurs when some of the population is
excluded.
9.Nonresponse bias occurs when some of those sampled do not
participate.
10. Poorly worded survey items can cause response bias.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 135
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
Exercises for Section 15
1.–10. True or False? Decide whether each statement is true or
false. If it is false, give a reason why it is false.
1.Of observational studies, surveys, and experiments, random
sampling is only important in experiments.
2.It is possible for an experiment to achieve statistical significance
and a large effect size, but have low practical significance.
3.High quality professional surveys should have at least 30
questions.
4.Observational studies and experiments attempt to
manipulate a sample, whereas surveys try to capture the
sample in its natural form.
5.If you conducted an observational study on the number of
times your best friend laughed each day, the sample would
simply be your best friend.
6.Another term for random assignment is random sampling.
7.If subjects in an experimental group did not know they were
being watched and performed better on a mathematics test,
this would be an example of the Hawthorne effect.
8.Random sampling helps to account for the effects of all the
variables we are not measuring.
9.If you discovered that eating granola for breakfast improved
English scores in a random sample of 20 males from your
school, it would not be valid to generalize these results to the
females.
10. You would expect the following question to have an answer
with variability: “How fast can high school girls run 800 m?”
Observational studies, surveys, and experiments. In Exercises 11 and 12, consider the three situations and then answer the
questions.
11. Winter Attire and Illness
I. You have a sample of doctors complete a questionnaire
about the health risks involved with going out in the cold
without a jacket.
II. On cold days, you record which of your classmates come
to school without a jacket. Later, you record how many of
those students become sick.
III. You find 30 adults and divide them into two groups. The
first group is told not to wear jackets on cold days; the
other group is told to wear jackets on cold days. You then
compare the number from each group who get sick after
a string of cold days.
(a) Which of these three situations is an example of an observational study? Why?
(b) Which of these three situations is an example of a survey?
Why?
(c) Which of these three situations is an example of an experiment? Why?
12. College Attendance and Grades
I. You randomly select 60 students from your biology class
and randomly assign them into three groups. The first
group must try to attend every lecture, the second group
must miss exactly five lectures, and the second group
must miss exactly ten lectures (missed lectures are randomly chosen). At the end of a semester, you collect and
compare grades for the 60 students.
II. In a large biology lecture, you randomly select 25 students and keep track of their attendance throughout
one semester. When the semester ends, you record their
grades from the list posted at the front of the classroom.
III. You design a short questionnaire that asks students about
their attendance and grades in biology class.
(a) Which of these three situations is an example of an observational study? Why?
(b) Which of these three situations is an example of a survey?
Why?
(c) Which of these three situations is an example of an experiment? Why?
13.–20. Determine which of the following situations are examples
of (a) observational studies, (b) surveys, or (c) experimental studies. What is the population of interest in each study?
13. Mrs. Jackson teaches American History and wanted to help
her students do well on exams. She decided to conduct her
own research on test formats. She flipped a coin for each student in her classes. If the coin landed heads up, the student
took a multiple-choice test. If the coin landed tails up, the
student received a fill-in-the-blank exam. Afterward, Mrs.
Jackson compared the averages for the two test formats.
14. In World History, Mr. Maris put a question at the bottom of
an exam: “Which do you prefer, multiple-choice or fill-in-theblank questions?” Afterward, Mr. Maris tallied the results.
15. For advertising purposes, a grocery store owner wants
to determine the age ranges of her customers. She hires a
student to watch the security camera and make a tally of the
estimated age group for each customer.
16. A mathematics teacher wants to see how much time pressure
affects student scores on a quiz. She divides her 170 students
into two groups and then gives the same quiz to each group.
One group must finish the quiz within 10 min and the other
group has no time limit for completing the quiz.
17. Derek and Lauren want to see who has greater social appeal.
They go to a large department store. Each randomly selects
and approaches 25 customers while the other one times the
length of the interaction. They then compute the average
times of interaction as a measure of social appeal.
18. A doctor goes back through her records for the past year to
see whether more of her patients were males or females.
19. A journalist wants to increase the readership of the school
paper. He designs a set of questions that will discern the
types of articles most students are interested in reading.
136 Advanced Quantitative Reasoning
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
20. A university student interested in ethnography wants to
see whether students from rural areas adapt to college life
differently from those from urban areas. She asks 10 rural
students and 10 urban students to fill out a questionnaire and
then analyzes the data.
Sampling methods. In Exercises 21–26, identify the type of
sampling method used. Then, describe one advantage and one
disadvantage of using this particular method for the situation.
21. Santosh wishes to analyze lung cancer rates in the United
States, so he uses a random number generator to choose
five states and then researches the lung cancer rate for each
county within those five states.
22. The high school yearbook staff wants to actively recruit
upcoming first-year students. To inform their campaign, the
staff members design a survey, obtain a list of all the middle
school students at the school, and then randomly select 30 of
these students to fill out the survey.
23. Ellie has a hunch that most of the students in her school
are against the new dress code. At lunch time, she takes an
opinion survey of the girls sitting at her table.
24. The new history instructor wants to get some feedback on
her teaching. Because she feels that males and females might
perceive her differently, she makes a list of all her students
and then separates them by gender. Within each group, she
chooses a random selection of 10 students to ask for feedback.
25. The manager of Happy Hamburgers wants to test out three
proposed varieties of pizza fries. He asks his employees to set
up a stand at the local farmer’s market and give a free sample
to anyone who agrees to rate the three varieties of fries.
26. A large department store pays an employee to stand outside
and hand out a coupon to every 20th customer. The employee
then asks this customer two quick questions about his or her
shopping preferences.
Extending What You Have Learned
27. Interpreting sampling strategy. A Gallup poll asked, “Do you
think the U.S. should take the leading role in world affairs, take
a major role but not the leading role, take a minor role, or take
no role at all in world affairs?” Gallup’s report said, “Results
are based on telephone interviews with 1,002 adults nationwide, aged 18 and older, conducted Feb. 9–12, 2004.”
(a) What is the population for this survey? What was the
sample size?
(b) Gallup notes that the order of the four possible responses
was rotated when the question was read over the phone.
Why was this done?
28. Choosing among sampling strategies. Jack and Ralph are
stranded on a medium-size island and would like to estimate
the number of pigs living on the island. They are trying to
decide on the most appropriate sampling method. Jack thinks
they should walk the perimeter of the entire island, counting
how many pigs they see, and then adjust this number. Ralph
thinks they should wander around the entire island separately
for a day, counting the pigs that they see, and then compare
their numbers. Whose sampling method do you think would
produce a better estimate? Can you design a third method that
would be preferable to both of the other two methods?
Choosing a statistical design. In Exercises 29–36, explain how
you might design a research study to answer the given question.
Include the type of study, sampling strategy, and any other
information you think is important. Justify each of your decisions.
29. “Does volunteering change a person’s happiness level?”
30. “Do teachers at my school have more stress than students?”
31. “What do my friends like most about me?”
32. “Are girls better than boys at statistics?”
33. “Does playing video games increase hand–eye coordination?”
34. “What percent of area teenagers know the differences among
mean, median, and midrange?”
35. “Do high school seniors have more definite plans for their
careers than high school juniors?”
36. “Do students who arrive early to school have a different personality type from those who arrive late?”
Writing to Learn
37. Choosing a statistical question. Come up with your own question for a statistical study. Explain why you would expect the
answer to have variability, the type of study that would best
answer the question, and a tentative plan for conducting the
study. Make sure to choose a topic that actually interests you!
38. Alternate designs. Would it be possible to use a different type
of study to answer the question you asked in Exercise 37?
(E.g., if you chose to do an experiment, would it be possible
to use an observation or a survey to help answer the question
instead?) Explain why or why not.
Generalizing. In Exercises 39–44, briefly describe the sample
and the intended population and then state whether it is reasonable to generalize the results from the sample to the population.
If it is not reasonable, provide an alternate population to which it
would be more reasonable to generalize the sample results. Justify your decisions.
39. Jeff wants to study the effects of technology on rural areas in
the U.S., so he investigates three small towns in Iowa that are
based around growing corn and wheat.
40. Roberto designs a survey to see how frequently first-year college students call their parents. He chooses a simple random
sample of 300 first-year students at a small Catholic university near his home. He will then extend his results to firstyear college students nationwide.
41. Soo Ja attends an international high school with 300 students (that is, the school hosts students from a wide range of
­nations). The school cafeteria staff has asked her to compile a
list of foods preferred by the students, so she obtains a list of
all students who attend the school, and then selects the first
30 from the list to contact as her sample.
42. Jessica would like to start up a small business in her town.
She chooses a simple random sample of 40 people from her
community, based on numbers listed in the phone book. She
hopes that her findings from surveying these 40 people will
represent the views of the entire town.
43. Edgar is trying to determine the average daily precipitation
for the city where he lives. Every Sunday throughout the
year, he sets out a graduated cylinder and collects the precipitation for the entire day.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 137
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
44. Jasmine loves fruit smoothies, and thinks that they put her in
a better mood all day long. To test this hunch with an experiment, she asks her friends to rate how happy she appears
every day for two months. Without telling her friends, every
Thursday and Friday morning, she drinks a smoothie. She
will then compare her friends’ ratings for these days with the
ratings from the other days of the week.
45. Greasy hair = higher scores? Some people claim that math
scores can be improved by following a superstition such as
not washing one’s hair the day before an exam. Design an
experiment, including a pretest and posttest, that could be
used to test this claim. Then describe the advantages and disadvantages of removing the pretest from your design.
46. Does meditation lower blood pressure? Ever since the 1970s,
studies have been coming out with claims about the health
effects of meditation. One of the strongest claims is that
meditation can lower blood pressure. Design an experiment,
including a pretest and posttest, that could be used to test this
claim. Then describe the advantages and disadvantages of
removing the pretest from your design.
47. Are pretests necessary? Refer to Exercises 45 and 46 for examples of how pretests can provide important information.
Describe an experiment where a pretest would not be feasible
or would not provide useful information.
48. The Hawthorne effect in the mathematics classroom. The
use of calculators in the mathematics classroom has been a
hotly debated issue. Carla wants to test whether the daily
use of calculators would improve standardized mathematics
scores at her school, which currently does not allow calculators. She receives permission to introduce calculators into
two of the five mathematics classes at her school and plans to
determine how scores have changed in the five classes. What
steps could she take to guard against the Hawthorne effect?
[Hint: Remember that the Hawthorne effect occurs when subjects in an experimental group change their behavior because
they know they are being watched.]
49. The John Henry effect. Legend has it that, in the late 1800s,
a folk hero named John Henry was determined to show that
humans were better workers than steam-powered technology. He was pitted against a drill in a mining competition
and won, even though it cost him his life. The John Henry effect describes a situation where the control group tries harder
than normal, in order to compete against an experimental
group. Explain how the John Henry effect could distort the
results of an experiment.
50. Ethical experimentation. One obvious way to counter both
the Hawthorne effect and the John Henry effect (see Exercise 49) is to not inform subjects that they are in an experiment.
However, this approach is often considered to be unethical,
unless it can be shown that (a) the experiment could produce
results that are important to society, (b) subjects would not be
harmed, and (c) it is necessary to deceive the subjects by not
informing them of the experiment. Provide one example of an
experiment that would meet these three conditions.
138 Advanced Quantitative Reasoning
Random assignment vs. random sampling. Randomization
is perhaps the most powerful tool in statistical studies, but it is
important to understand how to use it. In Exercises 51–56, state
whether the process describes random assignment, random sampling, or neither. Justify your answer.
51. To study the effects of a new medicine on cholesterol in older
U.S. residents, 100 people are chosen at random from the
membership list of the AARP (formerly the American Association of Retired Persons).
52. After securing a sample of 50 hens, a researcher uses a random number generator to place 25 hens into a group that
will receive a large amount of human affection and the other
25 hens in a control group that will have no interaction with
humans.
53. Using a property map, a farmer randomly chooses 20 small
plots. She then uses Fertilizer A on the first five sections, Fertilizer B on the second five sections, and no fertilizer on the
remaining 10 sections.
54. See Exercise 53. The same farmer instead chooses a single plot
of land that is not in use. She then divides it into 20 subsections and forms the two experimental groups and the control
group at random.
55. A social worker designs an Internet survey to distribute to
100 families. She uses a program to scramble the questions
randomly on each survey.
56. A biologist wants to test the water quality of a river. He takes
three samples from each mile mark along a 15-mi stretch of
the river.
Interpreting results. In Exercises 57–62, state whether the given
result expresses statistical significance, effect size, or practical significance. Provide an example of the other two types of interpretation (e.g., if the scenario describes an effect size, find a result of
statistical significance and of practical significance for the given
scenario).
57. Refer to Exercise 40. Roberto determines that first-year male
students call home more frequently than do first-year female
students.
58. Refer to Exercise 41. Based on Soo Ja’s findings, the kitchen
chefs decide to include more rice dishes on the school menu.
59. Refer to Exercise 44. Jasmine uses the final results of her
study as evidence for increasing the frequency with which
she drinks smoothies.
60. Refer to Exercise 48. Carla finds out that the students who
used calculators in their classrooms scored, on average, 20%
higher than those who didn’t.
61. Refer to Exercise 52. After two months, the hens are tested for
egg production rate. The hens in the high human interaction
group lay eggs, on average, at twice the rate of the control
group.
62. Refer to Exercise 55. Of the 100 surveys, 62 are completed and
returned. The social worker discovers that 20 of the families
feel that the local school provides a safe environment for
learning, and that the other 42 feel that the school is unsafe.
The social worker determines that there is significant concern
about school safety within the community.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Part II Probability and Statistical Reasoning
Section 15 Designing a Statistical Study
63. The people who watch the people who watch television.
Have you ever wondered how television show ratings are
determined? Do all of our televisions have some kind of
microchip built in that sends out signals to Hollywood? No,
but this isn’t far off. Conduct a search—either on the Internet
or at your library—on “people meters.” Then answer the
­following questions:
(a) How have researchers managed to solve this complex
problem of measuring the television watched by an entire
country?
(b) The method used is claimed to be a representative sample
of the U.S. population. Do you agree or disagree with this
claim? Explain your reasoning.
64. Discerning relevant information and writing to learn. Read
the following study on relaxing video games and then write a
paragraph analyzing the statistical design. Be sure to include
the type of study, sampling strategy, types of results, and
overall weaknesses and strengths. Explain why you think the
conclusion of the study was or was not valid.
What You Play Is What You Are
A recent study suggests that playing peaceful video games
may have beneficial effects on the player, just as playing aggressive video games have been linked to aggressive behavior
in players. Whitaker and Bushman conducted the study and
reported it in the January 2012 issue of Social Psychological and
Personality Science.
The authors designed two experiments to test the effects
of playing relaxing video games on players. In each experiment, 150 participants were randomly assigned to one of
three groups. The first group played a relaxing game such as
Endless Ocean; the second group played a neutral game such
as Super Mario Brothers; the third group played an aggressive
game such as Resident Evil. In the first experiment, after playing the video game for 20 min on the Wii system, participants
were given the opportunity to respond to an unseen opponent. (Participants did not realize that there actually was no
opponent.) In some scenarios, they could control the volume
of a noise blasted at their opponent, and in other ones, they
were able to choose the amount of money given to the opponent. In the second experiment, after playing the game for
20 min, participants reported their mood and were given the
option to help the experimenter by sharpening pencils.
Whitaker and Bushman found that participants from the
peaceful game group chose softer noises for their opponents
than did members of the neutral game group. The neutral
game players chose softer noises than the aggressive game
group. The peaceful game group gave more money to their
opponent than did members of either of the other groups.
In the second experiment, participants in the relaxing-game
group reported a better mood and were more helpful to the
experimenter than were members of the other two groups.
The researchers concluded that playing relaxing video
games made people kinder and less aggressive.
Copyright © 2015 by Gregory D. Foley, Thomas R. Butts, Stephen W. Phelps, and Daniel A. Showalter
Advanced Quantitative Reasoning 139