Research method- experimental design part I

The three principles of experimental design: replication, randomization, and randomized block design.

Treatment effect = signal / noise = the mean difference / the variability; the mean difference is still the same as it was for the entire sample. But also notice that the variability of the blocking is much less than it was for the entire sample.

Mill (1859) the principles: The first principle was that the putative or presumed cause must occur before the effect. The second was that the effect must always occur when the presumed cause occurs. Third, the effect must not occur when the presumed cause is absent. Fourth, the presumed cause must be isolated from other potential causes of the effect. Fifth, the presumed cause must be produced artificially to ensure that the cause is isolated from all other potential causes. - The fourth and fifth principles distinguish experimentation from other forms of research in that the experimenter must be able to manipulate the presumed cause and isolate the effects of the presumed cause from other events or variables.

Flannelly et al., (2018) - Threats to Internal Validity 

The 7 threats: history, maturation, testing, instrument decay, statistical regression, selection, and mortality (Campbell, 1957). - they pose alternate explanations for the apparent causal relationship between the independent variable and dependent variable of an experiment if they are not adequately controlled. 

Threat: “the operation of some extraneous variable causes the observed values of the dependent variable to inaccurately reflect the effect of the independent variable” - observed effect of the experiment is not due to the independent variable, but to the extraneous variable. 

History: might be called experience - a study participant experiences during the course of an experiment that are not part of the experiment itself.

Maturation: maturation involves bodily changes. - any biological changes that occur with the passage of time, such as age-related biological changes, becoming hungry, tired, or fatigued, wound healing, recovering from surgery, and disease progression (e.g., stages of cancer or other diseases). 

Ex. The physical condition of people who live in nursing homes is getting worse not due to poor care of employees. The physical functions of most elderly people will naturally degrade.

Testing: any form of measuring study outcomes in study participants. - testing becomes a threat to internal validity if the test itself can affect participants’ responses when they are tested again. - Campbell (1957) called such a test a “reactive measure.” 

Ex. This can happen when the test measures participants’ knowledge regarding a topic and they learn the correct answers to the questions before taking the test again. It could also happen when a test measures participants’ attitudes about a topic and they alter their responses when tested again to give more socially acceptable answers to questions. 

Improve: Measures or tests are less likely to be reactive when they are part of a regular routine, such as measures of temperature and blood pressure. - the degree to which a researcher can incorporate a measure of the interest outcome into the study participants’ usual routine. - embed it in an array of other measures, especially measures that may distract participants from the focus of the study

Instrument Decay: applied to any means of measuring the dependent variable. - reflect the fact that any change in measurement ability can pose a threat to internal validity a researcher using a battery-powered device to measure blood pressure, then the battery has decayed during the month so all the blood-pressure readings taken by the device are lower on the posttest than they were on the pretest.  

Statistical Regression: the tendency for individuals who score extremely high or extremely low, relative to the mean or average, on an initial measure of a variable to score closer to the mean of that variable the next time they are measured on it. Statistical regression is more accurately referred to as regression toward the mean. 

Reason: Even without any experimental intervention, these people are likely to score higher/lower on that measure the next time. 

Selection: a potential bias in selecting the participants who will serve in the experimental and control groups; hence it is also known as selection bias. 

The individuals who are assigned to the experimental and control groups differ from each other in some important ways; that is, that the groups are not equivalent. Selection bias would be an obvious problem if participants were allowed to choose whether they participated in the control group or the experimental group (i.e., self-selection), but selection bias is usually more subtle.

Ex. the parents who immediately enrolled their families in the study were more motivated to reduce their child’s weight than were the parents who enrolled their families later - the results of the experiment may reflect differences in the families assigned to the experimental and control groups, rather than the effectiveness of the experimental treatment. 

Mortality: (sometimes called experimental mortality) refers to the differential loss of study participants in the experimental and control groups. = the drop-out rate, or simply attrition.

The drop-out rate is more likely to be higher for an experimental group than a control group because experimental procedures usually make more demands on and require more commitment and effort from participants.  If less committed individuals drop-out of the experimental group, the results may suggest that the intervention is more beneficial than it actually is.

Ex. The subjects of experiment group need to exercise several weeks. Those who exercise regularly or want to exercise regularly are more likely than other individuals in the experimental group to complete an exercise intervention. If less committed individuals drop-out, then those who are willing to stay in the experiment (= the selective subgroup) have greater motivation to exercise. Hence, the result will show the effect is better than it actually is.

An example of a 2 by 3 factorial design- 2*3 factorial design, including 2 levels of one variable, and 3 levels of another variable. For example, we want to know the relationship between the way of teaching (2 levels: online and in-person) and student's performance (3 levels: low, middle, high). Through 2*3 factorial design, we will have six possible conditions: online-low, online-middle, online-high, in-person-low, in-person-middle, and in-person-high. We compute the mean score of exam of each condition, and then we compute the main effect. Next, we draw the graph to examine whether there is an interaction effect or not. If the graph shows "X-shaped," it means there is an interaction. Then we have to compute the interaction effect (in this case, the main effect does not meaningful).

The difference between selection bias and sampling bias- Generally, selection bias is related to internal validity, but sampling bias is related to external validity. Selection bias refers to the difference between treatment and control groups is due to the original difference not intervention. Hence, selection bias is internal validity threat because it shows the possibility of alternate explanation in the study. On the other hand, sampling error may happened by chance or human such as typo error. This leads to the sample of study cannot represent the population. Hence, sampling error affects generalization (external validity).

Reference

Flannelly, K. J., Flannelly, L. T., & Jankowski, K. R. (2018). Threats to the internal validity of experimental and quasi-experimental research in healthcare. Journal of health care chaplaincy, 24(3), 107-130.

Mill , J. S. ( 1859 ). A system of logic, ratiocinative and inductive; bring a connected view of the principles of evidence and the methods of scientific investigation . New York : Harper & Brothers .