Reliability and Validity

Reliability and Validity

Psychology Research Methods (IV)



Reliability and validity are essential to measurements and research procedure designs. These two concepts are distinct yet related. While high reliability does not warrant validity, a study cannot achieve validity without reliability. Reliability Reliability is the consistency of your measurement, or the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects and the testing procedure is free from random errors of measurement. It asks the question: "Did we measure accurately?" For example, if you have a test consisted of math questions. The test and the testing procedure have to be able to yield consistent results on repeated trials in order to satisfy the reliability criterion. Validity Validity is how close what is being measured on the paper is to what we intend to measure in our theory. It eventually leads to how close our conclusion based on the measurement results is to the truth. It asks the question: "Did we measure what we were supposed to measure?" For example, suppose the above mentioned math test is intended to be part of an IQ test. Even if it is a highly reliable test, one can still ask the question: Are the math skills measured by this test really a component of the IQ? Or, does the test result reflect a person's IQ? Or, to which degree does a person's math score indicate his or her intelligence level? While high reliability does not warrant validity, a study cannot achieve validity without reliability.

Correlation and Causality

Psychology Research Methods (I)

Correlation

In psychology, when two variables (e.g., self-esteem and popularity) are tested and determined to have a relationship, they are said to be "correlated". They can either be positively correlated, that is, the two variables increase and decrease at the same time. For example, when self-esteem increases, popularity increases at the same time. On the other hand, the relationship between two variables may also be identified as a negative correlation, which can be just as important as a positive correlation. Two variables are negatively correlated when one goes up while the other goes down, and vise versa. For example, when self-esteem increases the rate of teenage pregnancy decreases. Correlation coefficient, which can range from -1.00 to +1.00, is a measure of the direction and the strength of a correlation.

Correlation between variables is established through correlational studies in social psychology. (See "Types of Psychology Research".)

Causality

Correlation simply means that the two factors occur at the same time. It does not say anything about their causal relationship (AKA cause-effect relationship) - which variable is the cause; which variable is the result. Using the self-esteem and popularity example, as far as causality is concerned, there are three possibilities:

(1) high self-esteem leads to popularity,
(2) popularity leads to increased self-esteem, and
(3) there is no causal relationship between self-esteem and popularity.

Instead, both are the result of another factor or a group of other factors such as academic achievement and athleticism.

Again, an established correlation between two factors does not necessarily indicate the direction of the cause-effect relationships or that a cause-effect relationship definitely exists. However, if a strong relationship is found between two variables, causality can be tested by using experimental studies. (See "Types of Psychology Research".)

Theory and Hypothesis

Psychology Research Methods (II)

A theory is a set of statements or principles devised to explain a phenomenon, especially if they have been repeatedly tested or have been widely accepted and can be used to make predictions about natural phenomena.

A hypothesis is a tentative explanation for an observation, phenomenon, or scientific problem that can be tested by further investigation. Although in casual usage, the two terms are sometimes interchangeable, their differences in psychology research include the following:

A theory has already been tested or is widely accepted; A hypothesis is a tentative assumption to be tested in psychology studies.

Statements in a theory usually contain abstract and unobservable concepts; A hypothesis is usually specific and concrete.

A theory is about the bigger picture and the relationships of the parts in the bigger picture; A hypothesis is directly about a specific phenomena, usually the relationship between two variables.

The role of theories in psychology research is to organize old knowledge, guide new research, and predict future results; Hypotheses are necessary because those are what the studies are set out to prove.

Types of Research

Psychology Research Methods (III)

There are two major types of psychology research: Correlational studies and Experimental studies. Each has a variety of different settings and controls.

Correlational Studies

Purposes:

Correlational studies aim at identifying relationships between variables. There can be three kinds of turnouts: No relationship, positive correlation, and negative correlation.

Limitations: Correlation =/= Causal Relation

For example, two variables: (A) High Self-esteem and (B) Academic Achievement.
Does (A) lead to (B)? Or is the other way around? Or, are there some mysterious other factors that lead to both (A) and (B)?

As it turned out, two independent carefully conducted studies found that there is no causal relationship between these two factors. They are correlated because both of them are correlated to some other factors: intelligence and family social status. When these factors were controlled, the correlation between academic achievement and self-esteem disappeared.

Be very careful about this.

Another example, does college education lead to higher income? Or does socio-economic background lead to both college education and higher income? In order to study cause-effect relationship, we have to hold social-economic factors and all other factors constant. That can be done in experimental studies can. But not in correlational studies. Correlational studies may hint or suggest that one variable influences another, but they are never proof of causality.

Types of Research

Psychology Research Methods (III)

Types of Correlational Studies

1. Natural Observation

Advantages:

Subjects' behavior is natural when the observation is non-obtrusive.
It provides information useful in generating hypotheses, which can the tested in experiments.
When other methods are not feasible, observation is better than nothing.

Disadvantages:

Natural observation is not scientific, because in an uncontrolled environment it is unclear what all the influential factors are.
Time consuming.
Some things, such as thoughts, cannot be observed.
When the observation is obtrusive, subjects are aware of being watched, which influence their behavior.
It may violate people's privacy.

Make an Observation More Scientific

What to observe: Just the facts
Systematically record data
Random sampling helps generalize the results

Types of Correlational Studies

2. Scientific Survey and Test

Random sampling:

Survey procedure in which every person in the population being studied has an equal chance of being chosen to be in the study.

Random sampling is key; number of people surveyed is not. No matter how large the population is, a survey only needs about 500 to reach a 95% confidence with plus or minus 5% accuracy. Opinion polls run by media often are not scientific no matter how many people have participated in their polling, because they are not random samples.

It is almost impossible to achieve true, absolute random sampling. There will always be under-represented parts of the population. For example, phone surveys would exclude people who do not own telephones.

Scientific Design of Questionnaires / Tests

Factors:

Order of the questions
Options
Wording: (E.g., people tend to give affirmative answers.)
1. Avoid leading questions, which give hints about what kind of answers you are expecting. For example, "Psychology is an interesting subject, right?"
2. Make sure the questions are straightforward and easy to understand.
3. Ask question from both confirmatory (e.g., "Do you find psychology interesting?") and negative (e.g., "Do you find psychology boring?") perspectives because people have the tendency to answers "Yes" more often than "No".

Advantages of the Survey Method

Flexible: You can ask just about anything
Less time consuming than natural observation
Can collect a large amount of information

Disadvantages of the Survey Method

A survey is only as good as the sample and as good as the questions
Subjects' responses are not always truthful. Reasons include:
1. Social desirability bias: People may try to make themselves look better than they are. In other words, they lie.
2. Try to please the researcher.
3. Erroneous memories.
4. Not sure about the reason behind their behavior or feelings but make up answers as they go through the survey.

Types of Correlational Studies

3. Case Study

Advantages:

Rich source of information
Intensive study possible
Can focus on rare, unique, and extreme cases

Disadvantage:

This method is not scientific - nonrandom sample, non-generalizable conclusions.
Observer may be biased.
Subject is usually aware that they are being studied.

Types of Correlational Studies

4. Archive Research

Archive research is reviewing and analyzing records by previous researchers.

Advantages:

Experimenter cannot affect behavior
Large volume of data enables relationships to emerge

Disadvantage:

Researcher has no control over how data are collected
Important information may not have been recorded

Causal Studies: Experiments

An experiment is used to determine causal relationship between variables. An experiment is a cohort (group of subjects) study in which the investigator manipulates the predictor variable-otherwise called the treatment, program or intervention-and then observes the outcome. Experiments can be conducted in real situations (Field Experiments) or research labs (Lab Experiments). Experiments are conducted by following the rules and
guidelines of the scientific method.

A. Manipulate the Independent Variable

- This is the factor that the researcher manipulates.
- The hypothesized causal factor. E.g., Antidepressant Medication: Yes or No.
- Participants are random assigned to groups based on the independent variable. E.g., Group 1 gets antidepressant; Group 2 gets placebo.
- They are also called "experimental condition" and the "control condition".

B. Measure the Dependent Variable

- The variable being measured. E.g., relief in depression symptoms.
- A result of the independent variable.

C. Random Sampling

- A systematic way of recruiting experiment's participants, during which every person in the targeted population has an equal chance of being chosen to be in the study.
- Random sampling helps us generalize our findings to a population.

D. Random Assignment

- Random assignment is the process of assigning participants to the conditions of an experiment such that all persons have the same chance of being in a given condition.
- Random assignment neutralizes factors other than the independent variable and the dependent variable.
- Random assignment helps us infer cause and effect.

E.g., A random sample of people who suffer from depression is randomly assigned to two conditions: (a) Medication and (b) Placebo. Therefore, the two groups are equal in their socio-economic background, health, and looks. The only difference is in the independent variable -- in this case, medication condition: antidepressant or placebo. After both groups receive their pills for two weeks, the independent variable is measured.

E. Double-Blind Experiments

Double-blind experiments are the most objective kind of experiments, during which neither the experiment administrators nor the participants know who is receiving what experiment treatment. In the above example, each participant does not know if he or she is given the antidepressant or the placebo. Neither do the "experimenter's helpers", who dispense the pills and who measure the participants' symptoms.

F. Advantage of experiment

It can establish causation (i.e., cause-effect relationship) between variables

Science: Correlational vs. Experimental

Objectivity, reproducibility, measurements guided by theory, and bases for the development of new theory, which explains the research findings in a more general environment - these are some of the components of scientific methods of conducting psychology research.

In everyday language, the term "experiment" is used to refer to any scientific study. But in fact, nonexperimental does not mean nonscientific. As discussed before, through scientific design of the questions and random sampling of participants, correlational studies can be scientific, too. What correlational studies fall short in comparison with experimental studies is not that they can't be scientific; it is that they can't establish causality between the variables being investigated. But is that really a shortcoming. We, scientists and social scientists, as well as lay people, often favor the experimental method because there we can manipulate research conditions and we can make conclusions about cause and fact. We like doing those as we like to control and to explain why things happen. What we should also remember, however, correlational studies have the following advantages:

Correlational studies can provide useful information on what (if not why) people do, think, and feel.
Similar to everyday activities, they often are easier to carry out than experiments. What you do is look (observation), ask (survey / test), read and think (archives).
They are your only choice when an experiment cannot be carried out for ethical or practical reasons (e.g., certain variables can't be manipulated)
They are the main research method for many fields, especially those in behavioral science (e.g., psychology, sociology, and anthropology).

Reliability and validity are essential to measurements and research procedure designs. These two concepts are distinct yet related. While high reliability does not warrant validity, a study cannot achieve validity without reliability.

http://psychology.about.com/ (2002)