Questions for the course "Scientific Methods" with the topic: Validity threats (lecture 9).
Click the button to start the quiz
What is a Validity threat
Anything that can compromise the outcome of your study
Inappropriate election of statistical test
Incorrect measurement instrument in an experiment/survey
Incorrect formulation of interview/survey questions
Biased interpretation of interview transcripts
What are some typical examples of validity threats in interview or survey questions?
Leading questions
Double-negative
What is wrong with this interview question?
How likely are you to recommend working with the design team?
Nothing
What is wrong with this interview question?
What problems do you have with the design team?
Leading question
What is wrong with this interview question?
How would you rate the clarity of the instructions?
Nothing
What is wrong with this interview question?
Were the instructions not unclear?
Double-negative
Why is it important to discuss validity threats?
To communicate, that there is limitations to any study
To highlight research honesty and integrity
To view that the study is trustworthy
Correctly argue the finding of the results without misleading the reader
What are the categories of validity threats?
Internal
external
Construct
conclusion
Descriptive
Interpretive
What does internal validity threats refer to?
Factor/factors that might affect cause and effect relationships but is/are unknown to the researcher
What would a likely validity threat be in this study?
you would like to investigate the relationship between age and gameplay performance. The results indicated negative correlation between age and gameplay performance, which implies that the gameplay performance decreases as the age increases
Internal
What does external validity threats refer to?
Conerned with the ability to generalize the results
What does construct validity threats refer to?
Concerned with whether we measured or captured, what we intend to in relation to our hypothesis or theory to test
What would a likely validity threat be in this study?
Assume a researcher would like to assess the subject's experience in a programming language, measuring the number of courses at the university in computer science.
Construct
What does conclusion validity threats refer to?
The degree to which conclusions/inferences we draw (e.g. about relationships between variable, or based on qualitative data) are reasonable
What does descriptive validity threats refer to?
Related to factual accuracy of the account/data; that is, the researchers are not making up or distorting the things they observed and it is expected to produce descriptively same accounts/data for the same event or situation
What does interpretive validity threats refer to?
Concerns with wether the inferences/conclusions follow from the account (data), not biased by the researchers during analysis
What are some actiosn that can be done, in order to deal with validity threats?
Prevent
Mitigate
Admit
How would you prevent validity threats?
Ensure instruments are well developed
Ensure instruments are verified for correctness
Piloting the study whenever possible
Data collected is alligned with the theory or research question
Select appropriate analysis method for the data
How would you mitigate validity threats?
Redo the study
Avoid inferential statistics
Discard any invalid answers
Triangulate
What does triangulation refer to?
In regards to validity threats
Taking different angles of a phenomenon, to widen the view and increase precision
What types triangulation are there?
In regards to validity threats
Data triangulation
Observer triangulation
Methodological triangulation
Theory triangulation
How would you admit validity threats?
Discuss
Aknowledge
What are some typical examples of validity threats in systematic literature reviews in software engineering?
Incorrect or incomplete search terms in automatic search
iinadequate search strategy
Restricted time span
Bias in study selection
Subjective interpretation about the extracted data
Subjective quality assessment