Questions for the course "Scientific Methods" with the topics: Evidence-based software engineering (lecture 5).
Click the button to start the quiz
Why do we need evidence-based research?
To synthesise the best available evidence, based on multiple studies
Multiple studies might lead in different directions
Aggregate results from multiple studies
What does SLR stand for?
Systematic Literature Review
What is a primary study?
Empirical study in which we directy make measurements about the objects of interests
What is a secondary study?
An analysis of a set of studies, and usually seeks to aggregate the results from these, to provide a stronger forms of evidence
What are forms of secondary studies?
Systematic Literature Review (SLR)
Meta study
Meta analysis
Systematic Literature Mapping
Systematic Mapping Study (SM)
What are key aspects aspects of a systematic literature review?
As described by kitchenham et al. (2007)
A form of a research method
Qualitative or quantitative
What are the phases in a systematic literature review?
As described by kitchenham et al. (2007)
Plan review
Conduct Review
Document review
What are the steps in phase 1 of a systematic literature review?
As described by kitchenham et al. (2007)
Establish need for SLR
Specify RQ
Develop protocol
Validate protocol
What are the steps in phase 2 of a systematic literature review?
As described by kitchenham et al. (2007)
Identify research
Select studies
Assess quality
Extract data
Synthesise
What are the steps in phase 3 of a systematic literature review?
As described by kitchenham et al. (2007)
Document
What are the key factors in step 0, "Establish need for SLR", of a systematic literature review?
As described by kitchenham et al. (2007)
Summarise the existing evidence concerning a treatment or technology
Identify any gaps in current research in order to suggest areas for further investigation
Examine to the extent to which empirical evidence supports / contradicts theorectical hypotheses, or even to assist the generation of new hypotheses
What are the key factors in tep 1, "Specify RQ", of a systematic literature review, with focus on quantitative?
As described by kitchenham et al. (2007)
Comparison of two (or more) technologies to determine which one is more effective or efficient in a certain context.
What are the key factors in tep 1, "Specify RQ", of a systematic literature review, with focus on qualitative?
As described by kitchenham et al. (2007)
Understandingthe benefits, risk, impacts of adopting a specific technology
What does PICOC stand for?
Population: Roles (testers, developers), expertise, industry domain (telecommuncation, automative, etc.)
Intervention: Technology of interests, e.g., TDD, pair programming
Comparison: Test-last development, solo programming
Outcome: Quality (Number of defects, time to complete a task
Context: Academia - industry, SME - Large coporation
What are the key factors in step 2, "Develop protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Understand the background
Hinges on the research questions
Defines the search strategy
Defines the study selection criteria
Defines the study selection procedues
Defines the study quality assessment checklists and procedures
Defines the data extraction strategy
Defines how synthesis of the extracted data is handled
What are the key factors in tep 3, "Validate protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Pilot the protocol
Refine the protocol
What are some typical questions you can ask, in regards to refining the protocol in step 3, "Validate protocol", of a systematic literature review?
As described by kitchenham et al. (2007)
Is the motivation clear?
Are the research questions clearly stated?
Is the search strategy clearly defined?
Are the inclusion and exclusion criteria clearly defined?
What are the key factors in step 4, "Identify research", of a systematic literature review?
As described by kitchenham et al. (2007)
Search for relevant primary studies that fulfil the goals of the SLR and answer the research questions
An iterative process
Important to have a basis of completeness
What are the typical search strategies in step 4, "Identify research", of a systematic literature review?
As described by kitchenham et al. (2007)
Automated search
Manual search
Snowballing
What are some key factors of automated search, when searching for primary studies, in a systematic literature review?
Use digital libraries, e.g. IEEE Xplore, ACM Digital Library, etc.
Use services like scopus and Web of Science
Need to compose search strings
Search string usually consists of keywords and operators related to the research questions
Iterative process of composing and refining search strings
What are some key factors of manual search, when searching for primary studies, in a systematic literature review?
Manually search primary studies from journals and conference proceedings
Identify leading software engineering journals and conferences, examples being: ICSE, IEEE, FSE/ESEC, TSE, SoSym, etc.
What are some key factors of snowballing, when searching for primary studies, in a systematic literature review?
Citation anaylsis
Backward, where you look at the references of known papers
Forward, Where you look at other papers that have cited known papers
Suitable where there are a small number of pappers or in new areas
More precise in identifying relevant studies than automated search
Is a single search strategy for primary studies, in a systematic literature review, enough?
No
What are the key factors in tep 5, "Select studies", of a systematic literature review?
As described by kitchenham et al. (2007)
Select studies to include in the SLR
Define inclusion and exclusion criteria
Using a Kappa analysis on a scale from 0 to 1, with multiple reviewers
Funneling process
What are the key factors in step 6, "Assess quality", of a systematic literature review?
As described by kitchenham et al. (2007)
Determine the extent to which the results of an empirical sudy are valid and free (minimised) from bias
The variation of quality needs to be considered when synthesissing the results
A high quality study should carry a heavier weight when synthesising the results
Quality criteria coudl differ between SLR studies
Quality can be assessed using Rigour - Precision or exactness of the research method used
Quality can be assessed using Relevance - Evaluation of the academic relevance or research impact in industry
Multiple people can do the quality assessment, however either aggregation or consensus is needed for scores
In SLR, when accessing quality, what does Rigour include?
Based on Ivarsson and Gorschek, 2011.
Context described
Study design described
Validity discussed
In SLR, when accessing quality, what does Relevance include?
Based on Ivarsson and Gorschek, 2011.
Subjects
Context
Scale
Research method
What are the key factors in step 7, "Extract data", of a systematic literature review?
As described by kitchenham et al. (2007)
Extract the data needed to address the research questions
Could be numerical data or textual data
Uses tools like spreadsheets, analysis tools (NVivo), etc.
What are the key factors in step 8, "Synthesise", of a systematic literature review?
As described by kitchenham et al. (2007), with respect to the primary studies of a common topic.
Summarise
Integrate
Combine
Compare
What are typical types of combination strategy in step 8, "Synthesise", of a systematic literature review?
As described by kitchenham et al. (2007).
Aggregation (qualitative)
Meta-analysis (quantitative)
Vote-counting (quantitative)
What does uantitative synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Statistical infromation, like correlation, descriptive statistics, etc.
What does vote counting in synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Counting (or otherwise summarising) the number of studies that show significant efffect
What does meta-analysis in synthesis in step 8, "Synthesise", of a systematic literature review, focus on?
Statistical methods to analyse data from primary studies that compares two different treatments
What should you be aware of, when documenting the SLR in step 9, "Document", of a systematic literature review?
As described by kitchenham et al. (2007)
The audience
Format of the report
What is a Systematic mapping (SM) study?
Secondary study intended to identify and classify the set of publication on a topic
Give an overview of a research area through classification
Counting contributions in relation to the categories of that classification
What are research questions typical in a Systematic mapping (SM) study?
Aimed at discovering research trends
What are research questions typical in a Systematic literature review (SLR) study?
Aimed at aggregating evidence pertaining to the impact of an intervention
What are quality assesment tools in a Systematic mapping (SM) study?
Nice to have
What are the outcome typically in a Systematic mapping (SM) study?
Classification of primary studies
What are the outcome typically in a Systematic literature review (SLR) study?
Qualitative or quantitative synthesis
What are the steps in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Need identification & Scoping -> Aim & RQ
Conduct search -> All papers
Study selection -> Relevant papers
Keywording using abstract (classification) -> Classification scheme
Data extraction & mapping -> Systematic map
Document
What are the key factors of step 1, "Need identification & Scoping", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Aimed at identifying trends and gap in literature
What are the main search strategies step 2, "Aim & RQ", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Manual search
Automated search
Snowballing
What are the key factors of step 3, "Conduct search", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Similar to SLR
Relevant to the focus area and within scope
Within the domain
Language scope
What are some of the ways to find topic-dependenden classifications of step 4, "Classification", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Use keywords use in the abstract or list of keywords mentioned in the primary studies
Use existing classification from ISO standards or SWEBOK
Find other similar mapping studies
What are some of the ways to find topic-independenden classifications of step 4, "Classification", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Venue and research method
Research type (Wieringa et al., 2006): evaluation, solution proposal, validation, etc...
What are som keyfactors of step 5, "Data extraction & mapping", in a Systematic mapping (SM) study?
As described by Petersen et al. (2008)
Data is extraced and mapped to the classification scheme
Visualiaztion such as bubble plots, histograms and tables
What are tertiary studies?
Review of secondary studies
What are some of the motiviation behind doing tertiary studies?
Mapping of existing secondary studies
Aggregation of findings in secondary studies
Evaluation of existing secondary studies
What is grey literature?
Non peer-reviewed publications
Often done, when not extensively enough research is done on the topic
Common in education and nursing
What shades of grey literature are there?
1st: High outlet control & credibility: Books, magazines, government reports, white papers, etc.
2nd: Moderate outlet control & credibility: Annual reports, news articles, presentations, videos, Q&A sites, Wiki articles, etc.
3rd: Low outlet control & credibility: Blogs, emails, tweets, forum posts, etc.
What is multivocal literature review?
SLR of both white and grey literature
What is a replication study?
Repeating a study to verify the results are reproducible
Establish creddibility of scientific studies
Increase the validity of the original study
Generalize original study
What is the difference between reproduce and replicate?
Specifically in the context of replication studies.
Reproduce: Repeating the same study with the same data and methods
Replicate: Repeating the same study with different data and the same analysis methods