Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection
| Title: | Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection |
|---|---|
| Authors: | Herbold, Steffen, Trautsch, Alexander, Trautsch, Fabian, Ledel, Benjamin |
| Publication Year: | 2019 |
| Collection: | Computer Science |
| Subject Terms: | Computer Science - Software Engineering |
| More Details: | Context: The SZZ algorithm is the de facto standard for labeling bug fixing commits and finding inducing changes for defect prediction data. Recent research uncovered potential problems in different parts of the SZZ algorithm. Most defect prediction data sets provide only static code metrics as features, while research indicates that other features are also important. Objective: We provide an empirical analysis of the defect labels created with the SZZ algorithm and the impact of commonly used features on results. Method: We used a combination of manual validation and adopted or improved heuristics for the collection of defect data. We conducted an empirical study on 398 releases of 38 Apache projects. Results: We found that only half of the bug fixing commits determined by SZZ are actually bug fixing. If a six-month time frame is used in combination with SZZ to determine which bugs affect a release, one file is incorrectly labeled as defective for every file that is correctly labeled as defective. In addition, two defective files are missed. We also explored the impact of the relatively small set of features that are available in most defect prediction data sets, as there are multiple publications that indicate that, e.g., churn related features are important for defect prediction. We found that the difference of using more features is not significant. Conclusion: Problems with inaccurate defect labels are a severe threat to the validity of the state of the art of defect prediction. Small feature sets seem to be a less severe threat. Comment: Accepted at Empirical Software Engineering, Springer. First three authors are equally contributing |
| Document Type: | Working Paper |
| DOI: | 10.1007/s10664-021-10092-4 |
| Access URL: | http://arxiv.org/abs/1911.08938 |
| Accession Number: | edsarx.1911.08938 |
| Database: | arXiv |
| FullText | Text: Availability: 0 CustomLinks: – Url: http://arxiv.org/abs/1911.08938 Name: EDS - Arxiv Category: fullText Text: View this record from Arxiv MouseOverText: View this record from Arxiv – Url: https://resolver.ebsco.com/c/xy5jbn/result?sid=EBSCO:edsarx&genre=article&issn=&ISBN=&volume=&issue=&date=20191120&spage=&pages=&title=Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection&atitle=Problems%20with%20SZZ%20and%20Features%3A%20An%20empirical%20study%20of%20the%20state%20of%20practice%20of%20defect%20prediction%20data%20collection&aulast=Herbold%2C%20Steffen&id=DOI:10.1007/s10664-021-10092-4 Name: Full Text Finder (for New FTF UI) (s8985755) Category: fullText Text: Find It @ SCU Libraries MouseOverText: Find It @ SCU Libraries |
|---|---|
| Header | DbId: edsarx DbLabel: arXiv An: edsarx.1911.08938 RelevancyScore: 994 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 993.823486328125 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Herbold%2C+Steffen%22">Herbold, Steffen</searchLink><br /><searchLink fieldCode="AR" term="%22Trautsch%2C+Alexander%22">Trautsch, Alexander</searchLink><br /><searchLink fieldCode="AR" term="%22Trautsch%2C+Fabian%22">Trautsch, Fabian</searchLink><br /><searchLink fieldCode="AR" term="%22Ledel%2C+Benjamin%22">Ledel, Benjamin</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2019 – Name: Subset Label: Collection Group: HoldingsInfo Data: Computer Science – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+Science+-+Software+Engineering%22">Computer Science - Software Engineering</searchLink> – Name: Abstract Label: Description Group: Ab Data: Context: The SZZ algorithm is the de facto standard for labeling bug fixing commits and finding inducing changes for defect prediction data. Recent research uncovered potential problems in different parts of the SZZ algorithm. Most defect prediction data sets provide only static code metrics as features, while research indicates that other features are also important. Objective: We provide an empirical analysis of the defect labels created with the SZZ algorithm and the impact of commonly used features on results. Method: We used a combination of manual validation and adopted or improved heuristics for the collection of defect data. We conducted an empirical study on 398 releases of 38 Apache projects. Results: We found that only half of the bug fixing commits determined by SZZ are actually bug fixing. If a six-month time frame is used in combination with SZZ to determine which bugs affect a release, one file is incorrectly labeled as defective for every file that is correctly labeled as defective. In addition, two defective files are missed. We also explored the impact of the relatively small set of features that are available in most defect prediction data sets, as there are multiple publications that indicate that, e.g., churn related features are important for defect prediction. We found that the difference of using more features is not significant. Conclusion: Problems with inaccurate defect labels are a severe threat to the validity of the state of the art of defect prediction. Small feature sets seem to be a less severe threat.<br />Comment: Accepted at Empirical Software Engineering, Springer. First three authors are equally contributing – Name: TypeDocument Label: Document Type Group: TypDoc Data: Working Paper – Name: DOI Label: DOI Group: ID Data: 10.1007/s10664-021-10092-4 – Name: URL Label: Access URL Group: URL Data: <link linkTarget="URL" linkTerm="http://arxiv.org/abs/1911.08938" linkWindow="_blank">http://arxiv.org/abs/1911.08938</link> – Name: AN Label: Accession Number Group: ID Data: edsarx.1911.08938 |
| PLink | https://login.libproxy.scu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&scope=site&db=edsarx&AN=edsarx.1911.08938 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10664-021-10092-4 Subjects: – SubjectFull: Computer Science - Software Engineering Type: general Titles: – TitleFull: Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Herbold, Steffen – PersonEntity: Name: NameFull: Trautsch, Alexander – PersonEntity: Name: NameFull: Trautsch, Fabian – PersonEntity: Name: NameFull: Ledel, Benjamin IsPartOfRelationships: – BibEntity: Dates: – D: 20 M: 11 Type: published Y: 2019 |
| ResultId | 1 |