Report of the Digital Evidence Task Group Quality Study: 2020

Editor: Ava Gozo. 

Executive Summary

In 2020, the Organization of Scientific Area Committees (OSAC) for Forensic Science’s Digital Evidence Subcommittee formed a task group to identify the quality practices and management systems that are most effective within digital forensics laboratories. The group identified contemporary quality management practices used in forensic labs and interviewed laboratory management, senior staff, and digital evidence customers (e.g., law enforcement (LE), attorneys, judges). The report was based on the interviewees’ experiences with quality practices and management systems.

The study found significant problems with how quality management systems (QMS) are implemented within digital forensics laboratories generally. The study also found problems with timeliness and the ability of consumers of laboratory products to understand and make use of reports generated by digital forensic laboratories.

The study found that accreditation under ISO 17025:2017 and ISO 17020:2012 and formal QMSs are not having the desired outcome of enhancing the accuracy, timeliness, and clarity of reports. Current efforts to improve the processes are not resulting in significant gains and are, in many cases, reducing quality.

Some quality elements took significant resources with no benefit and others actually reduced timeliness and clarity. However, this study documented many examples where the implementation of a QMS had significant value. In most of these cases, the QMS had been optimized for digital forensics.

Based on these findings, the study recommended the development of a QMS optimized specifically for digital evidence processing. This new system should be tailored to maximize the improvement of quality that is most important to digital evidence. A digital forensics QMS should focus on quality elements relevant to digital evidence processing rather than shoehorning digital processing into existing frameworks.

Study Overview

The task group conducted interviews with the objective of identifying effective quality practices and management systems within digital forensics laboratories.

Key Findings

1. Quality Management Systems (QMS)

• Significant issues were found in the implementation of QMS in digital forensics laboratories.
• The current emphasis on accreditation under ISO 17025 and ISO 17020, as well as formal QMSs, is not achieving the desired outcome of enhancing the accuracy, timeliness, and clarity of reports.

2. Documentation

• Laboratories varied in their documentation practices and its role in quality management.

3. Tool Validation and Testing

• There was inconsistency in how laboratories conducted tool validation and testing.

4. Management Reviews and Audits

• Different interpretations and implementations of management reviews and audits were observed.

5. Peer Review

• The value and implementation of peer/technical reviews for forensic reports varied.

6. Quality Manager Role

• The role and impact of quality managers on laboratory work quality were assessed.

Sample Size

The study included interviews with 31 participants recruited through announcements at digital forensics conferences and forums, as well as personal outreach by the task group. The sample consisted of individuals from local, state, and federal law enforcement, private organizations, and digital forensic science practitioners, with most participants based in the United States.

Findings and Recommendations

Based on the responses, the task group recommended that the digital forensics community initiate and continue efforts to improve quality management practices. The current emphasis on accreditation under ISO 17025, ISO 17020, and formal QMSs is not having the desired outcome. One of the problems with the current implementations of QMSs is a lack of understanding of the tasks, methods, and sources of errors in digital forensic processes, particularly among Laboratory Management and Quality Managers.

In digital forensics laboratories, a major focus of quality systems has not been on improving quality; it has, instead, focused only on meeting 17025 requirements with little regard for enhancing the quality of the forensic services in the digital laboratory. The spirit of a quality system is lost in the dogmatic adherence to the quality system or the ISO standard rather than examining practices for their enhancement of quality management goals.

To be successful, a QMS must have accuracy, understandability, and timeliness as goals rather than a perception that the laboratory has quality. Successful quality systems focused on making things better rather than trying to create paperwork. The task group saw this in very few laboratories.

Another problem with current QMS implementations is that there has been a punitive culture rather than a quality culture. No amount of process will improve quality without strong management support.

Significance

This study underscored the need for tailored quality management approaches in digital forensics laboratories. It highlights areas for improvement and provides a foundation for developing more effective quality management systems specific to digital evidence handling and analysis.

By addressing these findings and implementing the recommended practices, digital forensics laboratories can improve their overall quality and reliability, ultimately benefiting the broader forensic science community and its stakeholders.

References

Guttman, B., Eber, W., Epstein, B., Fève, S., Ford, D., Hollway, J., Kelly, C., Lawrence, T., Lyle, J., Sauerwein, K., & Vodde, S. (2022). Report of the Digital Evidence Task Group Quality Study. Organization of Scientific Area Committees for Forensic Science. https://doi.org/10.29325/OSAC.DE.0001