I was recently asked a question which is basically this: “Justin, why don’t you stipulate to any forensic science result ever?”
My answer is simple. I haven’t found a single analytical test or a single forensic science result that does not have some area of legitimate inquiry whether it is in the data (the test itself), the QC, the traceability of the standards, the QA performed, or the technicians, the expert or the very foundational validation of the technique employed. I know that is a bold statement, but it is true. Having said that, the question becomes does the issue matter to a jury? Now that is the issue. These cases are tried by purposeful design to folks (meaning the jury) who have absolutely no idea of the subject matter. If the science was going to be evaluated by other scientists, then it would a lot easier and a lot fewer miscarriages would occur, in my humble opinion. However, they are not. This is where someone like me comes in who can use transferable concepts to bridge the gap between the world of science and the world of the jury.
A paper audit can only show so much. Even the most detailed instructions that I see still allow for some individual discretion by the user. There may very well be a disconnect between the existence of a truly validated method with robust instructions and actual execution that is not traceable by the paperwork. Knowledge of protocol and protocol adherence are two entirely different concepts. There is also the “x-factor” which is can you, as a witness who is likely very nervous in front of a jury, explain what you do, how you do it and how it results in a specific (or really a near specific) qualitative measurement with a quantitative measurement that is very free of calibration and bias error SO THAT THE JURY CAN UNDERSTAND IT.
Perhaps there will someday where crime laboratories that have truly validated methods for what they do such as what exists in the GLP or EPA regulated world using perhaps the USP guidelines of well-designed validation experiments that investigate and prove at a minimum accuracy (bias), precision (calibration), specificity, limit of detection, limit of quantitation, linearity and range, ruggedness, robustness and uncertainty in their qualitative and quantitative measurement that are all the while verified on the particular instruments used by the particular operators using them. Perhaps someday they will publish and make available these efforts in a true validation report that features:
- Objective and scope of the method (applicability, type).
- Summary of methodology.
- Type of compounds and matrix.
- All chemicals, reagents, reference standards, QC samples with purity, grade, their source or detailed instructions on their preparation.
- Procedures for quality checks of standards and chemicals used.
- Safety precautions.
- A plan and procedure for method implementation from the method development lab to routine analysis.
- Method parameters.
- Critical parameters taken from robustness testing.
- Listing of equipment and its functional and performance requirements, e.g., integration perimeters, baseline noise and column temperature range.
- Detailed conditions on how the experiments were conducted, including sample preparation. The report must be detailed enough to ensure that it can be reproduced by a competent technician with comparable equipment.
- Statistical procedures and representative calculations.
- Procedures for QC in routine analyses, e.g., system suitability tests.
- Representative plots, e.g., chromatograms, spectra and calibration curves.
- Method acceptance limit performance data.
- The expected uncertainty of measurement results.
- Criteria for revalidation.
- The person(s) who developed and validated the method.
- Summary and conclusions.
- Approval with names, titles, date and signature of those responsible for the review and approval of the analytical test procedure.
Maybe someday the process (sample preservation, sample prep, instrument running, data collection, and data interpretation) will be done by folks who are fully trained and fluent in the underlying theory of the technique they are employing as opposed to just simply the process employed.
Maybe someday all of the instruments will be control charted.
Maybe someday there will be truly blinded proficiency tests on each analyst on a relevant sample that contains a target analyte that is something that can be frequently misinterpreted that is intermixed with routine samples run with very tight acceptance criteria with the quantitative result not dependent on the results of other laboratories but rather on the targeted and designed value.
Maybe someday the Quality Assurance officer will be a person who is more qualified in the theory, have demonstrated greater proficiency it the assay, and can actually employ the technique better than the person who runs the assay that they are called upon to evaluate. A lot of what I see is the QA officer is not technically trained in the theory or the process of the assay they are called upon to double-check. Such a regime makes QA little more than a rubber stamp.
Maybe someday there will be verifiable information that is kept by the laboratory that they will readily make available that justifies their data and their opinions. I find that the difficulty in getting the data is due to the typical crime laboratory being ultimately administratively lead by a sworn police officer who has no technical training in science whatsoever. A promoted traffic cop, if you will. This person (who is not steeped in the scientific culture of openness or transparency or has not even been meaningfully exposed to science likely has no idea of the process involved and what constitutes relevant information) has very much a “us-versus-them” mentality. Having said that, I personally don’t have any problem getting whatever data I want because I can explain exactly what I am looking for, why it could matter in a given case, and how it can make a difference in the validity of the measure or opinion offered. I frequently tell my colleagues that if you don’t know what it is and why it matters, then don’t ask for it. I usually do it with a smile first by sending my subpoena to the laboratory directly followed up with a very nice mannerly introductory phone call to their subpoena compliance officer and the person who is in charge of the science of the laboratory. After chatting with them and getting them to understand that while I may have a JD after my name and although I freely admit that I am not a PhD that I am fairly fluent in the science of it all, they generally recognize that I am not asking them to do all of this work for no purpose or as an act of meaningless busywork all wrapped in a game of “got you” but instead in a wholly legitimate effort to meaningfully examine the validity of the opinion expressed. If they choose not to be open, then that is their choice. It is America after all. It just will be a costly and tough process for them as I will force them to be wholly open by going into Court.
Maybe, just maybe, I would consider adjusting my view.