In a series of posts, I am going to intro­duce the reader to the exis­tence of ISO 17025 and its impor­tance.  I am going to intro­duce it in bite-sized bits for easy diges­tion.  Just like all mat­ters of learn­ing, knowl­edge is incre­men­tal over time and builds upon pre­vi­ous exposure.

So far we have answered the fol­low­ing questions:

• What is ISO 17025?
• Why do we need stan­dards? Why ISO 17025 and pol­icy, pro­ce­dures and instruc­tions matter.
• Why is ISO 17025 so impor­tant to us in foren­sic science?
• Why should the crim­i­nal defense com­mu­nity care about ISO 17025?
• How can ISO 17025 pro­vide a sim­ple method to develop themes to cross-examine experts?
• How can ISO 17025 be used by the crim­i­nal law prac­ti­tioner to help get dis­cov­ery?
• Can ISO 17025 can answer the ques­tion as to who the actual ana­lyst is?
• How can ISO 17025 help min­i­mize the par­tic­u­lar prob­lem of fraud­u­lent cre­den­tials of foren­sic scientists?

In this post we exam­ine how ASCLD/LAB Inter­na­tional con­flicts with ISO 17025 and hon­est sci­en­tific report­ing of Uncer­tainty Mea­sure­ment (UM) in foren­sic science.

Sec­tion 5.3 (Accom­mo­da­tions and envi­ron­men­tal con­di­tions) requires a full doc­u­men­ta­tion into the envi­ron­men­tal and other test­ing accom­mo­da­tions of the lab­o­ra­tory to iden­tify poten­tial sources of error and vari­ance to val­i­date that the ana­lyt­i­cal devices and the per­son­nel involved in the area where the ana­lyt­i­cal devices are used are suf­fi­ciently free from envi­ron­men­tally caused error and yields valid test­ing results suit­able for its intended pur­pose. Although it is up to the lab­o­ra­tory to estab­lish a pol­icy, pro­ce­dure, and instruc­tions to meet Sec­tion 5.3, these require­ments include a full account­ing into such hereto­fore pos­si­bly ignored sources of poten­tial error that includes, but is not lim­ited to, “bio­log­i­cal steril­ity, dust, elec­tro­mag­netic dis­tur­bances, radi­a­tion, humid­ity, elec­tri­cal sup­ply, tem­per­a­ture, and sound and vibra­tion lev­els.” It also requires a lab­o­ra­tory to mon­i­tor, con­trol and record these envi­ron­men­tal con­di­tions that may change within the lab­o­ra­tory and may influ­ence the qual­ity of the results.

Per­haps the most fun­da­men­tal change in the way ISO 17025 lab­o­ra­to­ries will be con­duct­ing their test­ing and cal­i­bra­tion ser­vices if they seek and obtain ISO 17025 accred­i­ta­tion comes with the imple­men­ta­tion of Sec­tion 5.4 (Test and cal­i­bra­tion meth­ods and method val­i­da­tion)[i].

It is beyond the scope of this blog post to include the con­tri­bu­tions of Theodore “Ted” Vosk in this work con­cern­ing ISO 17025[ii]. I have also blogged on basic metrol­ogy and Uncer­tainty Mea­sure­ment (UM). They include:

• Metrol­ogy based posts at the Truth About Foren­sic Sci­ence Blog; and
• Metrol­ogy based posts at the Penn­syl­va­nia DUI Blog.

How­ever, it is within this sub­sec­tion, Sec­tion 5.4.6, that we find the encap­su­la­tion of the need to report, under cer­tain cir­cum­stances, Uncer­tainty Mea­sure­ments (UM)[iii]. Specif­i­cally 5.10.3.1© reads that UM shall be included in a Test Report to the cus­tomer “when it is rel­e­vant to the valid­ity or appli­ca­tion of the test results, when a customer’s instruc­tion so requires, or when the uncer­tainty affects com­pli­ance to a spec­i­fi­ca­tion limit”.

While a tremen­dous amount of empha­sis could be placed and should be placed on UM and whether or not the laboratory’s ver­sion of UM is fre­quen­tist or Bayesian in nature, Sec­tion 5.4 pro­vides addi­tional use­ful infor­ma­tion in the require­ment that the method to be employed must indeed be validated.

Although there is a com­mon mis­per­cep­tion among lawyers and even among lab­o­ra­tory man­agers that ISO 17025 pro­vides a method of val­i­da­tion spe­cific to the foren­sic sci­ence dis­ci­plines, this is not the case. Instead the require­ment of 5.4.5 is for the lab­o­ra­tory to have doc­u­men­ta­tion that includes specif­i­cally how it is deter­mined that a given method is to be applied and that its instruc­tions as well as its pro­ce­dures are in fact val­i­dated as pro­mul­gated and used.  This is to include lim­i­ta­tions on the assay to be performed.

This could be very use­ful for the crim­i­nal law prac­ti­tioner, for exam­ple, in the case of solid drug dose exam­i­na­tion and deter­mi­na­tion. It is pos­si­ble that within the laboratory’s own doc­u­ments there could be a dam­ag­ing admis­sion of its inabil­ity to deter­mine and dis­crim­i­nate between posi­tional iso­mers and chi­ral com­pounds[iv]. Dur­ing the val­i­da­tion process, although not specif­i­cally out­lined in ISO 17025, at a min­i­mum, the fol­low­ing should be addressed by the laboratory:

• matrix effects,
• sam­ple homogeneity,
• speci­ficity,
• demon­strated range of linearity,
• pre­ci­sion,
• inter­fer­ing substances,
• sta­bil­ity of targets,
• pop­u­la­tion dis­tri­b­u­tion, and
• mea­sure­ment uncertainty.

It is accept­able per ISO 17025 to use reli­able, pub­lished, and com­mer­cially avail­able infor­ma­tion to estab­lish each para­me­ter so long as after the imple­men­ta­tion of the val­i­dated process, it is effec­tively mon­i­tored while it remains in place. If there is devi­a­tion from the reli­able, pub­lished, and com­mer­cially avail­able infor­ma­tion upon which the method relies, then it is required that the lab­o­ra­tory rec­og­nize that the pre­vi­ous method was pro­duc­ing inap­pro­pri­ate results and there­fore embark upon a new process of val­i­da­tion that will insure that the process employed and the method­ol­ogy is one that is indeed suit­able for its intended purpose.

Of addi­tional prac­ti­cal use to us is Sec­tion 5.6 and specif­i­cally Sec­tion 5.6.2.1.1. It holds that the lab­o­ra­to­ries, when they con­struct a cal­i­bra­tion curve or do other types of cal­i­bra­tion of the instru­ment that can con­tribute to the uncer­tainty, must prop­erly doc­u­ment the mea­sure­ment trace­abil­ity of those ref­er­ence stan­dards to the clas­sic mea­sure­ment item (i.e., the Inter­na­tional Sys­tem of Units (SI)) or in the case of items that can­not be strictly made in SI units such as in the case of drugs and DNA pro­files, these ref­er­ence mate­ri­als are to be trace­able to an appro­pri­ate mea­sure­ment standard.

There is a dis­tinc­tion between ref­er­ence stan­dards and ref­er­ence mate­ri­als as out­lined in Sec­tion 5.6.2. While one can­not cer­tify street cocaine (and hence would be a ref­er­ence mate­r­ial), one can cer­tify Cer­ti­fied Ref­er­ence Mate­ri­als (CRM’s or SRMs) (ref­er­ence stan­dards). ASCLD/LAB in its inter­pre­ta­tion and grant­ing of ISO 17025 accred­i­ta­tion takes Sec­tion 5.6 to an addi­tional safe­guard step in that it requires that what­ever cal­i­bra­tion ser­vice provider is used by a lab­o­ra­tory must be ISO 17025 accredited.

Another poten­tial source of uncer­tainty that is addressed by ISO 17025 is the dis­tinc­tion that is made regard­ing the equip­ment itself. Sec­tion 5.5 requires the lab­o­ra­tory to have a method to iden­tify and clas­sify its instru­ments that are used through­out the process. There is a dis­tinc­tion between class 1, class 2, and class 3 instru­ments that is impor­tant for the prac­ti­tioner to be aware of so as to be able to deter­mine whether or not the very best sci­en­tific process was employed and whether or not the best cal­i­bra­tion of the equip­ment was under­taken. Per ASCLD/LAB’s inter­pre­ta­tion of ISO 17025, Class 3 instru­ments are the only type of instru­ments whose cal­i­bra­tion ser­vice providers do not need to be ISO 17025 com­pli­ant per ASCLD/LAB. It is required of the lab­o­ra­to­ries to not only state that the cal­i­bra­tion ser­vice providers are ISO 17025 accred­ited but that they be able to prove through doc­u­men­ta­tion the com­pe­tence, the trace­abil­ity, and the mea­sur­ing abil­ity of the ser­vice provider espe­cially if it is not ISO 17025 accredited.

Per­haps the sin­gle biggest area of poten­tial uncer­tainty and one of the most use­ful to expose remains an unde­clared poten­tial source for erro­neous results. It is encap­su­lated and addressed in Sec­tion 5.7 (Sam­pling). In all ana­lyt­i­cal mea­sure­ments the ana­lyt­i­cal device very sel­dom weighs and/or mea­sures the entire sam­ple as it organ­i­cally exists[v]. There­fore, only a very small part of the whole, called an aliquot or ali­quant, is actu­ally tested by any ana­lyt­i­cal device.  As a result, it becomes cru­cial and vitally nec­es­sary for the lab­o­ra­tory to ensure homo­gene­ity in the aliquot or ali­quant tested. Right now, shock­ingly, most lab­o­ra­to­ries do not have a writ­ten pol­icy or pro­ce­dure or instruc­tions that addresses this.

It is this cru­cial dif­fer­ence between sam­pling ver­sus sam­ple selec­tion that needs to be exposed by all of us. In essence, what hap­pens in the lab­o­ra­tory when an aliquot is pre­pared is to exer­cise a mas­sive amount of truly sub­jec­tive dis­cre­tion by select­ing a “pinch of this” or a “sec­tion of that” from the whole unknown sam­ple sub­mit­ted for exam­i­na­tion. It is clear that by doing such, even with a pol­icy, pro­ce­dure and instruc­tion in place, mas­sive rep­re­sen­ta­tion errors with respect to non-colloidal mix­tures can occur. Sam­ple selec­tion in the case of trace evi­dence, for exam­ple, per ISO 17025 Sec­tion 5.7 would require a writ­ten pol­icy, pro­ce­dure, and instruc­tion that is uni­ver­sally enforced, imple­mented and mon­i­tored by the lab­o­ra­tory down to the tech­ni­cians at the bench as to which hairs or fibers out of many or what part of a stain to swath and exam­ine, for exam­ple. This is an exam­ple of sam­ple selec­tion. This is to be dis­tin­guished from sam­pling itself wherein there must be a writ­ten pol­icy, pro­ce­dure, and instruc­tions to make sure that homo­gene­ity of a sam­ple, in fact, exists. A fine exam­ple of this would be blood and blood alco­hol sam­pling. With­out assur­ance of homo­gene­ity in such a sam­ple ran­dom sam­pling error is intro­duced and inac­cu­rate results may be reported. Per ISO 17025 and ASCLD/LAB, there must be rig­or­ous train­ing as well as a plan and pro­ce­dure in place for sam­ple selec­tion as well as sam­pling. If one were to obtain the pol­icy, for exam­ple, of either sam­pling or sam­ple selec­tion, then there could and should be some very use­ful lan­guage con­tained that admits to this very fun­da­men­tal source of sub­jec­tive­ness and iden­ti­fies sam­pling and sam­ple selec­tion as a large poten­tial source of error.

From strictly a sci­en­tific aspect, per­haps one of the dis­ap­point­ments in the pro­mul­ga­tion of ISO 17025 is in Sec­tion 5.8. Sec­tion 5.8 con­cerns the proper han­dling of spec­i­mens, which in our appli­ca­tion is the seized evi­dence. In only a few words, it states gen­er­al­i­ties of how the evi­dence is to be han­dled and how the items are tested. There is very lit­tle guid­ance and require­ments as to this in the ISO 17025 doc­u­ment. There only needs to be a pro­ce­dure in place per ISO 17025. ASCLD/LAB has right­fully taken the posi­tion that this is a cru­cial part of the crime lab­o­ra­tory analy­sis and there­fore ded­i­cates an addi­tional two and a half pages of require­ments in its “Inter­na­tional” program.

In Sec­tion 5.9 we find that there must be a pro­ce­dure as to the assur­ing of the qual­ity of the reported results. While not directly offer­ing or even sug­gest­ing such a method, one pos­si­ble process and method­ol­ogy that could be used is prop­erly called “con­trol chart­ing.” Con­trol chart­ing is a graph­i­cal and empir­i­cal sta­tis­ti­cal tool used to detect exces­sive process vari­abil­ity to try to iden­tify spe­cific assign­a­ble causes that can be cor­rected. It serves to deter­mine whether a process is in a state of sta­tis­ti­cal con­trol; that is, the extent of vari­a­tion of the out­put of the process does not exceed that which is expected based on the nat­ural sta­tis­ti­cal vari­abil­ity of the process[vi].  Con­trol chart­ing is a great way to iden­tify the source of sta­tis­ti­cal out­liers where a machine can get pulled, an envi­ron­ment checked or an inap­pro­pri­ate oper­a­tor stopped or re-trained[vii].

In Sec­tion 5.10.1 we can find lan­guage that has a great pos­si­bil­ity of abuse.  Per Sec­tion 5.10.1:

The results of each test, cal­i­bra­tion, or series of tests or cal­i­bra­tions car­ried out by the lab­o­ra­tory shall be reported accu­rately, clearly, unam­bigu­ously and objec­tively, and in accor­dance with any spe­cific instruc­tions in the test or cal­i­bra­tion methods.

The results shall be reported, usu­ally in a test report or cal­i­bra­tion cer­tifi­cate, and shall include all the infor­ma­tion requested by the cus­tomer and nec­es­sary for the inter­pre­ta­tion of the test or cal­i­bra­tion results and all infor­ma­tion required by the method used.

The sim­ple fact that the even­tual Test Report that is issued to the “cus­tomer” must be one that is unam­bigu­ous con­notes to the author of this blog that 5.4.6 and 5.10.3.1© requires the report­ing of UM. ASLCD/LAB does not think so unless cer­tain cri­te­ria are met. To present any mea­sure as an absolute with­out acknowl­edg­ing UM, does pre­cisely that—makes the report­ing ambigu­ous. The coun­ter­ar­gu­ment that is offered by ASCLD/LAB and other lab­o­ra­tory man­agers is the oppo­site in that, if UM report­ing were placed into every report when­ever a mea­sure is made, it would con­fuse the trier of fact.  As such, they argue that a restric­tion on the raw data is war­ranted unless it is around a crit­i­cal mea­sure and that the unnec­es­sary report­ing of UM would invite ambi­gu­ity in the over­all result. In essence, ASCLD/LAB instead wants to be the judge of the facts, which is, of course imper­mis­si­ble as that role is specif­i­cally reserved to the trier of fact.

Regard­less of ASCLD/LAB’s inter­pre­ta­tion of this sec­tion, there is hope in that in Sec­tion 5.10.1 and in 5.10.4, we find lan­guage that states clearly that it is a require­ment that any infor­ma­tion not listed on the Test Report “shall be read­ily avail­able in the lab­o­ra­tory” and there­fore should be acces­si­ble to all.

---

[i] Typ­i­cal foren­sic appli­ca­tions and dis­ci­plines that involve mea­sure­ment sci­ence include: tox­i­col­ogy includ­ing a Blood Alco­hol Con­tent (BAC) test­ing; mass deter­mi­na­tion; drug purity; and dis­tance to muz­zle just to name a few. There are also other mea­sur­ing dis­ci­plines that report mea­sure­ment and there­fore fall under the new require­ments of ISO 17025 and include: trig­ger pull; bar­rel lengths; atomic absorp­tion spec­tropho­tom­e­try (AAS) and induc­tively cou­pled plasma-atomic emis­sion spec­trom­e­try (ICP-AES) analy­sis of gun­shot residue; refrac­tive index­ing; micro­scopic dimen­sional analy­sis; and DNA.

[ii] The lead­ing crim­i­nal defense attor­ney who has been suc­cess­fully lit­i­gat­ing the lack of UM report­ing in ana­lyt­i­cal mea­sure­ments when pre­sented in the court­room is Attor­ney Theodore “Ted” Vosk of Wash­ing­ton State.  Attor­ney Vosk has pub­lished on the con­cepts of UM.   Foren­sic Metrol­ogy: A Primer on Sci­en­tific Mea­sure­ment for Lawyers, Judges, and Foren­sic Sci­en­tists; Edited by Ted Vosk, Ash­ley F. Emery; http://www.taylorandfrancis.com/books/details/9781439826195/

[iii] Crit­i­cal quan­ti­ta­tive UM con­cerns will address the fol­low­ing concepts:

• The iden­ti­fi­ca­tion and eval­u­a­tion of all sources of poten­tial error,
• The iden­ti­fi­ca­tion of sig­nif­i­cance of iden­ti­cal uncer­tain­ties must be eval­u­ated in the uncer­tainty bud­get, and
• The estab­lish­ing and close mon­i­tor­ing of results near crit­i­cal values

[iv] Iso­mers are com­pounds that pos­sess the same empir­i­cal for­mula, but are dif­fer­ent in struc­ture.  Cathine, which is not a con­trolled sub­stance, will likely be mis-identified as its diastere­omer, Phenyl­propanolamine, which is a con­trolled sub­stance, when solely a Gas Chro­mato­graph with Mass Spec­trom­e­ter (GC-MS) is uti­lized. While metham­phet­a­mine is a sched­ule II con­trolled sub­stance, the l-enantiomer of metham­phet­a­mine is found in the Vick’s Inhaler, which is a prod­uct exempted from con­trol.  Sim­i­larly, γ-Hydrox­y­bu­tyrate (GHB) and γ-Buty­ro­lac­tone (GBL) can be indis­tin­guish­able on some GC-MS. γ-Hydrox­y­bu­tyrate is a Sched­ule I con­trolled sub­stance that can­not be pos­sessed legally. γ-Buty­ro­lac­tone, on the other hand, may be pos­sessed and only becomes ille­gal if “intended for human consumption”.

[v] For exam­ple, in solid drug dose test­ing using Gas Chro­matog­ra­phy with Mass Spec­trom­e­try as the detec­tor, this issue becomes very patent.  In the lab, the tech­ni­cian starts with the whole sam­ple, then a small por­tion of the whole is removed that one hopes in rep­re­sen­ta­tive of the whole.  Next, a sol­vent is typ­i­cally applied such as methanol, ethanol or dichloromethane to deriv­a­tive the sam­ple that results in dilu­tion of the orig­i­nal item. An autosam­pler is employed that takes one micro­liter which is one mil­lionth of a liter of this deriva­tized or chem­i­cally altered and diluted sam­ple to inject it into the injec­tor port.  Typ­i­cally, a split injec­tor con­fig­u­ra­tion is used that results in a very, very small part of the micro­liter mak­ing it to the col­umn with the remain­der being vented out, not to be ana­lyzed. Of this very, very small amount that makes it onto the col­umn to be sep­a­rated into hope­fully unique ana­lytes, only 1% or 2% of this sep­a­rated mate­r­ial is ion­ized in the Mass Spec­trom­e­ter to be fur­ther frag­mented.  With the typ­i­cal scan level of between 40–400 times per sec­ond results that in a con­di­tion that in order for the ana­lyte to be detected at all a time frame of only 1/360^th occurs where all of these con­di­tions can be met is to record a result at all.

[vi] Every process has some inher­ent vari­abil­ity due to ran­dom fac­tors over which there is no con­trol and which can­not be elim­i­nated economically.

[vii] As can be the case, a lab’s stated but unproven error of +/- x stated to xx%  of con­fi­dence (really a pre­dic­tive inter­val) can grossly under­state real­ity as later empir­i­cally estab­lished once con­trol chart­ing and other mea­sures per ISO 17025 are imple­mented. Once the con­trols and processes per ISO 17025 are in place, labs may be able to iden­tify sources of pro­found error.  Then after the imme­di­ate triage is com­pleted to iden­tify and end the source of the error, then train­ing and per­son­nel could be selec­tively fired or up-trained to reduce error and there­fore tight­en­ing the cov­er­age fac­tor thereby caus­ing it to be 8, 9, 10 or more sigma to get to the crit­i­cal measure.