A gen­eral idea of under­stand­ing foren­sic sci­ence mea­sure­ment uncertainty

On May 20 of each year is a great day cel­e­brated all over the world as World Metrol­ogy Day.  Metrol­ogy is so very impor­tant in every­thing, but in foren­sic sci­ence it is fun­da­men­tal.  In honor of that day and in advance of it to hope­fully bring some aware­ness, I will exam­ine Metrol­ogy in this post.

What most attor­neys do not appre­ci­ate or under­stand is pre­cisely how a mea­sure­ment is made, what its sig­nif­i­cance is and how it needs to be reported in a sci­en­tif­i­cally respon­si­ble way that acknowl­edges and quan­ti­fies the lim­i­ta­tions of the mea­sure­ment itself.

Any time one mea­sures some­thing it is a unique event in the uni­verse, not to be exactly repeated ever again.  It depends on a lot of factors.

In binary or inte­ger reported val­ues arrived at by ana­lyt­i­cal mea­sure­ments in foren­sic sci­ence (e.g., the weight of a con­trolled sub­stance, the feet or meters between objects in an crime scene, angles of tra­jec­tory in anthropometric-based recon­struc­tions, blood alco­hol con­tent), there is a need to report uncer­tainty to account for and rec­og­nize the lim­i­ta­tions of each ana­lyt­i­cal mea­sure­ment made.

“Huh?” you may be thinking.

Think of it this way.  Imag­ine your bath­room spring scale.  You stand on it in the morn­ing, you get a result.  You frown, you step off, then you IMMEDIATELY step back on again two sec­onds later.  The result the sec­ond time is that you are two pounds lighter!  You per­haps think to your­self that the sec­ond mea­sure was more accu­rate and move on with your day.  But if you were a metrol­o­gist or some­one who is inter­ested in ana­lyt­i­cal mea­sure­ment, you would have a dif­fer­ent world view.  We all know that in that dif­fer­ence of less than two sec­onds, there is no way that you lost two pounds.  This sim­ple exam­ple is only but one exam­ple of the vari­abil­ity of a mea­sure and how each mea­sure is a unique event unique to itself in the uni­verse not to be repeated again.

While it is also impor­tant to report uncer­tainty in sub­jec­tive non-binary val­ued foren­sic sci­ence dis­ci­plines such as pat­tern recog­ni­tion, in ana­lyt­i­cal mea­sure­ment it is a must.

Every­one should agree that only hon­est and sci­en­tif­i­cally respon­si­ble reported results belong in a Court­room.  No true sci­en­tist would dis­agree.  But is that prac­ticed?  NO.

What is your mea­sure­ment and what is it based upon? What is the standard?

To report the total uncer­tainty bud­get as an expres­sion of Bayesian-based metrol­ogy is to essen­tially iden­tify and quan­tify all pos­si­ble sources of error (both Type I and Type II) and account for it in an empir­i­cal manner.

The end result is some­thing such as the following:

The result of the analy­sis is: x +/-y with a xx% coverage/confidence interval.

So, it must be expressed sim­i­lar to this:   0.164g/100ml +/- .004 with a 95% cov­er­age (con­fi­dence) interval.

With­out the data being hon­estly and sci­en­tif­i­cally reported, it should not be accepted.

Here is why.

If I tell you that some­thing is 3 feet in length does that really accu­rately describe that mea­sure based solely upon that infor­ma­tion? No.

If I said that I used a yard stick to mea­sure it, then you may have a bet­ter idea.  But if I used a yard stick that was all beat up or had been man­u­fac­tured impre­cisely and not accu­rately, then you may begin to see the prob­lem in part.  What if I told you that when I made the call that it was exactly 3 feet that I was not look­ing directly on at the end of the yard stick, but rather at a sig­nif­i­cant angle so my per­spec­tive was off?  What if I told you that my yard stick only had demar­ca­tions on it for every foot instead of each inch?  What if I told you I closed one eye when I made the call?  And on it goes.

The greater part of being metro­log­i­cally respon­si­ble in the report­ing of uncer­tainty of any ana­lyt­i­cal mea­sure­ment and mov­ing closer towards com­pli­ance with ISO 17025 would be to acknowl­edge and attempt to quan­tify the sources of uncer­tainty.  These indi­vid­ual dif­fer­ences both infra­sub­ject and inter­sub­ject and among sam­ples them­selves as well as among ana­lyt­i­cal devices/measurands and even the same ana­lyt­i­cal device/measurand in the case of dupli­cate or even repli­cate test­ing are far more impor­tant and will pro­duce the greater range of vari­abil­ity than sim­ply the machine’s stated or actual rate of ana­lyt­i­cal error as acknowl­edged by the manufacturer.

One needs to keep in mind that just like with the yard stick exam­ple above, this is com­pound­ing or con­trib­u­tory error.  They are not islands unto themselves.

To present any ana­lyt­i­cal mea­sure­ment as an absolute is not only sci­en­tif­i­cally irre­spon­si­ble and ignores metrol­ogy, but most impor­tantly for our pur­poses in pre­sent­ing truth in the COurt­room, it is unduly prej­u­di­cial as in doing so the Gov­ern­ment ben­e­fits inap­pro­pri­ately from the power of “loaded lan­guage” (such as the words “iden­ti­cal” and “match” in pat­tern recog­ni­tion based foren­sic sci­ence).  In essence, in doing so (report­ing it as an absolute), it mis­guides the trier of fact.

The absence of report­ing uncer­tainty in a metro­log­i­cally respon­si­ble man­ner gives the trier of fact no use­ful infor­ma­tion from which to eval­u­ate the mea­sure itself.  As such, it is not pos­si­ble to mean­ing­ful judge the value or attach a weight to the mea­sure­ment.  It has no context.

It is no dif­fer­ent if I told you that I am 5’5″ tall but if my process and mea­sur­ing device is +/- two feet with a 95% con­fi­dence level, what do you think the chances are that I am really truly 5’5″ tall?

Judges should pre­clude the pre­sen­ta­tion of the reported mea­sure as an absolute.  Hav­ing come from the Amer­i­can Acad­emy of Foren­sic Sci­ence (AAFS) annual meet­ing this past Feb­ru­ary, I can tell you that there is not a sin­gle sci­en­tist who dis­agrees that the metro­log­i­cally respon­si­ble and ISO 17025 method of report­ing is sci­en­tif­i­cally correct.

To be more direct, I mean that if a mea­sure is being reported as an absolute, it should be (read as must be) pre­cluded entirely from evi­dence as doing so in pre­sent­ing it as an absolute with­out prop­erly reported uncer­tainty per ISO 17025 or the well-developed body of  sci­ence known as metrol­ogy, the trier of fact does not have the rel­e­vant infor­ma­tion to make an hon­est assess­ment of the evi­dence itself.  As such a report­ing (as an absolute), it  is inher­ently and demon­stra­bly unduly prejudicial.

Until they start report­ing it in a metro­log­i­cally respon­si­ble fash­ion it is not sci­ence, but the func­tional equiv­a­lent of guess­ing or really sci­ence fic­tion.  We do not invite tri­ers of fact to guess.  Proof is demanded.  Spec­u­la­tion is shunned.

A total uncer­tainty bud­get must be pre­sented with con­fi­dence inter­vals by the Gov­ern­ment as the Gov­ern­ment is the pro­po­nent of the evi­dence claim­ing it to be rel­e­vant in its reported form.

The utter lack of report­ing the uncer­tainty in a metro­log­i­cally respon­si­ble way screams unfair and NOT sci­ence and should not be allowed into Court.

Whereas it is true that sci­ence in the long term is self-correcting, we need to be on the fore­front of this push.  Why wait for ISO 17025?  Let’s force it now.

The old say­ing is true:  “In God we trust. All oth­ers must bring data”.

_______

Spe­cial acknowl­edg­ment:  A great lit­i­gat­ing attor­ney who focuses almost exclu­sively on this very topic by the name of Theodore “Ted” Vosk intro­duced me first to the con­cept of Uncer­tainty Mea­sure­ment and metrol­ogy in 2009.  I thank him for the intro­duc­tion as from that moment I have spent a tremen­dous amount of time learn­ing about this sub­ject as well as Inter­na­tional Stan­dards in gen­eral includ­ing attend­ing other con­fer­ences and read­ing.  Attor­ney Vosk is a legal role model as he has quite suc­cess­fully lit­i­gated these mat­ters in the field of breath alco­hol test­ing in the state of Wash­ing­ton.  He is avail­able for hire to act as coun­sel on these matters.