We looked at the National Acad­emy of Sci­ences body of work wherein it iden­ti­fied weak­nesses in the way that foren­sic sci­ence in the United States is prac­ticed.  One of the major areas that it focused on and ulti­mately took issue with con­cerned the body of foren­sic sci­ences loosely called pat­tern recognition.

The basic pseudo-scientific prin­ci­ples of pat­tern recog­ni­tion based sci­ences is not unlike Sesame street in the video below.  I am being serious.

Like cookie mon­ster we ask “Did you guess which thing is not like the other things?  Did you guess which thing with all of your might?”

In pat­tern recog­ni­tion based foren­sic sci­ence dis­ci­plines, we take a known and com­pare it against an unknown that is typ­i­cally at the scene to try to guess with all of our might.

The prob­lem becomes fairly obvi­ous fairly quickly.

• First, we need to prove that these knowns that we are com­par­ing the unknown against are obtained as near as pos­si­ble to the real world con­di­tions as pos­si­ble that gen­er­ated the unknown so that the com­par­i­son can even begin to be valid.  For exam­ple, shoot­ing a pro­jec­tile from a gun into a body will mas­sively deform the bul­let along its path if it hits bone or dense mass, but typ­i­cally the known that is gen­er­ated for com­par­i­son pur­poses against this unknown is in fact cre­ated by fir­ing its bul­let into a water tank which cre­ates a near pris­tine object, not like the deformed one from the scene;
• Sec­ond, we need a suf­fi­cient pop­u­la­tion den­sity study to prove that the knowns and unknowns are uniquely con­tributed and trace­able uniquely to the orig­i­nat­ing source to the exclu­sion of all oth­ers.  For exam­ple, we need to prove through mean­ing­ful sci­en­tific study that only this type of seed with these types of observ­able and quan­tifi­able char­ac­ter­is­tics is in fact unique only to mar­i­juana to the exclu­sion of all other botan­i­cal substances;
• Third, can we use points of comparison/agreement at all?  If so, what defin­able, mea­sur­able and empir­i­cal points con­sti­tute these points of comparison/agreement?  Can we dis­tin­guish between class, sub­class and indi­vid­ual points of comparison/agreement based upon mean­ing­ful pop­u­la­tion den­sity studies?
• Fourth, have the meth­ods of analy­sis and the pro­to­cols been tested and val­i­dated to be declared suit­able for its intended use?
• Finally, can we iden­tify the sources of uncer­tainty in our analy­sis and can we quan­tify them so as to be able to report the uncer­tainty of our results in a metro­log­i­cally respon­si­ble way?

A lot of foren­sic sci­ence is caught up in pat­tern recog­ni­tion.  It includes the fol­low­ing gen­eral dis­ci­plines (note the list is not comprehensive):

1. Foren­sic Firearm Analysis
2. Foren­sic Tool­mark Analysis
3. Foren­sic Oden­tol­ogy Analysis
4. Impres­sion evi­dence such as Foren­sic Fin­ger­print Analy­sis, Foren­sic foot­print analy­sis and Foren­sic Tire Mark Analysis
5. Foren­sic Hair Analysis
6. Foren­sic Fiber Analysis
7. Some forms of trace evi­dence analysis
8. Foren­sic Ques­tioned Doc­u­ment analysis

The National Acad­emy of Sci­ences report states that these areas of sci­ence have poten­tial to be great tools of sci­ence and truth.  Yet, right now, the way they are stud­ied, prac­ticed and pre­sented in court is based upon unproven and fun­da­men­tal sys­temic flaws in that it is in fact based upon,  There are too many fun­da­men­tal assump­tions, much like the ones listed above, that have not been sci­en­tif­i­cally proven to be true or valid.

In other words, the jury is still out.