Read other letters about this article

• Tuesday, May 15, 2007 05:19 PM

  Practicability of data mining

  Michael Chertoff recently met with the EU parliament and said "We don't always capture with precision when the border inspector has relied upon [whether or not the database was used in a prosecution]. I don't know whether it would be possible to construct a system in practical terms that would do that."

  http://www.theregister.co.uk/2007/05/15/surveillance_scutiny/

  We are expected to believe that while the DHS can't manage to track the usage of their own databases (which must be connected to) to cases (which are all logged in police databases), they can still manage to catch the most statistically insignificant events (terrorist attacks) by subtle correlations of social networks in our email and web patterns.

  Widespread data mining is an application only useful for demographic information, there is no "needle in the haystack" algorithm that can produce the kind of information that is being claimed.

  It is a fundamental fact of information theory that a predictive algorithm (such as one used to catch terrorists) will degrade in performance the wider a variety of data that is given to it. (This is the same as saying that a predictive algorithm fails on high entropy data, for you nerds.)

  There is no reason to believe these iniatives are primarily aimed at catching terrorists, other than the administration's say-so. They have blocked every inquiry into the actual functionality of these systems.

  If the system is being used to monitor the American population (which is what those algorithms are good for, demographic analysis), then that would explain Ashcroft's adamant refusal to sign off on the program, as this would clearly be a felony.

  —prunes Read prunes's other letters
  Permalink
  Flag
  Click "Submit Flag" if you feel this post is inappropriate. Explain why below if you wish.

  Cancel