After spending $42 million since 2003 developing a data-mining software tool known as the Analysis, Dissemination, Visualization, Insight and Semantic Enhancement (ADVISE) program, the Department of Homeland Security (DHS) has reportedly scrapped it after investigators found that it was tested with information about real people without required privacy safeguards.

ADVISE was developed at the Lawrence Livermore and Pacific Northwest national laboratories to be used by components of DHS such as Immigration, Customs, Border protection, Biological Defense and its Intelligence office.

In March the U.S. Government Accountability Office (GAO) warned that “the ADVISE tool could misidentify or erroneously associate an individual with undesirable activity such as fraud, crime or terrorism.” The GAO also said the public should be notified by the DHS about how an individual’s personal information would be verified, used and protected before implementing ADVISE on live data.

Since then, the DHS Inspector General and the DHS Privacy Office discovered that tests conducted with ADVISE were using live data about real people instead of made-up data for one to two years without meeting privacy requirements.

The DHS Inspector General also noted that ADVISE was poorly planned, time-consuming for analysts to use and it lacked adequate justifications. ADVISE was one of the broadest of 12 data-mining projects in the DHS.

In 2004 a DHS research official said ADVISE would be able to ingest 1 billion pieces of structured information per hour such as databases of cargo shippers, and 1 million pieces of unstructured text per hour such as government intelligence reports.

In July and August the DHS Inspector General (PDF) and the DHS Privacy Office (PDF) released reports concluding that between 2004 and 2007 three pilot tests of ADVISE “created unnecessary privacy risks” by using personally identifiable information without issuing required privacy impact assessments first. Errors in the pilot programs involved weapons of mass effect, immigration enforcement and the DHS intelligence analysis office.

History Repeats Itself

The Secure Flight program to screen domestic air travelers was halted by Congress after it also acquired live personal data for testing. Since issuing a privacy impact assessment, dropping the use of commercial data such as personal credit histories, testing will begin this fall.

It’s hard to understand why the same mistakes with using personal data keep happening over and over. “All too often the Bush Administration has treated safeguards for databases as a disposable afterthought if at all” said Senate Judiciary Committee chairman Patrick Leahy, D-Vt.

Chairman of the Senate Homeland Security Committee Senator Joseph Lieberman, I-Conn. said the DHS “must follow federal privacy laws – in this case the E-Government Act which requires privacy impact assessments before personal information can be used – in order to maintain public support for these new technologies.”

Data fed into the ADVISE pilot projects included:

• The no-fly list of people barred from domestic air travel and the list of people who require special inspections before flying.
• More than 3.6 million shipping records from a commercial data provider with names of cargo shippers and consignees.
• Terrorist Screening Center lists of people who tried to cross the U.S.-Canadian border at a port-of-entry.
• Classified intelligence reports about illicit traffic in weapons of mass effect.
• Lists of foreign exchange students, immigrants under investigation and people from special interest countries.

DHS Lied About Using Data

Despite the fact that the DHS assured Congress in 2006 that ADVISE was not operational, the DHS Inspector General found that “on at least one occassion, the data was used to produce classified intelligence information.” At the Lawrence Livermore laboratory analysts used data from the pilot system “to uncover previously unknown connections between organized crime and terrorism.”

The DHS Privacy Office came to the conclusion that although required privacy analyses were ignored, the Privacy Act was not ‘technically’ violated because the live data was covered by privacy notices issued earlier for other programs that originally gathered the information.

Centralized Terror Watch List Full of Erroneous Information

An audit (PDF) released by Justice Department Inspector General Glenn Fine found duplication, erroneous information, incorrect tracking codes and poor coordination on the Centralized Terrorist Watch List used to screen 270 million individuals every month. As of 2004 the list reportedly contained more than 700, 000 entries.

105 records given by Terrorist Screening Center employees given to the auditors for “routine quality assurance reviews” found that 38 percent still contained inaccuracies. The audit also shows that some agencies didn’t include all the records in the database because of data sync issues. Problems with the faulty watch list listed in the Inspector Fine’s report (PDF) included:

“Deficiencies in the accuracy of watchlist data increase the possibility that reliable information will not be available to frontline screening agents, which could prevent them from successfully identifying a known or suspected terrorist during an encounter or place their safety at greater risk by providing inappropriate handling instructions for a suspected terrorist. Furthermore, inaccurate, incomplete, and obsolete watchlist information increases the chances of innocent persons being stopped or detained during an encounter because of being misidentified as a watchlist identity.”

A 2005 report by the Inspector General also called the accuracy and completeness of the nation’s watch lists into question. For more detailed findings of the report see the article on Wired.com.

More information can also be found at:

• DHS Data Mining System Shut Down After Privacy Slip Ups article from Wired.com
• The Blotter: FBI Terror Watch List ‘Out of Control’ article from ABC News
• DHS Re-Launches Watchlist Help Site After 27B Crushed the Old One – UPDATED article from Wired.com
• F.B.I. Data Mining Reached Beyond Initial Targets article from the New York Times
• Justice Lawyer Who Defied White House (‘The Law Required It’) article from Newsweek National News on MSNBC.com
• Spy chief: Oops! FISA changes didn’t aid arrests article from CNET News
• Watchdog asks: Why is Bush’s kid brother getting federal bucks? article from Raw Story
• Keith Olbermann’s Top 9/11 Story: The Promotion Of Failure in Bush Admistration article from Crooks and Liars

Back to Bill’s Blog | Bill’s Links and More