Brasil's shootdown program

Did you know that Brazil has a program to shoot down planes suspected of smuggling drugs?

One of the 512 topics identified from the #cablegate corpus has the following keywords:

provide comdabra aircraft notes braf exchange machado safety shootdown information traffic gob ref abd procedures control air program force

This identifies three cables:

04BRASILIA1938 2004-08-02 20:08
GOB would provide information to the USG about the status of Brazil's shootdown program once that program begins

06BRASILIA2002 2006-09-20 19:07
Mission Brazil herewith recommends annual recertification by the President of Brazil's Air Bridge Denial ('shootdown") Program/p>

09BRASILIA1142 2009-09-14 21:09
Mission is confident that there has been no deterioration in Brazilian safety standards over the last year and recommends that the Presidential Determination on the Brazilian Shootdown Law be renewed for 2009

This is one of five topics containing the keyword 'aircraft'. The others deal with rendition, Brazils purchase of the FX fighter, Varig's problems and their implications for Boeing, and airline security.

To explore these and other topics in the wikileaks corpus try the browser: use your web browser's search function to find topics mentioning your chosen keyword ('aircraft' in the example above), and explore the topics you uncover ...

Wikileaks Topics

cables

topics

This blog will document an experimental use of topic modelling to develop a site for browsing a multithreaded collection of documents. We start from a collection of documents (each viewed as a bag of words), and use Latent Dirichlet Allocation (LDA) to model the each document as a mixture of a number of topics. A topic is a probability distribution over words. Once we choose a fixed number of topics, LDA provides a set of topics and the proportions in which they should be mixed in each document to best approximate our collection.

We use the MALLET tools from UMASS to perform this analysis.

We use the Wikileaks #cablegate collection of cables as an example corpus.

This is a work in progress. We hope it will rapidly improve.

Two documents are similar if they are modelled by similar collections of topics, so we can present the structure of the collection of cables by linking each one to its nearest neighbour. You can access the source of each document by clicking on the node that represents it.

Just as two documents can be linked by a common topic, two topics can be linked by the documents they have in common. So we can also present the structure of the set of topics inferred by LDA in a similar (actually dual) fashion. You can inspect the most frequent words in each topic by hovering over the node that represents it.

We are now working on linking these two views.

We are eager to have feedback on the interface and how it can help to discover information in a previously unconnected collection of documents.