Exploring Inter Tagger Consistency Measures

Kipp and Campbell (2006) examined tags assigned to the same URL in del.icio.us and determined that MDS and frequency graphs showed clusters of related terms as well as divergences between synonyms. Professional indexers too exhibit convergence and divergence in indexing behaviour, which has been measured in inter-indexer consistency studies. Leonard (1977) and Markey (1984) examined the results of multiple inter-indexer consistency studies examining not only the levels of inconsistency which varied widely but also the level of indexing exhaustivity (number of terms assigned to each document), method of collecting indexing data and vocabulary size. The majority of inter-indexer consistency studies show high levels of inconsistency between indexers (Leonard 1977; Markey 1984). While inter-indexer consistency studies have traditionally compared the indexing terms used by a small group of indexers, it is possible to adapt some of the more common measures to be used with large groups of indexers. A number of measures were examined in this study to determine which measures provide the most ability to distinguish between different indexers. This study used Salton's Cosine measure, the Jaccard measure--also known as Hooper and Rolling's measures (Markey 1984), Wolfram and Olsen's Inter-indexer Consistency Density (Wolfram and Olsen 2007) and a Pairwise Jaccard measure (compares all indexers to each other without the need for a centroid or known good set of index terms). This study is part of a larger study examining measures of convergence and divergence in tagging systems. One goal of the larger study is to examine different ways of analysing tag data to see which methods provide the most useful analyses of the structures which develop in tagging. By calculating a number of different inter-indexer consistency measures it may be possible to make distinctions between tag lists to provide predictive analysis of tagging patterns.