Annotator normalisation

An interesting aspect of human annotations, of which figure 5.7 is a possible example, is that they appear to be self-normalising. In other words, when an annotator is assigned the task of dividing a dialogue ``into topics'', he or she has a preconception about how frequently topics might change. Reading a monotopical stretch of a document, it is theorised that the annotator begins to begin to feel that a topic change is due, and becomes more sensitive to possible changes. In this way, a text comprising regular, obvious changes will be marked with those changes, whereas a text with only subtle changes (if any) seems to be marked more sensitively by human annotators. When told to find topic changes, the annotators usually do. Examining the second half of 5.7, a very regular pattern of topic changes emerges after a gap (with, incidentally, no support from the TextTiling metric). In consulting with test subjects, it becomes clear that if they fail to find `enough' topics they feel they must be mistaken and begin to look more closely. This is in contrast with the system itself, which uses an absolute metric and is capable of assigning no topic breaks at all to some documents (for example figure 5.8).

Figure 5.6: American Culture advising session, $ps=20$ $bs=6$

Figure 5.7: CMU dialogue, $ps=20$ $bs=6$

Figure 5.8: CMU dialogue 2, $ps=20$ $bs=6$

Figure 5.9: ICSI dialogue, $ps=20$ $bs=6$

Figure 5.10: ICSI dialogue 2, $ps=20$ $bs=6$