A rule based Tibetan part-of-speech (POS) tagger for the creation of gold standard training data

Click here to view record
Main author: Garrett, Edward
Other authors: Hill, Nathan W.
Format: Datasets           
Online access: Click here to view record


id eprints-24155
recordtype eprints
institution SOAS, University of London
collection SOAS Research Online
language English
language_search English
topic PI Oriental languages and literatures
PL Languages and literatures of Eastern Asia, Africa, Oceania
description This rule based Tibetan part-of-speech (POS) tagger was prepared in the course of the research project 'Tibetan in Digital Communication' (2012-2015) hosted at SOAS, University of London and funded by the UK's Arts and Humanities Research Council (grant code: AH/J00152X/1). For a description of the tag set see Garrett et al. 2014. and Garrett et al. 2015. For a description of the tagger itself see Garrett et al. 2014. Note that the tagger must be used together with a lexicon (for example Hill & Garrett 2017a). One must use one's own script to tag all words with all tags in the lexicon and then apply the tagger to remove incorrect tags. On the associated corpus of 318,230 words (Hill & Garrett 2017b) the lexical tagger (i.e. simply applying all available tags to all words) tags 141,911 words with the correct unique tag, achieves as accuracy of 1.000 (by definition getting the right tag among others for each word) with an ambiguity of 2.73111. In contrast, the Rule Tagger tags 241,256 words with the correct unique tag, achieves an accuracy of 0.99893 and an ambiguity of 1.38577. Because this tagger does not achieve ambiguity 1.000 it is not suitable for tagging large scale corpora, but instead is useful for the creation of gold standard training data. N.B. In some rare cases the tagger removes all POS-tags.
format Datasets
author Garrett, Edward
author_facet Garrett, Edward
Hill, Nathan W.
authorStr Garrett, Edward
author_letter Garrett, Edward
author2 Hill, Nathan W.
author2Str Hill, Nathan W.
title A rule based Tibetan part-of-speech (POS) tagger for the creation of gold standard training data
publisher Zenodo
publishDate 2017
url https://eprints.soas.ac.uk/24155/

Similar items