langutils

Langutils is a natural language processing library for Common Lisp. It was written by Ian Eslick and is now maintained by Kevin Raison. The code is reasonably documented with doc strings and inline comments. Also peruse the LISP2005-langutils.pdf which is a more involved exposition of the implementation and performance issues in the toolkit.

The code can be downloaded from https://github.com/eslick/cl-langutils. It will also be available shortly via Quicklisp.

Mailing lists:

• Developers / Users
• Announcements

The library provides a heirarchy of major functions and auxiliary functions related to the structured analysis and processing of open text. The major functions working from raw text up are:

• String tokenization (string -> string)
• Part of speech tagging (string -> tokens -> vector-document)
• Phrase chunking (vector-document -> phrases)

We also provide auxiliary functions that operate on strings, tokens or vector-documents. The lisp functions implementing the functionality can be found under the appropriately labled section in the reference below.

Strings:

• Tokenize a string (separate punctuation from word tokens)
• POS tag a string or file returning a file, string or vector-document
• Identify suspicious strings that may become tokens

Tokens:

• String to token-id conversion routines
• Save/Load token maps
• Guess the POS tag for a token (lexicon-based, also includes the porter stemmer)
• Identify suspicious tokens
• Identify stopwords; words used primarily as syntactic combinators
• Lookup words in the lexicon
• Get possible parts of speech for known words
• Lemmatize a token (find the root lemma for a given surface form)
• Generate all surface forms of a root word

Vector-Documents:

• Generate phrases using the regex chunker

Miscellaneous:

• Concept representation
  Simple lemmatized noun or verb phrases can be treated as equal abstract notions; provides a CLOS class wrapper.

Back to Common-lisp.net.