Tokenization The process of segmenting running text into words and sentences. Electronic text is a linear sequence of symbols (characters or words or phrases). Naturally, before any real text processing is to be done, text needs to be segmented into linguistic units such as words, punctuation, numbers, alpha-numerics, etc. This process is called tokenization. In English, words are often separated from each other by blanks (white space), but not all white space is equal. Both “Los Angeles” and “rock 'n' roll” are individual thoughts despite... [More]

Tags:  semantics opennlp lexer tokenization natural_language_processi... parsing parse syntax apache tokens open_nlp rdf segementation parser stanford watson nlp lexical_analysis