2-1 말뭉치, 토큰, 타입

1. 말뭉치

2. 토큰

spaCy · Industrial-strength Natural Language Processing in Python

spaCy is a free open-source library for Natural Language Processing in Python. It features NER, POS tagging, dependency parsing, word vectors and more.

http://spacy.io/

import spacy
nlp = spacy.load('en')
text = "Don't give up, study with daiv."
print([str(token) for token in nlp(text.lower())])

>>>
output : ['do', "n't", 'give', 'up', ',', 'study', 'with', 'daiv', '.']

NLTK

NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum.

http://www.nltk.org/

from nltk.tokenize import TweetTokenizer
tweet=u"Snow White and the Seven Degrees#makeAMovieCold@midnight:-)"
tokenizer = TweetTokenizer()
print(tokenizer.tokenize(tweet.lower()))

>>> 
['snow', 'white', 'and', 'the', 'seven', 'degrees', '#makeamoviecold', '@midnight', ':-)']

# 공백을 기준으로 단어를 분리해줍니다.
text = "The journey is the reward."
text_low = text.lower()
text_low = text.replace('.', '')
words = text_low.split(' ')
print(words)

>>> 
['the', 'journey', 'is', 'the', 'reward']

3. 타입