Character-level convolutional networks for text classification

cleanUrl: 'paper/character-level-cnn'

1. 논문 선정 배경

2. Introduction

3. Character-level Convolutional Networks

3.1 Key Modules

• Stride : 입력데이터에 필터를 적용할 때 간격을 조절하는 것, 즉 필터가 이동할 간격의미.
ex) Stride = 1인 합성곱

• 오프셋 상수 : 동일 오브젝트 안에서 오브젝트 처음부터 주어진 요소나 지점까지의 변위차를 나타내는 정수형.

3.3 Model design

3.4 Data Augmentation using Thesaurus

4. Comparison Models

4.1 Traditional Method(기존의 전통적인 선형모델) : 모두 다항 로지스틱 회귀분석 사용

• Bag-of-words and its TFIDF : bag-of-words 모델은 각 데이터셋에 빈도가 높은 50000개 단어들로 구성됨
• Bag of Words : 단어들의 순서는 전혀 고려하지 않고, 단어들의 출현 빈도(frequency)에만 집중하는 텍스트 데이터의 수치화 표현 방법
TFIDF
[TFIDF 공식]

TF : 각 문서에서의 각 단어의 등장 빈도
DF : 특정 단어 t가 등장한 문서의 수
IDF : DF의 반비례
 
1) 해당 문서에서 나타난 횟수가 많을수록(TF)
2) 다른 문서에서 나타난 횟수가 적을수록(IDF)
→ 해당 문서를 대표하는 키워드
→ 그저 단어의 빈도만을 세는 bag-of-words와는 다름!

• Bag-of-ngrams and its TFIDF : 5-grams까지의 n-gram에서 가장 빈도가 높은 상위 500,000개로 구성됨, TFIDF는 동일한 과정
n-gram에 대한 이해

• Bag-of-means on word embedding : word2vec을 적용한 것에 대해 k-means을 사용함, 이를 통해 나온 단어를 클러스터링 된 단어들의 대표 단어로 사용함

4.2 Deep Learning Methods : 모두 word2vec을 이용하여 단어를 임베딩함(embedding size : 300)

• Word-based ConvNets
*우리의 모델은 character 기반이고, 사용한 비교군 딥러닝 모델은 단어 기반임

• LSTM(Long-short term memory)
학습 시 gradient clipping과 multinomial logistic regression을 사용하였음

5. Large-scale Datasets and Results

5.1 Dataset

• AG’s news corpus

• Sogou news corpus

• DBPedia ontology dataset

• Yelp reviews

• Yahoo! Answers dataset

• Amazon reviews

5.2 Result

6. Discussion

7. Conclusion and Outlook

8. Code

import numpy as np
import pandas as pd
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences

from keras.layers import Input, Embedding, Activation, Flatten, Dense
from keras.layers import Conv1D, MaxPooling1D, Dropout
from keras.models import Model

from google.colab import drive
drive.mount('/content/drive')

train_df=pd.read_csv('AG_news/train.csv')
test_df=pd.read_csv('AG_news/test.csv')

train_df.rename(columns={'Class Index':0,'Title':1,'Description':2},inplace=True)
test_df.rename(columns={'Class Index':0,'Title':1,'Description':2},inplace=True)

# concatenate column 1 and column 2 as one text
for df in [train_df, test_df]:
  df[1] = df[1] + df[2]
  df = df.drop([2], axis=1)

• 텍스트 소문자 변경

train_texts = train_df[1].values 
train_texts = [s.lower() for s in train_texts]

test_texts = test_df[1].values 
test_texts = [s.lower() for s in test_texts]

• Tokenizer

# Initialization
tk=Tokenizer(num_words=None, char_level=True,oov_token='UNK')

# Fitting
tk.fit_on_texts(train_texts)

• Construct Vocab

# construct a new vocabulary
alphabet = "abcdefghijklmnopqrstuvwxyz0123456789,;.!?:'\"/\\|_@#$%^&*~`+-=<>()[]{}"
char_dict = {}
for i, char in enumerate(alphabet):
    char_dict[char] = i + 1

# Use char_dict to replace the tk.word_index
tk.word_index = char_dict.copy()
# Add 'UNK' to the vocabulary
tk.word_index[tk.oov_token] = max(char_dict.values()) + 1
# oov_token: Out Of Vocabulary (oov) -> 모르는 단어로 인해 문제를 푸는 것이 까다로워지는 상황 처리함

# Convert string to index
train_sequences = tk.texts_to_sequences(train_texts)
test_texts = tk.texts_to_sequences(test_texts)

• Padding

train_data = pad_sequences(train_sequences, maxlen=1014, padding='post')
test_data = pad_sequences(test_texts, maxlen=1014, padding='post')

# Convert to numpy array
train_data = np.array(train_data, dtype='float32')
test_data = np.array(test_data, dtype='float32')

• Get Label

train_classes = train_df[0].values
train_class_list = [x - 1 for x in train_classes]

test_classes = test_df[0].values
test_class_list = [x - 1 for x in test_classes]

from tensorflow.keras.utils import to_categorical
train_classes = to_categorical(train_class_list)
test_classes = to_categorical(test_class_list)

• Parameter

input_size = 1014
vocab_size = len(tk.word_index)
embedding_size = 69
conv_layers = [[256, 7, 3],
               [256, 7, 3],
               [256, 3, -1],
               [256, 3, -1],
               [256, 3, -1],
               [256, 3, 3]]

fully_connected_layers = [1024, 1024]
num_of_classes = 4
dropout_p = 0.5
optimizer = 'adam'
loss = 'categorical_crossentropy'

• Embedding Layer

# Embedding weights
embedding_weights = []  # (70, 69)
embedding_weights.append(np.zeros(vocab_size))  # (0, 69)

for char, i in tk.word_index.items():  # from index 1 to 69
    onehot = np.zeros(vocab_size)
    onehot[i - 1] = 1
    embedding_weights.append(onehot)

embedding_weights = np.array(embedding_weights)
print('Load')

# Embedding layer Initialization
embedding_layer = Embedding(vocab_size + 1,
                            embedding_size,
                            input_length=input_size,
                            weights=[embedding_weights])

• Model Construction

# Input
inputs = Input(shape=(input_size,), name='input', dtype='int64')  # shape=(?, 1014)
# Embedding           
x = embedding_layer(inputs)
# Conv
for filter_num, filter_size, pooling_size in conv_layers:
    x = Conv1D(filter_num, filter_size)(x)
    x = Activation('relu')(x)
    if pooling_size != -1:
        x = MaxPooling1D(pool_size=pooling_size)(x)  # Final shape=(None, 34, 256)
x = Flatten()(x)  # (None, 8704)
# Fully connected layers
for dense_size in fully_connected_layers:
    x = Dense(dense_size, activation='relu')(x)  # dense_size == 1024
    x = Dropout(dropout_p)(x)
# Output Layer
predictions = Dense(num_of_classes, activation='softmax')(x)
# Build model
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])  # Adam, categorical_crossentropy
model.summary()

• Shuffle

indices = np.arange(train_data.shape[0])
np.random.shuffle(indices)

x_train = train_data[indices]
y_train = train_classes[indices]

x_test = test_data
y_test = test_classes

• Training

learning_history=model.fit(x_train, y_train,
          validation_data=(x_test, y_test),
          batch_size=128,
          epochs=10,
          verbose=2)

• Result

import matplotlib.pyplot as plt

hist = pd.DataFrame(learning_history.history)
hist['epoch'] = learning_history.epoch
hist.tail()


plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.plot(hist['epoch'], hist['accuracy'], label = 'Train accuracy')
plt.plot(hist['epoch'], hist['val_accuracy'], label = 'Val accuracy')
plt.legend()
plt.show()