Я пытаюсь использовать Наивный Байес для обнаружения юмора в текстах. У меня есть этот код, взятый из здесь, но у меня есть некоторые ошибки, и я не не знаю, как их решить, потому что я новичок в машинном обучении и этих алгоритмах. Мои данные о поездах содержат однострочники. Я знаю, что другие задают тот же вопрос, но я пока не нашел ответа.
import os
import io
from pandas import DataFrame
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
def readFiles(path):
for root, dirnames, filenames in os.walk(path):
for filename in filenames:
path = os.path.join(root, filename)
inBody = False
lines = []
f = io.open(path, 'r', encoding='latin1')
for line in f:
if inBody:
lines.append(line)
elif line == '\n':
inBody = True
f.close()
message = '\n'.join(lines)
yield path, message
def dataFrameFromDirectory(path, classification):
rows = []
index = []
for filename, message in readFiles(path):
rows.append({'message': message, 'class': classification})
index.append(filename)
return DataFrame(rows, index=index)
data = DataFrame({'message': [], 'class': []})
data = data.append(dataFrameFromDirectory('G:/PyCharmProjects/naive_bayes_classifier/train_jokes', 'funny'))
data = data.append(dataFrameFromDirectory('G:/PyCharmProjects/naive_bayes_classifier/train_non_jokes', 'notfunny'))
vectorizer = CountVectorizer()
counts = vectorizer.fit_transform(data['message'].values)
classifier = MultinomialNB()
targets = data['class'].values
classifier.fit(counts, targets)
examples = ['Where do steers go to dance? The Meat Ball', 'tomorrow I press this button']
examples_counts = vectorizer.transform(examples)
predictions = classifier.predict(examples_counts)
print(predictions)
И ошибки:
Traceback (most recent call last):
File "G:/PyCharmProjects/naive_bayes_classifier/NaiveBayesClassifier.py", line 55, in <module>
counts = vectorizer.fit_transform(data['message'].values)
File "C:\Users\mr_wizard\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 869, in fit_transform
self.fixed_vocabulary_)
File "C:\Users\mr_wizard\AppData\Local\Programs\Python\Python36\lib\site-packages\sklearn\feature_extraction\text.py", line 811, in _count_vocab
raise ValueError("empty vocabulary; perhaps the documents only"
ValueError: empty vocabulary; perhaps the documents only contain stop words
Вот некоторые данные от train_jokes
"[me narrating a documentary about narrators] ""I can't hear what they're saying cuz I'm talking"""
"Telling my daughter garlic is good for you. Good immune system and keeps pests away.Ticks, mosquitos, vampires... men."
I've been going through a really rough period at work this week It's my own fault for swapping my tampax for sand paper.
"If I could have dinner with anyone, dead or alive... ...I would choose alive. -B.J. Novak-"
Two guys walk into a bar. The third guy ducks.
Why can't Barbie get pregnant? Because Ken comes in a different box. Heyooooooo
Why was the musician arrested? He got in treble.
Did you hear about the guy who blew his entire lottery winnings on a limousine? He had nothing left to chauffeur it.
What do you do if a bird shits on your car? Don't ask her out again.
He was a real gentlemen and always opened the fridge door for me
train_jokes
содержит около 250 000 острот или твитов, а train_non_jokes
содержит простые предложения, которые не кажутся смешными. На данный момент у меня нет готового не смешного файла, только несколько предложений из Твиттера.
df
? - person Vivek Kalyanarangan   schedule 04.03.2018