MLP model with Keras and text2vec package

[KafeelI]

Hello

I have built a MLP model using keras to classify text. In order to predict the class of a given text, again I did below steps which I did for "training" set in the model building process

• q is some random question in text format
• ls is a defined function to convert text into lower case and do stemming

token=itoken(q,preprocess_function=ls,tokenizer=word_tokenizer)
vtxt=create_vocabulary(token,stopwords=stemDocument(stopwords('english')),ngram=c(1,1))
vectorizer=vocab_vectorizer(vtxt)
# #Document term matrix
dtmq=create_dtm(token,vectorizer)

To get TF-IDF, which is parsed in MLP, I have used below steps

model1=TfIdf$new(smooth_idf = TRUE,norm="l2")
dtm=model1$fit_transform(dtm)

Now the dimension of 'dtmq' is (1,35) but my model requires input_shape of 1462, how can I convert the given text in such a way that my MLP model accepts.

[KafeelI]

By using pruned vectorizer used for training part, we will be able to get same dimension.

#Preparation
tokenq=itoken(q,preprocess_function=ls,tokenizer=word_tokenizer)
vectorizer=vocab_vectorizer(pruned_vocab)

#Document term matrix
dtmq=create_dtm(tokenq,vectorizer)
dim(dtmq)

[jcblum]

If your question's been answered (even by you!), would you mind choosing a solution? It helps other people see which questions still need help, or find solutions if they have similar problems. Here’s how to do it:

FAQ: How do I mark a solution?

If your question has been answered, don't forget to mark the solution! How do I mark a solution? Find the reply you want to mark as the solution and look for the row of small gray icons at the bottom of that reply. Click the one that looks like a box with a checkmark in it: [image] Hovering over the mark solution button shows the label, "Select if this reply solves the problem". If you don't see the mark solution button, try clicking the three dots button ( ••• ) to expand the full set of options. When a solution is chosen, the icon turns green and the hover label changes to: "Unselect if this reply no longer solves the problem". Success! [solution_reply_author] …