Introduction

This package shorttext is a Python package that facilitates supervised and unsupervised learning for short text categorization. Due to the sparseness of words and the lack of information carried in the short texts themselves, an intermediate representation of the texts and documents are needed before they are put into any classification algorithm. In this package, it facilitates various types of these representations, including topic modeling and word-embedding algorithms.

The package shorttext runs on Python 3.11, 3.12, and 3.13.

Characteristics:

• example data provided (including subject keywords and NIH RePORT); (see Data Preparation)

• text preprocessing; (see Text Preprocessing)

• pre-trained word-embedding support; (see Word Embedding Models)

• gensim topic models (LDA, LSI, Random Projections) and autoencoder; (see Supervised Classification with Topics as Features)

• topic model representation supported for supervised learning using scikit-learn; (see Supervised Classification with Topics as Features)

• cosine distance classification; (see Supervised Classification with Topics as Features, Word-Embedding Cosine Similarity Classifier)

• neural network classification (including ConvNet, and C-LSTM); (see Deep Neural Networks with Word-Embedding)

• maximum entropy classification; (see Maximum Entropy (MaxEnt) Classifier)

• metrics of phrases differences, including soft Jaccard score (using Damerau-Levenshtein distance), and Word Mover’s distance (WMD); (see Metrics)

• character-level sequence-to-sequence (seq2seq) learning; (see Character-Based Sequence-to-Sequence (seq2seq) Models)

• spell correction; (see Spell Correctors)

Author: Kwan Yuet Stephen Ho (LinkedIn, ResearchGate) Other contributors: Chinmaya Pancholi, Minseo Kim

Contribution

If you would like to contribute, feel free to submit the pull requests to the develop branch. You can talk to me in advance through e-mails or the Issues page.

Home: Homepage of shorttext