Maximum Entropy (MaxEnt) Classifier

Maxent

Maximum entropy (maxent) classifier has been a popular text classifier, by parameterizing the model to achieve maximum categorical entropy, with the constraint that the resulting probability on the training data with the model being equal to the real distribution.

The maxent classifier in shorttext is impleneted by keras. The optimization algorithm is defaulted to be the Adam optimizer, although other gradient-based or momentum-based optimizers can be used. The traditional methods such as generative iterative scaling (GIS) or L-BFGS cannot be used here.

To use the maxent classifier, import the package:

>>> import shorttext
>>> from shorttext.classifiers import MaxEntClassifier

Loading NIH reports as an example:

>>> classdict = shorttext.data.nihreports()

The classifier can be instantiated by:

>>> classifier = MaxEntClassifier()

Train the classifier:

>>> classifier.train(classdict, nb_epochs=300)

After training, it can be used for classification, such as

>>> classifier.score('cancer immunology')   # NCI tops the score
>>> classifier.score('children health')     # NIAID tops the score
>>> classifier.score('Alzheimer disease and aging')    # NIAID tops the score

To save the model,

>>> classifier.save_compact_model('/path/to/filename.bin')

To load the model to be a classifier, enter:

>>> classifier2 = MaxEntClassifier.from_pretrained('/path/to/filename.bin')

shorttext.classifiers.bow.maxent.MaxEntClassification.logistic_framework(nb_features: int, nb_outputs: int, l2reg: float = 0.01, bias_l2reg: float = 0.01, optimizer: Literal['sgd', 'rmsprop', 'adagrad', 'adadelta', 'adam', 'adamax', 'nadam'] = 'adam') → tensorflow.keras.Model

Create a maximum entropy classifier neural network.

Args:

nb_features: Number of input features. nb_outputs: Number of output classes. l2reg: L2 regularization coefficient. Default: 0.01. bias_l2reg: L2 regularization for bias. Default: 0.01. optimizer: Optimizer. Options: sgd, rmsprop, adagrad, adadelta, adam, adamax, nadam. Default: adam.

Returns:

Keras Sequential model for maximum entropy classification.

class shorttext.classifiers.bow.maxent.MaxEntClassification.MaxEntClassifier(preprocessor: callable | None = None)

Bases: AbstractScorer, CompactIOMachine

Maximum entropy classifier.

A classifier that implements the principle of maximum entropy for text categorization using bag-of-words features.

Reference:

Adam L. Berger et al., “A Maximum Entropy Approach to Natural Language Processing,” Computational Linguistics 22(1): 39-72 (1996).

__init__(preprocessor: callable | None = None)

Initialize the classifier.

Args:

preprocessor: Text preprocessing function. Default: lowercase.

shorttext_to_vec(shorttext: str) → SparseArray

Convert short text to sparse vector.

Args:

shorttext: Input text.

Returns:

Sparse vector representation.

train(classdict: dict[str, list[str]], nb_epochs: int = 500, l2reg: float = 0.01, bias_l2reg: float = 0.01, optimizer: Literal['sgd', 'rmsprop', 'adagrad', 'adadelta', 'adam', 'adamax', 'nadam'] = 'adam') → None

Train the classifier.

Args:

classdict: Training data. nb_epochs: Number of training epochs. Default: 500. l2reg: L2 regularization coefficient. Default: 0.01. bias_l2reg: L2 regularization for bias. Default: 0.01. optimizer: Optimizer. Default: adam.

savemodel(nameprefix: str) → None

Save the trained model to files.

Args:

nameprefix: Prefix for output files.

Raises:

ModelNotTrainedException: If not trained.

loadmodel(nameprefix: str) → None

Load a trained model from files.

Args:

nameprefix: Prefix for input files.

score(shorttext: str) → dict[str, float]

Calculate classification scores for all class labels.

Args:

shorttext: Input text.

Returns:

Dictionary mapping class labels to scores.

Raises:

ModelNotTrainedException: If not trained.

classmethod from_pretrained(name: str, compact: bool = True) → Self

Load a MaxEntClassifier from file.

Args:

name: Model name (compact) or file prefix (non-compact). compact: Whether to load compact model. Default: True.

Returns:

MaxEntClassifier instance.

shorttext.classifiers.bow.maxent.MaxEntClassification.load_maxent_classifier(name: str, compact: bool = True) → MaxEntClassifier

Deprecated. Use MaxEntClassifier.from_pretrained.

Deprecated since version 4.0.1: This will be removed in 5.0.0.

Reference

Adam L. Berger, Stephen A. Della Pietra, Vincent J. Della Pietra, “A Maximum Entropy Approach to Natural Language Processing,” Computational Linguistics 22(1): 39-72 (1996). [ACM]

Daniel E. Russ, Kwan-Yuet Ho, Joanne S. Colt, Karla R. Armenti, Dalsu Baris, Wong-Ho Chow, Faith Davis, Alison Johnson, Mark P. Purdue, Margaret R. Karagas, Kendra Schwartz, Molly Schwenn, Debra T. Silverman, Patricia A. Stewart, Calvin A. Johnson, Melissa C. Friesen, “Computer-based coding of free-text job descriptions to efficiently and reliably incorporate occupational risk factors into large-scale epidemiological studies”, Occup. Environ. Med. 73, 417-424 (2016). [BMJ]

Daniel Russ, Kwan-yuet Ho, Melissa Friesen, “It Takes a Village To Solve A Problem in Data Science,” Data Science Maryland, presentation at Applied Physics Laboratory (APL), Johns Hopkins University, on June 19, 2017. (2017) [Slideshare]

Home: Homepage of shorttext