indexr.utils package

Submodules

indexr.utils.nlp module

indexr.utils.nlp.tokenize(text)

Convert a text to tokens.

Example:

>>> print(tokenize('Hello world!'))
['Hello', 'world', '!']

Parameters:text (str) – Text to tokenize. Returns:List of tokens. Return type:list

Module contents

indexr.utils.tokenize(text)

Convert a text to tokens. Example: >>> print(tokenize(‘Hello world!’)) [‘Hello’, ‘world’, ‘!’]

Parameters:text (str) – Text to tokenize. Returns:List of tokens. Return type:list