NLP Token Classification

Token: It is the smallest individual unit of text. Classification: It is a machine learning process of assigning data into predefined categories or classes. (OR) Simply can say that, grouping them together based on their group name.

Token Classification: 1.Text is first broken into tokens(words/subwords)using tokenizer. 2.Each token is then assigned a label(entity type). In simple words, It is an NLP task where the model assigns a label/category to each token (word) in a sentence. EXAMPLE: BillGates is a founder of Microsoft

Resultant output of token classification

It follows the process:

Token Classification Problem/Applications: 1.NER(Named Entity Recognition): It is an NLP task. It is used to identify an dclassify entities in the text. Extract real world entities. Example: i.person(name) ii.location iii.organization iv.email,skill,etc….. 2.POS(Parts Of Speech) tagging: It identifies grammar. It is a critical conversation,while using chatbots. Example: i.Noun ii.Verb iii.Pronoun iv.Object etc….. 3.Chunking: It groups adjacent tokens with same enteties or type into meaningful phrases. Example: Token like Bill Gates both belong to same entity then they will get combine, this avoids splitting your meaningful units.