Research any topic before you write.

Find related topics. | Discover entities. | See connections. | Build a topical map.

Lexical analysis

Lexical tokenization is conversion of a text into (semantically or syntactically) meaningful lexical tokens belonging to categories defined by a "lexer" program. In case of a natural language, those categories include nouns, verbs, adjectives, punctuations etc. In case of a programming language, the categories include identifiers, operators, grouping…

Details, Lexical token and lexical tokenization & Phrase structure

Use the mouse wheel or two fingers (on touchscreens) to zoom in and out of the map.

Research this topic

Explore the main themes, entities and connections around Lexical analysis. Start with the topic map, then use the sections below for research and deeper semantic analysis.

Explore this topic

Start with a few of the strongest sections from the source topic. These are research directions, not a list of keywords you must use.

Topics to explore

Browse the full topic structure. Each item opens a new analysis centered on that subject.

Overview

Rule-based programs

Disambiguation of "lexeme"

Lexical token and lexical tokenization

Lexical grammar

Details

Obstacles

Lexer generator

Phrase structure

Context-sensitive lexing

Advanced semantic analysis

Deeper signals for content research, entity SEO and topical coverage. The plain-language headings explain what each technical view is useful for.

Map overview Semantic statistics

Lexical analysis

Nodes93
Edges92
Triples34
Avg. degree1.98
Density0.021505
Components1

How this topic connects Entity context

See the strongest relationship patterns around the current topic before diving into the raw triples.

Lexical analysis

Top relations

related to External links · 23
Lexical analysis → Archived, Chan, Chey-Woei, Computer Languages, Craig, Developer Works, E-009-021, E-009-079, IBM, Jan, Jien-Tsai, NSC, On, Retrieved, S0096-0551, Structures, Systems, The Art, Tokenization, Trim
related to Phrase structure · 5
Lexical analysis → However, Less, Lexical, Omitting, This

Important terminology Word statistics

Use these terms to understand the vocabulary surrounding the topic, not as a checklist for keyword stuffing.

Important terminology

lexer lexical tokens token parser may language characters string lexeme often languages tokenization lexers example used also grammar analysis include

Entity relationships Subject–Predicate–Object triples

SubjectPredicateObjectConfidenceSrc
lexinstance ofwhich are understood by a lexical analyzer generator0.80text
or handcoded equivalent finite-state automatainstance ofwhich are understood by a lexical analyzer generator0.80text
contractionsinstance ofeven here there are many edge cases0.80text
hyphenated wordsinstance ofeven here there are many edge cases0.80text
emoticonsinstance ofeven here there are many edge cases0.80text
and larger constructs such as URIsinstance ofeven here there are many edge cases0.80text
Lexical analysisrelated to External linksYang0.60section
Lexical analysisrelated to External linksTsay0.60section
Lexical analysisrelated to External linksChey-Woei0.60section
Lexical analysisrelated to External linksChan0.60section
Lexical analysisrelated to External linksJien-Tsai0.60section
Lexical analysisrelated to External linksOn0.60section

Related concept clusters Concept neighborhoods

These clusters group vocabulary that occurs around closely connected concepts in the source material.

    Connections between topic areas Semantic bridges

    Bridges can reveal useful research angles that are easy to miss in a flat list of related terms.

    Min side: 3
    For writers, content strategists, SEOs, marketers and creators — from quick topic research to advanced semantic analysis.