lexical analysis
Lexical analysis is the process of converting a sequence of characters, such as source code, into a sequence of tokens. Tokens are meaningful sequences of characters that represent basic elements like keywords, operators, and identifiers. This step is crucial in programming languages, as it helps in understanding the structure and syntax of the code.
During lexical analysis, a tool called a lexer or scanner reads the input text and identifies these tokens. The output is typically used by a parser, which further analyzes the tokens to build a structured representation of the code, enabling easier interpretation and execution.