Lexical analyzer automaton

What is a lexical analyzer and where it is used?

A lexical analyzer is in charge of reading all the characters in a string, verifying and assigning its id, and creating a list of tokens. It would be used in a compiler, but it can be used wherever you need to recognize some characters.

Automaton

From Google, Automaton: “a machine that performs a function according to a predetermined set of coded instructions, especially one capable of a range of programmed responses to different circumstances.” Our task needs to be programmed to respond to different circumstances, or in our case, conditions.

What do we need to do?

Disclaimer: This is an example, the values can be adapted to your code or your goals. Our algorithm will be simple, its just a "conditions machine", being our first condition 0, then will be other conditions to recognize a character or a string and assign its token, and the condition 20 will be accepted, once its accepted, the token will be added to a list of all the tokens (This list may change depending on your code, it can be just values, a class, or an array, but mostly you will have to save the character or string and the token that may be a number) then, it returns to the condition 0 and it reads the next character, this will be a loop until there are no more characters.

1.- Automaton algorithm. (Example)

This way, following the algorithm, inside a while that reads all the characters, some if/else, you will have your lexical analyzer and your final list of tokens.