mikroC

mikroC - C Compiler for Microchip PIC microcontrollers

making it simple...

TOKENS

Token is the smallest element of a C program that is meaningful to the compiler. The parser separates tokens from the input stream by creating the longest token possible using the input characters in a left–to–right scan.

mikroC recognizes following kinds of tokens:

-keywords,

-identifiers,

-constants,

-operators,

-punctuators (also known as separators).

Token Extraction Example

Here is an example of token extraction. Let’s have the following code sequence:

inter = a+++b;

First, note that inter would be parsed as a single identifier, rather than as the keyword int followed by the identifier er.

The programmer who wrote the code might have intended to write

inter = a + (++b)

but it won’t work that way. The compiler would parse it as the following seven tokens:

page

38

inter

// identifier

=// assignment operator

a// identifier

++// postincrement operator

+// addition operator

b// identifier

;// semicolon separator

Note that +++ parses as ++ (the longest token possible) followed by +.

MikroElektronika: Development tools - Books - Compilers