Building Fermat: A Journey from Scratch to JIT Compiler (Part 2)

A deep dive into the Lexical Analyzer, the first critical component of a programming language compiler ( Github : https://github.com/satyik/fermat)

In my previous articles, I talked about my journey to create a new programming language from scratch. In this article, I want to share how I developed the first component of the language — the lexical analyzer.

Imagine trying to understand a sentence in a foreign language. Before you can grasp its meaning, you first need to identify individual words, punctuation marks, and understand the role each plays. A lexer does exactly this for programming languages — it transforms a raw stream of characters into meaningful tokens that the parser can understand.

A lexical analyzer — more commonly referred to as a lexer — is a software component that takes a string of characters and breaks it down into smaller units that are meaningful and understandable by the language.

The Architecture

Our first major design hurdle was token categorization: how do we distinguish between a simple mathematical symbol like + and a complex language keyword like def?

To solve this, we used an enumeration with negative values. By assigning negative values to our custom token types, we create a “safe zone” that prevents the compiler from confusing standard characters with our language-specific keywords.

• Positive numbers (0–255): Reserved for simple, single characters such as (, ), +, and -. The lexer simply passes their standard ASCII values.
• Negative numbers (-1, -2, -3, …): Reserved for special tokens like KEYWORDS, IDENTIFIERS, and END_OF_FILE.

With this design in place, we can now move on to implementation. We start by creating a header file, Lexer.h, and defining our tokens.

enum Token {
  tok_eof = -1,

  // Keywords
  tok_def = -2,
  tok_extern = -3,
  tok_let = -4,
  tok_mut = -5,
  tok_if = -6,
  tok_then = -7,
  tok_else = -8,
  tok_for = -9,
  tok_in = -10,
  tok_while = -11,
  tok_do = -12,
  tok_end = -13,
  tok_import = -14,
  tok_export = -15,

  // Loop control
  tok_break = -16,
  tok_continue = -17,

  // Type keywords
  tok_type = -18,
  tok_struct = -19,
  tok_int = -25,
  tok_float = -26,
  tok_string = -27,
  tok_bool = -28,

  // New keywords
  tok_static = -29,
  tok_abstract = -30,

  // Primary tokens
  tok_identifier = -20,
  tok_number = -21,
  tok_string_lit = -22,

Next, we move on to state management.

Think of the lexer like a person reading a book one letter at a time. If they see the letter 'f', they don’t yet know whether it begins the word for, the word false, or a variable named fast. They need to buffer those letters in memory until they encounter a space or a symbol that gives the sequence meaning. That temporary memory is our state.

For simplicity, we use a global state. Although this is generally considered poor practice, it is acceptable here because compilation currently occurs in a single-threaded context for each file.

std::string IdentifierStr;
std::string StringValue;
double NumVal;
FILE *InputFile = nullptr;
std::string CurrentFilePath;
static int LastChar = ' ';

• IdentifierStr: Holds the current identifier or keyword name
• StringValue: Stores string literal content
• NumVal: Contains parsed numeric values
• InputFile: The file we’re reading from
• CurrentFilePath: For better error messages (we know which file had issues)
• LastChar: The last character read—this is the heart of our one-character lookahead strategy (which we will explain next)

The Core

Let’s now talk about the main tokenization logic, piece by piece. We will create a Lexer.cpp file that will contain this implementation.

We begin by ignoring whitespace — spaces, tabs, and newlines — since they carry no semantic meaning.

while (isspace(LastChar))
  LastChar = fgetc(InputFile);

Next, we check for identifiers and keywords. Identifiers begin with a letter or an underscore, and after the first character, we allow letters, digits, and underscores.

Once we’ve built the complete identifier, we check whether it matches any known keywords. If it doesn’t, we treat it as a regular identifier.

if (isalpha(LastChar) || LastChar == '_') {
    IdentifierStr = LastChar;
    while (isalnum((LastChar = fgetc(InputFile))) || LastChar == '_')
      IdentifierStr += LastChar;

    if (IdentifierStr == "def")
      return tok_def;
    if (IdentifierStr == "extern")
      return tok_extern;
    if (IdentifierStr == "let")
      return tok_let;
    if (IdentifierStr == "mut")
      return tok_mut;
    if (IdentifierStr == "if")
      return tok_if;
    if (IdentifierStr == "then")
      return tok_then;
    if (IdentifierStr == "else")
      return tok_else;
    if (IdentifierStr == "for")
      return tok_for;
    if (IdentifierStr == "in")
      return tok_in;
    if (IdentifierStr == "while")
      return tok_while;
    if (IdentifierStr == "do")
      return tok_do;
    if (IdentifierStr == "end")
      return tok_end;
    if (IdentifierStr == "import")
      return tok_import;
    if (IdentifierStr == "export")
      return tok_export;
    if (IdentifierStr == "break")
      return tok_break;
    if (IdentifierStr == "continue")
      return tok_continue;
    if (IdentifierStr == "type")
      return tok_type;
    if (IdentifierStr == "struct")
      return tok_struct;
    if (IdentifierStr == "int")
      return tok_int;
    if (IdentifierStr == "float")
      return tok_float;
    if (IdentifierStr == "string")
      return tok_string;
    if (IdentifierStr == "bool")
      return tok_bool;
    if (IdentifierStr == "static")
      return tok_static;
    if (IdentifierStr == "abstract")
      return tok_abstract;
    return tok_identifier;
  }

Why string comparison over hash tables? For a language with ~20 keywords, linear string comparison is actually faster than hash table overhead. If our language grows to 100+ keywords, we’d switch to a hash map or trie.

Now, for string literals, we look for the " character to detect the start of a string. We then proceed character by character. If we encounter a \, we peek at the next character and translate the escape sequence; otherwise, we add the character as-is. We continue this process until we reach the closing " or encounter EOF.

if (LastChar == '"') {
  StringValue = "";
  while ((LastChar = fgetc(InputFile)) != '"' && LastChar != EOF) {
    if (LastChar == '\\') {
      LastChar = fgetc(InputFile);
      switch (LastChar) {
        case 'n': StringValue += '\n'; break;
        case 't': StringValue += '\t'; break;
        case '\\': StringValue += '\\'; break;
        case '"': StringValue += '"'; break;
        default: StringValue += LastChar; break;
      }
    } else {
      StringValue += LastChar;
    }
  }
  LastChar = fgetc(InputFile);
  return tok_string_lit;
}

Edge case handling: What if someone forgets the closing quote? We check for EOF to prevent infinite loops. we need to emit a proper error message here.

After the closing quote: Notice we read one more character (LastChar = fgetc(InputFile)) to advance past the closing quote, preparing for the next token.

We will treat all numbers as floating-point values (double). This significantly simplifies the lexer — there’s no need to distinguish between integer and floating-point literals at this stage.

Instead of parsing the number ourselves, we collect its string representation and let C’s strtod do the heavy lifting. This gives us support for edge cases like scientific notation (1.5e-10) for free!

if (isdigit(LastChar) || LastChar == '.') {
  std::string NumStr;
  do {
    NumStr += LastChar;
    LastChar = fgetc(InputFile);
  } while (isdigit(LastChar) || LastChar == '.');
  NumVal = strtod(NumStr.c_str(), nullptr);
  return tok_number;
}

For a few operators like ->, ==, and !=, we encounter the lookahead problem. How do we distinguish between - (minus) and -> (the function return type arrow)?

To solve this, whenever we encounter a -, we peek at the next character:

• If it’s >, we consume it and return tok_arrow.
• If it’s anything else, we put it back using ungetc and return - as a single-character token.

if (LastChar == '-') {
  int NextChar = fgetc(InputFile);
  if (NextChar == '>') {
    LastChar = fgetc(InputFile);
    return tok_arrow;
  }
  // Not arrow, just minus
  ungetc(NextChar, InputFile);
  int ThisChar = LastChar;
  LastChar = fgetc(InputFile);
  return ThisChar;
}

// Equal: == or =
if (LastChar == '=') {
  int NextChar = fgetc(InputFile);
  if (NextChar == '=') {
    LastChar = fgetc(InputFile);
    return tok_eq;
  }
  ungetc(NextChar, InputFile);
  int ThisChar = LastChar;
  LastChar = fgetc(InputFile);
  return ThisChar;
}

// Not Equal: !=
if (LastChar == '!') {
  int NextChar = fgetc(InputFile);
  if (NextChar == '=') {
    LastChar = fgetc(InputFile);
    return tok_ne;
  }
  ungetc(NextChar, InputFile);
  int ThisChar = LastChar;
  LastChar = fgetc(InputFile);
  return ThisChar;
}

We use # to denote comments. After consuming a comment, we recursively call gettok() to fetch the next meaningful token.

if (LastChar == '#') {
  do
    LastChar = fgetc(InputFile);
  while (LastChar != EOF && LastChar != '\n' && LastChar != '\r');
  if (LastChar != EOF)
    return gettok();
}

Finally, if we haven’t matched anything else, the character must be a single-character token like +, *, (, or ). In that case, we simply return its ASCII value.

int ThisChar = LastChar;
LastChar = fgetc(InputFile);
return ThisChar;

Importing module and Supporting Multi-File Compilation

File State Management

void setInputFile(FILE *file, const std::string &path) {
  InputFile = file;
  CurrentFilePath = path;
  LastChar = ' ';
}

The reset mechanism: When switching files (for imports), we need to:

1. Update the file pointer
2. Track the new file path (for error messages)
3. Reset LastChar to a space—this is crucial! It ensures we start in a clean state

State Save/Restore

LexerState saveLexerState() {
  LexerState state;
  state.File = InputFile;
  state.FilePath = CurrentFilePath;
  state.LastChar = LastChar;
  state.CurToken = CurTok;
  state.IdentStr = IdentifierStr;
  state.StrVal = StringValue;
  state.NumValue = NumVal;
  return state;
}
void restoreLexerState(const LexerState &state) {
  InputFile = state.File;
  CurrentFilePath = state.FilePath;
  LastChar = state.LastChar;
  CurTok = state.CurToken;
  IdentifierStr = state.IdentStr;
  StringValue = state.StrVal;
  NumVal = state.NumValue;
}

The module import problem: When we encounters an import statement, we needs to:

1. Save the current lexer state
2. Switch to the imported file
3. Lex and parse that file
4. Restore the original state and continue

saveLexerState() creates a complete snapshot of all lexer state and restoreLexerState() brings it all back. This is essential for recursive module loading.

Conclusion

We have built a very basic lexer, which we will likely improve in the future by using a trie-based keyword matcher or a DFA-based tokenizer. We will also need to add proper error handling, number validation, and line/column tracking. However, for now, this implementation works well for our needs.

So we have finally made a lexer which is capable of producing a stream of tokens ready for parsing. In the next post, we’ll explore how parser takes these tokens and builds an Abstract Syntax Tree (AST) — the structured representation of your program’s meaning.