Codename: Genesis

To content | To menu | To search

Friday, January 2 2009

Dynamically load the kernel

This post is related to rev.7

Our bootlader is now able to load a kernel written in C++ and compiled with Microsoft tools. It places the image at address 0x00010000 and jumps to the entry point. Now, the point is that this entry point is hard coded. It would be more robust to get dynamically this address. Luckily, this information is available in the header of the binary.

The Visual C++ compiler/linker generate an executable binary in a format called Portable Executable (PE). I'm not going to describe here in details this format: I'll have the opportunity to do this later. But, if you want more information, you could refer to the last version of the specifications.

What we need to know for the moment is that a PE header is made of several sections, or sub-header.

The first header is called DOS stub and contains some code used to maintain compatibility with the obsolete operating system. This stub begins with the two bytes "MZ" (for Mark Zbikowski, a developer of MSDOS) and has at offset 60 (0x3c) an offset to the next header.

This second header starts with the 4-byte signature "PE\0\0" that identifies this file as a PE format image file. This section contains lots of information for the image loader and the fields we read are SizeOfCode (offset 0x1C from 'PE'), AddressOfEntryPoint (offset 0x28) and BaseOfCode (offset 0x2C).

The first value is the size of our kernel code. We'll see in a future post that a PE image file defines several standard sections: a .text section contains code only, while .rdata and .data sections contains data, variables and constants. To simplify our present task, I have merged all these sections into a single .text one. Hence, the SizeOfCode field give us the size of the code and the data.

The BaseOfCode field gives the address of the beginning of the code section once loaded in memory. This value has to be added to the base of the image, which is its loaded address (0x00010000).

Finally, the AddressOfEntryPoint value is the value to add to the base of the image to get the address of the first instruction to execute.

We have modified our bootloader source in order to load and use all this information. We first load the file header in memory, at address 1000h:0000h

read_kernel_header:
    mov ax, 01000h        ; write into 1000h:0000h
    mov es, ax            ;
    xor bx, bx            ;

    mov ah, 02h           ; Read sector from drive
    mov al, 1             ; Number of sectors
    mov ch, 0             ; Cylinder
    mov cl, 2             ; Sector
    mov dh, 0             ; Head
    mov dl, [boot_drive] ; Bootable drive
    int 13h
    jc read_kernel_header  ; loop until success

Then, we parse the header to get the useful values:

    ; ds:bx points to the start of the image
    mov ax, 01000h
    mov ds, ax
    xor bx, bx
    
    ; Check MZ signature
    mov ax, [bx]
    cmp ax, 'MZ'
    jnz Error
    
    ; Read PE offset and move
    mov ax, [bx+3Ch]
    add bx, ax
    
    ; Check PE signature
    mov ax, [bx]
    cmp ax, 'PE'
    jnz Error
    
    ; Read code size
    mov eax, [bx+1Ch]
    shr eax, 9
    mov [kernel_size], eax
    
    ; Read address of entry point
    mov eax, [bx+28h]
    add eax, 10000h
    mov [entry_point], eax

    ; Read base of code segment
    mov eax, [bx+2Ch]
    mov [code_base], eax

Finally, we read the rest of the file:

read_kernel:
    mov ax, 01000h        ; write into 1000h:[code_base]
    mov es, ax            ;
    mov bx, [code_base]   ;

    mov ah, 02h           ; Read sector from drive
    mov al, [kernel_size] ; Number of sectors
    mov ch, 0             ; Cylinder
    mov cl, 3             ; Sector
    mov dh, 0             ; Head
    mov dl, [boot_drive]  ; Bootable drive
    int 13h
    jc read_kernel        ; loop until success

And we jump into the entry point, after switching into protected mode:

    mov ecx, [entry_point]
[...]
    call ecx

Next to these modifications, I've added several files in the kernel project. I've specifically create a Screen static class that manages all prints in the screen, and several helpers and typedefs. The kernel now starts by cleaning the screen, and then it prints the descriptive message "Starting GenOS".

Wednesday, December 31 2008

Write the kernel in C++

This post is related to rev.6

In the previous post, we have written some code that made the CPU run in 32 bits protected mode.

We can now use the Visual C++ compiler to produce a kernel binary from C/C++ code. That was not possible before, because this compiler cannot generate 16 bits code.

In fact, we'll not write C++ code for the moment, but we'll use inline assembly. This is only to make a smooth introduction to the C++ compiler.

We've created a new C++ Console Project with only one source file and the following function:

void __declspec(naked) kmain()
{
  __asm
  {
    mov eax, 0B8000h ; start of video memory
    mov [eax], 'B'   ; Put the ASCII-code of 'B'
    inc eax
    mov [eax], 1Bh   ; Assign a color code

hang:
    jmp hang
  }
}

This method is really simple - it prints a 'B' on the top-left of the screen and hangs into an infinite loop - but it needs some explanations.

The __asm { ... } statement is used to embed assembly-language instruction directly in the C++ source. It invokes the inline assembler built into the C++ compiler and doesn't require any additional link step. The compiler does not try to optimize the __asm block: what you write is what you get.

The __declspec(naked) annotation in the header tells the compiler not to add byte code at start nor at end of the function. Actually, a compiler often creates a prologue and an epilogue for each function, excepted when occur some optimizations or when a 'naked' declaration is specified.

The aim of a prologue is to:

  • set up EBP and ESP
  • reserve space on stack for local variables
  • save registers that should be modified in the body of the function

Here is a typical prologue:

push ebp                ; Save ebp
mov  ebp, esp           ; Set stack frame pointer
sub  esp, localbytes    ; Allocate space for locals
push <registers>        ; Save registers

An epilogue has to:

  • restore the saved register values
  • clean up the reserved space for local variables

This a standard epilogue:

pop  <registers>   ; Restore registers
mov  esp, ebp      ; Restore stack pointer
pop  ebp           ; Restore ebp
ret                ; Return from function

In our case, we have no parameters and no local variable, and we don't need to save the registers because this method never returns, so these prologue and epilogue are unneeded.

Now, we must modify the project configuration to fit our needs. Here are the most important parameters to change:

  • Ignore all default librairies: We don't want to link against Windows standard DLLs, not even against the C library that defines very standard functions like memcpy, memset, and so on.
  • Base address = 0x00010000: The binary will be loaded in memory at this address, and we tell the compiler to generate offsets in accordance with this.
  • Entry point = kmain: That tells the compiler to add in the file header the address of this function. This is the first function that will be called.
  • Exception handling = false
  • No basic runtime checks and No buffer security check: This would induce to link against several libraries.
  • Native subsystem: I'm not sure it's mandatory. This subsystem is used for Windows drivers, thus would fit well with our project. Other subsystems are Console, "Windows", "EFI" and "POSIX".

We can now compile the project to get an executable file named "kernel.exe" that will be placed just after the bootloader on the second sector of the floppy image.

The last thing to do is to perform several modification to the bootlader in order to load this file in memory and to jump to kmain()

The destination address of the read data is now 0x00010000 and we need to read 2 sectors on the disk.

read_sector:
    mov ax, 01000h             ; write into 1000h:0000h
    mov es, ax                 ;
    xor bx, bx                 ;

    mov ah, DISK_READ_SECTORS  ; Read sector from drive
    mov al, 2                  ; Load 1 sector
    mov ch, 0                  ; Cylinder=0
    mov cl, 2                  ; Sector=2
    mov dh, 0                  ; Head=0
    mov dl, [boot_drive]       ; Bootable drive
    INT_DISK
    jc read_sector             ; loop until success

Finally, the jump address is hard coded:

    jmp 010200h

Friday, December 26 2008

Switch from real mode to protected mode

This post is related to rev. 5

At boot time, the CPU runs in 16 bits real mode. This is a ancient execution mode of the Intel CPUs that is characterized by a 20 bit segmented memory address space (meaning that a maximum of 1 MiB of memory can be addressed), direct software access to BIOS routines and peripheral hardware, and no concept of memory protection or multitasking at the hardware level. In order to maintain backwards compatibility, the processor always starts in real mode. Therefore, we need to use a special mechanism to switch to an enhanced and more secured mode, called protected mode.

Protected mode adds the following features, used by most modern OSes:

  • 32 bits address space: 4 GiB of memory can be addressed
  • Paging: the OS could manage access rights of each page (4 KiB) of memory. This is also used for virtual memory.
  • Privilege levels: 4 privilege levels, or rings, are used to restrict tasks from accessing data or executing several privileged CPU instructions.
  • Multitasking: x86 processors introduced a mechanism of preemptive multitasking, but number of OSes don't use it.

To switch into protected mode, we just have to set the Protection Enable (PE) bit in the Control Register 0 (CR0). But before doing that, we have to setup two other things.

For historical reasons, all the memory cannot be accessed without enabling a feature called the "A20 line" (see this article for more information). A traditional way to enable the A20 line is to program the keyboard controller.

    cli                     ; no more interruptions
    
    xor cx, cx
    
clear_buf:
    in al, 64h              ; get input from keyboard status port
    test al, 02h            ; test the buffer full flag
    loopnz clear_buf        ; loop until buffer is empty
    mov al, 0D1h            ; keyboard: write to output port
    out 64h, al             ; output command to keyboard

clear_buf2:
    in al, 64h              ; wait 'till buffer is empty again
    test al, 02h
    loopnz clear_buf2
    mov al, 0dfh            ; keyboard: set A20
    out 60h, al             ; send it to the keyboard controller
    mov cx, 14h

wait_kbc:                   ; this is approx. a 25uS delay to wait
    out 0edh, ax            ; for the kb controler to execute our 
    loop wait_kbc           ; command.

The second thing is due to the fact that, in protected mode, the memory management is controlled through tables of descriptors. A descriptor is basically a structure of flags and addresses used by the CPU to perform safety checks and controls access to memory. Each descriptor defines a memory segment with a base address, a size, a granularity value and several security flags. All these descriptors are grouped into the Global Descriptor Table (GDT). We have to set up this table and to inform the CPU to use it.

We're going to set up a GDT with 3 entries: a mandatory NULL entry, and 2 entries for code and data segment in ring 0 (Kernel). In the future, we will add 2 other entries for code and data segment in ring 3 (User mode).

Moreover, this GDT will describe a flat memory model: each segment will start at address 0 and will cover the full range of memory.

I will not describe here the format of these entries because a lot of web pages have already done this better than me (see this article or this one)

So, here are the GDT definition:

gdt:
        dd 0            ; Null Segment
        dd 0

        dw 0FFFFh       ; Code segment, read/execute, nonconforming
        dw 0
        db 0
        db 10011010b
        db 11001111b
        db 0

        dw 0FFFFh       ; Data segment, read/write, expand down
        dw 0
        db 0
        db 10010010b
        db 11001111b
        db 0

gdt_desc:                       ; The GDT descriptor
        dw gdt_desc - gdt - 1    ; Limit (size)
        dd gdt                  ; Address of the GDT

and the code to load this GDT

    ; load GDT
    xor ax, ax
    mov ds, ax              ; Set DS-register to 0 - used by lgdt

    lgdt [gdt_desc]         ; Load the GDT descriptor

Once these operations are complete, we can switch into protected mode:

    
    ; enter pmode
    mov eax, cr0            ; load the control register in
    or  al, 1               ; set bit 0: pmode bit
    mov cr0, eax            ; copy it back to the control register

The CPU now runs in protected mode. To finalize the switch, we just have to set its different segments registers: DS and SS are set explicitely to point to the third decriptor (10h), whereas the code segment CS is set during the jmp instruction. This code jump is also needed to flush the CPU instruction pipeline.

    jmp 08h:protected_mode

[Bits 32]
protected_mode:   
    mov ax, 10h             ; Save data segment identifier
    mov ds, ax              ; Move a valid data segment into the data segment register
    mov ss, ax              ; Move a valid data segment into the stack segment register
    mov esp, 090000h        ; Move the stack pointer to 090000h

Read data from disk

This post is related to rev.4.

In the previous post, we have created a small program that boots when the computer is switched on. The only thing it does is to write a welcome message on the screen. Remember that this small bootloader was put in memory by the BIOS and that only 512 bytes were read from the boot device. This is a really tiny space to write a full OS ! Therefore, we will have to write some code that reads more data from data and puts it in memory.

To do that, we will invoke the BIOS interrupt 13h with several parameters passed through CPU registers.

You could get the full source here at Google Code.

The first thing to do is to create a new sector of data on the disk. This is easily done by adding some code after the AA55 signature which mark the end of the first sector. We simply move the code that prints the message, along with the infinite loop and the string data that is printed. This new code will be located in the second sector of the boot disk.

We now return to the main procedure, the entry point. We'll modify it in such a way it will be able to read the second sector of the boot device and to place it in memory just after the current code.

We start by reset the drive, in order to put it in a known configuration:

    mov ax, 00h                ; Reset the floppy drive
    mov dl, [boot_drive]       ; Set the drive
    int 13h

Then, we ask the BIOS to read the second sector of the boot drive and to place it at address 07C0:new_sector, just after the current memory sector:

    mov ax, cs                 ; write into 07C0:new_sector
    mov es, ax                 ;
    mov bx, new_sector         ;

    mov ah, 02h                ; Read sector from drive
    mov al, 1                  ; Load 1 sector
    mov ch, 0                  ; Cylinder=0
    mov cl, 2                  ; Sector=2
    mov dh, 0                  ; Head=0
    mov dl, [boot_drive]       ; Bootable drive
    int 13h

Cylinder/Head/Sector is a way to localize data chunks on a drive (see Wikipedia for more information).

The code is now placed in memory just after the current sector. All we have to do is to jump into this new code:

    jmp new_sector             ; Jump to the program

Saturday, December 20 2008

New project on Google Code

I've just created a new project in Google Code to have a SVN repository and several other good stuffs (bug track, wiki, ...). You can access it at this address: http://code.google.com/p/genos/.

The code will be always commited there with the BSD license and, from now on, each new note related to development of GenOS would point out a SVN revision, so you can get the related code.

Thursday, December 18 2008

First boot

Let start at the beginning: the very first thing an OS needs is to boot.

Our first job will be to write a boot sector that just prints a startup message and then loops forever.

To do this, we have to know how a computer starts. When we power on a computer, the BIOS starts up, initializes internal data and sets up several hardware devices. Then, it perform power-on self tests (POST) and display system settings on screen. After this initialization phase, it looks at the sequence of bootable devices (floppy, CD, DVD, hard disk, USB stick, ...) into its configuration parameters. It tries to initiate the boot sequence from the first device. If the BIOS does not find a valid boot sector, it will try the next device in the list. A valid boot sector is one that has the two bytes 0xAA 0x55 at offset 510. If it does not find a proper boot sector on a device, the startup process will halt. Once the BIOS has found the bootable device, it reads its full boot sector (512 bytes) and puts it into memory at linear address 7C00h. Then it jumps to that address and lets the bootstrap code get control.

At this point, the CPU runs in Real Mode [1] and the content of its registers is undefined, except for DL which contains the BIOS number of the bootable device (00h for floppy, 80h for first hard disk, ...)

You can download the sources of this bootstrap in the ZIP archive attached to this note. I don't have set up a real workspace for the moment, so the archive contains only several assembly sources and a Windows batch file used to build the boot sector. I'll write soon a new note to describe my workspace. The only thing you need for the moment is a recent version of NASM. Moreover, you probably want to install a virtual machine software to avoid rebooting your PC too often. I'm using Virtual PC, but there are several others that are freely downloadable.

So let describe our boot loader.

The first thing we're doing is to set the data segment DS to the current segment (07C0h) so we won't have to add an 7C00h to all our data

    mov ax, 0x7c0
    mov ds, ax

Then we set up the stack. We need this because we will use the call instruction, which uses the stack. We install this stack 2000h bytes after our boot sector to have enough room (~8 KiB).

    mov ax, 0x9c0
    mov ss, ax
    mov sp, 0

The next step is to display the welcome message.

    mov si, boot_message
    call PrintMessage

The procedure PrintMessage reads each byte pointed by ds:si and invokes BIOS interrupt 10h to print the character to the screen. The loop stop when a null character is read.

PrintMessage:          
    mov ah, 0eh              ; 'put character' function
PM_loop:
    lodsb                   ; load byte at ds:si into al
    or al,al                ; test if character is 0 (end)
    jz PM_done
    int 0x10                ; invoke BIOS interrupt
    jmp PM_loop
PM_done:
    ret

Finally, the CPU falls into a infinite loop which has the effect to hang the PC. I have inserted a hlt instruction inside the loop to halt the CPU until the next interrupt. Hence, the loop consumes very few CPU power.

Notes

[1] This theme (and a lot of others) will certainly be developped in a future note.

Friday, December 12 2008

Codename: Genesis

This is the first post of this new blog which aims to describe day after days the genesis of a new operating system.

To start, let me introduce myself. My name is Cedric Rousseau, I'm a 32 years old french developer and i've mainly coded since 10 years in C++ under Windows. I've started .NET and C# language in the first half 2006. I live near Paris and until now, I didn't wrote a lot of documents in english, so please be indulgent - English is not my mother tongue! I am married and have two (soon three) wonderful children. I work as software engineer for a start-up specialized in microelectronic. So the time that I reserve to work on this project will be probably reduced and not constant.

This project is really a challenge for me and it consists on creating a new operating system and describing in parallel what I am currently coding and all the choices that I am taking. Many developments will probably lead to a dead-end. Many other will start as "quick and dirty" code and will be enhance later. The aim is not to create a revolutionary OS to kill the good ol' Windows/OSX/Linux. Take it as an exercise. I only expect that this blog will help a lot of developers.