Codename: GenesisThe genesis of an Operating System2017-10-24T19:48:01+02:00Cédric Rousseauurn:md5:564beda28a290d1269163ba6060e7a36DotclearDynamically load the kernelurn:md5:5dcacc409c9029e755137f513e55aa332009-01-02T15:56:00+01:00cedrouBootbootkernelPE <h5><em>This post is related to <a href="http://code.google.com/p/genos/source/detail?r=7">rev.7</a></em></h5>
<p>Our bootlader is now able to load a kernel written in C++ and compiled with Microsoft tools. It places the image at address 0x00010000 and jumps to the entry point. Now, the point is that this entry point is hard coded. It would be more robust to get dynamically this address. Luckily, this information is available in the header of the binary.</p>
<p>The Visual C++ compiler/linker generate an executable binary in a format called <em>Portable Executable</em> (PE). I'm not going to describe here in details this format: I'll have the opportunity to do this later. But, if you want more information, you could refer to the last version of the <a href="http://www.microsoft.com/whdc/system/platform/firmware/PECOFF.mspx">specifications</a>.</p>
<p>What we need to know for the moment is that a PE header is made of several sections, or sub-header.</p>
<p>The first header is called <em>DOS stub</em> and contains some code used to maintain compatibility with the obsolete operating system. This stub begins with the two bytes <code>"MZ"</code> (for Mark Zbikowski, a developer of MSDOS) and has at offset 60 (<code>0x3c</code>) an offset to the next header.</p>
<p>This second header starts with the 4-byte signature <code>"PE\0\0"</code> that identifies this file as a PE format image file. This section contains lots of information for the image loader and the fields we read are <em>SizeOfCode</em> (offset <code>0x1C</code> from 'PE'), <em>AddressOfEntryPoint</em> (offset <code>0x28</code>) and <em>BaseOfCode</em> (offset <code>0x2C</code>).</p>
<p>The first value is the size of our kernel code. We'll see in a future post that a PE image file defines several standard sections: a <code>.text</code> section contains code only, while <code>.rdata</code> and <code>.data</code> sections contains data, variables and constants. To simplify our present task, I have merged all these sections into a single <code>.text</code> one. Hence, the <em>SizeOfCode</em> field give us the size of the code <em>and</em> the data.</p>
<p>The <em>BaseOfCode</em> field gives the address of the beginning of the code section once loaded in memory. This value has to be added to the base of the image, which is its loaded address (0x00010000).</p>
<p>Finally, the <em>AddressOfEntryPoint</em> value is the value to add to the base of the image to get the address of the first instruction to execute.</p>
<p>We have modified our bootloader source in order to load and use all this information. We first load the file header in memory, at address 1000h:0000h</p>
<pre>
read_kernel_header:
mov ax, 01000h ; write into 1000h:0000h
mov es, ax ;
xor bx, bx ;
mov ah, 02h ; Read sector from drive
mov al, 1 ; Number of sectors
mov ch, 0 ; Cylinder
mov cl, 2 ; Sector
mov dh, 0 ; Head
mov dl, [boot_drive] ; Bootable drive
int 13h
jc read_kernel_header ; loop until success
</pre>
<p>Then, we parse the header to get the useful values:</p>
<pre>
; ds:bx points to the start of the image
mov ax, 01000h
mov ds, ax
xor bx, bx
; Check MZ signature
mov ax, [bx]
cmp ax, 'MZ'
jnz Error
; Read PE offset and move
mov ax, [bx+3Ch]
add bx, ax
; Check PE signature
mov ax, [bx]
cmp ax, 'PE'
jnz Error
; Read code size
mov eax, [bx+1Ch]
shr eax, 9
mov [kernel_size], eax
; Read address of entry point
mov eax, [bx+28h]
add eax, 10000h
mov [entry_point], eax
; Read base of code segment
mov eax, [bx+2Ch]
mov [code_base], eax
</pre>
<p>Finally, we read the rest of the file:</p>
<pre>
read_kernel:
mov ax, 01000h ; write into 1000h:[code_base]
mov es, ax ;
mov bx, [code_base] ;
mov ah, 02h ; Read sector from drive
mov al, [kernel_size] ; Number of sectors
mov ch, 0 ; Cylinder
mov cl, 3 ; Sector
mov dh, 0 ; Head
mov dl, [boot_drive] ; Bootable drive
int 13h
jc read_kernel ; loop until success
</pre>
<p>And we jump into the entry point, after switching into protected mode:</p>
<pre>
mov ecx, [entry_point]
[...]
call ecx
</pre>
<p>Next to these modifications, I've added several files in the kernel project. I've specifically create a <code>Screen</code> static class that manages all prints in the screen, and several helpers and typedefs. The kernel now starts by cleaning the screen, and then it prints the descriptive message "Starting GenOS".</p>Write the kernel in C++urn:md5:ca19ab680d704301542bab5a5487b8a82008-12-31T16:35:00+01:00cedrouBootbootepilogkernelprologVisual C <h5><em>This post is related to <a href="http://code.google.com/p/genos/source/detail?r=6">rev.6</a></em></h5>
<p>In the previous post, we have written some code that made the CPU run in 32 bits protected mode.</p>
<p>We can now use the Visual C++ compiler to produce a kernel binary from C/C++ code. That was not possible before, because this compiler cannot generate 16 bits code.</p>
<p>In fact, we'll not write C++ code for the moment, but we'll use inline assembly. This is only to make a smooth introduction to the C++ compiler.</p>
<p>We've created a new C++ Console Project with only one source file and the following function:</p>
<pre>
void __declspec(naked) kmain()
{
__asm
{
mov eax, 0B8000h ; start of video memory
mov [eax], 'B' ; Put the ASCII-code of 'B'
inc eax
mov [eax], 1Bh ; Assign a color code
hang:
jmp hang
}
}
</pre>
<p>This method is really simple - it prints a 'B' on the top-left of the screen and hangs into an infinite loop - but it needs some explanations.</p>
<p>The <code>__asm { ... }</code> statement is used to embed assembly-language instruction directly in the C++ source. It invokes the inline assembler built into the C++ compiler and doesn't require any additional link step. The compiler does not try to optimize the <code>__asm</code> block: what you write is what you get.</p>
<p>The <code>__declspec(naked)</code> annotation in the header tells the compiler not to add byte code at start nor at end of the function. Actually, a compiler often creates a prologue and an epilogue for each function, excepted when occur some optimizations or when a 'naked' declaration is specified.</p>
<p>The aim of a prologue is to:</p>
<ul>
<li>set up EBP and ESP</li>
<li>reserve space on stack for local variables</li>
<li>save registers that should be modified in the body of the function</li>
</ul>
<p>Here is a typical prologue:</p>
<pre>
push ebp ; Save ebp
mov ebp, esp ; Set stack frame pointer
sub esp, localbytes ; Allocate space for locals
push <registers> ; Save registers
</pre>
<p>An epilogue has to:</p>
<ul>
<li>restore the saved register values</li>
<li>clean up the reserved space for local variables</li>
</ul>
<p>This a standard epilogue:</p>
<pre>
pop <registers> ; Restore registers
mov esp, ebp ; Restore stack pointer
pop ebp ; Restore ebp
ret ; Return from function
</pre>
<p>In our case, we have no parameters and no local variable, and we don't need to save the registers because this method never returns, so these prologue and epilogue are unneeded.</p>
<p>Now, we must modify the project configuration to fit our needs. Here are the most important parameters to change:</p>
<ul>
<li><strong>Ignore all default librairies</strong>: We don't want to link against Windows standard DLLs, not even against the C library that defines very standard functions like memcpy, memset, and so on.</li>
<li><strong>Base address = 0x00010000</strong>: The binary will be loaded in memory at this address, and we tell the compiler to generate offsets in accordance with this.</li>
<li><strong>Entry point = kmain</strong>: That tells the compiler to add in the file header the address of this function. This is the first function that will be called.</li>
<li><strong>Exception handling = false</strong></li>
<li><strong>No basic runtime checks</strong> and <strong>No buffer security check</strong>: This would induce to link against several libraries.</li>
<li><strong>Native subsystem</strong>: I'm not sure it's mandatory. This subsystem is used for Windows drivers, thus would fit well with our project. Other subsystems are <em>Console</em>, "Windows", "EFI" and "POSIX".</li>
</ul>
<p>We can now compile the project to get an executable file named "kernel.exe" that will be placed just after the bootloader on the second sector of the floppy image.</p>
<p>The last thing to do is to perform several modification to the bootlader in order to load this file in memory and to jump to <code>kmain()</code></p>
<p>The destination address of the read data is now 0x00010000 and we need to read 2 sectors on the disk.</p>
<pre>
read_sector:
mov ax, 01000h ; write into 1000h:0000h
mov es, ax ;
xor bx, bx ;
mov ah, DISK_READ_SECTORS ; Read sector from drive
mov al, 2 ; Load 1 sector
mov ch, 0 ; Cylinder=0
mov cl, 2 ; Sector=2
mov dh, 0 ; Head=0
mov dl, [boot_drive] ; Bootable drive
INT_DISK
jc read_sector ; loop until success
</pre>
<p>Finally, the jump address is hard coded:</p>
<pre>
jmp 010200h
</pre>Switch from real mode to protected modeurn:md5:115c6a8b3897329ba07482a13ec7af6f2008-12-26T15:54:00+01:00cedrouBootA20 lineGDTProtected modeReal mode <h5><em>This post is related to <a href="http://code.google.com/p/genos/source/browse/trunk/?r=5">rev. 5</a></em></h5>
<p>At boot time, the CPU runs in 16 bits real mode. This is a ancient execution mode of the Intel CPUs that is characterized by a 20 bit segmented memory address space (meaning that a maximum of 1 MiB of memory can be addressed), direct software access to BIOS routines and peripheral hardware, and no concept of memory protection or multitasking at the hardware level. In order to maintain backwards compatibility, the processor always starts in real mode. Therefore, we need to use a special mechanism to switch to an enhanced and more secured mode, called protected mode.</p>
<p>Protected mode adds the following features, used by most modern OSes:</p>
<ul>
<li><strong>32 bits address space</strong>: 4 GiB of memory can be addressed</li>
<li><strong>Paging</strong>: the OS could manage access rights of each page (4 KiB) of memory. This is also used for virtual memory.</li>
<li><strong>Privilege levels</strong>: 4 privilege levels, or rings, are used to restrict tasks from accessing data or executing several privileged CPU instructions.</li>
<li><strong>Multitasking</strong>: x86 processors introduced a mechanism of preemptive multitasking, but number of OSes don't use it.</li>
</ul>
<p>To switch into protected mode, we just have to set the Protection Enable (PE) bit in the Control Register 0 (CR0). But before doing that, we have to setup two other things.</p>
<p>For historical reasons, all the memory cannot be accessed without enabling a feature called the "A20 line" (see <a href="http://wiki.osdev.org/A20">this article</a> for more information). A traditional way to enable the A20 line is to program the keyboard controller.</p>
<pre>
cli ; no more interruptions
xor cx, cx
clear_buf:
in al, 64h ; get input from keyboard status port
test al, 02h ; test the buffer full flag
loopnz clear_buf ; loop until buffer is empty
mov al, 0D1h ; keyboard: write to output port
out 64h, al ; output command to keyboard
clear_buf2:
in al, 64h ; wait 'till buffer is empty again
test al, 02h
loopnz clear_buf2
mov al, 0dfh ; keyboard: set A20
out 60h, al ; send it to the keyboard controller
mov cx, 14h
wait_kbc: ; this is approx. a 25uS delay to wait
out 0edh, ax ; for the kb controler to execute our
loop wait_kbc ; command.
</pre>
<p>The second thing is due to the fact that, in protected mode, the memory management is controlled through tables of descriptors. A descriptor is basically a structure of flags and addresses used by the CPU to perform safety checks and controls access to memory. Each descriptor defines a memory segment with a base address, a size, a granularity value and several security flags. All these descriptors are grouped into the Global Descriptor Table (GDT). We have to set up this table and to inform the CPU to use it.</p>
<p>We're going to set up a GDT with 3 entries: a mandatory NULL entry, and 2 entries for code and data segment in ring 0 (Kernel). In the future, we will add 2 other entries for code and data segment in ring 3 (User mode).</p>
<p>Moreover, this GDT will describe a flat memory model: each segment will start at address 0 and will cover the full range of memory.</p>
<p>I will not describe here the format of these entries because a lot of web pages have already done this better than me (see <a href="http://wiki.osdev.org/GDT">this article</a> or <a href="http://www.jamesmolloy.co.uk/tutorial_html/4.-The%20GDT%20and%20IDT.html">this one</a>)</p>
<p>So, here are the GDT definition:</p>
<pre>
gdt:
dd 0 ; Null Segment
dd 0
dw 0FFFFh ; Code segment, read/execute, nonconforming
dw 0
db 0
db 10011010b
db 11001111b
db 0
dw 0FFFFh ; Data segment, read/write, expand down
dw 0
db 0
db 10010010b
db 11001111b
db 0
gdt_desc: ; The GDT descriptor
dw gdt_desc - gdt - 1 ; Limit (size)
dd gdt ; Address of the GDT
</pre>
<p>and the code to load this GDT</p>
<pre>
; load GDT
xor ax, ax
mov ds, ax ; Set DS-register to 0 - used by lgdt
lgdt [gdt_desc] ; Load the GDT descriptor
</pre>
<p>Once these operations are complete, we can switch into protected mode:</p>
<pre>
; enter pmode
mov eax, cr0 ; load the control register in
or al, 1 ; set bit 0: pmode bit
mov cr0, eax ; copy it back to the control register
</pre>
<p>The CPU now runs in protected mode. To finalize the switch, we just have to set its different segments registers: DS and SS are set explicitely to point to the third decriptor (<code>10h</code>), whereas the code segment CS is set during the <code>jmp</code> instruction. This code jump is also needed to flush the CPU instruction pipeline.</p>
<pre>
jmp 08h:protected_mode
[Bits 32]
protected_mode:
mov ax, 10h ; Save data segment identifier
mov ds, ax ; Move a valid data segment into the data segment register
mov ss, ax ; Move a valid data segment into the stack segment register
mov esp, 090000h ; Move the stack pointer to 090000h
</pre>Read data from diskurn:md5:8a9545a6bfdd3c1524571dcf65c567b12008-12-26T11:21:00+01:00cedrouBoot <h5><em>This post is related to <a href="http://code.google.com/p/genos/source/browse/?r=4#svn/trunk">rev.4</a>.</em></h5>
<p>In the previous post, we have created a small program that boots when the computer is switched on. The only thing it does is to write a welcome message on the screen.
Remember that this small bootloader was put in memory by the BIOS and that only 512 bytes were read from the boot device. This is a really tiny space to write a full OS ! Therefore, we will have to write some code that reads more data from data and puts it in memory.</p>
<p>To do that, we will invoke the BIOS interrupt 13h with several parameters passed through CPU registers.</p>
<p>You could get the full source <a href="http://code.google.com/p/genos/source/browse/trunk/src/boot/boot.asm?r=4">here</a> at Google Code.</p>
<p>The first thing to do is to create a new sector of data on the disk. This is easily done by adding some code after the <code>AA55</code> signature which mark the end of the first sector. We simply move the code that prints the message, along with the infinite loop and the string data that is printed. This new code will be located in the second sector of the boot disk.</p>
<p>We now return to the main procedure, the entry point. We'll modify it in such a way it will be able to read the second sector of the boot device and to place it in memory just after the current code.</p>
<p>We start by reset the drive, in order to put it in a known configuration:</p>
<pre>
mov ax, 00h ; Reset the floppy drive
mov dl, [boot_drive] ; Set the drive
int 13h
</pre>
<p>Then, we ask the BIOS to read the second sector of the boot drive and to place it at address <code>07C0:new_sector</code>, just after the current memory sector:</p>
<pre>
mov ax, cs ; write into 07C0:new_sector
mov es, ax ;
mov bx, new_sector ;
mov ah, 02h ; Read sector from drive
mov al, 1 ; Load 1 sector
mov ch, 0 ; Cylinder=0
mov cl, 2 ; Sector=2
mov dh, 0 ; Head=0
mov dl, [boot_drive] ; Bootable drive
int 13h
</pre>
<p>Cylinder/Head/Sector is a way to localize data chunks on a drive (see <a href="http://en.wikipedia.org/wiki/Cylinder-head-sector">Wikipedia</a> for more information).</p>
<p>The code is now placed in memory just after the current sector. All we have to do is to jump into this new code:</p>
<pre>
jmp new_sector ; Jump to the program
</pre>New project on Google Codeurn:md5:702bc7e1afd12bd19909defc3ab526b62008-12-20T23:53:00+01:00cedrou <p>I've just created a new project in Google Code to have a SVN repository and several other good stuffs (bug track, wiki, ...).
You can access it at this address: <a href="http://code.google.com/p/genos/" hreflang="en">http://code.google.com/p/genos/</a>.</p>
<p>The code will be always commited there with the BSD license and, from now on, each new note related to development of GenOS would point out a SVN revision, so you can get the related code.</p>First booturn:md5:e15608f9550d509d35e9f78145816a5d2008-12-18T01:22:00+01:00cedrouBootboot <p>Let start at the beginning: the very first thing an OS needs is to boot.</p>
<p>Our first job will be to write a boot sector that just prints a startup message and then loops forever.</p>
<p>To do this, we have to know how a computer starts.
When we power on a computer, the BIOS starts up, initializes internal data and sets up several hardware devices.
Then, it perform power-on self tests (POST) and display system settings on screen.
After this initialization phase, it looks at the sequence of bootable devices (floppy, CD, DVD, hard disk, USB stick, ...) into its configuration parameters.
It tries to initiate the boot sequence from the first device.
If the BIOS does not find a valid boot sector, it will try the next device in the list.
A valid boot sector is one that has the two bytes <code>0xAA 0x55</code> at offset 510.
If it does not find a proper boot sector on a device, the startup process will halt.
Once the BIOS has found the bootable device, it reads its full boot sector (512 bytes) and puts it into memory at linear address <code>7C00h</code>.
Then it jumps to that address and lets the bootstrap code get control.</p>
<p>At this point, the CPU runs in Real Mode <sup>[<a href="http://genos.blog.free.fr/index.php?post/2008/12/17/#pnote-11927-1" id="rev-pnote-11927-1">1</a>]</sup> and the content of its registers is undefined, except for DL which contains the BIOS number of the bootable device (<code>00h</code> for floppy, <code>80h</code> for first hard disk, ...)</p>
<p>You can download the sources of this bootstrap in the ZIP archive attached to this note. I don't have set up a real workspace for the moment, so the archive contains only several assembly sources and a Windows batch file used to build the boot sector. I'll write soon a new note to describe my workspace. The only thing you need for the moment is a recent version of <a href="http://www.nasm.us/" hreflang="en">NASM</a>. Moreover, you probably want to install a virtual machine software to avoid rebooting your PC too often. I'm using Virtual PC, but there are several others that are freely downloadable.</p>
<p>So let describe our boot loader.</p>
<p>The first thing we're doing is to set the data segment DS to the current segment (<code>07C0h</code>) so we won't have to add an <code>7C00h</code> to all our data</p>
<pre>
mov ax, 0x7c0
mov ds, ax
</pre>
<p>Then we set up the stack. We need this because we will use the call instruction, which uses the stack. We install this stack 2000h bytes after our boot sector to have enough room (~8 KiB).</p>
<pre>
mov ax, 0x9c0
mov ss, ax
mov sp, 0
</pre>
<p>The next step is to display the welcome message.</p>
<pre>
mov si, boot_message
call PrintMessage
</pre>
<p>The procedure <code>PrintMessage</code> reads each byte pointed by ds:si and invokes BIOS interrupt <code>10h</code> to print the character to the screen. The loop stop when a null character is read.</p>
<pre>
PrintMessage:
mov ah, 0eh ; 'put character' function
PM_loop:
lodsb ; load byte at ds:si into al
or al,al ; test if character is 0 (end)
jz PM_done
int 0x10 ; invoke BIOS interrupt
jmp PM_loop
PM_done:
ret
</pre>
<p>Finally, the CPU falls into a infinite loop which has the effect to hang the PC. I have inserted a <code>hlt</code> instruction inside the loop to halt the CPU until the next interrupt. Hence, the loop consumes very few CPU power.</p>
<div class="footnotes"><h4>Notes</h4>
<p>[<a href="http://genos.blog.free.fr/index.php?post/2008/12/17/#rev-pnote-11927-1" id="pnote-11927-1">1</a>] This theme (and a lot of others) will certainly be developped in a future note.</p></div>
Codename: Genesisurn:md5:865079fddaad71fd93fbfcad18a253302008-12-12T00:47:00+01:00cedrou <p>This is the first post of this new blog which aims to describe day after days the genesis of a new operating system.</p>
<p>To start, let me introduce myself. My name is Cedric Rousseau, I'm a 32 years old french developer and i've mainly coded since 10 years in C++ under Windows. I've started .NET and C# language in the first half 2006.
I live near Paris and until now, I didn't wrote a lot of documents in english, so please be indulgent - English is not my mother tongue!
I am married and have two (soon three) wonderful children. I work as software engineer for a start-up specialized in microelectronic. So the time that I reserve to work on this project will be probably reduced and not constant.</p>
<p>This project is really a challenge for me and it consists on creating a new operating system and describing in parallel what I am currently coding and all the choices that I am taking. Many developments will probably lead to a dead-end. Many other will start as "quick and dirty" code and will be enhance later. The aim is not to create a revolutionary OS to kill the good ol' Windows/OSX/Linux. Take it as an exercise. I only expect that this blog will help a lot of developers.</p>