1 Overview

The Bochs virtual PC consists of many pieces of hardware. At a bare minimum there are always a CPU, a PIT (Programmable Interval Timer), a PIC (Programmable Interrupt Controller), a DMA controller, some memory (this includes both RAM and BIOS ROMs), a video card (usually VGA), a keyboard port (also handles the mouse), an RTC with battery backed NVRAM, and some extra motherboard circuitry.

There might also be a NE2K ethernet card, a PCI controller, a Sound Blaster 16, an IDE controller (+ harddisks/CDROM), a SCSI controller (+ harddisks), a floppy controller, an APIC ..

There may also be more than one CPU.

Most of these pieces of hardware have their own C++ class - and if bochs is configured to have more than one piece of a type of hardware, each will have its own object.

The pieces of hardware communicates over a couple of buses with each other - some of the things that the buses carry are reads and writes in memory space, reads and writes in I/O space, interrupt requests, interrupt acknowledges, DMA requests, DMA acknowledges, and NMI request/acknowledge. How that is simulated is explained later FIXME.

Other important pieces of the puzzle are: the options object (reads/writes configuration files, can be written to and queried while bochs is running) and the GUI object. There are many different but compatible implementations of the GUI object, depending on whether you compile for X (Unix/Linux), Win32, Macintosh (two versions: one for Mac OS X and one for older OS's), BeOS, Amiga, etc.

And then there is the supporting cast: debugger, config menu, panic handler, disassembler, tracer, instrumentation.

2 Weird macros and other mysteries

Bochs has many macros with inscrutable names. One might even go as far as to say that bochs is macro infested.

Some of them are gross speed hacks, to cover up the slow speed that C++ causes. Others paper over differences between the simulated PC configurations.

Many of the macros exhibit the same problem as C++ does: too much stuff happens behind the programmer's back. More explicitness would be a big win.

2.1 static methods hack

C++ methods have an invisible parameter called the this pointer - otherwise the method wouldn't know which object to operate on. In many cases in Bochs, there will only ever be one object - so this flexibility is unnecessary. There is a hack that can be enabled by #defining BX_USE_CPU_SMF to 1 in config.h that makes most methods static, which means they have a "special relationship" with the class they are declared in but apart from that are normal C functions with no hidden parameters. Of course they still need access to the internals of an object, so the single object of their class has a globally visible name that these functions use. It is all hidden with macros.

Declaration of a class, from iodev/pic.h:

...
#if BX_USE_PIC_SMF
#  define BX_PIC_SMF  static
#  define BX_PIC_THIS bx_pic.
#else
#  define BX_PIC_SMF
#  define BX_PIC_THIS this->
#endif
...
class bx_pic_c : public logfunctions {

public:
  bx_pic_c(void);
  ~bx_pic_c(void);
  BX_PIC_SMF void   init(bx_devices_c *);
  BX_PIC_SMF void   lower_irq(unsigned irq_no);
  BX_PIC_SMF void   raise_irq(unsigned irq_no);
...
  };

extern bx_pic_c bx_pic;

And iodev/pic.cc:

...
bx_pic_c bx_pic;
#if BX_USE_PIC_SMF
#define this (&bx_pic)
#endif
...
  void
bx_pic_c::lower_irq(unsigned irq_no)
{
  if ((irq_no <= 7) && (BX_PIC_THIS s.master_pic.IRQ_line[irq_no])) {
    BX_DEBUG(("IRQ line %d now low", (unsigned) irq_no));
    BX_PIC_THIS s.master_pic.IRQ_line[irq_no] = 0;
    BX_PIC_THIS s.master_pic.irr &= ~(1 << irq_no);
    if ((BX_PIC_THIS s.master_pic.irr & ~BX_PIC_THIS s.master_pic.imr) == 0) {
      BX_SET_INTR(0);
      BX_PIC_THIS s.master_pic.INT = 0;
    }
  }
...
  }
}
...

Ugly, isn't it? If we use static methods, methods prefixed with BX_PIC_SMF are declared static and references to fields inside the object, which are prefixed with BX_PIC_THIS, will use the globally visible object, bx_pic. If we don't use static methods, BX_PIC_SMF evaluates to nothing and BX_PIC_THIS becomes this->. Making it evaluate to nothing would be a lot cleaner, but then the scoping rules would change slightly between the two bochs configurations, which would be a load of bugs just waiting to happen.

Some classes use BX_SMF, others have their own version of the macro, like BX_PIC_SMF above.

2.2 CPU and memory objects in UP/SMP configurations

The CPU class is a special case of the above: if bochs is simulating a uni- processor machine then there is obviously only one bx_cpu_c object and the static methods trick can be used. If, on the other hand, bochs is simulating an smp machine then we can't use the trick. The same seems to be true for memory: for some reason, we have a memory object for each CPU object. This might become relevant for NUMA machines, but they are not all that common -- and even the existing IA-32 NUMA machines bend over backwards to hide that fact: it should only be visible in slightly worse timing for non-local memory and non-local peripherals. Other than that, the memory map and device map presented to each CPU will be identical.

In a UP configuration, the CPU object is declared as bx_cpu. In an SMP configuration it will be an array of pointers to CPU objects (bx_cpu_array[]). For memory that would be bx_mem and bx_mem_array[], respectively.

Each CPU object contains a pointer to its associated memory object.

Access of a CPU object often goes through the BX_CPU(x) macro, which either ignores the parameter and evaluates to &bx_cpu, or evaluates to bx_cpu_array[n], so the result will always be a pointer. The same goes for BX_MEM(x).

If static methods are used then BX_CPU_THIS_PTR evaluates to BX_CPU(0)->. Ugly, isn't it?

2.3 BX_DEBUG/BX_INFO/BX_ERROR/BX_PANIC -- logging macros

go through a generic tracing mechanism. Can be switched individually on/off. Might eat a lot of CPU time - I think there are some BX_INFO calls for each instruction executed.

2.4 BX_TICK1, BX_TICKN(n), BX_TICK1_IF_SINGLE_PROCESSOR

BX_TICK1_IF_SINGLE_PROCESSOR, only used in cpu.cc -- and only confuses the matter. It calls BX_TICK1 on a single-processor and nothing on SMP.

3 CHECK_MAX_INSTRUCTIONS(count) - only needed on SMP configurations without debugger support. I am going to change the CPU emulation a lot (hopefully cleaning it up in the process), so I've decided to lose every SMP thing that gets in the way for me. This is one of them. Later, when UP works faster and better, I fully intend to restore SMP functionality -- or work with somebody else who does.

3.1 BX_SIM_ID

When using cosimulation it has something to do with which simulator that is executing? In any case, I removed it from my own source tree.

3.2 BX_HRQ, BX_RAISE_HLDA, BX_INTR, BX_SET_INTR(b), BX_IAC()

3.3 Various macros associated with dynamic translation

Relics of Kevin Lawton's initial attempts of using dynamic translation to IA-32 machine code instead of interpretive emulation. That development continued in Plex86, which seems to be more or less abandoned for the moment. Bochs will probably go in the direction of dynamic translation at some point in the future but for we will concentrate on better GUIs, better configuration, better hardware emulation and better support for reverse engineering. We would also very much like bochs to be faster but we will use simpler methods for the foreseeable future. These relics will be cut out as soon as possible.

3.4 Cosimulation support

For debugging changes in the CPU emulation, especially really big optimizations, Kevin Lawton invented something he called "cosimulation". The idea is to run two different CPU emulators in lock-step and constantly compare their CPU state. The idea is very good -- and has been independently discovered by many people for decades -- but is hard to put into practice. As Kevin Lawton wrote in some early docs: fixme: something about every time he uses cosimulation he has to hack on the code to make it work. I think the prudent thing would be to remove it for the time being -- and hack in specific hooks the next time somebody wants to use it. It should be maintained as a separate patch until we have found a cleaner way of doing it.

4 Memory - An Introduction

Both RAM and BIOS'es. BIOSes can be loaded individually. physical_read(), physical_write(). All address translation and access checking has already taken place in the CPU.

Some hardware interaction takes place through this object: VGA. This is unfortunately hardcoded into the memory object at the moment :(

5 The Basic CPU

Simple CPU: no caches! Does have TLBs. Some real IA-32 implementations distinguish between TLBs for code and for data -- we don't. We save some time on having 1024 TLB entries, which is a lot more than almost all real CPUs have at the moment. Different CPU levels -- level 5 is not complete, yet.

5.1 Some of the things we have to emulate - The IA-32

Real mode. Protected mode. 16-bit code, 32-bit code. segments, TLB, instruction prefetch queue, writes to memory can be executed "immediately" (makes things a bit harder for us later on), extraordinarily complex and varied instruction formats. Four different privilege levels, and then a system management mode on top of that for some of the CPUs. Six different segment registers (four on <386), capable of overriding the default segment register for the instruction, usually DS, but sometimes SS. Prefixes, address and operand size changes, tons of flags, tons of special cases about which registers can be used for what purpose. Totally free alignment of both code and data. Instructions can be one to sixteen bytes. IO Privilege level, IO privilege map, V86 mode.

Yada yada, you get the picture...

5.2 Some example instructions

(real mode, Intel syntax)
INC AX
MOV CX,[23+BX]
(protected mode, 32-bit default size, AT&T syntax)
<something with two size prefixes, a 0x0F prefix, a lock prefix? and a
complicated address.  LOCK ADD [BX*4 + CX*2 + DX + 1234], 17 ? >

5.3 Decoding instructions is *HARD*

On nicer processors, decoding instructions is an easy task. It's especially nice on the MIPS and the Alpha.

On IA-32 it's just about as lousy as it can get :/ In order to reduce the complexity a bit, all the decoding of the operand fields is done first, by BX_CPU_C::FetchDecode(), and then the instruction is executed by one of many hundred small methods that don't have to care (much) about their operands.

5.4 BxInstruction_t

b1
mod
r/m
rep_used
imm8, imm16, imm32
jmp
...
execute
resolvemodrm16
resolvemodrm32

5.5 The Main Loop - First cut

5.6 The Main Loop - Interrupts/Traps/Exceptions

5.7 The Main Loop - SMP

5.8 "Prefetching"

Should be called something else.

5.9 FetchDecode

5.10 Execute pointers

5.11 The Anatomy of Memory Accesses

segment : offset -> 32-bit linear -> 32-bit physical

segments, segment caches, base + limit, type

5.12 The Main Loop - Interrupts/Exceptions/Traps

5.13 So how was the prefetching in detail again?

prefetch
revalidate_prefetch_q
invalidate_prefetch_q

when is it invalidated?
when is it revalidated?
when do we actually have to do any of these?

5.14 Things I lied about

A20, extend down segments, FPU, synchronization between CPU and (potentially external) FPU. Reset of the CPU by forcing a triple-fault. debugger interface, config interface temporary disabling of interrupts (after SS changes) ;; might have to go below the following section

5.15 Flag handling

lazy flags, 5 32-bit ints to describe the operation. Some macros that evaluate the flags on demand.

5.16 How are exceptions implemented?

all instructions restartable from the register state + BxInstruction_t. Commit EIP + ESP (why that?) after successful execution of the whole instruction.

Never possible to generate exception /after/ changing the visible state.

longjmp(), setjmp()

5.17 What if we trip on an assertion?

Lots of checks all over the place. Also deep inside routines called by the cpu main loop. Die/cont/alwayscont/quit in Control Panel - or a debugger. How does it do that? Some variation on the exception scheme?

6 Specific tricks

6.1 4GB in real mode

What is the trick and how does bochs make sure that it works

6.2 Switching from protected mode to real mode

reset + cmos

6.3 Typical reset thru keyboard controller

6.4 Triple-fault reset

6.5 Fast reset gate

6.6 A20 change Should probably hitch a ride on the TLB paging mechanism for speed.

6.7 "CMOS" NMI gate

6.8 V86

6.9 V86 with virtual interrupt flag

6.10 APIC: IRQ rerouting to NMI

6.11 SMP: IPI (Inter-Processor Interrupt)

6.12 SMP: cache bounces

6.13 SMP: locked read-modify-writes

6.14 SMP: spinlocks

6.15 SMP: TSC potentially out of synch

6.16 SMP: BIOS and necessary tables

6.17 SMM - System Management Mode

Not implemented yet. Required for ACPI, I think.

6.18 Huge amounts of memory

Don't want bochs to push out other programs - handle swapping manually. BIOS and memory size reporting PAE Small window - big memory file Only need to swap in/out when TLB changes Keep memory on 4K boundary and use mmap() Needs > IA-32 machine (e.g. Alpha or some other 64-bit behemoth) or LFS.

6.19 PnP

6.20 PCI - configuration

6.21 PCI controller

7 Things that make you go "hmmm"...

7.1 16K pages between 0xC0000 and 0xFFFFF with PCI

7.2 Why read_RMW_virtual_(byte|word|dword)?

8 Optimization Ideas

8.1 Traces

 "Almost all programming can be viewed as an exercise in caching"
                                               -- Terje Mathisen

resolve16/32 can't be cached like this (example that uses registers to generate an effective address)

8.2 Squish out flags handling

BX_NEED_FLAGS, BX_SETS_FLAGS

8.3 How to be lazy with addresses

only retranslate seg:ofs -> linear -> physical when strictly necessary

8.4 Handle repeating instructions in bigger globs

special versions of access_linear()

8.5 split access_linear into read and write versions

8.6 combine segment limits with TLB pages

A bit that says if everything is ok or the address has to be reevaluated

8.7 Better branch prediction for execute ptr calls

switch (len) {
  case 7: i[len-7].execute(i[len-7]);
  case 6: i[len-6].execute(i[len-6]);
  ...
  case 0: i[len-0].execute(i[len-0]);
}

9 Communication Between Devices

9.1 Ticks and hardware emulation

The non-cpu hardware in the Bochs virtual PC needs to run some code once in a while to either do some real work, synchronize with the rest of the machine or interact with the host OS.

Timers, based on simulated instructions retired count. The GUI is made like this too -- that is probably a bad idea. BX_TICK1_IF_SINGLE_PROCESSOR()

Examples of worker functions: xxxxx.

9.2 Interrupts

9.3 DMA

HOLDA

9.4 IRQ pins

ISA IRQ2/9, IRQ3, IRQ4, IRQ5, IRQ6, IRQ7, IRQ...

PCI INTA, INTB, INTC, INTD - routing i PCI controller + on motherboard.

9.5 Interrupt routing

Level/edge triggered.

PIC

APIC

PCI controller

9.6 NMI

10 Communication between VGA and GUI

10.1 idle (HLT) and GUI

10.2 GUI and configuration

10.2.1 Floppy disk

/dev/fd0
A:
<path to disk file>
inserted/ejected
icon click -> set_status(inserted/injected)
how to check with ioctl

10.2.2 CD ROM

/dev/cdrom
disk change
How does El Torito work?
Only in BIOS?  What about hardware ATAPI?

11 Various Hardware

11.1 CPU

11.2 CPU - SMP

11.3 APIC

11.4 PIT

11.5 PIC - master/slave

11.6 Slowdown

11.7 Realtime PIC

11.8 RTC + CMOS

11.9 FPU FWait, exception handling, something about a weird exception + an IRQ reserved for the FPU.

11.10 Memory Some of the address range is handled by the i440 PCI chipset, which may subdivide it further.

11.11 i440 PCI chipset Also handles shadow ROMs.

11.12 AGP

11.13 DMA address bus sizes? built into PCI chipset? speed? limited to ISA bus speed? -- no. DMA happens as fast as devices want, provided the CPU allows it.

11.14 Floppy Controller

11.15 IDE

11.16 Harddisk

11.17 CDROM

11.18 Speaker

11.19 Sound Blaster

11.20 NE2K NIC

11.21 Mouse

11.22 Keyboard

11.23 Parallel port

11.24 Serial Port

11.25 USB

11.26 SCSI

11.27 IRQ in general

11.28 Ordinary BIOS

11.29 VGA BIOS

11.30 LBE both some BIOS calls and some "hardware"

12 How to register a new device

13 How to make snapshots

14 How to suspend/resume

15 How to make configurations easier

 #! ....
 bochs --help
 bochs -h
 bochs --version    // also prints compile options
 bochs -V (version)
 bochs <config filename>
 bochs -v  // tells us which config file is used + all the options read from
           // it.

16 Dreams for the future:

17 Error messages

Check that floppy/cdrom/disk are accessible with the current privileges and give the poor user some sensible error messages if not, INCLUDING examples of commands to fix the problem(s).

With bigmem support: check that 1) glibc supports LFS, 2) that the kernel supports it, 3) that the file system supports it, 4) that there is room enough in the designated directory.

18 Tools/links