Part VI: Software — The operating system

How an operating system works#

What an operating system is#

An operating system (OS) is a program that runs on the CPU like any other program, but with special privileges. It is allowed to control the CPU’s instruction pointer, manage memory, configure hardware, and decide which other machine code is allowed to run and when. The OS is software that takes control of the machine and stays in control for as long as the computer or device is on.

The OS does not add new capabilities to the hardware, it only organizes and coordinates the capabilities that already exist.

The reason an OS is needed becomes clear as soon as we want to do multiple things simultaneously. Modern computers are shared machines: multiple programs need to run, memory must be divided and shared, hardware devices must be accessed safely, and long-running tasks must coexist with programs that react to clicks, keystrokes, and other user actions. The CPU alone cannot manage all this, as it only executes the instruction at the address it is given. The OS plays two complementary roles:

• Manager: The OS decides which program runs, how long it runs, how much memory it can use, and which hardware resources it may access.
• Mediator: The OS provides controlled interfaces that allow programs to interact with hardware and with each other. Programs do not talk to hard drives, keyboards, or screens directly, they ask the OS to do it on their behalf.

Programs, processes and concurrency#

A program is just data, stored on an SSD or a hard drive. A sequence of bytes that represent instructions and static information.

A process is a program in execution. More precisely, a process is a particular instance of a program that has been loaded into memory and is being managed by the OS. When we say that a program is “running”, what we really mean is that the OS has created a process for it and is allowing the CPU to execute its instructions.

From the CPU’s point of view, only one thing is happening at any given instant: The CPU has a single instruction pointer, and it executes one instruction after another, in strict sequence. The OS maintains the illusion that many programs are running at the same time by rapidly switching the CPU between different processes. This mechanism is called context switching. Because these switches happen thousands or millions of times per second, each process appears to make continuous progress.

In reality, the CPU is executing a small slice of instructions from one process, then moving on to another, and so on. What we perceive as simultaneous execution is the result of very fast alternation, not true parallelism (at least on a single CPU core).

In conclusion, a process is not just code. Two processes can be created from the same program and still be completely independent. This is why you can open the same application multiple times and have each instance behave as a separate entity.

Kernel and user modes#

An OS is not a single program that runs entirely under one set of rules. It’s a collection of code, some of which is meant to run with full control of the machine, and some other which is meant to run with more restrictions. Which rules apply at any given moment depends on the CPU’s execution mode, which can be kernel mode and user mode. When the CPU executes code in kernel mode, that code has full control over the system.

It can execute privileged instructions, access any memory, control hardware devices, and modify other critical parts of the system. On the other side, when the CPU executes code in user mode, that code is restricted and cannot directly access hardware, cannot touch memory outside its own assigned region, and cannot execute privileged instructions. This separation exists because unrestricted access is dangerous.

A single faulty or malicious program with full hardware access could cause problems like overwriting memory belonging to other programs, interfering with devices, or crashing the entire system. This separation limits what most code is allowed to do, isolates programs from each other and protects the overall system from errors.

The kernel is the part of the OS’s code that is intended to run in kernel mode. It’s called kernel because it’s the minimal, central core of the OS, and is trusted with full control of the machine. This includes scheduling which program runs next, managing memory, handling hardware, controlling devices, and enforcing security rules. Other parts of the OS run in user mode, just like the rest of non-OS ordinary programs and applications.

However, user-mode programs still need to perform privileged operations such as reading files, allocating memory, and interacting with hardware. Since they are not allowed to do these things directly, they must ask the kernel to do them. This is done through a mechanism known as system calls, where user programs request the kernel to execute privileged operations on their behalf, after verifying that they are allowed.

Interaction with hardware#

User programs do not interact directly with hardware. If any program could freely control the hard drive or screen, programs would interfere with each other: one program could overwrite another’s data, lock up devices, or take control of the machine. A shared system would quickly become unstable and unsafe. For this reason, direct hardware access is restricted to OS code. Programs can only request actions (“read this file”, “send this data”), and the OS decides if and how those actions are carried out.

Hardware devices communicate with the CPU using mechanisms called interrupts. An interrupt is a signal generated from a hardware device that temporarily transfers control to OS code. This allows the OS to react to external events such as a key being pressed or a disk operation completing.

To manage different kinds of hardware, operating systems use device drivers. A driver is OS code that knows how to control a specific device, so the rest of the system can interact with devices through these drivers, without needing to know specific hardware details.

In conclusion, hardware interaction is centralized in the OS so the machine remains controllable, shareable, and reliable, even while many programs are running.

How the OS starts running#

When a device is powered on, the CPU does not know what an OS is. It does not know about disks, files, or programs. All it can do is start executing machine code from a fixed, predefined location. That first code that gets executed is not the OS, it’s a very small program stored permanently in a dedicated non-volatile memory chip inside the computer, physically connected to the CPU. This program is called firmware. Firmware exists before RAM, disks, or the OS are usable.

Its purpose is limited and very specific: initialize basic hardware, locate the OS on persistent storage (SSD or hard drive), load it into memory (RAM), and transfer control to it.

In simplified terms, the starting sequence (often called bootstrapping) follows these steps:

1. The device is turned on, causing the CPU to reset to an initial state.
2. The CPU begins executing firmware code from a fixed address.
3. The firmware initializes basic hardware and looks for where the OS is stored.
4. A small program (bootloader) is loaded into memory (RAM), whose only job is to load the OS.
5. This bootloader program loads the OS kernel into memory.
6. The CPU is redirected to the kernel’s first instruction.

At that moment, the OS begins running like any other program: the CPU fetches its instructions from memory and executes them. The firmware and bootloader have done their job and are no longer involved. From there, the kernel sets up everything the firmware could not: memory management, process scheduling, device drivers, and the rules that govern all other programs.

Other responsibilities of the OS#

The OS also handles several responsibilities that are related to mechanisms described above:

• Scheduling: Deciding which process runs next and for how long.
• Time management: Using hardware timers to interrupt running programs, implement delays, and make sure the system stays responsive.
• Failure containment: Detecting crashes or illegal behavior, terminating faulty programs, and preventing failures from spreading to other processes.
• Resource accounting: Tracking and limiting the use of finite resources such as CPU time, memory, open files, and device access.
• Enforcement of policies: Applying system-wide rules about priorities, permissions, and limits consistently across all running programs.

See also: User space and kernel space

Operating systems in practice#

Types and families of operating systems#

There are distinct categories of operating systems, depending on the device they are designed to run on, and by how they are expected to be used.

Personal computers and servers#

Personal computers and servers are the most general-purpose computing devices. They are expected to run many unrelated programs at the same time, support multiple users, and interact with a wide variety of hardware. Common examples include:

• General-purpose desktop operating systems: Windows, macOS, desktop Linux distributions
• Server operating systems: Linux distributions, Windows Server, BSD variants

Mobile devices#

Mobile operating systems are also general-purpose systems, but they are optimized for battery-powered devices with touch-based screens and a wide range of integrated sensors (cameras, accelerometers, etc). Their design is primarily focused on energy efficiency and security. Common examples include Android and iOS.

Embedded and real-time systems#

These operating systems are designed for devices that perform a specific, predefined function, often under strict hardware constraints. These systems commonly run on limited memory and storage, and may lack features such as graphical user interfaces. These operating systems prioritize simplicity, reliability, and efficiency.

• Embedded systems: Found in devices like routers, washing machines, smart thermostats or industrial controllers. OS examples include embedded variants of Linux, Zephyr, VxWorks or even custom firmware without a full OS.
• Real-time systems (RTOS): For systems where timing is critical. Used in things like airbags, medical devices, robotics, avionics, automotive braking systems. Common examples include FreeRTOS, QNX and VxWorks.

History and modern operating systems#

Designing an OS means deciding how CPU time, memory, devices, and programs are managed, shared, and isolated. These same problems have existed since the earliest computers, even though the solutions have changed.

Origins and Unix#

Early operating systems (1950s–1960s) were minimal and ran one program at a time. As systems became multi-user and interactive, concepts such as processes, scheduling, memory protection, and file systems were introduced.

In 1969, Unix was created at Bell Labs. Unix introduced a simple, modular design, a hierarchical file system, and a clear separation between kernel and user programs, which influenced nearly all modern operating systems.

Unix descendants#

Linux (initially released in 1991) is a Unix-like kernel written mainly in C. It’s open source, uses a monolithic kernel design and is combined with other user-space tools to form complete systems. It runs on servers, desktops and embedded devices. Linux by itself is only the kernel. A usable system is created by combining the Linux kernel with user-space software, most notably tools from the GNU project.

These combinations are what people usually call Linux distributions, such as Ubuntu, Debian, Fedora, Arch Linux, and Red Hat Enterprise Linux.

macOS descends from Unix via NeXTSTEP. It combines a Unix user environment with a hybrid kernel and emphasizes hardware integration and security. It is written primarily in C, C++, and some assembly.

Android is built on the Linux kernel but it uses its own system components designed for mobile devices, and focused on power efficiency, app isolation and security.

Windows#

Windows follows a different origin. Modern Windows systems are based on the Windows NT architecture (1990s), designed for multitasking and portability. Windows uses a hybrid kernel and is written mostly in C and C++, with strong emphasis on backward compatibility.

In conclusion, most modern operating systems:

• Are written mainly in C, with assembly for low-level code
• Use C++ and increasingly Rust in some components
• Evolve incrementally rather than being rewritten
• Preserve compatibility with older systems, even at the cost of more internal complexity

Virtualization#

Virtualization is a technique that allows multiple independent computing environments to exist on the same physical machine. Each environment behaves as if it were running on its own separate system, even though all of these virtual environments are sharing the same hardware.

On personal computers and phones, there is usually a single OS running directly on the hardware. Virtualization is mainly used in servers and data centers, where a single physical machine is expected to run many independent workloads at the same time, being shared by many users, many applications, or even many different companies.

There are several reasons why virtualization is useful:

• Efficient use of hardware: Modern servers are powerful enough that a single workload often uses only a small fraction of the available CPU and memory. Instead of running ten small applications on ten separate servers, a company can run them all on one physical machine, each isolated from the others, using virtualization.
• Isolation and fault containment: Virtualized environments are isolated from each other, so errors, crashes, or misconfigurations stay contained. One system failing does not automatically affect the rest.
• Portability and management: A virtualized system is easy to duplicate, back up, or move between different hardware machines without reinstalling everything.

The two main common forms of virtualization are virtual machines (VMs) and containers:

A virtual machine (VM) is a complete, self-contained computer simulated in software. Each VM runs its own OS, including its own kernel. The OS inside the VM believes it’s alone in the machine. In reality, the access to the CPU, memory, and devices is controlled by another software layer called a hypervisor. The hypervisor controls which VM uses the CPU, memory and devices, and prevents VMs from interfering with each other.

They are useful in cases like when very strong isolation is required, or when different OS (e.g. Windows and Linux) need to run on the same physical hardware.

A container is a lightweight way of running an application in isolation, without simulating a whole computer. Containers do not run their own OS or kernel. Instead, they all share the host OS. The host OS controls CPU, memory, disk, and networking, and uses kernel features to keep containers separated and prevent them from interfering with each other. Because there is no extra OS per container, they start faster and use far fewer resources than VMs.

Containers are useful when you want to package an application with everything it needs to run, deploy many copies efficiently, and keep environments consistent across machines.

The layers above and below#

So far we have described how the machine runs and controls software internally. The next chapter explains how this internal activity becomes visible and interactive: how bits turn into pixels, sound, and responses to user actions. This next chapter crosses the boundary between the machine and the human using it.