As I first started using Linux, the terminal was “just there”, it was a given. Need to run some commands? Open the terminal and type away.

Think about it for a second. You clicked on an icon or pressed some hotkey, and a new window appears on your screen: A familiar prompt, maybe preceded by a MOTD, and you can start processes, read their output, move around in the filesystem etc.

  • What just happened?
  • How does this new window know how to interpret a CTRL+C?
  • Where are you actually typing?

Glossary#

Lets start with a couple of definitions, in some circumstances these terms are used interchangeably but it is important to clarify the differences

terminal: Broad term used to describe the whole interface used to input commands
terminal emulator: Executable, like xterm, alacritty, ghostty or the GNOME terminal
TTY: TeleTYpe subsystem of the Linux kernel
tty (1): Executable, prints the file name of the controlling terminal to stdout, part of GNU coreutils
virtual TTY: Character device, usually living in /dev, eg:/dev/tty1
PTY: PseudoTerminal / PseudoTTY, are pairs of pseudodevices file, eg /dev/pts/1 (slave) and /dev/ptmx (master)
shell: Executable, command line interpreter. Takes input from a user and execute programs, scripts, manage files etc.
bash: One of the most used shells

A couple more important terms#

PID: Process ID
PPID: Parent Process ID (PID of the parent process)
syscall / system call: Interface between userland and kernel, more info here

Launching a terminal emulator#

Every Linux system with a GUI I ever used comes with a terminal emulator.
To run it, some interaction with the desktop environment is needed

On my system I can simply click the “Application Menu” and select “Terminal Emulator” to start the xfce4-terminal

Debian menu -> Terminal emulator

Mouse clicks#

Lets focus on those two clicks.
The desktop environment knows nothing about you hardware, it’s the kernel’s job (if the right drivers are available) to interpret the raw inputs from your mouse and generate events for processes in userland.

Those events are picked up by the Xorg server (a process running in userland) and dispatched to the appropriate application (X client) using the X11 protocol

Wayland: exists (modern alternative to Xorg/X11)

Launching a program#

The “Terminal Emulator” icon / menu entry is part of the xfce4 desktop environment, more precisely the xfce4-panel process: This is the X11 client alerted by Xorg when the mouse is clicked

On my system, the PID of xfce4-panel is 2441: 

$ ps -ef | grep xfce4-panel
fbtd       2441    2301  0 Aug24 ?        00:04:42 xfce4-panel
fbtd    1268092    3864  0 19:49 pts/8    00:00:00 grep --color=auto xfce4-panel

stracing#

Feel free to skip this section if you are not interested in the underlying syscalls to spawn new processes

The strace (1) utility can show us what is going on within the xface4-panel process as we click “Terminal Emulator”, in particular which syscalls are being invoked to spawn new processes.

Arguments to strace:

  • --follow-forks Also track child processes
  • -p 2441 Attach to the xfce4-panel process
  • -e trace=%process syscalls to trace: We are only interested in processes related syscalls

Some lines, for example failed syscalls, have been omitted for clarity 

 0$ strace --follow-forks -p 2441 -e trace=%process
 1strace: Process 2441 attached with 20 threads
 2[pid  2441] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 1276856 attached , child_tidptr=0x7fd51a017610) = 1276856
 3[pid 1276856] execve("/usr/bin/exo-open", ["exo-open", "--launch", "TerminalEmulator"], 0x55b808f4d5b0 /* 33 vars */) = 0
 4
 5[pid 1276856] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f565d059d10) = 1276857
 6[pid 1276857] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f565d059d10) = 1276858
 7[pid 1276858] execve("/usr/bin/xfce4-mime-helper", ["/usr/bin/xfce4-mime-helper", "--launch", "TerminalEmulator"], 0x5645e2842400 /* 33 vars */ <unfinished ...>
 8[pid 1276858] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLDstrace: Process 1276859 attached , child_tidptr=0x7f8253e19d90) = 1276859
 9[pid 1276859] execve("/usr/bin/x-terminal-emulator", ["/usr/bin/x-terminal-emulator"], 0x5601f78aa410 /* 33 vars */) = 0
10[pid 1276859] execve("/usr/bin/xfce4-terminal", ["xfce4-terminal"], 0x55cf733433b0 /* 33 vars */) = 0
11
12[pid 1276859] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fc604c54fd0) = 1276879
13[pid 1276879] execve("/bin/bash", ["bash"], 0x55ef77e99110 /* 36 vars */) = 0

Using the typical fork/exec technique, and a couple intermediary processes we end up with two new processes:

  • line 10: PID 1276859 as the terminal emulator itself (xfce4-terminal)
  • line 13: PID 1276879 as a shell (bash)

We can verify the PIDs with a couple of pscommands from within the new terminal window: 

$ ps -f -p "$$"
UID          PID    PPID  C STIME TTY          TIME CMD
fbtd     1276879 1276859  0 20:35 pts/5    00:00:00 bash

$ ps -f 1276859
UID          PID    PPID  C STIME TTY      STAT   TIME CMD
fbtd     1276859       1  1 20:35 ?        Sl     0:00 xfce4-terminal

The emulator and the shell#

Our new “Terminal” is made up of these two very important processes, lets see what are their roles

Terminal Emulator#

xfce4-terminal in our case

  • Emulate a physical terminal, like the VT100
  • Handle inputs (keyboard / mouse events) from the X server
  • Display output in its window
  • Interpret ANSI escape codes

Many fancy terminals provides more features, like a semi transparent background, tabs, clipboards etc

Shell#

It is called a shell, because it surrounds the kernel.
On my system the default shell is bash

  • Interpret user input and commands
  • Provide a CLI
  • Start processes by forking / exec-ing as needed
  • Handle working directory context
  • Offer some builtin commands to manipulate its own environment

IPC: InterProcess Communication#

There are many ways for processes to communicate between each other: pipes, sockets, shared memory, etc.
The main IPC between terminal emulator and shell takes place on a pseudoterminal (PTY). More precisely, a PTY pair:

  • Master, /dev/ptmx, created by the terminal emulator with a regular open (2)syscall
  • Slave, /dev/pts/5, obtained by calling ptsname(3) on the master. The shell opens it three times: as stdin, stdout and stderr
    we can verify which PTY the terminal is using by running tty without any arguments
  • The kernel takes care of transmitting data between master and slave

Here is the situation so far:

the situation so far

Legend:

legend

Pseudotreminals#

Why not use a simple pipe between terminal emulator and shell?
Here are some of the feature provided by pseudoterminals:

  • echoing: what you type on the master side ( when you use the write syscall) is echoed back ( when you use the read syscall), line 10 below
  • signal generation: interpret special character in order to generate signals for the shell, eg CTRL+Z sends the SIGSTOP signal to the shell, line 4 below (if using bash, the process can be resumed with fg)
  • line buffering
  • pauses and resume input/output

Line Discipline#

Line discipline is the kernel component that handles the communication between master and slave.
It can be configured at runtime with tools like stty (1), here is the list of settings:
line breaks added and some line omitted for clarity 

 0$ stty -a
 1speed 38400 baud; rows 24; columns 80; line = 0;
 2intr = ^C;
 3quit = ^\;
 4susp = ^Z;
 5...
 6isig
 7icanon
 8iexten
 9echo
10echoe
11...

Text editor and other TUI based programs also modify line discipline setting in order to supress echoing and signal handling.

Controlling Terminal#

Each PTY has a process who’s designated as its controlling terminal. Usually that’s the foreground process, for more informations look up the JOB CONTROL section of your favorite shell.
This will be the process receiving signals and inputs from the kernel (if line discipline is configured to send them)

Running a program#

Ok, we have a terminal emulator and its shell. Let’s do something useful with them by running the yes (1) program.

Keypresses#

As already discussed, pressing the Y E S keys triggers new events, picked up by Xorg and transmitted to the terminal emulator. The emulator calls write on the PTY master side and the shell read on the slave to get the character.

Readline#

What about backspace and other editing commands like CTRL + A to jump at the beginning of the line? Those are provided by the GNU Readline library that ships with the bash shell (and many others)

echoing#

bash won’t need to write out “yes” to the PTY slave, as the line discipline’e echo takes care of it

typing y e s

What should the shell run?#

We are almost there, as soon as we press ENTER bash (our shell) has to find out what yes actually is.
There are several possibilities, which bash checks one after the other:

  • alias: a shortcut for another command
  • built-in command: those are part of / implemented by the shell itself, like cd, pwd, etc
  • function: at least in bash, you can define your own function
  • executable: a file with execute permission that can be run by the system, like yes
  • keyword: stuff like while, for, etc implemented by the shell

Let’s see an example of each. The type builtin (line 5) tells us how each of the provided arguments (type, f, yes, ll and while) should be interpreted by bash.
We start by defining a function (f, line 1) and an alias (ll, line 4) ourself 

 1$ function f() { echo "you called f"; }
 2$ f
 3you called f
 4$ alias ll="ls -al"
 5$ type type f yes ll while
 6type is a shell builtin
 7f is a function
 8f ()
 9{
10    echo "you called f"
11}
12yes is /usr/bin/yes
13ll is aliased to 'ls -al'
14while is a shell keyword

$PATH#

So, yes is neither a builtin or a function, but how does bash knows that it is an executable?
The answer lies in the $PATH environment variable, which contains all the… paths… to check for executables, separated by a : 

$ echo $PATH
/home/fbtd/.local/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/fbtd/scripts
In our case yes is located in the /usr/bin folder

Fork / Exec#

Now our shell knows what to execure: it forks and execs the new process /usr/bin/yes, which inherits the PTY slave’s file descriptors and promptly floods it with y

running yes

stop it!#

The new process took over our terminal! Nothing a CTRL+C can’t fix.

sending SIGINT to yes
The process got a SIGINT signal and exits, giving back control to the shell:

bash is alone

That’s it! I hope you enjoyed this short journey in the world of terminals, see you next time!

Further readings#

The TTY demystified, by Linus Åkesson
TTY subsystem, Linux Kernel doc

***
Comments, suggestions or ideas? Let me know here