The terminal, core utils, and shell, what's the difference?
The terminal, core utils, and shell, what's the difference?
I’ve been writing a lot of PowerShell recently, and one of the questions I’ve always had is where exactly the line between the terminal, the core utils, and the shell itself are. It all seems so tightly integrated and that they would all be useless without one another.
To answer this question, well start with the terminal. Back in the early days of computing, the only interface they had was a bunch of switches and indicators. But, as computers became more advanced, there was a need for a better way to interact with them.
Enter the Teletype (TTY). This innovation originally emerged as a replacement for the Telegraph, which necessitated understanding of Morse code for long-distance message transmission. The Teletype revolutionized this communication landscape, functioning much like an electromechanical typewriter. It seamlessly converted transmitted characters into printed text on paper.
The Teletype catered to the nascent programmers' interface requirements. Instead of sending messages across great distances to other Teletype machines, the Teletype was repurposed to facilitate direct communication with computers. These Teletypes could be situated within the same room as the computer or elsewhere within the building. Flexibility reigned as computers could be connected to one or multiple Teletypes. Over decades, Teletypes stood as the primary interface between individuals and computers, but inherent issues arose.
While revolutionary, they were always a retrofit of an existing solution for a new problem. This meant that they were not perfectly suited for their tasks. From the sustained expenses of ink and paper, their loud mechanical noises, and heaviness, they had their problems. These problems were exasperated when computers generated output for multiple Teletypes simultaneously. Additionally, maintenance proved to be a burden as all the mechanical components required regular maintenance.
This was until the Computer Terminal Corporation, later renamed Datapoint, announced the first glass Teletype. This innovation marked a departure from ink-based paper printing, instead displaying characters on a screen. As data flowed in, the lines would progressively ascend the screen before eventually fading away. While this configuration lacked a permanent data record, it often sufficed for numerous applications.
These terminals varied in sophistication. Some could solely display text and scroll data up the screen, while others accepted byte sequences for actions like screen clearing or cursor movement. While the most common configuration was a terminal connected to a computer through a data cable, there were some exceptions. In certain instances, computers integrated terminals within them, as all operating systems assumed terminal interfaces were available by default.
Transitioning from glass terminals, video terminals emerged as another breakthrough. Earlier terminals possessed limited RAM and lacked processing capabilities. Disconnecting the data cable meant text remained static on the screen, with no further interactivity. The arrival of Video Terminals brought considerable enhancement, exemplified by the renowned VT-100 (introduced in 1978), which incorporated an Intel 8080 processor. Offering advanced graphics, line editing features, and expanded character storage, the VT-100 utilized ANSI escape codes to manage cursor position, color, font style, and other parameters.
Today, the concept of integrated terminals in computers has evolved into terminal emulators. Unlike a physical terminal linked through a serial data cable, a terminal emulator manifests as a graphical window on your desktop. One of the oldest terminal emulators is Xterm. Xterm was started in the 1980s to provide DEC VT102 and Tektronix 4014 compatible support for programs that required a terminal to work properly. Today, many terminal emulators exist and are integral to a developer’s workflow. Some popular ones are Konsole, Alacrity, Cygwin, and Gnome Terminal.
Remarkably, due to their heritage and purpose, many characteristics of the early terminals endure even after decades. From their abbreviation in Linux documentation as "tty" (Teletype Writer), to the employment of ANSI escape characters for specific actions, these traits persist.
The behavior of Unix-style ttys hinges on their mode. By default, input operates in "cooked mode" with a provided discipline. The disciplines control the default behavior of the terminal emulator. The default discipline on Linux, line editing, furnishes basic command line editing capabilities. Users can input a line, possibly modifying it before pressing enter, after which the program processes it. This mode offers rudimentary input editing for simple C programs, as well.
Chapter 1 of the "Unix Programming Environment" by Kernighan and Rob Pike extensively covers terminal nuances, discussing how to activate certain behaviors and troubleshoot anomalies. How characters and lines were deleted was not standardized in early terminals. Because of this some terminals required typing the "@" symbol to discard a whole line before pressing enter. To "delete" a character, adding a "#" after it before pressing enter to submit the line was required.
Here is some text that I royally messed up@ -- Deletes whole line
You put the lin#me in the coconut -- # deletes the last word you typedContemporary systems usually manage these actions through shortcuts like cmd/ctrl + backspace and backspace characters, respectively, but those are modern shortcuts. For more complicated features, combinations of ansi escape characters are available. Now all of these are handled by the terminal emulator and the tty driver. It's clear that terminals have made substantial progress.
Besides Cooked mode, there is also Raw mode. It is particularly crucial for developing Text User Interface (TUI) based programs. The popular Rust tutorial for the hecto text editor (originally based on the c implementation called kilo, as it was under 1024 lines of code) extensively familiarizes users with raw mode. In raw mode, the tty driver forwards each character to the program as it's typed, a necessity for most TUI applications aiming to manage inputs themselves.
The terminal stands as a relic of the teletype interface that fundamentally transformed human to computer interaction. Now that we understand more about the terminal and what it is, let's shift our focus to the core utilities.
The Core Utils
In my article titled Tracing the Lines: From the Telephone to Unix, I delve into the historical evolution of Unix. Within this article, a specific section is particularly relevant to this discussion.
As Ken continued to tinker with Unix, he kept adding more and more features. One feature he added was redirecting IO. This allowed him to implement a feature first described by Douglas McIlroy, his direct supervisor, back in 1968, in his paper titled Mass Produced Software Components. This paper described what would eventually become the piping process that is used today with shell programs. After whipping up the first Unix shell from scratch, and with few modifications to existing Unix programs, he had a working prototype. He immediately showed Dennis Ritchie, who was blown away by the possibilities of these small composable programs. They both set to work porting the rest of the Unix tools so that their outputs could be piped in and out of each other. This led to the birth of many of the common core utils used in command line programs today like ls, cp, grep, and awk.
Ken Thompson, Dennis Ritchie, Brian Kernighan, and their colleagues were instrumental in crafting many of the utilities residing within /usr/bin on various Linux/mac/ and BSD distributions. However, the notion of linking these tools via the piping process emerged only after the principles outlined in Douglas McIlroy's paper, Mass Produced Software Components, were concretely implemented. When Richard Stallman set forth to develop a fully free and open-source operating system—emphasizing liberty— The GNU Core Utilities was an important component of this process. These utilities aimed to furnish rudimentary functions for file handling, shell operations, and text manipulation on Unix-like systems, which had often been proprietary and incompatible.
The GNU Core Utilities were originally separate components developed by David MacKenzie, who announced the GNU fileutils in 1990, the GNU shellutils in 1991, and the GNU textutils in 1991. They were reimplementation of their POSIX standard equivalents, licensed under the GPL, with extensions and improvements over the existing utilities. In 2002 these three projects were merged into what we now know as the Gnu Core Utils by Jim Meyering. The GNU Core Utilities are widely used and distributed in various Linux distributions, such as Debian, Ubuntu, Fedora, and Arch Linux. They are also available for other operating systems, such as Windows, macOS, FreeBSD, and Solaris. There are over 100 commands in the core utils, and not a work day goes by that I don’t use at least one.
When executed within the terminal, these utilities channel their output through standard input, standard error, and standard output. By default, these streams link to your terminal, though they can be reconfigured to target other destinations. So, our terminals are emulations of the dumb terminals that used to allow us to interface with old computers. The GNU Core Utilities are C programs that perform useful operations on our computer and whose output is redirected to our terminal, that leads us to our final component in this pipeline…
The Shell
At last, we arrive at the realm of the shell. A shell serves as a software program bridging the user and the operating system, acting as a conduit for users to input commands and execute them. It supports this both interactively and as a script. Beyond this fundamental role, shells boast diverse capabilities encompassing variable substitution, command substitution, redirection, piping, and more. A range of shell variants exist, encompassing the Bourne shell (sh), C shell (csh), Korn shell (ksh), and others, with Bash (Bourne again Shell) emerging as the default shell for the majority of Linux systems. These shells all trace their lineage back to the Thompson Shell (sh), authored by Ken Thompson in 1969 to complement his Unix operating system.
A fundamental component of a shell is shell scripting. By assembling a sequence of commands within a file, and subsequently executing them as a coherent program, useful scripts can be created. This scripting approach yields the power to automate tasks, orchestrate complex operations, and personalize your system. While the various shell types exhibit distinct syntaxes for constructing these scripts, they all share the common support for a shebang line, denoted as follows…
#!/bin/bash
Which allows users to run chmod +x on the script to create an executable version of it. And that covers it. The terminal, the core utils, and the shell. These three tools make up the command line workflow that provide so many developers with a fast and functional user interface to their computers.
Call To Action 📣
If you made it this far thanks for reading! If you are new welcome! I like to talk about technology, niche programming languages, AI, and low-level coding. I’ve recently started a Twitter and would love for you to check it out. I also have a Mastodon if that is more your jam. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine! Thank you for your valuable time.
