Deus In Machina

AI is just the latest Monoculture

Diego Crespo — Thu, 05 Feb 2026 14:11:56 GMT

There’s growing anxiety that AI will be the death of innovation. The argument goes like this: LLMs are trained on existing code, so they recommend existing tools, so people use existing tools, so future LLMs are trained on even more of the same. New programming languages, libraries, and frameworks will face an impossible bootstrapping problem. How do you gain adoption when the coding assistant everyone uses has never seen your syntax?

It’s a reasonable concern. But I think it misses something important. LLMs aren’t introducing lock-in to software. They’re inheriting it. The feedback loop that people are worried about has been running for fifty years.

We optimize for familiarity, not quality

Look at the tools we celebrate as modern breakthroughs. Bun is a faster Node.js that runs the same code. Zig cc is llvm with better cross compilation. uv is a faster pip that installs the same packages. These are genuine improvements, and uv specifically has made my life as a Python developer significantly better. But notice what they have in common: none of them ask you to learn anything new. They compete on speed and reliability while carefully preserving the interfaces we already know.

This isn’t a criticism. It’s a survival strategy. Tools that demand new mental models face an uphill battle that most don’t survive. The path of least resistance is to make your new thing look exactly like the old thing. And that’s been true long before anyone was worried about training data.

The lock-in runs deep

The C ABI

Every language that wants to participate in systems programming eventually implements extern “C”. Rust does it. Go does it. Zig does it. Odin does it. Not because the C calling convention is well-designed, but because it’s the only calling convention that everything else understands.

The C ABI wasn’t created to be a universal standard. It just got there first. Now it constrains what languages can efficiently express at their boundaries. Features that don’t map cleanly onto C’s model, things like tagged unions, multiple return values, or non-nullable pointers, require awkward workarounds or simply get abandoned at the FFI layer. C++ carries decades of baggage specifically because it needed to interoperate with C. Until WebAssembly matures, there’s no real alternative. You speak C’s ABI, you spin up a webserver and communicate through REST or gRPC, or you don’t talk to anyone.

Curly braces everywhere

When Brendan Eich built JavaScript in ten days, management gave him an explicit instruction: make it look like Java. Not because Java’s syntax was suited to a dynamic scripting language, but because Java was familiar. And Java looked like C++ because C++ looked like C.

Follow the chain forward. Every mainstream language of the last thirty years uses curly braces for blocks. Java, JavaScript, C#, Go, Rust, Swift, Kotlin, TypeScript. The languages that tried something different, Python’s significant whitespace, Ruby’s end keywords, Lisp’s parentheses, either carved out niches or stayed marginal.

This pressure is so strong that the Racket team, when designing their new language Rhombus, deliberately abandoned s-expression syntax. From their documentation:

Rhombus is a new language that is built on Racket. It offers the same kind of language extensibility as Racket itself, but using traditional (infix) notation.

Even the people building a language specifically to showcase the power of Lisp-style macros felt they had to wrap it in familiar clothing. The syntax isn’t better. It’s just what people already know.

The x86/ARM/NVIDIA chokepoints

For decades, x86 dominated servers and desktops while ARM dominated mobile. Alternatives existed. MIPS, PowerPC, SPARC, Itanium. The weight of existing software, compilers, toolchains, and developer familiarity crushed all of them.
RISC-V is the current challenger, and look at how it’s positioning itself: as a drop-in replacement that slots into existing workflows, not as a fundamentally different way of thinking about processors. The pitch is “you won’t have to change anything.” That’s the only pitch that works.

The same dynamic plays out in GPUs. NVIDIA’s dominance isn’t just about hardware quality. It’s about CUDA. Try starting a new GPU company and see how far you get when every machine learning framework assumes CUDA exists. Intel is trying. It’s not going well. AMD has a chance, but ROCm has only recently started turning into a serious framework.

The browser is a bad operating system we’re stuck with

The web was designed for linked documents. JavaScript was bolted on as an afterthought. CSS was meant for styling academic papers. Somehow this accidental stack became the universal application platform.

Every attempt to replace it failed. Flash is dead. Silverlight is dead. Java applets are dead. Native app stores exist but haven’t displaced the web. So now we build complex applications by fighting against a document viewer, papering over its limitations with thousands of npm packages.

WebAssembly was supposed to change things. But notice what actually happened: it runs alongside JavaScript rather than replacing it. You still can’t escape the DOM. The gravitational pull of existing infrastructure is too strong.

So what about AI?

I could keep going. Git’s specific model of version control won and now everyone thinks in commits and branches. Unix file permissions from the 1970s still shape what’s possible on modern systems. QWERTY persists despite tradeoffs being made due to typewriter limitations.

The point is this: lock-in isn’t something that might happen if we’re not careful with AI. It’s the default state of computing. It’s what happens whenever switching costs accumulate faster than the benefits of switching.

Will LLMs accelerate this? Probably. A coding assistant that’s never seen your new language is a real barrier to adoption. But it’s a barrier stacked on top of dozens of existing barriers. The monoculture was already here. AI is just learning to speak it fluently.

Call To Action 📣

Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter, Mastodon, and Threads if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G

Subscribe now

Share Deus In Machina

An Amusing New Year's Eve

Diego Crespo — Thu, 01 Jan 2026 01:57:14 GMT

Photo by Ray Hennessy on Unsplash

Calling Raylib from PowerShell via P/Invoke

It was a quiet New Year’s for me, which caps off a pretty eventful year. I had some time off recently and have been cranking through some fun but challenging personal projects that I hope to share more about in 2026. In the meantime, I decided to blow off some steam with a silly project just to remind myself that programming can still be fun.

The Setup

PowerShell can call native C libraries directly through .NET’s P/Invoke mechanism. That means I can bind to raylib entirely from PowerShell. No C# project files, no compilation step, just a simple .ps1 script. Is this useful? Probably not, but that didn’t stop me.

I grabbed the latest version of raylib from GitHub (version 5.5), threw it into a folder called PowerShell, and created my test_raylib.ps1:

PS ~/Documents/Programming/PowerShell> (Get-ChildItem).name
raylib-5.5_linux_amd64
test_raylib.ps1
PS ~/Documents/Programming/PowerShell>

Since I’m on Fedora 43, I had to tell the dynamic linker where to find the library at the top of my script:

$env:LD_LIBRARY_PATH = "$PWD/raylib-5.5_linux_amd64/lib"

PowerShell’s Add-Type cmdlet can compile inline C# code, including P/Invoke declarations, so we can just do this to bind the library:

Add-Type -TypeDefinition @"
using System;
using System.Runtime.InteropServices;

[StructLayout(LayoutKind.Sequential)]
public struct Color {
    public byte r, g, b, a;
    public Color(byte r, byte g, byte b, byte a) {
        this.r = r; this.g = g; this.b = b; this.a = a;
    }
}

public static class Raylib {
    const string dll = "libraylib.so";

    [DllImport(dll)] public static extern void InitWindow(int width, int height, string title);
    [DllImport(dll)] public static extern void CloseWindow();
    [DllImport(dll)] public static extern bool WindowShouldClose();
    [DllImport(dll)] public static extern void BeginDrawing();
    [DllImport(dll)] public static extern void EndDrawing();
    [DllImport(dll)] public static extern void ClearBackground(Color color);
    [DllImport(dll)] public static extern void DrawCircle(int centerX, int centerY, float radius, Color color);
    [DllImport(dll)] public static extern void DrawRectangle(int posX, int posY, int width, int height, Color color);
    [DllImport(dll)] public static extern void DrawText(string text, int posX, int posY, int fontSize, Color color);
    [DllImport(dll)] public static extern int GetKeyPressed();
    [DllImport(dll)] public static extern bool IsKeyReleased(int key);
    [DllImport(dll)] public static extern void SetTargetFPS(int fps);
}
"@

Then we set up some variables:

# Colors
$red = [Color]::new(230, 41, 55, 255)
$green = [Color]::new(0, 228, 48, 255)
$raywhite = [Color]::new(245, 245, 245, 255)
$darkgray = [Color]::new(80, 80, 80, 255)

# Key codes
$KEY_A = 65
$KEY_D = 68
$KEY_LEFT = 263
$KEY_RIGHT = 262

# Game state
$playerX = 400
$ballX = 400
$ballY = 300
$ballDX = 3
$ballDY = 3
$score = 0
$movingLeft = $false
$movingRight = $false

And then call the functions from PowerShell:

# Initialize window
[Raylib]::InitWindow(800, 600, "PowerShell Pong-ish!")
[Raylib]::SetTargetFPS(60)

# Game loop
while (-not [Raylib]::WindowShouldClose()) {
    # We've got to do this manually
    $pressed = [Raylib]::GetKeyPressed()
    if ($pressed -eq $KEY_LEFT -or $pressed -eq $KEY_A) { $movingLeft = $true }
    if ($pressed -eq $KEY_RIGHT -or $pressed -eq $KEY_D) { $movingRight = $true }

    # Check for key releases
    if ([Raylib]::IsKeyReleased($KEY_LEFT) -or [Raylib]::IsKeyReleased($KEY_A)) { $movingLeft = $false }
    if ([Raylib]::IsKeyReleased($KEY_RIGHT) -or [Raylib]::IsKeyReleased($KEY_D)) { $movingRight = $false }

    # Apply movement
    if ($movingRight -and $playerX -lt 700) { $playerX += 6 }
    if ($movingLeft -and $playerX -gt 0) { $playerX -= 6 }

    # Ball movement
    $ballX += $ballDX
    $ballY += $ballDY

    # Ball collision with walls
    if ($ballX -le 10 -or $ballX -ge 790) { $ballDX = -$ballDX }
    if ($ballY -le 10) { $ballDY = -$ballDY }

    # Ball collision with paddle
    if ($ballY -ge 480 -and $ballY -le 500 -and $ballX -ge $playerX -and $ballX -le ($playerX + 100)) {
        $ballDY = -$ballDY
        $score++
    }

    # Ball fell off bottom - reset
    if ($ballY -gt 620) {
        $ballX = 400
        $ballY = 300
        $score = 0
    }

    # Draw
    [Raylib]::BeginDrawing()
    [Raylib]::ClearBackground($raywhite)
    [Raylib]::DrawRectangle($playerX, 500, 100, 20, $darkgray)
    [Raylib]::DrawCircle($ballX, $ballY, 10, $red)
    [Raylib]::DrawText("Score: $score", 10, 10, 20, $darkgray)
    [Raylib]::DrawText("A/D or Arrow keys to move", 10, 40, 16, $green)
    [Raylib]::EndDrawing()
}

[Raylib]::CloseWindow()

And that’s it!

Well, not quite. It took a few tries to get it right. And you’ll quickly see one of the many flaws with this approach if you try it yourself.

The Gotchas

When iterating on your P/Invoke definitions, you will run into errors like this:

Cannot add type. The type name 'Color' already exists.

Once Add-Type compiles and loads a type into the .NET runtime, it cannot be unloaded or redefined within the same PowerShell session. Your options are:

Restart PowerShell every time you change the type definitions
Rename your types during development (Color → RLColor2 → RLColor3... definitely not what I did)
Use namespaces with version numbers
Compile to a DLL once your bindings are stable

The other major challenge was on Fedora 43 with Wayland: IsKeyDown() always returned false, even though GetKeyPressed() worked fine. Who knows why this happens.

The solution was to track key state manually using the event-based functions:

# We've got to do this manually
$pressed = [Raylib]::GetKeyPressed()
if ($pressed -eq $KEY_LEFT -or $pressed -eq $KEY_A) { $movingLeft = $true }
if ($pressed -eq $KEY_RIGHT -or $pressed -eq $KEY_D) { $movingRight = $true }

# Check for key releases
if ([Raylib]::IsKeyReleased($KEY_LEFT) -or [Raylib]::IsKeyReleased($KEY_A)) { $movingLeft = $false }
if ([Raylib]::IsKeyReleased($KEY_RIGHT) -or [Raylib]::IsKeyReleased($KEY_D)) { $movingRight = $false }

Not something I’d want to do in a larger program, but it works for this small example.

And that’s all I’ve got for today. Happy New Year’s everyone!

Call To Action 📣

Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter Mastodon, and Threads if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G

Subscribe now

Share Deus In Machina

The Dart Programming Language Is Underrated

Thu, 05 Dec 2024 13:02:58 GMT

A few months ago, I started programming in Dart. I was drawn to it because of Flutter, and the frustrating experience I've had using other Desktop GUI libraries like GTK3, GTK4, FLTK, WxWidgets, AvaloniaUI, Raygui, QT, etc. In my time using the language it was easy to pick up, and I felt like it had a good amount of features for a modern OOP language. But that wasn't always the case, and Dart had to reinvent itself multiple time to get to where it is today. I'll go over some of the history of Dart, and then give some thoughts on my experience using it in this language review. If you'd just like to skip the history lesson and read my thoughts on using the language, jump to the section titled "My Experience with Dart so far".

Origins of Dart

In October 2011, Google unveiled Dart at the GOTO conference in Aarhus, Denmark. The language was developed by Lars Bak and Kasper Lund, both renowned for their work on virtual machines and language design. Dart's initial aim was ambitious... To address the shortcomings of JavaScript and provide a language that could scale from small scripts to large, and complex applications inside of the browser

At the time, web applications were becoming increasingly complex, and JavaScript, despite its ubiquity, was showing its limitations in terms of performance and maintainability for large codebases. Dart was designed to improve on JavaScript by offering optional static typing, a more advanced virtual machine, and improved ergonomics by having more expressive language constructs.

Early Years and Dart 1.x

The initial versions of Dart focused on being a superior language for web development. Developers could write Dart code and compile it to JavaScript using the Dart2js compiler, allowing it to run in any modern browser, but the ultimate goal was for all browsers to support the Dart VM directly, allowing developers to have another blessed language to program in the browser. Google went as far as getting Dart certified as an ECMA standard with ECMA-408, and created a custom fork of Chromium called "Dartium", which integrated the Dart VM directly. Despite these efforts, Dart struggled to gain traction in the web development community.

In an effort to strengthen Dart's position in web development, Google developed AngularDart, a port to Dart of the popular Angular web framework. AngularDart offered strong typing and performance benefits over its JavaScript counterpart and was used internally at Google for building complex web applications. However, the broader web development community continued to favor JavaScript, especially after the major updates to the language that came with ES6, and the rise of TypeScript, Microsoft's language that added optional static typing to JavaScript.

In 2015, Google introduced Flutter, a UI toolkit for building natively compiled applications for mobile, web, and desktop from a single codebase. Dart was chosen as the language for Flutter due to its familiar syntax, performant VM, and excellent tooling with tools like Observatory, a profiler, debugger, and introspection tool. Dart's Hot reload feature also enhanced developer productivity, allowing developers to see code changes in real time without restarting the application. This combination enabled the creation of a GUI toolkit with very fast development cycles, significantly improving developer experience.

Dart 2.0 and a New Direction

But JavaScript was advancing quickly and Dart was failing to gain traction. Recognizing the challenges in replacing JavaScript, Google shifted Dart's focus. In 2018, they released Dart 2.0 with a renewed mission... To optimize client side development for web and mobile platforms. This brought significant changes to the language. They ditched the optional typing and instead went for a sound type system. They removed legacy features and improved the language based on feedback from the Flutter Team, AdWords, AdSense and Dart community. With these changes Dart modified its focus to lean into its ability to compile into efficient JavaScript, its ability to modify programs as they ran through hot reload, and the safety of the language through its sound type system.

On the web front, Dart powered mission critical applications like Google AdWords and AdSense, significantly boosting developer productivity while maintaining performance and reliability. The Team emphasized Dart's importance to Google, noting how it was being used to power a major source of Google's revenue through AdWords. They also highlighted Dart's capabilities in the mobile space by showing off how the official Hamilton: The Musical app, was built using Flutter.

While this shift in focus definitely helped save Dart, it began to sow doubt in the community about whether Google would keep Dart around long term. Google already had Go, had made significant investments in Kotlin, and Dart had failed in its original mission in being the web's next programming language. Given Google's tendency to kill products people began to wonder if Dart was next on the chopping block.

Unfortunately Dart has never been able to shake this feeling even now, 6 years and another major release later. Whenever Google announces layoffs, or kills another product, people start speculating on Dart and Flutters future. The Dart team continues to have to calm the community every time this happens, despite its best efforts. The Dart website even displays a link to the roadmap

prominently at the very top of the page, as if to say "See, Google still has long term plans for this language". This Sword of Damocles over Dart's head hurts its adoption, and is one of the largest reason for people's hesitancy to adopt it as a technology, even though languages like TypeScript have been able to make significant inroads into the web community.

Dart 3.0

In 2023, after five years and numerous incremental releases, Google announced Dart 3.0. The largest update to Dart to date, coming with 3 headlining features. The first was 100% sound null safety which the team had been incrementally working towards for the last 6 years. The "sound part" was a big deal and meant that if a value was declared non nullable, it was guaranteed by the compiler. This along with sound type safety makes Dart a very safe language overall.

The second major feature was Records, Patterns, and Class Modifiers. This gave Dart some additional functional programming features, as well as an increase in its ability to create rich and expressive programs.

Finally, Dart 3 introduced support for compiling Dart code to WebAssembly, enabling the delivery of Dart apps as native code across all modern browsers. Previously, Dart's use of garbage collection hindered its compilation to Wasm, but collaboration with the WebAssembly community led to the development of the new WasmGC feature, now nearing stability in Chromium and Firefox. This allows Dart web apps to have improved load times, and enhanced performance.

My Experience with Dart so far

I've used Dart for 3 projects. A CLI that grabbed the profile sprite of every FREEDOOM and DOOM sprite, so I could compile them into a reference document, a library that parses DOOM wad files so I can understand how the binary format works (notice a trend?), and a File browser app that indexes files by content instead of filename allowing you to do full text search on a specified set of folders. While working on these projects I was exposed to a lot of Dart's interesting features, quirks, and drawbacks, and I learned some new things about programming along the way. Here's what I found

Dart is expressive, and very OOP

This one you'll either love or hate. In Python when writing platform dependent code I usually write it like this

import platform

current_platform = platform.system()

if current_platform == "Windows":
    print("Running on Windows!")
elif current_platform == "Linux":
    print("Running on Linux!")
elif current_platform == "Darwin":  # macOS
    print("Running on macOS!")
else:
    print(f"Unknown platform: {current_platform}")

In Dart you use a class called Platform, but instead of doing string comparisons you have specifically defined boolean values like isLinux or isMacoOs. You also use things like Platform.environment['HOME'] to get the home directory whereas in Python you'd do os.path.expanduser("~"). Here is an example in some of my code

  String findConfig() {
    String configDir;
    if (Platform.isLinux) {
      configDir =
          join(Platform.environment['HOME']!, '.local', 'share', 'TerboMine');
    } else if (Platform.isWindows) {
      configDir = join(Platform.environment['UserProfile']!, 'AppData', 'Local',
          'TerboMine');
    } else if (Platform.isMacOS) {
      configDir = join(Platform.environment['HOME']!, 'Library',
          'Application Support', 'TerboMine');
    } else {
      throw UnsupportedError('Unsupported platform');
    }
    return configDir;
  }

You work with objects a lot instead of raw strings, which leads to chaining code that looks like this

import 'dart:io';

void main() {
  final directory = Directory('/usr/Me/Documents');

  final textFiles = directory.listSync()
      .whereType()
      .where((file) => file.path.endsWith('.txt'))
      .toList();

  print(textFiles);
}

This makes the language feel like Ruby, Crystal, or even Pharo, which I like because I like those languages, but I understand that is subjective.

One draw back of being OOP means that there are a lot of different modifiers you can add to a classes which change the behavior, and it can get very confusing when a class has many keywords associated with it. This leads me to my next point

There are a lot of keywords

This unfortunately comes with the territory of most statically typed programming languages, but when classes support

abstract
base
final
interface
mixin
sealed

and variables can be const, late, or final, I find myself type masturbating as I try to craft the perfect type hierarchies. It also clutters the code visually, and I end of making liberal use of var. In total their are 61 plus or minus a few keywords listed in the Dart reference. As a library author, this is great, and as a consumer of a library the intellisense I get is very informative. But that is double what Python, JavaScript, and Go have, which can be a little overwhelming.

I feel like Dart knows this is an issue, because they try to limit the noise where they can, like how private variables identified by being prefixed with a _. It also supports the typedef keyword allowing you to alias types for when they get too long

typedef IntList = List;
IntList il = [1, 2, 3];

Dart Supports Metadata

Dart supports metadata, which are like Python decorators

class Television {
  /// Use [turnOn] to turn the power on instead.
  @Deprecated('Use turnOn instead')
  void activate() {
    turnOn();
  }

  /// Turns the TV's power on.
  void turnOn() {...}
  // ···
}

One thing I found cool is that you can use this to specify little TODO annotations like in this example

class Todo {
  final String who;
  final String what;

  const Todo(this.who, this.what);
}

@Todo('Dash', 'Implement this function')
void doSomething() {
  print('Do something');
}

It's very asynchronous

I'm not used to programming in languages like JavaScript so this was a relatively new concept for me. Dart supports Async Await, so you'll see methods that are explicitly blocking having Sync in the name like

final fileDate = file.lastModifiedSync().toIso8601String();

Or you'll write a function and all of a sudden you are returning a List wrapped in a future Future>> as in this code

  // Search documents in the database using raw SQL
  Future>> searchDocuments(String query) async {
    if (query.isEmpty) {
      // Return an empty list if the query is empty
      return [];
    }

    final db = await instance.database;
    final escapedQuery = _escapeSpecialCharacters(query);
    return await db.rawQuery('''
      SELECT rowid, name, location FROM $tableDocuments WHERE $tableDocuments MATCH ?
    ''', [escapedQuery]);
  }

but that beats having to litter the code with then everywhere so I'll take it

instance.database.then((db) {
    final escapedQuery = _escapeSpecialCharacters(query);
    db.rawQuery('''
      SELECT rowid, name, location FROM $tableDocuments WHERE $tableDocuments MATCH ?
    ''', [escapedQuery]).then((results) {
      callback(results);
    }).catchError((error) {
      // Handle errors here
      print(error);
    });
  });

Since Flutter is Dart's killer app, it makes sense that this type of code pops up everywhere as asynchronous code allows you to create non blocking functions that would otherwise slow down your app if they were synchronous. I believe this is one of the reasons Dart bills itself as a client optimized programming language, a term I've only ever heard used in the context of Dart.

The tooling is excellent

Dart allows you to try before you buy with Dartpad.dev. There is even sample flutter code so you can see what a simple Flutter app looks like in the browser.

It has extensions for VS Code and Android Studio, as well as an Emacs Major mode, so editor support is good. The Dart cli comes with a lot of different commands allowing you to generate documentation from docstrings, format the code, run tests etc.

A command-line utility for Dart development.

Usage: Dart  [arguments]

Global options:
-v, --verbose               Show additional command output.
    --version               Print the Dart SDK version.
    --enable-analytics      Enable analytics.
    --disable-analytics     Disable analytics.
    --suppress-analytics    Disallow analytics for this `Dart *` run without changing the analytics configuration.
-h, --help                  Print this usage information.

Available commands:
  analyze    Analyze Dart code in a directory.
  compile    Compile Dart to various formats.
  create     Create a new Dart project.
  devtools   Open DevTools (optionally connecting to an existing application).
  doc        Generate API documentation for Dart projects.
  fix        Apply automated fixes to Dart source code.
  format     Idiomatically format Dart source code.
  info       Show diagnostic information about the installed tooling.
  pub        Work with packages.
  run        Run a Dart program.
  test       Run tests for a project.

Dart also supports creating projects using the create command

Dart create --template console-application my_app

Which will generate a folder that typically includes...

pubspec.yaml: Defines your project's dependencies and metadata.
pubspec.lock: Locks the versions of your dependencies.
analysis.yaml: Configuration file to customize the behavior of the static analysis tool.
lib/main.Dart: Main entry point of your application.
test/: Contains test files for your project.

You can generate executables with dart compile and it has a package manager and a website for browsing packages called pub.dev where I've been able to find pretty much any library I need. This is good because of my next point...

The Standard Library is kind of small

Because Dart has a large mobile and web focus, It has to be careful about features that could mess with the highly sandboxed nature of Android and IOS. This means the standard library that comes with a vanilla install only covers these core areas

Dart:core
Built-in types, collections, and other core functionality. This library is automatically imported into every Dart program.
Dart:async
Support for asynchronous programming, with classes such as Future and Stream.
Dart:math
Mathematical constants and functions, plus a random number generator.
Dart:convert
Encoders and decoders for converting between different data representations, including JSON and UTF-8.
Dart:io
I/O for programs that can use the Dart VM, including Flutter apps, servers, and command-line scripts.
Dart:html
DOM and other APIs for browser-based apps. We now recommend using package:web over Dart:html.

To get around this, you'll notice when browsing pub.dev that the Dart Team publishes under three different publishers

Dart.dev - Core packages. Foundational packages that complement the core libraries.
tools.Dart.dev - Tooling packages. Used by the Dart team to build various Dart tools. Can be used for building other tools, but the support commitment is lower than for core packages.
labs.Dart.dev - Experimental packages. Shared for early feedback. Some will likely be discontinued.

Many of the additional packages I needed where available from the Dart.dev, and the tools.Dart.dev publishers. But it seemed a little silly that I had to download the yaml package which is maintained by tools.Dart.dev, considering that every Dart project includes a yaml file. On the other hand, a smaller standard library meanings easier porting to different platforms, and smaller programs overall, so it is a trade off.

One challenge with Dart running on so many platforms is that you have to be careful with using certain features in the standard library or the ecosystem at large. For example... In the dart io library you'll see the message

/// **Important:** Browser-based apps can't use this library.
/// Only the following can import and use the dart:io library:
///   - Servers
///   - Command-line scripts
///   - Flutter mobile apps
///   - Flutter desktop apps

There are also certain libraries on pub.dev which don't work with Flutter, Dart's killer app, and these tend to be less popular.

Dart code is very Tall

Dart compared to other languages I've used, loves its vertical space. It loves its trailing commas (I've become pro trailing comma), and will line break on every one, which leads to code like this in one of my apps

        "validExtensions": [
          ".txt",
          ".md",
          ".rst",
          ".yaml",
          ".yml",
          ".json",
          ".xml",
          ".csv",
          ".tsv",
          ".c",
          ".cpp",
          ".h",
          ".hpp",
          ".java",
          ".class",
          ".py",
          ...]

This is definitely influenced by Flutter, which has a declarative nested syntax, but I don't consider that a bad thing, because the language is 'client optimzied' and the minute you start highlighting widgets and using the Wrap with padding, Extract Widget, Extract Method etc, you start seeing and editing code as structured chunks which is hard to explain if you've never experienced it.

Concurrency in Dart

Dart provides robust support for concurrent programming using Async Await patterns, and Isolates. Isolates allow you to create independent units of execution that can run concurrently on multi core processors. This enables efficient handling of computationally intensive tasks without blocking the UI thread.

Isolate Isolate = await Isolate.spawn(heavyComputation, data);
Isolate.receivePort.listen((message) {
  // Process the result from the Isolate
});

Isolates offer several advantages over traditional thread-based parallelism including...

Memory Safety: Isolates have their own memory space, preventing them from interfering with each other. This eliminates the risk of data races and other concurrency related bugs that are common in thread based systems.
Simplified Concurrency: Dart's Isolate model simplifies concurrent programming by providing a more intuitive and less error prone approach. You can spawn Isolates to handle independent tasks while enjoying a higher level of abstraction.
Efficient Resource Utilization: Isolates can be efficiently scheduled on multiple cores, allowing you to take full advantage of modern hardware.
Graceful Error Handling: If an Isolate crashes, it won't affect the rest of your application. This makes your application more resilient to errors and failures.

Isolates are no silver bullet though as they do come with some downsides. There is overhead that comes with spawning Isolates, so you have to weigh the startup cost of spawning an Isolate vs just letting the task complete synchronously. There is also overhead when passing messages in Isolates when compared to direct memory access in traditional threads, whichs makes them less efficient. Finally, Isolates can cause weird behavior in some libraries on certain platforms, like when trying to use the popular graphics framework Raylib on macos

Still, Isolates are a good solution when you still want to have parallelism but want to avoid some of the pitfalls associated with it. Given Dart's focus on frontend development, incurring the performance hit for safer parallelism seems like a good tradeoff. And the fact that Isolates make it easier than traditional threading make people more inclined to actually parallelism where they can.

Dart Does Tree Shaking but there is an asterisk

Tree shaking, AKA Dead code elimination, results in smaller code sizes due to only including code in your app that you actually use. This is an awesome feature, and not one that is common in programming languages. Unfortunately this is also under documented in Dart, and it is unclear what is and isn't tree shakeable. This has been raised in this Github Issue, but there has been no resolution to it yet.

It has a large Medium community

This is a interesting one. There are a lot of Dart articles on Medium. I think this is because both Dart and Flutter announcements are syndicated to Medium like the annoucent of Dart version 3.5 and have been for a long time.

There are a lot of Materials about Flutter

Dart can be used for both backend and frontend development, but Flutter is king. This does come with a huge benefit though, pretty much everyone in the Dart community knows Flutter, and the amount high quality tutorials teaching it are numerous compared to the size of its community

This is very different from the Python world where I live that has many people using different subsets of the language. For instance Data Scientists use Pandas, NumPy and Matplotlib, ML People use TensorFlow, Scikit-learn, and Keras, and web developers using use Django or Flask. The drawback to this is that if you aren't doing anything Flutter based you are kind of on your own, but I believe with libraries like Jaspr, Dart Frog, ml_algo, and many others on awesome-Dart that is changing.

It's pretty fast

1 billion loop challenge

Benchmark Games

Micro benchmarks aren't entirely representative of a programming language’s performance. But people love them as seen by Twitter’s viral billion loop challenge, and the success of the billion row challenge last year.

With that being said Dart is fast for a garbage collected language. When your virtual machine was created by the same guy who made the V8 JavaScript engine (Lars bak), and the language was designed to be easier to optimize than JavaScript, you end up with a language that is pretty quick.

It has a cool Mascot (Objective)

As is tradition, Dart has an animal mascot and her name is Dash. She is a Hummingbird and there is a website dedicated to creating your own version of a Dashatar.

TLDR/Conclusion

Dart is an interesting language that is in a spot most programming languages would kill to be in. It has corporate investment and a team of people who are paid to develop it. It has a killer app through Flutter that has large total addressable market (anywhere a GUI can run).

It is cross platform across all 3 major desktops, including mobile which many programming languages struggle to target. It is capable of compiling an exe with dart compile exe bin/main.dart, and compiling to the browser with Wasm and JavaScript. And its foundation is built on cutting edge programming language theory. Yet it has been overlooked compared to other languages that have come out in a similar time frame namely TypeScript (2012), Kotlin (2011), Go (2009), Rust (2010), and Swift (2014).

While the language is not perfect, I hope you might consider given the language a shot in the future (maybe in this years advent of code?), and I will continue to use it in my personal projects going forward. If you are interested in more Dart content, I have a Flutter article coming soon, where I will speak to my experience learning and using it. Stay tuned!

Call To Action 📣

Subscribe now

Share Deus In Machina

The Story of Nel

Diego Crespo — Fri, 29 Nov 2024 03:17:59 GMT

Happy 🦃 Day! I hope everyone has a wonderful day

Real Programmers code in the terminal.

Maybe they dabble in slick IDEs now, in this decadent era of AI assistants, auto completion, and "intelligent" debuggers. But back in the good old days, when the term "cloud" referred to the weather and not where you stored your code, Real Programmers wrote in plain text editors. Not Visual Studio Code. Not even Helix or Fleet. Just GNU Emacs and Screen.

Lest a whole new generation of developers grow up in ignorance of this glorious past, I feel duty bound to describe, as best I can through the generation gap, how a Real Programmer writes code. I'll call her Nel, because that's her name.

I first met Nel when I joined a startup aiming to disrupt the world with yet another social app. The team was young, vibrant, and glued to their screens, eyes darting between Stack Overflow tabs and AI generated code snippets. Then there was Nel, a quiet figure hunched over a terminal window, typing furiously.

Nel didn't approve of AI assistants.

"If a programmer can't write code without an autocomplete," she said, "what good are they?"

Nel had written, using nothing but Emacs and her wits, the most critical module our company had. The real time data processor that kept everything running smoothly. It was a marvel. While the rest of us wrestled with dependency hell and cursed the day JavaScript was invented, Nel's code hummed along with these things called static types that, she said, were all the documentation she needed.

One day, disaster struck. An outage in us-east-1 left us disconnected from our beloved cloud services. Panic ensued. Without Stack Overflow, AI help, or even our code repositories, we were paralyzed. Features needed fixing, bugs needed squashing, and deadlines loomed ominously.

All except Nel were in despair.

"How can we code without the internet?" someone wailed.

"I've reached my 2-hour monthly in office quota anyway. Message me on Microsoft Teams when everything is back up," said another.

Nel looked up from her terminal, a faint smile playing on her lips.

"Maybe we can, you know, stay here and write code," she suggested.

"Without documentation? Without Google? Impossible!"

Nel shrugged and returned to her screen. Curiosity got the better of me, and I wandered over to see what she was doing. Lines of code scrolled past at an impossible speed. Functions, classes, even the occasional comment, all appearing as if by magic. She wasn't even using a mouse or trackpad.

"Are you... writing a new feature?" I asked, incredulous.

"Fixing the data parser," she replied without looking up.

"From memory?"

She nodded.

"But how do you remember the syntax? The APIs? Don't you need to look up anything?"

Nel chuckled softly. "I've been coding in this language longer than most of you have been alive. Muscle memory, I suppose."

I watched in awe as she deftly wrote a for loop, complete with iterator variables and proper scope handling, all without a single reference. It was as if I'd witnessed a unicorn.

"Could you help me with a for loop?" I asked sheepishly.

She paused and turned to face me fully for the first time.

"Sure," she said, gesturing for me to sit. "What are you trying to do?"

I explained the problem, expecting her to whip out her phone to search for an answer. Instead, she grabbed a notepad and sketched out the logic, walking me through each step.

"See? No need for the internet. Just think through the problem," she said.

She laughed as I gawked.

"I used to have to do stuff like this on whiteboards for interviews," She smiled while shaking her head.

Word spread through the office. One by one, team members approached Nel, seeking guidance. She became an impromptu mentor, her desk a shrine to the lost art of independent thinking.

Meanwhile, the managers were in a frenzy. Deadlines were at risk! Productivity was plummeting! They called an emergency all hands.

"We need solutions, people!" the CTO barked. "How can we work without our tools?"

Nel stood up.

"Perhaps we've become too reliant on them," she said calmly. "Maybe it's time we trusted our own abilities."

Silence fell. The idea was radical, almost heretical.

"But how will we manage without AI suggestions? Without code linters? Without..."

"By programming," Nel reiterated, "by understanding our code, not just piecing together fragments from the internet."

An idea sparked in Nel. "We could deploy the app on premises", "Mark has a 64-core 7.5GHz laptop with 256GB of Ram and two 10TB NVME drives. If we turned off all of the startup and electron apps we'd be able to run the entire thing locally."

Eyes turned to Mark, who nodded slowly. "It just might be possible. We have as many employees here as we have users so it would be a good test of the app"

The CTO considered this. "Can we set it up quickly?"

"Absolutely," Nel interjected. "It might even run better than in the cloud."

Emboldened, we set up the on premise deployment. With Nel's guidance, the process was smooth. We bypassed the cloud entirely, running everything locally.

Her words stirred something in us. Emboldened, we returned to our desks. The hum of conversation was replaced by the clatter of keyboards. Slowly, code began to take shape...not copied, not generated, but crafted. Whenever we had issues, we would physically walk over and discuss them with our teammates. Questions that would be left unread for hours before a response was finally given, were answered in a matter of seconds.

By the time the internet was restored, it had been 6 hours, the most I've ever worked without distractedly scrolling through short form video. We had accomplished more in those disconnected hours than we had in months.

The CTO called another all hands, praising the team's resilience. They gushed about how much VC money we would be able to raise from all the featuers we had developed. But we knew who deserved the real credit.

Nel's approach had sparked a revival of genius level programming. Developers started challenging themselves to code without immediately resorting to Google. Pair programming sessions became the norm, fostering collaboration and learning. People even began reading manuals. Not everyone could hack it though, but they eventually found their way into middle management positions.

I asked Nel one day why she preferred Emacs over the myriad of sophisticated tools available.

"Because it does exactly what I need...no more, no less," she replied. "It's just me and the code."

I haven't kept in touch with Nel since she moved on to greener pastures...or perhaps terminals. But I like to think she's out there somewhere, still coding in Emacs, still eschewing the trappings of modern development.

In any event, I was impressed enough that I've started relying less on AI assistants and more on my own understanding. And while I may never reach Nel's level of mastery, I still use VS Code after all, I did download the Vim Emulation plugin. I think I might do a lunch and learn once I finally master hjkl.

For those who don’t know this is a parody of The brilliant Story of Mel if you’ve never read it I highly recommend it!

Call To Action 📣

Subscribe now

Share Deus In Machina

Coding without braces: An alternate C Syntax

Diego Crespo — Thu, 10 Oct 2024 12:02:53 GMT

Photo by Lorenzo Herrera on Unsplash

I recently stumbled upon an interesting piece of C language trivia that underscores how different computers were when C was first created. Specifically, C allows the use of digraphs (two letter combinations that represent a single letter) like <% and %> as alternatives of { and } and <: and :> as substitutes for [ and ]. The purpose of this was to provide an alternate syntax to support keyboards and character sets where braces and brackets were not available. The Gnu C Language manual mentions these digraphs as well as %: as an alternative to #. Since nothing on the internet can be trusted anymore I wrote a simple program and compiled it to verify the claims.

#include 

int main() <%
    printf("Hello, World!\n");
    int array<:5:> = <%1, 2, 3, 4, 5%>; 
    for (int i = 0; i < 5; i++) <%
        printf("%d\n", array<:i:>); 
    %>
    return 0;
%>

And lo and behold (don't take my word for it) it worked! While it's hard to imagine a keyboard without these symbols today, and I might be showing my ignorance of international keyboards here, I'm glad someone thought to include these alternatives.

But if the cpp reference is anything to go by there is is an expecation that all keyboards must have ?,< and ) as there is an even more verbose version of this behavior in the form of trigraphs, which allow a three character sequence alternative instead of two. Examples include ??< for { and ??) for ] but there are many more which can be viewed in the cpp reference. While tripgrahs appear to be going away in C23, I have no doubt people have abused these as well as digraphs in code obfuscation competitions.

Call To Action 📣

Subscribe now

Share Deus In Machina

Changes to My Blogging Schedule: A Focus Shift

Diego Crespo — Thu, 25 Apr 2024 12:02:08 GMT

Photo by Fabien Bazanegue on Unsplash

Dear Readers,

I'm writing to share an important update about my Substack journey. For the past 17 months I’ve set out with a goal to deliver an article a week. I’m happy to say that I’ve never missed a week in over 75 articles. It's been an incredible learning experience, and your support and engagement have motivated me every step of the way.

However, life is all about adapting to new opportunities, and exciting changes at my job necessitate a shift in focus. Due to my knowledge of AI and LLMs (In no small part to the articles I’ve written on it here) I’ve been selected to work on a project implemeting large language models at a Fortune 500 company. I want to give this opportunity my full attention, so I need to make some adjustments to my schedule.

What This Means for the Blog

I won't be able to maintain my weekly posting schedule moving forward. I realize this may be disappointing, but I want to assure you this isn't goodbye. I'll still be publishing content, but on a less frequent basis.

My goal is to continue sharing my knowledge and experiences in the tech field and I hope that the experience I will gain with this project, will translate into new articles I can share with all of you.

Thank you!

Your readership has played a significant role in my personal and professional growth. I genuinely appreciate the comments, questions, and overall enthusiasm you've brought to this space.

I'm excited about the contributions I can make in the LLM space, and I hope you'll continue to follow my journey. With that I’ll be signing for this week. Thank you all for your support, it means a lot to me.

With gratitude,

Diego Enrique Crespo

Tree-sitter: Revolutionizing Parsing with an Incremental Parsing Library

Diego Crespo — Thu, 18 Apr 2024 12:06:04 GMT

No… Not that Tree Sitter..

Programmers love to add the word “tree” to everything with concepts like binary trees, red-black trees, trie, tree-shaking, and now, Tree-sitter. While 'Tree-sitter' might sound whimsical, it represents a significant innovation in parsing, the fundamental task in understanding and processing code. Traditional parsing methods often face limitations in flexibility and efficiency, but Tree-sitter seeks to improve on this and more, while also doing this in real time with every key stroke. This is an ambitious goal and requires solving many difficult problems.

The Fundamentals of Parsing

To understand the significance of Tree-sitter, it is essential to understand the basics of parsing. Parsing involves analyzing a sequence of tokens to determine its syntactic structure, often represented by a parse tree or an abstract syntax tree (AST). Traditional parsing methods, such as hand-written parsers or parser generators like Bison or ANTLR, have long been the standard tools for this task.

They work by using a grammar file often written in a version of Extended Backus-Naur Form. For many languages this is challenging due to the many quirks and edge cases. Here is a sample of some C grammar.

grammar C;


primaryExpression
    :   Identifier
    |   Constant
    |   StringLiteral+
    |   '(' expression ')'
    |   genericSelection
    |   '__extension__'? '(' compoundStatement ')' // Blocks (GCC extension)
    |   '__builtin_va_arg' '(' unaryExpression ',' typeName ')'
    |   '__builtin_offsetof' '(' typeName ',' unaryExpression ')'
    ;

genericSelection
    :   '_Generic' '(' assignmentExpression ',' genericAssocList ')'
    ;

genericAssocList
    :   genericAssociation (',' genericAssociation)*
    ;

genericAssociation
    :   (typeName | 'default') ':' assignmentExpression
    ;

postfixExpression
    :
    (   primaryExpression
    |   '__extension__'? '(' typeName ')' '{' initializerList ','? '}'
    )
    ('[' expression ']'
    | '(' argumentExpressionList? ')'
    | ('.' | '->') Identifier
    | '++'
    | '--'
    )*
    ;

argumentExpressionList
    :   assignmentExpression (',' assignmentExpression)*
    ;

unaryExpression
    :
    ('++' |  '--' |  'sizeof')*
    (postfixExpression
    |   unaryOperator castExpression
    |   ('sizeof' | '_Alignof') '(' typeName ')'
    |   '&&' Identifier // GCC extension address of label
    )
    ;

unaryOperator
    :   '&' | '*' | '+' | '-' | '~' | '!'
    ;

castExpression
    :   '__extension__'? '(' typeName ')' castExpression
    |   unaryExpression
    |   DigitSequence // for
    ;

For other languages like Schemes or Common Lisp, the grammar is much simpler due to how the languages’ syntax is constructed. Pharo, a Smalltalk, even boasts about how simple it is to parse, highlighting their LL(1) grammar, which only needs one symbol of look ahead. This is why Pharo's syntax fits on a postcard.

But since most programming languages are not developed by people with PhDs in Programming Language Theory, they usually accumulate language specific quirks that need to be handled as special cases within a compiler to fully parse the language. Because of this, creating plugins for IDEs that actually understand a programming language on a fundamental level is rare. The ugly truth is that most of the syntax highlighting you see for plugins on your favorite text editor are just a series of regex hacks. And we all know the issue with regexes

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

Enter Tree-sitter.

The Birth of Tree-sitter:

Tree-sitter was announced to the public in a GitHub blog post in 2018. When it was first announced it had support for 11 languages.

Bash
C
C++
ERB
EJS
Go
HTML
JavaScript
Python
Ruby
TypeScript

The author of the post, Max Brunsfeld was the creator of Tree-sitter. He developed it out of the need for an incremental parsing solution that was faster than the state of the art at the time. Incremental parsing allows IDEs to parse the file when it is first opened into a syntax tree. Then when edits are made in the file, Tree-sitter can give you a new syntax tree that reflects the edit. This is in contrast to how most IDEs worked pre Tree-sitter. They would reparse the whole file from the beginning to show you the new syntax highlighting. On very large files this contributed to noticeable lags in highlighting.

In Tree-sitter since the new tree shares the part of the old tree that you didn’t edit, creating the new syntax tree is fast and does not consume much memory. This makes it possible to parse source code every time a new key is pressed in the editor. But that is only part of the solution.

If an IDE tries to perform syntax highlighting on every keystroke it will invariably run into the situation where the code is in an incomplete state. This can cause errors when trying to build the syntax tree. But Tree-sitter is able to perform error recovery by determining where the start and end of every error is and give you back a working syntax tree to that point.

How it works

To use Tree-sitter you need to hope that someone has written a context free grammar for your language, or write it yourself. You do this by creating a grammar.js file. By developing the grammar in a programming language like Javascript it makes it easy to programmatically define grammars, and define grammar in terms of other grammars. An example of this is the C++ parser which is defined in terms of the C grammar. This grammar can then be parsed using a "GLR-based" algorithm. This algorithm allows Tree-sitter to handle ambiguous grammars effectively, where multiple parse trees can be generated for a given input and the right one can be picked.

After the grammar is written, you run a command line tool that spits out a parser.c file. The first part of the parser.c file is a lexer function that tokenizes the characters in a file. The second part is the parse table, which is a large 2 dimensional array that tells the parser what to do in a given state.

From there you can compile the parser.c and put the compiled library in a place that an editor can find it. See my most recent article to see what that looks like.

While parsing is Tree-sitter's primary focus, it offers a flexible API that allows developers to leverage its capabilities for various use cases beyond parsing. For example, since the code is parsed into a tree, the tree can be navigated and modified using the parse tree. This leads to navigation in files that is "aware" of the language it is in.. For example, many editors have the ability to jump to the end of a function from the beginning of a function, by finding the closing curly brace

int find_max(int num1, int num2) {
    if (num1 > num2) {
        return num1;
    } else {
        return num2;
    }
}

But that same logic fails in a language like Python which uses whitespace for scope.

def find_max(num1, num2):
    if num1 > num2:
        return num1
    else:
        return num2

With Tree-sitter, when you execute a "move to the end of the function" command, it knows where the end of a function is based on the tree, and does what you expect. This structured navigation and editing makes it easy to do all sorts of things that used to require ugly hacks, or horrific regexes. See Mickey Peterson's Combobulate library for examples of how this works in Emacs.

Tree-sitter's impact has been undeniable, with adoption in Vscode, Emacs, Neovim, and many other editors. It unlocks structured editing which allows unified movement and behavior between editors based on a parse tree. It advances on what came before, and is one of the few tools that lets you have your cake and eat it to. To top it all off it is open source, allowing everyone to benefit from its incredible achievements.

Call To Action 📣

Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter and a Mastodon, if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G

Subscribe now

Share Deus In Machina

Zig Saves the Day for cross platform tree-sitter compilation

Diego Crespo — Thu, 11 Apr 2024 12:12:26 GMT

Zig Mascot

I recently decided to give my Emacs configuration a makeover. Over time, startup had become sluggish, and I wanted to try out the new tree-sitter modes. Setting these up involves compiling tree-sitter grammars, which is easy enough on Linux. By following the excellent article, How to Get Started with Tree-Sitter by Mickey Peterson I was able to get everything up and running.

But the problem is, all the guides I found work great for Mac and Linux, but not for Windows so unfortunately this code...

(setq treesit-language-source-alist
   '((bash "https://github.com/tree-sitter/tree-sitter-bash")
     (cmake "https://github.com/uyha/tree-sitter-cmake")
     (css "https://github.com/tree-sitter/tree-sitter-css")
     (elisp "https://github.com/Wilfred/tree-sitter-elisp")
     (go "https://github.com/tree-sitter/tree-sitter-go")
     (html "https://github.com/tree-sitter/tree-sitter-html")
     (javascript "https://github.com/tree-sitter/tree-sitter-javascript" "master" "src")
     (json "https://github.com/tree-sitter/tree-sitter-json")
     (make "https://github.com/alemuller/tree-sitter-make")
     (markdown "https://github.com/ikatyang/tree-sitter-markdown")
     (python "https://github.com/tree-sitter/tree-sitter-python")
     (toml "https://github.com/tree-sitter/tree-sitter-toml")
     (tsx "https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")
     (typescript "https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")
     (yaml "https://github.com/ikatyang/tree-sitter-yaml")))
     
    (mapc #'treesit-install-language-grammar (mapcar #'car treesit-language-source-alist))

Which downloads and compiles the grammars flawlessly on my Linux system would not work for me long term.

But what is tree sitter?

I have an article about this coming soon but in brief...

Tree-sitter is a parsing library and tool that helps analyze and understand the structure of code. By understanding the structure of the code, it is able to more effectively provide syntax highlighting, code folding, symbol search, and structured editing. It is more flexible, and easier to work with than any other prior solution in the text editor space. We will look at how it does this in more depth next week :)

Zig to the Rescue

Ideally, I needed a way to automate compiling the grammars, so I could seamlessly transition across all my systems. That meant wrestling with different compilers (GCC, Clang, MSVC) on different operating systems. While doable, it sounded like a major headache, especially with the notoriously finicky MSVC.

Then I remembered reading about Zig's cross-compilation superpowers. I knew about this from Andrew Kelly's (creator of Zig) article titled... zig cc: a Powerful Drop-In Replacement for GCC/Clang.

This came in handy when I was writing the Experimenting with GUIs on the Pi Zero article. Being able to test examples beforehand by cross compile the C/C++ examples on my 8 core 4.3Ghz cpu, before committing to timing them on the Pi Zero's single core 1Ghz processor, sped things up immensely. So with a potential solution in mind, I turned to writing the script.

Automating the download PowerShell

I whipped up a PowerShell script to automate the process. I've put the core logic below, but don't worry about trying to understand all of it.

$ZigVersion = "0.11.0"

$fileExt = if ($IsWindows) { ".dll" } elseif ($IsLinux) { ".so" } elseif ($IsMacOS) { ".dylib" } else { throw "Unsupported OS" }

# Get the temp directory
$tempDir = Get-TempDirectory

# Determine OS and Architecture
$osArch = if ($IsWindows) { "windows-x86_64" } elseif ($IsMacOS) {
    if ([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture -eq "Arm64") {
        "macos-aarch64"
    } else {
        "macos-x86_64"
    }
} elseif ($IsLinux) { "linux-x86_64" } else { "unsupported" }

# I.E zig-windows-x86_64-0.11.0
$zigOsArch = "zig-$osArch-$ZigVersion"

# Construct the download URL with the Zig version variable
$downloadZigUrl = switch ($osArch) {
    "windows-x86_64" { "https://ziglang.org/download/$ZigVersion/$zigOsArch.zip" }
    "macos-aarch64" { "https://ziglang.org/download/$ZigVersion/$zigOsArch.tar.xz" }
    "macos-x86_64" { "https://ziglang.org/download/$ZigVersion/$zigOsArch.tar.xz" }
    "linux-x86_64" { "https://ziglang.org/download/$ZigVersion/$zigOsArch.tar.xz" }
    Default { throw "Unsupported platform: $osArch" }
}

# Check if Zig directory already exists and -ReinstallZig is not used
if ((Test-Path -Path "./$zigOsArch") -and (-not $ReinstallZig)) {
    Write-Host "$zigOsArch already downloaded. Skipping install. Pass -ReinstallZig to redownload."
}
elseif ($ReinstallZig -And (Test-Path -Path "./$zigOsArch")) {

    Write-Host "Removing old Zig version $zigOsArch"
    Remove-Item -Path "./$zigOsArch" -Recurse -Force
    # Get the temp directory
    $tempDir = Get-TempDirectory

    # Download and extract the file
    Download-And-Extract-Zig -url $downloadZigUrl -tempDir $tempDir
    Write-Host "Platform specific Zig has been redownloaded and extracted to: $PSScriptRoot"
}
else {

# Download and extract the file
Download-And-Extract-Zig -url $downloadZigUrl -tempDir $tempDir

Write-Host "Zig has been downloaded and extracted to: $PSScriptRoot"
}

I picked Zig version 0.11.0 because master is being actively worked on so the urls are changing.

It gets the operating system the script is running on which determines the name of the architecture, and the extension for the shared library.

It then interpolates the string to construct the appropriate url, and download the appropriate Zig compiler using that url. I download it to the platform specific temp folder, then extract the folder to my .emacs.d. The Zig executable is pre-compiled, so once I have the folder unzipped it's ready to go

Downloading and Compiling the Grammars.

After I've downloaded the Zig folder it's time for the second part of the script, downloading the tree-sitter grammars. I have all the grammars saved in an array, which I iterate through, cloning each repository. Every tree-sitter grammar repo I've encountered has either a parser.c, scanner.c, or scanner.cc located in the src folder. These are the files you need to compile to get your tree-sitter mode to work properly. The script ultimately gets these files and figures out which ones need to have these Zig commands run on them to create a dynamic library.

zig c++ -c -fPIC $fileName -o $cppObject $($includeDirs -join ' ' ) -lc"
zig cc -c -fPIC $fileName -o $cObject $($includeDirs -join ' ')"
zig cc -c -fPIC $fileName -o $cObject $($includeDirs -join ' ')"

It does some more logic to create the appropriately named libtree-sitter-.ext, and finally, copies the files over to the tree-sitter folder in my .emacs.d. And that is pretty much it. Here is a snippet of that part of the code

$grammarUrls = @(
    "https://github.com/camdencheek/tree-sitter-dockerfile",
    "https://github.com/tree-sitter/tree-sitter-bash",
    "https://github.com/uyha/tree-sitter-cmake",
    "https://github.com/tree-sitter/tree-sitter-css",
    "https://github.com/tree-sitter/tree-sitter-c-sharp",
    "https://github.com/tree-sitter/tree-sitter-javascript",
    "https://github.com/tree-sitter/tree-sitter-json",
    "https://github.com/tree-sitter/tree-sitter-python",
    "https://github.com/ikatyang/tree-sitter-yaml"
)
$zigCompiler = (Get-ChildItem zig-*/zig).FullName

foreach ($url in $grammarUrls) {
    # I.E tree-sitter-markdown 
    $repoName = $url -split '/' | Select-Object -Last 1
    # I.E tree-sitter-markdown to markdown
    $languageName = $repoName -replace 'tree-sitter-', ''
    $cloneDir = Join-Path $tempDir $repoName

    # Check if grammar directory already exists and -ReinstallGrammar is not used
    if ((Test-Path -Path $cloneDir) -and (-not $ReinstallGrammar)) {
        Write-Host "$repoName already cloned. Pass -ReinstallGrammar to reclone."
        continue
    }

    # If -ReinstallGrammar is used, clean up the current grammar directory
    if ($ReinstallGrammar -And (Test-Path -Path $cloneDir)) {
        Write-Host "Removing old version of $repoName"
        Remove-Item -Path $cloneDir -Recurse -Force
    }

    Write-Host "Downloaded $url to $cloneDir"
    # Clone the grammar repository
    git clone $url $cloneDir --depth=1

    # Navigate to the cloned directory
    Push-Location -Path $cloneDir

    # Compile the grammar using zig cc
    $libName = "libtree-sitter-$languageName$fileExt"
    $cSrcFiles = (Get-ChildItem src/*.c).FullName
    # Hack to filter schema.generated.cc from the yaml ts grammar
    # It's already included in scanner.cc
    $cppSrcFiles = Get-ChildItem src/*.cc
    | Where-Object { $_.Name -ne "schema.generated.cc" } | ForEach-Object { $_.FullName }
    $includeDirs = @('-Isrc -Isrc/tree-sitter')
    # Check if there are C++ source files
    if ($cppSrcFiles -ne $null) {
	# Compile C++ files using zig c++
	[System.Collections.ArrayList]$cppObjectFiles = @()
	foreach ($fileName in $cppSrcFiles) {
	    $cppObject = $fileName + ".o"
	    $cppObjectFiles.Add($cppObject) | Out-Null
	    $cppCmd = "$zigCompiler c++ -c -fPIC $fileName -o $cppObject $($includeDirs -join ' ' ) -lc"
	    Write-Host "Compiling C++ files with the command: $cppCmd`n"
	    Invoke-Expression $cppCmd
	}

	[System.Collections.ArrayList]$cObjectFiles = @()
	foreach ($fileName in $cSrcFiles) {
	    $cObject = $fileName + ".o"
	    $cObjectFiles.Add($cObject) | Out-Null
	    $cCmd = "$zigCompiler cc -c -fPIC $fileName -o $cObject $($includeDirs -join ' ')"
	    Write-Host "Compiling C files with the command: $cCmd`n"
	    Invoke-Expression $cCmd
	}

Modifying the init.el

After I've downloaded and compiled everything it's time to modify my init.el. The init.el file is what Emacs runs before it starts up, and can be modified by the user to make Emacs do pretty much whatever you want. I have logic in mine to make sure that tree-sitter support is compiled into my Emacs. If it is, it goes through the different languages, checks if the grammar is installed, and then maps the corresponding file extension to the appropriate -ts-mode.


(if (treesit-available-p)
    (progn
        (message "Tree-Sitter is available")
        ;; Shell script files
        (if (treesit-language-available-p 'sh)
            (add-to-list 'auto-mode-alist '("\\.sh\\'" . bash-ts-mode)))

Some people use major-mode-remap-alist instead, but I find that solution inelegant.

Here is the link to the full PowerShell file that does all of this. That is also the link to my Emacs configuration if you would like to use it yourself. I dedicate all of it to the public domain, or the MIT license if you prefer a concrete license.

If you've never given Zig a try, I'd recommend it. I think it has a lot of compelling arguments to exist, and is filling a niche that most new programming languages don't care to fill. And of course, I'd also recommend PowerShell as a modern cross-platform shell language. I've written two articles about it so my bias is clear :)

1.Getting work done with PowerShell on Linux

2.The case for PowerShell On macOS and Linux

Call To Action 📣

Subscribe now

Share Deus In Machina

The most vital knowledge in human history

Diego Crespo — Thu, 04 Apr 2024 15:18:10 GMT

Boston Public Library

I recently treated myself to a new Kindle to replace my decade old one. With a whopping 32Gb of storage, it was overkill for novels, but it sparked a curious idea (perhaps inspired by reading Asimov's Foundation)... Could I fit all of English Wikipedia onto it?

Of course, not all of Wikipedia, that currently encompasses 6,806,195 articles which is like 86gb uncompressed. But then I learned about Wikipedia's "Vital Articles." Out of all the sea of articles on Wikipedia, only a few select articles make this list, with five levels of "vitalness" that encompass larger and larger sections of Wikipedia.

Level 1 - 10
Level 2 - 100
Level 3 - 1000
Level 4 - 10,000
Level 5 - 50,000

I encourage everyone to browse the list. It's interesting to see what other people consider important throughout human history, even if it's not the list you would have made. That's definitely worth reflecting on. We programmers get a nod in Level 1 under Science, followed by a more prominent 'Computers' section in Level 2. By Level 3, there's an entire 'Computing and Information Technology' branch, a testament to our field. It did surprise me to see Cryptography get its own section at this level, but then I remembered its historical role in early computing during the World Wars with code breaking.

By level 4 there are 10,000 articles, and we get put in a higher level section called 'Technology' which includes 'Computing and information technology' and now encompasses 78 articles ranging from...

Basics
Operating Systems
Networks
Computer science
User interface
Internet
Computer hardware
Cryptography
Programming
Computer Software
Data Storage

Four programming languages get dedicated subsections in the section on Programming.

Assembly
C
Java
Javascript

Which I know will ruffle some feathers. Disappointingly, Most of the articles in the Computing and information technology section have a C grade, which Wikipedia describes as

"Useful to a casual reader, but would not provide a complete picture for even a moderately detailed study"

I'd like to spend some time in the future improving this.

By level 5 we get one final high level category, 'Communication, media, and computing technology'. It consists of 729 articles, broken down into 12 broad sections and many more subsections.

But while it's interesting to see at a high level what the community thinks is important in my field, it's time to figure out how to download all of this

Downloading articles

I found a nifty Calibre plugin called Wiki Reader for grabbing Wikipedia pages

The interface is simple. It allows you to put in the name of the book associated with the page you want to download...

Then it generates a MOBI file.

If instead of a single URL, you copy and paste a bunch of URLS to your clipboard, it can auto populate the urls, and combine them all into a single ebook.

Unfortunately, while bulk downloading is possible, it's agonizingly slow after about 100 articles. But when I tried to write a Python script to scrape the pages manually, the results weren't much better.

I was about to throw in the towel when I stumbled another project, Wikibooks.

Enter Wikibooks

Wikipedia itself has limitations as a learning resource...

Unfriendly for Reading: Links and references disrupt the flow.
Teaching Gaps: It's not a structured learning environment.

Wikibooks (also from Wikimedia) solves this. It offers deeper dives on certain subjects, with more organized learning paths, while keeping the approach to community contribution. All the information for a topic exists in one book, and you don't need to navigate a maze of links to get the full picture. The tradeoff is less overall coverage than its big brother Wikipedia, and a smaller community, but there are still plenty of content to choose from. I'm just glad a community of volunteers exist that make this knowledge freely available to all of us.

The nice thing about downloading Wikibooks over Wikipedia articles is that they are generally much friendlier to offline reading. Many Wikibooks offer printable versions of the books, and some even have e-reader friendly PDFs.

This means better formatting and a smoother reading experience on my Kindle. Plus, it's trivial to combine PDFs with tools like PyPDF2 in Python, after they are downloaded.

The Joy of Learning

I went through Wikibooks and I downloaded many of the books I was interested in to put them on my Kindle. While going through the Philosophy section, I came across a Wikibook that was called Lun Yu" (论语), also known as "The Analects of Confucius". It is a collection of sayings and ideas attributed to the Chinese philosopher Confucius and his disciples. The first quote from the first chapter resonated deeply with the spirit of this whole experiment...

The Master said, Is it not pleasant to learn with a constant perseverance and application?

Indeed, it is.

Call To Action

Subscribe now

Share Deus In Machina

Star Trek's Optimism Towards the Future

Diego Crespo — Thu, 28 Mar 2024 12:02:19 GMT

I've been watching the OG Star Trek series, having never consumed any of the TV shows before, and I've been surprised at how refreshingly optimistic its take on humanity has been. It's a stark contrast to the dystopian visions that dominate much of contemporary science fiction, I.E The Hunger Games, Blade Runner, Snowpiercer, and Black Mirror. As a millennial who grew up on such bleak narratives, the setting of Star Trek feels foreign.

Science fiction, unlike fantasy, often strives for a sense of plausibility. The shift toward dark, dystopian futures within the genre mirrors my generation's growing pessimism. Millennials and Gen Z have witnessed economic struggles, the escalating effects of climate change, and increasing social division. Understandably, this has shaped our outlook on what lies ahead.

That's not to say that there aren't people who are optimistic about the future though. Effective Accelerationism (E/acc) challenges this pessimistic trend. Like the techno-optimists and transhumanists of the past, E/acc supporters embrace technological progress and the potential of Artificial Intelligence. They reject the notion that the future must inevitably be bleak. While I'm not sold on the E/acc crowd, I also generally have a positive outlook on the future. It certainly makes thinking about the world I will inherit as I grow older a less daunting task.

It's also worth noting that this pessimism overlooks significant advancements in the last 15 years. Consider the rise of smartphones, breakthroughs in AI and deep learning, the promise of CRISPR gene editing, reusable rockets, and immersive VR experiences. These examples suggest that while the challenges are great, our trajectory isn't entirely negative. For example just this year, Make-A-Wish downgraded Cystic Fibrosis as an automatic qualifier for wish granting, due to ongoing advances in care. That's incredible and should be celebrated.

I don't think Star Trek's optimism is sourced in blind faith in progress. The show acknowledges social issues and conflict but envisions a humanity that has overcome its worst impulses. Perhaps we can draw inspiration from Star Trek, not to disregard the challenges we face, but to rekindle a sense of measured optimism. What if we imagined solutions along with the problems? What if we believed that progress, while never guaranteed, remains possible?

It's natural to be concerned about the future, but we shouldn't let pessimism cloud our ability to find solutions. The future is ours to shape, and perhaps a spark of the "Star Trek spirit", that belief in our potential to work together and build a better world, is what we need to chart a brighter course. Anyway, just some thoughts I had as we go into the weekend.

Call To Action 📣

Subscribe now

Share Deus In Machina

Learning to use Docker

Diego Crespo — Thu, 21 Mar 2024 12:13:55 GMT

Docker was released 13 years ago and has quickly become a mainstay in the programming community. At first I avoided it, considering it too complex, intimidating, and a waste of time for small projects. But Docker is not any of those things. Sure, wrapping a complex project with a Docker bowtie can be challenging, but that's because non trivial projects usually require non trivial solutions. The benefits you get once this is accomplished though, is immense. But before we get to that, let's start simple. What is Docker?

Docker is a technology that helps you package and run your software applications in a consistent and efficient way, regardless of the computing environment they're running on.

This package is typically called a Docker Image. The running instance of an Image is called a Docker Container. You can think of a Docker Image like a programming language and technology agnostic version of a C# or Java project, or a Python virtual environment. In those examples, the language settings and all of the packages that you install are only for that specific project/language you are working on.

If others want to collaborate on the project, they will need to have the same settings and packages as you. For example, if you have a Python virtual environment you can run pip freeze > requirements.txt to get the list of packages your project needs, then pass along the txt file to someone else who can then run pip install -r requirements.txt to download the same packages. But what if the person trying to install the dependencies for your project has a different version of Python than you? They may run into dependency issues, due to functions being deprecated, or behavior being changed as a library matures. Docker avoids this problem.

But there are more than just dependency conflicts that can doom a project. Sometimes projects require more than what the standard library, or 3rd party package ecosystem can provide.

Let’s say you want to do OCR on some pdfs. You might decide to use pytesseract. In the installation instructions it says that you need Google Tesseract OCR, as pytesseract is a wrapper over it.

Install Google Tesseract OCR (additional info how to install the engine on Linux, Mac OSX and Windows). You must be able to invoke the tesseract command as tesseract. If this isn’t the case, for example because tesseract isn’t in your PATH, you will have to change the “tesseract_cmd” variable pytesseract.pytesseract.tesseract_cmd. Under Debian/Ubuntu you can use the package tesseract-ocr. For Mac OS users. please install homebrew package tesseract.

Unfortunately Tesseract can't be pip installed, so getting it set up can be tricky. I can't count how many times I've joined a project, and my initial experience looks like this

Try to install additional dependencies
Run into an error preventing the install
Message Teams Chat, “Hey, has anyone run into this error when installing X?”
"Oh yea, the link they give in the documentation is to an old version of X, this Stack Overflow post has the correct link"
*Continues trying to install and runs into problem Y*
Ask in the chat again
"Oh it’s because your using version ZZZ for this, you have to downgrade it because Y is expecting a version QQQ instead."

Eventually you get it all working, and go on your merry way. 3 months later, it is summer and you have new interns. One of them has been assigned to this project to help. All of a sudden you see a Teams message in chat from the new intern.

“Hey has anyone run into this error when installing X?”
You: *Thinks very hard*. "Oh I remember this error! Hold on I have a link to a Stack Overflow post that has the fix"

Repeat steps 4-7 as your intern struggles with the same issues you had 3 months ago.

If only there was some way you could construct your software applications, that would allow it to behave in a consistent and efficient way, regardless of the computer environment it's running on. Wait! What's that I hear you say? There is? Why yes! Docker of course!

Docker In Practice

Let’s use our previous example for a simple Python app that performs ocr, and prints out the text to the screen.

The following code takes a trimmed pdf from the Autobiography of Benjamin Franklin pages 7-17, and prints it out to the console.

import pytesseract
from pdf2image import convert_from_path

pdf_file = "trimmed_autobiography.pdf"

# Convert PDF pages to images
pages = convert_from_path(pdf_file, 300)  # DPI set to 300

# OCR each page and extract text
text = ""
for page in pages:
text += pytesseract.image_to_string(page)

print(text)

Now let’s look at the Dockerfile we need to build this.

FROM python:3.12-slim-bookworm

# Set the working directory
WORKDIR /app

# Install Tesseract OCR, poppler-utils, and their dependencies
RUN apt-get update && \
apt-get install -y tesseract-ocr poppler-utils && \
apt-get clean

# Install python packages
RUN pip3 install --no-cache-dir pytesseract pdf2image

# Copy the main.py and PDF file into the Docker image
COPY main.py trimmed_autobiography.pdf /app/

# Command to run the Python script
CMD ["python3", "main.py"]

Here's a break down each line of this Dockerfile:

FROM python:3.12-slim-bookworm: This line specifies the base Image to use for building the Docker Image. In this case, we are using an Image named python with the tag 3.12-slim-bookworm, which means it's based on Python 3.12 and the Debian 'slim' variant. Slim is a smaller version of Debian without extra packages. We use the Python version of Debian instead of the normal Debian Image due to the fact that regular Debian disallows global pip installs (this is normally a good thing).
WORKDIR /app: This line sets the working directory inside the Docker container to /app. It's common practice to put the code for a project in /app but your working directory can be anywhere you want. The folder will be created if it isn’t there already. Subsequent commands will be executed from inside this directory
RUN apt-get update &&:
1. RUN Is a Docker instruction that executes commands in the container during the build process.
2. apt-get update Uses the Debian Advanced Package Manager (Apt) to update the package lists for packages that are available for installation.
3. && Allows for multiple commands to be run in a single RUN instruction and the \ allows the instruction to continue onto the next line.
4. apt-get install -y tesseract-ocr poppler-utils &&: This line installs the packages tesseract-ocr and poppler-utils along with their dependencies using the apt-get install command. The -y flag automatically confirms any prompts during installation.
5. apt-get clean: This command removes any unnecessary files and packages that were downloaded during the installation process. It helps to keep the Docker image size smaller.
RUN pip3 install --no-cache-dir pytesseract pdf2image: This command installs Python packages pytesseract and pdf2image using pip. The --no-cache-dir option disables caching of downloaded packages and metadata.
COPY main.py trimmed_autobiography.pdf /app/: This line copies files main.py and trimmed_autobiography.pdf from the host machine into the /app directory within the Docker container.
CMD ["python3", "main.py"]: This is the default command to run when the container starts. It specifies to run the main.py Python script using the python3 interpreter.

A Dockerfile is kind of like as a script you would run if you wanted to automate setting up a new laptop. It would specify all the tools and their dependencies you would need and then go and fetch and install them.

Building the Docker Image

Now that we have a Dockerfile we can use it to make a Docker Image. If you open your terminal and run docker build -t book-slim . with this Dockerfile in the current directory, Docker will build a Docker Image based on the instructions provided in the Dockerfile. Once the image is created, you can run the Docker Image with the command docker run --name book-slim-container book-slim. This executes your docker Image named book-slim and gives the running container the name book-slim-container. This is helpful because all running Images need a name. If you don’t provide one than docker will make up a name.

Looking at my Docker Desktop app, I can see that when I executed my Docker Image the first couple of times without specifying a name, Docker gave it the name “trusting_nightingale”, and “nostalgic_shaw”. But when I specified a --name, that was the name it gave my container instead.

Getting back to our running program, after waiting patiently you should start seeing text printed out to the screen

My father had also, by the same wife, four children born in America, and ten others by a second wife, making in all seventeen. I remember to have seen thirteen seated to- gether at his table, who all arrived at years of maturity, and were married. I wus the last of the sons, and the youngest chiid, ex- cepting two daughters. I was born at Bos- ton, in New England. My mother, the sec- ond wife, was Abiah Folger, daughter of Peter Folger, one of the first colonists of New England, of whom Cotton Mather makes . honorable mention, in his Ecclesiastical His-

So, we built our Docker Image which specified all the dependencies we need to run the program, then we ran our Image, and it executed the main.py to print the pdf's text to the terminal like we asked it to. Run the command in the terminal again, but be warned it will fail.

docker run --name book-slim-container book-slim

This is the error message you will get

docker: Error response from daemon: Conflict. The container name "/book-slim-container" is already in use by container "5bf83a1f62e4e6191fd502b889282334a1e666b42f473ed43e3d61fc87ac013c". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

The reason for this is that Docker requires each container to have a unique name, and we already created one named book-slim-container

To resolve this, we have a few options...

Remove the existing container: You can remove the existing container with the conflicting name using the docker rm command.

docker rm book-slim-container
or
docker rm 5bf83a1f62e4e6191fd502b889282334a1e666b42f473ed43e3d61fc87ac013c

Rename the existing container: If you want to keep the existing container but use a different name for the new one, you can rename the existing container using the docker rename command

docker rename book-slim-container new-name

Restart and re-execute the container: An existing container can be restarted, and then re-executed.

docker restart book-slim-container && docker exec book-slim-container python3 main.py

Avoid the issue altogether: Because an already created Docker Image is easy to execute as a container, it is common to pass the parameters --rm to delete the container after it is run, so subsequent runs can just recreate themselves using the same container name and Image

docker run --rm --name book-slim-contaner book-slim

Exploring containers

Let’s peel back the veil a bit on the container itself. Run the following command in the terminal.

docker run --name book-slim-container -it book-slim /bin/bash

Here’s what these new commands do

-i: Stands for "interactive mode". This means that the container's standard input will remain open even if not attached
-t: Stands for "tty" or "pseudo-terminal". It allocates a pseudo-terminal for the container
Together, -it ensures that the container session is interactive, allowing you to interact with the shell running inside the container */bin/bash: Overrides the default command specified in the Dockerfile (python3 main.py) and starts a bash shell (/bin/bash) inside the container.

If the interactive command ran successfully, you should be inside the container in the WORKDIR /app.

If you type ls you can see the two files that were copied into the container when the Docker Image was built. You can do python --version and see that the python version is 3.12 and run the main.py file manually. It all works like you would expect. When you are done exploring the container you can type exit in the terminal.

So the advantage of this approach is that you do all the hard work of figuring out all the dependencies for a project, and then encode that into a Dockerfile. Then your coworkers, friends, or collaborators just need to have Docker installed, and then they can run two simple commands to get up and running with your project

docker build -t  book-slim .
docker run --rm --name book-slim-contaner book-slim

Mounting volumes

Let’s make our python program more useful. Instead of printing out the text from the pdf, let’s save it to a file. Here is the new main.py

import pytesseract
from pdf2image import convert_from_path

pdf_file = "trimmed_autobiography.pdf"

# Convert PDF pages to images
pages = convert_from_path(pdf_file, 300)  # DPI set to 300

# OCR each page and extract text
text = ""
for page in pages:
text += pytesseract.image_to_string(page)

output_file = "extracted_text.txt"

with open(output_file, "w", encoding="utf-8") as file:
   file.write(text)

print("Text extracted and written to:", output_file)

If we were to run the Docker Image now with

docker run --rm --name book-slim-container book-slim

The extracted_text.txt would not be written in the same directory you ran the Docker Image. It would write inside the container, and then get cleaned up once the container finishes running. To make the written file accessible outside of the container, you want to mount a directory using the -v command.

Mounting a host directory into a Docker container allows you to share files and directories between the host machine (where Docker is running) and the container. When you mount a directory, you're essentially creating a link between a directory on your host system and a directory inside the container. This has 3 advantages

Access to Host Files: Any files or directories in the mounted directory on the host machine are accessible from within the container.
Persistence of Data: Changes made to files or directories in the mounted directory from within the container persist on the host machine, and vice versa. This means that if a file is created, modified, or deleted in the mounted directory from either the host or the container, the change will be reflected in both places.
4Sharing Resources: Mounting a directory can be useful for sharing resources such as code, configuration files, or data between the host and the container. This is particularly handy during development or when you need to provide input data or retrieve output data from a container.

Here is the full command to replicate this behavior.

docker run --rm --name book-slim-container -v .:/app book-slim

To make sure the file is written locally, I pass .:/app to the -v command. This mounts the current directory ‘.’ on the host machine to the /app directory inside the container.

After a few moments, extracted_text.txt appears, confirming that I successfully wrote the file to disk

And that's the basics of Docker. There are still many other aspects of Docker worth getting into (Docker-Compose, managing multiple docker containers, exposing ports, etc) but this should at least get you started. In the mean time, I hope that this article made Docker more approachable, and I hope you use it in one of your next projects!

Call To Action 📣

Subscribe now

Share Deus In Machina

The DOIT philosophy: How I'm avoiding Decision Fatigue and the Paralysis of Choice this year

Diego Crespo — Thu, 14 Mar 2024 12:02:23 GMT

My mantra for this year is DOIT and it stands for

DO
Overcome
Iterate
Triumph

I have a lot of projects that are in various states of completion, and I want to get them over the finish line. I just need to DOIT. In service of this, another goal of mine in 2024 is simplicity. By focusing on simplicity I will be able to more effectively achieve my project goals. Einstein's famous quote is applicable here

It can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience

Amusingly, the more familiar version of this quote is actually a simplified version of the former. It illustrates the point well...

Everything should be made as simple as possible, but no simpler

The key is the 2nd part of the quote "but no simpler". Complexity is the bane of Programmers, and I've been making my life more complicated than it needs to be recently. Sure, there are things that are complex just by their very nature, and just like how I wouldn't fear 1 duck, but 100,000 would give me pause, a 100,000 line code base would still be challenging to work with no matter what the language. But at least languages that value simplicity leave more space in your brain for the actual code part, so you don't end up looking like this

Or as Andrew Kelly puts it with Zig

Focus on debugging your application rather than debugging your programming language knowledge.

Ultimately what we are trying to do as Programmers is create things, and complexity gets in the way of that. But the complexity which has inhibited my ability to complete projects doesn't just stop at what programming languages I am using, there are other things that cause this complexity, and they result in two things. Decision fatigue and the Paralysis of Choice. Since these are so similar, let's get a baseline for how they are defined

Decision Fatigue: The deteriorating quality of decisions made by an individual after a long session of decision making. Essentially, the more decisions you make, the more your ability to make further decisions diminishes. This is because making decisions is cognitively taxing, and over time, the mental resources needed to weigh choices, foresee outcomes, and make judgments become depleted. As a result, an individual might resort to simpler decision-making strategies, exhibit reduced self control, or even avoid making decisions altogether as they become fatigued.

Paralysis of Choice: When an individual is faced with too many options and finds it difficult to make a decision. Instead of making a choice, the person becomes overwhelmed by the options available and may feel anxious or unable to choose at all. The paralysis of choice suggests that having too many options can lead to less satisfaction with the eventual choice made, due to the stress of the selection process and the potential for regret over the alternatives not chosen.

How it affects people

The important thing to note about the two definitions above is that it doesn't say anything about a specific scenario where this happens. From the time you get up to the time you go to bed, the act of decision making, and the number of choices presented cause the problems. It doesn't matter whether the decisions you are making are about lunch, or what Steam game to play, they all negatively affect your downstream decision making abilities. This is why successful people employ the trick of wearing the same close to reduce the amount of choices they have to make. But while this is definitely not a live like a "billionaire" grindset sort of article, the principles are sound. CEOs simplify their lives so they can make big decisions that steer the direction of companies and billions of dollars in capital. But I can do the same thing to be more successful in achieving more modest goals.

How it affects me

To this end I've been simplifying many aspects of my life. One way I've done this is cutting down to one large monitor. This might seem controversial as the productivity gain from two is almost double a single monitor. But a second monitor also invites temptation. It's easy to put things on that 2nd screen that are just for entertainment and don't serve the current task at hand. Furthermore, I find the little snags that come with a 2nd monitor interrupt my flow while working. For example

Windows opening on the wrong screen
Windows opening between two screens
Resolution issues when both monitors are not the same size
Deciding which monitor a window should be on

The point is not that these things can't be fixed, it's that they require three things of mine that are in limited supply. My concentration, my decision making energy, and my will power to not watch YouTube videos. I would rather alt tab to another window when I really need it, or split my window in two if I need to read documentation. Plus with one monitor I don't have to worry about one screen being in a suboptimal position. And as eye strain becomes more of an issue, this is important for me. I'm not the only one who feels this way too, and while my back would never let me program from a couch, I agree with a lot of the things Theo says in this video.

Another way I am simplifying my life is to stick to one operating system. I would switch distributions multiple times a year, as the sheen of the new one faded. This required me to spend the better part of the day just setting up everything again, and then another month running into little things I had forgotten to. This year I'm forcing myself to stick with Pop!_OS as my main desktop's operating system. Pop!_OS is good, so this isn't the hardest decision, but speaking it into existence as opposed to privately muttering it to my self provides some accountability.

I'm also cutting down on the number of programming languages I will start new projects in. I've been hopping programming languages like I hop distributions, and while I've appreciated the breadth of experience this has given me, I find that it has caused me to have very shallow experience in a lot of languages that aren't Python. Furthermore, a lot of projects start with me just digging into what programming language I'm going to use, what libraries are available for the task, and how to make my code more idiomatic. I end up trying to Goldilocks myself into the right programming language for the project, and that never works. This paralysis of choice also has a subtle but deadly side effect that I have been feeling more frequently too. Let's revisit part of the definition

leading to less satisfaction with the eventual choice made, due to the stress of the selection process and the potential for regret over the alternatives not chosen.

Less satisfaction and regret, two things I don't need more of in my life. Along these same lines I will reduce the number of categories these projects fall under. From GUI to Games, Programming Language Theory to Machine Learning, and Art to Level Design, I've doing a lot of things that cover a lot of different topics. They are all interesting, and I wish I could spend every waking moment hopping between them, but I will never reach the level of proficiency I desire if I spread myself so thin.

Finally as programmers there are tons of decisions we have to make related to...

Programming languages
Frameworks
Windows vs Mac vs Linux
Mechanical Keyboard Switches
frameworks
Wall Paper Color
Text Editor Font
Text Editor
ECS, Functional, OOP
Web, Mobile Desktop

Before we've even written a single line of code. It's no wonder so many of us want to start a farm instead. But I'm saying let's not throw the baby out with the bath water. Maybe I can pare things down a bit and focus on actually putting code to text editor. This was a lot easier back in the 80s when the options were Basic, Assembly, or Pascal (and let's be real, just Assembly when you were writing something serious). But in the context of today, if I don't pick the perfect stack for my project, who cares, just deal with it. My programming ancestors did more with less. Now I'm not saying to scale your product to millions of users using just Google Sheets, but it's hard to go wrong with a mainstream language today.

Of course, I will still learn new things for my job, and I will still write about the interesting things I do here. But I will cut back in places where I can. As it currently stands I have over a years worth of drafts saved up, so even if I just draw inspiration from my drafts it will cover me well into 2025. But, as the year is not even close to over, I suspect I'll come across more little nuggets that will spark ideas for new articles regardless.

Maybe some of you also struggle with this problem, and there are ways you can simplify your lives to make shipping easier too. I hope we all simplify our stacks this year, so that we can DOIT and be proud of what we created.

Call To Action 📣

Subscribe now

Share Deus In Machina

A World without C

Diego Crespo — Thu, 07 Mar 2024 13:01:20 GMT

The Whitehouse released a report this week encouraging the use of memory safe languages, over non memory safe languages, specifically calling out C/C++. This has obviously caused a stir in the programming community, especially considering most people's low opinion of the government’s ability to produce quality software. But if you’ve used a computer in the last year you’ve probably been subjected to poor quality software made by the brightest minds in Silicon Valley, so it's a moot point

What I find surprising in all of this discourse is that people have missed the point, intentionally or not. Memory unsafe languages have footguns that lead to unsafe software, and that leads to CVEs that cause harm to real people. The context for the report is not your hobby project, nor your single player 2D platformer. It’s about software that power governments and institutions, places that store large amounts of people’s data etc. It’s about companies that provide live services to millions of people, and store their credit card numbers, phone numbers, addresses, names, and dates of birth. This also not about it becoming illegal to program in memory unsafe languages. To sum it up

Programmers writing lines of code do not do so without consequence; the way they do their work is of critical importance to the national interest.

A lot of the discourse revolves around C, with the main point being that you will never get rid of it. That the world runs on C, your operating system runs on C, your programming language runs on C etc, etc. I know how important C is, I've written multiple articles that feature C prominently, and I'll probably write some more

There are a lot of reasons for C's dominance in the modern era, but the one that sticks out to me is that it is the lowest common denominator for just about everything. If you write something in C, it is straightforward in most languages to bind to it through an FFI. That means if I for instance, come across a cool C library that does exactly what I need, I can just bind the functions in my Python code without knowing all the nitty gritty details of how it's implemented. I can do it again in Go, and Java and C# and on and on. Just look at the list of programming languages that have Raylib bindings to see how powerful of a concept this can be

But Cs greatest strength is also its greatest weakness. It's simple. My 2nd Edition of K&R is just shy of 300 pages, but a good chunk of that is a reference manual, and a chapter on lexical conventions which can be skipped on a first read through. You could skim through the book in a day. But while the syntax may be simple, there are a lot of concepts in C that are tricky to get right, especially in large programs

Pointer Arithmetic
Memory Management
Headers/Preprocessor
Macros
Undefined Behavior
Strings and Cross Platform string behavior/support
Actually building large projects

And you don't have to take my word, here is C++ being brought up again as an alternative to C in the Linux Kernel just this year (2024), by someone who has programmed more C than I ever will in my lifetime. And here is a page that details GCC's transition to C++ (the irony is not lost on me). If these people are the best C programmers in the world, what chance do I have? Even John Carmack talks about being surprised on the Lex Fridman podcast about how many mistakes he found when he attacked his code base with sanitizers and fuzzing.

You might ask, well can't we just make C a better language? And the answer is yes, but that takes time. I'm really looking forward to many of the changes that are in C23, but it will be many years before that will be widely available across MSVC, Clang, and GCC. And getting any feature into C is an exercise in frustration. You only have to read JeanHeyd Meneide's article titled Finally, Embed in C23 to understand how thankless of a task this is. I'll give you one excerpt, but I've seen this same sentiment time and time again when it comes to improving the language

It’s deeply depressing and ultimately a great source of burnout being at the grindstone for 4 years for things people were casually discussing about in September of 1995 (and earlier). It’s almost as depressing as putting typeof in the C Standard and then realizing this was something they’d been discussing doing since after C89 (1989). Am I destined to just keep closing the loop on old, unrealized dreams because a bunch of people were too tired/scared/busy/combative to standardize what has literally been decades of existing practice?
It was hard to realize how tired I was of this kind of grind until the day the feature was put into the C Standard, this past Tuesday. I quite literally could not even muster a “yes!” after the vote finished. I just felt empty and numb, because quite literally dragging an entire community of implementers through several hurdles, to finally get them to acknowledge the existence of a problem and its solution, is just… soul-crushing. It is a level of effort that I recommend to nobody in any sphere of life, for any task. At least when you are on the ground and organizing and helping people, you’re providing some kind of material value. Water. Food. Improving something. Here? I’m talking about things that are quite literally older than I am. Just trying to bring ideas from the last 3 decades - things you would think were Table Stakes for foundational languages like C and C++ languages - would not be seen as a radical new paradigm or shift in the language. Nevertheless, I spent (burned?) that energy, and finally. It's in.

That doesn't sound like a fun process to me...

Now if you're still here I want to make it clear that there are qualities of C that I admire. It's an important language in the annals of programming history, and it is well worth learning. I love C's Spartan nature when I'm practicing Algorithms and Data Structures, and I appreciate that it produces software that is small and fast. Today's modern programming languages have done a shit job at filling in the C's shoes, which is why it's still around so prominently. I think of all the programming languages out there, Zig is the one I'm most hopeful for replacing C in what C does best. And it does so while being safer, easier to cross compile and build, and without adding a ton of complexity to the syntax like languages like Rust, D, and C++ do. Now to my main point...

It is worth noting that C didn't appear on the scene until 1972. That means that 60% of the population in the United States has been around longer, and there are plenty of programmers that remember a world where C didn't exist. C was also absent in the early 8-Bit era compared to programming languages like Basic, Pascal, and Assembly due to its difficulty to compile. So really it's only like 45 years that C has been a major player in the programming space. But people keep repeating the same mantra over and over again "C will never die", which feels like a shocking amount of small mindedness coming from the programming community. I thought we were supposed to be innovative thinkers, problem solvers, and engineers. Must we forever be shackled to a programming language we created 50 years ago, in field that is only 75 years old? Is it really not possible to envision a world that isn’t powered by C or C++ even if it takes another 50-100 years?

Two Years ago no one was talking about Large Language Models. We were still 30 years away from anything remotely resembling a useful AI assistant. Now, one year later my Church is using AI Generated images from ChatGPT in their service, and Nvidia has doubled its stock priced in a year. 8 Years ago Visual Studio Code was released, and now it is the most popular text editor by far. The programming space can move quickly if the value is there, and I believe that moving away from C is one of those things. If our best years of programming are still to come (and I hope they are), and we expect more code to be written in the next few years than at any other point in history, then the percentage of C code should dwindle every year. If the financial incentives are there, and if governments will start biasing favor towards more memory safe languages, then there will be a strong incentive for certain companies to comply.

Replacing C does not mean erasing its legacy. I will continue to write about C, and tell its part in the ever evolving history of programming, and I hope you will continue to enjoy the story as it is written.

Call To Action 📣

Subscribe now

Share Deus In Machina

Experimenting with GUIs on the Pi Zero

Diego Crespo — Thu, 29 Feb 2024 13:01:20 GMT

I recently brought an old Raspberry Pi Zero back to life. With the oldest files dating back to the summer of 2020, it was safe to say that it was time for a fresh install. Thankfully there is still a version of Raspberry Pi OS that supports 32 bit Raspberry Pi's like the Zero W

It comes with Bookworm which is a recent Debian release, which means we get nice goodies like Python 3.11, version 6+ of the Linux kernel (up from 4.19 in the old install), and newer versions of lots of other software. Not that I'll be using the RPI Zero much like a Desktop. A 32-bit single core CPU clocked at 1 Ghz, with 512Mb of RAM, is basically abandonware at this point as a Desktop platform. Even the default installer from the Raspberry Pi foundation installs software you can't use. It asks whether you want to install Chromium and or Firefox, but both programs say this when you try to use them

Chromium/Firefox is not supported on the Raspberry PI Zero W

Whether it's actually realistic to use the Raspberry Pi Zero as a Desktop is a boring question. Who cares! It's fun. But anyway, I'm more interested in using the Zero W to test the baseline performance of some apps that I'm writing anyways, so this isn't a problem. Let's start by looking at a couple of programs on the Pi Zero and see how they perform out of the box. The answer... Pretty slow, but not as bad as I was expecting. Opening up basic software like the terminal, and the file browser take anywhere from 3-6 seconds. That's definitely slow, but even on my beefy desktop with an SSD, most of the apps I use take at least 1-3 seconds to start up. That's not even an order of magnitude faster on a machine that is much more powerful. But complaining about software performance in 2024 is so passé so I'll just say it sucks. It did pique my interest in performing some highly unscientific testing though which is what you will get below.

Being nerdsniped by timing random apps opening, I decided to write some basic GUI apps that did nothing more than open an 800x600 window, and time them on my phone. Starting off with GTK, I modified the basic GTK4 tutorial example

#include 

static void
activate (GtkApplication* app,
          gpointer        user_data)
{
  GtkWidget *window;

  window = gtk_application_window_new (app);
  gtk_window_set_title (GTK_WINDOW (window), "Window");
  gtk_window_set_default_size (GTK_WINDOW (window), 800, 600);
  gtk_window_present (GTK_WINDOW (window));
}

int
main (int    argc,
      char **argv)
{
  GtkApplication *app;
  int status;

  app = gtk_application_new ("org.gtk.example", G_APPLICATION_DEFAULT_FLAGS);
  g_signal_connect (app, "activate", G_CALLBACK (activate), NULL);
  status = g_application_run (G_APPLICATION (app), argc, argv);
  g_object_unref (app);

  return status;
}

Compiling the tutorial with the -O2 optimization set, I was astonished to see the app take between 12-14 seconds to open a blank window. Now GTK4 does a lot for us out of the box, and that comes with an upfront cost. But for a blank window that is pretty bad. I looked around for some examples of deploying apps on resource constrained systems with GTK, and even asked in the GNOME forums but couldn't find any solutions. If you know of any please let me know.

With that shocking result, I decided to get a baseline for how fast a window could open up. I did this by making as simple X application that opens a window, and then timing the result...

#include 
#include 
#include  // For exit function

int main() {
    Display *display;
    Window window;
    XEvent event;
    int screen;

    // Open connection to the X server
    display = XOpenDisplay(NULL);
    if (display == NULL) {
        fprintf(stderr, "Cannot open display\n");
        exit(1);
    }

    screen = DefaultScreen(display);


    window = XCreateSimpleWindow(display, RootWindow(display, screen), 10, 10, 800, 600, 1,
                                 BlackPixel(display, screen), WhitePixel(display, screen));


    XSelectInput(display, window, ExposureMask | KeyPressMask);


    XMapWindow(display, window);


    while (1) {
        XNextEvent(display, &event);
        
        // Break out of the loop if the user presses any key
        if (event.type == KeyPress)
            break;
    }


    XCloseDisplay(display);
    return 0;
}

Except, I almost didn't have a chance to time the result, as the window opened up in ~0.5 seconds. I'll be the first to tell you this is not an Apples to Apples comparison, and writing a GUI using just X11 would be an exercise in suffering. (I mean but that's basically what Casey Muratori did in his Handmade Hero Series for Windows). But if all apps opened at that speed, my RPI Zero W wouldn't feel slow at all. Armed with this new knowledge, I decided to look at a couple of other ways of opening a window.

Since I've been on an SDL2 kick recently, it only made sense to do the same test with that library. You make recognize this code as essentially the example I wrote in SDL Tutorial Part 1: Opening A Window from a couple weeks ago

#include 
#include 
#include 

#define SCREEN_WIDTH 800
#define SCREEN_HEIGHT 600

int main(){
 if (SDL_Init(SDL_INIT_VIDEO) < 0){
   printf("Couldn't initialize SDL: %s\n", SDL_GetError());
   return EXIT_FAILURE;
 }

 SDL_Window *window = SDL_CreateWindow("Example: 0", SDL_WINDOWPOS_UNDEFINED,
   				SDL_WINDOWPOS_UNDEFINED, SCREEN_WIDTH, SCREEN_HEIGHT, 0);
 if (!window){
   printf("Failed to open %d x %d window: %s\n", SCREEN_WIDTH, SCREEN_HEIGHT, SDL_GetError());
   return EXIT_FAILURE;
 }

 SDL_Renderer *renderer = SDL_CreateRenderer(window, -1, 0);
 SDL_SetRenderDrawColor(renderer, 255, 255, 255, 250);
 SDL_RenderClear(renderer);
 SDL_RenderPresent(renderer);
 SDL_Delay(2000);
 SDL_DestroyWindow(window);
 SDL_DestroyRenderer(renderer);
 SDL_Quit();
 return EXIT_SUCCESS;
}

And our results... A little over 3.1 seconds. While SDL2 is not a retained mode GUI, I could certainly do more with it than I could with just a raw X app. So, so long as people didn't need a screen reader for the program, this might be a viable option. (I'm not even sure you could run any sufficiently complex app, and a screen reader on the RPI Zero, at the same time anyway though).

I recently made a simple GUI app in Python with Tkinter, and since it's already built into Python I wrote my example in it next to see how it would perform

import tkinter as tk

app = tk.Tk()
app.title("Simple 800x600 Window")

app.geometry("800x600")

app.mainloop()

~2.9 seconds. For an language that has to interpret the code upfront, that is not bad. I think that Tkinter is an underrated gem of the Python world, and I'll be writing more about that soon. But, while the Tk integration with Python is great, I wanted to look at two more implementations. One using the de facto Tcl bindings to Tk, and the other using just raw C and Tk

#!/usr/bin/env wish

wm title . "Simple Window"
wm geometry . 800x600

tkwait window .

Tcl is an interesting language that warrants its own discussion at a later date, but on my RPI Zero I was able to open up the window in ~2.0 seconds which is a great improvement. Considering we are entering the territory of as fast, or faster than the default apps, I feel like we have reached acceptable speeds. But Python and Tcl are just binding the underlying C Tk library, so let's look at C next.

#include 

int main(int argc, char **argv) {
    Tcl_Interp *interp;
    Tk_Window window;

    // Initialize Tcl and the Tk toolkit
    Tcl_FindExecutable(argv[0]);
    interp = Tcl_CreateInterp();

    if (Tcl_Init(interp) == TCL_ERROR) {
        return -1;
    }

    if (Tk_Init(interp) == TCL_ERROR) {
        return -1;
    }

    // Create a new top-level Tk window
    window = Tk_MainWindow(interp);

    // Set the title of the window using Tcl command
    Tcl_Eval(interp, "wm title . \"Simple Tk C App\"");

    // Set the size of the window
    Tk_GeometryRequest(window, 800, 600);

    // Start the Tk event loop
    Tk_MainLoop();

    return 0;
}

Honestly maybe a tenth of a second faster than Tcl example but much less ergonomic. I did compile everything with -O2 when I could, but I think that is just a testament to how well Tcl and Tk are integrated.

Next I looked at the Fast Light Toolkit (FLTK), a cross platform GUI library I used many years ago when I first attempted to learn C++. Fun fact this is also the GUI library used to create the Eureka Doom Level editor, a cross platform tool I have extensive experience with.

It's certainly got the right name for this test, so let's see if it can live up to it.

#include 
#include 

int main() {
    // Create a window with dimensions 800x600
    Fl_Window *window = new Fl_Window(800, 600);

    // Set the window label
    window->label("My FLTK Window");

    // Show the window
    window->show();

    // Run the FLTK event loop
    return Fl::run();
}

Aaand at 1.1 seconds we have a new winner. That was really impressive. I would be curious to see how much of it's speed it could retain with a more complex application.

To finish this test off, I'll add two late additions. A simple Qt6 app using QT Widgets and C++

#include 
#include 

int main(int argc, char *argv[])
{
    QApplication app(argc, argv);

    QWidget window;
    window.resize(800, 600);
    window.setWindowTitle("Simple Qt6 App on Raspberry Pi Zero");
    window.show();

    return app.exec();
}

and a QtQuick example with qml

import QtQuick 2.15
import QtQuick.Window 2.15

Window {
    width: 800
    height: 600
    visible: true
}

Compiling these apps made my little Pi Zero chug, but it was worth it. At a little under 2.5 seconds for QT widgets, and 2.75 seconds for the QML example, that's not bad. QT is used a lot in the auto industry for infotainment systems which are not known for their specs, so it's not surprising that we could get something running on our RPI Zero. Qt also comes with all the same goodies that GTK and other fully featured cross platform GUIs would need to build any type of app, so its performance here indicates it would be pretty snappy on any computer more powerful than a Raspberry Pi Zero. It's not my favorite cross platform GUI for a few reasons, but every non electron based desktop GUI developer job I've seen uses it, so that's a strong point in its favor.

Now, I'll be the first one to tell you that there are a lot more that goes into making a GUI app than how fast it opens. There is the set of widgets that it provides, how cross platform it is, what its memory footprint looks, its ability to provide a native "look and feel" etc. All things we didn't test here. But I hope that I've at least convinced you to give one of these great libraries a try

Call To Action 📣

Subscribe now

Share Deus In Machina

SDL Tutorial Part 2: Drawing Text

Diego Crespo — Thu, 22 Feb 2024 13:01:47 GMT

This is part 2 of my SDL series using C. If you missed part 1 you can find it here.📌

Displaying Text Cheat Sheet

Displaying Text in SDL2 requires a very similar set of operations to how we handled creating a window and renderer in part 1. But before we get to displaying text, we need to include the appropriate header. Since SDL2 is designed to be modular, the libraries that work with text are in their own separate headers/files. To work with text, you will need SDL_ttf.h. You can either include it and the appropriate files directly in your project, linking them at compile time, or if you are on Linux like I am, use your package manager to install them globally. While we won’t be going over the specifics for how to set up SDL in this tutorial, the SDL Wiki provides a page on how to properly set up SDL that is pretty thorough. Now, for those of you who just want to see the code, here is the entire source.

#include 
#include 
#include 
#include 

#define SCREEN_WIDTH 1280
#define SCREEN_HEIGHT 720
#define MY_FONT "/usr/share/fonts/truetype/freefont/FreeSans.ttf"

int main(){
  if (SDL_Init(SDL_INIT_VIDEO) < 0){
    printf("Couldn't initialize SDL: %s\n", SDL_GetError());
    return EXIT_FAILURE;
  }
    // Initialize SDL_ttf
    if (TTF_Init() < 0) {
        printf("SDL_ttf could not initialize! TTF_Error: %s\n", TTF_GetError());
        SDL_Quit();
        return EXIT_FAILURE;
    }

  SDL_Window *window = SDL_CreateWindow("Drawing Text", SDL_WINDOWPOS_UNDEFINED,
					SDL_WINDOWPOS_UNDEFINED, SCREEN_WIDTH, SCREEN_HEIGHT, 0);

  if (!window){
    printf("Failed to open %d x %d window: %s\n", SCREEN_WIDTH, SCREEN_HEIGHT, SDL_GetError());
    return EXIT_FAILURE;
  }

  SDL_Renderer *renderer = SDL_CreateRenderer(window, -1, 0);

  if (!renderer){
    printf("Failed to create renderer: %s\n", SDL_GetError());
    return EXIT_FAILURE;
  }

  SDL_SetRenderDrawColor(renderer, 255,255,255,255);
  SDL_RenderClear(renderer);

  TTF_Font *font = TTF_OpenFont(MY_FONT, 64); // specify the path to your font file and font size

  if (!font){
    printf("Failed to load font: %s\n", TTF_GetError());
    return EXIT_FAILURE;
  }
  
  // Create surface with rendered text
  SDL_Color textColor = {0, 0, 0, 255}; // black color
  SDL_Surface *textSurface = TTF_RenderText_Solid(font, "Hello World!", textColor);

  if (!textSurface) {
    printf("Failed to create text surface: %s\n", TTF_GetError());
    return EXIT_FAILURE;
  }

  // Create texture from surface
  SDL_Texture *textTexture = SDL_CreateTextureFromSurface(renderer, textSurface);

  if (!textTexture){
    printf("Failed to create text texture: %s\n", SDL_GetError());
    return EXIT_FAILURE;
  }

  // Render text
  SDL_Rect textRect = {50, 50, textSurface->w, textSurface->h}; // rectangle where the text is drawn 
  SDL_RenderCopy(renderer, textTexture, NULL, &textRect);

  SDL_RenderPresent(renderer);
  SDL_Delay(2000);
  SDL_DestroyWindow(window);
  SDL_DestroyRenderer(renderer);
  SDL_Quit();

  return EXIT_SUCCESS;
}

Starting off we #include which gives us access to functions that help us display text. Since the ttf module works with fonts, which are normally .ttf files, we need a font to properly display the text in our Window. You can either download a font from the internet, or use a .ttf file that you have on your computer. To keep it simple, I’ll just use one of the default ones that comes on my computer. At the top of my program, I've created a define that simply points to a default font

#define MY_FONT "/usr/share/fonts/truetype/freefont/FreeSans.ttf"

On macOS you should be able to find some .ttf files located at /Library/Fonts and on Windows C:\Windows\Fonts

Following the same paradigm we used to initialize our video module, we have to initialize our ttf library. The same pattern is used, where we check to see if the return value of the TTF_Init function is less than zero, and if it is, we display an error to the user

   // Initialize SDL_ttf
   if (TTF_Init() < 0) {
       printf("SDL_ttf could not initialize! TTF_Error: %s\n", TTF_GetError());
       SDL_Quit();
       return EXIT_FAILURE;
   }

If the ttf library was installed and linked correctly then we can attempt to load a font using TTF_OpenFont. Again we want to check that the font was actually loaded and we do that with our if statement

  TTF_Font *font = TTF_OpenFont(MY_FONT, 64); // specify the path to your font file and font size

  if (!font){
    printf("Failed to load font: %s\n", TTF_GetError());
    return EXIT_FAILURE;
  }

Now that we've loaded our font we need to create a text surface using TTF_RenderText_Solid This surface is essentially an image containing the rendered text. Since this is an operation that can fail, we want to make sure that we check this using an if statement like we've done before

 // Create surface with rendered text
  SDL_Color textColor = {0, 0, 0, 255}; // black color
  SDL_Surface *textSurface = TTF_RenderText_Solid(font, "Hello World!", textColor);

  if (!textSurface) {
    printf("Failed to create text surface: %s\n", TTF_GetError());
    return EXIT_FAILURE;
  }

The next step uses SDL_CreateTextureFromSurface to convert our SDL_Surface (your rendered text) into an SDL_Texture. This is necessary because modern SDL2 uses the GPU for rendering, and textures are the GPU-friendly way of handling images. This conversion process uploads the surface data to the GPU memory, enabling efficient rendering. This operation can also fail, so it is important to check that it doesn't before doing anything with the texture

  // Create texture from surface
  SDL_Texture *textTexture = SDL_CreateTextureFromSurface(renderer, textSurface);

  if (!textTexture){
    printf("Failed to create text texture: %s\n", SDL_GetError());
    return EXIT_FAILURE;
  }

Finally, we call SDL_RenderCopy. This function tells SDL2 to copy the text texture to the renderer. The SDL_Rect textRect that is defined is used to position and size the text on the screen.

It is important to understand that nothing will be drawn on the screen yet. It's more like preparing what needs to be drawn. The actual drawing happens when you call SDL_RenderPresent.

  // Render text
  SDL_Rect textRect = {50, 50, textSurface->w, textSurface->h}; // rectangle where the text is drawn 
  SDL_RenderCopy(renderer, textTexture, NULL, &textRect);

If all goes well then when you run the code, you should see some text on the screen. With that, you are well on your way to learning the core components of using the SDL2 library. With just the information that you’ve learned from part 1 and part 2, you should be able to make a graphical version of the guess my number game, a common beginner programming task, but with an SDL flavored twist. You can even create a rudimentary text adventure, if you were so inclined.

One final note. Between this and the last tutorial, you may be seeing a lot of repetitive code that might be better off abstracted away in functions. That’s good, and means you are getting familiar with the patterns in the library. Opinions about the structure of code should be based off the needs of the code base. While I’m keeping it simple for these tutorials, eventually we should abstract away some common patterns.

FAQ

Why Not Draw Directly to the Renderer?

SDL2 uses two main types of objects for drawing: SDL_Surface and SDL_Texture. SDL_Surface is a CPU-based bitmap, useful for manipulating pixel data directly. SDL_Texture, on the other hand, is GPU-based. Modern SDL2 applications use the GPU for rendering (via SDL_Renderer), which is faster and more efficient but requires textures, not surfaces.

The reason you can't directly render the font to the renderer is that SDL2_ttf is designed to work with surfaces, as historically SDL handled rendering with surfaces before textures and GPU rendering were introduced.

Does that mean that SDL2 can render just the surface instead of the texture if I don't want to use the GPU?

Yes, SDL2 can render using just an SDL_Surface without converting it to an SDL_Texture if you prefer or need to render without using the GPU. This approach is known as "software rendering" and it's done entirely by the CPU, bypassing the GPU.

Software rendering with SDL_Surface was the primary method used in SDL before SDL2 introduced hardware-accelerated rendering with SDL_Texture and SDL_Renderer. While software rendering is less efficient and slower than hardware accelerated rendering, it can be useful in certain scenarios, like on systems with limited or no GPU capabilities. Instead of using a renderer like we are using you would instead create a SDL_Surface with SDL_GetWindowSurface

SDL_Surface *windowSurface = SDL_GetWindowSurface(window);

Call To Action 📣

Subscribe now

Share Deus In Machina

SDL Tutorial Part 1: Opening A Window

Diego Crespo — Thu, 15 Feb 2024 13:01:23 GMT

I was recently learning SDL2, and I found the lack of tutorials in plain C frustrating. I've decided to be the change I wanted to see in the world, and write my own. This will be a multi-part series, and will serve as a quick reference for how to do common things in SDL2. I've listed the main functions at the top of the tutorial kind of like the Raylib Cheatsheet, if you just need a quick reference.

Before we dive in, I want to point out an immensely helpful resource that will be useful as you become more experienced. The SDL2 Wiki contains an index which has a link to every SDL function wiki page. This will teach you everything you need to know about calling the function and handling its output properly. If a big list of functions is too disjointed, there is also a wikipage that links to the same resources, but organized into API categories as well. This can be useful as common operations usually require multiple functions that are related.

Open a Window Cheat Sheet

Tutorial

The simplest and fastest way to get a window open in SDL2 is to write the following code.

#include 
#include 
#include 

#define SCREEN_WIDTH 1280
#define SCREEN_HEIGHT 720

int main() {
    SDL_Window *window = SDL_CreateWindow("Example: 0", SDL_WINDOWPOS_UNDEFINED, SDL_WINDOWPOS_UNDEFINED, SCREEN_WIDTH, SCREEN_HEIGHT, 0);
    SDL_Renderer *renderer = SDL_CreateRenderer(window, -1, 0);
    SDL_SetRenderDrawColor(renderer, 255,255,255,250);
    SDL_RenderClear(renderer);
    SDL_RenderPresent(renderer);
    SDL_Delay(2000);
    SDL_DestroyWindow(window);
    SDL_DestroyRenderer(renderer);
    SDL_Quit();
    return EXIT_SUCCESS;
}

The start of something great

But this is not recommended. It's possible that many of the functions that we just called can fail, so we want to include a few more checks to make sure that everything is working correctly. Let's do that now

int main(){
  if (SDL_Init(SDL_INIT_VIDEO) < 0){
    printf("Couldn't initialize SDL: %s\n", SDL_GetError());
    return EXIT_FAILURE;
  }

  SDL_Window *window = SDL_CreateWindow("Example: 0", SDL_WINDOWPOS_UNDEFINED,
					SDL_WINDOWPOS_UNDEFINED, SCREEN_WIDTH, SCREEN_HEIGHT, 0);
  if (!window){
    printf("Failed to open %d x %d window: %s\n", SCREEN_WIDTH, SCREEN_HEIGHT, SDL_GetError());
    return EXIT_FAILURE;
  }

  SDL_Renderer *renderer = SDL_CreateRenderer(window, -1, 0);
  SDL_SetRenderDrawColor(renderer, 255, 255, 255, 250);
  SDL_RenderClear(renderer);
  SDL_RenderPresent(renderer);
  SDL_Delay(2000);
  SDL_DestroyWindow(window);
  SDL_DestroyRenderer(renderer);
  SDL_Quit();
  return EXIT_SUCCESS;
}

The first thing we do is call SDL_Init and pass the flag SDL_INIT_VIDEO, which initializes the video subsystem. SDL has many initialization flags that you can bitwise OR (|) together to initialize multiple subsystems at once. For example like this

    // Initialize SDL with video and audio subsystems
    if (SDL_Init(SDL_INIT_VIDEO | SDL_INIT_AUDIO) != 0) {
        printf("SDL could not initialize! SDL_Error: %s\n", SDL_GetError());
        return 1;
    } else {
        printf("SDL video and audio initialized successfully.\n");

But we will stick to the video subsystem for now. Next we will call SDL_CreateWindow. It takes in a title, x and y for the window position, width and height for the window size, and some flags. If we go to the wikipage for SDL_CreateWindow we will see the function prototype, a description of its parameters (including flags that can be passed), and the expected return values on failure. This is how I knew what that I wanted to pass SDL_WINDOWPOS_UNDEFINED for the x and y, and let the operating system handle window placement.

Moving on, we create our renderer with SDL_CreateRenderer. We give it the window we created, so we can render to it, and pass its index parameter -1 to initialize the first rendering driver that can support our requested parameter flag of 0. You will always pass -1 as the index parameter unless you have a good reason not to. For the render flag we pass 0 which defaults to SDL_RENDERER_ACCELERATED (using hardware acceleration for the graphics) if it is supported on your system. If you want to be more explicit you can pass SDL_RENDERER_ACCELERATED instead. Just like with the SDL window, we write an if statement to check if the renderer was created so that we don't accidentally use it if it hasn't been initialized.

Next we call SDL_SetRenderDrawColor() and pass it our renderer, as well as 4 integers with values from 0-255. The 4 integers correspond to Red, Green, Blue, and Alpha values of a color. The alpha controls the transparency of the color, while setting the R, G, and B values to various levels allow you to create any color you want between 0,0,0,255 (Black) and 255,255,255,255 (White). SDL_RenderClear() takes our renderer and clears it with the white color we set using SDL_SetRenderDrawColor. SDL_RenderPresent() displays our graphics to the screen (just a white window for now) and then we call SDL_Delay(2000) which specifies a number of milliseconds to wait before moving onto the next piece of code. Normally the rendering code will be in a while loop that will constantly update until the user quits the game, but for simplicities' sake, we will stick with a delay for now.

SDL_DestroyWindow safely removes our window once we are done with it. If for whatever reason the window was not initialized, it is still safe to call this on the window variable. The same goes for SDL_DestroyRenderer. It does the same thing as SDL_DestroyWindow and cleans up our renderer. Finally we call SDL_Quit which cleans up the systems we initialized before we close our program.

And that is it, we've created our first window in SDL. In the next tutorial we will look at how to draw text on the screen.

One last thing to note. I will not be abstracting many things to functions or multiple files in these first few tutorials. I believe it's important when you are learning a new library, to keep it as simple as possible. As the code gets longer this will become a bigger pain point, and that is good. You will have a much clearer picture about what we should abstract, and why we should abstract it, as you become more familiar with SDL. Feel free to come up with your own abstractions as you see fit

FAQ

Why do we clear the screen with a color?

We clear the screen with a color so that the previous frames pixels don't interfere with the next frames pixels. For example, if we have a square on the screen that we are moving with the WASD keys, if we don't clear the screen, then square will smear across the screen as the old pixels from it's old position are not "wiped" away.

Why SDL and not Raylib?

Raylib is an awesome C library for creating games, and Raysan specifically made it for teaching. He did a comparison at one point between the two frameworks, though it is a little dated at this point. It is easier to learn and make new FFI bindings for Raylib than it is for SDL and Raylib comes with more convenience features out of the box. You are not making a mistake if you choose Raylib. But SDL has been around over twice as long as Raylib. Many of the open source projects I'm interested in contributing to (Doom Source Ports, OpenMW, and others were started before Raylib existed and use SDl2. Since these projects use SDL so will I. SDL2 is also a little lower level than Raylib, and requires you to work at a lower level with it's subsystems. This means instead of placing drawing code between a BeginDrawing and EndDrawing block like in Raylib, you have to set up the renderer, create textures, copy them to the GPU, before finally presenting it to the screen. I find this process to be enlightening.

Call To Action 📣

Subscribe now

Share Deus In Machina

The continued rise of Handheld gaming

Diego Crespo — Thu, 08 Feb 2024 13:00:37 GMT

My very first handheld game was probably the ring toss game… But in all seriousness it was my fathers Game Boy in the late 90s. I distinctly remember playing Star Wars and I don’t think I ever beat it.

Game Boy by Evan-Amos

This is what peak gaming in 1992 looked like

I remember handheld gaming being all the rage growing up, and I have fond memories of playing on Game Boys, Game Boy Colors, Game Boy Advances, PSPs, and DS'. Like many, after about the PSP era, I moved away from consoles and handhelds in favor of PCs. But since getting back into handheld gaming with my Steam Deck, I was shocked to see there had been a Cambrian explosion of handhelds. Many are from major players in the tech space, though some are from companies I've never heard of. It feels like we are in a renaissance, or, that is what I thought the angle of this article was going to be when I started writing it. But in actually researching handheld history, I've come to the conclusion that while I had abandoned it, it hadn't gone anywhere.

A lightning Tour

Let's take a look back at one of the earliest handhelds. For the sake of brevity I'll only consider handhelds that could play more than one game, so no Game and Watches, or single game electronics. In this vein the honor to the first handheld gaming console that fits our criteria is the Microvision created in 1979.

Microvision - Evan-Amos

It looked more like the iClicker I had in College than a gaming device, but it had 12 different games that it could play. Clocking in on a CPU that ran at 1kHZ, a frequency I have never seen next to a CPU, It's honestly hugely impressive it could play anything.

But just 10 years later we would be introduced to one of the most popular handhelds of all time, by one of handheld gaming’s most prolific supporters, Nintendo. The Game Boy would be a smash hit, bringing handheld gaming to the masses. Nintendo has certainly not slowed down since then and if we look at the timeline from the Game Boy to the Switch we get a history that spans from 1989-2021 with no signs of stopping.

Game Boy: Released on April 21, 1989, in Japan, July 31, 1989, in North America, and September 28, 1990, in Europe. The OG.
Game Boy Pocket: Released in 1996, it was a smaller, lighter Game Boy that required fewer batteries.
Game Boy Light: Released only in Japan on April 14, 1998, this model featured a backlit screen.
Game Boy Color: Released on October 21, 1998, in Japan, November 18, 1998, in North America, and November 23, 1998, in Europe. Backwards compatible with the Game Boy! Contained a 15-bit color palette instead of the 4 in the original Game Boy.
Game Boy Advance: Released on March 21, 2001, in Japan, June 11, 2001, in North America, and June 22, 2001, in Europe. Backwards compatible again! Hardware that was more powerful than the SNES from a decade prior.
Game Boy Advance SP: Released in February 2003, it featured a front-lit screen and a clamshell design.
Game Boy Micro: Released on September 13, 2005, in Japan, and subsequently in other regions. A slimmer version of the original GBA.
Nintendo DS: Released on November 21, 2004, in North America, December 2, 2004, in Japan, February 24, 2005, in Australia, and March 11, 2005, in Europe. Successor to and backwards compatible with the Game Boy Advance.
Nintendo DS Lite: Released in early 2006, it was a slimmer and lighter version of the original DS.
Nintendo DSi: Released on November 1, 2008, in Japan, and in 2009 in other regions. No slot for GBA Games. Two cameras, SD Card slot, and other enhancements.
Nintendo DSi XL/LL: Released in 2009, it was a DS that larger screens.
Nintendo 3DS: Released on February 26, 2011, in Japan, March 25, 2011, in Europe, March 27, 2011, in North America, and March 31, 2011, in Australia. Successor to the DS with the most notable feature being the 3d graphics, 3 cameras, and AR capabilities.
Nintendo 3DS XL: Released in 2012, this version featured larger screens.
Nintendo 2DS: Released on October 12, 2013, as a budget version of the 3DS without 3D capability.
New Nintendo 3DS: Released on October 11, 2014, in Japan, and 2015 in other regions. Enhanced performance, better 3d, a C-stick, and built in NFC.
New Nintendo 3DS XL: Released alongside the New Nintendo 3DS, with larger screens.
New Nintendo 2DS XL: Released on July 13, 2017, as an upgraded version of the 2DS.
Nintendo Switch: Released on March 3, 2017, a hybrid console, usable as docked as a home console and undocked as a handheld device.
Nintendo Switch Lite: Released on September 20, 2019, features non removeable joycons.
Nintendo Switch OLED: Released on October 8, 2021, it's a Switch, but with a nicer screen.

Phew! That was a lot. While compiling this list a couple of things became apparent to me.

Nintendo has been keeping the handheld dream alive, even after other hardware manufactures had left the space.
Nintendo loves to sell you the same handheld over and over again.

Multiple Game Boy's DS' and Switches. I wouldn't be surprised if we see a Switch XL at some point. And you already know there will be a Switch 2 Lite.

But while Nintendo never stopped making Handhelds, the list of handhelds we got from 1989 to ~2013 ranged for weird, wacky, cool, and everything in between. Here are some of them

Atari Lynx (1989) - First handheld console with a color LCD screen.
Sega Game Gear (1990) - Competitor to the Nintendo Game Boy, known for its backlit color screen.
Sega Nomad (1995) - Handheld version of the Sega Genesis, allowing players to use Genesis cartridges.
Tapwave Zodiac (2004) - Short lived device that functioned as both a PDA and a gaming console.
Sony PlayStation Portable (PSP) (2005)- Known for its powerful hardware, multimedia capabilities, and wide game selection. I had one with a skin covered in green skulls, and it was incredible.
GP2X Wiz (2009) - An open-source, Linux-based handheld console produced by GamePark Holdings.
Pandora (2010) - A combination of a handheld game console and a mini PC, running Linux.
Sony PlayStation Vita (2012) - Successor to the PSP. Featured a touch screen, rear touchpad, and high-quality graphics. Please bring it back Sony.
Nvidia Shield Portable (2013) - An Android-based device that combined a game controller with a screen, capable of streaming games from a PC.

Yes this was a real Ad. The early 2000s were wild

One interesting thing to note about this time. With devices like the Nvidia Shield, we begin to see the rise of handheld streaming devices. This is a lot more common now with the likes of the PlayStation Portal, and Logitech G Cloud (with an honorable mention to the Steam Link).

I would be remiss if I didn't at least talk about the most common forms of handheld gaming of our generation, mobile gaming. While I do remember playing games like Dragon's lair on my father's Sprint flip phone...

It was when we got smartphones where I really saw the quality of mobile gaming blow up. I remember games like Zenonia back in 2009 being awesome. I couldn't believe I was playing such a fully featured rpg right in the palm of my hand at that time. I remember sitting back and playing it for hours on my landscaped iPod in a wooden chair with no cushion, my back and eyes would never permit that now.

This was pre candy crush too, so the dark patterns were either non existent, or much simpler at the time. You could even buy a game once, and never pay for it again (shocking I know). And remember Infinity Blade? Mobile gaming may have truly peaked in 2010.

No longer available on the App Store :’(

But if we include phones on this tour, we'd be here all day so let's get back to dedicated handhelds. Not only have we had great handhelds for the last 35 years, if you look at the top 5 best selling consoles of all time...

PlayStation 2
Nintendo DS
Nintendo Switch
Game Boy/Game Boy Color
PlayStation 4

We see that three out of the top 5 are actually handhelds. So with that fact, and with this article having mentioned over two dozen handheld devices at this point, It's safe to say handhelds, and handheld gaming, never went away. In fact I'd say it's more popular now that it has ever been!

The experience is also so much better than it used to be. For one, I don't have to go through dozens of AA batteries a month anymore. And two, they don't feel cut down. Using my previously mentioned Star Wars Game Boy game as an example, I recently found out that this is what the NES version looked like.

It looks so good!

Shocking! I would have loved to have my Game Boy version look like that back in the 90s. By the Game Boy Advanced era in 2001, we were able to create incredible 2D handheld experiences, but we were only just getting to the point where we could render anything impressive in 3D. Sure you could program a 3D GBA game if you were a literal programming God like Randal Linden, but unless you are willing to write 200,000 lines of highly optimized assembly then we still needed a few generations before this was feasible for your average programmer.

I don't know how this is even possible

But fast forward to the PSP era, and we were beginning to see what no compromise 3d handheld experiences looked like.

And by the time we got to the Switch, it was clear that while certain games might have struggled to hit their 30fps target sometimes, we weren't getting stripped down version of a game originally made for PCs and consoles anymore.

And now in the more modern times (a sentence that will age like milk) we see many other players in the space besides Sony and Nintendo. There are Android, Windows, and Linux based handhelds, some from well known brands, and others from rising stars...

GPD XD: 2015
GPD Win: 2016
GPD XD Plus: 2018
Abernic RG351P: 2020
Retroid Pocketc 2S: 2020
Miyoo Mini: 2021
Ayn Odin: 2022
Steam Deck: 2022
Miyoo Mini +: 2023
Logitech G Cloud: 2023
Lenovo Legion Go: 2023
Ayn Odin 2: 2023
Asus Rog Ally: 2023
Steam Deck OLED: 2023
PlayStation Portal: 2023
Msi Claw: Sometime in 2024

And it's exciting. It feels like we've hit an inflection point with handhelds. With devices like my Steam Deck, I don't feel like I'm giving anything up when I'm using it. But at the same time, I'm excited for every new iteration. It feels just like how it was in the late 90s and 2000s where every console generation, or graphics card generation, was a noticeable improvement over the last. And with the Switch 2 rumored to be announced later this year, it seems like that excitement isn't going to be slowing down anytime soon.

Call To Action 📣

Subscribe now

Share Deus In Machina

Lets compile Linux DOOM

Diego Crespo — Thu, 01 Feb 2024 13:00:46 GMT

In honor of Doom’s 30th birthday, I decided I would try to compile the Doom source code on GitHub. Since I had no idea what I was doing, I searched around and found the this incredible video by hot_coffee_guy. It has no sound, but by just watching it, and with a little trial and error, I was able to get the game up and running.

I encountered some challenges along the way, and want to fill in explanations where I can, both as a record for anyone who might experience the same issue, and for my own edification so let’s get to it!

The Setup

The first step is to get the source code. It is available in the id-software repository here. You can clone it, or fork it, whichever you desire. I decided to make a fork and named it DOOMLEARN.

Using CMake and building with an IDE

The first time I did this, I used Make and the Makefile that came with the project. However, on my second round for this article, I wanted to make code edits using CLion, so I opted to create a CMake file. I’ve added a CMakeLists.txt file to my version of the project, and if you want to use it, you can find it here.

Using Make

If you use Make the setup is already done for you. Just navigate to where you downloaded the DOOM source code, cd into linuxdoom-1.10, and run…

# the executable automatically writes to a folder called linux
mkdir linux
make

You can expedite the compilation process by specifying the -j parameter with the desired number of parallel jobs

make -j8

Getting Started

Now that the prep work is out of the way, we need to build the project. You can build it with either Make or CMake. I'll demonstrate the Make method first before transitioning to the CMake build for the rest of the article.

deca@pop-os:~/programming/c_programming/DOOMLEARN/linuxdoom-1.10$ make -j8
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c doomdef.c -o linux/doomdef.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c doomstat.c -o linux/doomstat.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c dstrings.c -o linux/dstrings.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c i_system.c -o linux/i_system.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c i_sound.c -o linux/i_sound.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c i_video.c -o linux/i_video.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c i_net.c -o linux/i_net.o
gcc   -g -Wall -DNORMALUNIX -DLINUX  -c tables.c -o linux/tables.o
dstrings.c:25:1: warning: ‘rcsid’ defined but not used [-Wunused-const-variable=]
   25 | rcsid[] = "$Id: m_bbox.c,v 1.1 1997/02/03 22:45:10 b1 Exp $";
      | ^~~~~
Assembler messages:
Fatal error: can't create linux/dstrings.o: No such file or directory
make: *** [Makefile:91: linux/dstrings.o] Error 1
make: *** Waiting for unfinished jobs....
doomdef.c:26:1: warning: ‘rcsid’ defined but not used [-Wunused-const-variable=]
   26 | rcsid[] = "$Id: m_bbox.c,v 1.1 1997/02/03 22:45:10 b1 Exp $";
      | ^~~~~
Assembler messages:

After building the project we get a lot of warnings, which is to be expected for a game that came out in 1993, but eventually we hit our first fatal error.

i_video.c:49:10: fatal error: errnos.h: No such file or directory
   49 | #include 
      |          ^~~~~~~~~~

In CLion, since I setup my CMakeLists.txt file, I can build the program in one click by pressing the run button.

To make finding errors easier, I recommend suppressing most of the warnings to focus on compilation errors that stop the build. I did this by adding the following commands to my CMakeLists.txt file

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-but-set-variable")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-unused-but-set-variable")

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-pointer-to-int-cast")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-pointer-to-int-cast")

set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-const-variable")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-unused-const-variable")

These commands effectively suppress most of the warnings the project generates. Now, let's address the initial problem…

i_video.c:49:10: fatal error: errnos.h: No such file or directory
   49 | #include 
      |          ^~~~~~~~~~

Thankfully our first error is simple. There is no header file called errnos.h. It is errno.h and it exists in the C standard library. To fix this we just have to open the i_video.c file and remove the “s” from the header.

#include

Now lets recompile the project again. This time we get a different error

gcc   -Wno-unused-but-set-variable -Wno-pointer-to-int-cast -Wno-unused-const-variable -g -Wall -DNORMALUNIX -DLINUX -g -fdiagnostics-color=always -MD -MT CMakeFiles/linuxxdoom.dir/m_misc.c.o -MF CMakeFiles/linuxxdoom.dir/m_misc.c.o.d -o CMakeFiles/linuxxdoom.dir/m_misc.c.o -c /home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c
/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c:257:48: error: initializer element is not constant
  257 |     {"sndserver", (int *) &sndserver_filename, (int) "sndserver"},
      |                                                ^
/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c:257:48: note: (near initialization for ‘defaults[14].defaultvalue’)
/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c:264:35: error: initializer element is not constant
  264 |     {"mousedev", (int*)&mousedev, (int)"/dev/ttyS0"},
      |                                   ^
/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c:264:35: note: (near initialization for ‘defaults[16].defaultvalue’)
/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/m_misc.c:265:37: error: initializer element is not constant
  265 |     {"mousetype", (int*)&mousetype, (int)"microsoft"},
      |                                     ^

The issue is related to a cast on the defaultvalue field in the default_t struct in our m_misc.c file.

typedef struct
{
    char*	name;
    int*	location;
    int		defaultvalue;
    int		scantranslate;		// PC scan code hack
    int		untranslated;		// lousy hack
} default_t;

I think what is happening is we are using the pointer to the string as an number, but this won’t work on 64 bit systems. A pointer is 8 bytes on 64bit systems and and an int holds four. We can fix this by changing the defaultvalue to a long long (or long int whichever you prefer) which will be large enough to hold the address of our pointer

typedef struct
{
    char*	name;
    int*	location;
    long long	defaultvalue;
    int		scantranslate;		// PC scan code hack
    int		untranslated;		// lousy hack
} default_t;

This means we need to make a change on every cast to defaultvalue in the file to be a long long instead of an int. The first is on line 257

// UNIX hack, to be removed. 
#ifdef SNDSERV
    {"sndserver", (int *) &sndserver_filename, (long long) "sndserver"},
    {"mb_used", &mb_used, 2},

The next are on lines 264 and 265

#ifdef LINUX
    {"mousedev", (int*)&mousedev, (long long)"/dev/ttyS0"},
    {"mousetype", (int*)&mousetype, (long long)"microsoft"},

and finally 288-297

    {"chatmacro0", (int *) &chat_macros[0], (long long) HUSTR_CHATMACRO0 },
    {"chatmacro1", (int *) &chat_macros[1], (long long) HUSTR_CHATMACRO1 },
    {"chatmacro2", (int *) &chat_macros[2], (long long) HUSTR_CHATMACRO2 },
    {"chatmacro3", (int *) &chat_macros[3], (long long) HUSTR_CHATMACRO3 },
    {"chatmacro4", (int *) &chat_macros[4], (long long) HUSTR_CHATMACRO4 },
    {"chatmacro5", (int *) &chat_macros[5], (long long) HUSTR_CHATMACRO5 },
    {"chatmacro6", (int *) &chat_macros[6], (long long) HUSTR_CHATMACRO6 },
    {"chatmacro7", (int *) &chat_macros[7], (long long) HUSTR_CHATMACRO7 },
    {"chatmacro8", (int *) &chat_macros[8], (long long) HUSTR_CHATMACRO8 },
    {"chatmacro9", (int *) &chat_macros[9], (long long) HUSTR_CHATMACRO9 }

Now let’s build the project again.


/usr/bin/ld: errno: TLS definition in /lib/x86_64-linux-gnu/libc.so.6 section .tbss mismatches non-TLS reference in CMakeFiles/linuxxdoom.dir/i_sound.c.o
/usr/bin/ld: /lib/x86_64-linux-gnu/libc.so.6: error adding symbols: bad value
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Cool a linker error! That means that we were able to compile our code to obj files. It looks like a thread local storage error in the way we are using errno. errno is a global variable used in C to indicate the error status of many library functions, particularly those involving system calls. When these functions fail, they typically set errno to an integer value representing what kind of error occurred. Not totally unexpected as we changed the way we handled errno earlier on in the program. The issue is in the i_sound.c so let’s go there.

errno is thread safe, but we were using a non standard version of errno in our program. Searching for it in our file we see the offending code on line 166. We have to delete the line extern int errno…

myioctl
( int	fd,
  int	command,
  int*	arg )
{   
    int		rc;
    extern int	errno;
    
    rc = ioctl(fd, command, arg);  
    if (rc < 0)
    {
	fprintf(stderr, "ioctl(dsp,%d,arg) failed\n", command);
	fprintf(stderr, "errno=%d\n", errno);
	exit(-1);
    }
}

and then include the errno.h header file like we’ve done before. Then we will be using the errno value that comes with the standard library.

#include 
#include 
#include 
#include

Building it again and we get an executable!

Buuut it will segfault when we try to run it

What is the error saying? No files found? Let’s look where this error is called

void W_InitMultipleFiles (char** filenames)
{	
    int		size;
    
    // open all the files, load headers, and count lumps
    numlumps = 0;

    // will be realloced as lumps are added
    lumpinfo = malloc(1);	

    for ( ; *filenames ; filenames++)
	W_AddFile (*filenames);

    if (!numlumps)
	I_Error ("W_InitFiles: no files found");
    
    // set up caching
    size = numlumps * sizeof(*lumpcache);
    lumpcache = malloc (size);
    
    if (!lumpcache)
	I_Error ("Couldn't allocate lumpcache");

    memset (lumpcache,0, size);
}

Line 292 of w_wad.c is where this function is defined, but where is it used? Using our IDE we find its first usage in d_main.c in line 1021

    W_InitMultipleFiles (wadfiles);

The variable wadfiles is an array of wadfiles that are defined earlier in our d_main.c on line 37. We set the array of wadfiles in our function called D_AddFile on line 543

void D_AddFile (char *file)
{
    int     numwadfiles;
    char    *newfile;
	
    for (numwadfiles = 0 ; wadfiles[numwadfiles] ; numwadfiles++)
	;

    newfile = malloc (strlen(file)+1);
    strcpy (newfile, file);
	
    wadfiles[numwadfiles] = newfile;
}

And our first usage of our D_AddFile comes on line 621

    if (M_CheckParm ("-shdev"))
    {
	gamemode = shareware;
	devparm = true;
	D_AddFile (DEVDATA"doom1.wad");
	D_AddFile (DEVMAPS"data_se/texture1.lmp");
	D_AddFile (DEVMAPS"data_se/pnames.lmp");
	strcpy (basedefault,DEVDATA"default.cfg");
	return;
    }

Hmm seems like we have a bunch of parameters which tell Doom where the wadfile is. Looking further up the lines of source code, we get our indication of what we need to do

#ifdef NORMALUNIX
    char *home;
    char *doomwaddir;
    doomwaddir = getenv("DOOMWADDIR");
    if (!doomwaddir)
	doomwaddir = ".";

    // Commercial.
    doom2wad = malloc(strlen(doomwaddir)+1+9+1);
    sprintf(doom2wad, "%s/doom2.wad", doomwaddir);

    // Retail.
    doomuwad = malloc(strlen(doomwaddir)+1+8+1);
    sprintf(doomuwad, "%s/doomu.wad", doomwaddir);
    
    // Registered.
    doomwad = malloc(strlen(doomwaddir)+1+8+1);
    sprintf(doomwad, "%s/doom.wad", doomwaddir);

It appears that we can set an environment variable to tell DOOM where to look for our wad. If we don’t set this variable, it will assume it is in whatever directory the wadfile is called. That’s useful information, but before we can do anything with that, we need to get an actual wad file.

Getting a DOOM WAD

There are two ways you can get this.

If you don’t own DOOM you can go to the doomworld website and download the shareware version ( this is a downloadable link)
If you own a copy of DOOM, get the wad from the game files

If you download the shareware version you will want to rename the version from DOOM1.wad to doom1.wad (that’s the string the code looks for), and then follow the steps in the next section. I own DOOM on GOG so I will outline how do get the wad file from there because it requires a few steps. First go to your GOG library and find your DOOM Files

Clicking on the 18Mb Ultimate Doom Link will download the game. In my downloads I now have a file called setup_the_ultimate_doom_1.9_(28044).exe. To get the wad from the .exe, I need to extract it using a tool called innoextract. Make a folder called doom_unpack, and move the setup.exe to it. Once that is done, in the terminal run

innoextract setup_the_ultimate_doom_1.9_(28044).exe

Doing this will extract all the files in the game including our DOOM.WAD

Remember! In the code, doom.wad is all lowercase. Make sure to change the name!

// looking for lowercase doom
D_AddFile (DEVDATA"doom.wad");

Once you have the doom1.wad or doom.wad

After moving our doom.wad into our linuxdoom-1.10/linux folder (where the executable is), lets start running our code again in our IDE.

V_Init: allocate screens.
M_LoadDefaults: Load system defaults.
Z_Init: Init zone memory allocation daemon. 
W_Init: Init WADfiles.
Error: W_InitFiles: no files found

What gives! The error is still there? Before you freak out, it’s because of the way the IDE is calling the executable and the paths. If you manually cd into the linux folder, and run the executable (with the doom1 or doom wad in the same folder) you will see two things.

V_Init: allocate screens.
M_LoadDefaults: Load system defaults.
Z_Init: Init zone memory allocation daemon. 
W_Init: Init WADfiles.
 adding ./doom.wad
===========================================================================
                 Commercial product - do not distribute!
         Please report software piracy to the SPA: 1-800-388-PIR8
===========================================================================
M_Init: Init miscellaneous info.
R_Init: Init DOOM refresh daemon - [.Segmentation fault (core dumped)

One is that the game successfully added the wad, and two, that we have a new error. If you remember back in the code that deals with the DOOM wad, there was a section where it checked the DOOMWADDIR environment variable, which sets the path to our DOOMWADDIR.

    doomwaddir = getenv("DOOMWADDIR");
    if (!doomwaddir)
	doomwaddir = ".";

Let’s try setting that environment variable. Since this happens during the “run” phase of the program, we have to make sure the environment variable is set during that phase in our IDE

Since I’m using Clion this requires clicking the drop down next to the hammer icon and clicking “Edit Configurations”. This will open up a window that will allow me to set an environment variable.

If you are following along using Make and the terminal the same effect can be had by just running this code before running the executable

export DOOMWADDIR=/path/to/folder/where/doom.wad/is

After making this change, you should see a new erro when you build the project.

M_LoadDefaults: Load system defaults.
Z_Init: Init zone memory allocation daemon. 
W_Init: Init WADfiles.
 adding /home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/doom.wad
===========================================================================
                 Commercial product - do not distribute!
         Please report software piracy to the SPA: 1-800-388-PIR8
===========================================================================
M_Init: Init miscellaneous info.
R_Init: Init DOOM refresh daemon - [.
Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)

Progress! Now it’s crashing on the R_Init DOOM step. Which is called on line 1095 in d_main.c

    printf ("R_Init: Init DOOM refresh daemon - ");
    R_Init ();

The definition of this function is defined in line 773 in r_main.c

void R_Init (void)
{
    R_InitData ();
    printf ("\nR_InitData");
    R_InitPointToAngle ();
    printf ("\nR_InitPointToAngle");
    R_InitTables ();
    // viewwidth / viewheight / detailLevel are set by the defaults

Considering that I didn’t see the printf ("\nR_InitData") in the terminal, I’m assuming the crash was in R_InitData. That is defined on line 654 of r_data.c

void R_InitData (void)
{
    R_InitTextures ();
    printf ("\nInitTextures");
    R_InitFlats ();
    printf ("\nInitFlats");
    R_InitSpriteLumps ();
    printf ("\nInitSprites");
    R_InitColormaps ();
    printf ("\nInitColormaps");
}

More printfs that didn’t print to the terminal, The errors must be in R_InitTextures.

I_Init: Setting up machine state.
Could not start sound server [sndserver]
D_CheckNetGame: Checking network game status.
startskill 2  deathmatch: 0  startmap: 1  startepisode: 1
player 1 of 1 (1 nodes)
S_Init: Setting up sound.
S_Init: default sfx volume 8
HU_Init: Setting up heads up display.
ST_Init: Init status bar.
Error: xdoom currently only supports 256-color PseudoColor screens

I admit this is where I got stuck, but hot_coffee_guy’s video we see that we have to make a few changes in r_data.c including…

Adding and include for at the top of the file.

static const char
rcsid[] = "$Id: r_data.c,v 1.4 1997/02/03 16:47:55 b1 Exp $";

#include 

#include "i_system.h"

This allows the use of fixed width integer types like intptr_t. These types have a consistent size across different platforms. Next we have to change our maptexture_t struct to be an int columndirectory instead of an void **columndirectory

typedef struct
{
    char		name[8];
    boolean		masked;	
    short		width;
    short		height;
    int                     columndirectory;
    short		patchcount;
    mappatch_t	patches[1];
} maptexture_t;

Next we remove the hard coding of the pointer sizes to the textures on line 484-487 in our r_data.c. We can do this and make the code more flexible using the sizeof operator

    textures = Z_Malloc (numtextures*sizeof(*textures), PU_STATIC, 0);
    
texturecolumnlump = Z_Malloc (numtextures*sizeof(*texturecolumnlump), PU_STATIC, 0);
    texturecolumnofs = Z_Malloc (numtextures*sizeof(*texturecolumnofs), PU_STATIC, 0);
    texturecomposite = Z_Malloc (numtextures*sizeof(*texturecomposite), PU_STATIC, 0);

On line 644 in our R_InitColormaps function, we want to change the cast from int to an intptr_t. This concludes are changed to r_data.c

void R_InitColormaps (void)
{
    int	lump, length;
    
    // Load in the light tables, 
    //  256 byte align tables.
    lump = W_GetNumForName("COLORMAP"); 
    length = W_LumpLength (lump) + 255; 
    colormaps = Z_Malloc (length, PU_STATIC, 0); 
    colormaps = (byte *)( ((intptr_t)colormaps + 255)&~0xff); 
    W_ReadLump (lump,colormaps); 
}

The next file we wanto to change is r_draw.c. In r_draw.c we want to add the #include to the top again. Then we want to jump down to line 459 with the function R_InitTranslationTables and change our cast from an int to an intptr_t for the translationtables variable on line 464

    translationtables = Z_Malloc (256*3+255, PU_STATIC, 0);
    translationtables = (byte *)(( (intptr_t)translationtables + 255 )& ~255);

Moving back to r_data.c, We then cast the variable colormaps in the function R_InitColormaps to an intptr_t

//
void R_InitColormaps (void)
{
    int	lump, length;
    
    // Load in the light tables, 
    //  256 byte align tables.
    lump = W_GetNumForName("COLORMAP"); 
    length = W_LumpLength (lump) + 255; 
    colormaps = Z_Malloc (length, PU_STATIC, 0); 
    colormaps = (byte *)( ((intptr_t)colormaps + 255)&~0xff); 
    W_ReadLump (lump,colormaps); 
}

and again we add sizeof to make our code more flexible and allocate the right amount of memory.

    texturecolumnlump = Z_Malloc (numtextures*sizeof(*texturecolumnlump), PU_STATIC, 0);
    texturecolumnofs = Z_Malloc (numtextures*sizeof(*texturecolumnofs), PU_STATIC, 0);
    texturecomposite = Z_Malloc (numtextures*sizeof(*texturecomposite), PU_STATIC, 0);

Now with all of that done we compile and run our executable again. No linking errors or segfaults this time! So everything was successful. Let’s see what happens when we run our program

Could not start sound server [/home/deca/programming/c_programming/DOOMLEARN/linuxdoom-1.10/sndserver]
S_Init: default sfx volume 8
Error: xdoom currently only supports 256-color PseudoColor screens

A new error! We are making more progress! This one is because the original DOOM was designed to run in an 8-bit color mode, which is not natively supported by most modern systems. Modern displays and graphics interfaces usually operate at much higher color depths (like 24-bit or 32-bit). When you try to run an 8-bit application on a system that doesn't support this color depth natively, you can get color distortion and errors.

I can check what my current color depth is by using xdpyinfo.This is a commandline utility on Linux that displays information about an X server. The X server is part of the X Window System which provides the framework for creating and managing graphical user interfaces on Linux desktops.

xdpyinfo provides various details like screen dimensions, supported visuals, and color depth. Since we are only interested in the color depth we can grep for it in the output.

xdpyinfo | grep 'depth of root window'
  depth of root window:    24 planes

So my X server is running with a 24-bit color depth and I need to run at 8-bit. What do I do? Thankfully I can get around this by starting a Separate X Session. By starting a new session with specific parameters, I can configure it to use an 8-bit color depth, which will allow DOOM to display correctly. Xephyr is the tool I will use to do this. It is a nested X server that operates as a client to a host X server. By using Xephyr, I can construct a display with the characteristics I need. Type this into the terminal

sudo Xephyr :2 -ac -screen 640x480x8

What this is doing, is executing Xephyr as a superuser, without access control (-ac), with a screen resolution of 640x480 pixels and a color depth of 8-bits (x8). The result is an empty display.

Now we need to make sure the game launches in this second display. If you are following along in the IDE, you can set the DISPLAY environment variable to :2, which is the first argument we passed to Xephyr. If you are in a terminal, just type this , before launching the DOOM executable

DISPLAY=:2

When we run our executable it should pop up in the Xephyr window. But all the colors will be messed up.

Googling this error, leads to this stack overflow post that highlights the solution. We want to add a call to XInstallColormap near line 820 before the call to XDefineCursor in the i_video.c file. The XInstallColormap function installs the specified colormap into the X server's colormap database. Once installed, the colormap becomes available for use by windows on the specified X server. This means that any window that is using this colormap will be able to display colors from that colormap, fixing our messed up colors.

XInstallColormap( X_display, X_cmap );

Finally we have to change this line in p_setup.c on line 536

linebuffer = Z_Malloc (total*sizeof(*linebuffer), PU_LEVEL, 0);

Now we run the executable for one final time and…. It works! The game is running!

On my 4k monitor the screen is waay too small, but I’m just happy it’s running at all.

While this was an interesting exercise, and a good way to see how to organize large C projects, If you want to actually play vanilla Doom on modern hardware, you should use Chocolate Doom ( a pun on Vanilla Doom). The project aims to accurately reproduce the game so that it can run on new hardware, but without altering gameplay. They are so dedicated to this that they have a NOT-BUGS.md, for bugs that you would experience in the original game, that they won’t fix.
As a fun fact this is the DOOM source port that Modern Vintage Gamer used to port DOOM to the Nintendo Switch. It’s simpler than other source ports like GZDoom, which while much more feature packed, is much more difficult ot port

Also if you’d like to look at the changes I made for this project you can find then on my GitHub. You can see the specific changes I made compared to the original code using this link.

And if you want to know more about the DOOM source code from someone who actually knows what there doing, might I point you to Fabien Sanglard and his Game Engine Black Book: DOOM. It goes over the history of Id Software, the hardware in computers at the time, and of course lots of information about the DOOM Engine internals. It is well worth the read and price.

Call To Action 📣

Subscribe now

Share Deus In Machina

Small Joys when Programming

Diego Crespo — Thu, 25 Jan 2024 13:01:18 GMT

Skitarii Rangers

I used to paint Warhammer 40K mini figures, and one of the things I always enjoyed about the hobby, was building up my toolkit of home brewed accessories that I would use to paint. During my time painting, I repurposed a fishing tackle box as a mobile painting station, used bamboo spice racks for organizing my paints, and had the big brain idea of drilling magnets into my paintbrush handles so I could hang them for storage. I even crafted my own washes from diluted inks. My personal workflow fixed deficiencies I encountered over the course of years of painting.

This concept of customization is something I’ve developed as I become a more seasoned programmer as well. From configuring custom key bindings in Emacs, to writing little scripts that automate problems only I have, and agonizing over the ideal keyboard for programming, I’ve been meticulously honing my programming environment for years.

517 lines and counting!

Even the Operating System my code runs on has not escaped customization. People have asked me, “What if you are working on a machine that isn’t set up with your workflow? ”, as if I would somehow be unable to work in that environment. Sure there might be some pain, but it’s not like I would suddenly have forgotten how to program.

This reminds me of an anecdote of my CTO a few years ago. He was using my Macbook and trying to copy and paste something out of my terminal. His Mac was set up to copy as soon as the mouse was done highlighting, but mine wasn’t. He kept going to paste the output into another program and getting nothing. It took him a few tries to realize what was going on, but he got it eventually. So sure, my muscle memory might betray me in the beginning, but after a time things will go smoothly again. Plus I like to think it’s good for my brain. You know, neuroplasticity and stuff.

I consider these difficulties a blessing. It makes me appreciate more what I have built, which otherwise spends most of its day unappreciated in the background. It’s also a sentiment that can be applied in other domains as well. James Hoffman, a World Barista Champion turned coffee YouTuber, touches on this in his video What No One Tells You About Learning To Taste

As someone who worked tirelessly to achieve perfection in coffee, he realized that there we downsides that were making him unhappy. He avoided bad coffee and was a burden on his friends due to his high expectations around it. But he realized that he could still enjoy good coffee. He just needed to change his perspective. He said…

In the case of coffee I started to drink bad coffee. Now, I didn't go and seek it out, but when it was gonna happen I just let it happen.

I think that’s a healthy way of thinking about it. And, bringing this back to programming, the inability to always work in your preferred environment, should not rob you of the little things that make programming enjoyable. Programming is as much an art as science, and a little bit of personal expression is always fun. So remember…

Without a little ugliness there can be no beauty
- James Hoffman

And go fiddle with some customizations.

Call To Action 📣

Subscribe now

Share Deus In Machina

Gambit-C: Embedding C code directly in Scheme

Diego Crespo — Thu, 18 Jan 2024 13:00:43 GMT

Photo by Piotr Makowski on Unsplash

Every programming language bills itself as a general purposes programming language, but that always comes with tradeoffs. Working in a systems programming language gives you access to pointers and raw memory, affording you fine grained control over your code. However, this comes at the cost of slower development and platform specific headaches. Working in a high level programming language likes Python lets you crank out code quickly, but the performance suffers as you are many layers removed from the hardware. So if one programming language can’t cover all of your needs, how about two? One for high level programming, and one for low level programming? Thankfully just about every language has a way of interacting with C code through a foreign interface. But some make that easier than others. Enter Gambit-C.

Gambit-C is a Scheme that compiles to C code, and can be built using a C compiler. There are two ways of using C code in Gambit. One is calling C functions in Gambit, the other is by calling Gambit code from C. Lets look at both.

Using C code in Gambit

Gambit defines 5 special forms for working with C data

c-declare
c-lambda
c-define
c-initialize
c-define-type

These are executed during the “C phase” of a Gambit-C program, before the execution of Scheme code. Here is how you would use them.

C-Declare

Lets create an empty file called embed.scm and save it. We can compile this with the Gambit compiler gsc with the -c flag to see the corresponding C file that is created.

gsc -c embed.scm

The resulting C file contains preprocessor defines, information about the version of Gambit and an include for gambit.h. Here is a portion of what that file looks like

#ifdef ___LINKER_INFO
; File: "embed.c", produced by Gambit v4.9.3
(
409003
(C)
"embed"
(("embed"))
(
"embed"
)
(
)
(
"embed#"
)
(
)
(
)
 ()
)
#else
#define ___VERSION 409003
#define ___MODULE_NAME "embed"
#define ___LINKER_ID ___LNK_embed
#define ___MH_PROC ___H_embed
#define ___SCRIPT_LINE 0
#define ___SYMCOUNT 1
#define ___GLOCOUNT 1
#define ___SUPCOUNT 1
#define ___SUBCOUNT 1
#define ___LBLCOUNT 2
#define ___MODDESCR ___REF_SUB(0)
#include "gambit.h"

c-declare is a special form that allows us to write our own includes, function declarations and variable declarations in the resulting C file. The code defined here can be used in the rest of our c-special forms.

Let’s test this out by adding a simple c-declare to our embed.scm file that includes stdio.h

(c-declare #<
c-declare-end
)

Now if we compile our code with the same flags as last time

gsc -c embed.scm

We see that stdio.h is now included in our embed.c file.

___BEGIN_SUB
 ___DEF_SUB(___X0)
___END_SUB




#include 



#undef ___MD_ALL
#define ___MD_ALL ___D_R0 ___D_R1

C-Lambda

C-lambda allows you to create a Scheme procedure that corresponds to a C function, or a sequence of C code. It is used to bridge between Scheme and C, allowing you to call C functions or execute C code directly from Scheme. It has 3 parts.

A list of types for each argument the C function expects.
The type for the value the C function returns.
The body with the name of the C function or a string of C code to execute.

Since we defined stdio.h in the c-declare section, lets write a simple hello world program in our c-lambda.

(c-declare #<
c-declare-end

)

((c-lambda () void
#<

Now we compile our Scheme code with gsc

gsc -exe embed.scm

and execute to get our Hello World!

./embed
 
Hello World!

And now, if we look at the C file we can see our printf.

#undef ___AT_END
#define ___return goto ___return_embed_23_0
printf("Hello World!\n");
___return_embed_23_0:;

I don’t know about you, but this is starting to get interesting. Being able to execute C code by just embedding it directly into a Scheme file is super cool and convenient.

We can expand our example further by writing an actual C function and using it in Scheme. Here is our new embed.scm file

;; create our square function
(c-declare #<

It contains a new function in the c-declare section that squares numbers. In the c-lambda section, I tell Scheme what the C function is called, and what the arguments and return values are.

Then I write some regular Scheme code to prompt the user for an integer, read said integer, and then call the bound C function “square” using the square-scheme function. Now we compile like last time, execute the code, type in an integer, and bam!

We square values. So we can not only define c functions directly in our Scheme code, but also bind and execute those functions as well, without ever having to leave Scheme.

But I admit, writing C in a macro block inside a Scheme file is not the best user experience. There is neither syntax highlighting nor formatting. Thankfully you can define a C function in a regular ole C file and just use that. Here is a simple file named triple.c that does just that

//triple.c
int triple(int num){
  return num * 3;
}

In our embed.scm file, using our trusty c-declare and c-lambda, we can bind the c code.

(c-declare #<string num) " is " (number->string (triple num))))
(newline)

Then we compile and execute…

And just like that we are using C code in Scheme with no fuss.

`Inline Assembly`

A quick detour. In C you can drop down to inline assembly when the moments require it. Since a c-lambda allows you to write any valid C code, it stands to reason you would be able to do this in Gambit-C as well. Here is a small example in a file called asm.scm

((c-lambda () void
#<

After compiling and running the file like normal, we can get the result of our assembly operation.

`Using Gambit Code in C`

`C-Define`

The c-define special form in Gambit-C is a way to connect Scheme procedures with C, allowing C functions to call Scheme code directly. When C calls the Scheme code, its arguments are converted to Scheme types, and passed to the Scheme procedure. The Scheme procedure's result is then converted back to a C type, and returned by the function.

Lets look at an example. Let’s say I want to read in a file using Scheme instead of C. I can do it by creating a read.scm file that looks like this using c-define…

(c-define (read-file-into-string filename) (char-string) char-string "c_read_file_into_string" ""
  (let ((port (open-input-file filename))  
        (content ""))
    (let loop ((line (read-line port)))  
      (if (eof-object? line)  
          (begin
            (close-input-port port)  
            content)  
          (begin
            (set! content (string-append content line "\n"))  
            (loop (read-line port)))))
    ))

The c-define is simple, consisting of the five parameters.

The name for the function in Scheme (read-file-into-string)
The number of arguments (in this case just filename)
the type of the input arguments (char-string) and the type of the output value (char-string)
The name of the function in C
The scope. In our case ““ but it can be “static” as well

After that it’s just a regular ole body of Scheme code. We can create a read.h file which has the prototype for our Scheme/C function

#ifndef READ_H
#define READ_H

char* c_read_file_into_string(const char* filename);

#endif // READ_H

And then we create our main.c function from which we will call our Scheme code from. I struggled with this part a bit, but I found this old conversation on GitHub that cleared things up for me. Here is the boiler plate for how you set up C to call Scheme in your main.c. I’ve bolded the sections that you have to change. The rest can stay the same

#include 

#include "gambit.h"

#include "your_scheme.h"

#define SCHEME_LIBRARY_LINKER ____20_your_scheme_

___BEGIN_C_LINKAGE
extern ___mod_or_lnk SCHEME_LIBRARY_LINKER (___global_state_struct*);
___END_C_LINKAGE


int main(int argc, char** argv) {
// C code like normal 
  printf("Hello World, this is from C\n");

  ___setup_params_struct setup_params;
  ___setup_params_reset (&setup_params);

  setup_params.version = ___VERSION;
  setup_params.linker = SCHEME_LIBRARY_LINKER;

  ___setup (&setup_params);
// scheme code
  your_scheme_function();

  ___cleanup();
  
  return 0;
}

Our files we want to use will be called read.h, and read_.c. To create our read_.c file properly, we need to generate c code from Scheme that does not contain a main entry point, or it will conflict with our main.c file. I’ll let Marc Feeley (the maintainer of Gambit-C) in the GitHub discussions explain why…

By default the link file (here somescheme_.c generated from somescheme.c) contains a C “main” function so that without any additional step the code can be compiled and linked (at the C level) to obtain an executable program. In other words the entry point of the program will be the Gambit runtime system.
However your file main.c also defines a C “main” function so you end up with a duplicate definition. The way around this, shown below, is to compile the link file with the -D___LIBRARY C compiler option. This will cause the somescheme_.c file to avoid the inclusion of a C “main” function (through a series of #ifdefs). In other words, the Scheme code and the Gambit runtime system are acting as a library to the main C program.

Let’s create a quick build.sh file, that will build our Scheme code without a main, and then call gcc to glue everything together…

#!/bin/bash
gsc -c read.scm
gsc -link read.c
gsc -obj -cc-options -D___LIBRARY read.c read_.c
gsc -obj main.c

# don't forget to include the location of the Gambit headers and dynamic libraries in the gcc command!

gcc read.o read_.o main.o -I/usr/include/gambit.h /usr/lib/x86_64-linux-gnu/libgambit.so.4 -lm -ldl -lutil -lssl -lcrypto -o main

As you can see, the read.scm file is converted into a read.c using the commands we talked about before, and then is linked. The linked file is then compiled with the gsc -obj -cc-options -D___LIBRARY and converted to a read_.c file. We then compile an object file out of our main.c. Finally we just call gcc and pass it our object files and specify all our libs and voilà! We have a main program. Here is what our main.c function looks like before we compile it. I’ve bolded the sections of the boiler plate that I changed.

#include 

#include "gambit.h"

#include "read.h"

#define SCHEME_LIBRARY_LINKER ___LNK_read__

___BEGIN_C_LINKAGE
extern ___mod_or_lnk SCHEME_LIBRARY_LINKER (___global_state_struct*);
___END_C_LINKAGE


int main(int argc, char** argv) {
  printf("Hello World, this is from C\n");

  ___setup_params_struct setup_params;
  ___setup_params_reset (&setup_params);

  setup_params.version = ___VERSION;
  setup_params.linker = SCHEME_LIBRARY_LINKER;

  ___setup (&setup_params);

  char *text = c_read_file_into_string("dickinson.txt");
  printf("%s\n", text);

  ___cleanup();
  
  return 0;
}

And when we run the main executable…

Boom! we are calling Scheme code in C.

`C-Define-Type`

At its simplest, c-define-type lets you define a name in Scheme that refers to a C type. You use this when you have to interact with c-types that are not your standard, int, char, float etc. For example the manual shows how it can be used with Cs “File” type, and how it can be used to create a pointer to a file type.

(c-define-type FILE "FILE")
(c-define-type FILE* (pointer FILE))

I’ve found it the most useful when binding C code that requires custom structs. In Raylib, Raylib uses a custom Color struct in many of it’s functions

typedef struct Color {
    unsigned char r;        // Color red value
    unsigned char g;        // Color green value
    unsigned char b;        // Color blue value
    unsigned char a;        // Color alpha value
} Color;

We can make the Color struct available from our Gambit-C code by using c-define-type

(c-define-type color (struct "Color"))

This allows us to annotate functions like clear background with the “color” type

(define ClearBackground (c-lambda (color) void "ClearBackground"))

But what if we actually need to create a color? Well we have to define a c-lambda that constructs the color for us. Here we define our make-color function, which takes in 4 unsigned-int8s (don’t annotate with unsigned-char because unsigned-char is not 8 bit on the Scheme side), and returns a color. The body of our c-lambda creates a color from the arguments we passed to the c-lambda, which we can access with ___arg1, ___arg2, etc.

(define make-color (c-lambda (unsigned-int8 unsigned-int8 unsigned-int8 unsigned-int8) color
#<

If we want to get a individual value of the color, unfortunately we will have to create functions for each value I.E color-r, color-g, color-b etc. Here is an example of me doing this using another type, Vector2, which is a struct with two float components, an X, and a Y.

(define make-vector2 (c-lambda (float float) vector2
#<

but this is Scheme, so defining a hygenic macro to do this should be easy enough.

`C-Initialize`

I’ll be honest I haven’t needed to use this so I don’t know much about it. All’s I know is it is good if you need to start a resource before your code runs. I.E connect to a database.

`Binding other C-Libraries`

Because Gambit-C interfaces with C so well, it is trivial to bind to C libraries. As an example, lets expand upon the Raylib bindings (A popular game library written in C) we introduced earler. Since Raylib builds the releases for every major platform, we don’t have to compile the code ourselves. Download the build for your platform on their release page. Since I”m on linux I’ll download the tar.gx

# Download the code
wget https://github.com/raysan5/raylib/releases/download/5.0/raylib-5.0_linux_amd64.tar.gz

# decompress
tar -xzf raylib-5.0_linux_amd64.tar.gz

Once we have Raylib on our system we see two folders in the downloaded directory. include contains all of our C header files, and lib contains our compiled code (libraylib.a, libraylib.so, etc). So as not to confuse Gambit with multiple definitions for the same code, lets delete everything in the lib folder except libraylib.a. This lets us use Raylib statically (Gambit can use dynamic libraries too!)

Now that we’ve done that we can work on writing the Gambit code. To simplify things we will just write it directly in this directory. First lets create a file called main.scm. Once we’ve done that, we can use our c-declare’s and c-lambda’s, just like we’ve been doing before, to bring in the header files, and bind our functions.

(c-declare #<

Once we’ve written our binding code, it’s time to compile. If this were C code we would compile our program with these flags

gcc -o main main.c -Iinclude/ -Llib/ -lraylib -lGL -lm -lpthread -ldl -lrt -lX11

Thankfully, gsc, the Gambit compiler, allows us to compile it in much the same way. The Gambit-C compiler works in two phases, the first is a compilation step to create .o files and the next is a linking phase. If you specify the above commands to run in the compilation phase (--cc-options) and the linking phase (-ld-options) then it will build the executable just fine

gsc -exe -cc-options "-Iinclude/ -Llib/" -ld-options "-Llib/ -lraylib -lGL -lm -lpthread -ldl -lrt -lX11" main.scm

Now we run our main program and…

We can use Raylib to make games.

`One Final Experiment`

I’ve been recently getting back into C and translated my classic brainfuck interpreter to C for practice.

It consists of 3 files, a main.c, bf_interpreter.c, and a bf_interpreter.h. I wrote this before I ever started using Gambit, so there are no tricks to make this code easier to embed. We know how easy it is for Gambit to call the C code on it’s own, all languages with an FFI can do that. But what if I copy and paste the code directly into a c-declare and then call it with a c-lambda? Would it still work? Let’s start with by defining our c-declare. I'll create a brainfuck.scm file and I’ll copy and paste my bf_interpreter.h into the declare…

 (c-declare #<
#include 
#include 
#include 
#define STACK_SIZE 30000

struct Interpreter {
    unsigned char stack[STACK_SIZE];
    unsigned char *stack_ptr;
    char *code;
    char *code_ptr;
    bool is_valid;
};
struct Interpreter createInterpreter(const char code[]);

int interpreterInterpret(struct Interpreter *interpreter);

)

Then in the same block I’ll copy and paste all of bf_interpreter.c (truncated for brevity)

(c-declare #<
#include 
#include 
#include 
#define STACK_SIZE 30000

struct Interpreter {
    unsigned char stack[STACK_SIZE];
    unsigned char *stack_ptr;
    char *code;
    char *code_ptr;
    bool is_valid;
};


struct Interpreter createInterpreter(const char code[]);

int interpreterInterpret(struct Interpreter *interpreter);


struct Interpreter createInterpreter(const char code[]) {
    struct Interpreter interpreter;
    interpreter.is_valid = false;
    // Initialize stack to zero
    memset(interpreter.stack, 0, STACK_SIZE);

    // Set the data pointer to the beginning of the stack
    interpreter.stack_ptr = interpreter.stack;

    // Allocate memory and copy the Brainfuck code for the interpreter
    // Ensure the code is null-terminated
    interpreter.code = strdup(code);
    if (interpreter.code == NULL) {
        perror("Failed to copy code to interpreter");
        return interpreter;
    }

    // Set the code pointer to the beginning of the code.
    interpreter.code_ptr = interpreter.code;
    interpreter.is_valid = true;

    return interpreter;
}

int interpreterInterpret(struct Interpreter *interpreter) {
    printf("===BEGINNING INTERPRETATION===\n");

    int user_number;
    char *code = interpreter->code;
    for (int i = 0; i < strlen(code); i++) {
        char instr = code[i];
        switch (instr) {
.......

then in a c-lambda I’ll copy over my main.c code.

((c-lambda () void
#<

And it should run right? It will, with two minor updates. For the sake of simplicity I added the code from the bf_interpreter.h and .c file into the c-declare. Since I did that, I no longer need the includes in my main.c, because these are already visible from our c-declare (I know that from the documentation 😃)

///No needed anymore in our c-lambda
#include 
#include 
#include "bf_interpreter.h"

Lastly, I removed the return EXIT_SUCCESS line from the c-lambda that was present in my main.c. The reason is it will cause our program to segfault. This is understandable because Gambit expects to run Scheme code after it’s executed all the c stuff, but we prematurely exit. But other than that, all the rest of the code is the same. Let’s execute it!

Pretty…Darn…Cool.

Before we wrap up, I have one more thing to talk about in regards to Gambit’s CFFI overhead. You may be wondering, as I was, what the overhead between Gambit calling C is, or C calling Gambit. I asked this very question on the gambit-c gitter, and Marc Feeley the maintainer for Gambit actually answered!

Diego: Since Gambit Compiles to C code, there shouldn't be any overhead for calling C functions like there is when using a C-FFI in other languages right?
Marc Feeley: People often misunderstand the type of “Scheme to C” compilation that is performed by Gambit. In many other compilers that compile a certain language X to C there is a 1-to-1 mapping of functions in language X and functions in C. If you define function f in your X program then you can expect some function to exist in the generated C code that is a translation of f. Some Scheme to C compilers actually do this (Bigloo, and I think Chicken also). But Gambit takes another approach in order to correctly and fully implement Scheme tail-calls and continuations, which don’t have a direct equivalent in (standard) C. Gambit uses a trampoline to implement control transfers between pieces of Scheme code (whether they are calls to a procedure or returns from a procedure). So if you use the C calling convention as a reference, then Scheme calls and returns typically have a higher cost than a C call/return.
Gambit's C FFI has the additional burden of converting the data representation of parameters and return values. A Scheme integer is not represented the same way as a C integer, so some conversion must happen when going from Scheme to C and back. This is true for all types (floats, booleans, characters, strings, etc) except the scheme-object type that keeps the same representation. There is also the burden of protecting Scheme values (and converted values) from being garbage collected and moved in a way that would cause issues on the C side. Finally there’s an overhead in supporting bidirectional calls between Scheme and C, and exceptions, and continuations. So calls between Scheme and C using c-lambda and c-define are certainly not as cheap as a C function calling another C function. Surprisingly, the use of trampolines and some compiler optimizations can make Scheme to Scheme calls faster than C to C calls (but that is a different subject). When performance is critical the lesser known (##c-code "") which has no direct overhead can be quite useful. If x is a variable containing a fixnum, then computing the square of x can be done with essentially no overhead with (##c-code "___RESULT = ___FIX(___INT(___ARG1)*___INT(___ARG1));" x). Note however that this code does no type checking and no overflow checking (but that’s sometimes what you want to avoid).

A very thorough answer!

I hope that this has given you a brief overview of Gambit-C’s, C interfacing capabilities, and I hope that you will give the language a try. If you could only pick two programming languages to cover all of your programming needs, what would you choose? Comment down below, I’d love to hear your thoughts.

`Gitlab Repo`

You can find all of the code in this example in the Gitlab repo. Also if you haven’t had enough Gambit yet check out Gerbil Scheme! It is built on top of Gambit-C and adds some cool features!

`Call To Action 📣`

Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter and a Mastodon, if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G

Share

Subscribe now

Share Deus In Machina