The Evolution of Valve's Source Engine
The Evolution of Valve's Source Engine
In 1996, Mike Harrington and Gabe Newell, two Microsoft employees working on Windows NT, left their jobs to start a game company. They were inspired by Michael Abrash, who had also left Microsoft earlier to go to ID Software and work on Quake. Through their connection to Abrash, Gabe and Mike were able to meet with John Romero of Doom fame. Romero was instrumental in helping the two fledgling co founders get their new company Valve, off the ground. He also kick started them with a copy of the Quake source code, which they would use as the base for their first game engine.
This modified Quake engine, later dubbed the Half-Life engine, would catapult Valve into renown with its namesake game, Half-Life. The Verc Collective (Valve Editing Resource Collective) interview with Ken Birdwell and Jay Stelly about the modified engine.
At its core, it's a Quake 1 engine. You can tell this by comparing Half-life's map compiling tools with those shipped with Quake1. You'll find very minor differences -- none of them are fundamental. The core rendering is architecturally identical to Quake1, the only "significant" change is removing the fixed palette, making map lighting RGB instead of 8 bit, and converting software rendering to be 16 bit color instead of 8 bit color, which was pretty easy and only required minor code changes. Our skeletal animation system is new, though it was heavily influenced by the existing model rendering code, as were a lot of our updated particle effects, though less so with our beam system.
Decals are totally new, our audio system has some major additions to what already existed, and at ship time our networking was almost totally Quake1 / QuakeWorld networking but about a year later Yahn rewrote most of all of it to be very different in design. The most highly changed sections are the game logic; ours being written in C++ and Quake's being in written interpreted "Quake C". Our AI system is very very different from anything in Quake, and there's a lot of other significant architectural changes in the whole server and client implementations, though if you look hard enough you can find a few remnants of some nearly unmodified Quake1 era entities buried in places.
A mere two years post establishment, Valve would release Half-Life in November of 1998, the details of which are documented thoroughly documented in their 25th Anniversary Documentary.
Half-Life was a smash hit, and Valve wasn’t content to rest on their laurels. They immediately started working on their next game. But they soon realized a problem. By making modifications to the Half-Life engine they could potentially introduce bugs into Half-Life. Erik Jonson a Valve engineer describes their solution.
When we were getting very close to releasing Half-Life 1 (less than a week or so), we found there were already some projects that we needed to start working on, but we couldn't risk checking in code to the shipping version of the game. At that point we forked off the code in VSS to be both $/Goldsrc and /$Src. Over the next few years, we used these terms internally as "Goldsource" and "Source". At least initially, the Goldsrc branch of code referred to the codebase that was currently released, and Src referred to the next set of more risky technology that we were working on. When it came down to show Half-Life 2 for the first time at E3, it was part of our internal communication to refer to the "Source" engine vs. the "Goldsource" engine, and the name stuck.
Now, with two versions of the engine, GoldSrc and Src, development could continue without harming the stable codebase.
Through out Half-Life’s Development a popular Quake mod was going around the office called Team Fortress. Team Fortress modded Quake to have 9 different classes, each with their own unique playstyle.
Scout
Sniper
Soldier
Demoman
Medic
Heavy
Pyro
Spy
Engineer
Valve would go on to hire these modders to develop an official version of the game also called Team Fortress, which came out on GoldSrc on April 7, 1999. This approach, hiring modders to develop official ports of exciting mods, would end up serving Valve well. This led to a series of successful games, many originating as mods, such as Counter-Strike, Day of Defeat, and others, all developed on the evolving but stable GoldSrc engine. Finally with the release of Counter-Strike: Condition Zero March 23, 2004, the last game on the GoldSrc engine would be created.
During Condition Zero’s development, which was done by Ritual Entertainment and later finished by Gearbox, Valve would work on a successor to the GoldSrc engine.
The Source Era
The Source engine was developed in a modular fashion, through a combination of in house changes, and licensed middleware. This allowed modules to be updated piecemeal and distributed as updates through Steam. This also meant porting old games to the new engine was easier, as it could be updated in parts. Valve would do this many times in their history, starting first by updating Counter-Strike to run on the new Source engine. This would lead to the creation of Counter-Strike Source (CSS). CSS was released in October 7, 2004 and was huge hit. For many, it is the peak of the competitive first person shooter genre. Just a couple months later, Valve would deliver again with Half-Life 2, also on the Source engine, on November 16th. They would also port Half-Life to the engine as well, renaming it Half-Life Source.
These games would introduce many Gamers to the quirks of the Source Engine, including bunny hopping, crouch clipping, flying, wall climbing, teleporting npcs and many others. Gamers would use this for great effect in Speedruns.
and despite attempts like the server plugin zBlock to stop these exploits, players usually found a way around them.
Over the next few years, many of Valve’s other games would get the Source treatment including
Day of Defeat: Source: September 26, 2005
Half-Life Deathmatch: Source May 1, 2006
Half-Life 2: Episode One: June 1, 2006
Half-Life 2: Episode Two: October 10, 2007
Portal: October 10, 2007
Team Fortress 2: October 10, 2007
But porting wasn’t the only thing the Source engine was good at, and very soon Source would see a significant update.
The Orange Box Update and Beyond
On October 10th, 2007 Valve released the The Orange Box, which featured five in games one package: Half-Life 2 including, Episode One and Episode Two. Portal, and Team Fortress 2. Along with the update came a slew of changes to the Source engine including…
Threading
Soft Particles
Shadow Mapping
Improved High Dynamic Range (HDR) Rendering
Motion Blur
Hardware Morph (HWM) Flexes and Wrinkle Maps
Conditional Statements
Phong shading Improvements
Self-Shadowing Bump Maps
Automatic Displacement Texturing
Alpha to Coverage
Flowing Emissive Textures
Fresnel for Specular Reflections and Self-Illumination
Cinematic Physics
Visclusters
Bink Video
Maplist Thumbnails
This did not come easy, as Tom Leonard described in his talk, Dragged Kicking and Screaming: Source Multicore
This update was able to finally kill forward bunny hopping, a problem that had been around since Quake 1 and the GoldSrc days, and limit physics ones like prop flying and wall climbing. But it also introduced some new bugs as well.
The next major update would come in 2009. This would include Mac OS X support, Quick time support, particle effects updates, and the ability to turn off dynamic prop physics, for improved performance on low end hardware. This would also mark the beginning of Valve’s engine being multi platform.
A few years later, Valve would add support for an additional platform with their 2013 update. This would be the first time Linux and Android were supported. Additionaly, updates to VPK files and custom folders for easier modding, static prop lightmaps, Per-texel color tint masking for models, Swaying trees, BSP compression, and VScript rounded out the features. Little did we know this update would also lay the ground work for Valve’s foray into Linux, first with the Steam Machines in 2015, and more recently, the Steam Deck.
Source 2 is born
Valve was beginning to reach the limits of Source as far back as 2007, and began development on a successor to the Source engine soon after. As it would turn out, the development of this engine was one of the primary reasons Half-Life 2: Episode 3 never came out. Valve already had a track record of avoiding developing games while trying to develop an engine, which they consider a bad idea, and Source 2 was no different.
Source 2 beginnings were humble, first being used as a proof of concept for a re imagined Plantation level from Left 4 Dead 2 in 2010. A screen shot of this would be discovered in a leaked Valve PowerPoint presentation in 2014, but this was not the first time people had heard rumblings of Source 2. Gabe Newell would confirm that Valve was working on a succesor to Source in 2012, but noted that they had, “Just been waiting for a game to roll it out with”.
This first game would end up being Dota 2, first as an update to the Steam workshop tools in 2014, and then as the Dota 2 Reborn update on June 17, 2015. The update brought many improvements to the game including more responsive input, better tools for map creation, enhanced performance, additional multithreading, background streaming, lower latency audio, and support for the Vulkan graphics API. The Hammer editor was rebuilt from the ground up, and the Havok physics engine, which previously required a $25,000 license to use in Source 1, was replaced with a new physics engine called Rubikon.
Many people have done comprehensive comparisons between Source and Source 2, including this one by Rikan if you’d like to see a visual comparison.
Since the Dota reborn update there have been many other games produced using Source 2 including The Lab, Artifact, Dota Underlords, Half-Life: Alyx, and most recently, Counter Strike 2. The Source engine's history is a testament to Valve's dedication to innovation and quality. It is also a test of their fans patience as it along with many other Valve endeavors operate on Valve Time. Despite it’s bugs, and indeed because of them, it has endeared itself to many fans, and is continually being used to create new an exciting games to this day. As we look beyond 2023, the Source engine's legacy endures, and I’m exciting to see what comes next. But with the Source 2 engine being a decade old next year, the real question is “Hey Gabe, When will we see Source engine 3?”
Call To Action 📣
Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter and a Mastodon, if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G
