The most vital knowledge in human history

Boston Public Library

I recently treated myself to a new Kindle to replace my decade old one. With a whopping 32Gb of storage, it was overkill for novels, but it sparked a curious idea (perhaps inspired by reading Asimov's Foundation)... Could I fit all of English Wikipedia onto it?

Of course, not all of Wikipedia, that currently encompasses 6,806,195 articles which is like 86gb uncompressed. But then I learned about Wikipedia's "Vital Articles." Out of all the sea of articles on Wikipedia, only a few select articles make this list, with five levels of "vitalness" that encompass larger and larger sections of Wikipedia.

• Level 1 - 10

• Level 2 - 100

• Level 3 - 1000

• Level 4 - 10,000

• Level 5 - 50,000

I encourage everyone to browse the list. It's interesting to see what other people consider important throughout human history, even if it's not the list you would have made. That's definitely worth reflecting on. We programmers get a nod in Level 1 under Science, followed by a more prominent 'Computers' section in Level 2. By Level 3, there's an entire 'Computing and Information Technology' branch, a testament to our field. It did surprise me to see Cryptography get its own section at this level, but then I remembered its historical role in early computing during the World Wars with code breaking.

By level 4 there are 10,000 articles, and we get put in a higher level section called 'Technology' which includes 'Computing and information technology' and now encompasses 78 articles ranging from...

1. Basics

2. Operating Systems

3. Networks

4. Computer science

5. User interface

6. Internet

7. Computer hardware

8. Cryptography

9. Programming

10. Computer Software

11. Data Storage

Four programming languages get dedicated subsections in the section on Programming.

1. Assembly

2. C

3. Java

4. Javascript

Which I know will ruffle some feathers. Disappointingly, Most of the articles in the Computing and information technology section have a C grade, which Wikipedia describes as

"Useful to a casual reader, but would not provide a complete picture for even a moderately detailed study"

I'd like to spend some time in the future improving this.

By level 5 we get one final high level category, 'Communication, media, and computing technology'. It consists of 729 articles, broken down into 12 broad sections and many more subsections.

But while it's interesting to see at a high level what the community thinks is important in my field, it's time to figure out how to download all of this

Downloading articles

I found a nifty Calibre plugin called Wiki Reader for grabbing Wikipedia pages

The interface is simple. It allows you to put in the name of the book associated with the page you want to download...

Then it generates a MOBI file.

If instead of a single URL, you copy and paste a bunch of URLS to your clipboard, it can auto populate the urls, and combine them all into a single ebook.

Unfortunately, while bulk downloading is possible, it's agonizingly slow after about 100 articles. But when I tried to write a Python script to scrape the pages manually, the results weren't much better.

I was about to throw in the towel when I stumbled another project, Wikibooks.

Enter Wikibooks

Wikipedia itself has limitations as a learning resource...

1. Unfriendly for Reading: Links and references disrupt the flow.

2. Teaching Gaps: It's not a structured learning environment.

Wikibooks (also from Wikimedia) solves this. It offers deeper dives on certain subjects, with more organized learning paths, while keeping the approach to community contribution. All the information for a topic exists in one book, and you don't need to navigate a maze of links to get the full picture. The tradeoff is less overall coverage than its big brother Wikipedia, and a smaller community, but there are still plenty of content to choose from. I'm just glad a community of volunteers exist that make this knowledge freely available to all of us.

The nice thing about downloading Wikibooks over Wikipedia articles is that they are generally much friendlier to offline reading. Many Wikibooks offer printable versions of the books, and some even have e-reader friendly PDFs.

This means better formatting and a smoother reading experience on my Kindle. Plus, it's trivial to combine PDFs with tools like PyPDF2 in Python, after they are downloaded.

The Joy of Learning

I went through Wikibooks and I downloaded many of the books I was interested in to put them on my Kindle. While going through the Philosophy section, I came across a Wikibook that was called Lun Yu" (论语), also known as "The Analects of Confucius". It is a collection of sayings and ideas attributed to the Chinese philosopher Confucius and his disciples. The first quote from the first chapter resonated deeply with the spirit of this whole experiment...

The Master said, Is it not pleasant to learn with a constant perseverance and application?

Indeed, it is.

Call To Action

Hi 👋 my name is Diego Crespo and I like to talk about technology, niche programming languages, and AI. I have a Twitter and a Mastodon, if you’d like to follow me on other social media platforms. If you liked the article, consider liking and subscribing. And if you haven’t why not check out another article of mine listed below! Thank you for reading and giving me a little of your valuable time. A.M.D.G