Human Search Engine

Remember everything

Have you ever tried to remember the name of a product that someone shared in a slack/discord community and didn't work?

Then you went to google, tried a few searches, and you got no results for what you were looking for.

What if I tell you that you can remember anything?

The “trick” is to build your own search engine for your knowledge base.

First, let understand some basic concepts like search and index.

---

What is Search?

Search is the process of finding something.

The best example of a search engine is Google.

Google can help you find many different things: websites, images, places, content, news, product, flights, and more.

Google Search started using the PageRank algorithm. PageRank tries to find important websites based on how many “good” links that the website receives.

Google images and Google lens let us search using images. We can find related content based on a static image or extracting text from videos using our camera app.

Shazam lets us find music based on parts of the music or songs.

Mac Spotlight lets us find "anything" on our desktop. It can find documents, applications, and more.

Webstorm lets search in our codebase for filenames, variables, functions, classes, components, and more.

SaaS let us search any of our data using an Omnibox

What is indexing?

Indexing is the process of creating a data structure to make it fast to retrieve information.

We have billions of webpages on the internet, each webpage has many words on it. How can we make it fast to find a particular query?

Text search engines use an inverted index to reduce the sample search space. Instead of searching in billions of webpages, you search in a very shorter index making it much faster.

---

How to build our own search engine?

To build our own search engine, we need to build our own index.

The first thing you need to do is to keep your knowledge of the information in the same place. I keep mine in Notion.

We are going to follow many principles of Zettelkasten.

For each information that we want to remember we are going to create a note.

A note will be the basic unit for our index.

A note should contain with our own words the main concepts of what we want to remember. If you want to remember some book concept, post the link to the book and also the main concepts that you extracted from it.

A note can also contain links, images, videos to help you remember.

A note should also contain text and tags with many possible ways to find it, how would you search about this topic in the future?

You should link all notes to other notes to create a web of knowledge similar to the internet. So one concept/topic can lead to finding related ones.

Never delete a note, let it evolve over time.

This can also be called Human-based computation where a human helps the machine to provide the best solution.

---

Practical examples

Indexing a book

When reading a book I create 1 new note for the book, where I put the link to the book. I also put the core concepts/ideas of the book on that note.

I also create one new note for each concept so I can link the concept to other books, blog posts, or content that I've read before or I would read in the future.

I also add where I found the book, and who recommended the book to me.

I also link articles that I wrote about the book.

Indexing a Person

I write down a short description and contact information of the person.

I write their ideas, projects, books, music, and related persons.

Indexing Video/Audio

Video/Audio is hot media.

They are high-resolution media having much more information than text. The idea of indexing is to transform high-resolution into a low-resolution and compressed information.

I keep draft notes while listening to audio or watching a video, at the end of it I review and compile the notes only in important concepts. I also link the final note with related content.

---

Thoughts about this process

This can sound weird at first, but when I moved to this method of keeping my own index I never forget about that tweet that someone sent me on slack 3 months ago.

This is a Thinking Tool, as it let me offload my thinking to notes that I know that I will be able to retrieve when I needed.

You can find a more complete intro to build a search engine at Building a Search Engine From Scratch by Cliqz