A quick introduction
--------------------

Hopefully, you already know that [MeiliSearch](https://github.com/meilisearch/MeiliSearch) is a powerful and fast open-source search engine built in [Rust](https://www.rust-lang.org). It was designed to provide users with a very useful and customizable search experience including out-of-the-box features such as typo-tolerance, filtering, and synonyms for any kind of project.

Running a Meilisearch instance for testing purposes is incredibly easy and can be done in [many ways](https://docs.meilisearch.com/learn/tutorials/getting_started.html): using Docker, brew, aptitude, binaries, a simple curl, or even the source code. If you are new to MeiliSearch, we suggest that you take a tour around the documentation.

Using MeiliSearch on your own machine for your weekend project is fun, let's agree on that. However, you may want to go live and deploy a project in production, to take it to the next level. What steps and details would you need to **deploy MeiliSearch in production** and ensure it is **safe and ready to use**?

Get your MeiliSearch ready for production
-----------------------------------------

For this tutorial, we will be using a Debian 10 server, running on DigitalOcean. You can easily try it on your own, with plans starting at $5/month. And if you want some credits to start running your MeiliSearch and are not already registered on DigitalOcean, you can get $100 for free using [this referral link](https://m.do.co/c/7c67bd97e101).

Prerequisites
-------------

*   An up-to-date server that runs Debian 10
*   An ssh keypair to connect to that machine
*   MeiliSearch is a database which means that it needs a file system it can write to, and which must be persistent.

> _TIPS:_ learn how to connect via SSH to your [DigitalOcean droplet](https://www.digitalocean.com/docs/droplets/how-to/connect-with-ssh/) or any [Linux or windows server](https://phoenixnap.com/kb/ssh-to-connect-to-remote-server-linux-or-windows)

Step 1: Install MeiliSearch
---------------------------

Installing and running MeiliSearch is easy and straightforward. In order to keep this tutorial as simple as possible, let's use a script that will carry out the installation process. It will copy a binary of MeiliSearch to your machine and enable you to use it immediately.

Once you are logged in into your machine via SSH, ensure your system and its dependencies are up-to-date before proceeding with the installation.

```bash
# Update the list of available packages and their versions
$ apt update
# Install curl which is required to install MeiliSearch in the next step
$ apt install curl -y
# Install MeiliSearch latest version from the script
$ curl -L https://install.meilisearch.com | sh

```

> The different options to achieve a MeiliSearch installation are detailed in **[this guide](https://docs.meilisearch.com/reference/features/installation.html#download-and-launch)**.  
> It is important to know that there are different ways to get MeiliSearch running on your machine. As an open source project, you can always compile the latest stable release of MeiliSearch from the source code to ensure the binary uses your achitecture in the best possible way.

You can always check the latest MeiliSearch stable version, and get MeiliSearch for the Operating System of your choice, by visiting the following link:

**[Latest MeiliSearch Stable Version](https://github.com/meilisearch/MeiliSearch/releases/latest)**

MeiliSearch is finally installed and ready to use. To make it accessible from everywhere in your system, move the binary file into your system binaries folder:

```bash
# Move the MeiliSearch binary to your system binaries
$ mv ./meilisearch /usr/bin/

```

You can now start using MeiliSearch! In your terminal, run:

```bash
$ meilisearch

# --- Expected output for v0.10.1 ---
888b     d888          d8b 888 d8b  .d8888b.                                    888
8888b   d8888          Y8P 888 Y8P d88P  Y88b                                   888
88888b.d88888              888     Y88b.                                        888
888Y88888P888  .d88b.  888 888 888  "Y888b.    .d88b.   8888b.  888d888 .d8888b 88888b.
888 Y888P 888 d8P  Y8b 888 888 888     "Y88b. d8P  Y8b     "88b 888P"  d88P"    888 "88b
888  Y8P  888 88888888 888 888 888       "888 88888888 .d888888 888    888      888  888
888   "   888 Y8b.     888 888 888 Y88b  d88P Y8b.     888  888 888    Y88b.    888  888
888       888  "Y8888  888 888 888  "Y8888P"   "Y8888  "Y888888 888     "Y8888P 888  888

[2020-05-04T11:47:13Z INFO  meilisearch] Database path: "./data.ms"
[2020-05-04T11:47:13Z INFO  meilisearch] Start server on: "127.0.0.1:7700"

```

Step 2: Run MeiliSearch as a service
------------------------------------

In Linux environments, a `service` is a process that can be launched when the operating system is booting and which will keep running in the background. One of its biggest advantages is making your program available at any moment. Even if some execution problems or crashes occur, the service will be restarted and your program will be run again.

> If you are new to services and `systemd`, you can learn more about the basics of Linux services [here](https://www.hostinger.com/tutorials/manage-and-list-services-in-linux/).

In Debian and other Linux distributions, `systemd` allows you to create and manage your own custom services. In order to make sure that MeiliSearch will always respond to your requests, you can build your own service. This way, you will ensure its availability in case of a crash or in case of system reboot. If any of these occur, `systemd` will automatically restart MeiliSearch.

### 2.1 Create a service file

Service files are text files that tell your operating system how to run your program, and when. They live in the `/etc/systemd/system` directory, and your system will load them at boot time. In this case, let's use a very simple service file that will run MeiliSearch on port `7700`.

To run MeiliSearch in a production environment, use the `--env` flag. To generate a master key that will let MeiliSearch create reading and writing keys, use the `--master-key` flag. With those keys, you can easily control who can access or create new documents, indexes, or change the configuration. You can change the `Master Key` to any value in the following command. However, for security concerns, it's better to choose a safe and random key, never share it and, just, **keep it safe**.

```bash
$ cat << EOF >/etc/systemd/system/meilisearch.service
[Unit]
Description=MeiliSearch
After=systemd-user-sessions.service

[Service]
Type=simple
ExecStart=/usr/bin/meilisearch --http-addr 127.0.0.1:7700 --env production --master-key Y0urVery-S3cureAp1K3y

[Install]
WantedBy=default.target
EOF


```

> For more information on MeiliSearch authentication and API keys see the **[Authentication Docs](https://docs.meilisearch.com/reference/features/authentication.html)**. For more information on MeiliSearch options and flags see the **[MeiliSearch configuration options](https://docs.meilisearch.com/reference/features/configuration.html)**

As for now, it is not time yet to expose your MeiliSearch instance to the external world. To keep running it safely inside your own environment, make it available locally at `127.0.0.1`. This means that only programs running on your machine are allowed to make requests to your MeiliSearch instance.

### 2.2. Enable and start service

The service file you just built is all you need for creating your service. Now you must `enable` it to tell the operating system that we want it to run MeiliSearch at every boot. You can then `start` the service to make it run immediately. Ensure everything is working smoothly by checking the service `status`.

```bash
# Set the service meilisearch
$ systemctl enable meilisearch

# Start the meilisearch service
$ systemctl start meilisearch

# Verify that the service is actually running
$ systemctl status meilisearch

-# --- Expected output ---
● meilisearch.service - MeiliSearch
   Loaded: loaded (/etc/systemd/system/meilisearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-04-10 14:27:49 UTC; 1min 8s ago
 Main PID: 14960 (meilisearch)

```

At this point, MeiliSearch is installed and running. It is protected from eventual crashes, system restarts, and most of the problems it could find while running. But it is still hidden and protected inside the walls (or firewalls) of your machine, and unreachable from the outside world. You can stop here if all the requests you do to MeiliSearch are done by another application living in the same machine.

But you probably want to open your MeiliSearch to the outside world, and for now, it is isolated. Let's fix that in a safe way.

Step 3: Secure and finish your setup. Using a Reverse Proxy, domain name and HTTPS
----------------------------------------------------------------------------------

It's time to safely make your brand new MeiliSearch available to be requested from the outside world. For this purpose, you will use two of the main technologies available on the web: a Reverse Proxy and SSL/TLS.

### 3.1. Creating a Reverse Proxy with [Nginx](https://www.nginx.com/)

A reverse proxy is basically an application that will handle every communication between the outside world and your internal applications. Nginx will receive external HTTP requests and redirect them to MeiliSearch. When MeiliSearch has done its amazing job, it will communicate its response to Nginx, which will then transfer the latter to the user who originally sent the request. This is a common way to isolate and protect any application by adding a robust, secure, and fast gate-keeper such as Nginx, one of the safest and most efficient tools available online, and of course, open-source!

> Reverse proxies are very useful regarding security, performance, scalability, and logging concerns. If you are new to Reverse proxies, you may enjoy this article explaining the why and the how of [reverse proxies](https://www.keycdn.com/support/nginx-reverse-proxy).

Configuring Nginx as a proxy server is really simple. First of all, install it on your machine.

```bash
# Install Nginx on Debian
$ apt-get install nginx -y

```

First, deleting the default configuration file is important as the default port for HTTP, the `port 80`, is used by Nginx by default. Thus, trying to use it for MeiliSearch will create a conflict. Replace the default file by your own configuration file. You can also make MeiliSearch listen to another port by specifying it in the Nginx configuration file, but we will not cover this option in this tutorial.

```bash
# Delete the default configuration file for Nginx
$ rm -f /etc/nginx/sites-enabled/default

# Add your configuration file specifying the Reverse Proxy settings
$ cat << EOF > /etc/nginx/sites-enabled/meilisearch
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    location / {
        proxy_pass  http://127.0.0.1:7700;
    }
}
EOF

```

Finally, enable and start the Nginx service again to make sure it is still available.

```bash
# Reload the operating system daemons / services
$ systemctl daemon-reload

# Enable and start Nginx service
$ systemctl enable nginx
$ systemctl restart nginx

```

MeiliSearch is now up, deployed in a production environment, using a safe API key, and being served by a Reverse Proxy Nginx. You should now be able to send requests to your server from the outside world. Open your web browser and visit: ([http://your-ip-address](http://your-ip-address)). The IP address is the same you used to connect to your machine via SSH in Step 1.

> If you want to learn more about using Nginx as a Reverse Proxy, see [this dedicated documentation](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/).

The only remaining problem is that MeiliSearch processes requests via HTTP without any additional security. The content that is being transmitted over HTTP could easily be read or modified by attackers, and someone could get full or partial access to your data. In order to prevent this to happen, it's important to use the HTTPS, which will enable you to use a SSL/TLS certificate, and securely transmit data.

### 3.2. Set up SSL/TLS for your MeiliSearch

SSL will let the user or client establish an authenticated connection to MeiliSearch. In this way, a user can verify server's identity before sending sensitive data or making any request to it. Then, data is sent in an encrypted way that only MeiliSearch server will be able to decrypt, providing you a fast, reliable, and automatic layer of security.

In most cases, when enabling SSL, you may want to use your own domain name (or a sub-domain). The first step you need to follow is to register your own domain name and change the DNS records. To make your domain name point to your newly installed MeiliSearch server, you just need to add an `A record` pointing to the IP address used to connect to your own server. This process is simple and fast but can vary for every domain name provider. Thus, we will not cover that process in this article.

> When you register a domain name and add an `A record`, you should be automatically able to request MeiliSearch directly by using that domain name.  
> To illustrate this, if you had registered your domain name `example.com`, requesting indexes would be done at [http://example.com/indexes](http://example.com/indexes)

Once your domain name has been set up, you are ready to configure SSL/TLS and use HTTPS. You have two different options to achieve this goal. The first one is using [Certbot](https://certbot.eff.org/), an amazing, free, and very easy to use tool. If you already have SSL certificates issued from a `Certificate Authority or CA` for your domain name, the second option covers the steps you need to follow. Then, you will be ready to use MeiliSearch safely in production!

#### 3.2. Option A: Certbot

Using certbot in your Linux server is very easy and straightforward. This tool will generate a free SSL/TLS certificate for your domain name, and automatically handle its installation on your server. The certbot documentation contains detailed instructions for many operating systems and servers, but we will follow the instructions for [Certbot on Debian with Nginx](https://certbot.eff.org/lets-encrypt/debianbuster-nginx).

First of all, install the packages on your system:

```bash
$ sudo apt-get install certbot python-certbot-nginx -y

```

Let's run the Certbot script to be guided through the installation process:

```bash
certbot --nginx

```

Enter your email address, agree to the Terms and Conditions, and input your domain name. You will then be prompted with these options:

```bash
Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.

```

We recommend that you choose option 2, to redirect HTTP to HTTPS and always use a secure connection. You should be able to request your domain name with SSL as in `https://example.com` or `https://example.com/indexes`.

#### 3.2. Option B: Custom SSL/TLS Certificates

When a `Certificate Authority` issues a SSL certificate for you, you receive at least two files with encrypted keys:

*   Your **certificate** (commonly named `your_domain_name.pem` or `example.pem`)
*   Your **key** (commonly named `your_domain_name.key` or `example.key`)

> `example.pem` and `example.key` will be used in the following examples. Make sure to replace `example` by the names of your own certificate files.

All you need to do is store the certificate files in a secure location and use appropriate file system security permissions. Then, set the location of the certificates in Nginx configuration. It is also strongly recommended to redirect all HTTP requests to HTTPS (port 80 to 443).

First, let's copy your certificate files in their conventional directory so the server can find them:

```bash
# Create a directory /etc/ssl/example to store the certificate files
$ mkdir -p /etc/ssl/example

# Move your files to /etc/ssl/example. We will suppose that your
# files are called example.pem and example.key

$ mv path-to-your-files/example.pem /etc/ssl/example/
$ mv path-to-your-files/example.key /etc/ssl/example/

```

Finally, we create a new Nginx configuration file, and restart the daemons and Nginx service

> Remember to replace `example.com` in both `server_name` fields with your **own domain name**

```bash

# Replace example.com in both `server_name` fields with your own domain name

$ cat << EOF > /etc/nginx/sites-enabled/meilisearch
server {
      listen 80 default_server;
      listen [::]:80 default_server;

      server_name example.com;

      return 301 https://\$server_name\$request_uri;
}
server {
    server_name example.com;

    location / {
        proxy_pass  http://127.0.0.1:7700;
    }

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;

    access_log /var/log/nginx/nginx.vhost.access.log;
    error_log /var/log/nginx/nginx.vhost.error.log;
    ssl_certificate /etc/ssl/example/example.pem;
    ssl_certificate_key /etc/ssl/example/example.key;
}
EOF

$ systemctl restart nginx

```

Your SSL certificates should be working and Nginx should be able to find them. Every request to `http://example.com` will now be redirected to `https://example.com`

Conclusion
----------

You have followed the main steps to provide a safe and stable service. Your MeiliSearch instance should be up and running, in a safe environment and ready to stay available even when the most common issues occur. In addition, it is protected by a reverse proxy with your own domain name and API key, so your data and configuration are accessible only to trusted clients. Communication with your server is now encrypted. Furthermore, its identity will be verified every time before sending sensitive data in a fast and automated manner.

You are now ready to start using your **Production-ready MeiliSearch**!

MeiliSearch in production: taking it to the next level

As a Rails and Ruby lover, I often search for the gem that will perfectly cover my use case. When I need to solve a problem I'm having, I want to choose the _right one_, the most suitable solution.

Ruby gems are extensive libraries created inside the Ruby community. The best place to find a ready solution for any task is the website [rubygems.org](https://rubygems.org/), a public repository of gems that can be searched using the search box on the home page. RubyGems’ website is an efficient tool that facilitates the sharing and the installation of packages. But although its search bar is very helpful, I decided to create an [alternative search bar](https://rubygems.meilisearch.com/) more suited to our needs.

![](https://unable-actionable-car.media.strapiapp.com/rubygemsv3_510e7b8445.gif)

The search-as-you-type experience
---------------------------------

First, I wanted to implement a search-as-you-type experience. Which means:

*   a response time lower than 50 milliseconds
*   presenting all matching results immediately under the search box while the user is typing and without them pressing enter

That is not yet the case in the RubyGems' website since a new page is loaded each time a request is made.

The relevancy
-------------

You can get relevant and accurate results by using the RubyGems' search bar, but most of the time only by performing an Advanced Search, which is not always convenient. You have to decide what to fill in the different sections. Are you going to search for a specific package by entering its name (e.g. "devise") or find a package whose summary would match a keyword (e.g. "deployment")?

However, despite this functionality, you may not find gems that meet your needs. For example, if you enter "pagination" you would expect to see the gem "kaminari", which is the most popular gem for pagination among the RoR community, showing up in the results. Here is [the return we get from RubyGem's search bar](https://rubygems.org/search?utf8=%E2%9C%93&query=pagination) when submitting the keyword "pagination". As you can see, "kaminari" doesn’t appear before the 9th result.

Even when [refining the search](https://rubygems.org/search?utf8=%E2%9C%93&query=summary%3A+pagination+description%3A+pagination), the first result to show is "kanimari-core" which is not exactly the more appropriate and famous "kaminari" package we would like to find, but it’s still better than nothing.

Then if we do a search [containing a typo in the request](https://rubygems.org/search?utf8=%E2%9C%93&query=pagintion), like "pagintion", the page is displayed without any result and suggests you a similar word for your next search.

After this experience as a user, I aimed to create **a single search bar able to understand what you want and to instantly find it**!

![](https://unable-actionable-car.media.strapiapp.com/speed_b868b434f8.jpg)

Meilisearch checks all these points, and more!
----------------------------------------------

I've never implemented a search engine; I've never even used one, except a basic Elasticsearch instance with no configuration for a proof of concept. For that reason, all I needed was an easy-to-setup tool, able to handle both speediness and relevancy at the same time. That's why Meilisearch was a perfect fit for this project.

**[Meilisearch](https://github.com/meilisearch/MeiliSearch) is an ultra-relevant and fast search engine**. In other words, it can return the most relevant results of your dataset under 50 ms, and therefore it gives a strong sense of immediacy.

Plus, without the need to configure anything, it handles search miss-spelling: i.e. Typos. Try to submit "devose" instead of "devise" and Meilisearch will return "devise" as the first result.

Finally, **Meilisearch is open-source and integrates a simple [RESTful API](https://www.meilisearch.com/docs/reference/api/overview)**. You can seamlessly communicate with the API using cURL or [one of Meilisearch's wrappers](https://www.meilisearch.com/docs/learn/what_is_meilisearch/sdks).

Creating the alternative search bar
-----------------------------------

All the [gems data](https://rubygems.org/pages/data) are available on the RubyGems' website as a PostgreSQL dump file and are updated daily. Thus, I wrote a Ruby script to download the latest dataset, parse the PostgreSQL dump file, and push all the data into my Meilisearch instance. Of course, it uses the [meilisearch-ruby wrapper](https://github.com/meilisearch/meilisearch-ruby) to communicate with the API. This script is hosted in Heroku and runs every day thanks to the Heroku Scheduler.

About the Meilisearch instance, at Meili we manage an internal Kubernetes cluster, which is a handy tool to host demos like this one. For curious readers who would like to find out more, Meilisearch is quite [easy to download and run](https://www.meilisearch.com/docs/learn/getting_started/installation) by yourself (Homebrew, APT, Docker...).

Regarding HTML and CSS, I kept much of the original structure of the RubyGems' website. My intention was to develop a "search-as-you-type experience" in the same spirit as the original website. The front is deployed using GitHub Pages.

![](https://unable-actionable-car.media.strapiapp.com/chang_duong_F1_UI_4q_Dv1r_Q_unsplash_e212aa22e9.jpg)

Easily improving the relevancy
------------------------------

Without having to set anything up, Meilisearch returns pretty relevant results. Our search engine can quickly find the most suitable packages when inputting gem names like "devise" or "faraday". Unfortunately, for now it’s not always the case with keywords.

Let’s get back to my "pagination" example. If I run the search again without configuring anything, the first result to be displayed by Meilisearch will be the [Pagination](https://rubygems.org/gems/pagination) gem. I don't see [Kaminari](https://rubygems.org/gems/kaminari) at all in the results. That’s because by default, finding a document with a requested word in the title takes precedence over a document with a requested word in the description. Since there are many gems containing "pagination" in their title in the dataset, it explains why Kaminari does not appear at all.

I needed Meilisearch to include the libraries popularity as well. In my dataset, the popularity of the Ruby gems is indicated by the number of downloads. I classified my gems in eight groups of fame (downloaded more than 50M times, more than 30M, and so forth) from `0` to `7`. The latter being considered as the most famous group.

I added this information to every document (i.e. gems) as a field named `fame`. Then, I integrated this rule in Meilisearch settings as a custom ranking rule.

![](https://unable-actionable-car.media.strapiapp.com/5_519da9ed48.png)

Take a look at this snippet above. To put it simply, Meilisearch will execute one by one all of these rules (`_sum_of_typos`, `_number_of_words` ...) and sort your documents following this sequence. When I add my custom rule, i.e. `fame` in `rankingOrder` and `fame: 'dsc'` in `rankingRules`, I'm in fact asking Meilisearch to sort by fame in descendant order.

You might have noticed I have a second custom rule in the example: `total_downloads`, so that my results will be sorted by the number of downloads. But since I chose to place this rule at the end of the list, meaning it was considered less important than the other ones, it will be the last one to be applied. The order of sequence definitely matters.

I'm not going to give further details about the [Meilisearch default ranking rules](https://docs.meilisearch.com/learn/core_concepts/relevancy.html#ranking-rules), even if it’s a particularly interesting topic. Describing how our search engine works deserves indeed a separate article of its own! 😉 _Spoiler alert: Meilisearch uses a bucket sort!_

Now, if you type a [global keyword like "pagination"](https://rubygems.meilisearch.com/?q=pagination) you will find Kaminari in the first place; and if you try again with a [less famous gem name like "pagy" for example](https://rubygems.meilisearch.com/?q=pagy) you will still get the gem you expected! 🎉

Meilisearch + you = 💛
----------------------

These minor settings were really easy to integrate, and your projects might require the same kind of behavior.

Here are some useful links if you want to get ready for your own Meilisearch experience:

*   the [documentation](https://docs.meilisearch.com/)
*   the [Meilisearch's repository](https://github.com/meilisearch/MeiliSearch) on GitHub: support Meili by starring it! ⭐️
*   the [repository of this project](https://github.com/meilisearch/finding-rubygems/)

If you are interested in our project, how it works, or if you have any feedback, do not hesitate to [contact the team](mailto:bonjour@meilisearch.com)! 😁

Meilisearch finds Rubygems

Today, I am about to guide you in the depths of crates.io and how [I made an alternative search bar](https://crates.meilisearch.com) using our instant search engine: [Meilisearch](https://github.com/meilisearch/MeiliDB).

![](https://unable-actionable-car.media.strapiapp.com/crates_io_demo_798a562ddd.gif)

The Meili instant, relevant, and typo-tolerant search engine

[Crates.io](https://crates.io) is the official website that stores Rust community crates (packages), and it's the place where the cargo package manager uploads, updates, and downloads those.

[Sean Griffin](https://twitter.com/sgrif) is part of the crates.io team and maintains the current search engine of it along with the whole website. [Kornel Lesinski](https://kornel.ski/) built [lib.rs](https://lib.rs/) an alternative to [crates.io](http://crates.io/) and uses [Tantivy](https://github.com/tantivy-search/tantivy/) to power its search bar. To be honest, I prefer its color design, that's why I took it for our search demo.  
  
I decided to run our instant search engine and test its relevancy overtime against these existing solutions. Our search engine uses completely different algorithms; it is based on prefix search and is typo tolerant.

![](https://unable-actionable-car.media.strapiapp.com/current_results_1_73457862c1.png)

Meili is typo-tolerant and supports a whole lot of other features

So, I asked myself: Why not use our new instant search engine and make it useful to my beloved community? It would give us lots of feedback and probably some pull requests during the process.

* * *

At Meili we manage an internal Kubernetes cluster, this is useful to host demos for clients. The Meilisearch server for this demo currently run on a pod in this cluster.

To make Meilisearch exhibit those crates, we needed to find all the currently available packages on crates.io. Fortunately, this index is available on GitHub in the form of several subfolders with the names and versions of the packages, containing something like _32 000_ files. A commit is done each time a crate is updated to a new version, or a version is yanked.

So, I used the [crates.io-index repository](https://github.com/rust-lang/crates.io-index) to **initialize** our newly created Meili search engine but needed more data first, like, the description, keywords, and categories of each of those crates. Again, the Rust crates.io team was here for us, I talked to [Pietro Albini](https://twitter.com/pietroalbini), and he pointed me to the not rate-limited servers that deliver packages content.

Now that we can retrieve useful data, I create an [async crawler](https://github.com/Kerollmops/meili-crates/blob/67770fe179436eee6478cdebff7d8613f9263049/src/lib.rs) that downloads, extracts, retrieves the Cargo.toml, and upload the essential data to Meilisearch.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_2019_11_08_Meili_Dashboard_c9d2ae42f1.png)

The Meili dashboard interface showing raw documents

Meilisearch now understands that data and gives us instant, relevant and typo tolerant responses. But, what about new crates? We want to be notified about new crates and be able to send those to Meilisearch.

[Docs.rs](https://docs.rs) is the official website to compute and store the documentation of all the Rust packages hosted by crates.io. It diffs the crates.io index each minute to know about new crates updates. Luckily, it provides an Atom feed of those updates.

It is where [Heroku](https://heroku.com) enters the game. Heroku offers something like _1000_ free hours per month of computing power on their servers and provides us schedulers. We can consume those credits and freely [ask docs.rs about newly updated crates](https://github.com/Kerollmops/meili-crates/blob/67770fe179436eee6478cdebff7d8613f9263049/src/bin/live.rs), every 10 minutes, by fetching the Atom feed, download the updated packages like previously, and finally update our search engine, **live**!

![](https://unable-actionable-car.media.strapiapp.com/heroku_live_timings_1_fbb7661c94.png)

Fetching new crates takes us 4 seconds

Search results were satisfying but not as expected, there was something wrong. When we wrote "serde", for example, the first result was relevant, but the next ones were not. It was related to the fact that Meilisearch did not have enough data to rate crates except the query matching words.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_2019_11_10_Meili_crates_browser_2000_1_5cfc7e603c.png)

Meilisearch doesn't know how to settle equal crates with only the matching words

**Downloads counts** amazingly improved the search results. [This data is available through crates.io](https://crates.io/data-access). A full database export is done each day, and it contains the number of downloads of each crate. I decided to use those as the last ranking criterion to help Meilisearch settle about crates considered equal.

![](https://unable-actionable-car.media.strapiapp.com/Screenshot_Meili_crates_browser_2000_good_1_21f3eb0098.png)

Meilisearch shows better results thanks to the number of downloads

I deployed a Heroku scheduler to run each day to update all the crates downloads; it takes something like 30 seconds to download the tarball, extract it, read the CSV and upload the _32 000_ crates downloads counts to Meilisearch. So we are far from _1000_ free hours per month.

* * *

I think this search demo is pretty good, but I also thought about adding synonyms and stop-words as Meilisearch supports those. For example, it would be pleasant to write "db" and see results associated with "database". Stop-words would help ignore useless words like "the" or "that" which could pollute search results sometimes, but we need to be careful as crates names could be composed of stop-words.

Meilisearch also supports basic filtering, and it would be great in the future to be able to search for crates in a category or with a specific keyword.

All these improvements can be made by you; [the demo is open source](https://github.com/Kerollmops/meili-crates). The [core engine source code](https://github.com/meilisearch/MeiliDB) is also available on GitHub, that's the whole point of this article! Take a look and talk about it, more people involved, more features!

Meilisearch latest news and company updates

MeiliSearch in production: taking it to the next level

Meilisearch finds Rubygems

Search for Rust crates with Meili