Squeezing millions of documents in 128 TB of virtual memory

<Callout>
  Due to the recent documentation site migration, our documentation doesn't currently use Vale. You can still view our style guide on [GitHub](https://github.com/meilisearch/documentation/tree/b6837449f09587f9d6df92e1eea0c44cc394f398/.vale/styles/Meilisearch).
</Callout>

Writing is a time-consuming process. Depending on the writer and the review process, it can take a while to get content ready for publication. Whether you’re a solo writer or a big team, prose linters can help ensure a consistent tone and style.  

Similar to code linters, prose linters automatically check your text for errors. Unlike grammar checkers, which highlight violations of grammatical rules, prose linters focus on how you can make your text better by addressing common usage problems like extra spaces, repeated words, excessive use of jargon, sexist language, and incorrect capitalization.  

Prose linters can also provide a framework for creating and enforcing an [editorial style guide](https://www.writethedocs.org/guide/writing/style-guides/). This helps with the review process as you can now focus on reviewing the content itself instead of pointing out typos and preferred usage patterns. This is particularly important when working in open-source projects like Meilisearch, where you have many contributors unfamiliar with your style guide.  

What is Vale?
-------------

[Vale](https://vale.sh/docs/vale-cli/overview/) is an open-source, highly customizable, syntax-aware prose linter. It supports documents written in many different formats such as Markdown, HTML, reStructuredText, AsciiDoc, DITA, and XML.  

Vale isn’t your only option when it comes to prose linting. There are many other open-source tools available, including [proselint](http://proselint.com/), [textlint](https://textlint.github.io/), and [alex](https://alexjs.com/).  

At Meilisearch, we decided to go with Vale because it’s fast, easy to set up, flexible, and comes with existing rules to help you get started.

Where do I start?
-----------------

Though it may seem daunting, setting up Vale can be fairly straightforward if you start small and keep things simple. In this post, we’ll go over how to use Vale in a project much like [Meilisearch’s documentation](https://github.com/meilisearch/documentation).

### Step 1: Style guide

The first step is to create a style guide. A style guide ensures a consistent tone and style regardless of how big your team gets. It establishes standard practices when people may have different opinions, such as whether or not to use the oxford comma.  

If you don’t have an in-house style guide, you can check out [Google’s](https://developers.google.com/style) or [Microsoft’s](https://learn.microsoft.com/en-us/style-guide/welcome/) to help you get started. Over time you will memorize most of the rules, but it’s possible to overlook mistakes and sometimes forget the rules altogether. We’re only human.  

And that is where Vale comes in. It allows you to “codify” a style guide and checks your text against all the rules in that style guide. If it detects any issues, it displays suggestions, warnings, or errors on the console.

![](https://lh4.googleusercontent.com/jqa7GUDjATPh8UaNybvMrcuCURAdHu1mMrrm_M6T9MdKq_M3CPf0i8Gs60_yEt08zZGrCd43ZxsrdRDQ3nP8EBEXitjEeqFdDHYlD6HZ19CcGhkvpHHlR6NwZxbjkOMl6IpzUR1RCJ9P6myZoEHIOhs)

### Step 2: Install Vale

I’m using macOS and ran `brew install vale` in my console to install Vale.

If you’re using a different operating system, check out Vale’s documentation for [installation instructions](https://vale.sh/docs/vale-cli/installation/).  

Verify the installation by typing  `vale -v` in your console. If this command returns Vale’s version number, the installation was successful.  

Finally, create the following files and folders:

```json
├── .vale.ini
│   ├──  styles
│   │	    ├── Style guide
│   │       └── Vocab
```

### Step 3: Configuring Vale - vale.ini

Create a `vale.ini` file at the root of your repository. This is the Vale configuration file where you define what you want Vale to do and what files to lint. Let’s start with a basic setup—you can always add to it later based on your project’s needs.

```json
StylesPath = .vale/styles
MinAlertLevel = suggestion

Vocab = word_list

[*.md]
BasedOnStyles = Meilisearch
```

*   `StylesPath` is where Vale looks for your style guide (more on that in the next step). The path can be relative or absolute to the location of the `vale.ini` file.  
    
*   `MinAlertLevel` specifies the minimum alert level that Vale will report. By default, it’s set to `warning`. The other options are `error`  and `suggestion`.  
    An error indicates you did something wrong, like using extra spaces or making a typo. A warning isn’t as severe as an error, but indicates something you should avoid, like making sure your sentences don’t become too long. A suggestion is a recommendation to do something that is usually—but not always—a good idea, like breaking your sentence into two instead of using a semicolon.  
    If a rule is set to `suggestion`, you will see suggestions, warnings, and errors. If it is set to `warning`, Vale will only show errors and warnings, no suggestions.  
    
*   `Vocab`: This is where you create a directory containing the `accept.txt` and `reject.txt` files. Both files accept words, phrases, and [regular expressions](https://ubuntu.com/blog/regex-basics). If your text contains words that do not exist in the dictionary (e.g. “Meilisearch”), you can add them to `accept.txt` and Vale won’t angrily scream at you for making a “typo”. Vale will flag all occurrences listed in `reject.txt` as errors. This can be useful when you want writers to avoid a specific word—if you are writing about search engines and databases, for example, using “indexation” can be confusing.  
    
*   `[*.md]` tells Vale to only lint markdown files. If you want to lint plain text files, use `[*.txt]`.  
    
*   `BasedOnStyles` specifies the style guide Vale should use for linting.  
    

You can specify other settings, including what tokens and HTML tags to ignore and what Vale should consider an individual word. You can read more about the `vale. ini` file in [Vale’s documentation](https://vale.sh/docs/topics/config/).

### Step 4: Rules and the styles folder

As I mentioned earlier, you need a style guide to use Vale. You then convert this style guide into something Vale can understand: rules.

Rules use different [extension points](https://vale.sh/docs/topics/styles) to perform specific tasks. For example, the `existence` extension point looks for the existence of a particular token, `repetition` looks for repeated tokens, `spelling` implements spell checking, and so on. In Vale, each rule is a YAML file. The `styles` folder contains the individual rules that make up the style guide.  

If you don’t want to create your own style guide or need a starting point to build from, Vale comes with ready-to-use style guides that you can apply to your docs and start linting. Here are some style guides that helped us get started:

*   [Google](https://github.com/errata-ai/Google)
*   [GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.vale/gitlab)
*   [RedHat](https://github.com/redhat-documentation/vale-at-red-hat/tree/main/.vale/styles/RedHat)

You can find more on [Vale’s GitHub](https://github.com/errata-ai/packages) repository.  

Let’s start with a rule for sentence length. Your style would say something like “Ensure sentences don’t exceed 40 words”. This is what the rule looks like as a YAML file:

```json
# Warning: Meilisearch.SentenceLength

# Counts words in a sentence and alerts if a sentence exceeds 40 words.

extends: occurrence
message: 'Shorter sentences improve readability (max 40 words).'
scope: sentence
link: https://docs.gitlab.com/ee/development/documentation/styleguide/index.html#language
level: warning
max: 40
token: \b(\w+)\b
```

This rule counts the words in a sentence and throws a warning if it exceeds 40 words. The `scope` is set to `sentence`: this ensures that Vale does not apply this rule to other parts of the text, like headings or tables.  

The `level` is set to `warning`. Written text is complicated, and Vale will find false positives. There are no sure-fire ways of deciding when a rule should be a suggestion, warning, or an error. You will need to make decisions and learn as you go.  

I suggest reviewing your rules over time to update and, in some cases, delete outdated rules. Initially, the Meilisearch docs didn’t have a rule on sentence length. When we added it, the maximum length of a sentence was 45. Now it’s 40, and we plan on bringing it down to 35.  

You can also enable or disable specific rules within a style guide by adding them to `vale.ini`:  

```json
Meilisearch.Headings = NO
Meilisearch.Spelling = NO
Meilisearch.Semicolons = NO
```

The above lines disable the Heading, Spelling, and Semicolons rules from the [Meilisearch style guide](https://github.com/meilisearch/documentation/tree/main/.vale/styles).  

### Step 5: Run Vale

Now, when you use the following command on your console to lint your whole project:  

```json
vale .
```

Vale will check all your files against the rules stored in `BasedOnStyles`. If Vale detects any issues, it will display suggestions, warnings, and errors on the console.  

You can also lint individual files using:  

```json
vale {file_path} 
```

### Step 6: Automate Vale checks

All the checks we’ve discussed so far are for your local files. Once you’re confident the rules work as intended, you can automate these checks using the [Vale GitHub action](https://github.com/errata-ai/vale-action)! At the Meilisearch documentation repository, we configured it to run for every pull request. As mentioned before, Vale may find false positives. Since you don’t want a PR blocked because Vale isn’t working as it should, I recommend starting with a few rules, and slowly tweaking them to avoid failing checks.  

### Conclusion

That is all, folks! I hope I was able to help you get started with Vale (and a style guide). This was a quick overview to introduce you to Vale’s features. Tweaking it to your needs takes time and many, _many_ iterations.  

![](https://lh4.googleusercontent.com/zp5NecwV9Bc8pMsSf2QeV8yOMMGWoJR8bENrOuQNNFWat7J0KtzDq1n9oBrEW1lKgJO8utxivJS9UqXuMutCmft2o8x3LIE3bUIQFuqcCTkgQ6H5y6hzVAEXgEPYkGHlwnTZnaJ3r3rEq4LyWGFYOVU)

Once configured, Vale can automate parts of the review process and allow you to focus on the parts of a text that computers are not very good at. At least not yet.  

Oh, and if you’re curious, check out [our style guide on GitHub](https://github.com/meilisearch/documentation/tree/main/.vale/styles) to see how we use Vale!

Prose linting with Vale

Meilisearch currently maintains more than 50 open-source projects on GitHub. In the last six months alone we merged over 1000 Pull Requests (PRs), 160 of which came from external contributors.

Ensuring the codebase remains stable and that code quality is always top-notch requires us to be very careful when it comes to merging a new PR. As the number of contributions grows, the automation of this process becomes a necessity in order to keep everything simple for the contributors and safe for Meilisearch's codebase.

In this post I will describe why we decided to use [Bors](https://github.com/bors-ng/bors-ng?ref=blog.meilisearch.com), a merge bot for Github pull requests, to create a better environment for both maintainers and contributors.

## Our merging process
During an open-source contribution, any changes must be done on their own separate branch. Afterwards, when the work on this branch is finished, one of our main requirements before merging is that the new branch is up-to-date with the primary branch—called `main` in most of our repositories.

If the PR is not brought up to date with `main`, you could say that most of the work is done, but *most of the work* is not enough for us. Even if all the automated tests pass on the PR itself, they are not guaranteed to pass when merged if the PR branch and `main` are not aligned.

Before using Bors, this meant we often had to ask PR authors to **rebase their branch** if, for instance, a different PR had been merged meanwhile. This made contribution more complex than we wanted, since not everyone is comfortable with Git’s `rebase` command.

### Why rebasing?
At Meilisearch, we believe keeping a clear Git history is really important for two main reasons:
1 It allows everyone to follow all changes made to the project
2 When debugging, a clear history helps identify which commit introduced the issue

The goal of a rebase is to move the commits from a PR branch to right after the last commit on `main`. Therefore, **rebasing keeps your Git history clear and consistent with the order in which you have merged your PRs**.

Rebasing is not the only way to bring a branch up-to-date with the `main` branch: you can also pull the changes from `main` to the PR branch, for instance. The issue with this approach is that the pull would add a merge commit to the PR branch, which would not keep the repository history clear.

Keeping in mind how important it is to maintain a clear commit tree, **rebasing is the only way to guarantee that the work that is going to be merged fits the current work in production**. If the tests pass on an up-to-date branch, they will also pass once merged into `main`, ensuring your production branch will not break.

## Finding the perfect tool
With our rebase requirements in mind, I began my search for the perfect tool. I wanted to be efficient, so I narrowed my scope and only evaluated tools that were:
* **Free and open source:** the tool had to fit the values of Meilisearch and its community
* **Highly rated:** the tool had to be stable and reliable, since any mistakes when handling PRs are bound to have impactful consequences
* **Well documented and easy to set up:** good documentation saves time that we can use improving Meilisearch

Unfortunately, I did not find as many solutions as I expected within this scope.

The first tool I tried was [Shipits](https://github.com/shipitjs/shipit?ref=blog.meilisearch.com), which I discovered thanks to [this article](https://engineering.shopify.com/blogs/engineering/successfully-merging-work-1000-developers?ref=blog.meilisearch.com) on Shopify’s engineering blog. Though Shipits seemed really solid and promising, I felt it was a better fit for complex integration processes instead of open-source projects just trying to keep their Git history clean.

Another tool I tested was [Kodiak](https://github.com/chdsbd/kodiak?ref=blog.meilisearch.com). The set up was really easy, in large part due to their great documentation. It was a good candidate for our use-case, but the automatic merge process is based on labels that we would have to manage, which does not really fit our workflow. For example, our minimum requirement when merging a PR is obtaining the approval of one reviewer; sometimes, though, I want two people reviewing one PR and it is easy to forget to communicate this to Kodiak. The moment it receives the first approval, Kodiak automatically merges the PR because the one-reviewer-requirement has been respected.

Another thing we noticed was that Kodiak does not update the branch originating the PR by rebasing it and instead creates a Git commit by pulling the main branch into the PR branch, which, as we saw, muddles the repository’s history. In the end, Kodiak did not fit our needs, even if its setup and configuration provided a seamless and hassle-free experience!

Contrary to all the other tools I looked into, **[Bors](https://github.com/bors-ng/bors-ng?ref=blog.meilisearch.com)** was built following a unique and powerful principle described by the original Bors creator in [this article](https://graydon2.dreamwidth.org/1597.html?ref=blog.meilisearch.com):

> *The Not Rocket Science Rule Of Software Engineering: automatically maintain a repository of code that always passes all the tests.*

This "rule" perfectly fit our requirements and made Bors our final choice.

## How does Bors work?
Bors is a merge bot—meaning, it is an application that **automates aspects of the PR merging process**. Its setup is straightforward and, like Kodiak, very well documented. Indeed, configuration can be a single `bors.toml` file containing the names of tests we want to execute before merging.

All Bors commands have to be given through comments in a PR. To merge a PR, all we need to do is use `bors merge`.
Bors first applies a rebase before running the tests and, if they succeed, merges them back into `main`. To be more accurate, Bors does not actually rebase the branch of the PR: it merges the commits of the PR into another branch, called `staging`, that is already up-to-date with `main`. If the tests do not pass on `staging`, Bors returns a failure and does not merge the PR.

The icing on the cake: **Bors is also able to manage merging multiple PRs at the same time**, which means we don't need to merge them one by one. Each PR is added to a queue and will be checked with the last version on `main` at regular intervals.

If Bors finds conflicts between the PR and the main branch, it cannot make the decision about which final changes to apply. In this situation, the contributors have to rebase the branch manually—if that happens to you, I wrote this [quick tutorial](https://gist.github.com/curquiza/5f7ce615f85331f083cd467fc4e19398?ref=blog.meilisearch.com) on how to rebase from a forked repository.

## Conclusion: which problems have we solved?
We at Meilisearch have used Bors for a few months now and are really satisfied. We know that keeping the Git history clear while providing a stable codebase can be very complex, but so far Bors has helped us greatly optimize our workflow. With Bors integrated in our workflow there is:
* No need to ask and wait for PR authors to rebase
* No need to test and merge PRs one by one
* No frustration for our contributors: they all do amazing work, but some of them are not really comfortable with Git’s rebase feature, which is indeed not always easy to use—especially when working from a forked repository!

**Bors makes both maintainers and contributors happier**; we can, therefore, recommend it for projects with the same use-case as ours.

Automate pull requests merging with Bors

SpO₂ is oxygen saturation and is used in medical person monitoring.

I am Kerollmops, I am the CTO of [Meili](https://www.meilisearch.com/) and today I am releasing [SpO₂](https://github.com/Kerollmops/spo2/).

At Meili, we needed a tool that allows us to monitor our pods, we already have [vigil](https://github.com/valeriansaliou/vigil) which health checks our front page and backend, but the number of these services is limited. We do not pop new front or backend servers dynamically (for now). When we create new search engines for the user we instanciate a kubernetes pod, we need to monitor the health of this service. Adding each of those URLs by hand in the vigil config file is not a solution.

So we decided that we needed a simple tool, a tool that can accept HTTP requests to register/unregister URLs to health check. We use the [new async/await Rust syntax](https://blog.rust-lang.org/2019/09/30/Async-await-hits-beta.html) along with [tide](https://github.com/http-rs/tide) for the http server, no big deal here.

Our current cloud provider is [Digital Ocean](https://www.digitalocean.com/), therefore, we cannot host our SpO₂ service there. We chose [Scaleway](https://www.scaleway.com) as it is way cheaper, and it works out of the box. We need persitent storage of the health checked URLs. What would you do if those are only stored in RAM? What if the server restarts? I already worked on [a disk backed key-value store in Rust named Sled](https://sled.rs/). So we chose to rely on it.

![](https://unable-actionable-car.media.strapiapp.com/screenshot_d586f10f06.png)

[@qdequele](https://quentin.llmops.com) built the front-end with vanilla javascript. Using Websocket we are able to display in realtime the pods status. As most humans do not stand in front of a TV the whole day, spotting color changes to notify the devops, we decided to implement notifications. We are using Slack for all of our monitoring tools, and because it is as simple as a webhook to implement, we went with it.

In the last release, we made some improvements to the Slack notification system. We now batch status changes events by 40; this means that SpO₂ sends one message with at most 40 events and regulates channel spamming. It also displays the HTTP status related to an unhealthy measurement and the error message on an unreachable one.

SpO₂ does not support SSL/TLS by itself, neither for the HTTP nor the WebSocket endpoints. We needed this kind of security so we looked at NGINX, a tiny little obscure reverse proxy server which we configured with basic authentication. It is not an easy task and because we are cool we made the documentation to help you do the same.

Do not hesitate to share or star this project, pull requests are welcome 😊

And just as a side note, we do not measure humans but machines actually.

SpO₂ the little dynamic monitoring tool

Today we are releasing [MeiliES](https://github.com/meilisearch/MeiliES), our event store which is fully implemented in Rust.

Event sourcing is a way to _"Capture all changes to an application state as a sequence of events"_ as [Martin Fowler describes it](https://martinfowler.com/eaaDev/EventSourcing.html). The first purpose of the server is to publish events of a stream to subscribed clients. A client can also specify from which event number it wants to read, therefore it is possible to recover from a crash by rebuilding a state with new events only.

Since [a message queue is not made for event-sourcing](https://medium.com/serialized-io/apache-kafka-is-not-for-event-sourcing-81735c3cf5c), there are few available applications for event sourcing: the main option being [EventStore](https://eventstore.org/), an event sourcing database made in C#.  
So we began using EventStore for our event sourcing needs and ran against some troubles:

*   Portability: EventStore is built with C# and uses mono, making it quite difficult to install and run on UNIX
*   Support: We asked for information about EventStore commercial support and never got any response
*   Stability: the Rust client is not production-ready, and despite making some merge-requests there was still a lot to do while many of the available features were not useful to us.

So we decided to implement our own event store to resolve these issues.

MeiliES is an _Event Sourcing database_ that uses the _RESP_ (REdis Serialization Protocol) to communicate. We use the Redis protocol to simplify clients implementation. The portability problematics are resolved by using a full Rust implementation (we are using [Sled as internal storage](https://github.com/spacejam/sled)).

Installation is as simple as doing a `cargo build --release`. If you do not want to clone the repository you can [download the binaries from the Azure pipeline](https://dev.azure.com/thomas0884/thomas/_build) or run it using docker `docker run -p 6480:6480 --rm getmeili/meilies:latest`.

If you have any questions regarding MeiliES, you can tweet [@meilisearch](https://twitter.com/meilisearch) or make an [issue on GitHub](https://github.com/meilisearch/MeiliES) 🎉

Meilisearch latest news and company updates

Squeezing millions of documents in 128 TB of virtual memory

Prose linting with Vale

Automate pull requests merging with Bors

SpO₂ the little dynamic monitoring tool

MeiliES - Event sourcing in Rust