SD Times Open Source Project of the Week: Quickwit

Quickwit is an open source search engine for object storage with a sub-second delay for large data sets.

The project, created by the authors of the search engine library Rust tantivy, recently raised $ 2.6 million in the initial round led by FirstMark and firstminute with many other participants.

Quickwit 0.2 was launched in January with new features such as the ability to use Kafka using exactly one semantics, a search flow API, a PostgreSQL meta-repository, tag trimming, and a proper indexing pipeline.

The search engine uses a different approach than Elasticsearch, which relies on document replication. Meanwhile, Quickwit indexes documents on a single node and leaves it in the object store to replicate the resulting index files.

“We need to repeat the documents somewhere to consider them swallowed,” wrote Paul Mazurel, co-founder of Quickwit. blog post. “We allow users to connect Quickwit directly to their favorite distributed queue. Now Quickwit only supports Kafka. “

Frequent options for using Quickwit include searching logs, adding full-text search capabilities to OLAP databases such as ClickHouse, and searching backups that are in the cloud storage by adding Quickwit index files to the same storage.