Tangentially related if you need search without the clustering and high availability story of elastic search and friends I highly recommend Xapian.
Its like the SQLite of search. Single library that provides the basic set of features you would expect in a quality search experience: facets, ranked search, boolean operators, stemming etc etc.
FWIW, at the end of the day ElasticSeach is just a set of Java classes.
If you’re using the JVM already and want the SQLLite of search, you can instantiate an index in your own JVM, load documents, and get compatible query features.
You can also use Lucene directly, which is simpler, but does have some nuances that differ if you want perfect query compatibility between a deployed ElasticSeach cluster and an in-memory search.
A great example of Xapian in action is the email client Mu or Mu4e (for terminal or Emacs, respectively), which queries massive amounts of emails blazingly fast.
That looks great, I could use that right now, thank you!
I was previously looking into embedding TypeSense somehow (but not portable), MeiliSearch (but their API triggered a panic response), or SQLite's full-text search extension (but limited features e.g. language support). Using a dedicated library would be much better and this seems to be packed with features.
>MeiliSearch (but their API triggered a panic response)
Could you expand on that with examples? I only played around with it lightly but it seemed ok via the Python SDK.
I've been recently leaning towards Meilisearch for some client facing search features. This was over Typesense because, i believe Typesense is totally in-memory, but in some cases my dataset might not fit in RAM(atleast not at a reasonable cost). I was fine with taking a performance hit for disk access, provided i could have thousands of separate indexes with separate permissions etc. and directly call them from the browser app.
Admittedly it's been almost a year since I tried it, but:
* If you have more than 65,535 words in a document, words after that limit will be SILENTLY IGNORED (not indexed). This was 1,000 words when I tried it, which was very limiting
* Documents where your query word happens sooner are ranked above documents where your query words happen later (in the document string). You can't turn that off
* You can't get the "match" information (i.e. the position where your query matched in the document) unless you retrieve that attribute (i.e. tell the database to send you the entire document content)
* The API uses HTTP verbs wrong, e.g. POST is used to "Add or replace documents" and PUT is used to update documents (it's exactly backwards from the intended use, per RFC 7231)
The GPLv2 license is potentially problematic. Even if your product is open source, it can cause problems with libraries that use licenses that aren't compatible with the GPL, such as openssl prior to v3.
To be clear, I don't hate the GPL license, although I think it is probably not the best choice for a library. But from a licensing perspective, xapian is pretty different from SQLite
I work on Typesense and I can speak to it from Typesense’s perspective.
Typesense holds the entire index in memory, in order to enable fast search where every keystroke returns results in 50-150ms. So it’s built primarily to enable user-facing search.
It’s not a good fit for log search since putting your entire log dataset in memory can become expensive depending on the size of your logs, and you typically don’t need search-as-you-type for log search.
Zinc on the other hand seems to be designed specifically for log search.
There's many "lightweight alternatives" to Elasticsearch (see this thread, which lists at least two already).
Alternative doesn't always mean they are API compatible, it's an alternative and not a drop-in replacement. My question was about how comparable it is apart from the API compatibility.
> Tired Of High Cost And Complexity Of Your Log Systems
Is this really the main thing ElasticSearch is used for now? In 2016, I perhaps naively chose ElasticSearch (via AWS's managed service) for an actual user-facing search feature. That service is still running, and now I'm looking to move it to self-hosted, but didn't want to actually run ElasticSearch (or more likely now, OpenSearch). So I looked at the ZincSearch home page, and the quoted heading seems odd.
Elasticsearch and ZincSearch both are general purpose search engines based on inverted indexes that can be used for app search as well as log search. In my stint at AWS I found that most companies used elasticsearch and anecdotally 90%+ users used it for log search. Log search was also the use case that I had in my mind when I built ZincSearch for my own needs and I found ES to be lacking. Here is a blog that I wrote on my motivation around building ZincSearch - https://prabhatsharma.in/blog/in-search-of-a-search-engine-b...
Elasticsearch is free as in beer and you can use the properly OSS Opensearch as an alternative. So, I don't really get the cost argument.,
As for hardware cost needed to run Elasticsearch/Opensearch, it scales down as well as it scales up these days. I have a couple of ~70$/month clusters with millions of documents in each. Not a problem that I need to fix. I can scale them up to cost a lot more and handle billions of documents easily. That's in Elastic Cloud, which would be the expensive option. If you go bare metal, you can get way more bang for your buck obviously. A couple of Xeon machines with some decent amount of CPU/Memory/SSD go a long way.
As for complexity, they make it pretty easy to get started. What you do beyond that is kind of a luxury problem. You get all of these features out of the box but that doesn't mean you have to use them. But it's nice to have the option when you do need them as opposed to running into a brick wall because you are missing key features in these so-called drop in replacements. Depends on your needs. If you don't know the first thing about your search needs, keep it simple and go with mainstream solutions. Sign up with some managed opensearch/elasticsearch provider, get your cluster ready in a few minutes and start throwing some data at it. It's that simple. You can get it running in docker as well if you need something to develop against.
It seems a bit of a rite of passage to be implementing your own search engine these days without being hindered too much by not being completely up to speed with the field of information retrieval and all the wonderful algorithms and sollutions. Guilty of that myself actually; I built a tf/idf implementation at some point. It's not that hard. From there to something that gets close to Elasticsearch/Opensearch/Solr/etc. takes a lot of work and skills, however. There are a few nice solutions out there but also lots that aren't so hot.
I've not used Zincsearch so I'll reserve my judgement. However, their main pitch seems to be that they are cheaper than a thing that is free and partially compatible with it (for ingesting data only) but a bit lightweight (not so scalable, way less features). So maybe pay the difference of 0$ and get something without the limitations that come with being "lightweight"? Just saying. Not a great pitch if you have to name your main competitor in it. Kind of emphasizes that you are trying to catch up rather than disrupt. Does it have any actual things it does better? That should normally be the center piece of a pitch.
I haven't used it for nearly a year now, but in my last role we used ES for user-facing search. We had to turn to documentation, forums and support for a lot of problems we faced and I got the impression ES was mostly being used for logs while user-facing search was the less common use case. Support didn't have answers for us on a number of occasions and online documentation was equally sparse.
Well everyone has logs, but not everyone has user-facing search features.
I personally wouldn't recommend self-hosti gElasticSearch for logs; it's such a commodity now that it cheaper to get something like Datadog or another manage service (but do check the pricing,the last time I checked the managed logging service from Azure was very expensive for what it did).
If you're on the jvm, lucene is actually pretty trivial to embed. The complexity comes around representing the data to be seached as "documents", ingesting the documents to be searched, reimporting when they change, and lucene's goofy query syntax.
Running ES for simple site is trivial. If you're not pushing hundreds of GB or hundred+ searches per second, ES/opensearch is just pretty much "just run it"
It sure didn't look easy when I looked at the OpenSearch docs a few months ago. Any pointers to an easy way of configuring OpenSearch (or ElasticSearch) when I only want to run it on one machine, and it will only be accessed by other processes on that same machine (so I can just access it via localhost)?
I guess I'll ask the same question that I ask every time a search product appears here – does it handle languages other than English? Last I checked Meili search was still English-only, for that the inbuilt Postgres search is honestly good enough for most usecases.
We do support other languages than English, it actually depends on what are the languages supported by our tokenizer https://github.com/meilisearch/charabia, so any language that uses whitespace to separate words(including English), Chinese, Japanese, Hebrew and Thai (Japanese and Thai might work a little bit less)
Thank you for posting back. I have a list of technologies to try for an upcoming project, and Meilli is one of them. Our language is Hebrew, and I had just written in my notes file that Meilli is a no-go because of the GP post. Good think that I thought to refresh the page before moving on.
Such a comment on HN can be very damaging, as I generally trust the posters here more than in other places. I know that I'll have very little time to properly assess technologies so I'd like to shorten the short-list as much as possible. Again, thank you.
being written in go does not imply anything about ease of deployment. kubernetes is written in go.
in fact, the type of project that sees something that works exceptionally well (like ES) and then decides to partially reimplement it (to the point of maintaining API compatibility) is a pretty big red flag that it is NOT going to be easier to use in the long run.
It is definitely super easy to deploy and run compared to ES. You can get up and running in less than 2 min. And you are right, messaging needs to be better.
it's almost certainly their "drop in ES replacement" story (https://news.ycombinator.com/item?id=32938630) since almost every other submission like this is "just" a search with a REST API. But there's _so much_ tooling out there which claims to blast text at ES/OpenSearch that it makes trying to replace ES hard since it requires changing every producer in that story, too
Interesting. I'm not sure why I would chose this over Elasticsearch though? How does performance stack up against ElasticSearch? If the API is compatible with ES does that mean I can use the ELK stack including logstash for ingestion, and kibana for analysis?
Also... One of the things I didn't like about ES was the JSON DSL. It would be nice to create a friendlier query language, maybe something like SQL to query data. I find ES queries written to be ugly to look at, and not intuitive.
I've been working on something similar as I dislike ES's JSON DSL or Kibana's interface: https://log-store.com/
It's not open source but old skool '90s-style freeware. The current single-node version will remain free forever, but the next multi-node (distributed) version will be for-pay.
Zinc is a little too early for performance benchmarks (Still not at 1.0). fluentbit, logstash, filebeat can be used as is with ZincSearch. Kibana is not yet supported but you could use the UI of Zinc (Still in very early phase) which is embedded and you don't have to install separately.
It looks like the selling point is easier setup and operation. Elasticsearch is kind of a pain in the ass to take care of. Maybe they've made that better? That'd be valuable to me.
> Elasticsearch is kind of a pain in the ass to take care of.
Any specific issues you are referring to? At work, we've been using ES heavily for many years and it isn't really something taking up time apart from updating the software every once in a while.
I guess you are referring to the "log search" use case? We don't use it for that so that isn't really an issue, and it's mainly about beefing up memory / cpu.
I’m not sure it’s possible or desirable to make a low memory search index. Like, fundamentally you’re going to want the indexes in RAM for performance, no?
Vespa.ai[12] is great but I don't really know in which category to put them, user facing and enterprise search seems to work well with them.
It's interesting to note that Elasticsearch and Opensearch are general purpose search engine, Solr as well. They are all powered by Lucene, the popular and performant search engine library.
Another general purpose search engine is Manticore search[13].
I would love to see some benchmarks by category :)
Note: I don't know well ZincSearch and put it in the log search as said on their front page.
> It's interesting to note that Elasticsearch and Opensearch are general purpose search engine, Solr as well. They are all powered by Lucene, the popular and performant search engine library.
Another search engine which can be considered general, is not based on Lucene and is not less powerful than Elasticsearch/Solr is Manticore Search [1]
> I would love to see some benchmarks by category :)
I'd love too. We started this work on db-benchmarks [2] , hopefully we'll have resources to continue it. Contributions are very welcome. It's 100% opensource [3]
As an Yahoo internal project, Vespa actually started out very similar to Elastic in that it was meant for fast text search. The team then added ANN search capabilities later on.
Vector search infrastructure is overall still fairly new (https://milvus.io/docs/text_search_engine.md), but up and coming for sure. Disclaimer: I'm a part of the Milvus community.
This will be interesting to kick the tires on. The last gig I was on that used ES - ES would eat tons of memory and quite often crash due to OOM. Wasn't a fan of the query language, either.
Is there a good full text search engine with pluggable storage? I’d like to implement a simple set of apis and use search with other backends.
It sort of annoys me to see people reimplementing storage engines over and over. If it’s embedded use rocks/leveldb/SQLite and it’s distributed use foundationDB. The end.
I'm immediately interested in that this seems to be aiming at being API compatible, which means it could in theory be used as an ES replacement for self-hosting services that hard-depend on ES?
Its like the SQLite of search. Single library that provides the basic set of features you would expect in a quality search experience: facets, ranked search, boolean operators, stemming etc etc.
https://xapian.org/