Expand description
HNSW as a compatibility view, not as an index (10 section 7).
Clients pass M, EF_CONSTRUCTION and EF_RUNTIME to VADD and
FT.CREATE and expect them to do something, because against Redis and
valkey they do. There is no graph here to point them at.
There are three things you can do about that and two of them are bad. You
can reject the parameters, which breaks every client that has ever created a
vector index. You can accept them and do nothing, which is worse, because
someone raises EF_RUNTIME to fix their recall and nothing happens and they
have no way to find out why. Or you can work out what each one was for and
do that thing, which is what this is.
§The mapping
EF_CONSTRUCTION is how hard the index works while building. Here that is
the size a posting is split and merged around, which is what decides how
many vectors a probe reads and how finely the space is cut.
EF_RUNTIME is the search beam, the pool of candidates a query keeps. Here
that is Tuning::probe and Tuning::rerank, which are the two things
that decide how much a query looks at.
INITIAL_CAP is how many vectors are coming, so the postings for them can
be allocated up front instead of grown into.
M is the out degree of a graph. There is no graph, so there is nothing for
it to mean, and it is recorded and echoed back by FT.INFO and VINFO and
changes nothing. That is not a fudge. Quietly mapping it onto some unrelated
knob so the number looks used would be worse than admitting it does not
apply.
§What the client is actually promised
Not that a given EF_RUNTIME reads the same number of vectors it would
under HNSW. It does not, and it could not: a graph walk at EF_RUNTIME 10
touches tens of vectors and a probe of eight postings here touches two
thousand, because measuring one against a code is a popcount and following a
graph edge is a cache miss. The absolute numbers are not comparable and
pretending they are would be the lie.
What is promised is the thing a client relies on, which is that the knob
responds: raise EF_RUNTIME and the search does proportionally more work
and finds more, lower it and it does less and finds less. That is what
somebody turning it up at three in the morning needs to be true, and
an_ef_runtime_client_gets_what_it_turned_the_knob_for measures it rather
than asserting it.
§When somebody really does want a graph
Compat::Strict exists for that, and it does not build one. It reports
Plan::Graph, and the layer above turns that into a refusal, so a client
that asked for HNSW and meant HNSW is told it is not here rather than being
served something else under the name. The default is Compat::Permissive,
where the parameters are honoured as above and the partition index serves.
use yo_vector::hnsw::{Compat, Plan, Requested};
// What FT.CREATE sends when nobody set anything.
let asked = Requested::default();
let Plan::Partitions { tuning, .. } = asked.plan(Compat::Permissive) else {
panic!("permissive serves it")
};
// Redis's defaults come out as ours, so a client that set nothing gets the
// index tuned the way it would have been anyway.
assert_eq!(tuning.posting, yo_vector::Tuning::default().posting);
// And a client that meant it is not quietly served something else.
assert!(matches!(asked.plan(Compat::Strict), Plan::Graph));Structs§
- Requested
- What a client asked for when it said HNSW.
Enums§
- Compat
- Whether an HNSW request is served by the partition index or refused.
- Plan
- What to do with an HNSW request.
Constants§
- EF_
CONSTRUCTION - Redis’s default
EF_CONSTRUCTION, which maps to our default posting size. - EF_
RUNTIME - Redis’s default
EF_RUNTIME, which maps to our default probe and rerank. - M
- Redis’s default
M, which is echoed and otherwise ignored.