aarondb - v4.1.0

aarondb/index/bm25

A local, in-memory BM25 index for one string attribute.

Contract

Documents are keyed by entity. add replaces an existing entity document atomically within the immutable index; callers do not need to retract the prior text first. remove is idempotent and removes the indexed entity regardless of the supplied historical text, which prevents stale input from corrupting document-frequency statistics.

Tokenisation lowercases ASCII letters and digits, splits every other grapheme as a boundary, and does not apply stemming or stop-word removal. Ranking uses BM25’s standard k1/b formula. Search returns only positive scores, ordered by descending score and then ascending entity ID to make ties deterministic. This module is an index primitive: database transaction and query-engine integration are not claimed by this contract.

Types

BM25Index 

pub type BM25Index {
  BM25Index(
    term_freq: dict.Dict(String, dict.Dict(fact.EntityId, Int)),
    doc_freq: dict.Dict(String, Int),
    doc_len: dict.Dict(fact.EntityId, Int),
    avg_doc_len: Float,
    doc_count: Int,
    attribute: String,
  )
}

Constructors

SearchResult

A deterministic BM25 search result.

pub type SearchResult {
  SearchResult(entity: fact.EntityId, score: Float)
}

Constructors

Values

add 

pub fn add(
  index: BM25Index,
  entity: fact.EntityId,
  text: String,
) -> BM25Index

Adds or replaces one entity document.

build 

pub fn build(
  datoms: List(fact.Datom),
  attribute: String,
) -> BM25Index

Builds an index from string datoms for attribute.

If input contains multiple datoms for an entity, the last datom in input wins, matching add replacement semantics. Callers that retain historical datoms should filter to their desired active snapshot before building.

empty 

pub fn empty(attribute: String) -> BM25Index

remove 

pub fn remove(
  index: BM25Index,
  entity: fact.EntityId,
  text: String,
) -> BM25Index

Removes an entity document. The supplied text is intentionally ignored: indexed state is the source of truth, so removal is safe and idempotent.

remove_entity 

pub fn remove_entity(
  index: BM25Index,
  entity: fact.EntityId,
) -> BM25Index

Removes an entity document using the index’s own term and length state.

score 

pub fn score(
  index: BM25Index,
  entity: fact.EntityId,
  query: String,
  k1: Float,
  b: Float,
) -> Float

Scores an entity for a query using standard BM25 parameters.

k1 must be non-negative and b must be in [0.0, 1.0]; invalid parameters return 0.0 to preserve this total compatibility API.

pub fn search(
  index: BM25Index,
  query: String,
  k1: Float,
  b: Float,
  limit: Int,
) -> List(SearchResult)

Searches every indexed entity and returns deterministic positive-score hits. limit must be positive; invalid BM25 parameters or limit return [].

✨ Search Document