Exam API

Exam API#

The exams module provides the functionality to create and manage exams. Here, an exam is used to measure how well the corpus or the model performs on a given task.

The module provides the following functionality:

  • `Examinator`s are responsible for launch exams.

  • `Exam`s are the tests that take in an accumulated history of model / corpus and returns a score.

  • `Manager`s are responsible for managing results across runs.

class bocoel.core.exams.Examinator(exams: Mapping[str, Exam])[source]#

The examinator is responsible for launching exams. Examinators take in an index and results of an optimizer run, and return a DataFrame of scores for the accumulated history performance of the optimizer.

__init__(exams: Mapping[str, Exam]) → None[source]#
examine(index: Index, results: OrderedDict[int, float]) → DataFrame[source]#

Perform the exams on the results. This method looks up results in the index and runs the exams on the results.

Parameters:

  • index – The index of the results.

  • results – The results.

Returns:

The scores of the exams.

Todo

Run the different exams in parallel. Currently the exams are run sequentially and can be slow.

classmethod presets() → Examinator[source]#
Returns:

The default examinator.

class bocoel.core.exams.Exam(*args, **kwargs)[source]#

Exams are designed to evaluate the performance of a particular index, using a particular set of results generated by the optimizer.

run(index: Index, results: OrderedDict[int, float]) → ndarray[Any, dtype[_ScalarType_co]][source]#

Run the exam on the given index and results.

Parameters:

  • index – The index to evaluate.

  • results – The results generated by the optimizer.

Returns:

The scores for each entry in the index. The length must be the same as the results.

__init__(*args, **kwargs)#
class bocoel.core.exams.Manager(root: str | Path | None = None, skip_rerun: bool = True)[source]#

The manager for running and saving evaluations.

__init__(root: str | Path | None = None, skip_rerun: bool = True) → None[source]#
Parameters:

  • root – The path to save the scores to.

  • skip_rerun – Whether to skip rerunning the optimizer if the scores already exist.

Raises:

ValueError – If the path is not a directory.

run(steps: int | None = None, *, optimizer: Optimizer, embedder: Embedder, corpus: Corpus, model: GenerativeModel | ClassifierModel, adaptor: Adaptor) → DataFrame[source]#

Runs the optimizer until the end. If the root path is set in the constructor, the scores are saved to the path.

Parameters:

  • optimizer – The optimizer to run.

  • embedder – The embedder to run the optimizer with.

  • corpus – The corpus to run the optimizer on.

  • model – The model to run the optimizer with.

  • adaptor – The adaptor to run the optimizer with.

  • steps – The number of steps to run the optimizer for.

Returns:

The final state of the optimizer.

Keys are the indices of the queries, and values are the corresponding scores.

save(*, scores: DataFrame, optimizer: Optimizer, corpus: Corpus, model: GenerativeModel | ClassifierModel, adaptor: Adaptor, embedder: Embedder, md5: str) → None[source]#

Saves the scores to the path. If the root path is not set in the constructor, the scores are not saved.

Parameters:

  • scores – The scores to save.

  • optimizer – The optimizer used to generate the scores.

  • corpus – The corpus used to generate the scores.

  • model – The model used to generate the scores.

  • adaptor – The adaptor used to generate the scores.

  • embedder – The embedder used to generate the scores.

  • md5 – The md5 hash of the identifier columns.

Raises:

ValueError – If the path is not set.

with_cols(df: DataFrame, columns: dict[str, Any]) → DataFrame[source]#

Adds identifier columns to the DataFrame.

Parameters:

  • df – The DataFrame to add the columns to.

  • mappings – The columns to add to the DataFrame.

Returns:

The md5 hash of the identifier columns and the DataFrame with the columns added.

static load(path: str | Path) → DataFrame[source]#

Loads the scores from the path.

Parameters:

path – The path to load the scores from.

Returns:

The loaded scores.

Raises:

  • ValueError – If the path does not exist or is not a directory.

  • ValueError – If no csv files are found in the path.

static md5(*, optimizer: Optimizer, embedder: Embedder, corpus: Corpus, model: GenerativeModel | ClassifierModel, adaptor: Adaptor) → str[source]#

Generates an md5 hash from the given data.

Parameters:

  • optimizer – The optimizer used to generate the scores.

  • corpus – The corpus used to generate the scores.

  • model – The model used to generate the scores.

  • adaptor – The adaptor used to generate the scores.

  • embedder – The embedder used to generate the scores.

  • time – The time the scores were generated.

Returns:

The md5 hash of the given data.

class bocoel.core.exams.Accumulation(acc_func: Callable[[ndarray[Any, dtype[_ScalarType_co]]], ndarray[Any, dtype[_ScalarType_co]]])[source]#

Accumulation is an exam designed to evaluate the min / max / avg of the history.

acc_func: Callable[[ndarray[Any, dtype[_ScalarType_co]]], ndarray[Any, dtype[_ScalarType_co]]]#

The function for accumulation.

__init__(acc_func: Callable[[ndarray[Any, dtype[_ScalarType_co]]], ndarray[Any, dtype[_ScalarType_co]]]) → None#