Maggy User API

maggy.experiment module

Experiment module used for running asynchronous optimization tasks.

The programming model is that you wrap the code containing the model training inside a wrapper function. Inside that wrapper function provide all imports and parts that make up your experiment, see examples below. Whenever a function to run an experiment is invoked it is also registered in the Experiments service along with the provided information.

maggy.experiment.launch(map_fun, searchspace, optimizer, direction, num_trials, name, hb_interval=1, es_policy='median', es_interval=300, es_min=10)

Launches a maggy experiment for hyperparameter optimization.

Given a search space, objective and a model training procedure map_fun (black-box function), an experiment is the whole process of finding the best hyperparameter combination in the search space, optimizing the black-box function. Currently maggy supports random search and a median stopping rule.

Parameters:
  • map_fun (function) – User defined experiment containing the model training.
  • searchspace (Searchspace) – A maggy Searchspace object from which samples are drawn.
  • optimizer (str, AbstractOptimizer) – The optimizer is the part generating new trials.
  • direction (str) – If set to ‘max’ the highest value returned will correspond to the best solution, if set to ‘min’ the opposite is true.
  • num_trials (int) – the number of trials to evaluate given the search space, each containing a different hyperparameter combination
  • name (str) – A user defined experiment identifier.
  • hb_interval (int, optional) – The heartbeat interval in secondss from trial executor to experiment driver, defaults to 1
  • es_policy (str, optional) – The earlystopping policy, defaults to ‘median’
  • es_interval (int, optional) – Frequency interval in seconds to check currently running trials for early stopping, defaults to 300
  • es_min (int, optional) – Minimum number of trials finalized before checking for early stopping, defaults to 10
Raises:

RuntimeError – An experiment is currently running.
Returns:

A dictionary indicating the best trial and best hyperparameter combination with it’s performance metric
Return type:

dict

maggy.searchspace module

class maggy.Searchspace(**kwargs)

Create an instance of Searchspace from keyword arguments.

The keyword arguments specify name-values pairs for the hyperparameters, where values are tuples of the form (type, list). Type is a string with one of the following values:

  • DOUBLE
  • INTEGER
  • DISCRETE
  • CATEGORICAL

And the list in the tuple specifies either two values only, the start and end point of of the feasible interval for DOUBLE and INTEGER, or the discrete possible values for the types DISCRETE and CATEGORICAL.

The parameter names are added as attributes of Searchspace object, so they can be accessed directly with the dot notation searchspace._name_.

add(name, value)

Adds {name, value} pair to hyperparameters.

Parameters:
  • name (str) – Name of the hyperparameter
  • value (tuple) – A tuple of the parameter type and its feasible region
Raises:
  • ValueError – Hyperparameter name is reserved
  • ValueError – Hyperparameter feasible region in wrong format
get(name, default=None)

Returns the value of name if it exists, else default.

names()

Returns the dictionary with the names and types of all hyperparameters.

Returns:Dictionary of hyperparameter names, with types as value
Return type:dict
to_dict()

Return the hyperparameters as a Python dictionary.

Returns:A dictionary with hyperparameter names as keys. The values are the hyperparameter values.
Return type:dict

maggy.callbacks module

class maggy.callbacks.KerasBatchEnd(reporter, metric='loss')

A Keras callback reporting a specified metric at the end of the batch to the maggy experiment driver.

loss is always available as a metric, and optionally acc (if accuracy monitoring is enabled, that is, accuracy is added to keras model metrics). Validation metrics are not available for the BatchEnd callback. Validation after every batch would be too expensive. Default is training loss (loss).

Example usage:

>>> from maggy.callbacks import KerasBatchEnd
>>> callbacks = [KerasBatchEnd(reporter, metric='acc')]
class maggy.callbacks.KerasEpochEnd(reporter, metric='val_loss')

A Keras callback reporting a specified metric at the end of an epoch to the maggy experiment driver.

val_loss is always available as a metric, and optionally val_acc (if accuracy monitoring is enabled, that is, accuracy is added to keras model metrics). Training metrics are available under the names loss and acc. Default is validation loss (val_loss).

Example usage:

>>> from maggy.callbacks import KerasBatchEnd
>>> callbacks = [KerasBatchEnd(reporter, metric='val_acc')]