downhill.base.Optimizer

class downhill.base.Optimizer(loss, params, inputs, updates=(), monitors=(), monitor_gradients=False)

An optimizer computes gradient updates to iteratively optimize a loss.

Parameters:

loss : Theano expression

Loss function to minimize. This must be a scalar-valued expression.

params : list of Theano variables

Symbolic variables to adjust to minimize the loss.

inputs : list of Theano variables

Symbolic variables required to compute the loss.

updates : list of update pairs, optional

A list of pairs providing updates for the internal of the loss computation. Normally this is empty, but it can be provided if the loss, for example, requires an update to an internal random number generator.

monitors : dict or sequence of (str, Theano expression) tuples, optional

Additional values to monitor during optimization. These must be provided as either a sequence of (name, expression) tuples, or as a dictionary mapping string names to Theano expressions.

monitor_gradients : bool, optional

If True, add monitors to log the norms of the parameter gradients during optimization. Defaults to False.

__init__(loss, params, inputs, updates=(), monitors=(), monitor_gradients=False)

Methods

__init__(loss, params, inputs[, updates, ...])
evaluate(dataset)	Evaluate the current model parameters on a dataset.
iteropt(train[, valid, patience, ...])	Optimize a loss iteratively using a training and validation dataset.
minimize(*args, **kwargs)	Optimize our loss exhaustively.
set_params(targets)	Set the values of the parameters to the given target values.

evaluate(dataset)

Evaluate the current model parameters on a dataset.

Parameters:

dataset : Dataset

A set of data to use for evaluating the model.

Returns:

monitors : OrderedDict

A dictionary mapping monitor names to values. Monitors are quantities of interest during optimization—for example, loss function, accuracy, or whatever the optimization task requires.

iteropt(train, valid=None, patience=5, validate_every=10, min_improvement=0, max_gradient_norm=10000000000.0, max_gradient_clip=10000000000.0, learning_rate=0.0001, momentum=0, nesterov=False, **kwargs)

Optimize a loss iteratively using a training and validation dataset.

This method yields a series of monitor values to the caller. After every optimization epoch, a pair of monitor dictionaries is generated: one evaluated on the training dataset during the epoch, and another evaluated on the validation dataset at the most recent validation epoch.

The validation monitors might not be updated during every optimization iteration; in this case, the most recent validation monitors will be yielded along with the training monitors.

Parameters:

train : sequence or Dataset

A set of training data for computing updates to model parameters.

valid : sequence or Dataset, optional

A set of validation data for computing monitor values and determining when the loss has stopped improving. Defaults to the training data.

patience : int, optional

Number of validation “failures” that we are willing to tolerate before stopping the optimization process. A validation failure happens whenever the loss on the validation dataset decreases by less than min_improvement (relative) over the previous best validation loss. Defaults to 5.

validate_every : int, optional

Evaluate the loss on the validation dataset after making this many passes over the training data. Defaults to 10.

min_improvement : float, optional

Insist that the validation loss must improve by this relative amount before considering that the optimization has made progress. The optimization process halts when patience validations have failed to make this relative improvement. Defaults to 0; set to a larger value (e.g., 0.01 for 1% improvement) to halt the optimization process sooner.

max_gradient_norm : float, optional

Rescale each parameter’s gradient so that it has at most this L2 norm. Defaults to 1e10, i.e., very little rescaling.

max_gradient_clip : float, optional

Perform elementwise clipping on gradient values (this happens after rescaling). Defaults to 1e10, i.e., very little clipping.

learning_rate : float, optional

Many SGD-based optimization algorithms require a learning rate hyperparameter that scales the gradient step. Defaults to 1e-4.

momentum : float, optional

Apply momentum to the parameter updates for this optimizer, with the given strength. Typically this value ranges from 0 (no momentum) to \(1 - \epsilon\) (large momentum). Defaults to 0.

nesterov : bool, optional

If True, and momentum is nonzero, apply Nesterov-style momentum to parameter updates for this optimizer. If False and momentum is nonzero, “regular” momentum is applied. See NAG for a description of Nesterov momentum.

Returns:

train_monitors : dict

A dictionary mapping monitor names to values, evaluated on the training dataset.

valid_monitors : dict

A dictionary containing monitor values evaluated on the validation dataset.

minimize(*args, **kwargs)

Optimize our loss exhaustively.

This method is a thin wrapper over the iteropt() method. It simply exhausts the iterative optimization process and returns the final monitor values.

Returns:

train_monitors : dict

A dictionary mapping monitor names to values, evaluated on the training dataset.

valid_monitors : dict

A dictionary containing monitor values evaluated on the validation dataset.

set_params(targets)

Set the values of the parameters to the given target values.

Parameters:

targets : sequence of ndarray

Arrays for setting the parameters of our model.