lasagne.regularization

Functions to apply regularization to the weights in a network.

We provide functions to calculate the L1 and L2 penalty. Penalty functions take a tensor as input and calculate the penalty contribution from that tensor:

l1 Computes the L1 norm of a tensor
l2 Computes the squared L2 norm of a tensor

A helper function can be used to apply a penalty function to a tensor or a list of tensors:

apply_penalty Computes the total cost for applying a specified penalty to a tensor or group of tensors.

Finally we provide two helper functions for applying a penalty function to the parameters in a layer or the parameters in a group of layers:

regularize_layer_params_weighted Computes a regularization cost by applying a penalty to the parameters of a layer or group of layers, weighted by a coefficient for each layer.
regularize_network_params Computes a regularization cost by applying a penalty to the parameters of all layers in a network.

Examples

>>> import lasagne
>>> import theano.tensor as T
>>> import theano
>>> from lasagne.nonlinearities import softmax
>>> from lasagne.layers import InputLayer, DenseLayer, get_output
>>> from lasagne.regularization import regularize_layer_params_weighted, l2, l1
>>> from lasagne.regularization import regularize_layer_params
>>> layer_in = InputLayer((100, 20))
>>> layer1 = DenseLayer(layer_in, num_units=3)
>>> layer2 = DenseLayer(layer1, num_units=5, nonlinearity=softmax)
>>> x = T.matrix('x')  # shp: num_batch x num_features
>>> y = T.ivector('y') # shp: num_batch
>>> l_out = get_output(layer2, x)
>>> loss = T.mean(T.nnet.categorical_crossentropy(l_out, y))
>>> layers = {layer1: 0.1, layer2: 0.5}
>>> l2_penalty = regularize_layer_params_weighted(layers, l2)
>>> l1_penalty = regularize_layer_params(layer2, l1) * 1e-4
>>> loss = loss + l2_penalty + l1_penalty

Helper functions

lasagne.regularization.apply_penalty(tensor_or_tensors, penalty, **kwargs)[source]

Computes the total cost for applying a specified penalty to a tensor or group of tensors.

Parameters:

tensor_or_tensors : Theano tensor or list of tensors

penalty : callable

**kwargs

keyword arguments passed to penalty.

Returns:

Theano scalar

a scalar expression for the total penalty cost

lasagne.regularization.regularize_layer_params(layer, penalty, tags={'regularizable': True}, **kwargs)[source]

Computes a regularization cost by applying a penalty to the parameters of a layer or group of layers.

Parameters:

layer : a Layer instances or list of layers.

penalty : callable

tags: dict

Tag specifications which filter the parameters of the layer or layers. By default, only parameters with the regularizable tag are included.

**kwargs

keyword arguments passed to penalty.

Returns:

Theano scalar

a scalar expression for the cost

lasagne.regularization.regularize_layer_params_weighted(layers, penalty, tags={'regularizable': True}, **kwargs)[source]

Computes a regularization cost by applying a penalty to the parameters of a layer or group of layers, weighted by a coefficient for each layer.

Parameters:

layers : dict

A mapping from Layer instances to coefficients.

penalty : callable

tags: dict

Tag specifications which filter the parameters of the layer or layers. By default, only parameters with the regularizable tag are included.

**kwargs

keyword arguments passed to penalty.

Returns:

Theano scalar

a scalar expression for the cost

lasagne.regularization.regularize_network_params(layer, penalty, tags={'regularizable': True}, **kwargs)[source]

Computes a regularization cost by applying a penalty to the parameters of all layers in a network.

Parameters:

layer : a Layer instance.

Parameters of this layer and all layers below it will be penalized.

penalty : callable

tags: dict

Tag specifications which filter the parameters of the layer or layers. By default, only parameters with the regularizable tag are included.

**kwargs

keyword arguments passed to penalty.

Returns:

Theano scalar

a scalar expression for the cost

Penalty functions

lasagne.regularization.l1(x)[source]

Computes the L1 norm of a tensor

Parameters:

x : Theano tensor

Returns:

Theano scalar

l1 norm (sum of absolute values of elements)

lasagne.regularization.l2(x)[source]

Computes the squared L2 norm of a tensor

Parameters:

x : Theano tensor

Returns:

Theano scalar

squared l2 norm (sum of squared values of elements)