Batchnorm layer

class layers.batchnorm_layer.BatchNorm_layer(scales=None, bias=None, input_shape=None, **kwargs)

Bases: NumPyNet.layers.base.BaseLayer

BatchNormalization Layer

It performs a Normalization over the Batch axis of the Input. Both scales and bias are trainable weights.

Equation:

output = scales * input_normalized + bias

Parameters

• scales (array-like (default=None)) – Starting scale to be multiplied to the normalized input, array-like of shape (w, h, c). If None, the array will be initialized with ones.

• bias (array-like (default=None)) – Bias to be added to the multiplication of scale and normalized input of shape (w, h, c). If None, the array will be initialized with zeros.

• input_shape (tuple (default=None)) – Shape of the input in the format (batch, w, h, c), None is used when the layer is part of a Network model.

Example

>>> import os
>>>
>>> import pylab as plt
>>> from PIL import Image
>>>
>>> img_2_float = lambda im : ((im - im.min()) * (1./(im.max() - im.min()) * 1.)).astype(float)
>>> float_2_img = lambda im : ((im - im.min()) * (1./(im.max() - im.min()) * 255.)).astype(np.uint8)
>>>
>>> # I need to load at least to images, or made a copy of it
>>> filename = os.path.join(os.path.dirname('__file__'), '..', '..', 'data', 'dog.jpg')
>>> inpt = np.asarray(Image.open(filename), dtype=float)
>>> inpt.setflags(write=1)
>>> w, h, c = inpt.shape
>>>
>>> batch_size = 5
>>>
>>> np.random.seed(123) # set seed to have fixed bias and scales
>>>
>>> # create a pseudo-input with batch_size images with a random offset from the original image
>>> rng  = np.random.uniform(low=0., high=100., size=(batch_size, w, h, c))
>>> inpt = np.concatenate([np.expand_dims(inpt, axis=0) + r for r in rng], axis=0) # create a set of image
>>>
>>> # img_to_float of input, to work with numbers btween 0. and 1.
>>> inpt = np.asarray([img_2_float(x) for x in inpt ])
>>>
>>> b, w, h, c = inpt.shape # needed for initializations of bias and scales
>>>
>>> bias   = np.random.uniform(0., 1., size=(w, h, c)) # random biases
>>> scales = np.random.uniform(0., 1., size=(w, h, c)) # random scales
>>>
>>> bias = np.zeros(shape=(w, h, c), dtype=float)
>>> scales = np.ones(shape=(w, h, c), dtype=float)
>>>
>>> # Model Initialization
>>> layer = BatchNorm_layer(input_shape=inpt.shape, scales=scales, bias=bias)
>>>
>>> # FORWARD
>>>
>>> layer.forward(inpt)
>>> forward_out = layer.output
>>> print(layer)
>>>
>>> # BACKWARD
>>>
>>> layer.delta = np.random.uniform(low=0., high=100., size=layer.out_shape)
>>> delta = np.ones(shape=inpt.shape, dtype=float) # delta same shape as the Input
>>> layer.backward(delta)
>>>
>>> # Visualizations
>>>
>>> fig, (ax1, ax2) = plt.subplots(nrows=2, ncols=2, figsize=(10, 5))
>>> fig.subplots_adjust(left=0.1, right=0.95, top=0.95, bottom=0.15)
>>>
>>> fig.suptitle('BatchNormalization Layer')
>>>
>>> ax1[0].imshow(float_2_img(inpt[0]))
>>> ax1[0].set_title('Original image')
>>> ax1[0].axis('off')
>>>
>>> ax1[1].imshow(float_2_img(layer.mean))
>>> ax1[1].set_title("Mean Image")
>>> ax1[1].axis("off")
>>>
>>> ax2[0].imshow(float_2_img(forward_out[0]))
>>> ax2[0].set_title('Forward')
>>> ax2[0].axis('off')
>>>
>>> ax2[1].imshow(float_2_img(delta[0]))
>>> ax2[1].set_title('Backward')
>>> ax2[1].axis('off')
>>>
>>> fig.tight_layout()
>>> plt.show()

References

• https://arxiv.org/abs/1502.03167

backward(delta=None)

BackPropagation function of the BatchNormalization layer. Every formula is a derivative computed by chain rules: dbeta = derivative of output w.r.t. bias, dgamma = derivative of output w.r.t. scales etc…

Parameters

delta (array-like) – delta array of shape (batch, w, h, c). Global delta to be backpropagated.

Return type

self

epsil = 1e-08

forward(inpt)

Forward function of the BatchNormalization layer. It computes the output of the layer, the formula is:

output = scale * input_norm + bias

Where input_norm is:

input_norm = (input - mean) / sqrt(var + epsil)

where mean and var are the mean and the variance of the input batch of images computed over the first axis (batch)

Parameters

inpt (array-like) – Input batch of images in format (batch, in_w, in_h, in _c)

Return type

self

load_weights(chunck_weights, pos=0)

Load weights from full array of model weights

Parameters

• chunck_weights (array-like) – Model weights and bias

• pos (int (default=0)) – Current position of the array

Returns

pos – Updated stream position.

Return type

int

property out_shape

Get the output shape

Returns

out_shape – Tuple as (batch, out_w, out_h, out_c)

Return type

tuple

save_weights()

Return the biases and weights in a single ravel fmt to save in binary file

update()

Update function for the batch-normalization layer. Optimizer must be assigned externally as an optimizer object.

Return type

self