Torch-1 Fundamentals

24 Dec 2015

Abstract

In this blog I aim to analyze the working of Torch and build a simple Neural Network. Much of this work has been adapted from numerous blog posts by inspiring researchers.

For basic understanding of Lua please refer to this blog..

For building and training networks we use torch/nn package.

‘Module’ is a class which defines methods for training a neural net. They have two state variables ‘output’ and ‘gradInput’.

‘Output’ contains the ouput of the module which is computed with last call of inputs to that module. ‘gradInput’ contains the gradients with respect to the inputs of the module.

‘Containers’ are used to build Neural networks in modular manner using ‘container’ classes.

require 'nn'

-- Linear means we are doing linear transformation y = Ax + b
module = nn.Linear(5,2) -- 5 inputs and 2 output 

-- Use a container sequential which creates a feed-forward fully-connected net
net = nn.Sequential()
net:add(module)

-- Since x is 5x1, the weights A would be 2x5 and bias b would be of size 2x1
print(module.weight)
print(module.bias)

-- Giving random input x and feed forwarding the network to print the output y.
x = torch.rand(5)
print(x)
y = net:forward(x)
print(y)

‘Criterions’ are used to compute a gradient based on the loss function used and the target outputs. These are of Classification, Regression, Embedding types depending on the scope of problem we are working on. These have two important methods called 1) forward 2) backward used to compute the loss and update the weights respectively.
This link descibes use of Criterions with example codes. Example:

require 'nn'
-- This performs the backpropogation algo. 
function gradientUpgrade(model, x, y, criterion, learningRate)
    local prediction = model:forward(x) -- calculates the predicted o/p when input is x.
    local err = criterion:forward(prediction, y) -- calculates the error based on criterion 
    local gradOutputs = criterion:backward(prediction, y) -- calculates the gradients 
    model:zeroGradParameters()
    model:backward(x, gradOutputs)
    model:updateParameters(learningRate)
end

model = nn.Sequential()
model:add(nn.Linear(5,1))

x1 = torch.rand(5)
y1 = torch.Tensor({1})

--criterion = nn.MarginCriterion(1)
criterion = nn.AbsCriterion()


for i = 1, 1000 do
    gradientUpgrade(model, x1, y1, criterion, 0.01)
    print('loss after training for x1 = ' .. criterion:forward(model:forward(x1), y1))

end

-- with y1[1] we extract the first value in the tensor
print('prediction for x1 = ' .. model:forward(x1)[1] .. ' expected value ' .. y1[1])

print('loss after training for x1 = ' .. criterion:forward(model:forward(x1), y1))