Fully Connected Layer
The neurons in the last layer of the feature extractor, including the convolution and pooling layers, are converted into a one-dimensional vector. The layers that come after this process are fully connected layers. Fully connected layer neurons perform the following calculation
where 𝑓 is the activation function and M is the number of previous layer neurons aligned in a vector, 𝑜𝑞 is the 𝑞th neuron’s value, 𝑓𝑐1 is the value of the first neuron in the first hidden layer of the fully connected network and 𝑤1,𝑞 is the weight value between the neurons 𝑜𝑞 and 𝑓𝑐1. Fully connected neural network structure composed of fully connected layers is generally used as a classifier in deep learning structures. This structure is usually followed by a SoftMax output layer for classification purposes.
DropOut technique is used for avoiding overfitting. In this technique, some of the hidden layer neurons are determined to be dropped out by using a dropout probability. Selected neuron activities are zeroed during the forward and backward step for a phase of the training process. As different neurons become active for each iteration, more consistent and reliable learning is provided, although the training time is extended (Krizhevsky et al, 2012).
Their activations can thus be computed as an affine transformation, with matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).