DenseNet Architecture with Explanation

DenseNet is a Densely Connected Neural Network derived from ResNet with the exception that ResNet perform summation of the inputs received from the previous layers by the current layer whereas Dense Net concatenates the features received from previous layers. The DenseNet basically comprises of three main parts as shown in Fig 1.

1. Convolutional Layers
2. Hidden Layers comprising of Dense Blocks and Transition Layers
3. Fully Connected Output Layer.

The DenseNet 121 architecture consists of Convolutional 7x7 layer with stride 2 followed by the Pooling layer of 3x3 filter size. The downsampled features are then given to the Dense Block 1. The Densenet 121 consists of 4 Dense Block each followed by 3 Transition layers respectively. The output layer is Fully Connected layer with softmax activation function.

Fig. 1 DenseNet 121 architecture

DenseNet functionality is similar to the other variants of Convolutional networks with the exception that DenseNet along with downsampling also performs concatenation of features which are well explained below.

Dense Block:

The Dense Block performs the concatenation of features which means aligning the features together. Like, if first layer is extracting 10 features and next layer is extracting 5 features, then the next layer will receive 15 features. This is shown in Fig.2. In DenseNet 121 the first Dense Block consists of 6 convolutional layers of size 1x1 followed by 3x3.

Fig. 2

Each layer comprises of the following sublayers:

Dense Block Convolutional layer

For understanding the Batch Normalization and Dropout layer usage, refer to my previous blogs on the same. As shown in Fig. 2 each layer will extract certain features that concatenates to the features of the previous layers.

Kindly note that the Dense Block doesn’t comprise of any Pooling layers as for the concatenation the size of the input features should be same as the size of the output features. Consider the Table 1 that illustrate the input and output of each Dense Block layer.

Table 1. Dense Block features concatenation

In Dense Block each layer is connected to every other layer in the block so if there are n layers, there would be n(n+1)/2 link connections. If there are 3 layers, then there would be 6 connections. So, every layer will receive the features of every other layer as can be understood by the Table 1. and the last layer will output the concatenated features to the next block in the architecture. The Conv 1x1 layer acts as a Bottleneck layer to reduce the number of features to improve computational efficiency. In General, each Conv 1x1 layer reduces the inputs to 4 times the number of feature maps produced by the subsequent Conv 3x3 layer.

Growth Rate:

To limit the network from growing too wide a Growth rate is specified for the network which is usually a smaller integer value. It is denoted by k. Each layer adds the k feature maps to the concatenated feature maps which are recognized as “Global state” of network as each layer has access to every other layer feature maps. If growth rate is 3, it denotes that, a[l] composite function of each layer as explained below will produces 3 features map and each layer will produce

input features equation

input feature maps, where l is nth layer and k0 is number of channels in input of the given dense block. For k =3 and k0 =3 the number of feature maps input to the 6th layer of the above Dense Block 6 will be 16. Using the above input features equation, the features at each layer can be calculated as shown in Table 2.

Table 2. showing input features to each dense layer using input features equation considering growth rate factor k=3 and number of channels k0=1

The growth rate thus, regulates the amount of information flow through the network.

Transition Layer:

This layer performs the down sampling of the concatenated features obtained from the Dense Block. Thus, as shown in Fig. 1 the DenseNet 121 architecture consists of three Transition layers consisting of Convolutional layer and Avg pooling layer for the down sampling purposes. The Convolutional Layers of Transition Layer consists of following layers:

Compression Factor:

The compression factor is a factor by which the number of features are reduced in the Transition layer. If the Dense layer generates m feature maps that the Transition layer will generate (cf x m) feature maps where cf is compression factor and lies between 0 and 1. If cf is 1 then there would be no change in feature maps. If cf is 0.5 then considering above discussed feature maps in Table 2. the input features to Transition layer will be 19 and output features will be rounded to 10 (19 x 0.5).

Advantages of DenseNet

1. Improves the information flow as compared to ResNet and other architectures as information from each layer from to every layer whereas in ResNet the information flows through skip connection.

Source: PaperwithCode

where a[l] is a composite function of three type of operations: BN(batch normalization), Relu and Conv

2. It alleviates the problem of vanishing gradient as during backward pass error of each layer will be propagated to the previous layers also. So, if at any layer gradient is vanishing the error from the next layer will get added to it thus eliminating the vanishing gradient.

Fig. 3 Forward pass and backward pass link connections

3. It is parametric efficient as compared to the other architectures due to the presence of Transition layers and bottleneck structure.

4. It is memory efficient as compared to ResNet.

Thus, the DenseNet Neural Networks can be used for classification purposes where there is more no. of features to be extract out. For DenseNet Implementation in Tensorflow stay tuned to my blog.

Thanks for Reading,

References

G. Huang, Z. Liu and L. van der Maaten, “Densely Connected Convolutional Networks,” 2018.