How to extract kernel values in CNN using NETRON and generate feature maps
Netron is an open-source multi-platform visualizer and editor for artificial intelligence models.
There are a couple of assumptions we make before we start.
- The reader already knows about building CNN model.
- We have made a simple model. We are not concerned here with the accuracy of the model. Our primary aim is to learn to extract kernel values from NETRON and generate feature maps for better understanding of working of CNN.
Note:-
Just for the sake of simplicity we have not downloaded the bias values and hence not included it in our calculations.
from keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense#Part 1 — Creating & Compiling the Model
#Initialising the CNN
classifier = Sequential()# Adding first convolutional layer# Step 1 - Convolution
classifier.add(Conv2D(32, (5, 5), input_shape = (194,258, 3), activation = 'relu'))# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size = (2, 2)))# Adding a second convolutional layer# Step 3 - Convolution
classifier.add(Conv2D(16, (3, 3), input_shape = (194,258, 3), activation = 'relu'))# Step 4 - Pooling
classifier.add(MaxPooling2D(pool_size = (2, 2)))# Flattening the data so as to feed to Dense layer# Step 5 - Flattening
classifier.add(Flatten())# Adding Dense layers# Step 6 - Full connection
classifier.add(Dense(units = 128, activation = 'relu'))classifier.add(Dense(units = 6, activation = 'softmax'))# Compiling the CNNclassifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Part 2 — Fitting the CNN to the images
from keras.preprocessing.image import ImageDataGeneratortrain_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)training_set = train_datagen.flow_from_directory(r'link-from-desktop',target_size = (256,256), batch_size = 32, class_mode = 'categorical')test_set = test_datagen.flow_from_directory(r'link-from-desktop',
target_size = (256,256),
batch_size = 32,
class_mode = 'categorical')model = classifier.fit_generator(training_set,
steps_per_epoch = 500,
epochs=5,
validation_data = test_set,
validation_steps = 100)
#Saving the Model on our Desktop
classifier.save("Model_MultiClass_2layers.h5")
print("Saved model to disk")
This is how the above saved file in the folder will look like. This is an “h5” file. NETRON needs “h5” file to convert the model into visual map.
The two files on the left hand side are the kernel files downloaded in .npy format. How this can be done is the subject of this blog and we will deal with it in detail in the following section.
For now all we need to know is that in total we will be downloading three files. Important thing to note here is that all these three files should be in the same directory of our python file..
What is NETRON ?
Netron is an open-source multi-platform visualizer and editor for artificial intelligence models. It supports many extensions for deep learning, machine learning and neural network models. Netron is using Electron/ NodeJS and it has a binary application release for Windows, Linux and macOS.
Source : Link
Either we can download NETRON and install it on our desktop or we can directly access NETRON through web page.
In our case, we will be directly accessing NETRON through web page.
For that we will be using the link :- netron.app
On clicking the above link the following page will open.
As you can see in the above picture, we need to click on the “open Model…” tab to load our model.
Once the model is loaded, it looks something like this as below.
As we can see above, this is the visualisation of the model we have created. This is the graphic of the model which comes after loading the .h5 file.
On the left side we have used naming conventions for the two kernels so that it becomes easy for us to communicate and refer to the kernels.
As mentioned on the right side of the flow chart, if we want to access the kernel values of “Conv2D kernel 1” we need to click on that block.
Let us also see the Block diagram of the Model as below:-
Now, let us try to understand the dimensions of input image, filters 1 & 2.
Dimensions are as follows:-
Input image :- 256 X 256 X 3.
Here, 3 in the end represents the three channels RGB.
Conv2D Kernel 1:-
Note here that in the we model have taken 32 filters of 5X5.
But the input image has 3 dimensions(i.e. RGB) & hence each filter of 5X5 has 3 dimensions. Hence each filter dimensions is 5X5X3. This means that there are 3 learnable filters 5X5 in each of the 32 kernels.
Hence the total filters in Conv2D Kernel1 is [( 5X5)X3]X 32.
The kernel values which we will be downloading from NETRON will be a 4 dimensional Matrix.
Imp Note:-
Important thing to note here is that the output image of 5X5X3 kernel will be a single image and not three images and hence total feature maps out of this Conv2D Kernel 1 will 32 feature maps.
Conv2D Kernel 2:-
Here, as in code, we have taken total 16 kernels(3X3), each kernel with 32 filters. This is so, because the feature maps generated in the previous layer is 32. That means in layer “Conv2D Kernel 2" there are total 16 kernels each with 32 dimensions or filters.
So, the dimension of the kernel of this layer will be [(3X3) X 32]X 16
Let us continue on our journey to extract kernel values.
So, when we click on the Cnv2D kernel 1 block, on the right hand side of the flow chart the NODE PROPERTIES open up.
We are only concerned about kernel values and bias values.
To download the kernel values, we need to click on the “+” sign in the line of kernel.
After, we click on the “+” sign of kernel, the block below the kernel opens which shows the data type of kernel, values of kernel and there is one small save sign on the top right corner of kernel block.
This save sign can be used to download the kernel values.
Kernel values will be download in the form of “.npy” format.
As there are two kernels, we will have two files in our download section of PC. We need to relocate those files to the directory of our python file.
Those kernel files with “.npy” extension can be loaded as below.
This is how Conv2D kernel 1 values look like after loading it in out jupyter notebook.
Understanding Extraction of Kernel Values
So this is the most awaited part. Let’s start the adventure of Extraction of Kernel values.
Let us take example of the shape of “kernel_layer1”.
The shape of “kernel_layer1” is [( 5X5)X3]X 32.
The below diagram simplifies the kernel dimensions and helps us understand how to extract the values.
So, let us try to extract one of the three filters of first kernel out of 32 kernels.
kernel_layer1[:,:,0,0] :- This syntax will extract the first filter of the first kernel out of 32 kernels. This will generate a 5X5 matrix.
kernel_layer1[:,:,2,19] :- This will extract the third filter of the 20th kernel(as the counting starts from 0) out of 32 kernels. This will generate a 5X5 matrix.
Now, let us take example of the shape of “kernel_layer2”.
kernel_layer2[:,:,15,9] :- This will extract the 16th filter of the 10th kernel out of 16 kernels in layer 2. This will generate a 3X3 matrix.
Input Image
Below is the input image we will be using to generate feature maps.
The input image is show with its three channels.
test_image = image.load_img(r'image_link_in_desktop' target_size = (256,256))test_image = image.img_to_array(test_image)font = {'family': 'serif',
'color': 'darkred',
'weight': 'normal',
'size': 16,
}fig=plt.figure(fig size=(22,12))ax=fig.add_subplot(1,4,1)
red_image=test_image[:,:,0]
plt.imshow(red_image)
ax.set_title('Red Channel', fontdict=font)ax=fig.add_subplot(1,4,2)
green_image=test_image[:,:,1]
plt.imshow(green_image)
ax.set_title('Green Channel', fontdict=font)ax=fig.add_subplot(1,4,3)
blue_image=test_image[:,:,2]
plt.imshow(blue_image)
ax.set_title('Blue Channel', fontdict=font)ax=fig.add_subplot(1,4,4)
plt.imshow(test_image[:,:,:].astype('uint8'))
ax.set_title('All 3 Channels', fontdict=font)
Layer 1 — Generating Feature Maps for Layer 1
To generate feature map we will have to manually convolve the original input image with the extracted kernel values.
We have already separated the input image into its three channels viz: RGB
Here, we will use the “ndimage.convolve” feature of “scipy” library to convolve the kernel values with the three channels of the image separately.
Below is the code for that.
from scipy import ndimagefeature_map_layer1=[] #Creating an empty list to append feature maps.
for k in range(32):red_filter=kernel_layer1[:,:,0,k]
green_filter=kernel_layer1[:,:,1,k]
blue_filter=kernel_layer1[:,:,2,k]
red_conv=ndimage.convolve(red_image, red_filter, mode='constant', cval=0.0)
green_conv=ndimage.convolve(green_image, green_filter, mode='constant', cval=0.0)
blue_conv=ndimage.convolve(blue_image, blue_filter, mode='constant', cval=0.0)
add_images=red_conv+green_conv+blue_conv
feature_map_layer1.append(add_images)
We have created a list of feature maps. Now, we will use for loop to iterate over this list to generate our 32 feature maps.
fig=plt.figure(figsize=(20,30))for i in range(len(feature_map_layer1)):
ax=fig.add_subplot(8,4,i+1)
plt.imshow(feature_map_layer1[i])
Max pooling of the above generated feature maps.
We will do here 2X2 max pooling of feature maps.
For this, we will us the function “measure.block_reduce” from “skimage” library.
After max pooling the image we will create a list of max pooled images which will be fed to kernels in layer 2.
from skimage import measurefig=plt.figure(figsize=(20,30))for i in range(len(feature_map_layer1)):
max_pool=measure.block_reduce(feature_map_layer1[i], (2,2), np.max)
max_pool1.append(max_pool)
Layer 2 — Generating Feature Maps for Layer 2
Here, we will use the output of the max pooling. We have already created a list of max pooled images. Now, we will iterate of the max pooled list to generate feature maps for layer 2.
feature_map_layer2=[]for k in range(kernel_layer2.shape[3]):
featuremap=0
for i in range(kernel_layer2.shape[2]): channel=ndimage.convolve(max_pool1[i], kernel_layer2[:,:,i,k], mode='constant', cval=0.0) featuremap= featuremap + channel
feature_map_layer2.append(featuremap)
Let’s print the feature maps of layer 2.
fig=plt.figure(figsize=(15,15))for i in range(len(feature_map_layer2)):
ax=fig.add_subplot(4,4,i+1)
plt.imshow(feature_map_layer2[i])
This shows that every layer in CNN is transparent. We can visualize feature maps of any layer and have a good understanding of the working of our model.
