Compare tensorflow and dlib models

[UlrichVonRekkenin]

Hi there,

I'm trying to reproduce few tf example and I faced with the problem of mapping tf model to dlib. I check using dot notation. For instance first one:

inputs = Input(shape=(26, 34, 1))

net = Conv2D(32, kernel_size=3, strides=1, padding='same', activation='relu')(inputs)
net = MaxPooling2D(pool_size=2)(net)

net = Conv2D(64, kernel_size=3, strides=1, padding='same', activation='relu')(net)
net = MaxPooling2D(pool_size=2)(net)

net = Conv2D(128, kernel_size=3, strides=1, padding='same', activation='relu')(net)
net = MaxPooling2D(pool_size=2)(net)

net = Flatten()(net)

net = Dense(512)(net)
net = Activation('relu')(net)
net = Dense(1)(net)
outputs = Activation('sigmoid')(net)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])

and second one:

namespace dnn {
    using namespace dlib;

    template<size_t NUMS, typename SUBNET> using layer = max_pool<3, 3, 1, 1, relu<con<NUMS, 1, 1, 1, 1, SUBNET>>>;
    using netType = loss_binary_log<sig<fc<1, relu<fc<512, layer<128, layer<64, layer<32, input<matrix<unsigned char>>>>>>>>>>; 
}

[arrufat]

You don't need a sigmoid in the dlib model, it's already included in that loss layer.

[arrufat]

you have to forward one image first, then the network will initialize itself with random parameters.

matrix<unsigned char> dummy(26, 34);
net(dummy)
const auto p = count_parameters(net);

[UlrichVonRekkenin]

Super :)

57'945'345 dlib's parameters against tf model.count_params() = 880'129.

[arrufat]

Something is off. Most the parameters are taken by the first fc layer:

layer<0>	loss_binary_log
layer<1>	output size=(num:1, k:1,    nr:1,   nc:1)	fc	(num_outputs=1) learning_rate_mult=1 weight_decay_mult=1 bias_learning_rate_mult=1 bias_weight_decay_mult=0
layer<2>	output size=(num:1, k:512,  nr:1,   nc:1)	relu
layer<3>	output size=(num:1, k:512,  nr:1,   nc:1)	fc	(num_outputs=512) learning_rate_mult=1 weight_decay_mult=1 bias_learning_rate_mult=1 bias_weight_decay_mult=0
layer<4>	output size=(num:1, k:128,  nr:26,  nc:34)	max_pool (nr=3, nc=3, stride_y=1, stride_x=1, padding_y=1, padding_x=1)
layer<5>	output size=(num:1, k:128,  nr:26,  nc:34)	relu
layer<6>	output size=(num:1, k:128,  nr:26,  nc:34)	con	(num_filters=128, nr=1, nc=1, stride_y=1, stride_x=1, padding_y=0, padding_x=0) learning_rate_mult=1 weight_decay_mult=1 bias_learning_rate_mult=1 bias_weight_decay_mult=0
layer<7>	output size=(num:1, k:64,   nr:26,  nc:34)	max_pool (nr=3, nc=3, stride_y=1, stride_x=1, padding_y=1, padding_x=1)
layer<8>	output size=(num:1, k:64,   nr:26,  nc:34)	relu
layer<9>	output size=(num:1, k:64,   nr:26,  nc:34)	con	(num_filters=64, nr=1, nc=1, stride_y=1, stride_x=1, padding_y=0, padding_x=0) learning_rate_mult=1 weight_decay_mult=1 bias_learning_rate_mult=1 bias_weight_decay_mult=0
layer<10>	output size=(num:1, k:32,   nr:26,  nc:34)	max_pool (nr=3, nc=3, stride_y=1, stride_x=1, padding_y=1, padding_x=1)
layer<11>	output size=(num:1, k:32,   nr:26,  nc:34)	relu
layer<12>	output size=(num:1, k:32,   nr:26,  nc:34)	con	(num_filters=32, nr=1, nc=1, stride_y=1, stride_x=1, padding_y=0, padding_x=0) learning_rate_mult=1 weight_decay_mult=1 bias_learning_rate_mult=1 bias_weight_decay_mult=0
layer<13>	input<matrix>

In layer<3>, the number of parameters is: 128×26×34×512 = 57 933 824

It is weird to have a CNN depend on the image size, that's why we always put a Global Average Pooling (avg_pool_everything in dlib). If you do that, the number of parameters goes down to 77057.

[arrufat]

Anyway, after further inspection, you got the kernel sizes for convolutions and max pooling layers wrong.
Here's my attempt:

#include <dlib/dnn.h>

namespace dnn
{
    using namespace dlib;

    template <long num_filters, long ks, int s, typename SUBNET>
    using conp = add_layer<con_<num_filters, ks, ks, s, s, (ks - 1) / 2, (ks - 1) / 2>, SUBNET>;

    template <size_t NUMS, typename SUBNET>
    using layer = max_pool<2, 2, 2, 2, relu<conp<NUMS, 3, 1, SUBNET>>>;

    using netType = loss_binary_log<fc<1, relu<fc<512, layer<128, layer<64, layer<32, input<matrix<unsigned char>>>>>>>>>;
}

auto main() -> int
{
    dnn::netType net;
    dlib::matrix<unsigned char> image(26, 34);
    net(image);
    std::cout << net << '\n';
    std::cout << dlib::count_parameters(net) << '\n';
}

This outputs 880129.

[UlrichVonRekkenin]

Yes, it seems the same.

Note that you don't need the sigmoid when training, but when inferring from the network, it will output the logits, which you then need to pass through the sigmoid if you want the same output as the TF version.

However, if you're just training a binary classifier and deciding based on the output of the sigmoid (> 0.5) you can just use the logit (> 0) and you don't need to apply the sigmoid to the network output.

Hmmm

what kind of number should I get without sigmoid layer? In a range [0, 1] or something else?

[arrufat]

An unbound number from -inf to +inf

[UlrichVonRekkenin]

An unbound number from -inf to +inf

lol

1. May the target image size (width 17-20 and height about 6-10) be the problem during using trained net?
2. I use dlib::resize_image() to adjust the image size to the training dataset (26*34).
3. But unbounded number always the same negative const.

[arrufat]

Did you train the network?

[UlrichVonRekkenin]

from keras.applications.mobilenet_v2 import MobileNetV2

mobilenet = MobileNetV2(weights="imagenet",include_top=False,input_tensor=Input(shape=(160,160,3)))
mobilenet.trainable = False

And this one.
How to disable training for the loaded model tail?

[arrufat]

As you've seen in the example, you can use set_all_learning_rate_multipliers or set_learning_rate_multipliers_range to set the learning rate multipliers to zero in the layers you want to freeze.
Note that you should only do that if you plan to fine-tune a network, if you just want to do inference, you don't need to do anything. There's no train/eval mode in dlib.