Add QDepthwiseConv2D, DepthwiseConv2D, DepthwiseConv1D support #834

jmitrevs · 2023-07-17T20:06:27Z

Description

There was a request to add support for QDepthwiseConv2D for QKeras. Previously we had Keras support for DepthwiseConv1D and DepthwiseConv2D but only as part of separable convolutions. This PR adds support for standalone DepthwiseConv1D and DepthwiseConv2D, and also QDepthwiseConv2D. Currently only depth_multiplier=1 is supported.

Note that QKeras doesn't support QDepthwiseConv1D.

Adding this found an issue in that arguments to activations passed as parameters to, e.g. QDense, such as activation='quantized_relu(3, 0)', were ignored. This had the same effect as activation='quantized_relu()'.

Type of change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change which adds functionality)

Tests

Tests added to both test_keras_api.py and test_qkeras.py

Checklist

I have read the guidelines for contributing.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
I have installed and run pre-commit on the files I edited or added.
I have added tests that prove my fix is effective or that my feature works.

jmitrevs · 2023-07-17T20:15:35Z

I should also mention that this is only for 'io_stream` and only for Vivado and Vitis. There is not Quartus support.

bo3z

Looks good to me - for some reason the PyTests weren't triggered, so probably need to be triggered manually.

As a separate PR it could be worth including a Resource implementation of depth-wise multiplications and parameterise the new tests for both strategies.

bo3z · 2023-07-19T13:54:57Z

hls4ml/templates/vivado/nnet_utils/nnet_sepconv1d_stream.h

@@ -57,6 +57,21 @@ void depthwise_conv_1d_buffer_cl(hls::stream<data_T> &data, hls::stream<res_T> &
    }
 }

+template <class data_T, class res_T, typename CONFIG_T>


For the sake of code brevity - this function can be directly called, instead of the switch statement on lines 112 - 122

You are right. The reason is of course historic, since the function on 112 -122 existed before. But I think I'll make the suggested change.

We should check if using it this way it get inlined or not, otherwise it costs a cycle.

Anyway, I pushed the change. I expect that it would cause no difference in the produced code.

bo3z · 2023-07-19T13:55:53Z

hls4ml/templates/vivado/nnet_utils/nnet_sepconv2d_stream.h

@@ -75,6 +75,22 @@ void depthwise_conv_2d_buffer_cl(
    }
 }

+template <class data_T, class res_T, typename CONFIG_T>


Same comment for using this function below, like in the 1D case.

jmitrevs · 2023-07-19T15:35:04Z

I think we should also support depth_multiplier!=1. Or is that not used often?

jmitrevs · 2023-08-28T14:20:48Z

@bo3z, are there any other things needed for this PR?

bo3z · 2023-08-28T15:55:01Z

@bo3z, are there any other things needed for this PR?

Looks good to me. Maybe worth opening an issue to keep track of the TODOs from this PR (resource strategy and multiplier != 1).

vloncar · 2023-08-28T18:04:04Z

test/pytest/test_keras_api.py

+    model.add(DepthwiseConv2D(kernel_size=(3, 3), input_shape=(32, 32, 3)))
+    model.compile()
+
+    config = hls4ml.utils.config_from_keras_model(model, granularity='name', default_precision='fixed<32,12>')


If the input is clipped to exactly match <16,6>, why do we need to increase the precision this way? Should only accum be increased?

It's not clipped for the keras input, just the hls4ml input. The input is actually all in the [0,1) range. I didn't try to optimize the precision in the model and just sought to make it irrelevant.

test/pytest/test_keras_api.py

vloncar · 2023-08-28T18:06:03Z

test/pytest/test_qkeras.py

+            kernel_size=(3, 3),
+            input_shape=(32, 32, 3),
+            depthwise_quantizer='quantized_bits(6, 0, alpha=1)',
+            depthwise_initializer='ones',


Isn't this test cheating a bit? All the weights will be 1s and the biases 0s, no point in using quantizers. It doesn't really test the proper handling of quantizers and the accuracy of the implementation.

That's a leftover from me playing around with the initializer. Let me find better ones. For depthwise, the default is probably good ("he_normal"), so I'll move to that. The default for bias is 0. Let me see if I should change it.

Probably worth changing the bias initialiser - just to be sure the bias is not ignored in the C++ implementation.

Hi @jmitrevs .
Did you tested with large kernel size?
For example: model.add(DepthwiseConv2D(kernel_size=(16,16), input_shape=(32, 32, 24))).

I'm facing some errors regarding parameters that are bigger than the int256 representation in the parameters.h file.

jmitrevs added 2 commits July 15, 2023 17:39

Add support for DepthwiseConv[12]D in Vitis and Vivado io-stream

330a4ca

add basic QDepthwiseConv2D support, fix activation argument parsing

0014903

jmitrevs added this to the v0.8.0 milestone Jul 17, 2023

jmitrevs added bug enhancement please test Trigger testing by creating local PR branch labels Jul 17, 2023

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Jul 17, 2023

bo3z reviewed Jul 19, 2023

View reviewed changes

remove redundant case statement

923e7ae

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Jul 19, 2023

jmitrevs added 2 commits August 27, 2023 16:03

increase bitwidth to make test comparisons succeed

c160318

Merge branch 'main' into qdepthwiseconv2d

38530a1

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Aug 27, 2023

bo3z approved these changes Aug 28, 2023

View reviewed changes

vloncar reviewed Aug 28, 2023

View reviewed changes

test/pytest/test_keras_api.py Outdated Show resolved Hide resolved

vloncar reviewed Aug 28, 2023

View reviewed changes

jmitrevs added 2 commits August 28, 2023 17:57

Fix docstring text, increase default precision for sensitive tests

b26621b

Merge branch 'main' into qdepthwiseconv2d

a09fb59

jmitrevs added please test Trigger testing by creating local PR branch and removed please test Trigger testing by creating local PR branch labels Aug 28, 2023

vloncar merged commit 4b4b5a0 into fastmachinelearning:main Aug 28, 2023
7 checks passed

jmitrevs mentioned this pull request Aug 29, 2023

Add Resource, io_parallel, depth_multiplier>1 and Quartus support for (Q)DepthwiseConv #859

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add QDepthwiseConv2D, DepthwiseConv2D, DepthwiseConv1D support #834

Add QDepthwiseConv2D, DepthwiseConv2D, DepthwiseConv1D support #834

jmitrevs commented Jul 17, 2023 •

edited

Loading

jmitrevs commented Jul 17, 2023

bo3z left a comment

bo3z Jul 19, 2023

jmitrevs Jul 19, 2023 •

edited

Loading

vloncar Jul 19, 2023

jmitrevs Jul 19, 2023

bo3z Jul 19, 2023

jmitrevs commented Jul 19, 2023

jmitrevs commented Aug 28, 2023

bo3z commented Aug 28, 2023

vloncar Aug 28, 2023

jmitrevs Aug 28, 2023

vloncar Aug 28, 2023

jmitrevs Aug 28, 2023

bo3z Aug 28, 2023

georgenardeswebphy Jan 31, 2024

Add QDepthwiseConv2D, DepthwiseConv2D, DepthwiseConv1D support #834

Add QDepthwiseConv2D, DepthwiseConv2D, DepthwiseConv1D support #834

Conversation

jmitrevs commented Jul 17, 2023 • edited Loading

Description

Type of change

Tests

Checklist

jmitrevs commented Jul 17, 2023

bo3z left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs Jul 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs commented Jul 19, 2023

jmitrevs commented Aug 28, 2023

bo3z commented Aug 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmitrevs commented Jul 17, 2023 •

edited

Loading

jmitrevs Jul 19, 2023 •

edited

Loading