Implement Rescaling-ACG, which is a improved version of ACG attack. #2460

yamamura-k · 2024-07-02T13:20:41Z

Description

A new version of our ACG algorithm, Rescaling-ACG (ReACG), has been implemented. ReACG outperforms APGD and ACG, and also performs better for ImageNet models in particular. Our paper will be presented at International Conference on Pattern Recognition and Artificial Intelligence (ICPRAI), 2024.

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

Same Tests as ACG
Style check

Test Configuration:

OS: Ubuntu 24.04
Python version: 3.10.14
ART version or commit number: 1.18.0
TensorFlow / Keras / PyTorch / MXNet version

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
My changes have been tested using both CPU and GPU devices

…Conjugate Gradient (ACG) attack Signed-off-by: yamamura-k <[email protected]>

Signed-off-by: yamamura-k <[email protected]>

art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+                self.count_condition_1 = np.zeros(shape=(_batch_size,))
+                gradk_1 = np.zeros_like(x_k)
+                cgradk_1 = np.zeros_like(x_k)
+                cgradk = np.zeros_like(x_k)


art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+                cgradk = np.zeros_like(x_k)
+                gradk_1_best = np.zeros_like(x_k)
+                cgradk_1_best = np.zeros_like(x_k)
+                gradk_1_tmp = np.zeros_like(x_k)


art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+                gradk_1_best = np.zeros_like(x_k)
+                cgradk_1_best = np.zeros_like(x_k)
+                gradk_1_tmp = np.zeros_like(x_k)
+                cgradk_1_tmp = np.zeros_like(x_k)


art/attacks/evasion/rescaling_auto_conjugate_gradient.py

codecov · 2024-07-02T13:27:02Z

Codecov Report

Attention: Patch coverage is 87.15596% with 42 lines in your changes missing coverage. Please review.

Project coverage is 85.25%. Comparing base (1207d0a) to head (5411f45).
Report is 13 commits behind head on dev_1.19.0.

Files with missing lines	Patch %	Lines
...tacks/evasion/rescaling_auto_conjugate_gradient.py	87.11%	20 Missing and 22 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           dev_1.19.0    #2460    +/-   ##
============================================
  Coverage       85.24%   85.25%            
============================================
  Files             329      330     +1     
  Lines           30143    30470   +327     
  Branches         5173     5228    +55     
============================================
+ Hits            25696    25976   +280     
- Misses           3019     3043    +24     
- Partials         1428     1451    +23

Files with missing lines	Coverage Δ
art/attacks/evasion/__init__.py	`100.00% <100.00%> (ø)`
...tacks/evasion/rescaling_auto_conjugate_gradient.py	`87.11% <87.11%> (ø)`

... and 3 files with indirect coverage changes

Signed-off-by: yamamura-k <[email protected]>

beat-buesser · 2024-07-18T21:37:22Z

Hi @yamamura-k Thank you very much for you pull request! I'm starting the review now, please apologise the delay.

beat-buesser

Hi @yamamura-k The pull request looks good. Could you please take a look at the minor review comments below?

beat-buesser · 2024-07-18T21:43:46Z

art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+
+# MIT License
+#
+# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2020


Suggested change

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2020

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2024

Thank you for your comment. This suggestion is reflected to the latest commit.

beat-buesser · 2024-07-18T21:44:24Z

art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+"""
+This module implements the 'Rescaling-ACG' attack.
+
+| Paper link:


Do you have a link to your paper? Please insert it here.

The paper link does not appear to be publicly available yet. If it is not published after waiting for a while, we will consider uploading to arxiv.

beat-buesser · 2024-07-18T21:44:48Z

art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+    Implementation of the 'Rescaling-ACG' attack.
+    The original implementation is https://github.com/yamamura-k/ReACG.
+
+    | Paper link:


Please add paper link.

beat-buesser · 2024-07-18T21:49:51Z

art/attacks/evasion/rescaling_auto_conjugate_gradient.py

+        # if self.loss_type not in self._predefined_losses:
+        #     raise ValueError("The argument loss_type has to be either {}.".format(self._predefined_losses))


Do we need to keep these commented lines?

I don't think these comments are necessarily necessary. They have been removed in the revised commit.

beat-buesser · 2024-07-18T21:50:09Z

tests/attacks/evasion/test_rescaling_auto_conjugate_gradient.py

+
+# MIT License
+#
+# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2020


Suggested change

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2020

# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2024

Thank you for your comment. This suggestion is reflected to the latest commit.

beat-buesser · 2024-07-18T22:48:25Z

tests/attacks/evasion/test_rescaling_auto_conjugate_gradient.py

+
+            x_train_mnist_adv = attack.generate(x=x_train_mnist, y=y_train_mnist)
+
+            assert np.max(np.abs(x_train_mnist_adv - x_train_mnist)) > 0.0


Does the attack always create the same adversarial example? Could we also test for the expected pixel values in the adversarial image?

A CPU-only calculation with the same initial point should produce the same adversarial example each time.
However, when using a GPU, the adversarial example may be slightly different each time because the gradient calculation on the GPU is not deterministic.

Signed-off-by: yamamura-k <[email protected]>

yamamura-k · 2024-07-22T05:06:24Z

@beat-buesser Thank you for your review! I have modified my codes to reflect review comments. Although I just removed redundant comments and updated some comment lines, some checks were not successful.

Our paper is not publicly available now, so I will add a link to the paper after it becomes available.

beat-buesser · 2024-08-05T09:31:49Z

@yamamura-k Thank you very much. I will fix the issue with pytest-flake8 in a separate pull request and merge this pull request afterwards.

Signed-off-by: yamamura-k <[email protected]>

yamamura-k · 2024-08-15T06:00:44Z

@beat-buesser Thank you for your review and comments. We published our paper via arxiv, and I added the link. I think your review comments are addressed.

beat-buesser · 2024-08-26T21:58:33Z

Hi @yamamura-k Thank you very much for the updates and the published paper! Please apologise the delay because of vacation and fixing a bug in the unit tests. I have updated your branch with the fixed style checks and as soon as the test run has completed I'll take a final review and merge this pull request.

beat-buesser · 2024-10-07T20:31:15Z

Hi @yamamura-k Thank you very much for adding your new attack to ART! It will be part of the next release ART 1.19 around December 15 and immediately available on release branch dev_1.19.0.

yamamura-k added 4 commits July 2, 2024 17:49

Implement Rescaling-ACG (ReACG), which is a improved version of Auto …

e17fc53

…Conjugate Gradient (ACG) attack Signed-off-by: yamamura-k <[email protected]>

formatting

4d22c12

Signed-off-by: yamamura-k <[email protected]>

Add comments

0291898

Signed-off-by: yamamura-k <[email protected]>

update documentation

2615500

Signed-off-by: yamamura-k <[email protected]>

github-advanced-security bot found potential problems Jul 2, 2024

View reviewed changes

yamamura-k added 2 commits July 3, 2024 07:48

formatting with black

2ceb11f

Signed-off-by: yamamura-k <[email protected]>

raise NotImplementedError if the unexpected loss type is specified

ce42288

Signed-off-by: yamamura-k <[email protected]>

beat-buesser self-assigned this Jul 2, 2024

beat-buesser self-requested a review July 2, 2024 23:43

beat-buesser added this to the ART 1.19.0 milestone Jul 2, 2024

beat-buesser added the enhancement New feature or request label Jul 3, 2024

beat-buesser changed the base branch from main to dev_1.19.0 July 3, 2024 17:46

beat-buesser requested changes Jul 18, 2024

View reviewed changes

yamamura-k added 2 commits July 22, 2024 10:47

remove comments and update license

0ed7f69

Signed-off-by: yamamura-k <[email protected]>

update license

a5a61d3

Signed-off-by: yamamura-k <[email protected]>

add link to paper

2ee873d

Signed-off-by: yamamura-k <[email protected]>

beat-buesser added 3 commits August 26, 2024 23:58

Merge branch 'dev_1.19.0' into main

9b3feac

Merge branch 'dev_1.19.0' into main

1ae8ccb

Merge branch 'dev_1.19.0' into main

5411f45

beat-buesser approved these changes Oct 7, 2024

View reviewed changes

beat-buesser merged commit 7983374 into Trusted-AI:dev_1.19.0 Oct 7, 2024
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Rescaling-ACG, which is a improved version of ACG attack. #2460

Implement Rescaling-ACG, which is a improved version of ACG attack. #2460

yamamura-k commented Jul 2, 2024

codecov bot commented Jul 2, 2024 •

edited

Loading

beat-buesser commented Jul 18, 2024

beat-buesser left a comment •

edited

Loading

beat-buesser Jul 18, 2024

yamamura-k Jul 22, 2024

beat-buesser Jul 18, 2024

yamamura-k Jul 22, 2024

beat-buesser Jul 18, 2024

beat-buesser Jul 18, 2024

yamamura-k Jul 22, 2024

beat-buesser Jul 18, 2024

yamamura-k Jul 22, 2024

beat-buesser Jul 18, 2024

yamamura-k Jul 22, 2024

yamamura-k commented Jul 22, 2024

beat-buesser commented Aug 5, 2024

yamamura-k commented Aug 15, 2024

beat-buesser commented Aug 26, 2024

beat-buesser commented Oct 7, 2024

	# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2020
	# Copyright (C) The Adversarial Robustness Toolbox (ART) Authors 2024

		# if self.loss_type not in self._predefined_losses:
		# raise ValueError("The argument loss_type has to be either {}.".format(self._predefined_losses))


		x_train_mnist_adv = attack.generate(x=x_train_mnist, y=y_train_mnist)

		assert np.max(np.abs(x_train_mnist_adv - x_train_mnist)) > 0.0

Implement Rescaling-ACG, which is a improved version of ACG attack. #2460

Implement Rescaling-ACG, which is a improved version of ACG attack. #2460

Conversation

yamamura-k commented Jul 2, 2024

Description

Type of change

Testing

Checklist

codecov bot commented Jul 2, 2024 • edited Loading

Codecov Report

beat-buesser commented Jul 18, 2024

beat-buesser left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yamamura-k commented Jul 22, 2024

beat-buesser commented Aug 5, 2024

yamamura-k commented Aug 15, 2024

beat-buesser commented Aug 26, 2024

beat-buesser commented Oct 7, 2024

codecov bot commented Jul 2, 2024 •

edited

Loading

beat-buesser left a comment •

edited

Loading