From: "René Scharfe." <l.s.r@web•de>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail•com>, git@vger•kernel.org
Cc: "Junio C Hamano" <gitster@pobox•com>, "Jeff King" <peff@peff•net>,
"SZEDER Gábor" <szeder.dev@gmail•com>, "Eric Wong" <e@80x24•org>
Subject: Re: [PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks
Date: Mon, 22 Mar 2021 19:05:06 +0100 [thread overview]
Message-ID: <365dd12a-b96e-8daf-ba34-bc7d40f39634@web.de> (raw)
In-Reply-To: <3bca3239cb84488a1638f2bb269392f47f78c6da.1616414951.git.avarab@gmail.com>
Am 22.03.21 um 13:11 schrieb Ævar Arnfjörð Bjarmason:
> Change the coccicheck target to run on all of our *.c and *.h files
> with --include-headers-for-types, instead of trusting it to find *.h
> files and other includes to modify from its recursive walking of
> includes as it has been doing with only --all-includes.
>
> The --all-includes option introduced in a9a884aea57 (coccicheck: use
> --all-includes by default, 2016-09-30) is needed because we have rules
> that e.g. use the "type T" construct that wouldn't match unless we
> scoured our headers for the definition of the relevant type.
>
> But combining --all-includes it with processing N files at a time with
> xargs as we've done since 960154b9c17 (coccicheck: optionally batch
> spatch invocations, 2019-05-06) introduced a subtle bug with duplicate
> hunks being written to the generated *.patch files. I did not dig down
> to the root cause of that, but I think it's due to spatch doing (and
> failing to do) some magical locking/racy mtime checking to decide if
> it emits a hunk. See [1] for a way to reproduce the issue, and a
> discussion of it.
>
> This change speeds up the runtime of "make -j8 coccicheck" from ~1m50s
> to ~1m20s for me. See [2] for more timings.
Without this patch, /usr/bin/time -l reports:
95.08 real 598.29 user 32.62 sys
103727104 maximum resident set size
With --include-headers-for-types, I get this:
76.37 real 483.83 user 26.77 sys
97107968 maximum resident set size
This is similar to your numbers. And adding that option to the demo
script in your [1] gives consistent results for all xargs -n values,
so that option alone fixes the subtle bug you refer to above, right?
However, with the second part of this patch also applied (adding %.h
to FOUND_C_SOURCES), I get this:
90.38 real 558.38 user 38.32 sys
100073472 maximum resident set size
Is this still necessary?
>
> We could also use --no-includes for a runtime of ~55s, but that would
> produce broken patches (we miss some hunks) in cases where we need the
> type or other definitions from included files.
>
> If anyone cares there's probably an optimization opportunity here to
> e.g. detect that the whole file doesn't need to consider includes,
> since the rules would be unambiguous without considering them.
>
> Note on portability: The --include-headers-for-types option is not in
> my "man spatch", but it's been part of spatch since 2014. See its
> fe3a327a (include headers for types option, 2014-07-27), it was first
> released with version 1.0.0 of spatch, released in April 2015. The
> spatch developers are just inconsistent about updating their TeX docs
> and manpages at the same time.
>
> 1. https://lore.kernel.org/git/20210305170724.23859-3-avarab@gmail.com/
> 2. https://lore.kernel.org/git/20210306192525.15197-1-avarab@gmail.com/
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail•com>
> ---
> Makefile | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index eef99b4705d..e43a9618df5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1199,7 +1199,8 @@ SPARSE_FLAGS ?=
> SP_EXTRA_FLAGS = -Wno-universal-initializer
>
> # For the 'coccicheck' target
> -SPATCH_FLAGS = --all-includes --patch .
> +SPATCH_FLAGS = --all-includes --include-headers-for-types --patch .
> +
> # For the 'coccicheck' target; setting SPATCH_BATCH_SIZE higher will
> # usually result in less CPU usage at the cost of higher peak memory.
> # Setting it to 0 will feed all files in a single spatch invocation.
> @@ -2860,7 +2861,7 @@ check: config-list.h command-list.h
> exit 1; \
> fi
>
> -FOUND_C_SOURCES = $(filter %.c,$(shell $(FIND_SOURCE_FILES)))
> +FOUND_C_SOURCES = $(filter %.c %.h,$(shell $(FIND_SOURCE_FILES)))
> COCCI_SOURCES = $(filter-out $(THIRD_PARTY_SOURCES),$(FOUND_C_SOURCES))
>
> %.cocci.patch: %.cocci $(COCCI_SOURCES)
>
next prev parent reply other threads:[~2021-03-22 18:06 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-12 23:48 [RFC PATCH] *.h: remove extern from function declarations Denton Liu
2019-04-13 1:24 ` Jeff King
2019-04-13 5:45 ` Junio C Hamano
2019-04-15 18:24 ` [PATCH v2 0/3] " Denton Liu
2019-04-15 18:24 ` [PATCH v2 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-15 19:19 ` Thomas Gummerer
2019-04-15 18:24 ` [PATCH v2 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-15 18:24 ` [PATCH v2 3/3] cocci: prevent extern function declarations Denton Liu
2019-04-17 7:58 ` [PATCH v3 0/4] remove extern from " Denton Liu
2019-04-17 7:58 ` [PATCH v3 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-17 7:58 ` [PATCH v3 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-17 7:58 ` [PATCH v3 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-17 7:58 ` [PATCH v3 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-22 5:44 ` [PATCH] cache.h: fix mismerge of 'dl/no-extern-in-func-decl' Denton Liu
2019-04-22 6:30 ` Junio C Hamano
2019-04-22 11:19 ` Junio C Hamano
2019-04-22 21:49 ` [PATCH v3 0/4] remove extern from function declarations Jeff King
2019-04-25 12:07 ` SZEDER Gábor
2019-04-25 18:05 ` Denton Liu
2019-04-30 23:21 ` Johannes Schindelin
2019-05-01 10:01 ` Denton Liu
2019-05-01 18:56 ` Jeff King
2019-05-02 0:04 ` SZEDER Gábor
2019-05-03 9:32 ` Johannes Schindelin
2019-05-03 14:42 ` SZEDER Gábor
2019-05-03 14:58 ` SZEDER Gábor
2019-05-03 17:45 ` Jeff King
2019-05-03 18:44 ` SZEDER Gábor
2019-05-05 5:28 ` Junio C Hamano
2019-05-05 18:09 ` Jacob Keller
2019-05-05 18:08 ` Jacob Keller
2019-05-06 5:11 ` [PATCH] coccicheck: optionally process every source file at once Jeff King
2019-05-06 9:34 ` Duy Nguyen
2019-05-06 23:43 ` [PATCH] coccicheck: optionally batch spatch invocations Jeff King
2019-05-07 1:41 ` Jacob Keller
2019-05-07 2:04 ` Jeff King
2019-05-07 2:42 ` Junio C Hamano
2019-05-07 2:55 ` Jeff King
2019-05-07 3:04 ` Jacob Keller
2019-05-07 4:52 ` Junio C Hamano
2019-05-08 7:07 ` Jeff King
2019-05-08 12:36 ` Denton Liu
2019-05-08 22:39 ` Jeff King
2019-05-07 10:20 ` Duy Nguyen
2019-05-07 11:19 ` SZEDER Gábor
2021-03-02 20:51 ` [PATCH] Makefile: fix bugs in coccicheck and speed it up Ævar Arnfjörð Bjarmason
2021-03-03 9:43 ` Denton Liu
2021-03-03 11:45 ` Ævar Arnfjörð Bjarmason
2021-03-04 23:18 ` Junio C Hamano
2021-03-05 11:17 ` Ævar Arnfjörð Bjarmason
2021-03-05 10:24 ` Jeff King
2021-03-05 17:20 ` Ævar Arnfjörð Bjarmason
2021-03-06 10:59 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 0/4] Makefile/coccicheck: fix bugs " Ævar Arnfjörð Bjarmason
2021-03-05 19:10 ` René Scharfe.
[not found] ` <xmqqim659u57.fsf@gitster.c.googlers.com>
2021-03-06 11:26 ` René Scharfe.
2021-03-06 12:43 ` René Scharfe.
[not found] ` <xmqqft16914r.fsf@gitster.c.googlers.com>
2021-03-13 16:10 ` René Scharfe.
2021-03-06 17:27 ` Ævar Arnfjörð Bjarmason
2021-03-06 17:41 ` René Scharfe.
2021-03-06 17:52 ` Ævar Arnfjörð Bjarmason
2021-03-06 19:08 ` René Scharfe.
2021-03-05 17:07 ` [PATCH v2 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-06 10:45 ` Jeff King
2021-03-06 19:29 ` Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-06 10:51 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-06 19:25 ` [PATCH v2 5/4] Makefile/coccicheck: use --include-headers-for-types Ævar Arnfjörð Bjarmason
2021-03-18 20:49 ` SZEDER Gábor
2021-03-19 10:32 ` Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-22 18:04 ` René Scharfe.
2021-03-22 12:11 ` [PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe. [this message]
2021-03-24 19:19 ` Jeff King
2021-03-22 19:09 ` Junio C Hamano
2021-03-22 12:11 ` [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-24 19:26 ` Jeff King
2021-03-25 2:29 ` Ævar Arnfjörð Bjarmason
2021-03-26 4:11 ` Jeff King
2021-03-22 12:11 ` [PATCH v4 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe.
2021-03-24 19:27 ` Jeff King
2021-03-27 17:43 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Junio C Hamano
2021-03-27 19:46 ` Ævar Arnfjörð Bjarmason
2019-05-03 9:40 ` [PATCH v3 0/4] remove extern from function declarations Denton Liu
2019-04-23 23:40 ` [PATCH v4 " Denton Liu
2019-04-23 23:40 ` [PATCH v4 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-23 23:40 ` [PATCH v4 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-24 4:56 ` Junio C Hamano
2019-04-25 19:00 ` Denton Liu
2019-04-23 23:40 ` [PATCH v4 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-23 23:40 ` [PATCH v4 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-29 8:28 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-04-29 8:28 ` [PATCH v5 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-29 8:28 ` [PATCH v5 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-29 8:28 ` [PATCH v5 3/3] *.[ch]: manually align parameter lists Denton Liu
2019-04-29 8:30 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-05-06 11:03 ` Ævar Arnfjörð Bjarmason
2019-05-06 15:34 ` Denton Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=365dd12a-b96e-8daf-ba34-bc7d40f39634@web.de \
--to=l.s.r@web$(echo .)de \
--cc=avarab@gmail$(echo .)com \
--cc=e@80x24$(echo .)org \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=peff@peff$(echo .)net \
--cc=szeder.dev@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox