From: Junio C Hamano <gitster@pobox•com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail•com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail•com>
Cc: git@vger•kernel.org, "René Scharfe" <l.s.r@web•de>,
"Jeff King" <peff@peff•net>,
"SZEDER Gábor" <szeder.dev@gmail•com>, "Eric Wong" <e@80x24•org>
Subject: Re: [PATCH] Makefile: fix bugs in coccicheck and speed it up
Date: Thu, 04 Mar 2021 15:18:51 -0800 [thread overview]
Message-ID: <xmqq35xacxt0.fsf@gitster.c.googlers.com> (raw)
In-Reply-To: <20210302205103.12230-1-avarab@gmail.com> ("Ævar Arnfjörð Bjarmason"'s message of "Tue, 2 Mar 2021 21:51:03 +0100")
Ævar Arnfjörð Bjarmason <avarab@gmail•com> writes:
> I've often wondered why "make coccicheck" takes so long. This change
> speeds it up by ~2x and makes it use much less memory. Or a reduction
> of a max of around ~2GB per-process (under the old
> SPATCH_BATCH_SIZE=0) to around ~200MB.
>
> Running the full "make coccicheck" now takes ~50 seconds with -j8 on
> my machine, v.s. ~2x of that before. I've got 64GB of memory on that
> machine, or it would be much slower.
>
> Why has it been so slow? Because I think we've always been running it
> in entirely the wrong mode for what we wanted, and much of the
> previous fixing of this target has involved re-arranging the deck
> chairs on that particular Titanic.
>
> What we really want to do with coccicheck is to do search/replacements
> in all our *.c and *.h files. This is now what we do, and we'll
> process a default of 64 files at a time.
>
> What we were doing before was processing all our *.c files, and for
> each of those *.c files we'd recursively look around for includes and
> see if we needed to search/replace in those too.
>
> That we did that dates back to [1] when we were only processing *.c
> files, and it was always very redundant. We'd e.g. visit the likes of
> strbuf.h lots of times since it's widely used as an include.
>
> Then in the most recent attempt to optimize coccicheck in [2] this
> anti-pattern finally turned into a bug.
>
> Namely: before this change, if your coccicheck rule applied to
> e.g. making a change in strbuf.h itself we'd get *lots* of duplicate
> hunks applying the exact same change, as concurrent spatch processes
> invoked by xargs raced one another. In one instance I ended up with 27
> copies of the same hunk in a strbuf.patch.
>
> Setting SPATCH_BATCH_SIZE=0 and processing all the files in one giant
> batch mitigated this. I suspect the author of [2] either mostly ran in
> that mode, or didn't test on changes that impacted widely used header
> files.
>
> So since we're going to want to process all our *.c and *.h let's just
> do that, and drop --all-includes for --no-includes. It's not spatch's
> job to find our sources, we're doing that. If someone is manually
> tweaking COCCI_SOURCES they can just tweak SPATCH_FLAGS too.
>
> I'm entirely removing SPATCH_BATCH_SIZE. If you want to tweak it you
> can tweak SPATCH_XARGS_FLAGS to e.g. "-n 256", or "-P 4 -n 128". But
> in my testing it isn't worth it to tweak SPATCH_XARGS_FLAGS for a full
> "make coccicheck".
>
> I'm also the whole "cat $@.log" introduced in [3]. Since we don't call
> this in a loop anymore (and xargs will early-exit) we can just rely on
> standard V=1 for debugging issues.
>
> 1. a9a884aea5 (coccicheck: use --all-includes by default, 2016-09-30)
> 2. 960154b9c1 (coccicheck: optionally batch spatch invocations,
> 2019-05-06)
> 3. f5c2bc2b96 (Makefile: detect errors in running spatch, 2017-03-10)
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail•com>
> ---
Nice, so in short, we've been redundantly running the checker code
over and over on the same header files wasting cycles.
Even though I saw you mentioned something about preparing for a
reroll, I'll tentatively queue this version to 'seen' for now.
THanks.
next prev parent reply other threads:[~2021-03-04 23:18 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-12 23:48 [RFC PATCH] *.h: remove extern from function declarations Denton Liu
2019-04-13 1:24 ` Jeff King
2019-04-13 5:45 ` Junio C Hamano
2019-04-15 18:24 ` [PATCH v2 0/3] " Denton Liu
2019-04-15 18:24 ` [PATCH v2 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-15 19:19 ` Thomas Gummerer
2019-04-15 18:24 ` [PATCH v2 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-15 18:24 ` [PATCH v2 3/3] cocci: prevent extern function declarations Denton Liu
2019-04-17 7:58 ` [PATCH v3 0/4] remove extern from " Denton Liu
2019-04-17 7:58 ` [PATCH v3 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-17 7:58 ` [PATCH v3 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-17 7:58 ` [PATCH v3 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-17 7:58 ` [PATCH v3 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-22 5:44 ` [PATCH] cache.h: fix mismerge of 'dl/no-extern-in-func-decl' Denton Liu
2019-04-22 6:30 ` Junio C Hamano
2019-04-22 11:19 ` Junio C Hamano
2019-04-22 21:49 ` [PATCH v3 0/4] remove extern from function declarations Jeff King
2019-04-25 12:07 ` SZEDER Gábor
2019-04-25 18:05 ` Denton Liu
2019-04-30 23:21 ` Johannes Schindelin
2019-05-01 10:01 ` Denton Liu
2019-05-01 18:56 ` Jeff King
2019-05-02 0:04 ` SZEDER Gábor
2019-05-03 9:32 ` Johannes Schindelin
2019-05-03 14:42 ` SZEDER Gábor
2019-05-03 14:58 ` SZEDER Gábor
2019-05-03 17:45 ` Jeff King
2019-05-03 18:44 ` SZEDER Gábor
2019-05-05 5:28 ` Junio C Hamano
2019-05-05 18:09 ` Jacob Keller
2019-05-05 18:08 ` Jacob Keller
2019-05-06 5:11 ` [PATCH] coccicheck: optionally process every source file at once Jeff King
2019-05-06 9:34 ` Duy Nguyen
2019-05-06 23:43 ` [PATCH] coccicheck: optionally batch spatch invocations Jeff King
2019-05-07 1:41 ` Jacob Keller
2019-05-07 2:04 ` Jeff King
2019-05-07 2:42 ` Junio C Hamano
2019-05-07 2:55 ` Jeff King
2019-05-07 3:04 ` Jacob Keller
2019-05-07 4:52 ` Junio C Hamano
2019-05-08 7:07 ` Jeff King
2019-05-08 12:36 ` Denton Liu
2019-05-08 22:39 ` Jeff King
2019-05-07 10:20 ` Duy Nguyen
2019-05-07 11:19 ` SZEDER Gábor
2021-03-02 20:51 ` [PATCH] Makefile: fix bugs in coccicheck and speed it up Ævar Arnfjörð Bjarmason
2021-03-03 9:43 ` Denton Liu
2021-03-03 11:45 ` Ævar Arnfjörð Bjarmason
2021-03-04 23:18 ` Junio C Hamano [this message]
2021-03-05 11:17 ` Ævar Arnfjörð Bjarmason
2021-03-05 10:24 ` Jeff King
2021-03-05 17:20 ` Ævar Arnfjörð Bjarmason
2021-03-06 10:59 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 0/4] Makefile/coccicheck: fix bugs " Ævar Arnfjörð Bjarmason
2021-03-05 19:10 ` René Scharfe.
[not found] ` <xmqqim659u57.fsf@gitster.c.googlers.com>
2021-03-06 11:26 ` René Scharfe.
2021-03-06 12:43 ` René Scharfe.
[not found] ` <xmqqft16914r.fsf@gitster.c.googlers.com>
2021-03-13 16:10 ` René Scharfe.
2021-03-06 17:27 ` Ævar Arnfjörð Bjarmason
2021-03-06 17:41 ` René Scharfe.
2021-03-06 17:52 ` Ævar Arnfjörð Bjarmason
2021-03-06 19:08 ` René Scharfe.
2021-03-05 17:07 ` [PATCH v2 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-06 10:45 ` Jeff King
2021-03-06 19:29 ` Ævar Arnfjörð Bjarmason
2021-03-05 17:07 ` [PATCH v2 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-06 10:51 ` Jeff King
2021-03-05 17:07 ` [PATCH v2 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-06 19:25 ` [PATCH v2 5/4] Makefile/coccicheck: use --include-headers-for-types Ævar Arnfjörð Bjarmason
2021-03-18 20:49 ` SZEDER Gábor
2021-03-19 10:32 ` Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Ævar Arnfjörð Bjarmason
2021-03-22 12:11 ` [PATCH v4 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-22 18:04 ` René Scharfe.
2021-03-22 12:11 ` [PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe.
2021-03-24 19:19 ` Jeff King
2021-03-22 19:09 ` Junio C Hamano
2021-03-22 12:11 ` [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-24 19:26 ` Jeff King
2021-03-25 2:29 ` Ævar Arnfjörð Bjarmason
2021-03-26 4:11 ` Jeff King
2021-03-22 12:11 ` [PATCH v4 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-22 18:05 ` René Scharfe.
2021-03-24 19:27 ` Jeff King
2021-03-27 17:43 ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Junio C Hamano
2021-03-27 19:46 ` Ævar Arnfjörð Bjarmason
2019-05-03 9:40 ` [PATCH v3 0/4] remove extern from function declarations Denton Liu
2019-04-23 23:40 ` [PATCH v4 " Denton Liu
2019-04-23 23:40 ` [PATCH v4 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-23 23:40 ` [PATCH v4 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-24 4:56 ` Junio C Hamano
2019-04-25 19:00 ` Denton Liu
2019-04-23 23:40 ` [PATCH v4 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-23 23:40 ` [PATCH v4 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-29 8:28 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-04-29 8:28 ` [PATCH v5 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-29 8:28 ` [PATCH v5 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-29 8:28 ` [PATCH v5 3/3] *.[ch]: manually align parameter lists Denton Liu
2019-04-29 8:30 ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-05-06 11:03 ` Ævar Arnfjörð Bjarmason
2019-05-06 15:34 ` Denton Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqq35xacxt0.fsf@gitster.c.googlers.com \
--to=gitster@pobox$(echo .)com \
--cc=avarab@gmail$(echo .)com \
--cc=e@80x24$(echo .)org \
--cc=git@vger$(echo .)kernel.org \
--cc=l.s.r@web$(echo .)de \
--cc=peff@peff$(echo .)net \
--cc=szeder.dev@gmail$(echo .)com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox