* [RFH] fedora: pedantic fails complaining that awk is missing @ 2025-04-15 23:58 Junio C Hamano 2025-04-16 4:40 ` Todd Zullinger 0 siblings, 1 reply; 4+ messages in thread From: Junio C Hamano @ 2025-04-15 23:58 UTC (permalink / raw) To: git GitHub Actions CI started failing the pedantic (fedora) job at the tip of 'master' few days ago, and the log claims the failure is due to missing "awk". Even though we have seen a few topics to rewrite Perl scriptlet, and I think at least one of them uses awk, but they haven't hit 'master' yet, so it is puzzling why this started failing all of a sudden. So here is a band-aid. I wouldn't be surprised if the base image was updated without telling us. We have seen Ubuntu 20.04 base image retired to cause CI failures as well. Almalinux does not even seem to have awk available as a package (or it may be called differently, but we are not suffering from the lack of awk on that platform anyway), so make sure we ask for awk only on fedora. Signed-off-by: Junio C Hamano <gitster@pobox•com> --- * As the scheduled retirement of U20.04 image broke CI jobs running for 'master', I started looking at making it pass again with the minimum change, i.e. by merging dd/sparse-glibc-workaround and js/ci-github-update-ubuntu topics. Then I somehow found that another job is broken. And this patch seems to make it work. Not knowing how it got broken is unsatisfactory, though. ci/install-dependencies.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 8700c0f292..a7c613ce4c 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -30,8 +30,10 @@ alpine-*) bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null ;; fedora-*|almalinux-*) + awk= + case "$distro" in fedora-*) awk=awk ;; esac dnf -yq update >/dev/null && - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null ;; ubuntu-*|i386/ubuntu-*|debian-*) # Required so that apt doesn't wait for user input on certain packages. -- 2.49.0-511-g173acbf1d8 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [RFH] fedora: pedantic fails complaining that awk is missing 2025-04-15 23:58 [RFH] fedora: pedantic fails complaining that awk is missing Junio C Hamano @ 2025-04-16 4:40 ` Todd Zullinger 2025-04-16 5:31 ` Johannes Schindelin 0 siblings, 1 reply; 4+ messages in thread From: Todd Zullinger @ 2025-04-16 4:40 UTC (permalink / raw) To: Junio C Hamano; +Cc: git Hi, Junio C Hamano wrote: > GitHub Actions CI started failing the pedantic (fedora) job at the > tip of 'master' few days ago, and the log claims the failure is due > to missing "awk". Even though we have seen a few topics to rewrite > Perl scriptlet, and I think at least one of them uses awk, but they > haven't hit 'master' yet, so it is puzzling why this started failing > all of a sudden. > > So here is a band-aid. I wouldn't be surprised if the base image > was updated without telling us. We have seen Ubuntu 20.04 base > image retired to cause CI failures as well. > > Almalinux does not even seem to have awk available as a package > (or it may be called differently, but we are not suffering from > the lack of awk on that platform anyway), so make sure we ask for > awk only on fedora. > > Signed-off-by: Junio C Hamano <gitster@pobox•com> > --- > > * As the scheduled retirement of U20.04 image broke CI jobs running > for 'master', I started looking at making it pass again with the > minimum change, i.e. by merging dd/sparse-glibc-workaround and > js/ci-github-update-ubuntu topics. > > Then I somehow found that another job is broken. And this patch > seems to make it work. Not knowing how it got broken is > unsatisfactory, though. I can hazard a guess (or several)... It is likely that the image pointed to by the fedora:latest tag has moved from fedora 41 to 42, which was released today. The fedora 41 container images have awk installed while the fedora 42 images do not. That change is, I suspect, just part of reducing the size of the base container images. In both AlmaLinux and Fedora (as well as other RHEL derivatives/relatives), awk is provided by the gawk package. On Fedora, `dnf install awk` uses the package filelist data to determine that /usr/bin/awk is provided by gawk and installs gawk as a result. On AlmaLinux (8 & 9, by my quick testing), that is not the case and you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get it installed. Though awk is included in the current AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's probably better to be explicit that we need it installed, as a defense against some future change to the AlmaLinux container removing awk. Using gawk (or even '*bin/awk') would likely be clearer. That avoids relying on 1) a case statement to set an awk variable; and 2) the different behavior of older and newer releases of dnf with respect to the install argument matching on the names of binaries provided by the package. But that's certainly subjective. ;) > ci/install-dependencies.sh | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh > index 8700c0f292..a7c613ce4c 100755 > --- a/ci/install-dependencies.sh > +++ b/ci/install-dependencies.sh > @@ -30,8 +30,10 @@ alpine-*) > bash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null > ;; > fedora-*|almalinux-*) > + awk= > + case "$distro" in fedora-*) awk=awk ;; esac > dnf -yq update >/dev/null && > - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null > + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null > ;; > ubuntu-*|i386/ubuntu-*|debian-*) > # Required so that apt doesn't wait for user input on certain packages. -- Todd ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [RFH] fedora: pedantic fails complaining that awk is missing 2025-04-16 4:40 ` Todd Zullinger @ 2025-04-16 5:31 ` Johannes Schindelin 2025-04-16 14:03 ` Junio C Hamano 0 siblings, 1 reply; 4+ messages in thread From: Johannes Schindelin @ 2025-04-16 5:31 UTC (permalink / raw) To: Todd Zullinger; +Cc: Junio C Hamano, git Hi Todd, On Wed, 16 Apr 2025, Todd Zullinger wrote: > Junio C Hamano wrote: > > GitHub Actions CI started failing the pedantic (fedora) job at the > > tip of 'master' few days ago, and the log claims the failure is due > > to missing "awk". Even though we have seen a few topics to rewrite > > Perl scriptlet, and I think at least one of them uses awk, but they > > haven't hit 'master' yet, so it is puzzling why this started failing > > all of a sudden. > > > > So here is a band-aid. I wouldn't be surprised if the base image > > was updated without telling us. We have seen Ubuntu 20.04 base > > image retired to cause CI failures as well. > > > > Almalinux does not even seem to have awk available as a package > > (or it may be called differently, but we are not suffering from > > the lack of awk on that platform anyway), so make sure we ask for > > awk only on fedora. > > > > Signed-off-by: Junio C Hamano <gitster@pobox•com> > > --- > > > > * As the scheduled retirement of U20.04 image broke CI jobs running > > for 'master', I started looking at making it pass again with the > > minimum change, i.e. by merging dd/sparse-glibc-workaround and > > js/ci-github-update-ubuntu topics. > > > > Then I somehow found that another job is broken. And this patch > > seems to make it work. Not knowing how it got broken is > > unsatisfactory, though. > > I can hazard a guess (or several)... > > It is likely that the image pointed to by the fedora:latest > tag has moved from fedora 41 to 42, which was released > today. The fedora 41 container images have awk installed > while the fedora 42 images do not. That change is, I > suspect, just part of reducing the size of the base > container images. Thank you for that excellent and thorough insight. This is indeed the case in the instances I looked at (for example, Git for Windows' `shears/main` branch was using F42 most recently, and failing: https://github.com/git-for-windows/git/actions/runs/14485163916/job/40629342280#step:2:38 whereas the preceding build used F41 and succeeded: https://github.com/git-for-windows/git/actions/runs/14473893559/job/40594656287#step:2:37). > In both AlmaLinux and Fedora (as well as other RHEL > derivatives/relatives), awk is provided by the gawk package. > > On Fedora, `dnf install awk` uses the package filelist data > to determine that /usr/bin/awk is provided by gawk and > installs gawk as a result. > > On AlmaLinux (8 & 9, by my quick testing), that is not the > case and you'd need to use `dnf install gawk` or `dnf > install '*bin/awk'` to get it installed. Though awk is > included in the current AlmaLinux 8 and 9 images, so it > isn't strictly needed. But it's probably better to be > explicit that we need it installed, as a defense against > some future change to the AlmaLinux container removing awk. > > Using gawk (or even '*bin/awk') would likely be clearer. > That avoids relying on 1) a case statement to set an awk > variable; and 2) the different behavior of older and newer > releases of dnf with respect to the install argument > matching on the names of binaries provided by the package. > > But that's certainly subjective. ;) This reasoning is very convincing, so I allowed myself to turn that into the following commit (which fixed the build: https://github.com/git-for-windows/git/actions/runs/14485317015/job/40629741228): -- snipsnap -- From 4890df8bf1c1311ef0c3b5a92412ccc4f833f944 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin <johannes.schindelin@gmx•de> Date: Wed, 16 Apr 2025 07:17:24 +0200 Subject: [PATCH] ci(pedantic): ensure that awk is installed The image pointed to by the fedora:latest tag has moved from fedora 41 to 42. The fedora 41 container images have awk installed while the fedora 42 images do not. That change is most likely just part of reducing the size of the base container images. In both AlmaLinux and Fedora (as well as other RHEL derivatives/relatives), awk is provided by the gawk package. On Fedora, `dnf install awk` would work, but for unintended reasons! It uses the package filelist data to determine that /usr/bin/awk is provided by gawk and installs gawk as a result. On AlmaLinux (8 & 9, by my quick testing), that is not the case and you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get it installed. Having said that, awk _is_ included in the current AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's probably better to be explicit that we need it installed, as a defense against some future change to the AlmaLinux container removing awk. Using the package name "gawk" is the right thing to do. Note that even '*bin/awk' would have worked, but it is less specific. And who knows, maybe in the far future a BSD variant of awk is offered, too, and would then cause ambiguities. Best to avoid that. Suggested-by: Todd Zullinger <tmz@pobox•com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx•de> --- ci/install-dependencies.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh index 8700c0f2924d..be9ba5e30a47 100755 --- a/ci/install-dependencies.sh +++ b/ci/install-dependencies.sh @@ -31,7 +31,7 @@ alpine-*) ;; fedora-*|almalinux-*) dnf -yq update >/dev/null && - dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null + dnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null ;; ubuntu-*|i386/ubuntu-*|debian-*) # Required so that apt doesn't wait for user input on certain packages. -- ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [RFH] fedora: pedantic fails complaining that awk is missing 2025-04-16 5:31 ` Johannes Schindelin @ 2025-04-16 14:03 ` Junio C Hamano 0 siblings, 0 replies; 4+ messages in thread From: Junio C Hamano @ 2025-04-16 14:03 UTC (permalink / raw) To: Johannes Schindelin; +Cc: Todd Zullinger, git Johannes Schindelin <Johannes.Schindelin@gmx•de> writes: > This reasoning is very convincing, so I allowed myself to turn that into > the following commit (which fixed the build: > https://github.com/git-for-windows/git/actions/runs/14485317015/job/40629741228): > > -- snipsnap -- > From 4890df8bf1c1311ef0c3b5a92412ccc4f833f944 Mon Sep 17 00:00:00 2001 > From: Johannes Schindelin <johannes.schindelin@gmx•de> > Date: Wed, 16 Apr 2025 07:17:24 +0200 > Subject: [PATCH] ci(pedantic): ensure that awk is installed Superb. Thanks for working well together. > > The image pointed to by the fedora:latest tag has moved from fedora 41 > to 42. The fedora 41 container images have awk installed while the > fedora 42 images do not. That change is most likely just part of > reducing the size of the base container images. > > In both AlmaLinux and Fedora (as well as other RHEL > derivatives/relatives), awk is provided by the gawk package. > > On Fedora, `dnf install awk` would work, but for unintended reasons! It > uses the package filelist data to determine that /usr/bin/awk is > provided by gawk and installs gawk as a result. > > On AlmaLinux (8 & 9, by my quick testing), that is not the case and > you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get > it installed. Having said that, awk _is_ included in the current > AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's > probably better to be explicit that we need it installed, as a defense > against some future change to the AlmaLinux container removing awk. > > Using the package name "gawk" is the right thing to do. > Note that even '*bin/awk' would have worked, but it is less specific. > And who knows, maybe in the far future a BSD variant of awk is offered, > too, and would then cause ambiguities. Best to avoid that. The last two paragraphs may want to be rephrased. The using scripts are saying "awk", not "gawk", so there is no specific reason to avoid mawk or other implementations at least for now. As long as installing one of them makes sure that end-user scripts that invoke "awk" ends up invoking the one that was installed, I have no objection to pick just one of them, but I do not think "we must be explicit to call for 'gawk' and not any other implementation" is what we want to say. Instead, we should say something like "Because we know that on both of these distros, our scripts that call for 'awk' had been using 'gawk' that was installed as part of the base image, let's make sure that we explicitly install 'gawk'. If the image already has it, it would be a no-op that does not cause breakage." Specifically, we should not say we want to avoid BSD awk, unless we know our awk scripts won't work with it and/or fixing a bug that was caused by the incompatibilities. > Suggested-by: Todd Zullinger <tmz@pobox•com> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx•de> > --- Thanks for not crediting my initial report ;-) The patch text is as expected. ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-04-16 14:03 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-04-15 23:58 [RFH] fedora: pedantic fails complaining that awk is missing Junio C Hamano 2025-04-16 4:40 ` Todd Zullinger 2025-04-16 5:31 ` Johannes Schindelin 2025-04-16 14:03 ` Junio C Hamano
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox