From: Ben Peart <benpeart@microsoft•com>
To: benpeart@microsoft•com
Cc: David.Turner@twosigma•com, avarab@gmail•com,
christian.couder@gmail•com, git@vger•kernel.org,
gitster@pobox•com, johannes.schindelin@gmx•de, pclouds@gmail•com,
peff@peff•net
Subject: [PATCH v6 11/12] fsmonitor: add a sample integration script for Watchman
Date: Fri, 15 Sep 2017 15:20:42 -0400 [thread overview]
Message-ID: <20170915192043.4516-12-benpeart@microsoft.com> (raw)
In-Reply-To: <20170915192043.4516-1-benpeart@microsoft.com>
This script integrates the new fsmonitor capabilities of git with the
cross platform Watchman file watching service. To use the script:
Download and install Watchman from https://facebook.github.io/watchman/.
Rename the sample integration hook from fsmonitor-watchman.sample to
fsmonitor-watchman. Configure git to use the extension:
git config core.fsmonitor .git/hooks/fsmonitor-watchman
Optionally turn on the untracked cache for optimal performance.
Signed-off-by: Ben Peart <benpeart@microsoft•com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx•de>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail•com>
Signed-off-by: Christian Couder <christian.couder@gmail•com>
---
templates/hooks--fsmonitor-watchman.sample | 119 +++++++++++++++++++++++++++++
1 file changed, 119 insertions(+)
create mode 100755 templates/hooks--fsmonitor-watchman.sample
diff --git a/templates/hooks--fsmonitor-watchman.sample b/templates/hooks--fsmonitor-watchman.sample
new file mode 100755
index 0000000000..2779d7edf3
--- /dev/null
+++ b/templates/hooks--fsmonitor-watchman.sample
@@ -0,0 +1,119 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+use IPC::Open2;
+
+# An example hook script to integrate Watchman
+# (https://facebook.github.io/watchman/) with git to speed up detecting
+# new and modified files.
+#
+# The hook is passed a version (currently 1) and a time in nanoseconds
+# formatted as a string and outputs to stdout all files that have been
+# modified since the given time. Paths must be relative to the root of
+# the working tree and separated by a single NUL.
+#
+# To enable this hook, rename this file to "query-watchman" and set
+# 'git config core.fsmonitor .git/hooks/query-watchman'
+#
+my ($version, $time) = @ARGV;
+
+# Check the hook interface version
+
+if ($version == 1) {
+ # convert nanoseconds to seconds
+ $time = int $time / 1000000000;
+} else {
+ die "Unsupported query-fsmonitor hook version '$version'.\n" .
+ "Falling back to scanning...\n";
+}
+
+# Convert unix style paths to escaped Windows style paths when running
+# in Windows command prompt
+
+my $system = `uname -s`;
+$system =~ s/[\r\n]+//g;
+my $git_work_tree;
+
+if ($system =~ m/^MSYS_NT/) {
+ $git_work_tree = `cygpath -aw "\$PWD"`;
+ $git_work_tree =~ s/[\r\n]+//g;
+ $git_work_tree =~ s,\\,/,g;
+} else {
+ $git_work_tree = $ENV{'PWD'};
+}
+
+my $retry = 1;
+
+launch_watchman();
+
+sub launch_watchman {
+
+ # Set input record separator
+ local $/ = 0666;
+
+ my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'watchman -j')
+ or die "open2() failed: $!\n" .
+ "Falling back to scanning...\n";
+
+ # In the query expression below we're asking for names of files that
+ # changed since $time but were not transient (ie created after
+ # $time but no longer exist).
+ #
+ # To accomplish this, we're using the "since" generator to use the
+ # recency index to select candidate nodes and "fields" to limit the
+ # output to file names only. Then we're using the "expression" term to
+ # further constrain the results.
+ #
+ # The category of transient files that we want to ignore will have a
+ # creation clock (cclock) newer than $time_t value and will also not
+ # currently exist.
+
+ print CHLD_IN "[\"query\", \"$git_work_tree\", { \
+ \"since\": $time, \
+ \"fields\": [\"name\"], \
+ \"expression\": [\"not\", [\"allof\", [\"since\", $time, \"cclock\"], [\"not\", \"exists\"]]] \
+ }]";
+
+ my $response = <CHLD_OUT>;
+
+ die "Watchman: command returned no output.\n" .
+ "Falling back to scanning...\n" if $response eq "";
+ die "Watchman: command returned invalid output: $response\n" .
+ "Falling back to scanning...\n" unless $response =~ /^\{/;
+
+ my $json_pkg;
+ eval {
+ require JSON::XS;
+ $json_pkg = "JSON::XS";
+ 1;
+ } or do {
+ require JSON::PP;
+ $json_pkg = "JSON::PP";
+ };
+
+ my $o = $json_pkg->new->utf8->decode($response);
+
+ if ($retry > 0 and $o->{error} and $o->{error} =~ m/unable to resolve root .* directory (.*) is not watched/) {
+ print STDERR "Adding '$git_work_tree' to watchman's watch list.\n";
+ $retry--;
+ qx/watchman watch "$git_work_tree"/;
+ die "Failed to make watchman watch '$git_work_tree'.\n" .
+ "Falling back to scanning...\n" if $? != 0;
+
+ # Watchman will always return all files on the first query so
+ # return the fast "everything is dirty" flag to git and do the
+ # Watchman query just to get it over with now so we won't pay
+ # the cost in git to look up each individual file.
+ print "*\0";
+ eval { launch_watchman() };
+ exit 0;
+ }
+
+ die "Watchman: $o->{error}.\n" .
+ "Falling back to scanning...\n" if $o->{error};
+
+ binmode STDOUT, ":utf8";
+ local $, = "\0";
+ print @{$o->{files}};
+}
--
2.14.1.548.ge54b1befee.dirty
next prev parent reply other threads:[~2017-09-15 19:21 UTC|newest]
Thread overview: 137+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-10 13:40 [PATCH v5 0/7] Fast git status via a file system watcher Ben Peart
2017-06-10 13:40 ` [PATCH v5 1/7] bswap: add 64 bit endianness helper get_be64 Ben Peart
2017-06-10 13:40 ` [PATCH v5 2/7] dir: make lookup_untracked() available outside of dir.c Ben Peart
2017-06-10 13:40 ` [PATCH v5 3/7] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files Ben Peart
2017-06-27 15:43 ` Christian Couder
2017-07-03 21:25 ` Ben Peart
2017-06-10 13:40 ` [PATCH v5 4/7] fsmonitor: add test cases for fsmonitor extension Ben Peart
2017-06-27 16:20 ` Christian Couder
2017-07-07 18:50 ` Ben Peart
2017-06-10 13:40 ` [PATCH v5 5/7] fsmonitor: add documentation for the " Ben Peart
2017-06-10 13:40 ` [PATCH v5 6/7] fsmonitor: add a sample query-fsmonitor hook script for Watchman Ben Peart
2017-06-10 13:40 ` [PATCH v5 7/7] fsmonitor: add a performance test Ben Peart
2017-06-10 14:04 ` Ben Peart
2017-06-12 22:04 ` Junio C Hamano
2017-06-14 14:12 ` Ben Peart
2017-06-14 18:36 ` Junio C Hamano
2017-07-07 18:14 ` Ben Peart
2017-07-07 18:35 ` Junio C Hamano
2017-07-07 19:07 ` Ben Peart
2017-07-07 19:33 ` David Turner
2017-07-08 7:19 ` Christian Couder
2017-06-28 5:11 ` [PATCH v5 0/7] Fast git status via a file system watcher Christian Couder
2017-07-10 13:36 ` Ben Peart
2017-07-10 14:40 ` Ben Peart
2017-09-15 19:20 ` [PATCH v6 00/12] " Ben Peart
2017-09-15 19:20 ` [PATCH v6 01/12] bswap: add 64 bit endianness helper get_be64 Ben Peart
2017-09-15 19:20 ` [PATCH v6 02/12] preload-index: add override to enable testing preload-index Ben Peart
2017-09-15 19:20 ` [PATCH v6 03/12] update-index: add a new --force-write-index option Ben Peart
2017-09-15 19:20 ` [PATCH v6 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files Ben Peart
2017-09-15 21:35 ` David Turner
2017-09-18 13:07 ` Ben Peart
2017-09-18 13:32 ` David Turner
2017-09-18 13:49 ` Ben Peart
2017-09-15 19:20 ` [PATCH v6 05/12] fsmonitor: add documentation for the fsmonitor extension Ben Peart
2017-09-15 19:43 ` David Turner
2017-09-18 13:27 ` Ben Peart
2017-09-17 8:03 ` Junio C Hamano
2017-09-18 13:29 ` Ben Peart
2017-09-15 19:20 ` [PATCH v6 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit Ben Peart
2017-09-15 20:34 ` David Turner
2017-09-15 19:20 ` [PATCH v6 07/12] update-index: add fsmonitor support to update-index Ben Peart
2017-09-15 19:20 ` [PATCH v6 08/12] fsmonitor: add a test tool to dump the index extension Ben Peart
2017-09-17 8:02 ` Junio C Hamano
2017-09-18 13:38 ` Ben Peart
2017-09-18 15:43 ` Torsten Bögershausen
2017-09-18 16:28 ` Ben Peart
2017-09-19 14:16 ` Torsten Bögershausen
2017-09-19 15:36 ` Ben Peart
2017-09-15 19:20 ` [PATCH v6 09/12] split-index: disable the fsmonitor extension when running the split index test Ben Peart
2017-09-19 20:43 ` Jonathan Nieder
2017-09-20 17:11 ` Ben Peart
2017-09-20 17:46 ` Jonathan Nieder
2017-09-21 0:05 ` Ben Peart
2017-09-15 19:20 ` [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension Ben Peart
2017-09-15 22:00 ` David Turner
2017-09-19 19:32 ` David Turner
2017-09-19 20:30 ` Ben Peart
2017-09-16 15:27 ` Torsten Bögershausen
2017-09-17 5:43 ` [PATCH v1 1/1] test-lint: echo -e (or -E) is not portable tboegi
2017-09-19 20:37 ` Jonathan Nieder
2017-09-20 13:49 ` Torsten Bögershausen
2017-09-22 1:04 ` Junio C Hamano
2017-09-18 14:06 ` [PATCH v6 10/12] fsmonitor: add test cases for fsmonitor extension Ben Peart
2017-09-17 4:47 ` Junio C Hamano
2017-09-18 15:25 ` Ben Peart
2017-09-19 20:34 ` Jonathan Nieder
2017-09-15 19:20 ` Ben Peart [this message]
2017-09-15 19:20 ` [PATCH v6 12/12] fsmonitor: add a performance test Ben Peart
2017-09-15 21:56 ` David Turner
2017-09-18 14:24 ` Johannes Schindelin
2017-09-18 18:19 ` Ben Peart
2017-09-19 15:28 ` Johannes Schindelin
2017-09-19 19:27 ` [PATCH v7 00/12] Fast git status via a file system watcher Ben Peart
2017-09-19 19:27 ` [PATCH v7 01/12] bswap: add 64 bit endianness helper get_be64 Ben Peart
2017-09-19 19:27 ` [PATCH v7 02/12] preload-index: add override to enable testing preload-index Ben Peart
2017-09-20 22:06 ` Stefan Beller
2017-09-21 0:02 ` Ben Peart
2017-09-21 0:44 ` Stefan Beller
2017-09-19 19:27 ` [PATCH v7 03/12] update-index: add a new --force-write-index option Ben Peart
2017-09-20 5:47 ` Junio C Hamano
2017-09-20 14:58 ` Ben Peart
2017-09-21 1:46 ` Junio C Hamano
2017-09-21 2:06 ` Ben Peart
2017-09-21 2:18 ` Junio C Hamano
2017-09-21 2:32 ` Junio C Hamano
2017-09-19 19:27 ` [PATCH v7 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files Ben Peart
2017-09-20 2:28 ` Junio C Hamano
2017-09-20 16:19 ` Ben Peart
2017-09-21 2:00 ` Junio C Hamano
2017-09-21 2:24 ` Ben Peart
2017-09-21 14:35 ` Ben Peart
2017-09-22 1:02 ` Junio C Hamano
2017-09-20 6:23 ` Junio C Hamano
2017-09-20 16:29 ` Ben Peart
2017-09-19 19:27 ` [PATCH v7 05/12] fsmonitor: add documentation for the fsmonitor extension Ben Peart
2017-09-20 10:00 ` Martin Ågren
2017-09-20 17:02 ` Ben Peart
2017-09-20 17:11 ` Martin Ågren
2017-09-19 19:27 ` [PATCH v7 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit Ben Peart
2017-09-19 19:46 ` David Turner
2017-09-19 20:44 ` Ben Peart
2017-09-19 21:27 ` David Turner
2017-09-19 22:44 ` Ben Peart
2017-09-19 19:27 ` [PATCH v7 07/12] update-index: add fsmonitor support to update-index Ben Peart
2017-09-19 19:27 ` [PATCH v7 08/12] fsmonitor: add a test tool to dump the index extension Ben Peart
2017-09-19 19:27 ` [PATCH v7 09/12] split-index: disable the fsmonitor extension when running the split index test Ben Peart
2017-09-19 19:27 ` [PATCH v7 10/12] fsmonitor: add test cases for fsmonitor extension Ben Peart
2017-09-19 19:27 ` [PATCH v7 11/12] fsmonitor: add a sample integration script for Watchman Ben Peart
2017-09-19 19:27 ` [PATCH v7 12/12] fsmonitor: add a performance test Ben Peart
2017-09-22 16:35 ` [PATCH v8 00/12] Fast git status via a file system watcher Ben Peart
2017-09-22 16:35 ` [PATCH v8 01/12] bswap: add 64 bit endianness helper get_be64 Ben Peart
2017-09-22 23:37 ` Martin Ågren
2017-09-23 23:31 ` Ben Peart
2017-09-24 3:51 ` Jeff King
2017-09-24 3:52 ` Junio C Hamano
2017-09-22 16:35 ` [PATCH v8 02/12] preload-index: add override to enable testing preload-index Ben Peart
2017-09-22 16:35 ` [PATCH v8 03/12] update-index: add a new --force-write-index option Ben Peart
2017-09-22 16:35 ` [PATCH v8 04/12] fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files Ben Peart
2017-09-22 16:35 ` [PATCH v8 05/12] fsmonitor: add documentation for the fsmonitor extension Ben Peart
2017-09-22 16:35 ` [PATCH v8 06/12] ls-files: Add support in ls-files to display the fsmonitor valid bit Ben Peart
2017-09-22 16:35 ` [PATCH v8 07/12] update-index: add fsmonitor support to update-index Ben Peart
2017-09-22 16:35 ` [PATCH v8 08/12] fsmonitor: add a test tool to dump the index extension Ben Peart
2017-09-22 23:37 ` Martin Ågren
2017-09-23 23:33 ` Ben Peart
2017-09-24 3:51 ` Junio C Hamano
2017-09-22 16:35 ` [PATCH v8 09/12] split-index: disable the fsmonitor extension when running the split index test Ben Peart
2017-09-22 16:35 ` [PATCH v8 10/12] fsmonitor: add test cases for fsmonitor extension Ben Peart
2017-09-22 16:35 ` [PATCH v8 11/12] fsmonitor: add a sample integration script for Watchman Ben Peart
2017-09-22 16:35 ` [PATCH v8 12/12] fsmonitor: add a performance test Ben Peart
2017-09-29 2:20 ` [PATCH v8 00/12] Fast git status via a file system watcher Junio C Hamano
2017-09-29 12:07 ` Ben Peart
2017-10-01 8:24 ` Junio C Hamano
2017-10-03 19:48 ` Ben Peart
2017-10-04 2:09 ` Junio C Hamano
2017-10-04 6:38 ` Alex Vandiver
2017-10-04 12:48 ` Ben Peart
2017-10-04 12:27 ` Ben Peart
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170915192043.4516-12-benpeart@microsoft.com \
--to=benpeart@microsoft$(echo .)com \
--cc=David.Turner@twosigma$(echo .)com \
--cc=avarab@gmail$(echo .)com \
--cc=christian.couder@gmail$(echo .)com \
--cc=git@vger$(echo .)kernel.org \
--cc=gitster@pobox$(echo .)com \
--cc=johannes.schindelin@gmx$(echo .)de \
--cc=pclouds@gmail$(echo .)com \
--cc=peff@peff$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox