From: Alexei Starovoitov <ast@fb•com>
To: "David S . Miller" <davem@davemloft•net>
Cc: Daniel Borkmann <daniel@iogearbox•net>, <netdev@vger•kernel.org>,
<kernel-team@fb•com>
Subject: [PATCH net-next 2/9] bpf: split bpf core interpreter
Date: Tue, 30 May 2017 13:31:28 -0700 [thread overview]
Message-ID: <20170530203135.3642768-3-ast@fb.com> (raw)
In-Reply-To: <20170530203135.3642768-1-ast@fb.com>
split __bpf_prog_run() interpreter into stack allocation and execution parts.
The code section shrinks which helps interpreter performance in some cases.
text data bss dec hex filename
26350 10328 624 37302 91b6 kernel/bpf/core.o.before
25777 10328 624 36729 8f79 kernel/bpf/core.o.after
Very short programs got slower (due to extra function call):
Before:
test_bpf: #89 ALU64_ADD_K: 1 + 2 = 3 jited:0 7 PASS
test_bpf: #90 ALU64_ADD_K: 3 + 0 = 3 jited:0 8 PASS
test_bpf: #91 ALU64_ADD_K: 1 + 2147483646 = 2147483647 jited:0 7 PASS
test_bpf: #92 ALU64_ADD_K: 4294967294 + 2 = 4294967296 jited:0 11 PASS
test_bpf: #93 ALU64_ADD_K: 2147483646 + -2147483647 = -1 jited:0 7 PASS
After:
test_bpf: #89 ALU64_ADD_K: 1 + 2 = 3 jited:0 11 PASS
test_bpf: #90 ALU64_ADD_K: 3 + 0 = 3 jited:0 11 PASS
test_bpf: #91 ALU64_ADD_K: 1 + 2147483646 = 2147483647 jited:0 11 PASS
test_bpf: #92 ALU64_ADD_K: 4294967294 + 2 = 4294967296 jited:0 14 PASS
test_bpf: #93 ALU64_ADD_K: 2147483646 + -2147483647 = -1 jited:0 10 PASS
Longer programs got faster:
Before:
test_bpf: #266 BPF_MAXINSNS: Ctx heavy transformations jited:0 20286 20513 PASS
test_bpf: #267 BPF_MAXINSNS: Call heavy transformations jited:0 31853 31768 PASS
test_bpf: #268 BPF_MAXINSNS: Jump heavy test jited:0 9815 PASS
test_bpf: #269 BPF_MAXINSNS: Very long jump backwards jited:0 6 PASS
test_bpf: #270 BPF_MAXINSNS: Edge hopping nuthouse jited:0 13959 PASS
test_bpf: #271 BPF_MAXINSNS: Jump, gap, jump, ... jited:0 210 PASS
test_bpf: #272 BPF_MAXINSNS: ld_abs+get_processor_id jited:0 21724 PASS
test_bpf: #273 BPF_MAXINSNS: ld_abs+vlan_push/pop jited:0 19118 PASS
After:
test_bpf: #266 BPF_MAXINSNS: Ctx heavy transformations jited:0 19008 18827 PASS
test_bpf: #267 BPF_MAXINSNS: Call heavy transformations jited:0 29238 28450 PASS
test_bpf: #268 BPF_MAXINSNS: Jump heavy test jited:0 9485 PASS
test_bpf: #269 BPF_MAXINSNS: Very long jump backwards jited:0 12 PASS
test_bpf: #270 BPF_MAXINSNS: Edge hopping nuthouse jited:0 13257 PASS
test_bpf: #271 BPF_MAXINSNS: Jump, gap, jump, ... jited:0 213 PASS
test_bpf: #272 BPF_MAXINSNS: ld_abs+get_processor_id jited:0 19389 PASS
test_bpf: #273 BPF_MAXINSNS: ld_abs+vlan_push/pop jited:0 19583 PASS
For real world production programs the difference is noise.
This patch is first step towards reducing interpreter stack consumption.
Signed-off-by: Alexei Starovoitov <ast@kernel•org>
Acked-by: Daniel Borkmann <daniel@iogearbox•net>
---
kernel/bpf/core.c | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 339289402b96..abd410d394bc 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -763,10 +763,10 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);
*
* Decode and execute eBPF instructions.
*/
-static unsigned int __bpf_prog_run(void *ctx, const struct bpf_insn *insn)
+static unsigned int ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn,
+ u64 *stack)
{
- u64 stack[MAX_BPF_STACK / sizeof(u64)];
- u64 regs[MAX_BPF_REG], tmp;
+ u64 tmp;
static const void *jumptable[256] = {
[0 ... 255] = &&default_label,
/* Now overwrite non-defaults ... */
@@ -874,9 +874,6 @@ static unsigned int __bpf_prog_run(void *ctx, const struct bpf_insn *insn)
#define CONT ({ insn++; goto select_insn; })
#define CONT_JMP ({ insn++; goto select_insn; })
- FP = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)];
- ARG1 = (u64) (unsigned long) ctx;
-
select_insn:
goto *jumptable[insn->code];
@@ -1219,7 +1216,17 @@ static unsigned int __bpf_prog_run(void *ctx, const struct bpf_insn *insn)
WARN_RATELIMIT(1, "unknown opcode %02x\n", insn->code);
return 0;
}
-STACK_FRAME_NON_STANDARD(__bpf_prog_run); /* jump table */
+STACK_FRAME_NON_STANDARD(___bpf_prog_run); /* jump table */
+
+static unsigned int __bpf_prog_run(void *ctx, const struct bpf_insn *insn)
+{
+ u64 stack[MAX_BPF_STACK / sizeof(u64)];
+ u64 regs[MAX_BPF_REG];
+
+ FP = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)];
+ ARG1 = (u64) (unsigned long) ctx;
+ return ___bpf_prog_run(regs, insn, stack);
+}
bool bpf_prog_array_compatible(struct bpf_array *array,
const struct bpf_prog *fp)
--
2.9.3
next prev parent reply other threads:[~2017-05-30 20:31 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-30 20:31 [PATCH net-next 0/9] bpf: stack depth tracking Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 1/9] bpf: free up BPF_JMP | BPF_CALL | BPF_X opcode Alexei Starovoitov
2017-05-30 20:31 ` Alexei Starovoitov [this message]
2017-05-30 20:31 ` [PATCH net-next 3/9] bpf: teach verifier to track stack depth Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 4/9] bpf: reconcile bpf_tail_call and stack_depth Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 5/9] bpf: track stack depth of classic bpf programs Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 6/9] bpf: fix stack_depth usage by test_bpf.ko Alexei Starovoitov
2017-05-31 18:15 ` David Miller
2017-05-31 18:39 ` Alexei Starovoitov
2017-05-31 18:43 ` David Miller
2017-05-31 18:45 ` Alexei Starovoitov
2017-05-31 22:41 ` Alexei Starovoitov
2017-05-31 23:30 ` David Miller
2017-06-06 10:19 ` Daniel Borkmann
2017-05-30 20:31 ` [PATCH net-next 7/9] bpf: use different interpreter depending on required stack size Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 8/9] bpf: change x86 JITed program stack layout Alexei Starovoitov
2017-05-30 20:31 ` [PATCH net-next 9/9] bpf: take advantage of stack_depth tracking in x64 JIT Alexei Starovoitov
2017-05-31 23:30 ` [PATCH net-next 0/9] bpf: stack depth tracking David Miller
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170530203135.3642768-3-ast@fb.com \
--to=ast@fb$(echo .)com \
--cc=daniel@iogearbox$(echo .)net \
--cc=davem@davemloft$(echo .)net \
--cc=kernel-team@fb$(echo .)com \
--cc=netdev@vger$(echo .)kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox