Subject: Re: [possible regression in 6.5-rc1] sendmsg()/splice() returns EBADMSG

From: Jakub Kicinski <[email-masked]>
Date: Fri, 7 Jul 2023 18:09:01 -0700

On Sat, 8 Jul 2023 08:45:50 +0900 Tetsuo Handa wrote:
> (Branched from https://lkml.kernel.org/r/[email-masked] .)
> 
> I found that the following program started returning EBADMSG. Bisection for sendmsg() reached
> commit c5c37af6ecad ("tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES") and bisection
> for splice() reached commit 2dc334f1a63a ("splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather
> than ->sendpage()"). Is this program doing something wrong?
> 
>   6.4.0-rc5-00892-g2dc334f1a63a-dirty    argc==1 ? splice()=EBADMSG, sendmsg()=EBADMSG : sendmsg()=success, splice()=EBADMSG
>   6.4.0-rc5-00891-g81840b3b91aa-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success
> 
>   6.4.0-rc2-00520-gc5c37af6ecad-dirty    argc==1 ? splice()=success, sendmsg()=EBADMSG : sendmsg()=success, splice()=success
>   6.4.0-rc2-00519-g270a1c3de47e-dirty    argc==1 ? splice()=success, sendmsg()=success : sendmsg()=success, splice()=success

> 	setsockopt(fd, SOL_TCP, TCP_REPAIR, &one, sizeof(one));

I think it's just because the repro puts the socket in repair mode, 
and the current code doesn't want to play with repair mode as nicely.

I added:
	// needs #include <linux/tcp.h>
        int val = TCP_SEND_QUEUE;                                                   
        setsockopt(fd, SOL_TCP, TCP_REPAIR_QUEUE, &val, sizeof(val)); 

here (i.e. after putting the socket in repair mode), and I don't get 
the EBADMSG any more. Both sendmsg and splice succeed.

Can you check if we're back to the KMSAN problem with those lines added?

FWIW you can also try to repro with real tls sockets (not in repair
mode) by adding cases to tools/testing/selftests/net/tls.c for example:

TEST_F(tls, bla_bla)
{
	struct iovec iov = {
		.iov_base = "@@@@@@@@@@@@@@@@",
		.iov_len = 16
	};
	struct msghdr hdr = {
		.msg_iov = &iov,
		.msg_iovlen = 1,
		.msg_flags = MSG_FASTOPEN
	};
	int pipe_fds[2] = { -1, -1 };
	static char buf[32768] = { };
	int ret;

	ret = pipe(pipe_fds);
	ASSERT_EQ(ret, 0);

	EXPECT_EQ(write(pipe_fds[1], buf, 2432), 2432);
	EXPECT_EQ(write(pipe_fds[1], buf, 10676), 10676);
	EXPECT_EQ(write(pipe_fds[1], buf, 17996), 17996);

	EXPECT_EQ(splice(pipe_fds[0], NULL, self->fd, NULL, 1048576,
			 SPLICE_F_MORE), 17996 + 10676 + 2432);
	EXPECT_EQ(sendmsg(self->fd, &hdr, MSG_DONTWAIT | MSG_MORE), 16);
}

Then compiling:

make -C tools/testing/selftests/net/

And running:

tools/testing/selftests/net/tls -r tls.13_aes_gcm_256.bla_bla