From: Roland Dreier <rdreier@cisco•com>
To: Heiko J Schick <schihei@de•ibm.com>
Cc: linux-kernel@vger•kernel.org, openib-general@openib•org,
linuxppc-dev@ozlabs•org, Christoph Raisch <RAISCH@de•ibm.com>,
Hoang-Nam Nguyen <HNGUYEN@de•ibm.com>,
Marcus Eder <MEDER@de•ibm.com>
Subject: Re: [openib-general] [PATCH 07/16] ehca: interrupt handling routines
Date: Thu, 04 May 2006 14:29:54 -0700 [thread overview]
Message-ID: <adaejz9o4vh.fsf@cisco.com> (raw)
In-Reply-To: <4450A196.2050901@de.ibm.com> (Heiko J. Schick's message of "Thu, 27 Apr 2006 12:48:54 +0200")
> +void ehca_queue_comp_task(struct ehca_comp_pool *pool, struct ehca_cq *__cq)
> +{
> + int cpu;
> + int cpu_id;
> + struct ehca_cpu_comp_task *cct;
> + unsigned long flags_cct;
> + unsigned long flags_cq;
> +
> + cpu = get_cpu();
> + cpu_id = find_next_online_cpu(pool);
> +
> + EDEB_EN(7, "pool=%p cq=%p cq_nr=%x CPU=%x:%x:%x:%x",
> + pool, __cq, __cq->cq_number,
> + cpu, cpu_id, num_online_cpus(), num_possible_cpus());
> +
> + BUG_ON(!cpu_online(cpu_id));
> +
> + cct = per_cpu_ptr(pool->cpu_comp_tasks, cpu_id);
> +
> + spin_lock_irqsave(&cct->task_lock, flags_cct);
> + spin_lock_irqsave(&__cq->task_lock, flags_cq);
> +
> + if (__cq->nr_callbacks == 0) {
> + __cq->nr_callbacks++;
> + list_add_tail(&__cq->entry, &cct->cq_list);
> + wake_up(&cct->wait_queue);
> + }
> + else
> + __cq->nr_callbacks++;
> +
> + spin_unlock_irqrestore(&__cq->task_lock, flags_cq);
> + spin_unlock_irqrestore(&cct->task_lock, flags_cct);
> +
> + put_cpu();
> +
> + EDEB_EX(7, "cct=%p", cct);
> +
> + return;
> +}
I never read the ehca completion event handling code very carefully
until now. But I was motivated by Shirley's work on IPoIB to take a
closer look.
It seems that you are deferring completion event dispatch into threads
spread across all the CPUs. This seems like a very strange thing to
me -- you are adding latency and possibly causing cacheline pingpong.
It may help throughput in some cases to spread the work across
multiple CPUs but it seems strange to me to do this in the low-level
driver. My intuition would be that it would be better to do this in
the higher levels, and leave open the possibility for protocols that
want the lowest possible latency to be called directly from the
interrupt handler.
What was the thinking that led to this design?
- R.
next prev parent reply other threads:[~2006-05-04 21:39 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2006-04-27 10:48 [PATCH 07/16] ehca: interrupt handling routines Heiko J Schick
2006-05-04 21:29 ` Roland Dreier [this message]
2006-05-05 13:05 ` [openib-general] " Heiko J Schick
2006-05-05 14:49 ` Roland Dreier
2006-05-09 12:35 ` Heiko J Schick
2006-05-09 16:23 ` Roland Dreier
2006-05-09 16:49 ` Michael S. Tsirkin
2006-05-09 18:27 ` [openib-general] " Shirley Ma
2006-05-09 18:36 ` Roland Dreier
2006-05-09 18:44 ` Shirley Ma
2006-05-09 18:44 ` Michael S. Tsirkin
2006-05-09 18:51 ` Shirley Ma
2006-05-09 18:55 ` Michael S. Tsirkin
2006-05-09 18:57 ` [openib-general] " Heiko J Schick
2006-05-09 19:46 ` Shirley Ma
2006-05-09 20:20 ` Michael S. Tsirkin
2006-05-09 21:28 ` Shirley Ma
2006-05-10 5:33 ` Michael S. Tsirkin
2006-05-09 23:35 ` [openib-general] " Segher Boessenkool
2006-05-09 13:15 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=adaejz9o4vh.fsf@cisco.com \
--to=rdreier@cisco$(echo .)com \
--cc=HNGUYEN@de$(echo .)ibm.com \
--cc=MEDER@de$(echo .)ibm.com \
--cc=RAISCH@de$(echo .)ibm.com \
--cc=linux-kernel@vger$(echo .)kernel.org \
--cc=linuxppc-dev@ozlabs$(echo .)org \
--cc=openib-general@openib$(echo .)org \
--cc=schihei@de$(echo .)ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox