On Mon, Nov 20, 2006 at 08:15:14AM +0100, Jens Axboe wrote:
> On Sun, Nov 19 2006, Paul E. McKenney wrote:
> > On Sat, Nov 18, 2006 at 09:46:24PM +0300, Oleg Nesterov wrote:
> > > On 11/17, Paul E. McKenney wrote:
> > > >
> > > > Oleg, any thoughts about Jens's optimization? He would code something
> > > > like:
> > > >
> > > > if (srcu_readers_active(&my_srcu))
> > > > synchronize_srcu();
> > > > else
> > > > smp_mb();
> > >
> > > Well, this is clearly racy, no? I am not sure, but may be we can do
> > >
> > > smp_mb();
> > > if (srcu_readers_active(&my_srcu))
> > > synchronize_srcu();
> > >
> > > in this case we also need to add 'smp_mb()' into srcu_read_lock() after
> > > 'atomic_inc(&sp->hardluckref)'.
> > >
> > > > However, he is doing ordered I/O requests rather than protecting data
> > > > structures.
> > >
> > > Probably this makes a difference, but I don't understand this.
> >
> > OK, one hypothesis here...
> >
> > The I/Os must be somehow explicitly ordered to qualify
> > for I/O-barrier separation. If two independent processes
> > issue I/Os concurrently with a third process doing an
> > I/O barrier, the I/O barrier is free to separate the
> > two concurrent I/Os or not, on its whim.
> >
> > Jens, is the above correct? If so, what would the two processes
>
> That's completely correct, hence my somewhat relaxed approach with SRCU.
OK, less scary in that case. ;-)
> > need to do in order to ensure that their I/O was considered to be
> > ordered with respect to the I/O barrier? Here are some possibilities:
>
> If we consider the barrier a barrier in a certain stream of requests,
> it is the responsibility of the issuer of that barrier to ensure that
> the queueing is ordered. So if two "unrelated" streams of requests with
> barriers hit __make_request() at the same time, we don't go to great
> lengths to ensure who gets there firt.
So the "preceding" requests have to have completed their I/O system
calls? If this is the case, does this include normal (non-direct/raw)
writes and asynchronous reads? My guess is that it would include
asynchronous I/O, but not buffered writes.
> > 1. I/O barriers apply only to preceding and following I/Os from
> > the process issuing the I/O barrier.
> >
> > 2. As for #1 above, but restricted to task rather than process.
> >
> > 3. I/O system calls that have completed are ordered by the
> > barrier to precede I/O system calls that have not yet
> > started, but I/O system calls still in flight could legally
> > land on either side of the concurrently executing I/O
> > barrier.
> >
> > 4. Something else entirely?
> >
> > Given some restriction like one of the above, it is entirely possible
> > that we don't even need the memory barrier...
>
> 3 is the closest. The request queue doesn't really know the scope of the
> barrier, it has to rely on the issuer getting it right. If you have two
> competing processes issuing io and process A relies on process B issuing
> a barrier, they have to synchronize that between them. Normally that is
> not a problem, since that's how the file systems always did io before
> barriers on items that need to be on disk (it was a serialization point
> anyway, it's just a stronger one now).
So something like a user-level mutex or atomic instructions must be used
by the tasks doing the pre-barrier I/Os to announce that these I/Os have
been started in the kernel.
> That said, I think the
>
> smp_mb();
> if (srcu_readers_active(sp))
> synchronize_srcu();
>
> makes the most sense.
If the user-level tasks/threads/processes must explicitly synchronize,
and if the pre-barrier I/O-initation syscalls have to have completed,
then I am not sure that the smp_mb() is needed. Seems like the queuing
mechanisms in the syscall and the user-level synchronization would have
supplied the needed memory barriers. Or are you using some extremely
lightweight user-level synchronization?
Thanx, Paul
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
