On Mon, 2007-08-13 at 09:43 +0200, Nick Piggin wrote:
> On Fri, Aug 10, 2007 at 05:08:18PM -0400, Lee Schermerhorn wrote:
> > On Wed, 2007-08-08 at 16:25 -0400, Lee Schermerhorn wrote:
> > > On Fri, 2007-07-27 at 10:42 +0200, Nick Piggin wrote:
> > > > Hi,
> > > >
> > > > Just got a bit of time to take another look at the replicated pagecache
> > > > patch. The nopage vs invalidate race and clear_page_dirty_for_io fixes
> > > > gives me more confidence in the locking now; the new ->fault API makes
> > > > MAP_SHARED write faults much more efficient; and a few bugs were found
> > > > and fixed.
> > > >
> > > > More stats were added: *repl* in /proc/vmstat. Survives some kbuilding
> > > > tests...
> > > >
<snip>
>
> Hi Lee,
>
> Am sick with the flu for the past few days, so I haven't done much more
> work here, but I'll just add some (not very useful) comments....
>
> The get_page_from_freelist hang is quite strange. It would be zone->lock,
> which shouldn't have too much contention...
>
> Replication may be putting more stress on some locks. It will cause more
> tlb flushing that can not be batched well, which could cause the call_lock
> to get hotter. Then i_mmap_lock is held over tlb flushing, so it will
> inherit the latency from call_lock. (If this is the case, we could
> potentially extend the tlb flushing API slightly to cope better with
> unmapping of pages from multiple mm's, but that comes way down the track
> when/if replication proves itself!).
>
> tlb flushing AFAIKS should not do the IPI unless it is deadling with a
> multithreaded mm... does usex use threads?
Yes. Apparently, there are some tests, perhaps some of the /usr/bin
apps that get run repeatedly, that are multi-threaded. This job mix
caught a number of races in my auto-migration patches when
multi-threaded tasks race in the page fault paths.
More below...
>
>
> > I should note that I was trying to unmap all mappings to the file backed pages
> > on internode task migration, instead of just the current task's pte's. However,
> > I was only attempting this on pages with mapcount <= 4. So, I don't think I
> > was looping trying to unmap pages with mapcounts of several 10s--such as I see
> > on some page cache pages in my traces.
>
> Replication teardown would still have to unmap all... but that shouldn't
> particularly be any worse than, say, page reclaim (except I guess that it
> could occur more often).
>
>
> > Today, after rebasing to 23-rc2-mm2, I added a patch to unmap only the current
> > task's ptes for ALL !anon pages, regardless of mapcount. I've started the test
> > again and will let it run over the weekend--or as long as it stays up, which
> > ever is shorter :-).
>
> Ah, so it does eventually die? Any hints of why?
No, doesn't die--as in panic. I was just commenting that I'd leave it
running ... However [:-(], it DID hang again. The test window said
that the tests ran for 62h:28m before the screen stopped updating. In
another window, I was running a script to snap the replication and #
file pages vmstats, along with a timestamp, every 10 minutes. That
stopped reporting stats at about 7:30am on Saturday--about 14h:30m into
the test. It still wrote the timestamps [date command] until around 7am
this morning [Monday]--or ~62 hours into test.
So, I do have ~14 hours of replication stats that I can send you or plot
up...
Re: the hang: again, console was scrolling soft lockups continuously.
Checking the messages file, I see hangs in copy_process(),
smp_call_function [as in prev test], vma_link [from mmap], ...
I also see a number of NaT ["not a thing"] consumptions--ia64 specific
error, probably invalid pointer deref--in swapin_readahead, which my
patches hack. These might be the cause of the fork/mmap hangs.
Didn't see that in the 8-9Aug runs, so it might be a result of continued
operation after other hangs/problems; or a botch in the rebase to
rc2-mm2. In any case, I have some work to do there...
>
> >
> > I put a tarball with the rebased series in the Replication directory linked
> > above, in case you're interested. I haven't added the patch description for
> > the new patch yet, but it's pretty simple. Maybe even correct.
> >
> > ----
> >
> > Unrelated to the lockups [I think]:
> >
> > I forgot to look before I rebooted, but earlier the previous evening, I checked
> > the vmstats and at that point [~1.5 hours into the test] we had done ~4.88 million
> > replications and ~4.8 million "zaps" [collapse of replicated page]. That's around
> > 98% zaps. Do we need some filter in the fault path to reduce the "thrashing"--if
> > that's what I'm seeing.
>
> Yep. The current replication patch is very much only infrastructure at
> this stage (and is good for stress testing). I feel sure that heuristics
> and perhaps tunables would be needed to make the most of it.
Yeah. I have some ideas to try...
At the end of the 14.5 hours when it stopped dumping vmstats, we were at
~95% zaps.
>
>
> > A while back I took a look at the Virtual Iron page replication patch. They had
> > set VM_DENY_WRITE when mapping shared executable segments, and only replicated pages
> > in those VMAs. Maybe 'DENY_WRITE isn't exactly what we want. Possibly set another
> > flag for shared executables, if we can detect them, and any shared mapping that has
> > no writable mappings ?
>
> mapping_writably_mapped would be a good one to try. That may be too
> broad in some corner cases where we do want occasionally-written files
> or even parts of files to be replicated, but if we were ever to enable
> CONFIG_REPLICATION by default, I imagine mapping_writably_mapped would
> be the default heuristic.
>
> Still, I appreciate the testing of the "thrashing" case, because with
> the mapping_writably_mapped heuristic, it is likely that bugs could
> remain lurking even in production workloads on huge systems (because
> they will hardly ever get unreplicated).
>
>
> > I'll try to remember to check the replication statistics after the currently
> > running test. If the system stays up, that is. A quick look < 10 minutes into
> > the test shows that zaps are now ~84% of replications. Also, ~47k replicated pages
> > out of ~287K file pages.
>
> Yeah I guess it can be a little misleading: as time approaches infinity,
> zaps will probably approach replications. But that doesn't tell you how
> long a replica stayed around and usefully fed CPUs with local memory...
May be able to capture that info with a more invasive patch -- e.g., add
a timestamp to the page struct. I'll think about it.
And, I'll keep you posted. Not sure how much time I'll be able to
dedicate to this patch stream. Got a few others I need to get back
to...
Later,
Lee
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
