Re: automated test? (was Re: Linux 2.6.17.7)

[David Lang <dlang@digitalinsight.com>]

On Tue, 25 Jul 2006, Matthias Andree wrote:

> On Tue, 25 Jul 2006, Arjan van de Ven wrote:
>
> > well you can do such a thing withing statistical bounds; however... if
> > the patch already is in -git (as is -stable policy normally).. it should
> > have been found there already...
>
> The sad facts I learned from Debian bug #212762 (not kernel related) that
> culminated in CVE-2005-2335 (remote root exploit against older
> fetchmail) and from various qmail bugs Guninski discovered:
>
> - a bug need not necessarily be found soon after introduction
>
> - a bug report may not convey the hint "look at this NOW, the shit
>  already hit the fan"
>  (sorry, I meant to write: look NOW, it's urgent and important)
>
> - an automated test to catch non-trivial mistakes is non-trivial in
>  itself, and - what I've seen with another project I was involved with,
>  and more often than I found amusing - is that the test itself can be
>  buggy causing bogus results.
>
> That doesn't mean I object to automated tests, but "it should have been
> found by now" (because the source is open, someone could have tested it,
> whatever) just doesn't work.

what I was intending with my origional question was a series of simple 'does it 
compile' tests that try all the config options that are affected by the patchset 
in question. the purpose being to catch simple errors like the one here where 
the patch was diffed against the wrong tree and the result doesn't compile

i.e. if the change is the the e1000 driver, try compiling a kernel with it on, 
off, and as a module

obviously such a test can't be done on the huge -rc patches (they would approach 
an exhaustive test of all config permutations), but for -stable patches (or 
better still the indivdual patches that form a -stable release)

so the first part of the question boils down to

1. Given a patch that modifies file X is it possible to know what config options 
could be affected?

and the second part would be

2. would it make sense for the LTP or one of the big compile farms to do a 
series of compiles prior to the release of a -stable kernel to catch mistakes 
like this?

David Lang
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/