Re: Merging relayfs?

[Tomasz Kłoczko <kloczek@rudy.mif.pg.gda.pl>]

On Tue, 12 Jul 2005, Vara Prasad wrote:
[..]
> O.K, Tomasz your point is we can do aggregation in the kernel and cut down 
> the amount of data that needs to be sent out from the kernel hence we don't 
> need an efficient, low overhead mechanism like relayfs to get the data out of 
> the kernel. Having relayfs doesn't prevent someone in aggregating the data in 
> the kernel, so it is not an argument for not including relayfs in the kernel 
> when it fills the need for those who needs raw data.

Of course you are right and (look again) this is what I told in first mail 
in this thread :)

> I am part of a team working on systemtap where we are are developing a tool
> similar to Dtrace that does some aggregation where appropriate but nothing 
> like fancy statistics etc. We use relayfs in our systemtap project and  based 
> on my reading of Dtrace paper they use exactly similar to relayfs buffering 
> mechanism as well.

If I can suggest something about order prepare some feactures:

1) prepare base infrastructure for counters,

this "tool" will take very small amount of data and can be performad
by very small pieces of binary codes. Even this will allow perform some
*very* interesting experinments on existing kernel code.
And after above:

2) prepare base infrastructure for association tables of couters (for
   collecting data for example about I/O operations or other two or more
   arguments operations),
3) prepare user space tool with some kind of language which will allow
   hanging ptrobes with aboove tho (simple counters and association tables
   of couters)
4) base functions for measure time (with KProbes overhead and without) and
   store them in couters and association tables,

All above base "tools" for above will take small or medium amount of data 
and can be performad small or medium pieces of binary codes. And after 
above:

5) prepare infrastrucrute for probes which will store data in diffrent
   containers depending on initiator process and/or thread (and maybe in
   next etap also will be good have something more common which will
   depend on stack path),
6) prepare base functions for tracing stack paths (counting them and store
   in association tables),
7) make some kind of study where is it will be good compute something
   more complicated like base "speculative probes" (lookin on
   working DTrace probably answer in this point will be "yes").

All to this moment will not require relayfs because amount of transfered
data will be _very low_.
Details of above will be probably different (I have only some very 
common knowledge about DTrace implementations details and some avarange 
about using dtrace tool) but I want count/pint *only* feactutres which 
will not require using relayfs.

And *after finish above* will be much easier perform some kind of study 
about "is relayfs is still neccessary ?" and *if* answer will be still 
"YES" try to integrate neccessary patches (or maybe something other .. 
maybe better adjusted to all non-above cases). Also add something like 
relayfs at this moment _will not require_ changes in existing code (if 
will require changes will be very small but maybe will ollow reduce 
existin now relayfs (?)).

But if you will build all infrastructure even for simple couters on 
relayfs fundament it will be (IMO) badly/incorrectly designed .. and using
even simple couters will introduce to high overhead for system.

*NOT using realyfs* if it is not neccessary for possibly big amout 
of feactures future KProbes IMO in this case is *fundamental*.

To time where this base not requiring relayfs feactures will not be
integrated in kernel code better IMO will be stop merging relayfs.

> There are tools like itrace and Intel has one (i forgot the name) they would 
> like to get the raw data into user space and do all kinds of  fancy 
> statistical analysis, visualization etc. Their value add is the analysis of 
> the data. I am sure you are not suggesting pushing capabilities of those 
> tools to the kernel, right.

I don't know any thing about this tool (can you sent URL?) but please ..
dont't be fool and do not try as first prepare something eye candy :)
Rest this area for other developers and focus on fundaments :)

regards

kloczek
--
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: kloczek@rudy.mif.pg.gda.pl*