On Thursday 28 April 2005 10:57, Lars Marowsky-Bree wrote:
> On 2005-04-27T18:38:18, Daniel Phillips <[email protected]> wrote:
> > Uuids's at this level are inherently bogus, unless of course you have
> > more than 2**32 cluster nodes. I don't know about you, but I do not have
> > even half that many nodes over here.
>
> This is not quite the argument. With that argument, 16 bit would be
> fine. And even then, I'd call you guilty of causing my lights to flicker
> ;-)
BlueGene is pushing the 16 bit node number boundary already, 32 bits seems
prudent. More is silly. Think of the node number as more like a PID than a
UUID.
> The argument about UUIDs goes a bit beyond that: No admin needed to
> assign them; they can stay the same even if clusters/clusters merge (in
> theory); they can be used for inter-cluster addressing too, because they
> aren't just unique within a single cluster (think clusters of clusters,
> grids etc, whatever the topology), and finally, UUID is a big enough
> blob to put all other identifiers in, be it a two bit node id, a
> nodename, 32bit IPv4 address or a 128bit IPv6.
>
> This piece is important. It defines one of the fundamental objects in
> the API.
>
> I recommend you read up on the discussions on the OCF list on this; this
> has probably been one of the hottest arguments.
Add a translation layer if you like, and submit it in the form of a user space
library service. Or have it be part of your own layer or application. There
is no compelling argument for embedding such a bloatlet to cman proper (which
is already fatter than it should be).
> "How is the kernel component
> configured which paths/IP to use" - ie, the equivalent of ifconfig/route
> for the cluster stack,
There is a config file in /etc and a (userspace) scheme for distributing the
file around the cluster (ccs - cluster configuration system).
How the configuration gets from the config file to kernel is a mystery to me
at the moment, which I will hopefully solve by reading some code later
today ;-)
> Doing this in a wrapper is one answer - in which case we'd have a
> consistent user-space API provided by shared libraries wrapping a
> specific kernel component. This places the boundary in user-space.
I believe that it is almost entirely in user space now, with the recent move
of cman to user space. I have not yet seen the new code, so I don't know the
details (this egg was hatched by Dave and Patrick).
> This seems to be a main point of contention, also applicable to the
> first question about node identifiers: What does the kernel/user-space
> boundary look like, and is this the one we are aiming to clarify?
Very much so.
> Or do we place the boundary in user-space with a specific wrapper around
> a given kernel solution.
Yes. But let's try and have a good, durable kernel solution right from the
start.
> > Since cman has now moved to user space, userspace does not tell the
> > kernel about membership,
>
> That partial sentence already makes no sense.
Partial?
> So how does the kernel
> (DLM in this case) learn about whether a node is assumed to be up or
> down if the membership is in user-space? Right! User-space must tell
> it.
By a message over a socket, as I said. This is a really nice property of
sockets: when cman moved from kernel to user space, (g)dlm was hardly
affected at all.
> For example, with OCFS2 (w/user-space membership, which it doesn't yet
> have, but which they keep telling me is trivial to add, but we're
> busying them with other work right now ;-) it is supposed to work like
> this: When a membership event occurs, user-space transfers this event
> to the kernel by writing to a configfs mount.
Let me go get my airsick bag right now :-)
Let's have no magical filesystems in the core interface please. We can always
add some later on top of a sane base interface, assuming somebody has too
much time on their hands, and that Andrew was busy doing something else, and
Linus left his taste at home that day.
> Likewise, the node ids and comm links the kernel DLM uses with OCFS2
> are configured via that interface.
I am looking forward to flaming that interface should it dare to rear its ugly
head here :-)
> If we could standarize at the kernel/user-space boundary for clustering,
> like we do for syscalls, this would IMHO be cleaner than having
> user-space wrappers.
I don't see anything wrong with wrapping a sane kernel interface with more
stuff to make it more convenient. Right now, the interface is a socket and a
set of messages for the socket. Pretty elegant, if you ask me.
There are bones to pick at the message syntax level of course.
> > Can we have a list of all the reasons that you cannot wrap your heartbeat
> > interface around cman, please?
>
> Any API can be mapped to any other API.
I meant sanely. Let me try again: can we have a list of all the reasons that
you cannot wrap your heartbeat interface around cman _sanely_, please.
> That wasn't the question. I was aiming at the kernel/user-space boundary
> again.
Me too.
> > > ... which is why I asked the above questions: User-space needs to
> > > interface with the kernel to tell it the membership (if the membership
> > > is user-space driven), or retrieve it (if it is kernel driven).
> >
> > Passing things around via sockets is a powerful model.
>
> Passing a socket in to use for communication makes sense. "I want you to
> use this transport when talking to the cluster". However, that begs the
> question whether you're passing in a unicast peer-to-peer socket or a
> multicast one which reaches all of the nodes,
It was multicast last time I looked. I heard mumblings about changing from a
udp-derived protocol to a sctp-derived one, and I do not know if multicast
got lost in the translation. It would be a shame if it did. Patrick?
> and what kind of
> security, ordering, and reliability guarantees that transport needs to
> provide.
Security is practically nonexistent at the moment, we just keep normal users
away from the socket. Ordering is provided by a barrier facility at a higher
level. Delivery is guaranteed and knows about membership changes.
> > Of course, we could always read the patches...
>
> Reading patches is fine for understanding syntax, and spotting some
> nits. I find actual discussion with the developers to be invaluable to
> figure out the semantics and the _intention_ of the code, which takes
> much longer to deduce from the code alone; and you know, just sometimes
> the code doesn't actually reflect the intentions of the programmers who
> wrote it ;-)
Strongly agreed, and this thread is doing very well in that regard. But we
really, really, need people to read the patches as well, especially people
with a strong background in clustering.
Regards,
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
