Chris Wright wrote:
Hi Chris, thank you for your comments. I've tried to answer as much as
I can - hopefully I found all your questions.
+ guest operating systems. In the future, we envision that additional
+ higher level abstractions will be added as an adjunct to the
+ low-level API. These higher level abstractions will target large
+ bulk operations such as creation, and destruction of address spaces,
+ context switches, thread creation and control.
This is an area where in the past VMI hasn't been well-suited to support
Xen. It's the higher level abstractions which make the performance
story of paravirt compelling. I haven't made it through the whole
patchset yet, but the bits you mention above as work to be done are
certainly important to good performance.
For example, multicalls, which we support, and batched page table
operations, which we support, and vendor designed virtual devices, which
we support. What is unclear to me is why you need to keep pushing
higher up the stack to get more performance. If you could have any
higher level hypercall you wanted, what would it be? Most people say -
fork() / exec(). But why? You've just radically changed the way the
guest must operate its MMU, and you've radically constrained the way
page tables and memory management structures must be layed out by
putting a ton of commonality in their infrastructure that is shared by
the hypervisor and the kernel. You've likely vastly complicated the
design of a virtualized kernel that still runs on native hardware. But
what can you truly gain, that you can not gain from a simpler, less
complicated interface that just says -
Ok, I'm about to update a whole bunch of pages tables.
Ok, I'm done and I might want to use them now. Please make sure the
hardware TLB will be in sync.
Pushing up the stack with a higher level API is a serious consideration,
but only if you can show serious results from it. I'm not convinced
that you can actually hone in on anything /that isn't already a
performance problem on native kernels/. Consider, for example, that we
don't actually support remote TLB shootdown IPIs via VMI calls. Why is
this a performance problem? Well, very likely, those IPI shootdowns are
going to be synchronous. And if you don't co-schedule the CPUs in your
virtual machine, you might just have issued synchronous IPIs to VCPUs
that aren't even running. A serious performance problem.
Is it? Or is it really, just another case where the _native_ kernel can
be even more clever, and avoid doing those IPI shootdowns in the
firstplace? I've watched IPI shootdown in Linux get drastically better
in the 2.6 series of kernels, and see (anecdotal number quoting) maybe 4
or 5 of them in the course of a kernel compile. There is no longer a
giant performance boon to be gained here.
Similarly, you can almost argue the same thing with spinlocks - if you
really are seeing performance issues because of the wakeup of a
descheduled remote VPU, maybe you really need to think about moving that
lock off a hot path or using a better, lock free synchronization method.
I'm not arguing against these features - in fact, I think they can be
done in a way that doesn't intrude too much inside of the kernel. After
all, locks and IPIs tend to be part of the lower layer architecture
anyways. And they definitely do win back some of the background noise
introduced by virtualization. But if you decide to make the interface
more complicated, you really need to have an accurate measure of exactly
what you can gain by it to justify that complexity.
Personally, I'm all for making lock primitives and shootdowns an
_optional_ extension to the interface. As with many other relatively
straightforward and non-intrusive changes. I know some of you will
disagree with me, but I think a lot of what is being referred to as
"higher level" paravirtualization is really an attempt to solve
pre-existing problems in the performance of the underlying system.
There are advanced and useful things you can do with higher level
paravirtualization, but I am not convinced at all that incredible
performance gain is one of them.
We do not want an interface which slows down the pace. We work with
source and drop cruft as quickly as possible (referring to internal
changes, not user-visible ABI changes here). Making changes that
require a new guest for some significant performance gain is perfectly
reasonable. What we want to avoid is making changes that require a
new guest to simply boot. This is akin to rev'ing hardware w/out any
backwards compatibility. This goal doesn't require VMI and ROMs, but
I agree it requires clear interface definitions.
This is why we provide the minor / major interface numbers. Bump the
minor number, you get a new feature. Bump the required minor version in
the guest when it relies on that feature. Bump the major number when
you break compatibility. More on this below.
+ VMI_DeliverInterrupts (For future debate)
+
+ Enable and deliver any pending interrupts. This would remove
+ the implicit delivery semantic from the SetInterruptMask and
+ EnableInterrupts calls.
How do you keep forwards and backwards compat here? Guest that's coded
to do simple implicit version would never get interrupts delivered on
newer ROM?
This isn't part of the interface. If it were to be included, you could
do two things - bump the minor version, and add non-delivery semantic
enable and restore interrupt calls, or bump the major version and drop
the delivery semantic from the originals.
I agree this is pretty clumsy. Expect to see more discussion about
using annotations to expand the interface without breaking binary
compatibility, as well as providing more advanced feature control. I
wanted to integrate more advanced feature control / probing into this
version of the VMI, but there are so many possible ways to do it that it
would be much nicer to get feedback from the community on what is the
best interface.
+ CPU CONTROL CALLS
+
+ These calls encapsulate the set of privileged instructions used to
+ manipulate the CPU control state. These instructions are all properly
+ virtualizable using trap and emulate, but for performance reasons, a
+ direct call may be more efficient. With hardware virtualization
+ capabilities, many of these calls can be left as IDENT translations, that
+ is, inline implementations of the native instructions, which are not
+ rewritten by the hypervisor. Some of these calls are performance critical
+ during context switch paths, and some are not, but they are all included
+ for completeness, with the exceptions of the obsoleted LMSW and SMSW
+ instructions.
Included just for completeness can be beginning of API bloat.
The design impact of this bloat is zero - if you don't want to implement
virtual methods for, say, debug register access - then you don't need to
do anything. You trap and emulate by default. If on the other hand,
you do want to hook them, you are welcome to. The hypervisor is free to
choose the design costs that are appropriate for their usage scenarios,
as is the kernel - it's not in the spec, but certainly is open for
debate that certain classes of instructions such as these need not even
be converted to VMI calls. We did implement all of these in Linux for
performance and symmetry.
clts, setcr0, readcr0 are interrelated for typical use. is it expected
the hypervisor uses consitent regsister (either native or shadowed)
here, or is it meant to be undefined?
CLTS allows the elimination of an extra GetCR0 call, and they all
operate on the same (shadowed) register.
Many of these will look the same on x86-64, but the API is not
64-bit clean so has to be duplicated.
Yes, register pressure forces the PAE API to be slightly different from
the long mode API. But long mode has different register calling
conventions anyway, so it is not a big deal. The important thing is,
once the MMU mess is sorted out, the same interface can be used from C
code for both platforms, and the details about which lock primitives are
used can be hidden. The cost of which lock primitives to use differs on
32-bit and 64-bit platforms, across vendor, and the style of the
hypervisor implementation (direct / writable / shadowed page tables).
+ 85) VMI_SetDeferredMode
Is this the batching, multi-call analog?
Yes. This interface needs to be documented in a much better fashion.
But the idea is that VMI calls are mapped into Xen multicalls by
allowing deferred completion of certain classes of operations. That
same mode of deferred operation is used to batch PTE updates in our
implementation (although Xen uses writable page tables now, this used to
provide the same support facility in Xen as well). To complement this,
there is an explicit flush - and it turns out this maps very nicely,
getting rid of a lot of the XenoLinux changes around mmu_context.h.
+
+ VMI_CYCLES 64 bit unsigned integer
+ VMI_NANOSECS 64 bit unsigned integer
All caps typedefs are not very popular w.r.t. CodingStyle.
We know this. This is not a Linux interface. This is the API
documentation, meant to be considerably different in style. Where this
ugliness has crept into our Linux patches, I have been steadily removing
it and making them look nicer. But the vast difference in the style of
the doc is to avoid namespace collision.
+ #define VMICALL __attribute__((regparm(3)))
I understand it's for ABI documentation, but in Linux it's FASTCALL.
Actually, FASTCALL is regparm(2), I think.
Cheers,
Zach
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[Index of Archives]
[Kernel Newbies]
[Netfilter]
[Bugtraq]
[Photo]
[Stuff]
[Gimp]
[Yosemite News]
[MIPS Linux]
[ARM Linux]
[Linux Security]
[Linux RAID]
[Video 4 Linux]
[Linux for the blind]
[Linux Resources]
