Alright. We all know OpenVZ VPS are generally oversold in either memory, disk space or bandwidth (or all of them). Overselling memory and disk space is easy. You just vzctl create a new virtual environment regardless whether there is sufficient memory or disk space. Piece of cake. But a sure fire-way to anger your clients if overdone.
That too is also one of many arguments against OpenVZ, especially when it is compared to Xen. “OpenVZ got oversold resources. Xen VPS has dedicated memory, blah blah blah.” Well. Not entirely true.
2 weeks ago I got a cheap VPS from a provider in Asheville NC. Someone who should probably remain anonymous here :) It is a Xen VPS running Ubuntu 10.04 and 64bit Linux 2.6.32 kernel. 512MB of memory, 25GB of disk space and more bandwidth than I ever need. Pretty good price too. So I logged in and check how much memory it got…
# free
total used free shared buffers cached
Mem: 543776 535360 8416 0 66516 130504
-/+ buffers/cache: 338340 205436
Swap: 1048568 136 1048432
Hmm. 330MB of memory already used for my brand new Xen VPS. Let’s see what processes are running.
# ps aux ... root 204 0.0 0.1 17028 780 ? S Oct27 0:00 upstart-udev-bridge --daemon 102 356 0.0 0.1 23548 1080 ? Ss Oct27 0:00 dbus-daemon --system --fork root 431 0.0 0.1 21068 788 ? Ss Oct27 0:00 cron root 3110 0.0 0.5 253832 2992 ? Sl Oct28 0:00 /usr/sbin/console-kit-daemon --no-daemon root 11037 0.0 0.1 49256 1012 ? Ss Oct28 0:00 /usr/sbin/sshd root 15427 0.0 0.1 12520 772 ? S Oct28 0:00 /usr/sbin/syslogd --no-forward root 31231 0.0 0.0 16748 436 ? S<s Nov12 0:00 udevd --daemon root 5716 0.0 0.6 79100 3772 ? Ss 01:02 0:00 sshd: root@pts/0 root 5731 0.0 0.3 19400 2148 pts/0 Ss 01:02 0:00 -bash root 5782 0.0 0.1 6072 724 ? Ss 01:08 0:00 /sbin/getty -8 38400 hvc0 root 5783 0.0 0.2 15248 1172 pts/0 R+ 01:08 0:00 ps --sort=start_time uax
Yes. That’s it — there is nothing memory intensive running on the box. SSH server, syslogd, cron and that’s about it. So where did my 330MB of used memory disappear into?
Before digging any further, here is the fact. You can most definitely oversell memory on Xen VPS. Something that is well known for years, although it is not something Xen providers want to talk about. It uses a technique called ballooning.
Basically a special Linux kernel driver is installed on your system — the “balloon driver”. When dom0 (the Xen server/hypervisor) needs more memory, and wishes to claim some from the guest VPS (domU), it asks the guest VPS’s balloon driver to inflate itself — by asking its Linux kernel for some memory. Kernel memory allocation will be requested from the available memory for that VPS, and cannot be paged out to swap. Once the balloon driver consumes the memory, it then passes to dom0/hypervisor to be used elsewhere (creating a new VPS for example). So the amount of your VPS’s “total memory” will stay the same, but there will be a big increase in “used memory”, as a big chunk has now been used by the balloon driver inside the kernel, and possibly now a part of another VPS. A user-land daemon “xenballoond” (a bash script actually) is also available to allow dynamic ballooning, although I did not see that in my VPS.
I guess it might explain why I have 330MB of memory used, while I only have a very small number of processes running.
I found this file in procfs interesting:
# cat /proc/xen/balloon Current allocation: 524288 kB Requested target: 524288 kB Minimum target: 173056 kB Maximum target: 532480 kB Low-mem balloon: 8192 kB High-mem balloon: 0 kB Driver pages: 224 kB
I can’t seem to be able to find sufficient document on explaining what those value mean. It looks like my VPS has requested 512MB of memory (Request target), and has currently allocated so (Current allocation). However at the same time it also has “Minimum target” — the minimum amount of memory reserved for this VPS to prevent it from FUBAR when the balloon requested too much — set to 169MB. Would that mean that is how much memory that is really guaranteed to my VPS?
It’s certainly something that I am not familiar with and maybe some providers can enlighten us. However the conclusion remains the same. Overselling on Xen is certainly possible.
Related Posts:
- 5 Reasons Why You Want a Low End Box - May 26, 2021
- Dead Pool January 2012 - February 2, 2012
- exit(0); - January 19, 2012
Mine don’t have /proc/xen/balloon, in fact /proc/xen dir is empty. Good or bad?
LEA thanks for information,
but my US vps (1 GB XEN, $6) dont have /proc/xen/balloon, any other way ?
Mine too, but I think the equivalent are:
/sys/devices/system/xen_memory/xen_memory0/info/current_kb
/sys/devices/system/xen_memory/xen_memory0/target_kb
/sys/devices/system/xen_memory/xen_memory0/info/{low_kb,high_kb,driver_kb}
don’t know where Minimum target and Maximum target.
circus, i dont have them on my vps.
I only found balloon.
/lib/modules/2.6.18-194.17.1.el5/kernel/drivers/xenpv_hvm/balloon
/lib/modules/2.6.18-194.17.1.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko
/lib/modules/2.6.18-194.17.4.el5/kernel/drivers/xenpv_hvm/balloon
/lib/modules/2.6.18-194.17.4.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko
/lib/modules/2.6.18-194.el5/kernel/drivers/xenpv_hvm/balloon
/lib/modules/2.6.18-194.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko
maybe because im on xen hvm ?
I guess so, at least based on this blog http://ian.blenke.com/xen/3.0/limitations CMIIW
so that’s why hvm is a bit more expensive :D
I have read [1] before :)
1. http://ian.blenke.com/xen/3.0/limitations
No matter how good a Xen VPS is, it is still a shared system. It depends on how good the Geek behind the keyboard (I wouldn’t say the man behind the gun, heh heh heh) is. So, to some extent it is possible.
This can help LEA? My file is really different xD
We need to research more :P
$ cat /proc/xen/balloon
Current allocation: 524288 kB
Requested target: 524288 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 220 kB
Xen hard limit: ??? kB
Interesting. Some values to compare from my Xen VPSs:
e:~# cat /proc/xen/balloonCurrent allocation: 131072 kB
Requested target: 131072 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 136 kB
Xen hard limit: ??? kB
n:~# cat /proc/xen/balloonDriver pages: 224 kB
s:~# cat /proc/xen/balloonDriver pages: 172 kB
w:~# cat /proc/xen/balloonDriver pages: 164 kB
All omitted values on the latter VPSs are exactly the same as the first. These are all from the same provider.
The “Driver pages” value changes a lot, just try using “cat” a lot of times and you will see :P
This is what I’ve got:
# cat /proc/xen/balloon
Current allocation: 131072 kB
Requested target: 131072 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 0 kB
Xen hard limit: ??? kB
@LEA
Can you maybe post ‘top’ output sorted by memory usage (Shift-M)?
It will also show RES/SHR/VIRT usage, along with buffers, cached and free all on one screen
mine:
[root@en ~]#cat /proc/xen/balloon
Current allocation: 524288 kB
Requested target: 524288 kB
Low-mem balloon: 5376 kB
High-mem balloon: 0 kB
Driver pages: 248 kB
Xen hard limit: ??? kB
nothing suspicious about memory usage
If you provider allow you to build your own kernel you’re ok as you can build it with balloon disabled.
I think some of us are talking about 2Host. I’m running into the same confusion about figuring out how the ballooning works, and to what degree it’s impacting my VPS.
2HOST totally sux, avoid them.
i have been looking at the output of /proc/xen/balloon and i think i have made some sense of it.
here is the output from one one of my vps’s
Current allocation: 262144 kBRequested target: 262144 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 176 kB
Xen hard limit: ??? kB
and here is the output from free
total used free shared buffers cachedMem: 262144 258760 3384 0 110120 54684
-/+ buffers/cache: 93956 168188
Swap: 393208 432 392776
at first glance it seems current allocation and total allocation are your VPS memory limits, its what its trying to reserve for your VPS.
to me it would seem that lowmem balloon is how much it can take at MOST with ballon (8mb in this case, not to bad)
highmem is how much it will take if I’m using all the memory (none)
driver pages would then be how much it has balloned into my vps (176kb) or rather how much memory the balloon is taking up
now i have absolutely no documentation to back this up but it seems to match over all my VPS’s so tell me what you think of it.
Here’s mine…
————————————————————————–#free
total used free shared buffers cached
Mem: 524288 394044 130244 0 34236 106712
-/+ buffers/cache: 253096 271192
Swap: 524280 386124 138156
————————————————————————–
#cat /proc/xen/balloon
Current allocation: 524288 kB
Requested target: 524288 kB
Low-mem balloon: 5376 kB
High-mem balloon: 0 kB
Driver pages: 224 kB
Xen hard limit: ??? kB
————————————————————————–
Fact is on SolusVM won’t let you create more DomUs if there are no available memory and disk space in the host node…
Good to know this, thank you
Thanks Joe
I agree,
So if you have an provider who uses SolusVM it’s really unlikely that you are on an oversold node.
yeah in terms of RAM and disk space, bandwidth and i/o can still be oversold though.
Joe,
poor i/o performance is actually caused by what?
Dirty neighbors…:P
Xen has a better io credit scheduler than OpenVZ so you’re less likely to experience io issues on xen vps than its openvz counterpart.
Any explaination on this:
xen:~# cat /proc/xen/balloon
Current allocation: 524288 kB
Requested target: 524288 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 188 kB
Xen hard limit: ??? kB
That ram used on Cache can be reclaimed easily on XEN:
echo 3 > /proc/sys/vm/drop_caches
then you will see how much really is oversold when you issue ‘free’ command.
Nice write up!
this is on EC2 micro:
root@ec2:~# cat /proc/xen/balloon
Current allocation: 629760 kB
Requested target: 629760 kB
Minimum target: 186240 kB
Maximum target: 637952 kB
Low-mem balloon: 8192 kB
High-mem balloon: 0 kB
Driver pages: 252 kB