LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

How to Tell Your Xen VPS is Overselling Memory?

Alright. We all know OpenVZ VPS are generally oversold in either memory, disk space or bandwidth (or all of them). Overselling memory and disk space is easy. You just vzctl create a new virtual environment regardless whether there is sufficient memory or disk space. Piece of cake. But a sure fire-way to anger your clients if overdone.

That too is also one of many arguments against OpenVZ, especially when it is compared to Xen. “OpenVZ got oversold resources. Xen VPS has dedicated memory, blah blah blah.” Well. Not entirely true.

2 weeks ago I got a cheap VPS from a provider in Asheville NC. Someone who should probably remain anonymous here :) It is a Xen VPS running Ubuntu 10.04 and 64bit Linux 2.6.32 kernel. 512MB of memory, 25GB of disk space and more bandwidth than I ever need. Pretty good price too. So I logged in and check how much memory it got…

# free
             total       used       free     shared    buffers     cached
Mem:        543776     535360       8416          0      66516     130504
-/+ buffers/cache:     338340     205436
Swap:      1048568        136    1048432

Hmm. 330MB of memory already used for my brand new Xen VPS. Let’s see what processes are running.

# ps aux
...

root       204  0.0  0.1  17028   780 ?        S    Oct27   0:00 upstart-udev-bridge --daemon
102        356  0.0  0.1  23548  1080 ?        Ss   Oct27   0:00 dbus-daemon --system --fork
root       431  0.0  0.1  21068   788 ?        Ss   Oct27   0:00 cron
root      3110  0.0  0.5 253832  2992 ?        Sl   Oct28   0:00 /usr/sbin/console-kit-daemon --no-daemon
root     11037  0.0  0.1  49256  1012 ?        Ss   Oct28   0:00 /usr/sbin/sshd
root     15427  0.0  0.1  12520   772 ?        S    Oct28   0:00 /usr/sbin/syslogd --no-forward
root     31231  0.0  0.0  16748   436 ?        S<s  Nov12   0:00 udevd --daemon
root      5716  0.0  0.6  79100  3772 ?        Ss   01:02   0:00 sshd: root@pts/0
root      5731  0.0  0.3  19400  2148 pts/0    Ss   01:02   0:00 -bash
root      5782  0.0  0.1   6072   724 ?        Ss   01:08   0:00 /sbin/getty -8 38400 hvc0
root      5783  0.0  0.2  15248  1172 pts/0    R+   01:08   0:00 ps --sort=start_time uax

Yes. That’s it — there is nothing memory intensive running on the box. SSH server, syslogd, cron and that’s about it. So where did my 330MB of used memory disappear into?

Before digging any further, here is the fact. You can most definitely oversell memory on Xen VPS. Something that is well known for years, although it is not something Xen providers want to talk about. It uses a technique called ballooning.

Basically a special Linux kernel driver is installed on your system — the “balloon driver”. When dom0 (the Xen server/hypervisor) needs more memory, and wishes to claim some from the guest VPS (domU), it asks the guest VPS’s balloon driver to inflate itself — by asking its Linux kernel for some memory. Kernel memory allocation will be requested from the available memory for that VPS, and cannot be paged out to swap. Once the balloon driver consumes the memory, it then passes to dom0/hypervisor to be used elsewhere (creating a new VPS for example). So the amount of your VPS’s “total memory” will stay the same, but there will be a big increase in “used memory”, as a big chunk has now been used by the balloon driver inside the kernel, and possibly now a part of another VPS. A user-land daemon “xenballoond” (a bash script actually) is also available to allow dynamic ballooning, although I did not see that in my VPS.

I guess it might explain why I have 330MB of memory used, while I only have a very small number of processes running.

I found this file in procfs interesting:

# cat /proc/xen/balloon
Current allocation:   524288 kB
Requested target:     524288 kB
Minimum target:       173056 kB
Maximum target:       532480 kB
Low-mem balloon:        8192 kB
High-mem balloon:          0 kB
Driver pages:            224 kB

I can’t seem to be able to find sufficient document on explaining what those value mean. It looks like my VPS has requested 512MB of memory (Request target), and has currently allocated so (Current allocation). However at the same time it also has “Minimum target” — the minimum amount of memory reserved for this VPS to prevent it from FUBAR when the balloon requested too much — set to 169MB. Would that mean that is how much memory that is really guaranteed to my VPS?

It’s certainly something that I am not familiar with and maybe some providers can enlighten us. However the conclusion remains the same. Overselling on Xen is certainly possible.

LEA
Latest posts by LEA (see all)

28 Comments

  1. circus:

    Mine don’t have /proc/xen/balloon, in fact /proc/xen dir is empty. Good or bad?

    November 13, 2010 @ 2:58 pm | Reply
  2. tommy:

    LEA thanks for information,
    but my US vps (1 GB XEN, $6) dont have /proc/xen/balloon, any other way ?

    November 13, 2010 @ 3:15 pm | Reply
    • circus:

      Mine too, but I think the equivalent are:
      /sys/devices/system/xen_memory/xen_memory0/info/current_kb
      /sys/devices/system/xen_memory/xen_memory0/target_kb
      /sys/devices/system/xen_memory/xen_memory0/info/{low_kb,high_kb,driver_kb}

      don’t know where Minimum target and Maximum target.

      November 13, 2010 @ 3:40 pm | Reply
      • tommy:

        circus, i dont have them on my vps.
        I only found balloon.

        /lib/modules/2.6.18-194.17.1.el5/kernel/drivers/xenpv_hvm/balloon
        /lib/modules/2.6.18-194.17.1.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko
        /lib/modules/2.6.18-194.17.4.el5/kernel/drivers/xenpv_hvm/balloon
        /lib/modules/2.6.18-194.17.4.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko
        /lib/modules/2.6.18-194.el5/kernel/drivers/xenpv_hvm/balloon
        /lib/modules/2.6.18-194.el5/kernel/drivers/xenpv_hvm/balloon/xen-balloon.ko

        maybe because im on xen hvm ?

        November 13, 2010 @ 3:48 pm | Reply
  3. No matter how good a Xen VPS is, it is still a shared system. It depends on how good the Geek behind the keyboard (I wouldn’t say the man behind the gun, heh heh heh) is. So, to some extent it is possible.

    November 13, 2010 @ 3:36 pm | Reply
  4. Yomero:

    This can help LEA? My file is really different xD

    We need to research more :P

    $ cat /proc/xen/balloon

    Current allocation: 524288 kB
    Requested target: 524288 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 220 kB
    Xen hard limit: ??? kB

    November 13, 2010 @ 4:14 pm | Reply
  5. InsDel:

    Interesting. Some values to compare from my Xen VPSs:
    e:~# cat /proc/xen/balloon
    Current allocation: 131072 kB
    Requested target: 131072 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 136 kB
    Xen hard limit: ??? kB

    n:~# cat /proc/xen/balloon
    Driver pages: 224 kB

    s:~# cat /proc/xen/balloon
    Driver pages: 172 kB

    w:~# cat /proc/xen/balloon
    Driver pages: 164 kB

    All omitted values on the latter VPSs are exactly the same as the first. These are all from the same provider.

    November 13, 2010 @ 4:20 pm | Reply
    • Yomero:

      The “Driver pages” value changes a lot, just try using “cat” a lot of times and you will see :P

      November 13, 2010 @ 4:41 pm | Reply
  6. This is what I’ve got:

    # cat /proc/xen/balloon
    Current allocation: 131072 kB
    Requested target: 131072 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 0 kB
    Xen hard limit: ??? kB

    November 13, 2010 @ 4:34 pm | Reply
  7. rm:

    @LEA
    Can you maybe post ‘top’ output sorted by memory usage (Shift-M)?
    It will also show RES/SHR/VIRT usage, along with buffers, cached and free all on one screen

    November 13, 2010 @ 4:36 pm | Reply
  8. mine:

    [root@en ~]#cat /proc/xen/balloon
    Current allocation: 524288 kB
    Requested target: 524288 kB
    Low-mem balloon: 5376 kB
    High-mem balloon: 0 kB
    Driver pages: 248 kB
    Xen hard limit: ??? kB

    nothing suspicious about memory usage

    November 13, 2010 @ 10:40 pm | Reply
  9. If you provider allow you to build your own kernel you’re ok as you can build it with balloon disabled.

    November 13, 2010 @ 11:16 pm | Reply
  10. CheapVPS:

    I think some of us are talking about 2Host. I’m running into the same confusion about figuring out how the ballooning works, and to what degree it’s impacting my VPS.

    November 13, 2010 @ 11:36 pm | Reply
  11. i have been looking at the output of /proc/xen/balloon and i think i have made some sense of it.

    here is the output from one one of my vps’s
    Current allocation: 262144 kB
    Requested target: 262144 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 176 kB
    Xen hard limit: ??? kB

    and here is the output from free
    total used free shared buffers cached
    Mem: 262144 258760 3384 0 110120 54684
    -/+ buffers/cache: 93956 168188
    Swap: 393208 432 392776

    at first glance it seems current allocation and total allocation are your VPS memory limits, its what its trying to reserve for your VPS.

    to me it would seem that lowmem balloon is how much it can take at MOST with ballon (8mb in this case, not to bad)

    highmem is how much it will take if I’m using all the memory (none)

    driver pages would then be how much it has balloned into my vps (176kb) or rather how much memory the balloon is taking up

    now i have absolutely no documentation to back this up but it seems to match over all my VPS’s so tell me what you think of it.

    November 14, 2010 @ 12:08 am | Reply
  12. Raniel:

    Here’s mine…

    ————————————————————————–#free
    total used free shared buffers cached
    Mem: 524288 394044 130244 0 34236 106712
    -/+ buffers/cache: 253096 271192
    Swap: 524280 386124 138156

    ————————————————————————–

    #cat /proc/xen/balloon
    Current allocation: 524288 kB
    Requested target: 524288 kB
    Low-mem balloon: 5376 kB
    High-mem balloon: 0 kB
    Driver pages: 224 kB
    Xen hard limit: ??? kB

    ————————————————————————–

    November 14, 2010 @ 2:29 am | Reply
  13. Fact is on SolusVM won’t let you create more DomUs if there are no available memory and disk space in the host node…

    November 14, 2010 @ 9:14 am | Reply
    • mina:

      Good to know this, thank you

      November 14, 2010 @ 9:57 am | Reply
    • ibk:

      Thanks Joe

      November 14, 2010 @ 1:30 pm | Reply
    • Richard:

      I agree,

      So if you have an provider who uses SolusVM it’s really unlikely that you are on an oversold node.

      November 15, 2010 @ 4:43 pm | Reply
      • yeah in terms of RAM and disk space, bandwidth and i/o can still be oversold though.

        November 16, 2010 @ 12:31 am | Reply
        • Joe,

          poor i/o performance is actually caused by what?

          November 16, 2010 @ 2:59 am | Reply
        • Dirty neighbors…:P

          Xen has a better io credit scheduler than OpenVZ so you’re less likely to experience io issues on xen vps than its openvz counterpart.

          November 17, 2010 @ 12:18 pm | Reply
  14. Hanh:

    Any explaination on this:

    xen:~# cat /proc/xen/balloon
    Current allocation: 524288 kB
    Requested target: 524288 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 188 kB
    Xen hard limit: ??? kB

    November 17, 2010 @ 11:10 am | Reply
  15. That ram used on Cache can be reclaimed easily on XEN:

    echo 3 > /proc/sys/vm/drop_caches

    then you will see how much really is oversold when you issue ‘free’ command.

    Nice write up!

    November 17, 2010 @ 1:22 pm | Reply
  16. this is on EC2 micro:


    root@ec2:~# cat /proc/xen/balloon
    Current allocation: 629760 kB
    Requested target: 629760 kB
    Minimum target: 186240 kB
    Maximum target: 637952 kB
    Low-mem balloon: 8192 kB
    High-mem balloon: 0 kB
    Driver pages: 252 kB

    November 17, 2010 @ 1:23 pm | Reply

Leave a Reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *