Welcome all LEBers! Today has been an unfortunate day for many hosts and indeed a shocking eye-opener for anyone using SolusVM to offer VPS’ to the public. Earlier on today the website localhost.re reported on a shocking SolusVM exploit that effects every SolusVM version – the now defunct/unused file centralbackup.php contained multiple blunders including SQL Injection, direct exec()ution of any command, and access to the SolusVM server-side binary which can execute any command. Unfortunately for hosts this was a surprise to say the least, and one of the first to be targetted seems to be RamNode.
In very short time since the original post was made on localhost.re one of our long term members @vld delivered the news to members via LowEndTalk and that was the first time many of the LE* members had heard about the exploit. Many hosts were quick to patch the exploit & SolusVM released an announcement via Email giving hosts clear instructions on how to patch the exploit not long after being contacted. Unfortunately one very reputable host was caught out and barely hours after the post on LowEndTalk their SolusVM index was replaced with an image giving anyone access to download the database and access the admin panel. (http://cdn-static.com/i/AiBpjTIC.png)
RamNode were quick to respond, removing the exploit and blocking access to anything that might be vulnerable – they are now busy dealing with this nightmare and getting clients back online as fast as possible. We must commend Nick for his perseverance in dealing with such an issue swiftly and calmly.
An announcement from RamNode was soon released and it was confirmed that Robert, was behind the initial breach of security at RamNode via the exploit. “As you are all aware, this has been a nightmare for [us]. Robert ran the SolusVM exploit on our control panel early this morning. Someone, him or else, then logged into several nodes and wiped the data.”
Members of LowEndTalk did post findings that correlate with the above statement that Robert was behind the attack/intrusion. Evidence such as IP-matches & even confirmation that the IP was indeed Roberts’ home network (via the welcome page for a HP media server which clearly stated “Robert’s Pictures” with the hostname ‘clarkeone.homeserver.com’) – not especially good news considering Robert’s previously dubious history and not so great reputation in the industry. While Robert has admitted to the initial “testing” of the exploit he still protests his innocence and vehemently denies doing any of the damange.
Whatever the case, we wish every host the best of luck in dealing with the aftermath of this shocking exploit, and commend RamNode for their quick response & level headed handling of the situation.
Thanks to @Zen for writing up the majority of this.
- VPSDime – $7/Month 6GB RAM OpenVZ (Dallas) + Backupsy – $7/Month 250GB KVM (US/UK) - September 2, 2013
- TortoiseLabs – $6.45/Month 512MB RAM Xen VPS in LA, FL, TX, CHI and London - August 25, 2013
- Crissic Solutions – $1.64/Month 256MB RAM OpenVZ VPS in Jacksonville - August 23, 2013
FYI,
The exploit was attempted against us from multiple people but did exactly zilch :) When we moved away from solusvm we straight up nuked the /www/ folder and replaced it with our own setup.
Long live the stallion!
Francisco
Signed up yesturday then this, had less than half a day use.
Just a little perspective.
Now that everyone has all but burnt Robert at the stake, it is worth considering that this exploit appeared on the net to be then immediately broadcast in many locations, not least the home of the child VPS provider and DDoS hive that is Lowendtalk.
If Robert did cause the issues at Ramnode it is likely, or actually definite that he was simply one of a much larger group of people cutting their way through providers trying to get a “hit”, he was lucky as it were and found Ramnode, others are in the same position. I know of at least 4 with varying degrees of repair work required.
Whilst I am not condoning what he did, if he did it, it is easy to focus in an target him, yet from what Nick has said he is can only be sure Robert accessed something rather than did anything. I am sure all you providers can check your logs and see countless others “all of a sudden” waking up and becoming active in the apparent name of “just testing to check everything is ok”.
Pretty sure even if Robert had not done whatever he is supposed to have, someone else would have been “checking” too and the same outcome would have followed.
Let’s not forget the developer provided the exploit, someone on the internet provided the instructions. All it needed once advertised was a willing “checker” and once it was posted on LET well you can’t tell me you did not expect something like this to happen!
As always tends to happen here the real issues/events that led to this are somewhat over looked as long as people have a body to dance around and claim victory against!
come on, you are deflecting, is he your friend?
1) He could have just emailed them….
2) What he did was disruptive and invasive. By society marked as an criminal offense.
Just a little perspective: someone discovered the gunpowder, an other build a gun to and bullets so we good put it to use. I shot someone, but do not blame me….
I am deflecting? You are just looking for an argument I am not willing to give you more like.
Perhaps you are missing my point. The whole issue is really not about Robert, it’s about the situation that existed in the first place that needs to be addressed. And that specifically is why solusvm had what most seem to be stating as a glaringly obvious puncture in their wheel but continued to ride on it?
Whether Robert did it or not makes little difference, why? Because if it had not been him or someone you could identify, you would have no body to beat, so instead you would have been shouting “WTF Solus, shit software!!”.
I certainly don’t blame Ramnode, they appear to be a great provider and are working hard to restore all they can, unluckly for them and the others that got caught. But it’s still not about the person, it’s about the cause.
Very well said, brother !
We call this acarul Paun syndrome, meaning that as long as there is someone to take the blame, the problem itself seems less important.
Someone had to take the fall for this.
Solusgate – a date which will live in infamy
I hope Ramnode customers understand, it’s not Ramnode problem, it’s problem of a SolusVM.
Ramnode, i wish to you be strong, and i hope you will the same as you was some time ago before hack. Goodluck with services, and your quality.
If you look at it, ramnode should be somehow blamed knowing that most if not all other host was able to quickly prevent the exploit. they somehow lack somthing to prevent this from happening when all other host out there was able to protect themselve
Don’t know how I got lucky, but apparently I was on the one (KVM) server that was only down for about fifteen minutes. I really hope all the critical data can be restored properly, as Ramnode has become one of my favorites (not the least because one of their PoPs is only a mile and change from my house).
Apparently a lot of servers are fine. RamNode has been working very hard to fix things. Kudos.
I’m a customer of servercrate since 3 months ago.
and as I recall, they had put they master SolusVM at buyvm and ramnode.
And he got kicked out of BuyVM (http://vpsboard.com/topic/733-ramnode-down/page-2#entry10594 ). He was known via IRC and in the community as a whole to dislike Nick_A and RamNode.
I’ve also created this thread in WHT, to support Nick’s great job:
http://www.webhostingtalk.com/showthread.php?t=1276341
Web Hosting is a 24/7 operation. I guess RamNode were caught sleeping. But they have been really calm and professional in damage control.
1. This is not webhosting
2. 24/7 is at the provider’s discretion, not because some shitty big hosts have support 24/7 doesn’t mean all do….
3. Just fuck off.
I have no real issues with how ramnode handled things. Their updates could have been a bit more timely and frequent, but other than that I think they handled things professionally.
24 hours later still no fix, Considering asking for a refund since just order then bang!
Hi Nick, SolusVM released an announcement via Email giving hosts clear instructions on how to patch the exploit
PLEASE READ THIS INFORMATION CAREFULLY. THIS INFORMATION IS RELEVANT TO ALL VERSIONS OF SOLUSVM, INCLUDING BETA VERSIONS.
A security update has now been released for the Stable and Beta versions of SolusVM. We advise you to make this update as soon as possible.
To run the update you can either do it from within the SolusVM admin area or from CLI on the master server. To preform the update from CLI the commands differ depending on the version of SolusVM you are running.
==================
Stable version:
/scripts/upcp
Beta version:
/scripts/upcp-beta
==================
Once the update is complete you will have the patched system.
We have included the original instructions in this email that were given when the exploit was announced and before we released the patched updates. If you feel the need to remove the originally exploited file after the update you can do the following:
==================
Instructions:
You will need root SSH access to your master server. You are then required to delete the following file:
/usr/local/solusvm/www/centralbackup.php
Example:
rm –f /usr/local/solusvm/www/centralbackup.php
==================
Due to this exploit we are conducting a full audit of the SolusVM client area code. The audit is already underway and any updates, if needed will be released in quick succession.
A full explanation of this exploit will be released in due course. We will also be reviewing the release status of version 1.14 due to the advanced security features it already contains.
Thank you for your continued support and apologies for any inconvenience caused.
Regards,
Soluslabs Security Team
You got to be kidding Nick? Get a refund, go to another provider Ramnode don’t need customers like YOU!
You might want to take this to IRC, Nick might set you up on a restored node.
Fortunately, both of my vps were not affected at ramnode :)
a sad situation, but well, as I can see there, it could have been easily avoided with mod-security, anyway, leason learned for many providers I guess not to just throw a standard setup and start milking money
“We will also be reviewing the release status of version 1.14 due to the advanced security features it already contains.”
I’m wondering how advanced are these security features – considering the “advanced” security in their centralbackup.php file, I’m expecting another flight of the ROFL copter…
Your writing style made me cringe.
Please do not attempt to write in styles you are terribly unfamiliar with.
Generally I don’t ‘report’ on things in my writing, neither do I write professionally. This article was a favour for Humza and took me less than 5 minutes, it was sent to Humza with no real edits (normally if I’m writing an article I will go through it a dozen times and change the general structure until I’m happy) and no idea whether it would be posted or not.
I would appreciate some constructive criticism, but simply telling me that I should not write ‘like this’ is useless to me, how can I take it on board? You aren’t telling me anything I don’t know already, that’s for sure.
Thanks :)
I can confirm, Zen did it really quickly. Sure, the writing may not be the best quality but for literally about 7 minutes it’s great. I was in a rush too so didn’t make too many edits. Just chill, it’s informative, and that’s all it needs to be.
The content in itself is fine, but trying to needlessly view things from a third party’s point of view just made it sound dumb.
That’s my own opinion, which I believe I’m entitled to. It shouldn’t need to be long winded to be informative.
From an English perspective, the sentences are too long – consistently too long. It lacks variation and proper pace. It’s hard to grasp and slightly confusing. There are lots of conjunctions. Doesn’t flow that well.
Aren’t you guys taking an automated spam comment too seriously?
Your writing style on the exploit seems perfectly alright to me.
This is not automated, Wintereise is a well known guy here…
Not good, this is why we are scripting our own control panel built into whmcs rather than using a commercial one, but I am sure solus will get it under control
Sorry to hear about the hosts effected by this, hopefully everything gets sorted
I’m working on a free & opensource panel in codeigniter right now. OVZ, KVM support, two panels, but we might make a module for WHMCS that doesn’t require it, however unsure.
If you look at it, ramnode should be somehow blamed knowing that most if not all other host was able to quickly prevent the exploit. they somehow lack somthing to prevent this from happening when all other host out there was able to protect themselve
Not to say ramnode never said a single sorry, as if he’s not liable!
Stop posting, Remi
Shit happen dude.
LEB and Other Linux Server forums should ban a person and his business who actually doesn’t respect Internet and customer protection. ServerBear already banned their site and business. Robert’s mindset shows he have no guts in business, likes to invade peoples data and life. I wonder! How much safe their (ServerCrate) own customers feels? IMHO! they are naked….
For Nick and RamNode, Thanks for handling so nicely and professionally.
He has been…
6 hours till another solusvm exploits surfaces on the internet.
Bull****.
Book, the fact is that RamNode is not responsible for that incident. The exploit in SolusVM was not their fault – it was a problem with the software itself.
“they somehow lack somthing to prevent this from happening when all other host out there was able to protect themselve”
There are 2 more hosts that suffered from data loss because of the exploit in SolusVM.
I haven’t had time to respond here since we’ve been underwater restoring service and responding to tickets. I appreciate the support we have received in the LEB/LET community. We will continue striving to be the best provider we can for you all.
ChicagoVPS now hacked, thanks to a 0day SolusVM exploit
Please, don’t remove this comment.
5 hours.
Wish you would hurry up and get on with it lol :P
man, your first post was at 11:01, and the next one at 12:31, and you said 6 and 5 hours, r u sure your clock is working right? Maybe try ‘5 hours and 26 minutes until next release’, or well make a website with a counter, that all default installers will refresh every second… etc
I have spoken, they have still not managed to patch it, however admit vulnerabilities. http://blog.soluslabs.com/2013/06/19/security-updates-available-for-all-solusvm-versions/
It should be noted:
SolusVM released a work-around, and then an update, two days ago. Posted to their blog and sent to subscribed customers via email on 6/16. Had those alerts been acted on, many hosts would not be scrambling now. The “work around” was pretty darned simple (remove one file), and only had to be performed on the Master(s) (to off-set the “it takes time to patch…” replies that will obviously come).
Just sayin’.
ChicagoVPS had problems too.
“Around 3am Eastern Standat Time (EST) today, there was a security breach, due to a vulnerability in SolusVM that allowed a command line to be run to dump the ChicagoVPS SolusVM client database and attempt to delete all data from our nodes. Our staff is working tirelessly to get everything back online, along working with SolusVM to address the root issue and no furthur impact is expected.”
I haven’t had a good experience with them at all, and migrated away before this incident.
Yep…I can confirm that. I have two VPSs with them, one is OK, the other is down.
Thankfully, it’s just a stand-by failover server.
So did chicagovps did not address the solusvm exploit yesterday. Wonderful.
did someone now bring down lowendtalk and all forums hosted my http://vanillaforums.org/
*by
its down for me too, I don’t know, maybe it’s ddosed or maybe they got hacked AGAIN
Seems like cluster 1 for vanilla is having problems..May be where LET is hosted. http://status.vanillaforums.com/
well there is one previous community member who was trying to avoid being banned of LET today. i have a strong feeling that he is probably the cause.
I have a strong felling it was Robert, the one who supposedly hacked ramnode yesterday.
It definitely was not Robert. I can confirm that 100%.
I think you overestimate him. He just ran a published exploit. It could have been done by script kiddie.
*any script kiddie
Well my Chicago VPS has now been down for at least 15 hours. Sigh.
It’s not Robert, but the same guy who was bragging around that he hacked LET.
Some idiots like BradND – NodeDeploy obviously support those criminal acts.
@Liam, Is Robert in jail now? Is that how you know? lmao
I think the focus should be on improving and mitigating security issues and not finding someone to target and focus on. Not sure I’d have included the name of the user that the exploit ran on since it’s hard to say at the time if it was said person (and even still you can’t know 100%). I have no idea who he is myself and fortunately no VPS of mine went down this time. I wonder how many VPS providers are still vulnerable given how there are so many out there.
ChicagoVPS is still down, control panels and servers. Hopefully they will fix this soon I’m hosting servers for a gaming league that has official match time tomorrow night. And 8 (4×2) teams are relying on my servers. :S
Greeting script kiddies,
I have stood by, watched, used & abused, disclosed information on SolusVM. I’m not stopping there however; WHMCS you’re next. SolusVM admitted to security issues, after much disbelief, I had disclosed vulnerabilities. I will be testing v1.13.05 (latest) shortly on SolusVM.
Let this be a warning;
No one knows how insecure you are until your security is breached.
You forget security is important, you rely on encoders to “keep you safe”, however this is not the case. Anything that can be encrypted, can easily be unencrypted. WHMCS: After a code review of your latest version, PayPal module has 1 exploit, Clientarea Register has 1 exploit.
HOSTBILL SOURCE CODE:
Exploits spotted: 6
Exploits patched: 0
Can someone get this buffoon off LEB.
Someone finds and publishes an exploit. Sites were exploited. Threaten to publish more, and providers panic and shutdown services. Threaten to target even more important stuff, and bask in the joy of seeing more panic.
This clown is most probably someone following the news, and trying to get people to lose their cool. He knows zilch.
I can confirm we spoke with SolusVM directly within minutes of this guy posting under name Lol in previous topic claiming to have more exploits; This is a fake scare tactic used by a VPS host in an attempt to get all others to shut down in meantime. IT WONT WORK BRO. We do directly speak with developers an SolusVM are very much aware we will all run to OnApp should SolusVM fail so they respond pretty fast.
They confirmed the bug reported had been patched an even released another patch for another undisclosed vun, However NOONE has reported to them any zero day vuns as you claimed in previous post.
Noob kiddie trying to scare hosts, We invest alot of money in keeping our clients happy so I think i speak for most vps company’s not just VpsCorner when i say, Noone will jump to your scare tactics which are quite pathetic.
Sure there might be good peeps out there with ability to find/create expliot’s for a range of software However, Those 99% of time report it an its patched, Kiddies who go about threatening to release stuff without alerting Dev’s are clearly just bluffing wannabe’s.
LEB has a duty to list that users IP so we can all make sure we purge the idiot from our database’s, As this is a direct target upon VPS Company’s to have people pull out for a few days.
Love the post Humza though.
Lets dig into WHMCS 5.2.5
http://img692.imageshack.us/img692/3293/4jq2.png
FAIL.
Shitty decoder skills, “decoded HostBill code” is still ionCubed.
admin directory is decoded for hostbill.
Oke, I’m waiting :p
Still hiding in IonCube…
http://img515.imageshack.us/img515/7231/z0e.png
HostBill XSS
XSS… wow actually show something interesting..
I love that people continue to underestimate XSS and have no idea how easy it is to exploit. I know a blackhat crew of carders who trade thousands of dumps a month – 80% of their exploits are XSS. the method they use to get people to execute them are definitely very novel (from what i’ve seen).
We know you find things like this, but wouldnt it be more worth it actually helping the companies out and showing them exploits? Maybe you will get a job, who knows.
Nice job blowing your cover. That image and the image you posted on vpsBoard show the same single Chrome extension.
http://vpsboard.com/topic/777-personal-arguments/?p=11495
http://i.imgur.com/NbMYlal.png
Odd how ramnode was effected yet on there own news page outright denial of any breach just using excuse its offline for security updates due to vun being discovered.
Surely for client reassurance wouldn’t a company be obliged to let client know of the breach esp where sensitive customer details are at risk.
I know if VPSCorner.co.uk was breached we’d announce it, 1 for privacy of clients so they can change password they use on site anywhere else, As yeah it might be hashed but if you get access to configuration then you have the encryption hash also. >.<
Not a personal dig at ramnode at all, Im sure they have there reasons, I.E not to scare clients, But when breached, That kinda goes out window an just admit the breach even if it was little breach esp if it concerned DB being publicly downloadable.
Let me also make all users aware, They was NOT breached just because they often see now vps.server.com on a mass of servers, This is when a FQDN is not provided, In middle of upgrade for SolusVM it does a check.
more info at : http://blog.soluslabs.com/2013/06/19/vps-server-com-what-is-it/
SolusVM Official response to us regarding the false rumours about more problems. : Also just for the record we had no reports. We will always act on reports and i'm sure you know that. Why would we want to ruin what we have? If we have a security problem we will fix it ASAP.
Ramnode e-mailed all customers with the gory details and recommendations to change passwords, particularly to ensure that login/root passwords are changed via ssh rather than solusvm.
So yes, they did push out a notice.
Didn’t know an email was sent out at all >.< sorry for jumping to conclusions. Know admitting the problem isn't easy least ramnode took decision to do it in favour of users security. so Users are made aware. :)
Guess its time to put my foot in my mouth for making opinions without full facts.
Simon is this a joke? We have been in constant communication with clients since the attack happened. Maybe you’re referring to a different RamNode or maybe you just need to do a bit more research before making that kind of statement.
I guess Simon did not noticed how many emails RamNode sent!
@Asim – like! I got one for each active service I have with them, then one for luck!
I have many, many positive things to say about RamNode in all of this. One of them is their excellent communication. Through Twitter, client area notices and mass e-mail we’ve all been kept very well informed indeed. It’s a crisis like this that tests the metal of a host, and I’m more convinced than ever that RamNode is solid.
Sounds like a little self-promotion at others expense there.
Ah Least ramnode pushed out notice, Offically on there website was unsure they was actually effected by time read all this, Posts on news just claims they turned off while patching not they was breached.
An No dig at ramnode i meant purely from what i seen on there website/twitter or all official channels, As clients often do not check emails daily but are lurking around facebook an twitter.
Didn’t know a email went out, I just gave my opinion based on what i seen via other channels regardless of email. Least client was notified :) Good job ramnode :)
Sorry nick_a if it did sound like a dig at ramnode its not at all, I ant a customer so i was unaware of the emails sent out. So i apologize for jumping to conclusions without full facts i just assumed the public posts on ramnode would also confirm it rather than suggest it was taken down to prevent a breach when in fact it was due to a breach, Slightly confusing as leads users to think they dont need to change passwords as was never breached. Although Emails sent if they confirm breach would sort that confusion. Sorry >.< I shouldn't comment without knowing full detail lol.
Still think its stupid why any user targets company's aiming to provide cheap services. Seems very counter-productive, I bet who ever did it also buys budget VPS services. Oh the irony. I bet it was someone who tried to sell bug to SolusVM for mad amount of money when turned down they go on a revenge mission post it knowing full well, Nublets skids will go use the info to effect people's work an lively hood, Regardless if ramnode is big or not Plays no part, its still effecting people who work there an damaging customer trust.
We posted on Twitter all day and our News page on our website:
http://gyazo.com/1a996ee2fa505c83fedb34c5a35b046f
Perhaps you didn’t look hard enough. Oh well.
Ah yes I totally missed that post on june 16th >.< Now i feel retarded.
Mainly as i stopped reading just as got to KVM topic didn't notice SolusVM issue continued below. Yeah ok you can kick me in the nuts now for my totally wrong opinion.
Oh look, solusvm whmcs module now vulernable. Ironic, EH?
Posting AFTER stuff is patched, Is ironic.
Notice how you can never post a bug before its already been known for hours LMAO.
Failzor.
As much as it might seem fun to yourself to carry out such actions, using expliots already published by others to try and expliot hosts or scare hosts, it isn’t fun for those hosts impacted and those who have to have techs on hand to keep monitoring for problems with software such as WHMCS/SolusVM or any other range of products have been extremely inconvenienced.
What is more ironic is that I can bet that the person doing this is scaring people yet I also bet that he is a client of many of the and that he couldn’t survive without a BUDGET VPS service yet he likes to go around impacting them. I find that extremely ironic.
What is even more ironic is the fact its clear the expliots are found by others and you’re just jumping on the bangwagon because anyone with a average IQ would know possesing such information is valuable and approaching vendors would be rewarding, although i assume any reports are like “OMG VULN STUPID SOFTWARE IM GONNA PUBLISH PUBLIC DESTORY U” when that isn’t very productive, you hate so much on the software but rather than assisting it and fixing its problems you just want to out them to have everyone suffer.
Users demand fair prices but then hate on the fact companys have to resort to WHMCS/SOlusVM and other products to be able to provide such cheap products.The days of moving things to other services and the cost to do such things is no longer viable for any company thus these kind of actions gain nothing but annoyance at your stupidity.
Any person finding expliots would be simply alerting the vendor then alerting public on the issue NOT winding people up. One of the expliots was posted by Localhost.re or where-ever it was published first and a single user jumped on the bangwagon to try scare hosts.
Have you noticed one thing, Ramnode for example has been affected massively by the expliots, yet recovered and patched and still continue to trade so it just goes to show that all hosts can be threatened or explioted and will patch and then continue to run. Hosts are not going to stop trading because of a ludacris post by someone who posts as Oh No with 0 facts provided just comments to scare?
Ramnode plus others had the correct response, patch the problem, recover, find out who carried out the exploit and remove, then continue providing services at the same prices. I personally jumped the gun with Ramnodes response and I didn’t follow any others who was effected, although I guess there was a few.I don’t see anyone pulling out of VPS hosting purely due to these problems. All companys face problems in software security, If using WHMCS/SolusVM/OnApp or a custom solution.
Just because the software such as Solus and WHMCS is globally used it doesn’t mean its more expliotable or less; No software is 100% secure neither is any server and I very much doubt that any host would claim it is. Either way though, no company wants to be faced with threats via emails or ludacris posts on forums such as posts you make here and all to achieve what? Force budget providers to stop trading? Or is this just a VPS company with 0 Sales trying to knock everyone out of the game.
What’s up, Can’t handle fair competition so you have to resort to underhand scare tactics so people and slow orders and things decline, I think some would agree with me, Noone is going to let any such threats impact business on any level. Measures are always taken to protect from problems but nothing is perfect. Hourly offsite backups of SQL+Files does minimize the problem an expliot may have on a company though.
Why does ServerCrate still exist?
Who the fuck is Robert?