Who killed Server Rabbit? Finale

The root of the cause was right there. How could I had missed it? Both the servers were hardened and rebooted on different dates due to the Windows Update. I got hold of the details of the hardening and released that due to the inheritance properties, the hardening of the permission in hte registry has been perform on the "service" branch instead of the individual services. Several critical roles such as "Power Users" were actually removed as part of the hardening processes.

Now, take a step back. So what? I still ahve admin running and it was not working when I log in as admin. The reason being even if you are admin, it does not mean all the services are run as admin. Some of them are still run as other roles and these role unfortunate to say happens to be one of the removed one!

My gut feeling this was the root cause. I checked with another working server / workstation and restored the deleted roles into one of the Server Rabbit and ta-da... Everything was working and kicking again. Actually as long as RPC service is up, I pretty much knew I hit it.

Well, here is the aftermath of the whole incident.
1. There are malware found. Incidentally, this means the malware protection is not enough. Even if this time the malware did not take it down, sometimes in the future, it will.
2. There was no backup. Having a good snapshot / backup will save the situation and minimize the downtime required to recover the whole problem.
3. There is not enough documentation. This applies to logging as well. If there logs and documents are properly put in place, the whole incident could had been easily tracked simply from the look of the documentations.
4. There is insufficent training. While its a good idea to have some of the admin learn how to harden or improve certain functions, it is always good to provide the admin sufficent training to do so. With proper training / certification, the admin would had avoided the incident.
5. Access control was not properly inplemented. The sharing of accounts had made the whole investigation a nightmare. If individual accounts were used, combined with properly logging, it would had provided the information who made some change at what time. This could had greatly help the investigation.
6. Logging is the last time that I felt was missing. In terms of the firewall, IDS etc. If proper logs were generated, I would had concluded whether this was a security incident or not from the very start based on who has accessed the server etc. Of course lets not forget the golden rule that if a machine is compromized, the log is as good as none. Therefore logs from hardware boxes (which is probably not the target of the attack) is greatly helpful in determining whether a real intrusion has taken place or was it simply a misconfigureation or software corruption.

I am glad this was not my day in day out job. :) Well if it is, I am hell not getting paid enough for this. :)

Samsung Omnia / Orca Firmware HH3

Its out! Finally. While a HI3 is already leaked, the official ROM is not at HH3.
What are you waiting for? Grab it at :
http://www.samsungmobile.com/promotion/omnia/sg/sub02_Vista.jsp

CWCheat 0.22D

Its been a while since I updated this. Anyway, same drill. The latest (as of 25 Sep 2008) GameDB is loaded in. This is a package for the PSP Slim. Just unpack all to the root directory of your memory card of your PSP. Take note of the few *.txt file. If you have other plugins, you may want to manually edit them in instead of replacing it. Especially GAME.TXT, if you are a big fan of mine, then you will probably have the USB Charge and the Save state of the 4.01 FW.

Here is the download :
http://rapidshare.com/files/148169481/CWCheat22D.zip

Yet another 8 films to die for

The Horrorfest 2009 is back! Jan 9-15. 5 of the films has been announced. Check them out at :

http://www.horrorfestonline.com/

Highly recommended for horror fans!

Who Killed Server Rabbit? Part 4

The RPC service is critical for Windows function and I dare say nobody can do without it. If there is no way to enable it, the server is pretty much dead. Most things will not function and I suspect the GUI errors and the network, users and other symptons I had seen was probably due to the fact that the RPC service is down.

So, my aim will be to find a way to restore the RPC service. If I tried to restart it, it gives me an error 5 that access is denied. Well, that even happens to me as an administrator. At that point of time. I failed to see the point that although I am logged in as the administrator, it doesn't means that everything I run, I run it as an administrator. This was later found in the service control panel under the run as section.

Based on some of the suggestions, RPC service should be ran as Local System instead of Network Services. I tried changing that, well of course I can't change it in the service control panel since the properties is down. So, I had to do it the caveman way by editing the registry. This is one reason why normal users should never have write access to the registry. If they can change the registry, they can control anything they want.

I rebooted and hope for the best. The best was not good enough. The server was still pretty dead. Neither the Local System or Network Services ran RPC service properly. Now, how could someone had changed the permission such that even the defaults accounts could not run it?

By now, you may have some idea what actually went wrong. But I was pretty certain this was not a security incident then. It could had been a bad patch or some wrong configuration that cause the problem. But why only these 2 rabbit server? What similarity do they have? Ok, one of them were killed right on Sep 11, but that doesn't explain the other one. Neither did the patches, they do not even contain 1 single similar patch. OK, except the malware removal tool Sep, but if that was the case, millions of people out there would had scream out loud.

I was certain I was close... but not close enough...

Who killed Server Rabbit? Part 3

The weekend was not very fruitful because there were other temptation and Assassin Creed also look pretty cool on a 9600GT. Anyway, the logs looks pretty normal and there was no sign of an external attack. Maybe its time to consider an insider job after all.

So, I took one step back and take a look at the records. So far, I notice the passwords had been changed recently and whoever did left the company would not have access to it. So that rules angry ex-employee out.

The VPN was a connection to another site across the globe. The sole purpose was for replication only. And by that, nobody will log in to the server. The channel was also encrypted. So, unless the machine on the other site was infected, its highly unlikely something could had came in through the VPN. And moreover, the VPN is only on one of the machine, not the other. The 2 machines were on seperate network which could not reach one another either. So not possible to infect via the network. Thumbdrive, maybe, but that's restricted only to some of the staff and highly unlikely anyone was there on Sat and Sun where the first server rabbit went down. So, this is out too.

I am almost at my witts end. Then I had to go around and drink coffee with the rest of the staff to find out what they had been doing, any progress etc. One thing caught my mind. They were in progress of hardening the server based on Microsoft Best Practice. Well, at least they started to.

Giving there were no other choices, I feel this is one possibility that could had went wrong. So, I had the staff show me what was performed. One of them was undoubtly the auditing, which we can safely ruled out as an attacker's trick to clear the log. Then there was also some tweaks so restrict the user's access to the event viewers and some other services. I probably did not need to know the details, but the mention about services has brighten my way a bit.

I went back to the rabbit servers and check if the services were up. My hunch was right. There were barely any services running. I tried to manually start some of the services but were met with a permission error. I run some quick check on the internet and found more than 10 ways to solve it, but none of them worked. At this point of time, the staff has already begin to repair server rabbit 2003 by a in place reinstall of the files using one of the cloned drives.

I knew my time is running short and I had to quickly nail the problem. Looking at the services tab which has a block of GUI error, I suddenly saw one critical service was not up. I knew I had hit the jackpot. This was where the problem began....

Who killed Server Rabbit? Part 2

It was patch Tuesday again. Microsoft released some really serious looking patches who nobody can say no to. What nobody expected was another murder right in the morning on Wed. Another dead server rabbit 2003. The scary part was... the symptons were the same. No network and user control panel. Services panels no properties and some GUI errors. Again network was dead.

Now, it really doesn't make sense for someone to attack a good machine and not ownz it, but kill it. I took a quick look at the network architecture and I notice this time, that the 2 machines were on different network. One of them were not directly accessible from the outside, except through a VPN tunnel. This make the whole situation even more creepy because if this is a outsider job, he probably has a VPN access.

In any case, the 2003 rabbit was cloned with encase and quarantined. Now I will have to do some serious debugging. Some of the blame flew to MS's patches. I took a look at the 2 new patches applied to the 2003 rabbit. One was serious, the graphics file format attack. I know its possible some site could had actually created a 0 day attack and maybe the server rabbit was just stupid enough to stumble upon it. But highly unlikely. The other patch was the standard once per month malware scan by Microsoft. While I wasted some time on the server rabbit 2003, it finally hit me that I should take a look at the server rabbit XP and see if these patches make sense. Well, the unfortunate case was that it didn't. The XP has around 20 patches. Well, at least 20 patches since the last reboot. This rabbit was not put on a regular reboot routine. All the patches were put in place, but not in effect until it got reboot. This is in general a bad practice as the patches are not effective and when shit happens, it is really hard to tell which patch is the one causing the problem.

Without better options, I scanned again with the same malware tools, hoping to find some similar malware that could explain the situation. The fact was, its clean as ever. Either the malware tool was useless or we are facing a serious 0 day malware.

Next, I examine the logs, which fortunately was not 512KB only. This time audit was also kicked in, but at least some logs make sense. I also extracted the WindowsUpdate.log which details the installation of the patches from both the machine's Windows directory. With these logs, I am looking at a really gloomy weekend...

Who killed Server Rabbit? Part 1

On the dark night of Xth Sep, something took place because right on the first thing in the morning of the Xth+1 Sep, we found server rabbit dead. Network control cannot be accessed no more. We tried checking users accounts, nothing turns up too. Right click on various properties in event logs and services, totally nothing. Not even a box.

This was how it all started. We were pretty sure something killed our rabbit server XP. But what? First thing, I checked the log. Emm. It was a fantastic 512KB for each of the 3 logs and it was flooded with audit info. No luck. This could has been a really clever trick to wipe the logs. Change the log size and turn on excessive audits. If someone did it, he is pretty creative.

Althought the users control panel turns up empty, I was able to check it using the Administrator tools, manage computer section. No additional users of funny priviledges. So that's safe I thought. If anything was compromized, its one of these legal accounts.

The next thing I checked was the network connection. I found that it was down. At least it seems to be. Nothing seems to be able to connect out or coming in. This is weird. If I am an attacker, I wouldn't bring down the network because I would want to come back again. Well, it could be just a bad hacker.

So, maybe lets find out what he used to kill the server. I run a few malware scanner on the machine. Well, after we did a clone with encase anyway. In case we need to perform forensic on it later. Well well, a few BHOs and trojans turns up along with a couple of adware registry. Nothing serious. Well except for the trojan which keylogs some potential sites, but I doubt anyone would be using this machine to do internet banking anyway. So, I gave it a clean up and rebooted. It's still dead. Either the malware did some permenant damages or we simply did not manage to find the cause. In any case, the HDD was quarrantined and the data was restored from a mirror.

Since I had a cloned drive to test with, I tried to run sfc /scannow and restore the critical system files. I thought some of the files were replaced, but even with this, the server rabbit was still dead after a reboot. Its either that the files that was damaged were not critical files or the sfc was not able to fix the files but kept quiet about it. In any case, the HDD was taken offline and kept securely. All network were unplugged from the server until they can get the server up and running again in the coming week.

Little did I know that this was only the beginning...

Good riddance to bad rubbish ~ SingTel

While I made my request to cancel my line to SingTel, I was told this will take place in 3 days time... I wait 1 week and the line was still active. So I called up again. This time I can't be bothered to wait in the queue which took like 45 mins. I just used one of the I lost my phone or something options to get an operator. I explained the situation and guess what? They notice it was not being processed. Then they tried to hold me back from cutting the line. The proposed a SPECIAL package which is only $6 / month. Emm... What can you get for $6? Well, nothing except 15 mins for airtime... so that whoever call you, you can still tell them you changed number. Lets do our maths and you will notice thats like 20 cents / min. Luckily I passed my maths. This is a really stupid plan as their last attempt to con my money. Without anything free means you will need to pay $5 for CallerID as well. Now, I told them plain and simple. CANCEL. I wont even want to explain why I am so fed up with them or how they had fcuked up over and over again for me. A quick answer, the operator say he can make a new request and it will be cut off before end of the day.

Now, doesn't that makes you wonder why they told me it requires 3 days to cut off in the first place? Anyway, I think I don't mind paying a bit more, but definitely I stay off the Evil Red company for now.

Microsoft Dot Net Framework 1.1 SP1 for Vista

I do not know if Microsoft is aware of this, but Dot Net Framework 1.1 simply will not install on Windows Vista without SP1. And there is no package with SP1 integrated out there. You will have to installl the vanilla 1.1 then perform a upgrade to 1.1.

There is a page that will show you how to extract the files and patch it before installation here:
http://kbase.gfi.com/showarticle.asp?id=KBID003100

Well, you do not need to do that anymore. I had decided to slipstream the whole package so that it can be installed directly on Windows Vista.

Simply download the package below, extract all of it. Then just run netfx.msi.
http://rapidshare.com/files/145885903/dotnet.zip

Beware of the SingTel Shop scam!

Yes, althought www.singtelshop.com seems really like a real shop, its a black shop. Look at what happened to me. I sign up, made the payment. Then they called me, told me that they cannot honor the purchase and slam the phone.

Guess what? They already charged to my credit card! I called and they insist they will not honor the purchase and will not send the phone to me.

Tell me if this is not a scam, what is?

I strongly advise anyone to think again if you are really buying from singtelshop.com. Or in the first place, does SingTelShop even belongs to SingTel? they have such differences in prices. For all you know, this is a phishing site!!!

Anyway, I am going to call my credit card company to cancel the purchase.

Singtel fcuks up again!

I am a bit boilling to talk abt the whole incident, but I would like to archive the letter to Singtel here. I will talk about it in another post.

To singtel :
I had placed an order 100****** online because I wish to avoid the queue at the shop. I had followed all the intruction, scanned the student pass for the student line etc, However, next working day I got a call to say the order is cancelled!!!I was told that ST cannot honor the signup done online and I had to do it at the Hello Shop? WTF? Then why have Singtelshop.com at all? It even wasted mt time to scan and upload the image!
Then I checked at the HelloShop... The price of the handphone is doubled!!! Why do you published price which you cannot honor at all? You had totally wasted my time and my opportunity to sign up with SH or M1!!! If you so desire to do so, you might as well put all your phone at $1 on Singtelshop.com since you cant honor the pricing anyway!

Goodbye Xiao P

Xiao P has past away on 2nd September after he was with me for over 2 years. I would estimate he lived about 2 yrs 3 months at least which is already far longer than an average pudding. He was still ok 2 night before when I played with him in his playground and I gave him a lot of his favorite food. He was still digging through them playfully.

Then the night before I saw him lying down weird. I knew the day was finally coming. After giving him some honey water, he seems better and he seems to do his very best to run around just to show me he is ok. What makes the whole thing even more sad... I comforted him and transfer him to somewhere quiet. Then after a while he slept peacfully and quietly. I couldn't sleep that night.

At about 5am I woke up and he was still and starts to harden. I did what I had to and give him a proper buiral.

I will alwasy miss you Xiao P. I have to stop here...

Facebook DoS PoC

An Facebook app call Photo of the Day :
http://www.facebook.com/apps/application.php?id=8752912084
demostrated a PoC of how Facebook can DoS another site. This app basically in short show 1 photos from National Geographic. But downloads 3 more huge files from somewhere else. Or for that matters, you can do upload, posts, requests or whatever you can imagine using hte Facebook API. Effectively, all you need is millions of people to sign up your app, then when they log in, they become your botnet to DoS someone.

For example, I can up my site counter to millions easily by creating a bogus app that gives free paypal money when you refer someone. Of course on the side line, I can con some paypal accounts as well. :) Then I can make a hidden read to my site in the app and not disaply it. Once millions sign up, millions will visit my site. OK, shake this off your head. This is a (C)Copyright idea... Hahaha. Seriously, its that simple. Make you wonder why it hasn't happen yet. (Or is has... We just did not hear of it)

Anyway, here is the full whitepaper :
http://blogs.zdnet.com/security/images/facebotisc08.pdf

Google Chrome Updates to v0.2.149.29

Quietly, Chrome has issued an update (which should happen quietly too) to v0.2.149.29 from v0.2.149.27. The older version is vulnerable to at least 1 auto downloading bug and at least 4 DoS / crashes error. I can't be bother to post these exploit here anymore because almost everyone will be updated and I will get tons of "How come it does not work?" post...

The good thing is Google is putting this in spotlight and fixing as we move along. Of course, thanks to the millions of Beta testers like you and me who is trying so hard to 0 days this. I guess I will work on something else now...

IE8? Maybe...

发誓 TVB剧集《搜神传》片尾曲 ~ 钟嘉欣

Legend Of The Demigods 搜神传 主题曲

Google Chrome - Zero Day Buffer Overflow

Yes, Google's browser is out. And Yes, a zero day can be found within 24 hours. Now you have to see it to believe it.

BTW, Google Chrome can be downloaded via :
http://www.google.com/chrome

After installing, just type this into the URL :
xiaop:%

or if you are lazy, browse to Crash Chrome

Yes, immediately, Google super friendly error message will greet you :


Well, if you had installed the right stuff, you can kinda find out what happen :

Of course, this is a buffer overflow.

Well, if this doesn't worry you about the way Google release this software, take a look at this :

This won't look too good if you had been surfing content rich multimedia sites (which Chrome is optimized for) such as Playboy or something...