[SecurityRatty] tag: entries

Internal Network Threat Encyclopedia

Thu, 21 Aug 2008 11:39:04 +0000

Promisec has announced what it calls the first encyclopedia of internal threats. The Internal Threat Encyclopedia contains both shady and clearly legitimate software that is subject to abuse. For instance, you'll find Laplink and Timbuktu in there, both straight-up remote control programs. The top 5 internal threats, according to the encyclopedia, includes (today) Google Talk, Skype and MySpace. These applications are well known for sure, but the encyclopedia entries are a handy collection of the problems each can cause. It could be useful if you need to explain why you're setting rules against one of them.

Keylogger Or False Positive Detected in Wall-E Demo?

Sun, 03 Aug 2008 07:23:19 +0000

I woke this morning to find an interesting set of blog entries regarding the Wall-E demo game from THQ - someone downloaded the demo and found their AV scanner flagging it as potentially dangerous.

A quick roundup of posts:

1) Security researcher Timeless Prototype downloads the Wall-E demo, only to find his antivirus software going crazy. It has detected Spyware.Ardakey.

2) Over at Spyware Sucks, Sandi Hardmeier decides to try downloading versions of the game from different regions, only to find the French, German ,Danish and Italian versions are all 177MB in size, whereas the US version is "only" 133MB. Furthermore, the 177MB versions all have different filenames. Note that (so far) it's the UK version (clocking in at 177MB) that has been snagged by an antivirus program. As Sandi notes, there is no way an extra 40-odd MB are needed for a keylogger, so why the extra filesize?

3) Wayne Porter contacted Cachefly (who manage the servers the game is downloading from), and they said this:

"I can confirm that our servers were not compromised, beyond that I can't offer much else.

Obviously we'd like to be as helpful as possible, but since it's related to customer data we're rather limited in what we can discuss. I've opened a ticket to make THQ aware of this, and we can/will work them on tracking stuff down if we need to (we do have a history of all versions of a file w/ filesizes/md5 checksums, and the dates/times/src ip of all revisions)."

The 177MB file is still available to download, I grabbed it a little earlier on today:

What we really need to know, is if this is anything to be worried about or not. I would have contacted THQ UK directly, but they don't seem to be available on a Sunday. Until this is resolved one way or another, I'd have to advise people not to download this demo as a precaution until THQ (or Norton, whose AV program flagged the file) have clarified exactly what is going on here. We're currently running some more antivirus / antispyware scans against the download in question, but as you can imagine, this takes some time. A particular problem here is that there are issues submitting a file like this to sites such as Virustotal.com, because of their 10MB file size limit.

Sorting this one out might take a while...

/ Update - some people are saying AVAST flags the file, too.

The Impact of Dans DNS Debacle on Internet Risk

Wed, 30 Jul 2008 04:11:30 +0000

Blogger: Pete Lindstrom

On July 8th, Dan Kaminsky of IOActive announced a major DNS “vulnerability” in conjunction with a number of major DNS vendors. The announcement was off the charts in fanfare and attention, but what was the real impact on risk?

First, it is worth noting that this “bug” is more properly classified as a new attack technique invented by Dan. It combines two vulnerabilities that have been well-known for some time – the ability to guess non-random transaction IDs and the use of Additional RRs to insert new entries into the DNS cache. A fix against either of these vulnerabilities also negates the attack itself.

The fundamental question that determines the risk impact revolves around whether it is reasonable to expect fewer or more incidents that use this technique when comparing the period prior to disclosure -- or, more properly, before the date of Dan’s invention of the technique (this also assumes prior art) – with the period after invention/disclosure and into the future. If the disclosure reduces the number of those incidents, then risk is reduced; if the disclosure increases the number of those incidents, then risk is increased.

With that litmus test as our guideline, it is useful to break down the functional elements of risk and look at the impact on threats, vulnerabilities, and consequences (we will cover consequences, then vulnerabilities, and finally threat).

Consequences
Though the consequences are the same before and after disclosure, it is worth discussing the impact here, given that the implication was that the “entire web” could be taken down. The nature of the attack requires the following:

An attacker must convince/trick a user into making a DNS request for a domain that doesn’t already exist in their DNS server’s cache. The expectation here is that s/he can be easily tricked into doing this.
Then, the attacker must simultaneously attack the DNS server by guessing the transaction ID. According to Kaminsky, the request/attack phase can be done reliably in about 10 seconds.
The attack is DNS server-specific. Only users on the same DNS server are affected.
Propagation: once the cache is poisoned, anyone requesting that domain will be routed to a malicious server.

Without combining this attack with other attack techniques, there can be three results:

Spoofing of a single website for multiple, perhaps many, users using the same DNS server. Presumably, this would be followed by more traditional phishing and malware attacks.
Denial-of-service by rerouting traffic from a legitimate site thereby taking potential customers or “eyeballs” away.
Denial-of-service be rerouting traffic from a legitimate high volume site to a legitimate low-volume site thereby overloading the servers on the low-volume site.

Because of the point-to-point (user-to-website) nature of the attack, to do something that constitutes “taking over the entire web” is infeasible by a longshot.

The bottom line analysis for the effect on risk due to a change in consequences from pre-invention to post-invention: no change, and therefore no impact.

Vulnerabilities
These vulnerabilities have existed for years, and there have been workarounds for years. Along with this announcement, new patches were introduced in all major DNS server solutions. It is reasonable to assume that many DNS server implementations have been patched, though public accounts have suggested that number is in the 66%-75% range.

Bottom line analysis: the vulnerability level has been reduced, probably significantly, and the affect is positive for risk reduction. If 100% of DNS servers were patched, then overall risk would be reduced for this attack (assuming that there were actual attacks using this technique in the past.)

Threats
The real question regarding risk impact comes in the arena of the less-controllable manipulation of threat. The general threat equation revolves around an attacker’s willingness to attack, based on his/her own cost/benefit analysis that compares the cost to attack to the expected benefits, tempered by the potential for being caught and penalized.

Cost to attack – prior to disclosing the invention, there were likely few, if any attackers with “prior art” that mirrored this technique. It is anybody’s guess how many potential attackers might have figured it out eventually, but they would have had to come from the pool of folks with enough expertise to do so – I am going to guess 500,000 people.

After the disclosure, the hints provided in the press release, the podcast, the sorted stories, and the blog entries made it much easier to figure out. Let’s guess that 5 million people could execute the attack. With automated tools, that number goes up to 50 million.

These numbers are estimates that illustrate the nature of the exercise. You are welcome to fill in your own estimates and come to your own conclusions.

Bottom line analysis: a significant increase in threat and corresponding risk.

Net Effect
The risk manager's challenge is to weigh the decrease in vulnerable systems compared with the corresponding increase in threat, within the context of number of incidents and anticipated future incidents. Given the sheer size differential, it is difficult to conceive of a situation where risk is not increased.

Sometimes it "feels" like someone is taking action for the greater good, when that action actually creates a negative impact for all. For example, it is common for people to believe that raising prices of scarce resources during times of trouble (e.g. gasoline in the hurricane Katrina aftermath) is unconscionable even though a majority of economists recognize that raising prices actually provides for the greater public good. Vulnerability discovery and disclosure, and attack inventions, might feel like the right thing to do, but the net result is almost always a negative impact.

Coding Spyware and Malware for Hire

Mon, 21 Jul 2008 23:52:14 +0000

What type of antivirus evasion do you want today? For the past several years, we have been witnessing the emerging customerization applied in malware and spyware for hire services. What used to be a situation where the malware authors would code and then start promoting a piece of malware including features that he thinks his potential customers would want by generalizing a cybercriminal's needs, is today's "listening to the customer" win-win situation that they've reached already.

The whole maturity from a product concept to customerization is in fact so prevalent these days, that malware authors wanting to preserve their intellectual property are forbidding their customers from reverse engineering their malware modules, presumably fearing that remotely exploitable flaws like this one in one of the most popular Ebanker malwares for the last two yers Zeus, could be discovered due to the malware author's insecure coding practices. Moreover, limiting the distribution of a single license they are given to more than three people will result in the malware author ignoring any future business relationships with the party that ruined the exclusiveness of the malware, thereby leaking it to the public, something that's been happening and will continue happening with web malware exploitation kits.

What would be the price of a custom malware module coded on demand? How much does it cost to have a built in email harvester that would sniff all the incoming and outgoing email addresses from the infected host to later on include them in upcoming spam and malware campaigns? Would the malware author also provide a managed hosting service for the command and control and the actual binaries on a revenue sharing

Here's an automatically translated, and fairly easy to understand random proposition for coding spyware and malware for hire, aiming to answer many of these questions, clearly demonstrating that today's malware is coded in exactly the same way the customer wants it to :

"As you can see in the history of its development turned directly into the combine, while almost no raspuh in weight, full-size pack аж 18 kb and minialno 5 kb, for all nampomnyu again, all descriptions below can be done as otdelnym bot, and any combination of cross except for a few restrictions. This product is targeted at mass-user and will not be all prodavatsya row. So, you can choose from:

Actually loader - is able to load a file from adminki, by country and other characteristics, such as the number of animals on board with a specific bot, a country group of countries, the availability of certain authors or Fire, sredenemu time online, etc. etc.. You can adjust the speed of shipping limits for each file, can load 1 as well as how files simultaneously
300 €

FTP and not only Graber
Analyzes user traffic and collects from the ftp acclamation, that is ftp acclamation would you regardless of how the customer uses ftp user, thus can be obtained most valuable ftp aka (even those to which the password is not saved), you can also grab other in a way not only acclamation acclamation and other tasty things more)
150 €

Assembler spam bases
Analyzes user traffic and collects from all email, snifit http pop3 smtp protocols, keeps records unikallnosti locally on each boat to reduce the burden on the server as well as globally on a server has 2 mode of operation - ie passive with only collects user to please and active - the very beginning to download the entire inet) in search of soap
220 €

Socks 4 / 5
Normal soks with competently implemented multithreading, is activated only if the user real Ip, otherwise not. And also optional, depending on the connection type and speed ineta.
70 €

Indicates
The primitive method, contamination fleshek avtoranom gives 2-3% increase in the first week and up to 7% in the next, a pleasant trifle)
35 €

Scripts
Loader supports internal scripting language - jscript, to carry out arbitrary actions on the victim machine, whether recording data in the register, setting authentic hon-Pago, opening URL in your browser (it was done so to please with 90% punching)), apload arbitrary files on a server, even theoretically possible to form and grabing inzhekty in IE) has only to write the script zaebetes, vobschem lyuboye actions soul who wish)
70 € basic functionality

Assembler passwords
Collects data such as passwords pstorage IE, MSN, etc., will be added at the request of other sources of passwords
70 €

Mini-AV
When installing loadera wheelbarrows to remove BHO shaped three, zevso-shaped, the majority of shit from all avtoranov, render most keylogerov until all) forward proposals to improve
70 €

File-default
In exe loadera program URL (in adminke) to the file which once progruzit 1 and run at first start loadera on wheelbarrows, while simultaneously helping progruzke Trojan for example, in its entire botnet that does not paired with challenges in adminke, the module operates in 20 seconds after the mini - av which excludes the removal of your Trojan bot, after progruza this exe bot continues to normal activities.
35 €

Form Graber
While in beta version, robbed IE. Sends logs in adminku, folding country. Logs are like logs agent. It consists of:

Graber certificats
On the idea is part formgrabera but could work and of itself, actually there is nothing to describe)

Injections
Literacy sold inzhekty, did not begin work after full progruza pages (as in bolshistve three) and immediately supported injection yavaskript code, which allows avtozalivy and DC inzhekty for data collection. For example not to yuzat acclamation at all is not yet introduce the necessary number of Britain, after which inzhekt ceases to operate. Вобщем mdelat can be anything and in any form) rather than the meager request field pin) And also inzhektov subspecies - a substitute for the issuance of search enginee.

Graber balances
Makes loot aka balances at the entrance to the user acclamation, detail added to the logs.

Screen
Universal method to grab information from absolutely any species and varieties klaiviatur screens, in particular html, flash, in one picture, with a drop-down fields after choosing your encrypted, as well as information such as "enter 3 yu secret letter word" etc. as well as any information which is visible a user but not seen in the logs. Screen settings of adminki, set URL where do screen as well as the type of screen: for virtual keyboard (done several small images of areas around the clique) or to "enter 3 yu secret letter words" (makes 1 full shot). With the withdrawal screen recorded in the log entry with the name of the file to the screen this position.

Antiabuznost for botneta
Feachem adminki, keep botnet enables fast, normal, bezglyuchnyh NEabuzoustoychivyh hosting, with features that you forget what abuzy, nohistory week saporta "abuzoustoychivogo" hosting inaccessibility host to half ineta etc., etc., also with the help of the supplement will be able to keep huge botnety (over SL) at 1 dedike with 512 Lake) and well on the price of hosting a savings, not $ 500 a month and 150. It may use this feature to stroronnim development, Trojans, bots, etc., actually is a separate product. And incidentally, if you do not understand the theory that nenado ask "and how does it work?" imagine that it works and point and neubivaemo in pritsnipe.
600 € +

All prices are in euros, the calculation is made at the rate of CB on the day of purchase. ps I will not disappear as most authors after months of sales, I DONT how to please you get to the assembly ftp, I DONT how many soap collects soap-graber, I DONT what otstuk from loadera, I DONT soksov how many will be from 1 to downloads, and how best To work load a file is not dead quickly, if you are confused my ignorance - that my loader so you do not need more tries)

Rules / Licence
-- Customer has no right to transfer any of his three 3 persons except options for harmonizing with me
-- Customer does not have the right to make any decompile, research, malicious modification of any three parts
-- Customer has no right where either rasprostanyat information about three and a public discussion with the exception of three entries.
-- For violating the rules - without any license denial manibekov and further conversations"

This malware coder seems to be participating in an affiliate program with a malicious ISP that is offering hosting services for the entire campaign, not just the malware binaries, so you have a rather good example that incentives and revenue-sharing models result in value-added services, a all-in-one shop for a customer to take advantage of without bothering to approach a third-party.

Cybercrime is getting even more easier to outsource these days, and with the malicious parties improving their communication and incentives model, the resulting transparency in the underground market

Related posts:
The Underground Economy's Supply of Goods and Services
The Dynamics of the Malware Industry - Proprietary Malware Tools
Using Market Forces to Disrupt Botnets
Multiple Firewalls Bypassing Verification on Demand
Managed Spamming Appliances - The Future of Spam
Localizing Cybercrime - Cultural Diversity on Demand
E-crime and Socioeconomic Factors
Russia's FSB vs Cybercrime
Malware as a Web Service
Localizing Open Source Malware
Quality and Assurance in Malware Attacks
Benchmarking and Optimising Malware

Review: Internet Cleanup 5.0

Sun, 20 Jul 2008 20:00:00 +0000

As you use the Internet, traces of your activities build up on your Mac in the form of things like cookies, caches, entries in your browser history, transcripts of instant messaging chats, and e-mail attachments. Smith Micro's Internet Cleanup provides a suite of tools to help you locate and delete such data to protect your privacy, which is especially important if you're using a shared computer. Several additional tools in the package protect your Mac against certain kinds of spyware, hacking, and other threats.

How Can I Find Them? They Haven't Gone Missing!

Wed, 09 Jul 2008 08:45:35 +0000

I've often highlighted the utterly worthless spam messages that seem to endlessly circulate on Facebook, usually warning not to add (insert random name here) because they're an evil hacker and will destroy your PC, kill your family and so on.

Well, today I came across another such message:

.....insert gag about them being related to Chuck here....but underneath that message was something far more interesting:

Sounds serious, right? It seems personal, because it's their friend missing which adds a little more urgency - they provide a contact email address to notify them on, and it mentions a real world example of someone who went missing and was found via the Internet.

However.

Dig into this a little bit, and it all becomes clear quite quickly that something isn't quite right here. For starters, search for the missing persons name and there is no mention of him ever "going missing". Nothing on websites, news pages....it's like the whole thing is a work of fiction. In fact, buried in unrelated entries is the following snippet from a page on myyearbook.com:

Click to Enlarge

Check out the name of the "hacker" you shouldn't add. It seems someone has simply swiped the name and started pasting it into spam messages. A quick search of Facebook confirms the name and face go together.

A quick search for the email address listed as a contact brings up more interesting posts, this time posted to a personal blog:

Click to Enlarge

Same text....same reference to "real world" example....same email address. This person sure does get through a lot of missing friends! Note that this "missing person" chain letter has now stepped outside of Facebook and into other websites and networks.

At this point, you're probably wondering about the validity of the "real world" example, aren't you? Well, that would be a good idea! Notice they don't give any detail - it simply says "That is how the girl from Stevens Point was found by circulation of her picture on TV", and expect you to accept it as is. If you go searching for that phrase, it doesn't take long to find a page on Snopes.com regarding a missing girl hoax that stretches back some years:

"Please look at the picture, read what her father says, then forward his message on. Maybe if everyone passes this on, someone will see this child. That is how the girl from Stevens Point was found by circulation of her picture on tv..."

An email hoax, wrapped up and repackaged for the Facebook generation.

Your 419 Mail Roundup

Wed, 02 Jul 2008 13:11:42 +0000

A handful of scam mails currently in circulation, including one mention of "groundnut oil" that seems so bizarre I had to highlight it in bold text. All this and more, after the jump...
Subject:
FROM THE DESK OF MR. STEVEN JAMES
From:
"Steven James"
Date:
Mon, 30 Jun 2008 19:17:03 +0100
BCC:

FROM THE DESK OF MR. STEVEN JAMES
CHAIRMAN INTERNATIONAL RELATION
FIRST BANK OF NIGERIA PLC
# 1 BANK ROAD WUSE FCT
ABUJA-NIGERIA.
PHONE: +234-80-66520277
Email: stevenjames809@live.co.uk

Very Urgent Attention,

Please permit me to introduce my humble self to you, my name is Mr. Steven James, I am the Manager of International Relation with First Bank of Nigeria Plc, I 'm 38yrs old, and I got your email address from a friend of mine, and my confidence reposed on you. I hope you read this message carefully and reply me immediately. Although we have not met before, but I suggest that this transaction will bring us together.

My dear, we had a customer, a foreigner but base here in Nigeria, his Name was Mr. Hamilton Creek. He is from Atlanta Georgia United State of America, but based here with his wife and his two children, Mr. Hamilton has being banking with us for the past 4yrs and some time in August 2002, Mr. Hamilton was on his way to his house, and unfortunately ran into a Trailer load of Groundnut Oil, and died   immediately, Their car got burnt, no single soul was saved, Mr. Hamilton Creek and His entire family was confirmed dead.

My Board of Directors and the Management of First Bank has mandated and instructed me to look for Mr. Hamilton Creek? Relation(s) and his Next of Kin to come and claim his fund, Since August 2003 till date, I have been looking for his relation's or his next of Kin to come and claim his fund which he Deposited with our bank, I have contacted his Embassy and after 3days, his Ambassador told me that Mr. Hamilton Creek has no relation and no next of Kin, their Ambassador told me that he used his first son as His next of kin, but it is quite unfortunate that Mr. Hamilton Creek Died with all his family members.

The reason why I contacted you is thus, Mr. Hamilton is dead, and his only son who supposed to inherit his properties and money also died with him. As at this moment, nobody or person[s] is coming to   claim this Money from our bank. The Board of Directors and management of our bank told me that if nobody or person[s] apply for the claim of Mr. Hamilton Fund, the bank will return the entire Fund into our Federal reserve. In the Light of the above, I want you to stand as the next of kin to Late Mr. Hamilton Creek; it might interest you to know that he had a Domiciliary Bank Account with our Bank and he has a total sum of US$9.2M Nine Million Two Hundred thousand Dollars, this is the exact amount which he had in his domiciliary account before the ugly incident occurred, and this money is still in his account as unclaimed money.

This transaction is very easy and simple, and it is 100% risk free, I'm the Manager for International Relations with First Bank of Nigeria Plc, and the Management and Board of Directors of the Bank are waiting for me to provide to them the Relation or next of Kin to late Mr. Hamilton Creek, of which I told them that I am still searching the next of kin to the deceased. Finally, if you are interested with this transaction, I will front you to the bank as the only next of kin to late Mr. Hamilton Creek, and I will let the bank know that you are the only right person to inherit Late Mr. Hamilton Funds and properties. If you are interested, just email me or call me on my   direct and private line#: +234-80-27536038 and late Mr. Hamilton's Funds will be credited into your account and all his Properties will be released to you either through Courier Services or the Bank will Cargo all his properties to you in any were you want it.

So reply me immediately and feel free to ask any question with regards to this transaction. You will take 50% of the US$9.2M. Which is? US$4.600, 000.00 Four Million Six Hundred Thousand Dollars, while the Balance of the same amount will be mine.

Your swift response will be highly appreciated.

Thanks and have a nice day.

Friendly Regards

Mr. Steven James

*******************************************************************************************

Subject:
REPRESENTATIVE NEEDED
From:
DFS SALES LTD UK
Date:
Tue, 01 Jul 2008 23:00:55 +0800
To:
undisclosed-recipients: ;

COMPLIMENT OF THE DAY TO YOU.

I am PETER WOODS from DFS SALES LTD UK.(
Website: www.dfs-online.co.uk ) Visit our site

We are into furnitures and we sell shares to people in
Canada,America, Australia and Europe.

We are in need of a book keeper. someone who can represent our company
in his/her country.

Our client in your location will contact you and make the company
payment to you.

You will be entitle to 11% of every payment been made out to you.

This is because most of our officer are from china and they do not

understand english very well.its hard for them to contact our
customers.

Our head office is located in CHINA. But we have a sub-office in the
uk.

If you are interested, Kindly send the entries for more understanding.

NAME IN FULL :.........
COMPANY NAME: .....
POSITION:......
FULL ADDRESS: .......
CITY/TOWN:........
STATE:............
ZIP CODE:........
COUNTRY:.......
MOBILE:.......
HOME TEL: .....
EMAIL ADDRESS: ........
OCCUPATION: ...........
BANK NAME :.......
AGE:............

You are to send the above details to

NAME : PETER WOODS.
EMAIL : dfs_woods@yahoo.co.uk
PHONE NUMBER : +44-704-575-0212

HOPE TO HEAR FROM YOU

*****************************************************************************************

To:
undisclosed-recipients:;

Good day!!!

We have been waiting for you since to contact me for your Confirmable Bank Draft of ?18 Million (Eighteen Million Pounds sterling) but we did not hear from you since for a couple of weeks now. Then we went to the bank to confirm if the draft that expired or getting near to expire and Metropolitan Police Uk told us that before the funds will get to your hand that it will expire.So I told him to cash the ?18 Million (Eighteen Million Pounds sterling) to cash payment to avoid losing this fund under expiration as I will be out of the country for a 6 Months Course.

What you have to do now is to contact FED EX COURIER SERVICES as soon as possible to know when they will deliver of your funds to you because of the expiring date. For your information we have paid for the delivering Charge Insurance premium. The only money you will send to the FED EX COURIER SERVICES to deliver your cheque direct to your postal Address in your country is ?250.00 being Security Keeping Fee of the Courier Company so far. Again don't be deceived by anybody to pay any other money except ?250.00 for the Security Keeping Fee.We would have paid that but they said no because they don't know when you will contact them and in case of demurrage. You have to contact FED EX COURIER SERVICES now for the delivery of your Draft with this
information below:

CONTROLLER: Mrs.Helen Williams
NAME: FED EX COURIER SERVICES
ADDRESS: fedexofficeuk@gmail.com
PHONE NUMBER: +447024080684

IF YOU ARE THE OWENER OF THE FUNDS AND YOU WILL SEND YOUR INFORMATION TO US SO THAT WE CAN DELIVERY YOUR FUNDS TO YOU WITHIN THE NEXT 84HRS TIME.IF YOU DO NOT RECEIVED YOUR FUNDS WITHIN THE NEXT 72HRS TIME AND YOU REPORT US THE UK FBI AND THE METROPOLITAN POLICE (SCOTLAND YARD) or YOU CONTACT YOUR LAWYER TO TAKE UP PROCEDURES AGAINST US.

Let me repeat again try to contact them as soon as you receive this mail to avoid any further delay and remember to pay them their Security keeping fee of ?250.00 for their immediate action. The FED EX COURIER SERVICES don't know the contents of the funds. This is to avoid them delaying with the funds.

Thanks as you contact them today.

Yours Faithfully

Mrs Helen Williams.

(The above actually comes with a nifty graphic that they've thrown in, thinking it makes it all look more legitimate. It doesn't, but here it is anyway):

....altogether now: oooooh. A slightly shorter 419 roundup than usual, but I'm sure I'll have piles of the things next week.

A Question of Integrity: To MD5 or Not to MD5

Wed, 25 Jun 2008 15:50:57 +0000

Cloud Storage offers pay per drink off-site storage. Data to be saved is shuffled from the customer to the Cloud Storage Provider by the network. This all works wonderfully most of the time, what you upload is what you get back later. But what happens where the gremlins strike and what you send is not what is received?

This happened recently to some Amazon S3 customers. There were complaints in the AWS forums about ‘S3 Corruption’. The first post in the forum was recorded at Jun 22, 2008 5:05 PM PDT (although in subsequent posts some people reported emailing Amazon prior to this):

we are having some serious S3 issues.

all data we store on S3 has gone through the same code path for months. starting a couple days ago a small percentage of the objects we are retrieving are not checksumming to the correct values. we hash and store objects by checksum and rehash the objects when we retrieve to ensure there is no data corruption. all the objects we’re having issues with were uploaded at approximately the same time period a few days ago.

we’ve stored 10’s of millions of objects in S3 and never encountered such problems. please let me know ASAP if you have any idea what could be going on here. thanks.

Amazon responded 6 minutes later (!) and started investigating. To troubleshoot they asked customers to email aws@amazon.com with the ‘Bucket-Name and few keys that you believe are having issues’.

Others weighed in reporting similar problems. Amazon provided status updates and on Monday Jun 23rd at 6:10pm PDT, provided the following explanation:

We’ve isolated this issue to a single load balancer that was brought into service at 10:55pm PDT on Friday, 6/20. It was taken out of service at 11am PDT Sunday, 6/22. While it was in service it handled a small fraction of Amazon S3’s total requests in the US. Intermittently, under load, it was corrupting single bytes in the byte stream. When the requests reached Amazon S3, if the Content-MD5 header was specified, Amazon S3 returned an error indicating the object did not match the MD5 supplied. When no MD5 is specified, we are unable to determine if transmission errors occurred, and Amazon S3 must assume that the object has been correctly transmitted. Based on our investigation with both internal and external customers, the small amount of traffic received by this particular load balancer, and the intermittent nature of the above issue on this one load balancer, this appears to have impacted a very small portion of PUTs during this time frame.

What are some of the takeaways?

If you are directly using the AWS S3 API, make sure to calculate and send MD5 checksums along with actual data. Check status return codes - an HTTP 400 error code means ’something went wrong’ - respond appropriately.
If you are relying on 3rd party tools to access S3, be sure to check with your software vendor that they are following the advice from Amazon to use MD5. If they are not then your data can get silently corrupted…
Downloads, aka HTTP GETs, can also be affected. The thread in the forum continues and questions are asked as to whether the corruption caused by the loadbalancer was affecting both incoming and outgoing traffic. The conclusion was yes. If you are hosting media on S3, and the browser is using partial GET requests (to download in chunks) then the corruption will not be automatically detectable.
If your business relies on Cloud Storage, are you prepared to wait a 36 hours for a resolution? This isn’t a swipe at Amazon, this is true for any provider. Check your SLA’s, check the trouble ticket resolution times, ask about availability of experts for troubleshooting etc.
Cloud Providers will increasingly need to instrument their services such that they can ‘early detect’ negative operational events. In this case, Amazon has stated plans to use better logging and analysis to automate detection of unusual error patterns (i.e. anomoly detection).
This incident - caused by an Amazon malfunctioning loadbalancer - did not make it onto the AWS status page at http://status.aws.amazon.com/. Taking Amazon at face value, this incident only affected a small number of transfers, relative to the total number of S3 transfers. But this begs the question, what level of outage or service problem needs to happen before Amazon will flag the issue on their status page? On a sidenote, based on the timestamps, 31 hours passed between the loadbalancer being taken out of service and Amazon providing the explanation on the forum.
When Amazon update their S3 API documentation, it would be useful to have entries in the S3 API index for ‘checksum’, ‘MD5′, ‘integrity’ and ‘corruption’.
Stepping back, will customers hold Cloud Service Providers to a higher standard than their own internal IT teams?

I’m sure there are more takeaways I didn’t cover. What say you?

###

Kudos for the heads-up on the S3 issue goes to my friend and colleague Jason Harper - network supremo and crypto-head. Thanks Jason!

New RSA Compliance Solutions Bloggers

Tue, 24 Jun 2008 20:00:00 +0000

Please join us in welcoming a new set of RSA Bloggers. The RSA Compliance Solutions team--including Dave Howell and Brad Davenport--will be penning a set of blog entries for "Speaking of Security" around the theme of Simplified Compliance. Please take advantage of the comments field to get answers to your compliance-related security queries!

Akihabara killer chronicled massacre plans online

Sun, 08 Jun 2008 20:00:00 +0000

The entries posted to a Japanese online discussion board are chilling.