Full Disclosure mailing list archives
Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass?
From: "Chris Weber" <chris () casabasec com>
Date: Sat, 06 Jun 2009 17:43:47 -0700
Here's a common pattern in software, especially transcoders, converters, parsers, filters, the good stuff: 1. accept input from any charset (e.g. shift_jis, iso-8859-x) 2. convert input to Unicode for further processing 3. convert input back to its original charset for output/display Now, lots can go wrong here, too much to get into right now, but in general, what happens when there's no clear mapping between 1/2 or 2/3? Quite often characters will get mapped to the private use area (PUA) and at that point it's up to the vendor/framework to handle it how they want. But that's not normalization. I just want readers to be clear that 'Normalization' in Unicode isn't a squishy 'maybe this maybe that' sort of thing where characters and strings transform by the whim of a given framework. It's a well-defined standard of both 'compatibility' and 'canonicalization' decomposition and recomposition. Some of the mappings are data-driven, X == Y, and some are algorithmic. If all frameworks implement this standard, they should all normalize the same, differences would occur when one framework uses an old version of the Unicode database, or of course if they botched something in the spec. Your discussion point #2 seems to digress, talking about the confusables and lookalikes don't seem to lend to the original subject. Unless, you're suggesting that they somehow add to the canonicalization of strings that White Hat is seeing? If someone wants to study the 'best-fit' mappings between charsets and Unicode, they're mostly available at http://unicode.org/Public/MAPPINGS/. Whether or not they've been implemented that way by each vendor is another study. -Chris -----Original Message----- From: arian.evans () gmail com [mailto:arian.evans () gmail com] On Behalf Of Arian J. Evans Sent: Friday, June 05, 2009 4:31 PM To: Stephen de Vries; Full-Disclosure; websecurity () webappsec org Subject: Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? response inline On Thu, Jun 4, 2009 at 11:23 PM, Stephen de Vries <stephen () twisteddelight org> wrote:
Hi Arian,
http://jeremiahgrossman.blogspot.com/2009/06/results-unicode-leftright-point ing.html
Was there a common library or framework in all the vulnerable sites that
was
responsible for this?
Excellent question. Chris's response covers part of this, but I will add below where I disagree with his 99% excellent response. Long and short: Yes I think so, though Blind Black Box testing only informs you is that the culprit is guilty, not *who* the culprit is. :) There are three classes of these issues, and they all occur for different reasons. Chris already excellently addressed this, ironically using three categories as well, though I label them a bit differently: 1. Valid but Alternate Encodings that are normalized 2. Literal Transcodings that occur to avoid one issue (security, false-familiar, name-collision, etc.) while creating a new vulnerability 3. Interpreters Bugs that were truly unintended --- 1. Valid Alternate Unicode Encodings that are Normalized: Some of these issues, like Fullwidth encoding, use valid and legitimate Unicode representations that the software normalizes to a canonical form. However uncommon and unexpected the encoding may be -- when you find these they tend to be broadly spread in an application, and my speculation is that they are the result of a framework or near-universally used library. These are the most common issues. (though not in the dataset I reported yesterday.) They tend to be "vendor" or "open source framework" issues. 2. Transliteral transcodings (a=A because they look the same) including: + in Unicode terms "whole & mixed-script confusables" + in language terms "false-familiars" + as Chris described "best-fit mappings". All names == the same. Which, incidentally, is the problem. :) These are usually found in one specific location, usually specific to a function or set of functions in an application, from what I see. My further speculation is that they are the result of a specific library (or emergent behavior due to a specific combination of libraries) to facilitate a specific function in that location. Chris addressed most of the "why" in his response, and I agree most of what Chris said. These are, to a degree, Unicode problems insomuch as I think their recommendations cause some of them. To solve the "confusables" and "false-familiars" problems that are leveraged for multiple types of fraudulent and criminal activity (phishing, luring, fraud and stalking on social networks) Unicode recommended a set of practices to minimize or avoid name-collisions for Security's Sake: Unicode Security Considerations http://unicode.org/reports/tr36/ Unicode Security Mechanisms http://unicode.org/reports/tr39/ These security recommendations lead to another set of security issues: the increase in available characters that can be used to launch any given type of syntax attack. These are the next most-common issues we see of the 3 types. These happen for a variety of reasons: sometimes custom coding, and increasingly in Europe we see libraries and functions baked into frameworks and packages to do these types of things, sometimes transparently to the end-developer using the framework. 3. Bizarre Behavior By Interpreters: And some are simply the result of interpreter bugs/unfathomable behavior. None of the examples I have given so far are ones that I put in the #3 category. Yosuke Hasegawa linked in Jeremiah's blog post on this gives some really good examples of what I mean here. Namely -- sometimes interpreters (browsers like IE) do really weird or unsafe things for no clear good reason. I spend very little time researching these as my focus is on web software and not compiled code interpreters, so I/we/WhiteHat tend to only find these through accident. There is a fuzzy line here as some interpreter bugs allow you to exploit an application with #3, but it's not my specialty. At WhiteHat we have been heavily researching #1 and #2 and actually have a wealth of exploit data to share. The problem was bigger than we thought so have continued to expand the scope and range of our testing in these areas, which sometimes allow you to stumble upon a #3 issue. These are all simply my speculations...to be clear. I do not feign expertise on this subject like I do when it comes to motorcycles, mistresses, and martinis. --- Arian Evans ---------------------------------------------------------------------------- Join us on IRC: irc.freenode.net #webappsec Have a question? Search The Web Security Mailing List Archives: http://www.webappsec.org/lists/websecurity/archive/ Subscribe via RSS: http://www.webappsec.org/rss/websecurity.rss [RSS Feed] Join WASC on LinkedIn http://www.linkedin.com/e/gis/83336/4B20E4374DBA _______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Current thread:
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 04)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Prasad Shenoy (Jun 04)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 04)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Thierry Zoller (Jun 05)
- Re: [WEB SECURITY] Re[2]: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Chris Weber (Jun 05)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 06)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Chris Weber (Jun 05)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 04)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Prasad Shenoy (Jun 04)
- Message not available
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 06)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Chris Weber (Jun 07)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 07)
- Re: [WEB SECURITY] Unicode Left/Right Pointing Double Angel Quotation Mark bypass? Arian J. Evans (Jun 06)