Nice. Would it make sense if I say the success of this might solely depend
on the order of the spoofed header? or that is obvious?

Different browsers have different order in which the request headers are
stacked. FF 8.0.1 added the actual system User-Agent header before the
custom request headers (User%Agent - created by XMLHttpRequest object).
Chrome and IE 7 both added the custom header before the actual header.

I did not see any ordering preferences in the RFCs.

[ ~ Prasad |  @prasadshenoy  ~]

On Thu, Jan 5, 2012 at 7:13 AM, Kousuke Ebihara kousuke@co3k.org wrote:

This is an artifact of the mismatch between the character set available for Unix shell environment variables and those legal in RFC 822 headers as used by HTTP.  The equivalence between at least - and _ is documented in RFC 3875.  http://tools.ietf.org/html/rfc3875#section-4.1.18

'Meta-variables with names beginning with "HTTP_" contain values read
from the client request header fields, if the protocol used is HTTP.
The HTTP header field name is converted to upper case, has all
occurrences of "-" replaced with "" and has "HTTP" prepended to
give the meta-variable name.'

That RFC is from 2004, but documents practices that began in 1993.

So I would expect this bug to be present on all CGI-style applications, and absent in frameworks that provide a better object mapping to HTTP values, e.g. servlets.

Given this is documented and standardized behavior, it seems that the XHR spec should be updated to compare at least "_" and possibly other characters not valid in Unix shell variables, as equivalent to "-" when blacklisting custom headers.

Brad Hill
Sr. MTS, Internet Standards and Governance
PayPal Information Risk Management
cell: 206.245.7844 / skype: hillbrad
email: bhill@paypal-inc.com

What were you able to do?

A couple things come to mind (assuming this works on single word headers which I've been unable to get working)

1. Modifying headers such as 'Host' to access other virtualhosts on the
   same ip, or breaking weak CSRF protections by modifying the 'Referer' header (http://www.securityfocus.com/archive/1/441014)

2. Potential shared virtual hosting browser cache poisoning?
   (http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/2008-June/003951.html)

3. Abusing transparent proxies via Host header modification
   (http://www.thesecuritypractice.com/the_security_practice/2010/03/abusing-transparent-proxies-with-flash-presentation-available-paper-update.html)

Regards,

• Robert A.
  http://www.webappsec.org/
  http://www.cgisecurity.com/
  http://www.qasec.com/

Actually scratch my last reply.

/me will drink some coffee before hitting the send button next time :)

On Thu, 5 Jan 2012, Robert A. wrote:

Perhaps it could be combined with the 307 flash header-forging
vulnerability (
http://lists.webappsec.org/pipermail/websecurity_lists.webappsec.org/2011-February/007533.html
) to forge user-agent headers cross domain; some sites use user-agents
as a weak CSRF protection (don't ask). Or another similar but unpatched
vulnerability.

albino

On Thu, Jan 5, 2012, at 09:13 PM, Kousuke Ebihara wrote:

Bypass authentication.

Many sites for Japanese feature phone implements weak authentication system which depends on user identifiers in HTTP header (e.g. X-Up-Subno).

The carrier takes measures to prevent spoofing the identifier.

One of them is a restricted XHR in its web browser. The XHR implementation denies sending "X-Up-Subno" and "X_Up_Subno" request header, but it allows "X.Up.Subno" header, so attacker could spoof the user identifier by using this technique.

Of course, authentication by only ID is too bad. However, in Japanese mobile sites, this "authenticate" method is popular for some reasons, so that vulnerability could create real threats.

--
Kousuke Ebihara kousuke@co3k.org
http://co3k.org/

Actually, at least in Unix, shell environment variables can have just
about any character, except for perhaps '\x00' and '=' (IIRC...
haven't looked at the storage format in a while).  For example:

$ env 'FOOBAR=BAZ' /bin/sh -c 'set' | grep FOO
FOOBAR='BAZ'

It's just that most shells restrict the characters you can use to set
these or access them.  For all I know, other scripting languages could
have similar restrictions.  So naturally when defining CGI, someone
felt the need to impose some sort of restrictions on variable names to
be compatible with existing scripting languages, including shell.

As a security guy, I find "squashing" characters down to a single safe
place-holder is always a bad idea, as we see it play out here with
namespace conficts.  A better approach would have been to come up with
an encoding/escaping scheme, or to just reject the names you don't
like up front.  But what is done is done.

When I first read Kousuke's email, I thought to myself that browsers
(in the XMLHttpRequest context) or web servers really ought to reject
those headers that contain '.' and '*', since those can't be valid
names.  As it turns out they are valid, based on the HTTP/1.1 spec.
Relevant excerpts from RFC2616:

   CHAR           = <any US-ASCII character (octets 0 - 127)>

...
CTL            = <any US-ASCII control character
(octets 0 - 31) and DEL (127)>
...
token          = 1*<any CHAR except CTLs or separators>
separators    = "(" | ")" | "<" | ">" | "@"
| "," | ";" | ":" | "" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT
...
message-header = field-name ":" [ field-value ]
field-name    = token
...

Somewhat surprisingly, all of the following are permitted in HTTP
header names:

!#$%&'*+-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz|~

I think it could vary widely from framework to framework, what is
squashed, permitted, and rejected.

I think given the clearly defined HTTP RFC, we're really looking at
age-old insecure practices in interpreting headers on the web server.
There's no reason that headers with permitted special characters
shouldn't be accessible through an appropriate API.  No good reason to
continue squashing and polluting the namespace.

tim

[Hill, Brad] I would say the opposite.  Given a documented, common practice that, however hacky, has been established for nearly 20 years, there is every reason for a new technology (or at least, relatively new) not to introduce vulnerabilities to a huge and established ecosystem.

I'm not saying XHR should actually coerce non-alphanumeric characters down when sending headers - only that the security checks for banned headers should do this for purposes of pre-flight comparison.  There is an obligation not to introduce new vulnerabilities into existing systems, and the "cost" to long-term compatibility with HTTP is very small: the blacklist becomes a set of simple regexes instead of string literals.

Brad Hill
Co-chair, WebAppSec WG

Well, you can either side with ancient de facto behavior and poor
choices made in a standard that is little used today (CGI), or you can
side with a very prevalent standard (HTTP).

Let us suppose that all browsers do implement work-arounds for these
de facto behaviors in CGIs, then we still don't address the other
problems created by this name squashing.

Consider an application that uses a front-end load balancer.  The
application wishes to restrict access to a particular set of external
IP addresses.  Since the load balancer terminates those connections,
the application needs the load balancer to pass on information about
what IP address connected.  This is normal done through
X-Forwarded-For or similar headers, added by the load balancer to the
request.

Of course this introduces a potential vulnerability right off, even
without problematic name squashing.  An attacker could connect from an
untrusted IP address and add his own X-Forwarded-For header.  Ok,
fine, we can address that by configuring the load balancer to strip
the X-Forwarded-For header from the intial request, and then add our
own.

But, if name squashing happens on the application server, then the
load balancer will have to strip "X_Forwarded-For", "X*Forwarded_For",
... ad nauseum.

This is just one specific example of the problem of name squashing.
Modern web frameworks should either reject HTTP header names with
special characters they don't like (nginx apparently does this), or
simply allow through any valid HTTP header names.  No squashing is
needed.

How would this harm web apps that actually rely on HTTP headers?

tim

On Fri, 6 Jan 2012, Hill, Brad wrote:

(disclaimer: I'm late to this discussion and haven't seen all the posts.)

The industry does this all the time, for example by accounting for
non-standard browser behavior with XSS defenses (as reflected in the XSS
cheat sheet, HTML5 Security Cheatsheet, etc).  Not that this has been
wildly successful given all the XSS variants that keep cropping up that
only apply to 1 or 2 browsers, but it seems like a reasonable approach
short of normalizing inputs/outputs in a predictable fashion
(non-standard), or modifying the standards to design out these kinds of
problems in the first place (unrealistic, infeasible, and maybe
impossible).  Anti-virus products, IDS, mail scanners, and other products
also have to wrestle with the same issue of accounting for
popular-but-non-standard behaviors in order to work properly.  Should they
be responsible for this kind of protection?  Philosophically speaking, no;
but operationally speaking, they have no choice.

• Steve

Hi Steve,

I appreciate what you're saying, and yes, many de facto hacks exist to
protect people.  The more I think about it, the more I think that
nearly all web frameworks probably do this header namespace squashing.
The squashing clearly creates security problems, as we have discussed.

Let us balance that then against the security down sides of making a
change to web frameworks.  If today, "X^MyHeader" and "X*MyHeader" are
treated as equivalent, what is the harm in not treating them as
equivalent in the future?  The only header an application can search
for is "X_MYHEADER".  If we make such a change, they're still going to
be looking up "X_MYHEADER".  Sure, applications sending that header
would need to be more careful about what format of the header they are
sending (such as sending "X_MyHeader" in all cases) , but that's not a
security issue, just a backward compatibility one.  How would the
application be harmed in a security sense?

Brad's suggested approach would be to start restricting the types of
headers that can be sent to begin with.  So if XMLHttpRequest simply
blocked the less common variants of these headers (essentially
rejecting "Accept[!#$%&'*+-._^`|~]Encoding", and so on), then great,
we've prevented these headers from working their way in to these
requests at the web layer.  But as I've pointed out, there are other
situations where this approach would need to be taken.  Essentially,
to enforce it, all servers would need to start rejecting the uncommon
characters.  This amounts to changing the HTTP spec.  From a security
perspective, this clearly prevents a lot of problems, but changing
HTTP header names from a permissive set of characters to a restrictive
set is difficult.  Any legacy servers will continue to be "vulnerable"
in a sense, basically forever.

Both changes are very difficult from an implementation perspective,
but if there are no security downsides to fixing web frameworks
(allowing most/all HTTP headers), then that seems like the better
approach to me, not to mention easier.

tim

Looks like this is a quite old issue, dating to 2007 with Flash:

http://kuza55.blogspot.com/2007/07/exploiting-reflected-xss.html

Adobe patched, it but the W3C editors for XHR won't:

http://lists.w3.org/Archives/Public/public-webapps/2012JanMar/0024.html

In my testing, Apache did not handle all "bypassed" headers the same
way. For example, it was not possible to submit a modified
"Transfer-Encoding" header (eg. Transfer.Encoding; Transfer*Encoding;
Transfer Encoding), and have Apache handle the request with the
specified encoding.

The environment variable HTTP_TRANSFER_ENCODING is still set, but the
actual request isn't treated with the provided encoding.

If you could get the browser to send a Transfer*Encoding Header, and
if Apache treats it like normal, then you could possibly abuse it for
Request Splitting/Smuggling.
http://www.mindedsecurity.com/MSA01240108.html

On Thu, Jan 5, 2012 at 4:13 AM, Kousuke Ebihara kousuke@co3k.org wrote:

Ack.  I am learning PHP today to write some test cases and I find that PHP also uses the "-" to "_" conversion before exposing HTTP headers, so this server vulnerability is much more widespread than just ancient CGIs.  :(

Yeah, after our conversation a while back, I looked into a bit more
and realized that just about every web framework does this foolish
transliteration of characters to _.  Almost every special character is
converted like this.  The CGI spec only requires - be transliterated
to _ and doesn't require anything else be.

Of course those other frameworks don't have to adhere to CGI, but even
if their developers wanted to claim "oh, we're just being backward
compatible with CGI", they don't have a sound argument.

tim

In your investigation did you find an instance of a framework converting single header names
such as _Referer or _Host?

Regards,

• Robert

On Thu, 2 Feb 2012, Tim wrote: