Re: [NSE] http-feed.nse

[David Fifield <david () bamsoftware com>]

On Sat, Aug 17, 2013 at 06:16:57PM +0300, George Chatzisofroniou wrote:
> On Fri, Aug 16, 2013 at 08:38:21PM -0700, David Fifield wrote: 
> > It didn't find feeds at secwiki.org. It doesn't seem to understand this
> > markup on the home page.
> >
> > <link rel="alternate" type="application/atom+xml" title="SecWiki Atom feed" 
> > href="/mediawiki/index.php?title=Special:RecentChanges&amp;feed=atom" />
>
> What is special with this reference is that it contains an encoded ampersand.
> For now, i updated the script to decode "&amp" to "&", but this is something
> that should be done in httpspider and http libraries.

To be clear, the &amp; is not the reason the feed wasn't detected with
the previous version of the script, right? It was because the script
lacked an "application/atom+xml" pattern.

The right way to get rid of the &amp; is with an HTML parser. Since we
don't have that, I think I would prefer that we not interpret the string
at all in the script. If we handle &amp;, we really should handle &lt;
and &gt; and especially &quot; and &apos; that are likely to appear in
attribute values. (Granted, &amp; is the most likely and most
problematic of all of these.) But there are also numeric character
entities and the large number of HTML named entities too.

Decoding just &amp; and nothing else creates ambiguities, for example
both of the input strings
        http://example.com/?a=b&quot;=d
        http://example.com/?a=b&amp;quot;c=d
map to the same output string
        http://example.com/?a=b&quot;=d

I agree with you that it shouldn't be handled by just this script, in
just this place.

In HTML5 there are some rules about when an ampersand is just an
ampersand and when it is part of a character reference.
http://www.whatwg.org/specs/web-apps/current-work/multipage/syntax.html#syntax-ambiguous-ampersand
http://www.w3.org/International/questions/qa-escapes#bytheway

David Fifield
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/