Re: Combatting automated download of dynamic websites?

[bugtraq () cgisecurity net]

http://www.google.com/search?hl=en&q=apache+prevent+image+hotlinking&btnG=Google+Search
http://www.alistapart.com/articles/hotlinking/

Also check out mod_throttle.
http://www.snert.com/Software/mod_throttle/

- zeno
http://www.cgisecurity.com

> Which preventive or repressive measures could one apply to protect
> larger dynamic websites against automated downloading by tools such as
> WebCopier and Teleport Pro (or curl, for that matter)?
>
> For a website like Amazon's, I reckon some technical measures would be
> in place to protect against 'leakage' of all product information by such
> tools (assuming such measures are justified by calculated risk). The
> data we publish online are important company gems which we want to be
> accessible by any visitor, but to be protected against systematic
> download in either non-intentional context (like Internet Explorer's
> built-in MSIECrawler) or intentional context (WebCopier, Teleport, ...).
>
> Consider this:
> detailpage.html?bid=0000001
> detailpage.html?bid=0000002
> detailpage.html?bid=0000003
> (...)
>
> Or with multiple levels:
> detailpage.html?bid=0000001&t=1
> detailpage.html?bid=0000001&t=2
> detailpage.html?bid=0000002&t=1
> detailpage.html?bid=0000002&t=2
> (...)
>
> In specific, I was wondering if it's possible and sensible to limit the
> allowed number of requests for certain pages per minute/hour. At the
> same time, the data displayed by detailpage.html should be indexable by
> Google, so the data itself can't be hidden behind a user login and it's
> not possible to use any client-side scripting as Google doesn't
> interpret it. I'm using Apache 2 on RedHat 4 Enterprise and know about
> mod_throttle (which doesn't work with Apache 2) and mod_security (which
> also offers some 'throttling' functionality, regression, but is only
> able to work with individual requests and can't remember request sequences).
>
> I'd also suppose that dealing with proxy servers of large ISPs, like
> AOL, is a big caveat.
>
> Any ideas?
>
> Best regards,
> Matthijs
>
> --------------ms060404070107050703030905
> Content-Type: application/x-pkcs7-signature; name="smime.p7s"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment; filename="smime.p7s"
> Content-Description: S/MIME Cryptographic Signature
>
> MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIII6TCC
> As8wggI4oAMCAQICAw875DANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE
> ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv
> bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwODAyMDc0MTI1WhcNMDYwODAyMDc0MTI1
> WjBDMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMSAwHgYJKoZIhvcNAQkBFhFt
> YXR0aGlqc0Brb290LmJpejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOYHFw4g
> ikLak42UMhxWsIOP+hUCIkF1BKdpjPRI1kRpnHj1coXFK2Gr70lf0kohAz2H1kh5VohpN87g
> BPiB7OOw0JSnvmXRDOqlTHGvqOtFblI8JZCzNNRuLkIGYFM24V1PGRtGNi5bFytXM5JcsI7I
> /6EO4/FQxyVrH62dBUBCwu6gXd+AMtK8ethQzA8mZrrht5k5g5HCUiVn+XhiXmt0kjH2Bf/a
> SQlf406PLDr3Kq8D9V4xj27spDweFmTorYnfozxX6a3h/nMz+wXYLmUyXcKScudFQP50ni8r
> GPf0KgNFSp2EetZ97/n3vO8+tVDk4fNr8t10zN6HEyUSluUCAwEAAaMuMCwwHAYDVR0RBBUw
> E4ERbWF0dGhpanNAa29vdC5iaXowDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQQFAAOBgQCq
> 62sBc3tVFkR8JIfpoEm7hH09hdiAnJDf7/fek/vt6d/hhvqD/Wv7bSWoRgGF8N8lUaKr2dtZ
> 52pFJrm0K2Oy7cCCmCd/78wRlnrHBuOOyM+eUroGVtRYL2n2PuLapf4pQsXut7K1XNAQY5lx
> z2PJvdtqffnxxNbobi/1+/gNljCCAs8wggI4oAMCAQICAw875DANBgkqhkiG9w0BAQQFADBi
> MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEs
> MCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwODAy
> MDc0MTI1WhcNMDYwODAyMDc0MTI1WjBDMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVt
> YmVyMSAwHgYJKoZIhvcNAQkBFhFtYXR0aGlqc0Brb290LmJpejCCASIwDQYJKoZIhvcNAQEB
> BQADggEPADCCAQoCggEBAOYHFw4gikLak42UMhxWsIOP+hUCIkF1BKdpjPRI1kRpnHj1coXF
> K2Gr70lf0kohAz2H1kh5VohpN87gBPiB7OOw0JSnvmXRDOqlTHGvqOtFblI8JZCzNNRuLkIG
> YFM24V1PGRtGNi5bFytXM5JcsI7I/6EO4/FQxyVrH62dBUBCwu6gXd+AMtK8ethQzA8mZrrh
> t5k5g5HCUiVn+XhiXmt0kjH2Bf/aSQlf406PLDr3Kq8D9V4xj27spDweFmTorYnfozxX6a3h
> /nMz+wXYLmUyXcKScudFQP50ni8rGPf0KgNFSp2EetZ97/n3vO8+tVDk4fNr8t10zN6HEyUS
> luUCAwEAAaMuMCwwHAYDVR0RBBUwE4ERbWF0dGhpanNAa29vdC5iaXowDAYDVR0TAQH/BAIw
> ADANBgkqhkiG9w0BAQQFAAOBgQCq62sBc3tVFkR8JIfpoEm7hH09hdiAnJDf7/fek/vt6d/h
> hvqD/Wv7bSWoRgGF8N8lUaKr2dtZ52pFJrm0K2Oy7cCCmCd/78wRlnrHBuOOyM+eUroGVtRY
> L2n2PuLapf4pQsXut7K1XNAQY5lxz2PJvdtqffnxxNbobi/1+/gNljCCAz8wggKooAMCAQIC
> AQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENh
> cGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAm
> BgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0
> ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1h
> aWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNV
> BAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQD
> EyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEF
> AAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B
> 1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79A
> gAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8E
> CDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3
> dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEa
> MBgGA1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7M
> DaNmrGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUa
> C4J0niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk1
> 3iSx0x1G/11fZU8xggM7MIIDNwIBATBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3
> dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl
> ZW1haWwgSXNzdWluZyBDQQIDDzvkMAkGBSsOAwIaBQCgggGnMBgGCSqGSIb3DQEJAzELBgkq
> hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA1MDgyOTA4MTgzMVowIwYJKoZIhvcNAQkEMRYE
> FMgQWAsDJ+yguLCuvoprrZuMV+pqMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYI
> KoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMHgG
> CSsGAQQBgjcQBDFrMGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0
> aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1
> aW5nIENBAgMPO+QwegYLKoZIhvcNAQkQAgsxa6BpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQK
> ExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29u
> YWwgRnJlZW1haWwgSXNzdWluZyBDQQIDDzvkMA0GCSqGSIb3DQEBAQUABIIBAB6KGoIDp9+1
> dhzgVdKRuwLuMDiWxs1IbULiJKLlzDNrSXy23Wbvq5Jg0xX05BXhMFgwdsDdgNPc5lBR94dD
> smdQiJv9+nN0CNtw9fCBuaZBrrZJ0x+BPtLrguwR1TTMmXd7r1KnB9nC2nW39LPs+5PYq/C6
> x/1dsHGMb4YTf1P7Kj2FbzKW39NbQz4g4LBo7DayHKSgps1i0ww5UcMqopIyqVrGspTW9w57
> /5Aup5H7kkGpJE39BrYWmRCP674+BizaXw4AXnDOwnRIRgrGYyrszj9Ksiv9yrP10WJC9fXq
> c40GJ4ewgMPOJ/UBNBdLKR1saP1nOsnasD97yOgwuuoAAAAAAAA=
> --------------ms060404070107050703030905--