WebApp Sec mailing list archives
Re: Combatting automated download of dynamic websites?
From: bugtraq () cgisecurity net
Date: Mon, 29 Aug 2005 11:50:52 -0400 (EDT)
http://www.google.com/search?hl=en&q=apache+prevent+image+hotlinking&btnG=Google+Search http://www.alistapart.com/articles/hotlinking/ Also check out mod_throttle. http://www.snert.com/Software/mod_throttle/ - zeno http://www.cgisecurity.com
Which preventive or repressive measures could one apply to protect larger dynamic websites against automated downloading by tools such as WebCopier and Teleport Pro (or curl, for that matter)? For a website like Amazon's, I reckon some technical measures would be in place to protect against 'leakage' of all product information by such tools (assuming such measures are justified by calculated risk). The data we publish online are important company gems which we want to be accessible by any visitor, but to be protected against systematic download in either non-intentional context (like Internet Explorer's built-in MSIECrawler) or intentional context (WebCopier, Teleport, ...). Consider this: detailpage.html?bid=0000001 detailpage.html?bid=0000002 detailpage.html?bid=0000003 (...) Or with multiple levels: detailpage.html?bid=0000001&t=1 detailpage.html?bid=0000001&t=2 detailpage.html?bid=0000002&t=1 detailpage.html?bid=0000002&t=2 (...) In specific, I was wondering if it's possible and sensible to limit the allowed number of requests for certain pages per minute/hour. At the same time, the data displayed by detailpage.html should be indexable by Google, so the data itself can't be hidden behind a user login and it's not possible to use any client-side scripting as Google doesn't interpret it. I'm using Apache 2 on RedHat 4 Enterprise and know about mod_throttle (which doesn't work with Apache 2) and mod_security (which also offers some 'throttling' functionality, regression, but is only able to work with individual requests and can't remember request sequences). I'd also suppose that dealing with proxy servers of large ISPs, like AOL, is a big caveat. Any ideas? Best regards, Matthijs --------------ms060404070107050703030905 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIII6TCC As8wggI4oAMCAQICAw875DANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwODAyMDc0MTI1WhcNMDYwODAyMDc0MTI1 WjBDMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMSAwHgYJKoZIhvcNAQkBFhFt YXR0aGlqc0Brb290LmJpejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOYHFw4g ikLak42UMhxWsIOP+hUCIkF1BKdpjPRI1kRpnHj1coXFK2Gr70lf0kohAz2H1kh5VohpN87g BPiB7OOw0JSnvmXRDOqlTHGvqOtFblI8JZCzNNRuLkIGYFM24V1PGRtGNi5bFytXM5JcsI7I /6EO4/FQxyVrH62dBUBCwu6gXd+AMtK8ethQzA8mZrrht5k5g5HCUiVn+XhiXmt0kjH2Bf/a SQlf406PLDr3Kq8D9V4xj27spDweFmTorYnfozxX6a3h/nMz+wXYLmUyXcKScudFQP50ni8r GPf0KgNFSp2EetZ97/n3vO8+tVDk4fNr8t10zN6HEyUSluUCAwEAAaMuMCwwHAYDVR0RBBUw E4ERbWF0dGhpanNAa29vdC5iaXowDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQQFAAOBgQCq 62sBc3tVFkR8JIfpoEm7hH09hdiAnJDf7/fek/vt6d/hhvqD/Wv7bSWoRgGF8N8lUaKr2dtZ 52pFJrm0K2Oy7cCCmCd/78wRlnrHBuOOyM+eUroGVtRYL2n2PuLapf4pQsXut7K1XNAQY5lx z2PJvdtqffnxxNbobi/1+/gNljCCAs8wggI4oAMCAQICAw875DANBgkqhkiG9w0BAQQFADBi MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEs MCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwODAy MDc0MTI1WhcNMDYwODAyMDc0MTI1WjBDMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVt YmVyMSAwHgYJKoZIhvcNAQkBFhFtYXR0aGlqc0Brb290LmJpejCCASIwDQYJKoZIhvcNAQEB BQADggEPADCCAQoCggEBAOYHFw4gikLak42UMhxWsIOP+hUCIkF1BKdpjPRI1kRpnHj1coXF K2Gr70lf0kohAz2H1kh5VohpN87gBPiB7OOw0JSnvmXRDOqlTHGvqOtFblI8JZCzNNRuLkIG YFM24V1PGRtGNi5bFytXM5JcsI7I/6EO4/FQxyVrH62dBUBCwu6gXd+AMtK8ethQzA8mZrrh t5k5g5HCUiVn+XhiXmt0kjH2Bf/aSQlf406PLDr3Kq8D9V4xj27spDweFmTorYnfozxX6a3h /nMz+wXYLmUyXcKScudFQP50ni8rGPf0KgNFSp2EetZ97/n3vO8+tVDk4fNr8t10zN6HEyUS luUCAwEAAaMuMCwwHAYDVR0RBBUwE4ERbWF0dGhpanNAa29vdC5iaXowDAYDVR0TAQH/BAIw ADANBgkqhkiG9w0BAQQFAAOBgQCq62sBc3tVFkR8JIfpoEm7hH09hdiAnJDf7/fek/vt6d/h hvqD/Wv7bSWoRgGF8N8lUaKr2dtZ52pFJrm0K2Oy7cCCmCd/78wRlnrHBuOOyM+eUroGVtRY L2n2PuLapf4pQsXut7K1XNAQY5lxz2PJvdtqffnxxNbobi/1+/gNljCCAz8wggKooAMCAQIC AQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENh cGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAm BgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0 ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1h aWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNV BAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQD EyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEF AAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B 1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79A gAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8E CDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3 dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEa MBgGA1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7M DaNmrGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUa C4J0niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk1 3iSx0x1G/11fZU8xggM7MIIDNwIBATBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3 dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJl ZW1haWwgSXNzdWluZyBDQQIDDzvkMAkGBSsOAwIaBQCgggGnMBgGCSqGSIb3DQEJAzELBgkq hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA1MDgyOTA4MTgzMVowIwYJKoZIhvcNAQkEMRYE FMgQWAsDJ+yguLCuvoprrZuMV+pqMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYI KoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMHgG CSsGAQQBgjcQBDFrMGkwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0 aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1 aW5nIENBAgMPO+QwegYLKoZIhvcNAQkQAgsxa6BpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQK ExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29u YWwgRnJlZW1haWwgSXNzdWluZyBDQQIDDzvkMA0GCSqGSIb3DQEBAQUABIIBAB6KGoIDp9+1 dhzgVdKRuwLuMDiWxs1IbULiJKLlzDNrSXy23Wbvq5Jg0xX05BXhMFgwdsDdgNPc5lBR94dD smdQiJv9+nN0CNtw9fCBuaZBrrZJ0x+BPtLrguwR1TTMmXd7r1KnB9nC2nW39LPs+5PYq/C6 x/1dsHGMb4YTf1P7Kj2FbzKW39NbQz4g4LBo7DayHKSgps1i0ww5UcMqopIyqVrGspTW9w57 /5Aup5H7kkGpJE39BrYWmRCP674+BizaXw4AXnDOwnRIRgrGYyrszj9Ksiv9yrP10WJC9fXq c40GJ4ewgMPOJ/UBNBdLKR1saP1nOsnasD97yOgwuuoAAAAAAAA= --------------ms060404070107050703030905--
Current thread:
- Combatting automated download of dynamic websites? Matthijs R. Koot (Aug 29)
- Re: Combatting automated download of dynamic websites? Jayson Anderson (Aug 29)
- Re: Combatting automated download of dynamic websites? Serg Belokamen (Aug 29)
- Re: Combatting automated download of dynamic websites? bugtraq (Aug 29)
- Re: Combatting automated download of dynamic websites? Matthijs R. Koot (Aug 29)
- Re: Combatting automated download of dynamic websites? Javier Fernandez-Sanguino (Aug 30)
- Re: Combatting automated download of dynamic websites? Eoin Keary (Aug 31)
- Re: Combatting automated download of dynamic websites? Javier Fernandez-Sanguino (Sep 05)
- Re: Combatting automated download of dynamic websites? Matthijs R. Koot (Aug 29)
- Re: Combatting automated download of dynamic websites? Michael Boman (Aug 30)
- Re: Combatting automated download of dynamic websites? Paul M. (Sep 05)
- Re: Combatting automated download of dynamic websites? Eoin Keary (Sep 07)
- Re: Combatting automated download of dynamic websites? Jayson Anderson (Aug 29)
- <Possible follow-ups>
- Re: Combatting automated download of dynamic websites? Tony Stahler (Aug 30)
- Message not available
- Fwd: Combatting automated download of dynamic websites? Mark Quinn (Aug 31)
- Message not available