{"id":304,"date":"2011-09-27T10:38:35","date_gmt":"2011-09-27T09:38:35","guid":{"rendered":"http:\/\/www.floyd.ch\/?p=304"},"modified":"2011-08-03T10:49:49","modified_gmt":"2011-08-03T09:49:49","slug":"sending-generic-https-requests-in-python","status":"publish","type":"post","link":"https:\/\/www.floyd.ch\/?p=304","title":{"rendered":"Sending generic HTTP(S) requests in python"},"content":{"rendered":"<p>During Web Application Penetration tests I always need to automate requests, e.g. for fuzzing. While most of the local proxy\/testing softwares (Burp, WebScarab, w3af, etc.) include a repeater\/fuzzer feature, I often want to do addtional computations in python (e.g. calculating a hash and sending it as a fuzzed value or comparing parts of the response). The following script will take an entire HTTP(S) request as a string, parse it and send it to the server. As I show with the POST parameter &#8220;fuzzableParam&#8221; in this example, values can easily be fuzzed.<\/p>\n<pre class=\"brush: python; title: ; notranslate\" title=\"\">\r\ndef send_this_request(http_request_string, remove_headers=None):\r\n    &quot;&quot;&quot;\r\n    Always HTTP\/1.1\r\n    &quot;&quot;&quot;\r\n    import urllib2\r\n    if remove_headers is None:\r\n        remove_headers=&#x5B;'content-length', 'accept-encoding', 'accept-charset', \r\n        'accept-language', 'accept', 'keep-alive', 'connection', 'pragma', \r\n        'cache-control']\r\n    for i, remove_header in enumerate(remove_headers):\r\n        remove_headers&#x5B;i] = remove_header.lower()\r\n    if '\\n\\n' in http_request_string:\r\n        headers, body = http_request_string.split('\\n\\n',1)\r\n    else:\r\n        headers = http_request_string\r\n        body = None\r\n    headers = headers.split('\\n')\r\n    request_line = headers&#x5B;0]\r\n    headers = headers&#x5B;1:]\r\n\r\n    method, rest = request_line.split(&quot; &quot;, 1)\r\n    url, protocol = rest.rsplit(&quot; &quot;, 1)\r\n\r\n    merge_host_header_into_url = False\r\n    if url.startswith(&quot;http&quot;):\r\n        merge_host_header_into_url = False\r\n    elif url.startswith(&quot;\/&quot;):\r\n        info(&quot;Warning: Defaulting to HTTP. Please write URL as https:\/\/ if you want SSL&quot;)\r\n        merge_host_header_into_url = True\r\n    else:\r\n        fatalError(&quot;Protocol not supported. URL must start with http or \/&quot;)\r\n\r\n    header_tuples = &#x5B;]\r\n    for header in headers:\r\n        name, value = header.split(&quot;: &quot;, 1)\r\n        if merge_host_header_into_url and name.lower() == 'host':\r\n            url = 'http:\/\/'+value+url\r\n        if not name.lower() in remove_headers:\r\n            header_tuples.append((name, value))\r\n            \r\n    opener = urllib2.build_opener()\r\n    opener.addheaders = header_tuples\r\n    urllib2.install_opener(opener)\r\n\r\n    try:\r\n        return urllib2.urlopen(url, body, 15).read()\r\n    except urllib2.HTTPError, e:\r\n        info('The server couldn\\'t fulfill the request. Error code:', e.code)\r\n    except urllib2.URLError, e:\r\n        info(&quot;URLError:&quot;, e.reason)\r\n    except Exception, e:\r\n        error(&quot;DIDNT WORK:&quot;, e)\r\n        \r\ndef info(*text):\r\n    print &quot;&#x5B;PY-INFO] &quot;+str(&quot; &quot;.join(str(i) for i in text))\r\n\r\ndef error(*text):\r\n    print &quot;&#x5B;PY-ERROR] &quot;+str(&quot; &quot;.join(str(i) for i in text))\r\n\r\nrequest = '''POST http:\/\/example.com\/ HTTP\/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla\/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko\/20100101 Firefox\/4.0.1\r\nAccept: text\/html,application\/xhtml+xml,application\/xml;q=0.9,*\/*;q=0.8\r\nAccept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3\r\nAccept-Encoding: gzip, deflate\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nKeep-Alive: 115\r\nContent-Type: application\/x-www-form-urlencoded;charset=utf-8\r\nReferer: http:\/\/example.com\r\nContent-Length: 132\r\nCookie: test=somevalue; abc=123\r\nDNT: 1\r\nConnection: keep-alive\r\nPragma: no-cache\r\nCache-Control: no-cache\r\n\r\nid=123&amp;fuzzableParam='''\r\n\r\nadditionalValue = &quot;&amp;anotherParam=abc&quot;\r\n\r\nfor i in &#x5B;'78', '-1']:\r\n    print send_this_request(request+i+additionalValue)\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>During Web Application Penetration tests I always need to automate requests, e.g. for fuzzing. While most of the local proxy\/testing softwares (Burp, WebScarab, w3af, etc.) include a repeater\/fuzzer feature, I often want to do addtional computations in python (e.g. calculating &hellip; <a href=\"https:\/\/www.floyd.ch\/?p=304\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[18,19],"tags":[21,20],"class_list":["post-304","post","type-post","status-publish","format-standard","hentry","category-coding","category-useful-scripts","tag-http","tag-python"],"_links":{"self":[{"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/posts\/304","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=304"}],"version-history":[{"count":5,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/posts\/304\/revisions"}],"predecessor-version":[{"id":309,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=\/wp\/v2\/posts\/304\/revisions\/309"}],"wp:attachment":[{"href":"https:\/\/www.floyd.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.floyd.ch\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}