Sending generic HTTP(S) requests in python

During Web Application Penetration tests I always need to automate requests, e.g. for fuzzing. While most of the local proxy/testing softwares (Burp, WebScarab, w3af, etc.) include a repeater/fuzzer feature, I often want to do addtional computations in python (e.g. calculating a hash and sending it as a fuzzed value or comparing parts of the response). The following script will take an entire HTTP(S) request as a string, parse it and send it to the server. As I show with the POST parameter “fuzzableParam” in this example, values can easily be fuzzed.

def send_this_request(http_request_string, remove_headers=None):
    """
    Always HTTP/1.1
    """
    import urllib2
    if remove_headers is None:
        remove_headers=['content-length', 'accept-encoding', 'accept-charset', 
        'accept-language', 'accept', 'keep-alive', 'connection', 'pragma', 
        'cache-control']
    for i, remove_header in enumerate(remove_headers):
        remove_headers[i] = remove_header.lower()
    if '\n\n' in http_request_string:
        headers, body = http_request_string.split('\n\n',1)
    else:
        headers = http_request_string
        body = None
    headers = headers.split('\n')
    request_line = headers[0]
    headers = headers[1:]

    method, rest = request_line.split(" ", 1)
    url, protocol = rest.rsplit(" ", 1)

    merge_host_header_into_url = False
    if url.startswith("http"):
        merge_host_header_into_url = False
    elif url.startswith("/"):
        info("Warning: Defaulting to HTTP. Please write URL as https:// if you want SSL")
        merge_host_header_into_url = True
    else:
        fatalError("Protocol not supported. URL must start with http or /")

    header_tuples = []
    for header in headers:
        name, value = header.split(": ", 1)
        if merge_host_header_into_url and name.lower() == 'host':
            url = 'http://'+value+url
        if not name.lower() in remove_headers:
            header_tuples.append((name, value))
            
    opener = urllib2.build_opener()
    opener.addheaders = header_tuples
    urllib2.install_opener(opener)

    try:
        return urllib2.urlopen(url, body, 15).read()
    except urllib2.HTTPError, e:
        info('The server couldn\'t fulfill the request. Error code:', e.code)
    except urllib2.URLError, e:
        info("URLError:", e.reason)
    except Exception, e:
        error("DIDNT WORK:", e)
        
def info(*text):
    print "[PY-INFO] "+str(" ".join(str(i) for i in text))

def error(*text):
    print "[PY-ERROR] "+str(" ".join(str(i) for i in text))

request = '''POST http://example.com/ HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Content-Type: application/x-www-form-urlencoded;charset=utf-8
Referer: http://example.com
Content-Length: 132
Cookie: test=somevalue; abc=123
DNT: 1
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

id=123&fuzzableParam='''

additionalValue = "&anotherParam=abc"

for i in ['78', '-1']:
    print send_this_request(request+i+additionalValue)

Ack-All-Happy-Scapy – Finding a hole in a corporate firewall

When being located in a corporate environment (internal network), it is sometimes interesting to know if there are ports that are not outbound filtered, or in other words, if there is a hole where an attacker could connect to the outside world (damn perimeter-security). For example Apple products need port 5223 to be open for push notifications. So if the iPhones and iPads of managers should work, you have to open that outbound port 😀 . Of course you can simply chose one of those ports for your reverse shell when you take over one of their web servers in a later step. So what’s the easiest way to check if there is an open port, apart from knowing that they use the Apple push notification?

The following script can be run on every server, that has a public IP and Python/Scapy installed. When this script is running, it will send back a TCP SYN/ACK to every SYN coming from outside. It doesn’t matter which port. So if you do a NMAP SYN-Scan (-sS switch), all ports will be shown as open. Unless the corporate firewall between you and the server is blocking the SYN probes. So simply do a nmap SYN-Scan from the internal network of the company to the server and each open port is an open outbound port (unless there is some more filtering active such as deep packet inspection).

#!/usr/bin/python
# -*- coding: utf-8 -*-
DEBUG_ON=False
def ack-all-happy-scappy():
    from scapy.all import sniff, send, Ether, IP, TCP
    import os
    #################
    #CONFIG OPTIONS
    #################
    
    #Standard options
    my_ip = "xxx.xxx.xxx.xxx" #your external IP
    my_interface = "eth0"
    exclude_ports = ["22"] # Exclude ports, that already have a service running 22 = SSH,
    DEBUG_ON = False
    
    #Advanced options
    static_seq = 1337 #Specify as None for random seq number
    start_iptables_command = "iptables -A OUTPUT -p tcp --tcp-flags RST RST -j DROP"
    end_iptables_command = "iptables -D OUTPUT -p tcp --tcp-flags RST RST -j DROP"
    
    #################
    #CONFIG END
    #################
    
    #Actual code start
    if os.geteuid() != 0:
      info("You must be root to run this script.")
      sys.exit(1)    
    
    info("##################################")
    info("The ACK-ALL-HAPPY-SCAPY script, written by floyd")
    info("This script can only be used with SYN-scans (nmap -sS)")
    info("Altough untested, this should work as well for IPv6")
    info("##################################")
    sleep(3)
    info("This is how the IPTABLES looked, before starting ACK-ALL-HAPPY-SCAPY:")
    executeInShell("iptables -L")
    
    def getSeqNumber():
        if static_seq:
            return static_seq
        else:
            import random
            return random.randint(1,4294967295)
        
    def handleEachSyn(synpacket):
        if DEBUG_ON:
            debug("IN:")
            synpacket.display()
        ethlen = len(Ether())
        iplen = len(IP())
        synpacket_raw = str(synpacket)
        i = IP(synpacket_raw[ethlen:])
        t = TCP(synpacket_raw[ethlen + iplen:])
        f = IP(src=i.dst, dst=i.src)/TCP(sport=t.dport, dport=t.sport, ack=t.seq+1, seq=getSeqNumber())
        if DEBUG_ON:
            debug("OUT:")
            f.display()
        send(f)
        
    try:
        #Setup
        info("Executing now:", start_iptables_command)
        executeInShell(start_iptables_command)
        info("Done!")
        #Work
        not_port_filter = " and not port "+" and not port ".join(exclude_ports)
        filter_string = 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) = 0 and dst '+my_ip+not_port_filter
        info("Using filter ", filter_string)
        info("Waiting for your scans on tcp ports 1-65535, except "+", ".join(exclude_ports)+", where already a real service should be waiting")
        info("Start your scan with: sudo nmap -PN -sS -p 1-65535 "+my_ip)
        sniff(filter=filter_string, iface=my_interface, prn=handleEachSyn)
    except KeyboardInterrupt:
        #Restoring
        info()
        info("You pressed Ctrl+C... please wait, restoring IPTABLES")
        info("Executing now:", end_iptables_command)
        for i in range(3):
            executeInShell(end_iptables_command)
        info("This is how the IPTABLES looks, after finishing ACK-ALL-HAPPY-SCAPY:")
        executeInShell("iptables -L")

def executeInShell(command):
    import subprocess
    process = subprocess.Popen(command, shell=True)
    process.wait()

def sleep(seconds):
    import time
    time.sleep(seconds)

def info(*text):
    print "[PY-INFO] "+str(" ".join(str(i) for i in text))

def debug(*text):
    if DEBUG_ON:
        print "[PY-DEBUG] "+str(" ".join(str(i) for i in text))

main()

Today it shouldn’t be a big problem to start this script on your server, even when you can’t use your corporate network internet access. Just use your mobile phone to connect to the server and start the script.

Btw, Scapy is one of the most amazing Python libraries I’ve ever seen. Extremely powerful.

DNS zone transfer

Today I thought it would be cool to have a list of all domains that exist in Switzerland. As it turns out, the swiss registrar (Switch) has configured their nameservers correctly, so you can not do a DNS zone transfer 🙁 . But I found out that a lot of other TLDs allow to make zone transfers. I don’t know if its on purpose, but I don’t think so, because not all of their DNS root servers allow to do the transfer… Try it yourself (bash script):

tlds="AC AD AE AERO AF AG AI AL AM AN AO AQ AR ARPA AS ASIA AT AU AW AX AZ BA BB BD BE BF BG BH BI BIZ BJ BM BN BO BR BS BT BV BW BY BZ CA CAT CC CD CF CG CH CI CK CL CM CN CO COM COOP CR CU CV CX CY CZ DE DJ DK DM DO DZ EC EDU EE EG ER ES ET EU FI FJ FK FM FO FR GA GB GD GE GF GG GH GI GL GM GN GOV GP GQ GR GS GT GU GW GY HK HM HN HR HT HU ID IE IL IM IN INFO INT IO IQ IR IS IT JE JM JO JOBS JP KE KG KH KI KM KN KP KR KW KY KZ LA LB LC LI LK LR LS LT LU LV LY MA MC MD ME MG MH MIL MK ML MM MN MO MOBI MP MQ MR MS MT MU MUSEUM MV MW MX MY MZ NA NAME NC NE NET NF NG NI NL NO NP NR NU NZ OM ORG PA PE PF PG PH PK PL PM PN PR PRO PS PT PW PY QA RE RO RS RU RW SA SB SC SD SE SG SH SI SJ SK SL SM SN SO SR ST SU SV SY SZ TC TD TEL TF TG TH TJ TK TL TM TN TO TP TR TRAVEL TT TV TW TZ UA UG UK US UY UZ VA VC VE VG VI VN VU WF WS XN XXX YE YT ZA ZM ZW"
    
for tld in $tlds
do
   echo "Doing TLD $tld"
   for f in `dig ns $tld. | grep "NS" | cut -f 7 | grep "$tld." | grep -v "ANSWER"`
   do
       echo "$tld : $f"
       dig axfr $tld @$f >> output.txt
   done
done

For me it worked for the following TLDs: an, bi, ci, cr, er, et, ga, ge, gy, jm, km, mc, mm, mo, mw, ni, np, pg, pro, sk, sv, tt, uk, uy, ye, zw. Might change in the future. For me the winner is… Slovakia (sk)! Never seen so many DNS entries in one file 😀

Update: I just uploaded my results here. When I talked to Max he decided to put his treasures (.DE for example!) up as well, you’ll find his domains here.

Detect shared hosting with Bing

Bing has a pretty cool IP advanced search operator. It can be used to detect shared hosting. It is quite annoying to type in each IP manually when you have to check several IPs (e.g. corporate IP network). The following script will scan an entire range of IPs. The examples below (Google IPs 74.125.39.103 to 74.125.39.106) give some pretty interesting results…

def printBingSharedHosting(ip_start, ip_end):
    import urllib2
    import re
    
    bing_url = 'http://www.bing.com/search?q=ip%3A'
    no_results_string = "No results"
    
    def bing_shared_hosting_get_matches(response):
        regex = '<h3><a\shref="(.*?)"\sonmousedown="'
        mo = re.finditer(regex, response)
        urls = []
        for i in mo:
            urls.extend(i.groups())
        return urls
    
    def bing_shared_hosting_query_bing(ip_str):
        body = urllib2.urlopen(bing_url+ip_str).read()
        if not no_results_string in body:
            for i in bing_shared_hosting_get_matches(body):
                result(ip_str+" : "+i)
        else:
            error("Bing page did not show '"+no_results_string+"' for ip: "+ip_str)

    info("Starting bing shared hosting search. I'm not printing anything until i find something")
    info("This method is only searching through the first result page of bing!")    
    for ip in getIpRangeList(ip_start, ip_end):
        bing_shared_hosting_query_bing(ip)


def getIpRangeList(ip_start, ip_end):

    result_list = []

    one, two, three, four = ip_start.split('.')
    one, two, three, four = (int(one), int(two), int(three), int(four))
    end_one, end_two, end_three, end_four = ip_end.split('.')
    end_one, end_two, end_three, end_four = (int(end_one), int(end_two), int(end_three), int(end_four))

    while one <= end_one:
        end_two_tmp = end_two
        if not one == end_one:
            end_two_tmp = 255
        while two <= end_two_tmp:
            end_three_tmp = end_three
            if not two == end_two:
                end_three_tmp = 255
            while three <= end_three_tmp:
                end_four_tmp = end_four
                if not three == end_three:
                    end_four_tmp = 255
                while four <= end_four_tmp:
                    result_list.append("%i.%i.%i.%i"%(one, two, three, four))
                    #debug(str(one)+" "+str(two)+" "+str(three)+" "+str(four))
                    four += 1
                four = 0
                three += 1
            three = 0
            two += 1
        two = 0
        one += 1
    
    return result_list

   
def error(*text):
    print "[PY-ERROR] "+str(" ".join(str(i) for i in text))
    if SLEEP_TIME_ON_ERROR > 0:
        sleep(SLEEP_TIME_ON_ERROR)

def result(*text):
    print "[PY-RESULT] "+str(" ".join(str(i) for i in text))

def info(*text):
    print "[PY-INFO] "+str(" ".join(str(i) for i in text))

def debug(*text):
    print "[PY-DEBUG] "+str(" ".join(str(i) for i in text))

printBingSharedHosting("74.125.39.103", "74.125.39.106")

I admit, the getIpRangeList function could be a little bit more elegant, but I didn’t want to use an external library, didn’t find any suitable code snippet and in the end, it does its job.