Historical stock data with curl

[AureliusMarcusHu]

hi @alvarobartt ,

Let us try a new alternative with curl instead of requests. Curl pass the cloudflare detection without any problems.
It's just a different approach, but I don't get the form data right. Maybe someone have a idea how to post the query data (from_date, to_date). All suggestions are welcome.
Run the test script below and check it, the script will print the url which you can use in chrome or any other browser as well:

from subprocess import run, PIPE
from htmlement import HTMLement
from json import dumps


def get_data(tabel):
    try:
        thead = tabel.find("thead")
        rhd = thead.find("tr")
        header = []
        cnt = 0
        for th in rhd:
            tmp = th.find('div').find('button').find('span')
            header.append(tmp.text)
            cnt += 1
        print(header)
        
        body = tabel.find("tbody")
        for tr in body.findall("tr"):
            row = []
            cnt = 0
            for td in tr:
                if cnt == 0:
                    tijd = td.find('time')
                    row = [tijd.text]
                else:
                    row.append(td.text)
                cnt += 1
            print(row)
            
    except:
        print('table ; data ; invalid')
    return


def get_document(val):
    url = f'https://www.investing.com{val}-historical-data'
    print('url:', url)
    params = {"from_date": "07/01/2024", "to_date": "08/31/2024", "interval": "Daily"}
    cnfg = ['curl.exe', '-s', '-A ' + 'Chrome/91.0.4472.114', '-H Content-Type: application/json', '-d ' + dumps(params), url]
    print('config:', cnfg, chr(10))

    htm_doc = run(cnfg, stdout=PIPE).stdout.decode('utf-8')
    try:
        parser = HTMLement("table", attrs={"class": "freeze-column-w-1 w-full overflow-x-auto text-xs leading-4"})
        parser.feed(htm_doc)
        table = parser.close()
        get_data(table)
    except:
        print('html ; table ; invalid')
    return

if __name__ == '__main__':
    get_document('/equities/aarons')
    exit()

[emilianomfortes]

Interesting, yes, this prints whatever is shown at the screen at the time of getting in the website. It is just not working the 'Apply' button to change the dates displayed.