I am interested in the effect of time on plane tickets. So I figured I’d go ahead and parse my favorite websites for plane tickets; hipmunk, world ticket center and skyscanner.
I go to hipmunk and type the date and the destinations. I can see that this generates the following http link:
This is good because this means that I can adjust the link and automate some of the data collection. I can change the date and the destinations. If I want to parse the website all I need to do is look for the proper tag and then I’d be done.
import urllib link = 'http://www.hipmunk.com/flights/ AMS-to-BUE#!dates=Jan27&pax=1' html = urllib.urlopen(link) website = html.read()
First I import the required python libraries. I take the original website link and store it as a string in the variable link. This then gets opened by calling the urllib.urlopen() method which gives me all the html generated from the website. This is turned into a string by using the html.read() method. You can print the website variable to confirm that it does return html.
Because the html variable is a string we do not need to use a parsing library on it per se, we can also just use website.find() for now. We want to check if we can find the cheapest ticket price in the string. It turns out that we cannot find it. Try it out for yourself.
If I inspect the element through Chrome’s console that lists the price then I can see that website.find(‘$921′) should return an index, instead it returned -1. What is happening?
import codecs import lxml.html as lh from selenium import webdriver browser = webdriver.Firefox() browser.get('http://www.hipmunk.com/flights/AMS-to-BUE#!dates=Jan27&pax=1') content = browser.page_source browser.quit()
Now typing content.find(‘$921′) returns a positive index. We can begin parsing the string.
Downside of this approach is that it is slow. You will see the browser window being opened and closed while python is executing this code. Still, this allows for a lot of clever hacking.
Big final tip: use browser.quit() at the very end of any for loop that you would like to make.
By doing this for about 80 days I was able to see a pattern occur in plane ticket prices:
It seems rather unwise to buy tickets less than 40 days in advance, unless you are buying a last minute. It must be said that this graph was made on December 1st 2012, so the spike in prices might be due to Christmas. The total code needed to parse skyscanner can be found on github but also here:
import codecs import lxml.html as lh from selenium import webdriver import time import re results =  days = [ '01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30'] months = [ '1212', '1301', '1302' ] for month in months: for day in days: browser = webdriver.Firefox() link2 = 'http://www.skyscanner.nl/vluchten/ams/buea/' + month + day + '/vliegtarieven-van-amsterdam-schiphol-naar-buenos-aires-in-januari-2013.html' browser.get(link2) bcontent = browser.page_source price = bcontent[bcontent.find('px EUR'):bcontent.find('px EUR')+25] if len(price) > 0: print 'got one' results.append( [ month+day, str(re.search(r"\d+", price).group())] ) else: print month + day browser.quit()