Tuesday, November 1, 2011

YQL: All the Data you can Eat

I love SQL and I love data. Just give me a database and some SQL and I'll analyse the bejeesuz out of it. I also like flexible interfaces. YQL provides all of this. The Yahoo Query Language is basically 2 things:
  1. A huge data source
  2. An easy query language which resembles SQL
  3. Standardized output format (JSON or XML)
OK, that was 3 things but I'm not going back to change my assertion just because I thought of an extra thing whilst typing. And, speaking of inaccuracies, I saaay "data source" but actually it's mostly just passing on data it gets from elsewhere. Let's dive in:

Say you wanna query an RSS feed:
SELECT * FROM rss WHERE url = 'http://sports.yahoo.com/top/rss.xml'

This will return the RSS items from the Yahoo! sports site. You can of course specify multiple feeds using IN (url1, url2) and you can also sort (example).

You can query weather, flickr for photos, find restaurants in San Francisco , list music videos ... I think you get the picture.

There is also the YQL Console, a place to test out your queries. It will generate the link which you can then use in your applications to query data. As you will see, you can specify the output format as XML or JSON:
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20geo.oceans&format=json&callback=showOceans

The callback function wraps your data in a javascript function which means you just AJAX the URL above and your callback function will automatically be executed.

Finally the really gr8 thing about YQL is that it's free and it's cross-domain compatible. If you've ever made an application which fetches data from a different domain you'll know of the issues involved. Modern browsers just don't allow it unless the server says it's OK. YQL always adds the "it's OK header" (Access-Control-Allow-Origin: *) which means it damn well works. Very nice of dem folks at Yahoo!.

Finally, the other of the final 3 great things is that you can literally query (scrape) any web page on the, er, web. You can query normal HTML pages or XHTML or XML sources.

Coming soon, my jQuery Mobile RSS Reader app which uses YQL and Local Storage to save your RSS.