Tuesday, November 1, 2011

YQL: All the Data you can Eat

I love SQL and I love data. Just give me a database and some SQL and I'll analyse the bejeesuz out of it. I also like flexible interfaces. YQL provides all of this. The Yahoo Query Language is basically 2 things:
  1. A huge data source
  2. An easy query language which resembles SQL
  3. Standardized output format (JSON or XML)
OK, that was 3 things but I'm not going back to change my assertion just because I thought of an extra thing whilst typing. And, speaking of inaccuracies, I saaay "data source" but actually it's mostly just passing on data it gets from elsewhere. Let's dive in:

Say you wanna query an RSS feed:
SELECT * FROM rss WHERE url = 'http://sports.yahoo.com/top/rss.xml'

This will return the RSS items from the Yahoo! sports site. You can of course specify multiple feeds using IN (url1, url2) and you can also sort (example).

You can query weather, flickr for photos, find restaurants in San Francisco , list music videos ... I think you get the picture.

There is also the YQL Console, a place to test out your queries. It will generate the link which you can then use in your applications to query data. As you will see, you can specify the output format as XML or JSON:

The callback function wraps your data in a javascript function which means you just AJAX the URL above and your callback function will automatically be executed.

Finally the really gr8 thing about YQL is that it's free and it's cross-domain compatible. If you've ever made an application which fetches data from a different domain you'll know of the issues involved. Modern browsers just don't allow it unless the server says it's OK. YQL always adds the "it's OK header" (Access-Control-Allow-Origin: *) which means it damn well works. Very nice of dem folks at Yahoo!.

Finally, the other of the final 3 great things is that you can literally query (scrape) any web page on the, er, web. You can query normal HTML pages or XHTML or XML sources.

Coming soon, my jQuery Mobile RSS Reader app which uses YQL and Local Storage to save your RSS.

Sunday, September 11, 2011

SOAP2REST Proxy with node.js

Given the rebirth of JavaScript in the past decade it shouldn't surprise us that some wise-guy asked why we don't use the same language on the server as we use on the client: JavaScript!

Actually Microsoft was doing this in the 90's with their <script runat="server"> where you could write either VBScript or JScript both client and server side. Then browsers like Firefox took off and VBScript was unsupported leading to a rebirth of the powerful JavaScript language.

Node.js (or node.exe) is basically an interpreter like php.exe except you can actually use it stand-alone as it's own web service. You write a simple server.js like this:
var http = require('http');
var srv = http.createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/html'});
  response.write("<h1>Hello World!</h1>");
Next you run your script from the command prompt:
C:\node.exe server.js
Finally, open up your browser to http://localhost:8080 and you should see Hello World! in large, friendly letters.

Because I'm planning a killer hip mobile/offline app which consumes SOAP web services I decided to write a node.js script which will allow me to make simple HTTP GET requests and receive JSON data in return. The soap2rest.js proxy I wrote converts the GET request to a SOAP request and should parse the SOAP response to return JSON:
myApp.html -> HTTP GET -> (soap2rest.js) -> SOAP Request -> SOAP Response -> JSON

The first thing I need is an HTTP client. It's important to remember that our soap2rest.js is going to proxy the SOAP request so I made a generic HTTP request function:

function doHTTP(options, success, error) {

  var req = http.request(options, function(res) {
    var data = '';
    res.setEncoding(options.encoding || 'utf8');
    res.on('data', function (chunk) { data += chunk; });
    res.on('end', function () { success({ data: data, response: res }); });

  if (typeof options.body != 'undefined') req.write(options.body);
  req.on('error', function(e) { error('HTTP ERROR: ' + e.message); });
As you can see the function simply takes all it's parameters via the options object and has two callback functions for success and error. Now all we have to do is map the incoming GET to an HTTP SOAP request. In order to make this proxy as generic as possible I defined the following syntax for the URL: http://whatever/service/method?parameters=values So, for a simple weather service I would call: http://localhost:8080/weather/fetch?city=Zurich Before I can call my soap2rest.js proxy I need to define the service call. This is done in a configuration file called fetch.json corresponding to the fetch method of my weather service. The configuration file sits in a folder called weather with other potential methods: weather/fetch.json:
 "url": "http://www.webservicex.net/globalweather.asmx?op=GetWeather",
 "headers": {
  "SOAPAction": "http://www.webserviceX.NET/GetWeather",
  "Content-Type": "text/xml"
This configuration file tells SOAP2REST.js almost everything it needs to know about calling the SOAP service globalweather.asmx. What it still needs is the request format. This can be discovered by parsing the WSDL of the service but this is seriously complicated and liable to error. And I'm too lazy. So, the developer "just" has to provide the sample request body in fetch.xml inside the weather/ folder:
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <GetWeather xmlns="http://www.webserviceX.NET">
The ##CITY## is a placeholder which will be replaced with the city parameter of the incoming GET request. Here is where the magic happens. The SOAP2REST function does the following:
  1. Parse the URL and determine the service and method
  2. Read the service/method.json configuration file
  3. Read the service/method.xml request file
  4. Replace all placeholders from the query string (e.g. city)
  5. Call the SOAP service POSTing the method.xml data
  6. Return the response to the client
So here's the magic SOAP2REST:
function SOAP2REST(URL, success, error) {

  URL = url.parse(URL, true);

  // Load Service Definition
  try {
    var soapJSON = fs.readFileSync('./soap' + URL.pathname + '.json', 'UTF-8');
  } catch (e) {
    return die({msg: "Service description " + URL.pathname + ".json could not be found!", err: e}, 404);
  // Load SOAP Request
  var soapXML = '';
  try {
    soapXML = fs.readFileSync('./soap' + URL.pathname + '.xml', 'UTF-8');
  } catch (e) {
    // No problem
  // Parse SOAP Configuration
  try {
    var soapJSON = JSON.parse(soapJSON);
	var soapURL = url.parse(soapJSON.url, true);
  } catch (e) {
    return die({msg: "Service JSON could not be parsed!", name: e.name, message: e.message}, 500);
  // Set Parameters
  for (p in URL.query) soapXML = soapXML.replace('##' + p.toUpperCase() + '##', URL.query[p]);

  // Prepare SOAP Request Headers
  soapJSON.headers = soapJSON.headers || {};
  soapJSON.headers["Content-Length"] = soapXML.length;
  soapJSON.headers["Connection"] = "close";
  // Do SOAP Call
  var httpOptions = {
    host:     soapURL.hostname,
    post:     soapURL.port || 80,
    method:   soapJSON.method || 'POST',
    path:     soapURL.pathname,
    headers:  soapJSON.headers,
  httpOptions.body = soapXML;
    function(d) {
      error(e, 500);

Technically, I guess, this ain't REST because I'm not using /weather/fetch/city/Zurich/ but I prefer /weather/fetch?city=Zurich because this is the standard HTTP way of passing parameters and it means AJAX forms will work automagically!

If I do say so myself, I like the way you can easily add a new service on the fly simply by adding a .json and .xml file.

Now to find an XML2JSON converter...

Note: This code is all for node.js v5.6 and the functions seem to change with every release so no doubt this won't work in future! Full Source Code

Tuesday, August 30, 2011

Tinkering with Javascript Libraries on jsfiddle.net

Man I'm out of it. Such great tools that I missed during my post-2.0 slumber.

jsfiddle allows you to quickly and easily make and test JavaScript apps using some of the cool libraries out there (e.g. jQuery or mootools).

Just go to http://jsfiddle.net and start hacking in your HTML, Javascript and CSS. Press Ctrl+Enter to run and your app will, er, run!

I think it's cool.

Check out my sandbox application which says hello to you via ajax. Oh the wonders of modern technology.

Monday, August 29, 2011

Mongoose: Free Easy Web Server

I've always been an IIS fan but the move to Windows 7 and IIS7 has shown me that I'm getting old. I can't get IIS to do anything right and so I've decided: let's re-rethink this. Back to basics. KISS!

I googled "simple http server" and discovered mongoose.

It's Simple™. It Works™. I Like It™.

Download the mongoose.exe and run it and you've just started your webserver.

By default, mongoose serves from C:\ so if you browse to http://localhost/ you should see a directory listing.

Mongoose is configurable via the command line or via a configuration file.

Example: Server on Port 81
C:\>mongoose.exe -p 81

or, via config file you create a mongoose.conf file with the following content:
p 81

You'll probably do something wrong at some stage so you need to tell mongoose where to log errors. This is done via the e parameter:
C:\>mongoose.exe -e error.log -p 81

e error.log

Check out the full manual explaining all commands.

This thing can even run PHP! Just tell it which file extensions you want to be processed via CGI:
c .php

Next, tell it where your php executable lies:
I c:\\php\\php-cgi.exe

(The double backslashes are required for windows PCs like mine)

Here is my sample/example mongoose.conf:
# Error Log
e error.log

# Ports (csv)
p 80

# Root directory
r c:\\mywebsite\\html

# Default documents
# (disabled with # comment)
#i index.htm,index.html

# Add your special mime types here
m .manifest=text/cache-manifest,.asp=text/html

# CGI File extensions (csb)
c .php

# Location of CGI interpreter (e.g. PHP)
I c:\\php\\php-cgi.exe

Happy serving!