Making Raw HTTP Requests in Node

If you’ve used Node for any amount of time you’ve probably worked with the http module. In most cases people use it to create their own server applications or for interacting with remote APIs. Using it to make API calls is pretty straightforward but if you’re using a newer version of Node (0.10+) then you may be tripped up when you get 400 Bad Request responses from the endpoint you’re communicating with. Here are the two trickiest issues I’ve come across. Hopefully this will help someone else who is scratching their head when they get these mysterious 400 responses back after making what would seem to be a perfectly valid HTTP request.

To give some context, I ran into this issue while building an API client for my company’s new public API. The API uses HTTP everywhere and has an authentication scheme similar to what Amazon does with AWS where you sign your request with an HMAC of your private key and request body plus some other information.

Issue 1: The plain HTTP request was sent to HTTPS port

I got a “The plain HTTP request was sent to HTTPS port” along with a 400 status when sending a request from Node to my API. The code looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
var http = require('http');

var request = http.request({
  host: 'example.com',
  path: '/endpoint',
  port: 443,
  method: 'POST',
  headers: {
    'Authorization': clientId + ':' + signature(endpoint, postBody), // Don't worry about this line, ignore it for now
    'Content-Type': 'application/json',
    'Content-Length': postBody.length
  }
}, function(response) {
  var reply = '';
  response.on('data', function(chunk) {
    reply += chunk;
  });

  response.on('end', function() {
    return cb(reply);
  });
});

request.write(postBody);
request.end();

There’s some missing code here which you can safely ignore. The above code was within a function that took the post body data and a callback. Using this code returned the 400 error telling me that I was sending plain HTTP data to an HTTPS port. Well, the thing is, that’s exactly what I wanted to do. I thought that by sending the request to port 443 Node’s http module would be smart enough to figure out that I was trying to make an HTTPS request and handle that. It doesn’t.

So what’s the solution? It was really simple actually. Node has a separate HTTPS module for just this scenario. So you just need to require('https') and update your code so all references to http are updated to read https (technically you can skip this but its easier to understand your code if you do). Once you start using the https module instead of the plain http one your requests will start working.

Issue 2: Mismatched HMAC signatures

My API requires you to sign your requests using HMAC like Amazon. We give you a public client ID and a private token. Signing a request looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# PSEUDO CODE FOLLOWS
#
# GET requests:
# For get requests you just sign your
# path including any query strings.
endpoint '/endpoint/?param=value&somethingelse=whatever';

signature = HMAC('sha1', token, endpoint);

# POST requests:
# For all POST/PUT/PATCH requests your
# signature is the endpoint with the request
# body appended to it as a string.
endpoint = endpoint '/post-stuff/?param=value&somethingelse=whatever';
body = '{"json_key": "value", "boolean": true, "more_stuff": "you get the point"}';

signature = HMAC('sha1', token, endpoint + body);

# Create custom header:
# The API requires you put your cient ID and computed
# signature in a special header when sending the request.
# Its an 'Authorization' header with the value being
# your public client ID with your signature appended to it
# using a colon ':' as a separator between the two like
# 'Authorization: my_public_id:my_computed_signature'.
header = 'Authorization: ' + clientId + ':' + signature;

Using this you should be getting the same signature no matter what language you use to generate your HMAC hash. But somehow ours weren’t matching up. There’s actually two issues here you should know about. The first is that you need to ensure that both hashes are of the same type. What this means is that the hash can either be binary or hex. Our API was computing signatures with hex encoding. Luckily the Node library was doing the same so that wasn’t an issue but I’ve seen people get tripped up by that.

So here’s the code that was sending the POST request. Note that here we fixed our issue with using the HTTPS module.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
var http = require('http');

var request = http.request({
  host: 'example.com',
  path: '/endpoint',
  port: 443,
  method: 'POST',
  headers: {
    'Authorization': clientId + ':' + signature(endpoint, postBody), // .signature does the HMAC work
    'Content-Type': 'application/json'
  }
}, function(response) {
  var reply = '';
  response.on('data', function(chunk) {
    reply += chunk;
  });

  response.on('end', function() {
    return cb(reply);
  });
});

request.write(postBody);
request.end();

Each time we ran this code we’d get a 403 because our signature didn’t match the one computed on the server. How though if both are using the same algorithm to compute the signature? The key here was the Content-Length header. In doing research I noticed that some people included this header in their requests. This was when it made sense. Running the API locally, I made the app write the request path and posted body to the console. What I found was that the API was adding random numbers and strings to the beginning and end of my request body.

So my API client was using this string to compute a signature:

1
/endpoint/whatever{"json_key": "value"}

On the server side, the string it was using to compute the signature looked like this:

1
/endpoint/whatever123\r\n{"json_key": "value"}0

Wait a sec? Where did that “123” and the trailing “0” come from? I have no idea still. I just needed to make it stop. So my theory was that if I sent over the Content-Length then the API would know when to start and stop reading the POST body. I made the API library count the length of the request body and send it over in a Content-Length header. Voila. It worked like a charm. Both the API client and the server were computing matching signatures.

So there you have it. HTTPS calls use a different module in Node. When working with low level modules like http you need to take care of low level problems like sending content lengths.

For most cases, unless you’re doing something really low level you’ll likely want to stick to a library like Request. I’m only using it because I want our API wrapper to have only dependencies that are absolutely required. So far its all that’s needed. If things get real hairy I may reach for something more high level.

Web development

« Turn a string into a hash with String.to_hash in Ruby Initializing a class in Node »

Comments