HTTP Client

Description

The module implements sending of HTTP requests to a server.
It is based on libcurl.

Handled message

Message name http.request.
Parameters:

• url: Required HTTP URL
• method: HTTP method to send. Defaults to GET if missing
• accept: Optional Accept header contents
• wait: Boolean (default: false). The sender is synchronously waiting for completion
• notify: Target id for notification. Ignored for synchronous requests
• local_failure_response_async: Boolean (default: false). Always emit a notification on local failure (don't return error in handled message).
  This parameter is ignored for synchronous requests or empty notify.
  This parameter may be used by modules using full asynchronous sending without needing to check if the request started.
  Further message processing by other module(s) will stop if this parameter is set to true.
• http_: Add header(s) to request from parameters starting with this prefix. This parameter may be repeated to add multiple header.
  E.g. http_X-Hdr=custom will add the header X-Hdr: custom
• header: Add header(s) to request. This parameter may be repeated to add multiple header.
  E.g. header=X-Hdr2: custom2 will add the header X-Hdr2: custom2
• ret_header: Return header(s) from response. Comma separated list of header name(s).
  E.g. ret_header=Location will return the Location header if present in response as xhttp_location parameter
• version_http: Force using the request HTTP version. Handled values: 1, 1.0, 1.1, 2, 2.0
• version_http_required: Boolean true: start connection with requested version. Applicable for HTTP ver. 2 only

Body handling:

• body_parse: Boolean. Parse received body and return it in message parameter(s) instead of return value
• body: Request body to set (plain character string). Ignored if method is GET or HEAD
• type: Optional Content-Type header contents
• multipart_subtype: Non empty indicates a multipart body subtype. E.g. related will build a multipart/related body. Bodies will be handled as body, body.1 until first not found index.
  For subsequent bodies the parameters are given with body prefix. E.g. body.1.type
• body.type: Alternative for type. Required for multipart bodies
• body.encoding: hex indicates the body value is a hex string containing binary data
• body.header_: Multipart bodies. Extra header(s) to be added to body. See the header_ description

The following parameters may be used to override the configured defaults: timeout, redirect, agent, max_hdr_length, max_body_length, server_keep_alive, ssl_verify_peer, ssl_verify_host, debug.
See Configuration section for all module parameters allowed to be overridden in handled message.

The module will return true if:

• Message was accepted (validated)
  • Synchronous requests: the sender should check the presence of code parameter on return to match success
  • Asynchronous requests: the request was queued
• Request was not accepted but a notification was asynchronously sent (see local_failure_response_async parameter)

The module will return false in all other scenarios. reason and error parameters may be present in the message.

Returning result

Success. Common parameters

• code: Answer code
• code_text: Answer text (e.g. OK)
• xcontent_length: The contents of received Content-Length header

Success. Body parsed

• xbody: The body contents
• xbody.type: The body content type with no header parameters
• xbody.encoding: hex if received body was detected to be a non text one or contains NULL char(s). The returned body is a string containing HEX chars with no spaces
• xbody.typeparam_: Single Content-Type header parameter. E.g. xbody.typeparam_charset=iso-8859-1

Multipart:

• xmultipart_subtype: Subtype of a received multipart body
• xbody.header_: Received multipart body header(s). Header name is lower case. E.g. xbody.header_content-id=1

Subsequent bodies will be added to message for multipart starting with index 1: xbody.1, xbody.1.type ...

Success. Body not parsed

• xtype: The contents of received Content-Type header
• xbody_encoding: hex if received body contains NULL char(s). The returned body is a string containing HEX chars with no spaces

The received body is set in message's return value.

Failure

• reason: Failure reason. Here are some reasons that worth to be mentioned:
  • timeout: Request timed out
  • congestion: An asynchronous request was not taken from queue fast enough by the processing thread
  • protocol-error: The module was not able to parse received response line to detect the code or received a header containing NULL char(s)
  • rejected: The module can't accept the request because it already holds the configured maximum allowed active requests
• error: String describing the failure reason

Notification message

Message: http.notify
Parameters:

• targetid: The value set in the request's notify parameter

See Returning result section for all other parameters.

Status report

Status parameters:

• total: Total number of requests
• maxrequests: Maximum number of active (accepted) requests ever
• maxqueue: Maximum queue length (asynchronous requests waiting to start) ever
• maxthread_concurrent: Maximum number of per processing thread concurrent requests ever
• total_accepted: Total number of accepted (validated and started/queued) requests
• queued: Current number of asynchronous requests waiting to be started
• total_sync: Total number of accepted synchronous requests
• total_async_notify: Total number of accepted asynchronous requests requiring notification
• total_async_no_notify: Total number of accepted asynchronous requests not requiring notification
• process_async: Current number of running asynchronous requests
• process_sync: Current number of running synchronous requests
• timeout: The number of timed out requests
• congestion: The number of congested (timeout in queue) requests
• rejected: The number of rejected (by module) requests

Configuration

httpclient.conf [general]

; General settings for the HTTP Client module

; redirect: integer: Enable HTTP 3xx redirects (max number of allowed redirects)
; This parameter is applied on reload and can be overridden in handled message
; Allowed interval: 0..3
; 0: disable redirect
;redirect=0

; timeout: integer: Maximum time, in milliseconds, that the request or a response is allowed to take
; This parameter is applied on reload and can be overridden in handled message
; Allowed interval: 100..600000
;timeout=10000

; queue_timeout: integer: Timeout, in milliseconds, of queued requests: how much time a request can wait
;  in input queue before beeing processed
; This parameter is applied on reload
; Allowed interval: 300..3000
;queue_timeout=2000

; max_hdr_length: integer: Maximum length for received headers
; This parameter is applied on reload and can be overridden in handled message
; Minimum allowed value s 200. Defaults to 16384
;max_hdr_length=16384

; max_body_length: integer: Maximum length for received body
; This parameter is applied on reload and can be overridden in handled message
; Minimum allowed value s 200. Defaults to 65535
;max_body_length=65535

; max_requests: integer: Maximum number of requests (sync or async) pending in the module
; This parameter is applied on reload
; New requests will be rejected by the module when this limit is reached
; Minimum allowed value is 100. Defaults to 20000
;max_requests=20000

; max_concurrent_requests: integer: Maximum number of concurrent requests per async processor
; This parameter is applied on reload
; Allowed interval: 10..3000
;max_concurrent_requests=3000

; server_keep_alive: boolean: Tell the server we want it to keep the connection alive
; If enabled a 'Connection: Keep-Alive' header will be added to request
; This parameter is applied on reload and can be overridden in handled message
;server_keep_alive=yes

; ssl_verify_peer: boolean: Verify peer certificate when SSL is used
; This parameter is applied on reload and can be overridden in handled message
; Empty value indicates nothing to be done: use library default
;ssl_verify_peer=

; ssl_verify_host: boolean: Verify peer host in certificate when SSL is used
; This parameter is applied on reload and can be overridden in handled message
; Empty value indicates nothing to be done: use library default
;ssl_verify_host=

; agent: string: User agent to set in request
; This parameter is applied on reload and can be overridden in handled message
;agent=YATE/${version}

; printmsg: boolean/keyword: Print sent/received message contents
; This parameter is applied on reload and can be overridden in handled message
; Boolean false: Don't print
; Boolean true: Print headers only
; 'verbose': Print headers and body
; 'firstline': Print request/response first line only
; 'headers': Print headers only (same as boolean true)
;printmsg=no

; printmsg_localaddr: boolean: Print locall address when printing a message
; This parameter is applied on reload and can be overridden in handled message
;printmsg_localaddr=no

; threads: integer: The number of threads used for async processing
; Allowed interval: 1..50
;threads=1

; async_priority: keyword: Priority of the thread processing the asynchronous requests
;async_priority=normal

; priority: integer: Priority level of http.request message handler
;priority=90

; debug: boolean/integer: Debug mode for CURL library (displaying SSL data, headers, sent and received data)
; This parameter is applied on reload and can be overridden in handled message by a debug parameters
; This will translate to a debug level: 0: use output, greater than 0 use debug
; Boolean true is translated to debug level 0 
;debug=no

Testing

httptest.js:

#require "lib_str_util.js"

Engine.debugName("httptest");
Message.trackName("httptest");

pending = new Engine.HashList(101);
targetid = 0;

// Default request parameters
http_params = {
    method: "GET",
    url: "http://www.null.ro",
    // Parameters overriding module defaults
//    timeout: 5000,
//    redirect: 0,
//    agent: "HTTP Test",
//    max_hdr_length: 100,
//    max_body_length: 100,
//    server_keep_alive: false,
//    debug: false,
    // Parameters for requests with body
//    accept: "application/json",
//    type: "application/json",
//    body: "{request:\"REQ\"}",
};

// Send a HTTP request
// params: Message parameters
// sync: Send synchronous
// notify: Async only, indicate if response is expected
// enq: true to enqueue, false to dispatch and check if request was started
//      Ignored if notify is not requested
function httpRequest(params,sync,notify,enq)
{
    var req = undefined;
    var m = new Message("http.request",false,params);
    m.wait = !!sync;
    if (!sync) {
	if (!notify) {
	    m.enqueue();
	    return true;
	}
	notify = Engine.debugName() + "/" + ++targetid;
	req = pending[notify] = {id:notify};
	req.info = m.method + ":" + m.url;
	m.notify = notify;
	if (enq) {
	    // Instruct the module to notify us if request failed to be started
	    m.local_failure_response_async = true;
	    m.enqueue();
	    return true;
	}
    }
    var ok = m.dispatch(true);
    if (req) {
	if (ok)
	    return true;
	// Failed to start, remove from tracking list
	delete pending[req.id];
    }
    handleHttpResult(m,req,ok);
    return ok;
}

// Handle HTTP result message
// req: pending request if async
// ok: boolean true/false for immediate response, undefined if this function is called from async notify
function handleHttpResult(msg,req,ok)
{
    if (req) {
	var info = req.info;
	// Async result in http.notify
	if (undefined === ok)
	    info += " (async notify)";
    }
    else
	var info = msg.method + ":" + msg.url;

    var result = "";
    ok = (undefined !== msg.code);
    if (ok) {
	result += " '" + msg.code + " " + msg.code_text + "'";
	if (msg.xtype)
	    result += " Content-Type='" + msg.xtype + "'";
	if (msg.xcontent_length)
	    result += " Content-Length=" + msg.xcontent_length;
	var b = msg.retValue();
	if (b.length) {
	    // Answer body may contain NULL chars
	    if ("hex" == msg.xbody_encoding)
		result += " body_len=" + (b.length / 2) + " (hex)";
	    else
		result += " body_len=" + b.length;
	}
    }
    else {
	if (msg.reason)
	    result += " reason=" + msg.reason;
	if (msg.error)
	    result += " error='" + msg.error + "'";
	if (!result)
	    result = " unknown error";
    }
    if (ok)
	Engine.debug(Engine.DebugInfo,info,"got answer" + result);
    else
	Engine.debug(Engine.DebugNote,info,"failed" + result);
}

function onHttpNotify(msg)
{
    var p = pending[msg.targetid];
    if (!p)
	return false;
    delete pending[msg.targetid];
    handleHttpResult(msg,p);
    return true;
}

function parseParams(line)
{
    var res = {};
    while (line.length) {
	var m = line.match(/^(.* )?([^= ]+)=([^=]*)$/);
	if (!m)
	    break;
	var s = "" + m[3];
	res[m[2]] = s.trim();
	line = "" + m[1];
    }
    return res;
}

// Handle commands
function onCommand(msg)
{
    if (!msg.line) {
	switch (msg.partline) {
	    case undefined:
	    case "":
	    case "help":
		oneCompletion(msg,"httptest",msg.partword);
		return false;
	    case "httptest":
		oneCompletion(msg,"send",msg.partword);
		return false;
	    case "httptest send":
		oneCompletion(msg,"sync",msg.partword);
		oneCompletion(msg,"async",msg.partword);
		oneCompletion(msg,"async_enq",msg.partword);
		oneCompletion(msg,"fire_and_forget",msg.partword);
		return false;
	}
	return false;
    }
    var s = msg.line;
    var m = s.match(/^httptest send (sync|async|async_enq|fire_and_forget)( .*)?$/);
    if (m) {
	var params = parseParams(m[2]);
	for (var p in http_params)
	    if (undefined === params[p])
		params[p] = http_params[p];
	Engine.debug(Engine.DebugAll,"Sending",m[1],params.method + ":" + params.url);
	var ok = false;
	switch (m[1]) {
	    case "sync":
		ok = httpRequest(params,true);
		break;
	    case "async":
		ok = httpRequest(params,false,true);
		break;
	    case "async_enq":
		ok = httpRequest(params,false,true,true);
		break;
	    default:
		ok = httpRequest(params,false);
	}
	if (ok)
	    msg.retValue("OK\r\n");
	else
	    msg.retValue("Failure\r\n");
	return true;
    }
    return false;
}

help = "  httptest send {sync|async|async_enq|fire_and_forget} [param=value ...]\r\n";
function onHelp(msg)
{
    if (msg.line) {
	if ("httptest" == msg.line) {
	    msg.retValue(help + "Control the HTTP Test operation\r\n");
	    return true;
	}
    }
    else
	msg.retValue(msg.retValue() + help);
    return false;
}

Engine.setDebug("level 10");

Message.install(onHttpNotify,"http.notify",100);
Message.install(onCommand,"engine.command",150);
Message.install(onHelp,"engine.help",150);