HTTP adapter configuration tips

(finally some documentation available)

Posted by JPLa on November 19, 2018

The last time I wrote about the http_adapter (article HTTP adapter available for testing) was a little over a year ago. Maybe it’s time to start documenting how to configure the new functionality that has been added since then…

Less and less limited in functionality

At that time the http_adapter only supported MT messages. During last spring I kept adding (and adding) functionality so that we could as much as possible avoid making changes to existing internal applications that were implemented using CGW HTTP APIs.

So I added support for:

  • RECEIVE_ONLY type of services
  • QUERY/REPLY type of services
  • even support for asynchronous handling of the above type of services (it turned out that Opaali notifications time out much sooner than how it was in CGW – many existing services could not handle these shorter timeouts)

But a little bit about Security first…

Now that the http_adapter has not only MT (Mobile Terminated) message support but also supports MO (Mobile Originated) messages, you should consider where to install it. The http_adapter uses https to make requests to OpaaliAPI, so the content of messages being sent is encrypted and not easily visible to external parties. But for incoming traffic the http_adapter implements a http (not https!) server, which means that incoming message requests are not encrypted. This may not be a big issue if the incoming traffic only contains a keyword, possibly followed by some non-personal data, but as GDPR has become effective since the http_adapter was first introduced, you probably really should use encryption for incoming messages also.

Security in Content Gateway

Security in CGW CGW by default

Content Gateway uses its own proprietary encrypted tunnel between the Operator Server and the Provider Server - the messages between them are not visible to outsiders. It is assumed that the the messages between the Provider Server and an Application Server are protected by the customer’s firewall.

Limited Security with http_adapter

Security in http_adapter http_adapter by default

  • http_adapter uses https to make requests to OpaaliAPI so the content of messages being sent is encrypted and not easily visible to external parties.
  • http_adapter implements a http server for incoming notifications, which means that incoming message requests are not encrypted.

There are at least a couple of approaches that could be used to implement https support for incoming notifications.

Implementing https server inside http_adapter

Security with https server implemented http_adapter if https server were implemented

The Java http server used by http_adapter contains a https version of the server, so you can modify the existing code by yourself to use this. (The main reason why I haven’t done this by myself is that I have been so busy with other things. And then there would also have been the trouble of obtaining certificates and finding the place where to put them…)

Putting http_adapter behind a Reverse Proxy

Security with Reverse Proxy http_adapter behind a HTTPS Reverse Proxy

It might be easier to put the http_adapter behind a reverse proxy which implements https support (you could use e.g. the Apache HTTP Server).

…and then a little more about Security

This may be a little far fetched, but as the http_adapter does not check where the Incoming Message Notifications come from, it is possible for evil hackers to feed bogus incoming messages to you system, especially if the traffic is not encrypted. I’ll leave it to your imagination to think if this can be used to harm you… You may want to create a Firewall rule that allows incoming notifications only from the IP-address where Opaali sends them from.

And I have probably said this before: don’t let your Opaali Application username and password leak to outsiders. That is all the information that is needed to send SMS messages from your account.

http_adapter block diagram

http_adapter block diagram http_adapter block diagram

The diagram above shows the message traffic through the http_adapter inserted between OpaaliAPI and your backend service. It also shows some of the internal modules of the http_adapter (of which only the CgwHttpApiHandler existed in the original version last year). The rest of this article shows how to configure the various services.

The common part of the configuration file

Some of the following configuration entries were already covered in the previous article about the http_adapter, but there are a couple of new enhancements:

  • simple log rotation
  • multiple threads
# HTTP adapter configuration file
#
# common config parameters are at the beginning
# server port
port=8081
            
# log file name            
log_file=OpaaliLog.txt   
# log level: 0=NONE, 1=ERROR, 2=WARNING, 3=INFO, 4=DEBUG,
log_level=3
# always log to stderr too: 0=no, 1=yes
log_stderr=1
# append to existing log file: 0=no, 1=yes
log_append=1
# log_rotate - insert current date before last dot in filename: 0=no 1=yes
#log_rotate=1


# by default the http server is single threaded
# which may lead to lockups especially if you
# have more than one service configured
#
# you probably want to configure multiple threads
# using a thread pool
#
# set the threadpool size for http-server (optional, default is 1)
threadPoolSize=10

port

  • you need to specify the server port where this will run on your local machine 
    port=8081

log configuration

  • you probably want to configure a log file and the level of logging
  • you can choose to output log to standard error output of the console window
  • you can also choose whether to clear existing log at start or append to it
  • there now is simple built-in support for log rotation: date will be inserted before the last dot in log filename, a new log file is started for the first incoming request after midnight 
    log_file=mylogfile.log
log_level=3
log_stderr=1
log_append=1

threadpool size

  • you may need to tune the amount of parallel threads that the http-server is allowed to use
  • by default the http-server is single threaded, which may be sufficient for light send-only use, but if you have receiving enabled you will need to have several simultaneous threads to avoid deadlocks or other performance problems 
    threadPoolSize=10

Configuring services

You will need at least one service section in the configuration file. Service sections start with a label containing a name and the service type separated by a colon.

# service sections [name:type]            
#
# service will run at the /name URL path
# if name is empty the service will run at root path
# but notice that all incoming requests will then match
# this service and you cannot have any other services

How to configure send service

The send service is almost always needed and it may be the only service you configure. This service sets up a http service that accepts MT SMS messages in the same format as the http send interface in CGW.

The outgoing message is processed by CgwHttpApiHandler which will call OpaaliAPI for actually delivering MT messages.

http_adapter send service http_adapter send functionality

The type of the send service is cgw (referring to Content Gateway http send API) and typically its name will be send (but you can use other names instead, if you like).

A configuration file example:
# service type "cgw" is for sending MT messages using
# Content Gateway type of http request
[send:cgw]
# character set for CGW API side
cgwCharset=ISO-8859-1
#cgwCharset=UTF-8
# replace these with your own credentials (these won't work)      
applicationUserName=b535b0c5e5ae815cea82db6b32b5095a
applicationPassword=1%AMCC?u

# mask specified request parameters with '*' in log at log_level info
# -format: (key, position, length)
# -replaces key value (from left to right) starting from given position with '*' up to given length or end of value
# -if position is negative, applies mask right to left from end of value
# -length is optional, if omitted applies mask until the end/start of value
#log_mask=(to,-2,4),(msg,15)

cgwCharset

  • you may explicitly set the character encoding used by the http server
  • you only need to explicitly set this if the default does not match what your calling application expects (depending on your platform, you may also need to change the Java Virtual Machine (JVM) default character encoding to get the desired result)

applicationUserName

applicationPassword

  • these are the credentials of your MT send application which you will find in the Manage Endpoints section of your Application Profile in the Developer Portal

log_mask

  • the log_mask can be used to hide personal details from URLs in the log file
  • a mask is defined inside parentheses, you can have a list of masks
  • a mask starts with a URL parameter name
  • followed by the starting position of data to be hidden (starting from left, unless when negative starting from right)
  • followed by optional length, if missing mask is applied to the end/start of the value
an example:

The following log_mask

log_mask=(to,-2,4),(msg,15)

could produce this kind of a log entry

2017-12-16 15:23:24,014 INFO    http request(port 8877): /send?to=04017****9&from=15609&msg=Täma+on+testiä* (response in 2582ms:200 OK)

How to configure receiveonly service

For incoming messages there is the receive service, which receives callback notifications from Opaali and forwards them as HTTP GET requests in the same format as Content Gateway does. There is support for keyword detection and URL variables. Currently only text messages are supported; there is no specific support for binary or MMS messages.

http_adapter receive service http_adapter receive functionality

A configuration file example:
# service type "receive" is for receiving callback notifications
# from Opaali for received MO messages
# (this service is available since v0.2.0)
[opaalinotif:receive]
##opaaliCharset=UTF-8
#
# see above for log_mask usage
log_mask=(msisdn,-2,4),(msg,15)

# incoming MO message notifications will be sent
# to a configurable target service using
# Content Gateway style HTTP GET requests
# using template URLs with macros filled in
# (notice that NOT all of the CGW macros are supported)
#
# a default URL is called when no there is no
# other matching configuration
defaultUrl=http://localhost:80/?msisdn=$(M)&shortnumber=$(R)&keyword=$(1)&msg=$(*)
#
# a separate mapping file can be used for choosing the target
# based on keyword and/or short code
mappingFile=mappings.txt
#
#

opaaliCharset

  • you may explicitly set the character encoding used by the http server
  • you only need to explicitly set this if the default does not match what Opaali notifications expect

log_mask

  • log_mask acts similarly to what whas described above for the send service

defaultUrl

  • this is the URL that is called to pass the received message on to a backend service (unless a keyword specific target is specified in a separate mapping file)
  • it supports many of the same variables as Content Gateway does and replaces them with values taken from the notification coming from Opaali

mappingFile

  • if you want to choose the backend service URL to be called based on the short code and keyword of the MO message you need to specify a separate mapping file where these are configured

Mapping file

The mapping file is a separate file where you can specify short code specific mappings to backend URLs and optionally keyword specific mappings to URLs. The purpose of these mappings is to provide similar functionality that you could configure using the Provider Admin UI in Content Gateway.

  • the file contains sections starting with a short code inside brackets
  • all the following configuration lines are specific to this short code
  • if there are more than one consecutive section lines before an actual URL mapping then these are interpreted as a list of short codes for which the mappings will apply
  • there is a global section before the first explicit short code section which is applied to any short code if none of the more specific rules fits
  • a mapping is of the format keyword=URL template, where @ matches to any keyword
An example of a mapping file:
# This is a mapping configuration file from
# - service_address and keyword to a target URL template
# - keyword to a target URL template
#
# The mappings are specified as lines containing
# SERVICE_KEYWORD=APPLICATION_URI
# 
# a default mapping is specified using special keyword @
# (you can only have one default mapping for a set of service addresses)
# @=APPLICATION_URI
#
# mapping lines at the beginning of this file are applied to any service address
#
# to specify service address specific mappings you insert any number of service address lines
# in front of the mapping lines, like this:
# [SERVICE_ADDRESS]
# SERVICE_KEYWORD=APPLICATION_URI
#
# mapping lines are applied to the preceding service address line or lines until the next 
# batch of service address lines
#
# service address specific mappings take precedence over service address independent mappings
#
# the following mappings are applied to all service addresses 
testi=http://localhost:8080/$(msisdn)?msg=$(msg)


# all the rest of the mappings are tied to one or more service addresses
[12345]
INFO=http://localhost:28077/tfolsms/sms.do?sender=$(M)&recp=$(R)&msg=$(MSG)
TILAA=http://localhost:28077/tfolsms/sms.do?sender=$(M)&recp=$(R)&msg=$(MSG)&tilaa=yes

[1234]
[54321]
@=http://localhost/testi/$(R)/$(M)

How to configure asynchronous receiveonly service

It turns out that Opaali has a much shorter timeout for getting a http response from a backend service than what Content Gateway had. Many services which used http_adapter as a gateway for incoming MO messages were unable to process the request quickly enough (because they performed lengthy database operations which was ok previously).

Symptoms of this were the sender of a MO message first getting a response that the service is unavailable and shortly afterwards receiving the actual response anyway! (Unless it was a service that does not send any response back…in this case the user thinks the action was not performed although it actually was…)

To support these services http_adapter can process received notifications asynchronously: the incoming http request is acknowledged as successful right away, without waiting for the backend to process it. Such requests are entered into an internal queue where requests are made to the backend service synchronously.

http_adapter async receive service http_adapter receive functionality (asynchronous processing)

Configuring internal queueing service

The internal queue is automatically enabled, but you may want to configure its size and especially log_mask manually (to avoid exposing user details in the log file). The service type is queue, its name is not important as only one queue is used.

# an internal queue is automatically generated so it is not necessary to list here,
# unless you want to configure a log_mask or need to change the queue size
# even if more than one queue is configured in this file only one will be created
# (...and you have to guess which one of them)
#[internalq:queue]
#queueSize=20
#log_mask=(to,-2,4),(msg,15)

queueSize

  • you can manually set the queue size if the default value is too small
  • if the queue happens to become full (due to messages arriving much faster than they can be processed) the processing will revert to synchronous mode which may lead to incoming requests timing out

log_mask

  • use the same mask as you used for receive service

Enabling asynchronous mode for a service

You need to explicitly specify which receiving services have asynchronous mode enabled.

# allow caller to request actual processing to be queued for later processing: 0=no, 1=yes
# this is done by appending /nowait to the request URL
# this may be needed if the backend service cannot process requests fast enough
# if the queue is full, we fall back to normal processing and take the risk of timing out
# notice that if the queued processing fails or http server crashes the data may be lost
#nowait=0

nowait

  • adding nowait configuration entry you can turn asynchronous mode on (1) or off (0)

Using asynchronous mode

There is a final piece of information you need to actually use asynchronous processing: requests are processed asynchronously only if they append /nowait to the URL they call.

synchronous processing asynchronous processing
http://server:port/receive http://server:port/receive/nowait

How to configure Query/Reply service

Query/Reply service is like receive service, but the backend service can return a reply to be sent back in the HTTP request response body. The service type is qr. The reply is sent through the queue service and requires configuring a URL to a service that can be used to send a MT message in Content Gateway request format (usually the send service of the http_adapter, but it could be an external service as well.)

http_adapter query/reply service http_adapter query/reply functionality

# service type "qr" is a variant of "receive" service where the response body (if there is one) 
# is returned to the caller in a separate, queued MT request to mimic the functionality of
# CGW QR services
# (this service is available since v0.3.0)
#[opaaliqr:qr]
# see above for log_mask usage
log_mask=(msisdn,-2,4),(msg,15)
#
# see above for defaultUrl and mappingFile usage
defaultUrl=http://localhost:80/?msisdn=$(M)&shortnumber=$(R)&keyword=$(1)&msg=$(*)
mappingFile=mappings.txt
#
# see above for nowait usage
# this affects only the MO notification processing, the MT response is always queued as it requires
# a separate Opaali API call
nowait=0
#
# defaultReplyUrl is used to specify how the MT message for a Query Reply is sent,
# this can be the "cgw" service from the top of this configuration file
# notice how the $(MSG) macro is escaped by doubling the $,
# here the macros are expanded in two passes, once when creating the queued request
# (for sender/recipient) and again when the message content is available
defaultReplyUrl=http://localhost:8081/send?to=$(M)&from=$(R)&msg=$$(MSG)

log_mask

defaultUrl

mappingFile

nowait

  • see receive service for details of the above configuration entries

defaultReplyUrl

  • this is the URL used for sending a MT message as a reply to a query
  • this is similar to the format of defaultUrl but with an extra twist: the sender and recipient values are taken from the incoming request, but the reply message macro contains two $-characters so that it is first expanded into “$(MSG)” which is ultimately replaced with the reply text returned in the response body by the backend service. (This will not work if you only have $(MSG) as the value will be extracted from the request, not from the response.)

Enough already!

The real reason why I had all the trouble writing this down is that I myself couldn’t remember how these services were supposed to be configured. (I have a feeling I should have stopped adding new functionality long ago. Or at least have written the documentation while the code was written…)

JPLa is a member of the Content Gateway (CGW) to Opaali migration team, specialising in programming related issues and API usage.