When an error message is trying to be too helpful

Posted by JPLa on August 19, 2019

Sometimes an error message tries to be helpful, but ends up obfuscating the real problem. When the http_adapter encounters an unrecoverable error while trying to authenticate towards the Opaali API, it will stop trying (after all, the situation seems to be unrecoverable) and returns an error message for all future requests:

2019-08-19 09:21:19,857 ERROR   unrecoverable error, returning 401 Unauthorized to request
2019-08-19 09:21:19,858 DEBUG   CGWHandler response: HTTP 401: <b>Access is denied due to probably invalid credentials in http_adapter configuration</b>

Recoverable vs Unrecoverable

The most typical case where authentication towards Opaali API fails is the use of wrong credentials. Maybe you didn’t edit the http_adapter configuration file before starting it and the default values were used. Or maybe you are using valid credentials but they belong to some other Opaali application. In any case, the situation will not be corrected until you change your configuration settings.

The http_adapter tries to spare Opaali from unnecessary load caused by these authentication requests that are destined to always fail, so it enters into a failure_mode where any subsequent requests will fail until the configuration is corrected and the http_adapter restarted.

Recoverable error?

A recoverable failure when authenticating might happen when your application exceeds its TPS limit (i.e. tries to make API requests in too fast succession). In these cases the http_adapter will pause and resume operation a little bit later.

Recovering from an unrecoverable error?

In the above cases it is assumed that the Opaali API itself is functioning normally. But what if there is a break in the service or network connectivity? The http_adapter sees this as an unrecoverable error and enters the failure_mode. It does not exit the failure mode automatically when Opaali API resumes its operation - it needs to be shut down and restarted.

In this case this error message you will see is misleading: Access is denied due to probably invalid credentials in http_adapter configuration

This time you actually would not need to make configuration changes, you just need to restart the http_adapter.


(maybe the http_adapter should be more intelligent and recover from this situation automatically…maybe by exiting the failure_mode after 15 minutes or something?)

Perhaps something like this might work:

$ diff http_adapter/src/smsServer/ServerConfig.java http_adapter_recoverable/src/smsServer/ServerConfig.java
55a56
>     public static final String CONFIG_FAILUREMODETIMEOUT = "failureModeTimeout";
$ diff http_adapter/src/smsServer/CgwHttpApiHandler.java http_adapter_recoverable/src/smsServer/CgwHttpApiHandler.java
64a65,69
>         failureModeTimeout = sc.getConfigEntryInt(ServerConfig.CONFIG_FAILUREMODETIMEOUT);
>         if (failureModeTimeout > 0) {
>               // convert from minutes to milliseconds
>               failureModeTimeout *= 60000;
>         }
76a82,83
>     long failureModeStartTime = 0;
>     long failureModeTimeout = -1;
81a89,97
>
>         // reset failureMode if failureModeTimeout reached
>         if (failureMode && failureModeTimeout > 0) {
>             if (startTime - failureModeStartTime > failureModeTimeout) {
>               failureMode = false;
>               failureModeStartTime = 0;
>               Log.logError("resetting unrecoverable error after reaching failureModeTimeout of "+failureModeTimeout/60000);
>             }
>         }
273a290
>                                 failureModeStartTime = RequestLogger.getTimeNow();

and then adding a new configuration entry in the [send:cgw] section:

[send:cgw]
# failureModeTimeout in minutes
failureModeTimeout=15

JPLa is a member of the Content Gateway (CGW) to Opaali migration team, specialising in programming related issues and API usage.