Developers »
Forecasting API » here
Programmer's guide for the Lokad Forecasting API v3
.NET developers do not have to use Forecasting API directly. There is an open source
Forecasting Client to jump-start the development.
This document contains reference documentation and guidance for the
Forecasting API v3 of Lokad. You can programmatically send time-series data toward Lokad and get forecasts back.
Sandbox service
If you are developing a new application against our API, then we suggest you to take advantage of our
sandbox service which is made available for free to our fellow developers. In order to do that you need
open a separate account on the sandbox, and then replace
api.lokad.com by
sandbox-api.lokad.com for all your API accesses (either REST or SOAP).
SOAP vs. REST
The Forecasting API v3 comes in two flavors:
- a REST endpoint at
http://api.lokad.com/rest/forecasting3/ - a SOAP endpoint at
http://api.lokad.com/forecasting3.svc
The two endpoints are strictly identical feature-wise and performance-wise. We suggest you pick the format that best suits your needs. In our experience, SOAP is best when you have a powerful SOAP toolkit on hand; if you don't, or if you have no experience with SOAP, then we suggest to use REST instead.
Authentication
Forecasting API v3 relies on the API keys that can be obtained through any
Lokad account. Just log in your account, go to
Users and generate a new key.
If you are developing a client app (in particular a web app) and you want to programmatically obtain API keys on behalf of your users, then check out
Lokad.TinyAuth, a tiny REST API to retrieve API keys.
API concepts
The API v3 introduces the notion of
datasets. A single Lokad account can hold many datasets, each one of them identified by a unique name (within the account).
Each dataset contains several
rich time-series, each one of them uniquely identified by its name (two series may have the same name if they belong to different datasets). Then, each time-series can contains multiple time-values, tags and events.
The dataset also defines the
forecasting settings which include:
- the Period defining the level of aggregation (day, week, month, ...)
- the Horizon defining the number of periods being forecasted ahead.
Dataset settings are
immutable: once defined, they cannot be altered. Although it is possible to delete the dataset, and to recreate one with different settings.
Datasets represent
data containers but with a forecasting angle. We suggest to keep all related data within the same container, as Lokad will be able to exploit more intensely the statistical correlations that exist in your data, if the correlated data happen to be gathered in the same dataset in the first place.
For example, a eRetailer who wants to forecast both sales (to optimize inventory) and call volumes (to optimize staffing), should create a first dataset for the sales (with
weekly forecasts) and a second dataset for the call volumes (with
quarter-hourly forecasts).
Datasets are
persistent. In particular, they can be
incrementally updated, sending only the most recent data to be merged with the older data (which has been already uploaded). Such incremental upload saves you the cost of re-uploading the entire history each time the forecasts need to be refreshed.
Call flow overview
The most typical usage of Forecasting API v3 is the
forecasting pipe:
- insert a dataset (if no dataset is already available)
- insert or update time-series in this dataset.
- trigger the forecast flow.
- checks forecasts status until they are ready.
- download forecasts associated to the dataset.
Once the
forecast flow has been triggered, the dataset cannot be altered any more until the forecasts are computed. Lokad guaranties that the forecasts are delivered in
less than 2h no matter the size of the dataset. In practice though, if your dataset is small, computation is much faster (only a few minutes).
The API also offers the possibility to:
- list the datasets (or delete them).
- list the time-series of a dataset (or delete them).
The
listing features are provided for debugging purposes rather than for production. We suggest not to rely on those feature in your production logic, and instead, get a client-side record of the
state of your datasets.
Continuation tokens
The two methods
list datasets and
list time-series let you enumerate all the data stored in your Lokad account. Since the amount of data can be arbitrarily large, a single call might not return all the data contained in your account.
Instead, the method returns a
continuation token. This token should be treated as a black-box (not to be parsed or tempered) and returned to Lokad in the next method call to retrieve the next batch of items. As long a continuation token is returned, there are items that remains to be enumerated.
Scalability notes: iterating on datasets or time-series is a strictly sequential process (hence not
exactly scalable). Yet, as we were outlining before, retrieving historical data from Lokad is provided to facilitate debugging, but those operations are not expected to be performed for production purposes.
Retry policies
When interacting with remote API such as the one offered by Lokad, we advise to setup
retry policies on the client side. In short, network connections are rather reliable but
not 100% reliable: a single failed is not supposed to cause the client app to implode.
In our own experience observing a random network error every 100,000 web requests is nothing out of the ordinary. Without retry policies, failures within process handling large datasets (requiring thousands of web requests) are going to become painful.
Most of the time, when a web request fails, it's just a transient error. Just wait to a few hundreds milliseconds and try again.
Obviously, retry polices come with a potential issue: when a request fails we can't be sure if the call has been processed or not, as the failure might have occurred randomly either while sending the request, or while retrieving the response.
Thus,
all methods exposed by the Forecasting API v3 are idempotent. Calling twice (or more) the same method with the same arguments has identical effect than the single initial call (because we have designed the API that way). Hence, the
API is retry-policy friendly.
Setting-up your own home-made retry policies is rather straightforward. We strongly
advise to setup retry policies in your client logic, as it will save you a lot a trouble later on.
Dataset async deletion
Deleting a dataset also delete all the series contained in the dataset. The deletion of datasets presents one subtlety: it follows an
asynchronous pattern. Indeed, it would been hard for Lokad to support atomic deletion of arbitrarily large amount of data. Hence, the deletion of a dataset is not immediate after the call to
delete dataset. Instead the dataset enter a state of
deletion pending, and the deletion typically occurs within a few minutes after the initial call.
Then, when designing your logic against the Forecasting API v3, you should keep in mind that you won't be able to immediately recreate an dataset with the same name than the dataset you've just deleted.
Incremental time-serie updates
The method
upsert time-series (i.e. insert or update) provides an optional
merge behavior. This behavior is provided to support incremental update of data already uploaded toward Lokad. Let assume that we have an
old time-series already stored in the Lokad account, we are now pushing a new time-series:
If the update is done with the option
merge=true, then, instead of simply overwriting the existing time-series with new value (plain overwrite is the default behavior), the resulting time-series is obtained as a merge of the old time-series and the new time-series.
It should be noted that all sections that overlap between the old and new time-series get overwritten by the data of the new time-series (most recent data take precedence).
Then, we are illustrating here the merge behavior with time-values, but the Forecasting API v3 behaves identically for events which could be incrementally updated too.
Object naming rules
Whenever a
string token is used in the API (dataset names, series names, tags, ...), the following restrictions apply:
- Length should be between 1 and 32 characters.
- Characters should be strictly alphanumerical [a-zA-Z0-9].
The equivalent
REGEX pattern is:
[a-zA-Z0-9]{1,32}$Capacity limitations
There is no limit in the number of datasets in a Lokad account. Although, unless you're acting on behalf of plenty of clients, there are usage not many compelling reasons to have more than a handfew datasets in your account. There is no limit in the number of time-series in each dataset.
Then, all arrays and compound operations are limited to
100 items; except for the time-series that may hold up to 100,000 time-values. Any web request beyond
4MB will also be rejected.
Those rules imply (among other examples):
- only 100 time-series can be updated at a time.
- only 100 forecasts can be downloaded at a time.
- a single time-series may not hold more than 100 tags or more than 100 events.
- a single event may not hold more than 100 tags.
- ...
Forecasting periods
The API supports the following periods (periods are part of the dataset settings):
quarterhourhalfhourhourdayweekmonth
Although this is not a strict requirement, we advise to aggregate your historical data to match the forecasting aggregation period. For example, if you want weekly forecasts, then we suggest to pre-aggregate your data into weekly values.
Error codes
Every single method of the API may return an error code (always enclosed in a
<ErrorCode/> element). The existing error codes are:
AuthenticationFailed: The authentication failed. This situation can be caused either because the API key is incorrect, or because the account has been locked for lack of payments.OutOfRangeInput: The input is well-formed XML, but it is not compliant with the naming rules (ex: invalid characters in dataset names) or the capacity rules (ex: too many series updated at once).DatasetNotFound: No such dataset available in your account. You need to insert the dataset first, and/or correct your logic.InvalidDatasetState: The dataset is not in an appropriate state to be the target of the specified operation. For example, no data could be uploaded once the forecast computation is in-progress, and the forecasts cannot be retrieved until the forecasts are ready. Also, if a dataset is being deleted, you should wait until completion.ServiceFailure: Transient failure on the Lokad side. We suggest to delay your call from 15min (for example), and then try again.
Service fees
The fees applied to the Forecasting API follows our public
pricing. The client gets charged for every forecasting points retrieved through the method
get forecasts.
Other methods are offered at no extra charge. Although, we are
considering introducing a flat fee per request of 0.10€ per 1000 web requests (i.e. $0.15). This extra charge is NOT activated yet, but depending on the observed usage patterns, we
might introduce this fee at some point.
We suggest to take advantage of our API to make bigger requests. For example, instead of uploading a single time-series at a time, the API lets you upload up to 100 time-series in a single web request.
REST API methods
The REST Forecasting API 3 follows canonical REST patterns. The only supported output format at this point is XML, and we rely on Basic Authentication. This section lists the methods available for the REST API.
Basic Authentication
All REST methods rely on the same
Basic Authentication pattern.
- The user name must be auth-with-key@lokad.com
- The password must be an API key, as retrieved from your Lokad account.
As far we know, nearly all decent development environments supports Basic Authentication.
Debugging with your web browser
One of the advantage of REST is that is allows you to start experimenting directly with your web browser, at least for all GET requests. For example, just
try and see for yourself (you will need to login). For other request verbs (PUT and DELETE), you can use simple extensions such as the excellent
Poster add-on for Firefox.
PUT /datasets
Insert a new dataset in your account.
URL: http://api.lokad.com/rest/forecasting3/datasets
HTTP method: PUT
Request:
<Dataset>
<Name>candysales</Name>
<Period>week</Period>
<Horizon>7</Horizon>
</Dataset>
Response:
<ErrorCode/>
Remarks:
If a dataset with the same name already exist in your account, then the call is simply ignored.
GET /datasets
List the datasets available in your account.
URL: http://api.lokad.com/rest/forecasting3/datasets/token
HTTP method: GET
{token} optional argument: continuation token (must have been retrieved from a previous call).
Request:
Request body should be left empty.
Response:
<DatasetCollection>
<ContinuationToken>02k29xk2c82ks</ContinuationToken>
<Datasets>
<Dataset>
<Horizon>6</Horizon>
<Name>candysales</Name>
<Period>month</Period>
</Dataset>
</Datasets>
<ErrorCode/>
</DatasetCollection>
DELETE /datasets
Delete one dataset from your account. Deleting a dataset also delete all the series contained in the dataset.
URL: http://api.lokad.com/rest/forecasting3/datasets/dname
HTTP method: DELETE
{dname} argument: name of the dataset.
Request:
Request body should be left empty.
Response:
<ErrorCode/>
Remarks:
If there is no such dataset, or if the dataset is already marked for deletion, then the call has no effect.
PUT /series
Add decorated time-series to a dataset.
URL: http://api.lokad.com/rest/forecasting3/series/dname?merge=B
HTTP method: PUT
{dname} argument: name of the dataset.
{B} optional argument: boolean value set as
false when the argument is omitted. You can enable time-series merge with
true.
Request:
<TimeSeries>
<TimeSerie>
<Name>lollipop</Name>
<Tags>
<string>sugar</string>
<string>candy</string>
</Tags>
<Events>
<Event>
<Time>2010-06-01</Time>
<Tags>
<string>promo</string>
</Tags>
</Event>
</Events>
<Values>
<TimeValue>
<Time>2010-05-01</Time>
<Value>42</Value>
</TimeValue>
<TimeValue>
<Time>2010-05-08</Time>
<Value>17</Value>
</TimeValue>
</Values>
</TimeSerie>
</TimeSeries>
Response:
<ErrorCode/>
When
B=false (or omitted), it represents an
overwrite semantic: the existing serie (if any) is overwritten by the new serie passed with the request. We suggest to use this behavior unless series are extremely long (which typically happens with hourly, half-hourly or quarter-hourly series).
When
B=true, the new serie is merged with the existing one (if any). Most recent overlapping time-values and events from the existing serie are overwritten with the values passed with the request; older values stay untouched.
Check also the
Incremental time-serie updates paragraph here above for a more visual illustration of the process.
GET /series
Retrieve time-series from your account.
URL: http://api.lokad.com/rest/forecasting3/series/dname/token
HTTP method: GET
{dname} argument: name of the dataset.
{token} optional argument: continuation token (must have been retrieved from a previous call).
Request:
Request body should be left empty.
Response:
<TimeSerieCollection>
<ContinuationToken>jsoxln5028s02ksix</ContinuationToken>
<ErrorCode/>
<TimeSeries>
<TimeSerie>
<Name>lollipop</Name>
<Tags>
<string>sugar</string>
<string>candy</string>
</Tags>
<Events>
<Event>
<Time>2010-06-01</Time>
<Tags>
<string>promo</string>
</Tags>
</Event>
</Events>
<Values>
<TimeValue>
<Time>2010-05-01</Time>
<Value>42</Value>
</TimeValue>
<TimeValue>
<Time>2010-05-08</Time>
<Value>17</Value>
</TimeValue>
</Values>
</TimeSerie>
</TimeSeries>
</TimeSerieCollection>
Remarks:
The number of series to be retrieved for each call is decided on the server side. In practice, you should expect about 100 time-series or 4MB of data (whatever limit is reached first) to be retrieved at each request.
DELETE /series
Delete one or several series from your account.
URL: http://api.lokad.com/rest/forecasting3/series/dname?n=snames
HTTP method: DELETE
{dname} argument: name of the dataset.
{snames} argument: name(s) of the series. In order to delete multiple series at a time, series names must be contacted with semicolon used as a separator:
serie1;serie2;serie3.
Request:
Request body should be left empty.
Response:
<ErrorCode/>
GET /status
Used both to trigger the forecast computation for a given dataset AND to check whether forecasts are available.
URL: http://api.lokad.com/rest/forecasting3/status/dname
HTTP method: GET
{dname} argument: name of the dataset.
Request:
The request body should be left empty.
Response:
<ForecastStatus>
<ErrorCode/>
<ForecastsReady>true</ForecastsReady>
</ForecastStatus>
Remarks:
Once you have finished uploading your data with
/put/series, you must
call this method once to trigger the forecast computations. Then, we suggest we call this method every 5min until the status indicates that the forecasts are ready.
GET /forecasts
Retrieves the forecasts associated to a dataset.
URL: http://api.lokad.com/rest/forecasting3/forecasts/dname?n=snames
HTTP method: GET
{dname} argument: name of the dataset.
{snames} argument: name(s) of the series to be retrieved. If you want to retrieve multiple forecasts (associated to distinct series) in a single request, you must concatenate the names of the series using the semicolon as a separator:
serie1;serie2;serie3.
Request:
Request body should be left empty.
Response:
<ForecastCollection>
<ErrorCode/>
<Series>
<ForecastSerie>
<Name>lollipop</Name>
<Values>
<ForecastValue>
<Time>2010-05-15</Time>
<Value>18</Value>
<Accuracy>0.17</Accuracy>
</ForecastValue>
<ForecastValue>
<Time>2010-05-22</Time>
<Value>21</Value>
<Accuracy>0.19</Accuracy>
</ForecastValue>
</Values>
</ForecastSerie>
</Series>
</ForecastCollection>
Remarks:
You should not assume that the
ForecastSeries will be returned in the same order than the one implicitly specified in the request. You should rely on the
ForecastSerie/Name element instead.
SOAP API methods
We advise NOT to use SOAP (instead of REST) unless you have a excellent SOAP toolkit available.
WSDL URL: http://api.lokad.com/forecasting3.svc?wsdlWeb methods:
string InsertDataset(string identity, Dataset dataset)DatasetCollection ListDatasets(string identity, string continuationToken)string DeleteDataset(string identity, string datasetName)string UpsertTimeSeries(
string identity, TimeSerie[] timeSeries, bool enableMerge)TimeSerieCollection ListTimeSeries(
string identity, string datasetName, string continuationToken)string DeleteTimeSeries(
string identity, string datasetName, string[] serieNames)ForecastStatus GetForecastStatus(string identity, string datasetName)ForecastCollection GetForecasts(
string identity, string datasetname, string[] serieNames)
The first argument is always
identity which should contain the API key as retrieved from your Lokad account.
We suggest to refer to the REST documentation to get an high-level understanding of each method, and then rely on the WSDL to get the XML right.
SOAP exceptions
Due to the lack of consistencies between the SOAP toolkits available in the market, the Forecasting API v3 is not using any SOAP exception, but hard-coded error codes instead.