Discuss (0)

Print

RSS

Developers » here

User guide for Tags and Events

This page is an introductory guide for the framework codenamed Tags+Events. The forecasting technology of Lokad goes beyond classical methods; in particular, we have an advanced framework that can refine its forecasts by taking into account more data than classical methods can handle. This page will tell you how you can take advantage of tags and events to refine your own forecasts.

Table of Contents [Hide/Show] User guide for Tags and Events    Classical forecasting model    Limitations of the classical model       Short time-series       External factors    The solution brought by Tags+Events       Tags       Events       Known-since property of events       Data acquisition for tags and events       Conclusions    Choosing Tags+Events in retail and manufacturing       Products through tags       Promotions through events       Product launches through tags and events       Other type of events    Going further with Tags and Events       Dealing with quantitative variables       Dealing with low frequency events       Privacy though obfuscation

Tags and Events are useful in retail or manufacturing to better forecast when there are:

• Product launches
• Promotions
• Marketing operations
• Shortages
• Cannibalization
• ...
  

For services companies such as call centers, Tags and Events help when there are:

• Marketing operations
• Disruption in the service
• Local events
• ...
  

This page does not focus on the nitty-gritty technical details how to manage Tags+Events data and how to communicate with the Forecasting API. The goal here is to give you a good understanding of the purpose of tags and events, and how they are intended to be used in real-life situations.

Classical forecasting model

At Lokad, we deliver time-series forecasts. A time-series is nothing but a list of date and value pairs. Any phenomenon that can be measured over time can be represented as a time-series.

For example, in retail, the sales of each product (each SKU actually) can be represented as a time-series, with one data point per week if a weekly data aggregation is chosen for example.

Classical forecasting models take time-series as inputs - those time-series represent the historical data of the company - and return time-series as output too, those outputs represent the forecasts of the company sales.

Simply put, forecasting consists in prolonging a time-series into the future. Although, there could be many ways to do this, it is now widely recognized that for most short-term and mid-term forecasts, if historical data is available, statistical forecasts tend to outperform all other approaches.

Limitations of the classical model

Our framework Tags+Events extends the classical time-series forecasting model though additional meta-data, namely tags and events. In order to understand why Lokad features such an extension, let us start by gaining further insights about the limitations of the classical forecasting model.

Short time-series

The classical model relies only on the information contained in the raw time-series to produce the forecasts. If we consider a short time-series, for example the monthly sales of a product aggregated per week that has been launched only 4 weeks ago, the amount of information contained in the time-series is very limited: only 4 data points in this situation. Obviously, it’s out of the question to consider even moderately complex patterns such as seasonality, since we would need 12 months of history for such an analysis. Then, even if we tried to match the 4 weeks with other time-series to obtain a seasonality through data correlation, 4 weeks is still very likely to be too short to provide any reliable matching to infer on a pure statistical basis a reliable seasonality for the product. With short time-series, the classical forecasting approach is crippled. Obviously, it’s out of question to forecast product launch as well.

External factors

The classical model does not apprehend external factors that are impacting time-series, as all the information is supposed to be contained within time-series. Yet, if we consider an external factor such as a retail promotion, it causes two problems:

1. past promotions have artificially inflated historical sales, and unless the data gets corrected (somehow), the forecasting model will over-estimate past sales level, and consequently over-estimate sales forecasts as well.
2. if a promotion is planned in the future, the classical model cannot take this element into account, as a result, the forecasts matching the date interval of the upcoming promotion will be under-estimated.
   

Although the classical model is appropriate to evaluation internal factors in data such as trend, it’s crippled by design when it comes to external factors. Note that classical model could theoretically handle external factors through covariate, but that’s somehow stretching the definition what is typically considered as classical and widespread in time-series forecasting. Most forecasting tools do not handle natively such extraneous information, those that do, usually provide little or no automation.

The primary purpose of our framework Tags+Events is to provide an efficient purely statistical solution to the problems outlined here above.

The solution brought by Tags+Events

The limitations listed in the previous section can be summarized as follow: time-series alone are not expressive enough to reflect the complexity of the phenomenons that we are trying to forecasts. Hence, we need to enrich the data model with extra elements that come on top of the raw time-values. Tags and events are precisely those extra elements.

Tags

A tag is nothing more than a keyword or term. We use the word tag instead keyword to emphasize that we don’t focus on the meaning of the keyword, the tag is just used as an identifier. Lokad focuses on raw statistical processing, not semantic and linguistic analysis. Each time-series can be associated with several tags (up to 1000 tags in current Lokad limitation) or eventually no tags at all.

Example: for candy retailer, the tag lemon could be applied to all time-series being associated with candy sales when the underlying candy comes with a lemon flavor.

Tags are user-defined; hence any combination of letters could be used as tag. It does not have to be a word that exists in the English language. The tag reflects a description of some sort specified by the user herself.

The usefulness of tags comes from comparisons that can be drawn from multiple occurrences of the same tag. This implies that you need several time-series (in practice a dozen or more) tagged with the same tag; otherwise no statistical patterns can be inferred from the presence (or absence) of the tag.

Example: for a candy retailer that has applied the tag lemon to all candies that come with a lemon flavor, Lokad can leverage the lemon tag to figure out if there are specific sales patterns for lemon-flavored candies, such as a shared seasonality for example.

Then, tags can be combined too. Although we said that a single time-series cannot be associated to more than 1000 tags at once, there is no limit on the total number of distinct tags. Hence, it is possible to create highly descriptive sets of tags. Lokad is able to leverage complex tag combinations to detect subtle statistical patterns to further refine the forecasts - not just considering naive partitions based on a single tag as illustrated here above.

Events

Events are similar to tags, but slightly more complex. An event is a basically dated tag that is to say a tag that comes with an extra date information that first reflects the time at which the event took place. More precisely, a event comes with 4 properties:

• the token (we don't use the word tag again to avoid confusion with Lokad tags).
• a start date.
• a duration.
• a known-since (this property is a bit more subtle and will be detailed later on).
  

Example: a candy retailer is performing every year for the week before Christmas a promotion on chocolate lollipops buy one and get one free. The time-series associated to the sales of chocolate lollipops gets decorated with events codenamed buy1get2. The event is starting on December 18 every year, and associated with a duration of 7 days. By applying the event in advance at the beginning of December, the retailer is able to anticipate 3 weeks in advance the right number of chocolate lollipops to be provisioned, while taking into account the upcoming promotional effect.

If the same event (i.e. events that have the same token) occurs multiple times within the same time-series, Lokad is able to extract the statistical pattern associated to the event occurrence. When the event gets applied to time interval located in the future, Lokad adjusts its forecasts by applying within the forecasts the statistical pattern associated to the event.

Like for the tags, Lokad needs several events with identical token to identify similarities in time-series and consequently refine its forecasts. By convention, events with same identical token are said to be similar. Yet, similar events can be spread over multiple time-series. The situation where a single time-series has similar events, is the easiest case to analyze, but in practice, situations are frequently more complicated and a single product may have only a single event in its whole lifetime. By comparing the event effect on other products, Lokad refines its forecasts, even if the event has never occurred before on the considered product.

Example: a candy retailer promotes every year during the week before Christmas a promotion on all lollipops buy one and get one free. Time-series associated to the sales of lollipops are decorated with tokens buy1get2. Yet, each year, the candy retailer choose a different set of flavors for the lollipops, in order both to renew the shop display, but also to closely fit the market trends. Each year, there are new lollipops flavors for sale, yet, for those new ones, the retailer does not have historical data related to previous Christmas promotions. Nonetheless, Lokad is able to include the effect of buy1get2 using the pattern previously observed with other lollipops.

Then, events are allowed to overlap. This property is used to create powerful event description through token combinations. Exactly like products that can be described with multiple tags, a single business event can be described with multiple Lokad events, sharing the same temporal properties, but having distinct tokens.

Known-since property of events

In the previous section, for the sake of simplicity, we detailed only the top 3 properties of an event: a token, a start date and a duration. Yet each event comes with a 4th property: the known-since date. Since the purpose of this property is slightly more subtle, we have kept the explanation for the present section.

The known-since property represents the point of time when the information concerning the event has become available to the system performing the forecast (i.e. Lokad). In other words, the existence of the event has been known since that particular date.

Certain event, such as promotions are typically known weeks in advance, while some others such as power outage are not. The known-since property indicates whether past events could have been considered as known in advance to refine the forecasts or not.

The reason why this subtle aspect matters is because accuracy measurement is the most critical aspect of any good forecasting technology (Lokad being no exception). The information carried by events typically improve the accuracy, but if in real business situations, some events are not known until after the situation occurred, the historical data, which include data acquired afterward, could give somehow a false sense of accuracy to the forecasting system, leading to sub-optimal optimizations while learning the forecasting model from the data.

Data acquisition for tags and events

We have described how tags and events can used to refine the forecasts. Although, we have said that both tags and events are arbitrary and user-defined (tags and tokens being an arbitrary combinations of letters), tags and events are typically not manually entered into Lokad.

Indeed, Lokad aims not only to enable process optimization but to enable process automation as well. For a retailer with thousands of product references, to enter thousands of tags and events manually would be not only be a very tedious, time-consuming task, but also an error-prone process.

Hence, we typically advise to setup a process to automatically extract tags and events from the underlying company IT solution (typically an ERP, MRP or CRM software). Although this process is typically domain specific, if not company specific, general guidelines are provided in the following.

Conclusions

We have detailed limitations associated to the classical forecasting model. Lokad addresses those limitations by introducing a richer data model that not only handle raw time-series, but generic meta-data as well named tags and events. Both tags and events are used to decorate time-series. Tags introduce an implicit notion of similarity between entire time-series, while events introduce an implicit notion of similarity between specific time intervals within time-series. Those similarities are exploited by Lokad to refine the forecasts in ways that are not available for classical models.

Choosing Tags+Events in retail and manufacturing

This section focuses on the usage of tags and events for retail and manufacturing. Typical situations in those verticals consist of forecasting product sales for each SKU (Stock Keeping Unit) in order to optimize the amount of inventory in each SKU. We detail here how tags and events can be used to improve inventory management through refined forecasts.

Products through tags

The primary purpose of the tags is to circumvent the limitations of the classical forecasting models that only take into account historical demand levels in order to compute the forecasts. Tags provide an extra set of information beyond strict historical data (that is to say historical sales).

Products or goods can be described in many and very detailed ways starting from their physical properties (color, weight, dimensions, material …) up to more abstract properties (product family, brand name, duration of the warranty, ...).

On the other hand, tags have a limited expressiveness: for the time-series associated to a product sale history, a tag can only be present or absent, and there is a limit of 1000 tags per time-series. Although tags are still remarkably expressive despite their simplicity, encoding an extensive fine-grained description of the physical properties of the product is out of question.

Yet, we focus on effective, not all-inclusive descriptions product descriptions. Since the task on hand here is demand forecasting, the effectiveness of the description is judged according to the amount of improvement brought to the forecast accuracy through the additional data.

Although this might appear as a complex endeavor at first, a couple of simple guidelines can be used to choose tags that will bring effective improvements.

The main idea is that the description should not focus on products themselves but rather on the relationships they entertain, and especially, on relationships that potentially uncover similar sales patterns. Typically, the product hierarchy as expressed in the retailer or manufacturer catalog is a very good starting candidate for tags.

Example: a candy retailer has the following catalog, the product references are put between parentheses:

• Gum
  • Coke flavor (1)
  • Lemon flavor (2)
  • Mint flavor
    • White color (3)
    • Light Green color (4)
• Lollipops
  • Apple flavor
    • Light Green color (5)
    • Green color (6)
  • Lemon flavor (7)
  • Mint flavor
    • White color (8)
    • Light Green color (9)
• Jelly Beans
  • Coke flavor (10)
    

There are 10 candies in the catalog. The following tags are chosen:

• Gum, JellyBeans, Lollipops, for the candy categories.
• Apple, Coke, Lemon, Mint for the flavors.
• Green, LightGreen, White for the colors.
  

Now, we can associate the following tags with each candy reference:

1. Gum, Coke
2. Gum, Lemon
3. Gum, Mint, White
4. Gum, Mint, Light Green
5. Lollipops, Apple, LightGreen
6. Lollipops, Apple, Green
7. Lollipops, Lemon
8. Lollipops, Mint, White
9. Lollipops, Mint, LightGreen
10. JellyBeans, Coke
    

We have said previously that having tags that appear only once in a single time-series is of little use since no comparison could be directly inferred with other time-series. Although this is roughly true, Lokad is still able to make limited use of tags that appear only once. It can still be useful to refine the amount of difference between two time-series as reflected by differing tags, even if some tags happen to be encountered only once.

In practice, we suggest to pay more attention the overall hierarchy, as opposed to try to micro-optimize tags. Lokad is resilient to noise. The presence of many tags that are not usable for forecasting purposes is not a problem as they will be filtered out. As long as valuable tags are not massively over weighted by noisy tags, forecasts won't be negatively impacted.

As a rule of thumb, the more meaningful and readable the hierarchy, once flattened as a set of tags, the more likely it will convey good and useful information to refine the forecasts.

Here is a brief list of properties that could be used as effective tags:

• Location codes for SKU, as the same product can be sold in different points of sales. The location tag can be used by Lokad to identify sales patterns that are specific of the point of sales.
• Price positioning, although tags are purely qualitative (a tag can’t represent a quantity), the price positioning is usually an important sales factor. Typical positioning would be: first price, best price/quality ratio, top quality.
• Facing options, such the row chosen for display.
  

Promotions through events

Promotions are important events that strongly influence the sales. Taking promotions into account is important for two reasons:

• To anticipate demand spikes that could not have been forecasted otherwise. This is the direct and most obvious use of the information concerning promotions.
• To remove bias from historical data that would be introduced in other patterns. For example, if a past promotion is not flagged as such, it could adversely impact the seasonality estimation, leading to incorrect figures.
  

Check also our post about Better promotion forecasts in Retail for more details.

Exactly like products, promotions could be described with an enormous amount of fine-grained details. Yet, our focus is not to provide all-inclusive details about promotions, but rather to emphasize the salient aspects that are the most useful to refine the promotion forecasts.

We have seen that each Lokad event has 4 properties: a token, a start date, a duration and known-since date. In order to represent a complex business event such as a promotion, several events are typically used. Since all those events are related to the same business phenomenon, we suggest letting them share the same values for the 3 temporal properties: start date, duration and known-since date.

Hence, a complex event can be viewed with 4 defining properties: a set of tokens, a start date, a duration and known-since date.

Let see how each one of these 4 properties should be defined. Temporal properties are rather obvious:

• the start date reflects the day the promotion is started,
• the duration reflects the duration of the operation,
• the known-since date represents the time when the promotion has been decided (typically weeks in advance compared to the start date of the promotion).
  

Since the known-since property is slightly unusual, typical ERP does not keep track of this information. Yet, for promotions, we can safely assume that they are always known in advance. Hence any date positioned well before the start of the promotion could be used. For example, January 1st, 1901 could be as a mock placeholder value expressing the fact that promotions are just known in advance,

The set of tokens is obviously the property that holds most of the information defining the promotion. What we said for tags remain true: tokens should focus on similarities that exist between business events.

Typically, a promotion comes with two sets of descriptors:

• The mechanism that describes the nature of the discount.
• The communication that describes how your customers get notified.
  

The mechanism is, for example, buy one and get one free. Large retailers typically have dozens of promotion mechanisms, ranging from immediate discount, to more complex schemes that involve multiple products and eventually multiple operations from customers.

Discounts are typically expressed as percentages (for example: -15% on the product price) yet tags are qualitative attributes, not quantitative ones. See the section Dealing with quantitative variable below to fine-tune your tag representation in presence of quantitative factors.

The communication typically involves several channels:

• A packaging for the promoted product.
• A specific facing such as shelf end.
• Local radio advertising.
• Direct customer mailing.
• ...
  

Each channel would typically be associated with its own codename that needs to be placed in the description of the promotion.

Example: a candy retailer has the following event representation

• Mechanism:
  • Buy1get2
  • Buy2get3
  • Buy1minus20percent
• Communication
  • WindowsDisplay
  • NeighborMailing
  • PromotionShelf
    

The retailer is planning a spring promotion where customers can buy 3 licorices for the price of 2. There will be a big sign displayed on the shop window and a direct mailing made to the neighborhood, but there will be no custom shelf dedicated to promoted licorices. The promotion is planned to start on April 15th and to last for one week. The description for the promotion in the Lokad framework is:

ProductRef: Licorice (also the name the time-series associated to the licorice sales)
Names: Buy2get3, WindowDisplay, NeighborMailing
Start-date: April 15th
Duration: 7 days
Known-Since: March 1st

It should be noted that product tags, that comes on top of the events, are also very useful for promotion forecasts. Indeed, Lokad will be able to match promotion patterns more effectively if product descriptions are provided on top of events.

Product launches through tags and events

Product launch is typically a pathological situation for most forecasting tools, because there are precisely no sales data available for the product being launched. Nonetheless, Lokad is able to forecast product launches in a fully automated manner. As usual, the process is purely statistical: Lokad leverages patterns observed for previous product launches in order to figure out the sales pattern for the product being launched.

Needless to say, forecasting a product launch requires a significant sales history with typically dozens or more of already-launched products. If the product being launched is the first product ever sold by the company (ex: start-up case), then no statistical approach is tractable.

From the Lokad viewpoint, product launches are very similar to promotions, except there are no historical sales data available for the product being launched. In order to represent a product launch, we suggest to include 3 distinct data:

• a zero time-value just before the launch.
• a set of tags to describe the product.
• a set of events to describe the launch itself.
  

The purpose of the zero time-value is to give Lokad a context so that a forecast can be made. Indeed, Lokad has a data centric approach when it comes to forecasting. Simply put: forecasts start where historical data ends. The zero provides a hint so that Lokad can position the first forecast correctly.

In the context of a product launch, tags are required because Lokad has no historical sales to rely on. Lokad is entirely dependent on the information contained within other product sales.

Finally, events are not required, but since the amount of information contained in the tags is thin compared to a usual forecasting situation with an available sales history, extra information that is provided through events is likely to significantly improve the accuracy.

Other type of events

Promotions are probably the most important patterns that can be modeled through events in retail. There is no need to tell Lokad about seasonal events such as Christmas. Lokad offers an implicit native management of such recurring that impact a large amount of businesses.

There are other types of events that could be modeled such as:

• Partial shortages, which would reflected as a drop of the demand in the data, while it actually reflects that, at some point, it wasn’t further possible to buy the product.
• Special closed day, that might be caused by miscellaneous causes such as inventory count, storm, strike or construction works.
  

At this point, Lokad does not handle weather data as input in order to refine the forecasts based on local weather patterns. This feature is part of our mid-term roadmap.

Going further with Tags and Events

This section deals with more subtle situations that have not been detailed previously for the sake of clarity.

Dealing with quantitative variables

The Tags+Events framework does not handle natively quantitative variables. For example, if we consider a candy retailer applying a 10% discount on lollipops and a 30% discount on jelly beans, we could create two tokens Discount10 and Discount30, yet it won’t tell Lokad that a relationship existing between the two discount tokens.

In order to reflect the relationship between quantitative values through qualitative variables (also called discreet or Boolean variables), we can establish a discrete graduation, for example

• -10% gets associated with: Discount10
• -20% gets associated with: Discount10, Discount20
• -30% gets associated with: Discount10, Discount20, Discount30
• ...
  

This method is efficient but should be abused as going for a too fine-grained granularity as such a fine-grained scale would be likely to worsen the accuracy. Indeed, it would somehow flood informative tokens into a large soup of not so informative ones. In practice, we suggest to keep graduations smaller than 5 tokens unless you have very good reason to do otherwise.

Dealing with low frequency events

Some variables could also implicitly reflect a quantitative phenomenon but with a subtle frequency twist. For example, the more extreme the phenomenon, the greater the impact on the business. Yet extreme events tend to be very rare as well, which make them harder to quantify.

Example: rain, storm and hurricane adversely impact the sales of a candy retailer, the worse the weather, the worse the greater the negative impact on the sales. Yet, rain, storm and hurricane happen with radically different frequencies. At the retailer's location, they average:

• Rain happens 10 weeks per year (1).
• Storm happens 5 days per years (2).
• Hurricane happens once every three year (3).
  

Considering that the retailer has 10 weeks of rain every year, just looking at the data of last year is sufficient to quantity the impact of the rain on its retail business (the statistical method might be complicated, but at least the data is present). On the other hand, last year's data might not even contain a single hurricane occurrence. Hence, it is hard to quantify the impact of a hurricane.

The challenge is that without further information, the statistical model is likely to ignore the hurricane event simply because it does not have enough data to establish a robust pattern based on very few past occurrences. As a matter of fact, there is only two hurricane occurrences in the whole business history.

Thus, following the same approach that we had in the previous section for quantitative variable, the candy retailer goes for a gradual representation of those whether events:

1. Rain
2. Rain, Storm
3. Rain, Storm, Hurricane
   

This will allow Lokad to refine its forecasts for hurricanes based on storm or rain events. If there is not enough data to support the quantification of the hurricane event, Lokad would still be able to approximate the effect of the hurricane through more frequent storms.

The approach that we have just presented is a powerful way to deal with low frequency extreme phenomenon. Again, we suggest not abusing the method, as it is usually best to stick with limited graduation of 2 or 3 tokens.

Privacy though obfuscation

Tags and events may represent sensitive information for your business. Although protecting the data of our customers is a top priority for Lokad, you can also decide to obfuscate the data send to Lokad. This process ensures that the data made available to Lokad cannot be used for any other purpose than strict statistical forecasting. For more details, check our page about obfuscation.