Near Parallel Data Loading using EPM Groovy REST API

Data Loading using EPMAutomate for a range of periods is always treated in sequential manner.  This in part due to how Data Management processes data between the staging and mapping tables under the covers, just look at the process log.  The Data Exchange framework offers some efficiencies and ways to speed up data loading.  Using EPMAutomate still processes 1 period at a time even Quick Mode.

In recent project when loading Forecast for 1 year it took 20 min using Quick Mode.

And we needed to load 3 years, and the forecast needed to be reloaded several times a day during the forecast cycle, so waiting nearly an hour was not an option.

As I pointed out in my blog post EPM Groovy RestAPI vs Groovy EPMAutomate for Data Loading REST APIs calling EPMAutomate uses POST and requires a GET to retrieve status.  

This presents an opportunity to think out of the box and after consultation with Joe Aultman the Evangelist of Groovy we decided that instead of sending a range in one EPMAutomate we would loop through the 12 periods and send POST to runIntegration for a period. And looking at the process details we saw 12 active integrations.

First (1) we need to create a List of the POST calls then iterate (2) through the list to GET the status of the runIntegration and once all 12 integrations finished the code proceeded to next step.

 

 

Using this approach we were able to cut the data load time down from 20 minutes per forecast year to less than 6 minutes per forecast year.

 

EPM Groovy RestAPI vs Groovy EPMAutomate for Data Loading

Lately I have been doing a lot more EPM Data Integration work.  I have been trying to move to more server side automation using Groovy in Business Rules to replace on-premises EPMAutomate scripts for most of my data loads.  

Here is link to documentation.

Before this was available, there was and still is the ability to call REST APIs to invoke actions within EPM.

Here is link to documentation.

One of the reasons I like using server side automation, is that it puts the responsibility of loading data back in the hands of the Business user. 

From a developer and development perspective Groovy and EPMAutomate I believe is easier and has less steep learning curve over Groovy and REST API. Additionally you can do same action with a lot less code using Groovy and EPMAutomate over REST API.

Here is example of running data integration with REST API.

One thing about REST API. There are 2 main commands used. POST sends action to request to remote server.  GET gets status or response of the POST.  So after you send POST you need another bit of code to check if the requested action of the POST has finished and if it was successful or was an error.  So you need a block of code like the following:

 

Now with EPMAutomate, it is different.  The framework behind the scenes has a wait, so all that is needed is to collect the status of the execution.

 

 

And by using a function logging is cleaner

Each approach has their merits and use cases, however with the introduction Groovy rules being able to run EPMAutomate commands the development time and lines of code are greatly reduced.