Skip to main content

Celonis Product Documentation

How to design the data pipeline? Replications vs Data Jobs

Replication Cockpit (RC) and Data Jobs (DJ) are 2 different tools that basically serve the same purpose - define and orchestrate how the data is fetched from the source system to the Celonis Platform. RC does not support all of the ETL use cases yet, and it has to co-exist with the DJs to enable a fully functional data pipeline.

The matrix below summarises for which use cases RC is the recommended tool. In the current public version RC supports real-time extractions and real-time transformations - both for Transactional and Metadata tables. Additionally, the Replication Cockpit allows you to execute Full Extractions (called Initializations).

While there is no exact formula to distinguish between Transactional vs Metadata tables, the general approach is that if a table includes a Case or Activity, then it is Transactional and should be extracted in real-time. Examples are EKKO, EKPO, CDPOS, CDHDR, BKPF, etc. As for the Metadata tables, they include relatively static information that is updated not very frequently and/or is not very important from an analytical perspective. These tables can be extracted both via RC and DJ. All of T.... and D.... tables fall in this category, and sometimes the master data tables (examples - KNB1, LFA1, MARA, MARC) can also belong here, assuming their data is not important for operational use cases.

Full

Delta

Transactional Tables

Extraction

RC

RC

Transformation

DJ

RC

Metadata Tables

Extraction

DJ or RC

DJ or RC

Transformation

DJ

DJ or RC

Data Model Load

DJ

Ultimately, RC can be utilized to run the continuous data replication from the source system, covering transactional and also if necessary metadata tables. The Initialization functionality allows you to execute full loads (of extractions) in the RC.  Meanwhile, data model loads can only be orchestrated via DJs.

Note

The executions of Full Extractions (in Data Jobs) and Replications (in Replication Cockpit) should be coordinated to avoid conflicts.

We recommend using the Initialization functionality in Replication Cockpit for Full Loads instead of doing it in the Data Jobs.

When Full Loads are still being executed in Data Jobs the corresponding tables should be deactivated in the Replication Cockpit because:

  • there can be only one Data Push Job per table at the same time

  • otherwise it can lead to data loss

This can be easily done by using the Replication Calendar.

On the data connection level, you can set a limit on the number of parallel extractions. This limit counts for Extractions from Data Jobs as well as Replication Cockpit. Therefore, ongoing full loads can delay the processing of replications.

image2022-4-20_16-31-54.png