Running transformation data jobs
Custom transformation data jobs define how data from your source systems is converted into objects and events in Celonis. Running these data jobs applies the rules you’ve set up, creating new objects and events or updating existing ones when attributes or relationships change. This ensures that your data model stays accurate and up to date, reflecting the latest structure and information from your data sources.
When you publish custom transformations, Celonis stores them in test:ocpm-data-job for the development environment and ocpm-data-job for production. These jobs include both your custom transformations and the built-in ones. Simple transformations without joins or filters are optimized into views, with the executed SQL included as a comment. You can run or re-run these transformations to ensure your data is up to date.
Transformation data job loads
When running your data jobs, you can choose between full and delta loads:
Full load: You extract and transform all of the table’s data. This is recommended when testing your data connection and data jobs and for the first time data is extracted from your source system.
Delta load: You extract and transform only new data that has been added to a table since the last load – rather than all of the table's data. Using delta loads reduces your execution time and allows you to extract and transform only new data.
For more information about delta load, see our Data Integration content: Executing delta extractions.
Before you begin
You can only run transformations after deploying at least one version of your object-centric data model to development or production.
To learn more about versioning and deployment, see: Versioning and deploying OCDM.
Tip
This procedure is shown using the Objects and Events dashboard. Alternatively, you can open your object-centric data pool from Data → Data Integration and continue from step 3.
To manually run your data transformation jobs from the Objects and Events dashboard:
Click Version history.
Click Deployments and then click to open either the development or production data pool.
The object-centric data pool opens.
With your object-centric data model displayed, click Data Jobs.
For either the development or production data jobs, click Options - Execute Data Jobs.
Development: Labeled as
test:ocpm-data-job.Production: Labeled as
ocpm-data-job.
Select the perspectives to include in the data job.
Notice
Any simple transformation scripts that don't include a join or WHERE clause, aren't new scripts, or haven't changed since the previous run will not be included in the transformation and will be replaced with a view.
Select either Delta Load or Full Load.
Click Execute Selection.
The data job runs and a your data job log screen is displayed, showing the current status of the job.
Schedules are used to automatically execute data jobs on a predefined basis, allowing you to control the day, time of day, and repetition of your executions. By default, schedules run sequentially, meaning that they attempt to successfully execute each task before moving on to the next. If a task fails to execute, all remaining tasks are canceled.
For more information about scheduling data jobs, see: Scheduling the execution of data jobs.
 |