Skip to main content

Celonis Product Documentation

Supporting content for SAP ECC and SAP S/4HANA data connections

The following sections can be used when configuring your connection between SAP ECC and SAP S/4HANA and the Celonis Platform:

Connection Test

When you test a connection, the following steps are performed:

  • Check whether all the function modules exist

  • Write an empty file on the SAP server in the defined folder

  • Compress the newly created file (if compression is enabled)

  • Send the file to the extractor

Therefore the following aspects are tested:

  • Existence and completeness of the RFC module

  • Writing rights on the SAP server in the respective folder

  • Reading rights on the SAP server in the respective folder

  • Correctly configured compression method

  • Open firewall ports to send RFC call to the SAP system

  • Open firewall ports to send files from the extractor to the cloud endpoint

Connection Test Troubleshooting

Below you can find the errors that may cause the connection test to fail and the respective solution(s), grouped by the connection checklist task.

When a checklist task fails in the SAP connection test, you can click on the button Help, to navigate to the relevant solutions in this page.

41195122.png
SAP JCo not found on classpath, ensure 'sapjco3.jar' exists

Solution

The JCo library Java part (sapjco3.jar) needs to be placed within the 'jco' folder of the extractor. You can download the respective package from the SAP Marketplace.

Note: This step should be performed by someone who has an SAP Service User (S User), which authorises to download software from SAP portals. Usually the SAP BASIS has such access.

A restart of the Extractor is required after the installation.

SAP JCo native library not found

Solution

The JCo native library file sapjco3 (.dll / .so / .sl / .dylib) needs to be placed within the 'jco' folder of the extractor. You can download the respective package from the SAP Marketplace.

Note

Only an SAP Service User (S User) can download software from SAP portals. Usually, a customer's SAP BASIS has this access.

The downloaded jco folder already contains two files: a Java part “sapjco3.jar”, and an operating system-specific part e.g. sapjco3.[.dll | .so | .sl ].

Copy the Library files into the external folder located at:

{installation_folder}/shared/libraries/external.

The Celonis Agent is configured to read the library from this directory.

Note

After setting up the jco, the Windows Service Celonis Agent needs to be restarted for the jco related changes to take effect.

Unsupported operating system for SAP JCo native library

Solution

The JCo native library file sapjco3 (.dll / .so / .sl) is not supported by your operating system. Make sure to place the correct library file for your respective operating system.

A restart of the Extractor is required after the installation.

Solution

The Visual 2013 C++ Redistributable Package needs to be installed on the server.

A restart of the Extractor is required after the installation.

The connection between the Extractor Server and SAP could not be established

Solution

Check if the Host and the System Number are specified correctly in the Data Connection.

If you are using Message Server as middleware, check if the Message Server Host and the Message Server Port are specified correctly.

Check if the following network connectivity requirements are satisfied:

Source System

Target System

Port

Protocol

Description

On premise extractor server

SAP ECC system

33XX (where xx is the system number)

TCP

RFC connection from on premise extractor server to the SAP system. The system number can be retrieved from the SAP basis team.

Error during connection validation

Solution

Please check the specific error message that is provided in addition.

If this does not help, open a support ticket.

Missing authorizations for the Celonis RFC User

Solution

The Celonis RFC user is missing required authorizations. Make sure you have downloaded the dedicates user role from the Download Portal.

Afterwards follow these steps to apply the correct role to the technical user of the RFC connection:

  1. Run the transaction PFCG

  2. Go to Role → Upload

  3. Select the provided role file (CELONIS_EXTRACTION.SAP)

  4. Apply the uploaded role CELONIS_EXTRACTION to the RFC user

If you wish to set up the user yourself, make sure that the user has the following permissions:

  • Permission to write on the server hard disk under the path specified in step B

  • Permission to execute all RFC modules contained in the transport imported in step A (package name /CELONIS/DATA_EXTRACTION )

  • Read access to all tables that should be extracted and DD02L and DD03L for retrieving the table and column names

Necessary Celonis Function not implemented in SAP

Solution

You have not installed the Celonis RFC Module or you have an outdated version in place.

For all the installation files, go to the Download Portal.

Please update your RFC Module to use the 'Zip' compression method

Solution

The minimum requirement is RFC Module version 1.6

For all the installation files, go to the Download Portal.

RFC Module call failed

Solution

The called RFC function resulted in an internal error.

You can check the specific error message. If that doesn't help to fix the error, open a support ticket.

Your RFC module version does not support usage of native SAP compression. Please update your RFC moduleto use this feature

Solution

The minimum requirement is RFC version 1.6. As Extractor version 2019-11-12is required.

For all the installation files, go to the Download Portal.

In case an upgrade is not possible, you can switch to gzip compression.

Checking writing of parquet files

Solution

The Extractor Service is lacking permissions to write to the 'temp' folder in the extractor directory.

If the permissions are fine, open a support ticket.

Test file writing failed

Solution 1: (Z_CELONIS_TARGET is set to its default value):

  1. Create a dedicated directory on your SAP server (preferably a network drive) and make sure that the user running the SAP system has reading and writing access to it

    Warning

    In case you are using Logon Groups, or there are multiple application servers where the extraction jobs can run, make sure that the directory is on a network drive, and is accessible from all the servers.

  2. Run the transaction FILE in your SAP system

  3. Find the Logical Path Z_CELONIS_TARGET in the list

  4. Edit the entry by clicking on the button to the left of the new entry and then double-click on the folder "assignment logical and physical paths"

  5. Set a path for UNIX and/or Windows (shown in the screenshot on the left); the path should be the directory on the system or a network drive that you created in step 1

Solution 2:

The logical path Z_CELONIS_TARGET is not accessible from the application directory.

If you are using load balancing, please make sure to use a network drive instead of a local directory.

Missing authorizations for the Celonis RFC User

Solution

The Celonis RFC user is missing required authorizations.

Afterwards follow these steps to apply the correct role to the technical user of the RFC connection:

  1. Run the transaction PFCG

  2. Go to Role → Upload

  3. Select the provided role file (CELONIS_EXTRACTION.SAP)

  4. Apply the uploaded role CELONIS_EXTRACTION to the RFC user

If you wish to set up the user yourself, please make sure that the user has the permissions below.

  • Permission to write on the server hard disk under the path specified in step B

  • Permission to execute all RFC modules contained in the transport imported in step A (package name /CELONIS/DATA_EXTRACTION )

  • Read access to all tables that should be extracted and DD02L and DD03L for retrieving the table and column names

Test file permissions failure

Solution

Please make sure that the operating system as well as the SAP system have reading and writing permissions for the logical path Z_CELONIS_TARGET.

Test file compression failed

Solution

Make sure the command (for your compression method) is configured in SM69. Check if the command is valid and that an executable exists.

Compressed test file not found

Solution

Check if the command (for your compression method) configured SAP transaction SM69 is configured correctly.

Test file deletion failed

Solution

Make sure to download the Celonis User Role from the Download Portal and add it to the RFC user.

Z_CELONIS_TARGET is set to its default value. Please check there is enough space in this location, and if you have multiple servers configure it to a network share

Solution

  1. Create a dedicated directory on your SAP server (preferably a network drive) and make sure that the user running the SAP system has reading and writing access to it

    Warning

    In case you are using Logon Groups, or there are multiple application servers where the extraction jobs can run, make sure that the directory is on a network drive, and is accessible from all the servers.

  2. Run the transaction FILE in your SAP system

  3. Find the Logical Path Z_CELONIS_TARGET in the list

  4. Edit the entry by clicking on the button to the left of the new entry and then double-click on the folder "assignment logical and physical paths"

  5. Set a path for UNIX and/or Windows (shown in the screenshot on the left); the path should be the directory on the system or a network drive that you created in step 1

Listing files failed

Solution

The application server cannot access Z_CELONIS_TARGET.

Make sure to download the Celonis User Role from the Download Portal and add it to the RFC user.

On average, you should expect that the following extraction throughputs:

  • Large tables - such as BSEG or 100k records/minute.

  • Mid size - such as EKKO, VBAK, or 250k records/minute.

However, these numbers are just an approximation and the actual duration can vary significantly depending on the source system. To understand the variance, it is important to understand the overall flow and services involved:

  1. The extraction is based on an SQL query, which is executed at the database level.

  2. The output is stored in the SAP memory.

  3. From memory, the data is written to .csv files.

  4. These .csv files are then fetched by the On-Premise-Client to be uploaded to the cloud.

A problem in any of these steps will impact the performance of the overall runtime. The following are the most common factors that can impact performance:

  • Database and SAP server capabilities

    The more resources the database and the application server have, the faster the extraction query will run.

  • Source system load

    Running the extractions during the busy hours will lead to slower performance since there are more requests competing for the same resource pool.

  • Complexity of the applied filter and table size

    This is a very important and often neglected factor. If you apply a filter to non-primary keys, the runtime will be way slower than with a PK-based filter which is indexed in the database. The table size also plays a big factor, especially with non-PK filters. Extracting 1 million records out of 100 billion won’t have the same duration as extracting 1 million out of 100 million.

  • Size of the table

    Depending on the business, the size of the same table can vary significantly. This will lead to different runtimes for the same table, even if the number of extracted records is the same.

  • Network bandwidth between the OPC and SAP Servers

    OPC fetches the files from the SAP server, and if the network has a low bandwidth, this will impact the data transfer and consequently the overall extraction speed.

When upgrading from SAP ECC to S/4 HANA, the Celonis data pipeline may be affected, particularly in the Extraction and Transformation layers. Below, we explore various scenarios and their expected impacts.

Celonis RFC Extractor

The RFC module is an ABAP application that is compatible with both SAP ECC and S/4 HANA. It uses no version specific functionality and runs seamlessly in both systems without any migration. The only exception is when the Real-Time Extension is activated, in which case a migration is needed.

Real Time Extension is not active

Minimal migration effort is needed in this scenario. The RFC Module is used only for batch-based extractions and there are no Celonis change logs or triggers installed in SAP.

In this scenario follow these steps:

  1. Confirm the successful transfer of the RFC Module in S/4 HANA. Search for the package /CELONIS/* (the full name varies depending on the extractor version).

  2. Set up the target path Z_CELONIS_TARGET as explained in Installing RFC Module.

  3. Check that the Celonis user exists in S/4 and has the appropriate permissions.

Real Time Extension is active

In this scenario, change log tables and triggers installed in SAP ECC need to be recreated in S/4 HANA to accommodate table schema changes. Recreating them will resolve this issue. In general, we advise that you perform the “Real Time Extension” setup from scratch by following the steps in Step 2: Set up and Configure the Real-Time Extractor.

Warning

In S/4 many tables have been replaced with compatibility views, i.e. MSEG. Since they no longer physically exist in the database, you won’t be able to install triggers on them. To extract these tables in real time you need to switch to the replacing table, i.e. MATDOC instead of MSEG.

Extraction and Transformation configuration in the Celonis Platform

S/4 HANA comes with compatibility views so that every deprecated SAP ECC table has a corresponding view in S/4. As a result, old integrations won’t break and can also be run against S/4. This also applies to the Celonis Platform, meaning that you can keep running the same Extraction=>Transformation against S/4 and expect the same dataset.

However, if the real-time extension is active and you have to replace tables that don’t exist anymore, then you have to modify the Extractions as well. In the MSEG example, you need to replace it with MATDOC. You can also rename it to MSEG during the extraction in order to keep the transformations intact downstream.

When starting an extraction in the Celonis Platform, every table being extracted maps to one background job in the SAP system.

The number of parallel extractions (and therefore background jobs) can be modified in the connection details of the data connection.

12321394.png

To see the jobs created by the Celonis Platform in SAP, you can go to run transaction SM37. To find the correct jobs, enter the prefix of the created job names (CEL_EX*) into the job name field and the name of the Celonis RFC user in the user field.

12321395.png

If one job on this day already ran, you may see a listing like on the left.

It shows the successful extraction of the table T001 from the Celonis Platform.

12321396.png

In this case the BSEG was running from one Data Job, the DD0* Tables from another job.

12321397.png

When cancelling one of the data jobs, the overview may look like the following.

The job extracting the DD0* Tables was cancelled from the Celonis Platform, the job extracting the BSEG is still running.

Cancelling a Job from the SAP System
12321399.png
12321398.png

If we want to cancel a job from the sap system, we can simply do so from the transaction SM37.

12321400.png

In the Celonis Platform, we can look at the job log and see something like the picture on the left. So the cancelling in SAP is picked up by the Celonis Platform and the Data Job is set to failed.

To remove the Celonis Extractor from SAP system perform the following steps:

  • (Optional) Stop all replications and/or the extraction data jobs from the Celonis cloud.

  • (Optional) If the Real-Time Extension is active:

    • Delete the Triggers via the transaction code /CELONIS/CLMAN_UI in DEV, QA and PROD;

    • Delete the Change Log tables in DEV and then move the transport forward to the QA and PRD systems;

    • Remove the CEL_CL_CLEAN_UP_SCHEDULED clean up job, if it was set up, in all systems.

  • Download the latest “Uninstall transport” for the RFC Module from the Downloads portal and import it into SAP.

    • This transport removes all objects shipped in the RFC Module.

    • After running it, verify that nothing exists in the /CELONIS/namespace.

  • Delete the Celonis user and role from SAP.

We are generating and storing detailed extraction logs on the Extractor server locally. By default they are stored in the root folder where the file connector-sap.jar is located. A new log file is generated on a daily basis, and with the time the number of logs can grow up significantly. To prevent this, the user can set up a retention period in the file application-local.yml, and the older logs will be cleaned up automatically.

  1. In the file application-local.yml, add the following line in the "logging" section, as shown below. Make sure that the spacing is correct.

    Logging.png
  2. Restart the Extractor service