1 of 42

3.14 Update 1

Overview

This online documentation describes how to use the migration-center software in order to migrate content from a source system into a target system (or into the same system in case of an in-place migration).

Please see the Release Notes for a summary of new and changed features as well as known issues in the migration-center releases.

Also, please make sure that you have read System Requirements before you install and use migration-center in order to achieve the best performance and results for your migration projects.

The supported source and target systems and their versions can be found in our knowledge base article List of supported versions.

The migration-center is a modular system that uses so called adapters in order to connect to the various source and target systems. The adapters to connect to a source system are called scanners and the adapters to connect to a target system are called importers. The capabilities and the configuration parameters of each adapter are described in the corresponding manual.

You can find the explanations of the terms and concepts used in this manual in the Glossary.

And last but not least, Legal Notice contains some important legal information.

Customized versions of adapters or entirely new ones (for other types of source and target systems) can be created and added to migration-center thanks to the open, API-based and documented architecture of the product. If such customized adapters are deployed in your migration project, the documentation provided with them should be consulted instead, since the features, behavior and parameters are likely to be different from the adapters described in these guides.

Further Resources

For additional tips and tricks, latest blog posts, FAQs, and helpful how-to articles, please visit our user forum.

In case you have technical questions or suggestions for improving migration-center, please send an email to our product support at support@migration-center.com.

Release Notes

migration-center 3.14 Update 1

Features & Enhancements

Veeva Importer
- New feature: Import documents using existing content from FTP server (#55303)
- New feature: Add support for importing relations (#54991)
SharePoint Online Importer (Batch)
- New feature: Allow assignment of any valid SP user (#55219)
- Support for OneDrive (#54262)
Filesystem Scanner
- New feature: Perform transformation of source object XML files before processing (#54776)

Fixes

SharePoint Online Importer (Batch)
- Update objects not marked as processed/imported (#55058)
Veeva Importer
- Import version tree with renditions fails (#55396)
SharePoint Importer
- Folder and document get created although a column cannot be set (#55001)
- SPO importer fails setting lookup column with a NULL value (#55140)
- CSOM Processor fails with 401 Unauthorized error if a job ran longer than 24 hours (#55323)
Opentext Scanner
- Keywords system rule is limited to 255 characters for Physical Objects migset (#55341)

Known Issues & Limitations

General / logging
- The installer does not update the location of the Job Server's log files. So if you do not want to use the default location for log files, which is <Job Server Home>/logs, you need to manually update the log file location in the <Job Server Home>/lib/mc-core/logback.xml configuration file (#54732)

System Requirements

A content migration with migration-center always involves a source system, a target system, and of course the migration-center itself.

This section provides information about the requirements for using migration-center in a content migration project.

For best results and performance it is recommended that every component will be installed on a different system. Hence, the system requirements for each component are listed separately.

The required capacities depend on the number of documents to be processed. The values given below are minimal requirements for a system processing about 1.000.000 documents.

Database Server

For a typical Oracle Database creation please refer to Installation Guide. For more advanced topics regarding Oracle Database configuration and administration please refer to Database administrator’s guide.

Server Components (Job Server)

Note that not all adapters are available on the Linux version of the jobserver.

Client

Installation Guide

Introduction

This guide describes the installation process of migration-center. The following main steps of the installation process are subsequently described:

Preparing the Oracle database for migration-center
Installing the Client components of migration-center
Installing the database of migration-center
Installing the Server Components of migration-center

Install

The following section gives an overview of the system requirements, and paragraph 2.2 describes the preparation of an Oracle Standard Edition database for migration-center.

System requirements

For best results and performance, it is recommended that every component will be installed on a different system. Hence, the system requirements for each module are listed separately.

The required capacities depend on the number of documents to be processed. The values given below are minimal requirements for a system processing about 1.000.000 documents. Please see the Sizing Guide for more details.

Database Server

For a typical Oracle database creation please refer the section Preparation of the Oracle database. For more advanced topics regarding Oracle database configuration and administration please refer to Database administrator’s guide.

Server Components (Job Server)

Note that not all adapters are available on the Linux version of the Job Server.

Client

For the communication between the individual components, it must be possible to have access to the database from the Job Server and the Client. In addition, the Client must be able to access the distributed Job Server. The figure below outlines the communication of the individual components.

Preparation of the Oracle database

Migration-center uses an Oracle database to store information about documents, migration sets, users, jobs, settings and others.

The steps in the creation of a new database for migration-center are described next. Even though the description is very detailed, the installing administrator is expected to have a good degree of experience in configuring databases.

It is also possible to use Oracle Express Edition, but this version is not recommended for performance reasons and scalability due to its built-in limitations.

Starting the database configuration assistant

Oracle 11g offers the Database Configuration Assistant for database creation. In Microsoft Windows, the assistant is in the Oracle program group in the Start menu. For all other operating systems, please refer to the corresponding Oracle documentation to start the assistant.

The assistant will provide the user with a step-by-step guide through all the stages necessary for creating and configuring a database. With the [Back] and [Next] buttons, the user can navigate between the individual steps of the database configuration process, to change options.

The screenshots in this chapter were created on a Microsoft Windows System with Oracle 11.2.0.1.0. For other operating systems, the menus and dialogs may be different

After the start, the assistant will display a welcome note which can be skipped with [Next].

Step 1: Select operation

The first step determines the operation to be performed with the Configuration Assistant. “Create a database” is the appropriate choice here.

Step 2: Database template

The next menu shows all available database templates. „General Purpose“ is appropriate for creating a database instance for migration-center.

Step 3: Database identification

Define the database name here. The name must be unique, meaning that there should not be another database with this name on the network. The SID is the name of a database instance and can differ – if desired - from the global database name. In order to make an easy identification of the database possible, a descriptive name should be selected, for instance „migration“.

Step 4: Management options

To manage the database, Oracle Enterprise Manager is recommended. In Oracle 11g, databases can be managed centrally using the Grid Control or locally using the Database Control. Since the choice of database management has no influence on the operability of migration-center, this can be configured according to personal preferences or company policies. For the purposes of this guide the local Database Control will not be selected here. E-mail notification and backup can be configured separately at a later time, if necessary.

Step 5: Passwords for system accounts

Once the database has been created, the passwords for the Oracle system accounts “SYS”, “SYSTEM”, “DBSNMP” and “SYSMAN” must be entered and confirmed. These accounts with their passwords are needed for the installation of the migration-center database schema.

Using the [Exit] button, the user can exit the database configuration and close the assistant.

These accounts/passwords grant full rights for the new Oracle database and should therefore be chosen and safeguarded carefully. The allocation of different passwords is recommended.

Step 6: Storage options

This step defines the storage method for the database. For use with migration-center the “File System” option is sufficient. For more information on the “ASM” method, please refer to the Oracle documentation.

Step 7: Storage location

In this step the paths in which the initialization parameters and the trace files are to be stored are determined. The preset values from the template can be retained.

Step 8: Recovery configuration

In this screen the initialization parameters for the database’s archive and recovery mode are specified. The default settings can be applied.

Step 9: Database content

In this step pre-defined example schemata can be added to the database. For migration-center no example schemata are required.

Step 10: Initialization parameters

In this step of the assistant there are several tabs on which the initialization parameters for the new database must be specified.

The memory values of the database need to be specified here. For migration-center, user-defined settings are required.

Please review the requirements in chapter System Requirements with regard to the recommended amount of memory allocated to the database instance

When processing mass data, it may be necessary to change the parameters after some time. This can be done retrospectively in the Oracle Enterprise Manager. To do so, select <Administration / Memory Parameters> in the menu. In the dialog box displayed the settings for the "SGA” and “PGA” values can be changed and saved.

At this point the block sizes and maximum number of user processes are defined. The default block size can be applied. The maximum number of processes should be set to 150.

The Unicode character set “AL32UTF8” should be selected here. This character set supports all languages specified in Unicode 3.1.

The figure below illustrates the selection.

Here the mode in which client applications access the database is specified. For migration-center, the “Dedicated Server Mode” connection type is appropriate and should be selected.

Step 11: Database storage

No modifications for the database storage are necessary for migration-center.

Step 12: Options

Optionally, at this point a script can be generated which contains all the settings for the creation of this database. With this script, the setup can be repeated later if necessary. Furthermore, this script can be used as documentation for the settings used.

At this point, the user has the last possibility to perform changes on previous pages by using the [Back] button. After confirming this step by selecting [Finish], the database will be created.

After the user selects [Finish], the assistant opens a window showing a summary of the previous settings. This summary can be stored for documentation purposes in an HTML file. By pressing the [OK] button, the database will be created and installed. This process can take some time.

The Oracle database Configuration Assistant generates the following tablespaces (we assume that unused tablespace is marked to be deleted. Refer to Step 4: Database- features):

INDX
SYSTEM
TEMP
TOOLS
UNDOTBS
USERS

Summary of the most important Oracle 11g installation parameters

Oracle components:

Oracle Database
Oracle XML DB
Oracle Net Listener
Oracle Call Interface

General options:

Oracle JVM
Enterprise Manager Repository (optional)

Initialization parameters:

Dependent on server

Character sets:

Database character set (AL32UTF8)

Data files:

Use standard parameters

Control files:

Use standard parameters

Redo log groups:

Use standard parameters

Installing migration-center

An installation assistant will help the user install migration-center. This assistant directs the user through all the steps necessary for the installation of migration-center by installing and configuring the components in the following order:

migration-center Client
migration-center database
migration-center Server Components

The three individual components of migration-center can also be distributed and installed with separate assistants. The following table shows the setup assistants on the installation medium:

This installation guide describes the installation by means of the main installation assistant, which initiates the setup routines of each individual component.

The installation is started with setupMigCenter.exe, which is to be found in the root directory of the installation medium/package.

The installation can be aborted before the component selection step by pressing the [cancel] button.

By pressing [Back], the user can navigate to the previous pages in order to change or edit options.

The migration-center installation package should not be placed in a folder/path containing special characters like “( ) ;” as this can interfere with the installation process.

Selecting components

At the following step, the user will choose the components of migration-center to install.

The following are different installation variants:

Full installation All components of migration-center are installed
Compact Installation This variant installs the migration-center Client and database only.
Custom installation The user-defined installation allows the user to select specific components.

The full installation is recommended as a first installation option, or a custom installation can be performed by selecting only components that need to be installed.

Although the components can be installed individually, for full functionality at least one instance of each component must be deployed and the components must be able to connect to one another.

By confirming the choice by selecting [Next], all chosen components will be installed one after another. For each of the components there will be an individual installation assistant.

migration-center Client

Installation of the migration-center Client starts with the welcome screen. The user will get to the next page by clicking [Next], while [Back] brings him back to the previous page, in case the entries there have to be corrected.

The installation can be cancelled by pressing [Cancel]. Only the installation of the current migration-center component will be interrupted. The setup assistant will continue installing the next component.

Select destination location

To install the Client, select the installation path for the application. The default installation path proposed by the installer is recommended but can be changed if needed.

Select start menu folder options

During the installation shortcuts in the Windows Start Menu will be created for migration-center. The name of the folder for these shortcuts can be set here.

Select shortcut placing options

During the installation shortcuts to the Desktop or to the Quick Launch Bar can be created.

Installation summary

Before starting the installation process, all previously set options are displayed. In order to change settings, the user can navigate to the previous pages by clicking [Back]. By clicking [Install], the installation is started.

Officially, Oracle Client 11gR2 32 bit is not supported on Windows 2012. By ignoring the warning during installation, it can be successfully installed and used properly on Windows 2012.

Database installation

This installation prepares the database for use with migration-center by creating a user, tables, installing the packages containing migration-center functionality and setting default configuration options. All of these objects will be grouped in a schema called FMEMC. Currently it is not possible to change the schema’s name or install multiple schemata on the same database instance.

The migration-center database installation program requires Oracle 11g/12c/18c/19c Client or Instant Client to connect to the Oracle server remotely or can be run locally on the Oracle server if the Oracle server machine is Windows-based.

Database connection

Next, the name of the Oracle database instance for migration-center, as well as the user account for accessing it must be entered. The specified database user must the predefined SYS user or an Oracle administrative user with enough privileges.

By selecting [Test Connection], the credential and connection to the selected database instance can be verified prior to starting the installation. A message box will confirm whether the connection could be established or return the error message from the server if not.

If selecting [Next], the connection test is performed and if the test is successful, the next window appears.

Enter your migration-center license key by copying and pasting it from the email message received from fme product support containing your licensing information.

Selecting the tablespaces path

At this point, the path for the tablespaces of the database is set. The default path provided by the selected database instance is recommended.

Also, the location for the database installation log file MigrationCenter.log can be set is set.

The migration-center database schema will be installed by clicking [Install].

The progress bar in the lower area of the screen informs the user about the progress of the database installation steps.

The log file records several database actions run by the setup routine. Therefore, this log data file is very useful for support in case problems occur during the installation, such as error messages due to insufficient privileges. The location for the log file should therefore be chosen carefully.

Selecting Listener port and date/time pattern

Listener port – is the port where the Oracle Listener is listening for incoming database connections. This is necessary for the scanner and importer to connect to the database instance. Configure accordingly to local Oracle configuration. By default, this is port 1521, but another port number can be entered here if the Oracle Listener is configured differently.

Date-time pattern – is the pattern used to display time and date values across the various migration-center components, such as the Client and adapters. Also, the scanner and importer will use this pattern for exporting/importing date-time attributes. Since migration-center is a distributed application and the various components can be installed on different machines with different regional settings, the definition of a common date/time format makes sense.

The dropdown menu includes of 4 the mostly widely used date/time formats.

If the migration-center database schema was successfully installed, the [Finish] button appears, clicking the [Finish] button closes the installer.

Database objects created during migration-center database schema installation:

During the installation, the following Oracle Users will be created for migration-center.

Additionally the user FMEMC will be granted to use the following Oracle packages:

SYS.UTL_TCP TO FMEMC;
SYS.UTL_SMTP TO FMEMC;
SYS.DBMS_LOCK TO FMEMC;
SYS.UTL_RAW TO FMEMC;
SYS.UTL_ENCODE TO FMEMC;
SYS.DBMS_SCHEDULER TO FMEMC;
SYS.DBMS_OBFUSCATION_TOOLKIT TO FMEMC;

Furthermore, during the installation, the following tablespaces will be created for migration-center:

FMEMC_DATA (40MB)
FMEMC_INDEX (20MB)

These tablespaces are set to expand in steps of 10MBs according to the usage and needs of migration-center.

The user role FMEMC_USER_ROLE will be created and granted the required privileges on FMEMC schema objects. This role must be granted to any database user that needs to use migration center.

Database advanced installation

If default settings for the user FMEMC and tablespaces do not meet the requirements of the customer or conflict with internal company policies and guidelines, the migration-center database installation can be customized. This way, one may create the user FMEMC and the required necessary tablespaces manually before running the provided database installer by using the scripts located in <MIGRATION-CENTER package>\Database\Util.

The tablespaces can be separated or merged into any number of individual datafiles, but the tablespace’s names cannot be changed (FMEMC_DATA and FMEMC_INDEX must not be changed).

These tablespaces must be created prior to the creation of the FMEMC user.

After creating the tablespaces and the FMEMC user, the migration-center database installer can be run connected as user FMEMC instead of SYS.

Updating from an older migration-center database version

Before starting the update process it is always advised to back up the existing data.

The migration-center database installer supports updating migration center databases starting with version 3.0. The process will be performed automatically by the installer and does not require any user intervention.

An update can take in excess of 1 hour to perform, depending on the amount of data and database system performance. This is normal and does not mean the installer has stopped responding. Please do not try to close the installer application or otherwise intervene in the process.

Prerequisites for upgrading from version 3.0.x to 3.13

Before starting the update process it is always advised to back up the existing data.

The following conditions need to be fulfilled for the update procedure to work (these are checked by the installer):

The old database’s version must be one of the following: 3.0.x, 3.0.1.985, 3.1.0.1517, 3.1.0.1549, 3.1.1.1732, 3.1.2.1894, 3.2.0.2378, 3.2.1.2512, 3.2.2.2584, 3.2.2.2688, 3.2.3.2808, 3.2.4.3124, 3.2.4.3187, 3.2.4.3214, 3.2.4.3355, 3.2.5.3609, 3.2.5.3768, 3.2.6.3899, 3.2.6.4131, 3.2.7.4348, 3.2.7.7701, 3.2.7.7831, 3.2.7.7919, 3.2.8.7977, 3.2.8.8184, 3.2.8.8235, 3.2.8.8315, 3.2.9.8452, 3.3.8573, 3.4.8700, 3.5.8952, 3.5.8952, 3.6.8970, 3.7.1219, 3.8.0125, 3.9.0513, 3.9.0606, 3.9.0614, 3.9.0606, 3.9.0704, 3.10.0823, 3.10.0905, 3.11.1002. You can check your current database version in the Client’s “About” window. If your version is not one of the above please contact our technical support at support@migration-center.com.
Any running jobs (scanners, importers, schedulers, transformations, validations) must be finished or stopped.

Considerations regarding updating database version older than 3.1.0.x

Due to the new update feature released with migration-center 3.1.0, a new instance of an existing scanner may detect updates for objects scanned with previous versions even though the objects haven’t changed from the previous scan. This behavior always applies to virtual documents or documents that have dm_relations and occurs due to the information used by the new features in migration-center 3.1 not being available in the previous release. For this reason, a new scan will recreate this information on its first run.
Transformation rules created with a version older than 3.1.0 which use the system attribute r_folder_path might need to be reconfigured. This is because migration-center 3.1 now stores absolute folder paths instead of paths relative to “scanFolderPaths” as was the case with the previous versions.
Database update from previous versions older than 3.1.2: as a result of the update process increasing the default size for attribute fields to 4000 bytes (up from the previous 2000 bytes), Oracle’s internal data alignment structures may fragment. This can result in a performance drop of up to 20% when working with updated data (apparent when transforming / validating / resetting). A clean installation is not affected, neither is new data which is added to an updated database, because in these cases the new data will be properly aligned to the new 4000 byte sized fields as it is added to the database.
If the database contains Virtual Documents and objects that are part of scanned Dctm Relations (this does not apply to FileSystem adapter) some additional checks are done by installer. In case any of these checks fail an error is raised by the installer. In this case stop the installation and contact our technical support at support@migration-center.com.

Considerations regarding updating database version 3.1.2

Due to changes and new functionalities implemented in migration-center version 3.2 that rely on additional information not present in older releases, the following points should be considered when updating a database to version 3.2 or later:

The concatenate function will only have 5 parameters for newly added transformation rules. Reason: the “Concatenate” transformation function has been extended to 5 parameters for version 3.2. Transformation rules using the “Concatenate” function and created with a previous version of migration-center will retain their original 3 parameters only.
The system rule “mc_content_location”, which allows the user to define or override the location where the objects’ content files are stored will be available for use only in migration sets created after the database has been updated to version 3.2 Reason: the “mc_content_location” system rule is new to migration-center version 3.2 and did not exist in previous versions
The Filesystem scanner won’t create the “dctm_obj_link” attribute anymore Reason: with version 3.2 of migration-center scanners and importers are no longer paired together. The “dctm_obj_link” attribute was created automatically by the Filesystem scanner in previous iterations because it was a Filesystem-Documentum adapter. Since this no longer applies to the current Filesystem scanner which is just a generic scanner for filesystem objects and has no connection to any specific target system, it won’t create any attributes specific to particular target systems either. If objects scanned with a Filesystem scanner are intended to be migrated to a Documentum system, the “dctm_obj_link” rule must be created and populated with transformation rules in the same way as any other user-defined rule.
The Filesystem scanner won’t detect changes in a file’s metadata during subsequent scans for files which have been scanned prior to updating to version 3.2 Reason: detecting changes in a file’s external metadata (in addition to the file’s content) is a new feature of the Filesystem scanner in migration-center 3.2; previous versions of migration-center did not store the information which would be required by the current Filesystem scanner to detect such changes. A change to a file’s modification date would trigger the update mechanism though and would also enable full 3.2 functionality on subsequent scans.

Updating a previous database version to current version

A previous version of the migration-center database can be updated to the current version. The installer will detect supported database versions eligible for update and offer to update the database structure, migration data and stored procedures to the current version.

Running the database installer in update mode

The first step is connecting to the database instance as user SYS. If an old database version is detected on that database instance, the following screen appears:

Before clicking [Next] make sure you backed up your old database first!

Enter a location for saving the database installation/update log file. In case of any errors this log file will be requested by our technical support staff.

After clicking “Next” the appropriate database scripts will be executed. Some of them might take several minutes to execute; this depends on the amount of existing data in the database being upgraded.

Backup a migration-center database

To back up a database used by migration-center it is sufficient to back up only the data within the FMEMC schema. The easiest way to do this is with Oracle’s “exp”. See screen shot below for the basic steps required to back up a schema. For more information consult the documentation provided by Oracle.

Starting with Oracle 11g release 2, the empty table might not be exported at all. This happens when the database parameter DEFERRED_SEGMENT_CREATION is set to TRUE. Initially this parameter is set to TRUE. To force exporting all tables from FMEMC schema the following commands should be run connected as user FMEMC:

ALTER TABLE SCHEDULER_RUNS ALLOCATE EXTENT;

ALTER TABLE SCHEDULERS ALLOCATE EXTENT;

Restoring a migration-center database

To restore the backup, follow the steps below:

If the database instance where the backup should be restored does not contain an FMEMC schema and FMEMC user, please create them first.
Use Oracle’s “imp” utility for importing the dump file previously created by the “exp” utility. See screen shot below for the basic steps required to restore a database schema from a dump file. For more information consult the documentation provided by Oracle.

Note: The same character sets in the Oracle Client should be used when exporting and importing the data.

Installation of the Server components on Windows

The migration-center Server Components include the adapters which connect to the various source and target systems, as well as the Job Server service which runs the jobs employing those adapters.

Before starting the installation note that JAVA_HOME or JRE_HOME need to be set appropriately.

After starting the installation, a welcome screen appears. Click [Next] to move on to the next step.

The installation of the Job Server can be cancelled anytime by clicking [cancel].

Select destination location

Next, the user will select the installation location for the Server Components. The default installation path proposed by the installer is recommended but can be changed if needed.

Server components folder in the start menu

During the installation, shortcuts are created for migration-center Server Components in the Windows start menu. The location of the shortcuts can be configured here. Furthermore, an entry is created in the Windows Start Menu Startup folder, in order to allow the automatic start of the Job Server.

Select Job Server port

At this point the TCP listening port of the Job Server is configured. The port specified here is used by the migration-center client to control the Job Server. The default port is 9700 and can be changed if needed.

Different Job Servers on different machines can all use the default port number, the same port number but different from the default port, or entirely different port numbers each.

The port specified for each Job Server during installation must be specified when adding a new Job Server location in the client pointing towards that respective Job Server.

Select the folder where the log files will be saved

Default value is the logs folder in the Server Components installation folder, but it can be set to any valid local path or UNC network share. Note the location set here must allow write access for the account used to run the migration-center Job Server service, which is the component writing the log files.

Since version 3.14 the log location is only set for the adapter logs. The server logs location is manually set in the mc-core/logback.xml file, using the FOLDER_LOG property.

Completion of the installation

The Job Server is installed on the system by selecting [Install].

Multiple installations of the Server Components

Depending on the requirements or possibilities of each migration project, it can make sense to install multiple Job Servers across the environment, either to share the workload, or to exploit performance advantages due to the geographical location of the various source and/or target system, i.e. installing the Job Server on a node as close as possible to the source or target system it is supposed to communicate with.

In terms of throughput, local installations will provide benefits over remote installations and local networks will always be faster than remote networks.

Besides throughput, latency needs to be considered as well, since increased latency can affect performance just as much, even leading to timeouts or connection breakdowns in severe cases.

Ending the installation process

If all selected components are installed, the installation will be ended by selecting [Finish].

Uninstalling previous versions of the mc Server Components

A clean installation of the mc Server Components may require uninstalling the previously installed version first and would be recommended if upgrading while skipping over several intermediate versions.

To uninstall the mc Server Components follow the steps below:

Stop the mc Job ServerJob Service if it is currently running; this can be done
- using the Windows Services application
- using the “Stop service” shortcut in Start Menu/All Programs/fme AG/migration-center Components <currently installed version>
Uninstall the mc Server Components; this can be done
- Using the Programs and Features applet in Windows Control Panel
- using the “Uninstall” shortcut in Start Menu/All Programs/fme AG/migration-center Components <currently installed version>

Uninstalling will not delete the log files having been generated in the mc Server Components installation folder, as the user may want to keep log files for future reference. Thus, it is up the user whether to delete or archive leftover log files after the uninstaller has completed removing the application’s files.

Installation of the Server Components on Linux

Not all adapters are available on the Linux version of Server Components. The ones available are:

Scanners: Documentum, Filesystem, Database, eRoom, Exchange
Importers: Documentum, Filesystem, InfoArchive

The Linux version of the migration-center Server Components can be found in the folder ServerComponents_Linux.

In order to install the Migration Center Job Server extract the archive Jobserver.tar.gz in the desired location using the command:

tar -zxvf Jobserver.tar.gz

All necessary Job Server files will be extracted in the “Jobserver” folder.

To install the Job Server as a service / daemon, follow these steps:

1. Switch to the “bin” folder of the “Jobserver” folder

2. Run the command sudo ./installDaemon.sh

Run the installDaemon.sh as a user that has administrative permissions (sudo).

Instead of installing the Job Server as a service/daemon, you can run it in the terminal by executing the script ./runConsole.sh in the bin folder

The default TCP listening port is 9701 and can be changed in “server-config.properties” file located in “lib/mc-core” folder.

For running Documentum Scanner or Importer on Linux the DFC (Documentum Foundation Classes) needs to be configured in “conf/dfc.conf” as it is described in the Scanner and Importer user guides.

Starting migration-center Client

During installation of migration-center Client, a desktop icon (optional) and a Start Menu entry were created. By default this can be found in Start-> (All) Programs-> fme AG-> migration-center Server Components <Version>. The migration-center Client can be started by using the shortcuts from these locations.

Starting the Job Server on Windows

The Job Server is set up as a Windows service during the installation and can be managed using the regular Services Microsoft Management Console snap-in.

By selecting the “Migration Center Job Server” entry, the service can be manually started, stopped or restarted if needed. If the service is required to run using a specific user account, this can be configured on the Log On tab in the services’ properties.

In addition, the Job Server has shortcuts in the Start menu by which the service can be stopped or started directly without having to access the Services console. By default these can be found in Start-> (All) Programs-> fme AG-> migration-center Server Components <Version>.

Starting and running the Job Server on Linux

The Job Server is setup as a service (daemon) and can be started or stopped by running the following scripts inside the “Jobserver/bin” folder

./startDaemon.sh for starting the daemon
./stopDaemon.sh for stopping the daemon

Instead of installing the Job Server as a service/daemon, you can run it in the terminal by executing the script ./runConsole.sh in the bin folder.

Uninstall

The following paragraphs describes how to uninstall the individual components of migration-center.

Uninstalling the migration-center Client

The migration-center Client can be uninstalled by using „Add or Remove Programs“ or “Programs and Features” item in the Control Panel (depending on the version of Windows used). You can find the Client listed under the name „migration-center Client <Version>“, it can be uninstalled by selecting the entry and clicking [Remove].

Uninstall links are also provided in the Start Menu program group created during installation. By default, this is Start-> (All) Programs-> fme AG-> migration-center Client <Version>.

Uninstalling the migration-center database schema

Uninstalling the migration-center database schema will delete all data within that schema. It is no longer possible to recover the information after that. Please back up the database schema before uninstalling it.

An uninstall script is provided with migration-center; it can be found in database/util/drop_fmemc_schema.sql. Run this script against the Oracle database instance the FMEMC schema should be removed from using the Oracle administration tool of your choice.

Make sure no connections with the user FMEMC exist, otherwise the script will fail to execute properly. Should this happen, the script can be re-run after closing all connections of the user fmemc.

Uninstalling the migration-center Server Components on Windows

The migration-center Server Components can be uninstalled by using “Add or Remove Programs” or “Program and Features” item in the Control Panel (depending on the version of Windows used). You can find the Server Components listed under the name “migration-center Server Components <Version>” and can be uninstalled by selecting the entry and clicking [Remove].

Uninstall links are also provided in the Start Menu program group created during installation. By default, this is Start-> (All) Programs-> fme AG-> migration-center Server Components <Version>.

Uninstalling the migration-center Server Components on Linux

To uninstall the Job Server as a service/daemon, follow these steps:

Go to the “bin” folder inside “Jobserver” folder
Run the command ./uninstallDaemon.sh

Run uninstallDaemon.sh as a user that has administrative permissions (sudo).

Client User Guide

Introduction

The focus with migration-center 3.x is on providing a lightweight, yet highly functional, usable and extensible migration solution which can grow to match the requirements of present and future migration projects through customization and connectivity to other document and content management systems.

The current release is primarily a release that consolidates the features and adapters developed in the versions 3.2.x, which itself is a significant update to the initial version 3.0 with major new features, usability, performance, and reliability improvements. Besides the incremental improvements and fixes, migration-center also brings along an entire suite of new adapters available directly from fme; currently the full list of adapters for migration-center available from fme is:

Documentum Scanner
eRoom Scanner
Filesystem Scanner
Database Scanner
Outlook Scanner
SharePoint Scanner
Documentum NCC Scanner
Alfresco Scanner
Domino Scanner
Exchange Scanner
OpenText Scanner
Documentum Importer
DCTM Audit trail Importer
Alfresco Importer
Filesystem Importer
FirstDoc Importer
D2 Importer
DCM Importer
box Importer
Alfresco Importer
SharePoint Legacy Importer
SharePoint Online Importer
SharePoint 2013 Importer
Documentum NCC Importer
InfoArchive Importer
OpenText Importer

migration-center client 3.13 cannot be used to connect to a migration-center database 3.4 as well as migration-center client 3.4 cannot be used to connect to migration-center database 3.13.

Other adapters can of course be developed according to customer or project requirements.

Using migration-center will tell why migration-center is “first in migration technology”.

Product features

As stated in the introduction, migration-center 3.x is a complete rebuild. The general concept and useful features from the previous version have been retained functionality-wise and improved in the current version. migration-center’s strengths remain:

The transformation of metadata through intelligent, rule-based mapping of objects
Incremental delta migrations for newly created and changed documents and folders
Identification of duplicates
one-time or regular batch migration of large volumes of documents and folders
Complete audit ability, error reporting and error handling
1:1 migration of folder structures and mapping of objects within, including metadata
1:1 migration of relations (i.e. Virtual Documents, etc.)
The possibility to export and import type templates, transformation rules, object grids (export only) to CSV files.

Features that were never/rarely used have been removed, resulting in a cleaner, more user-friendly and productivity-oriented GUI.

Compatibility

Migration-center 3.x is not compatible with migration-center 2.x. Migrations initially started with migration-center 2.x should be completed before upgrading to the new version; a version upgrade halfway through a project is not possible.

Architecture Diagram

From a technical point of view, migration-center is a distributed, client-server type application using a database management system as a backend. All these components interact to allow for a highly configurable, automated mass import of very large amounts of data from a source document/content management system into another content management system.

The solution centers on an Oracle database used for storing object related information like metadata, migration rules, and all configuration and status information necessary for managing and tracking the steps involved in the migration process.

All of the aforementioned information is defined and controlled via the migration-center Client, a GUI application which acts as the command and control center for migration-center.

Finally, there are the adapters which connect migration-center to various source and target systems via migration-center’s own API on one end and the respective system’s API(s) on the other end. These adapters which interact directly with the source and target systems are controlled by a service, which is also managed by the migration-center Client application (configuring and launching the appropriate jobs). There can be any number of adapters installed and managed with migration-center.

Multiple instances of the above components can be deployed across the network, even across geographical boundaries, and can work together on the same or multiple projects.

The actual content moved during a migration (files, documents, emails) is stored on the file system (local or remote) and does not require a specific migration-center component for managing.

Depending on the combination of adapters deployed in a particular migration project, the source and target systems supported can be same/different versions of the same solution or entirely heterogeneous systems sharing nothing but basic concepts of content management, like a content object with associated metadata.

Not all adapters illustrated in the above diagram are part of the standard product package.

Client

Migration-center Client is a GUI application and represents the interface between the user and the migration-center core features and functionalities. All migration-center functionalities like creating, monitoring and managing jobs, tasks, and migration sets are accessible through the client.

The client does not perform any tasks by itself, it’s merely a management interface; therefore the Client application can be disconnected or shut down safely at any time, without affecting any tasks that might be running in migration-center at that time.

The client is a standard Windows application that can be installed anywhere, provided it has a network connection to migration-center’s central database, and the prerequisites installed (as described in the Installation Guide). Any number of Clients can be deployed across a network, and connect to the same, or different migration-center databases.

After starting the migration-center Client application, the database login window opens. The Client application is not operational by itself, since all information required to operate migration-center is stored in a migration-center database, which is why a connection to such a database is required at all times to use the Client. Type the username and the password, select the migration database from the list of available database connections and click [OK]. migration-center Client will remember the last database it connected to and will automatically reconnect to this database the next time it is started.

During installation a default user is created. If no other users have been created, use the default user account shown below for logging on to the migration-center database.

Username: fmemc

Password: migration123

Job Server

The Job Server is a migration-center component installed with the migration-center Server Components; it runs as a Windows service and does not interact with the user. Its purpose is to execute various tasks in the background, like scanning or importing documents. The tasks running on a given Job Server can be monitored and managed by the user through the Client. A migration-center Job Server can be installed anywhere as long as there is a network connection to the migration-center database; several Job Servers can be deployed across the network and managed using migration-center Clients. Since the Job Server is the process actually accessing the source and target systems, the user account this service runs with should have the required permissions for proper access to the respective source and target systems.

Database

The migration-center database contains most of the data processing logic and stores all the migration sets, including their objects, metadata and transformation rules.

As with the other components, at least one database needs to be installed for migration-center to work, but several databases can be deployed in the same environment, depending on the requirements.

Both the Client and the Job Server need to connect to a migration-center database to function.

Client Functionalities

This chapter details the features and functionalities available to the user through migration-center Client.

Main application window

migration-center’s main application window adheres to typical windows applications’ user interface, offering a main menu bar and toolbar for accessing general, frequently used features.

Major features and functionalities each open in their own child windows, providing toolbars and context menus that apply to the items displayed in that window.

Any combination of child windows can be opened at the same time; child windows can be tiled or cascaded.

Some windows (e.g. windows displaying properties for items, or message boxes informing the user or requesting input from the user) always display on top of other windows and stay there until they are closed by the user.

All the important main menu commands are represented in the toolbar of the main window, are self-explanatory and will be described in the next chapter - Main window toolbar. The menu commands which do not have a correspondent in the toolbar menu will be described here.

<migration-center> <Connect to…> Calls up a dialogue window where the user can connect to a (another) migration- center specific database.
<migration-center> <Reconnect> Reconnects client to currently used migration-center DB
<migration-center> <Renew License> Allows entering a new license key
<migration-center> <About> Displays information about licensing and product versions
<migration-center> <Exit> Exits migration-center
<Manage> <Scanners…> opens the -Scanners- window for creating and managing scanner jobs
<Manage> <Migration Sets…> open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
<Manage> <Importers…> opens the -Importers- window for creating and managing importer jobs
<Manage> <Jobs…> open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
<Manage> <Jobservers…> open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
<Manage> <Schedulers…> opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
<Manage> <Object Types…> opens the -Object Types- window, where the object type definitions can be imported and edited (these are required for associating transformation rules with for the attribute model transformation are defined)
<Manage> <Global Mapping Lists> opens the -Global Mapping Lists- window, where global mapping lists can be created or imported. A global mapping is automatically available for use in any migration set.
<Manage> <Object History…> opens the -History- window where it is possible to look up any object that underwent processing in migration-center. Key steps of the objects lifecycle during the migration can be viewed, such as the objects status, metadata and the exact timestamp at that moment.

The main toolbar provides quick access to frequently used functionalities. These correspond with the same functionalities available from the main menu.

[Scanners] opens the -Scanners- window for creating and managing scanner jobs
[Migration Sets] open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
[Importers] opens the -Importers- window for creating and managing importer jobs
[Jobs] open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
[Jobservers] open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
[Schedulers] opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
[Cascade] Arranges all open child windows in a cascade
[Tile Horizontally] Tiles all open child windows horizontally
[Tile Vertically] Tiles all open child windows vertically

Job Servers

To input/output any data from/to migration-center there needs to be at least one location with a running Job Server defined.

The -Jobservers- window has its own toolbar and right-click context menu containing items that allow the user to create, edit, delete, and monitor the availability of the Job Servers running on the specified Job Server locations.

The location for a Job Server can be specified as the machine’s hostname or fully qualified domain name, if required, or simply as an IP address.

Note: Do not use loopback addresses or the “localhost” name.

The toolbar and the context menu of the -Jobservers- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[Refresh] is useful for rechecking the status of the locations listed
[New] opens a blank -Jobserver- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a location from list

Properties window

To create a new Job Server location

select <New> from the context menu or from the toolbar
Enter the IP address or hostname of the machine the Job Server is running on and the port number it is listening on.
Enter the appropriate port on which the Job Server on that machine was configured

The new location is displayed in the -Jobservers- window. Check the Status column to make sure the required Job Server is listed as Available and thus capable of accepting incoming connections. The Job Server on that particular location is now ready to be used by any input/output adapter.

The default port number is 9700, unless changed during setup.

Creating a scanner and trying to save it without having any Job Servers defined will prompt the user to define one Job Server location and save the scanner with the new created Job Server location.

Scanners

Scanners are responsible for filtering and extracting metadata and content from the source system and inputting this information into the migration-center database. Different types of scanners can be available in a migration-center installation; different types of scanners also offer different configuration parameters, depending on the source system they connect with migration-center. All scanners provide the same basic functionalities, such as extracting documents and folders form the source system and extracting the metadata associated with these objects and storing it in the migration-center database, as well as adding system specific information allowing migration-center to understand and process these objects correctly.

The toolbar and the context menu of the -Scanners- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[History] opens the -History- window where more information about all scans performed by the selected scanner can be viewed. The log files from these scans can also be accessed here.
[View Scanned Objects] will display a grid with the scanned objects in a new window, where these can be viewed, filtered, and exported to a CSV (comma separated values) file. This functionality does not allow any editing to be performed on the objects.
[Run] initiates the run of a scanner with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the scanners listed
[New] opens a blank -Scanner- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected scanner, such as the number of processed objects, the start and ending time and the status of each run.

Double clicking a history entry or clicking the [Open] button on the toolbar with a selected history entry opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

The log files generated by a scanner are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components 3.13\logs

All scanners, regardless of type share some common parameters

Name: must be a unique name for the current scanner

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the scanner depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this scanner does or add some comments related to the scanner

In the History of a scan, or after the objects have been assigned to a Migration set the metadata of these objects can be viewed in the -Scanned Objects- window which can be opened with the [View Scanned Objects] button.

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

The [Export to CSV] button allows the user to export the entire grid in its current configuration (i.e. including any applied filters and visible columns) into a comma separated file. The CSV field separator, and the repeating attribute separator can be specified by the user before exporting the file.

Filtering attributes and (de)selecting columns is neither permanent, nor does it affect the actual data in any way. It only affects the current display and serves to analyze the data on display.

Migration Sets

Migration sets are sets of individual objects but with a common set of transformation and validation rules applied to them. Operations like transformation, validation and import are always performed on such a set of objects and cannot be performed on individual objects that are not assigned to any migration set. Assigning the objects resulting from a scan to a migration set is therefore the first step towards making these objects operable for migration-center’s features.

It is possible to add multiple scans to one migration set, as it is possible to split a scan containing a large number of objects onto several migration sets.

It is also possible to filter the selection of objects before adding it to a migration set based on user specified selection criteria and conditions; objects which do not meet these criteria will not be added to the current migration set and will continue to be available and can be filtered further, added to another migration set, or left unassigned if they are not supposed to be migrated.

Buttons and context menu items:

[Edit Transformation Rules] opens the window where transformation rules are managed and rules are applied
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Transform and Validate] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
[Reset All] resets all objects of in the selected migration set to their initial attribute values, essentially removing any processing performed during the transformation process and resetting status changes resulting from validation and import. The [Reset All] button is available in all views (-Source Objects-, -Processed Objects-, and -Error Objects-).

Imported objects will NOT be reset by default and require an extra checkbox to be checked prior to executing the reset command. This is for security reasons, because resetting imported objects will erase all information linking the source objects to the objects created by migration-center during import, therefore rendering migration-center unaware of having imported the affected objects at all.

Resetting imported objects would typically be used during testing phases of a project and not during production.

It is strongly advised NOT to reset imported objects in cases where dependencies to these objects may exist in other, even future migration sets, such as newer versions of these objects or relations between these and other object which haven’t been migrated yet; this also applies if it is intended to update these objects during future migrations.

[Stop] appears as a button only when the selected migration set is busy being transformed or validated, and only if these processes take longer than 3 seconds for the selected migration set.
[Refresh] refreshes data in all info-columns
[View All Objects] (press drop-down list/button combo) opens 3 windows: Source Documents, Processed Documents, and Error Documents.
[Source Objects] (select from drop-down list/button combo) displays all scanned objects attributed to the respective migration set. Original attributes and attribute values are displayed.
[Processed Objects] (select from drop-down list/button combo) displays the objects after they have been processed by applying the transformation rules.
[Error Objects] (select from drop-down list/button combo) displays objects that failed transformation/validation criteria, or failed to import, indicating the cause
[New] opens a blank -Migration Set- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured migration set from list

Items in columns can be sorted by clicking on the column headers. Clicking the same column header repeatedly switches between ascending/descending sorting order.

In the lower part of the window, migration sets can be filtered by name. This can prove useful if there are a large number of migration sets, such as the migration sets generated automatically by repeated scheduled migration runs.

Migration Set Properties window

The migration set’s properties window opens when creating a new migration set and can be accessed for an existing migration set by double-clicking it or selecting [Properties] button or menu item from the -Migration Sets- windows toolbar or context menu.

The -Migration Set Properties- window also determines the selection of objects assigned to the migration set. The |Filescan Selection | allows scans to be selected and added to the migration set, | Exclude objects by values | offers a simple method of removing excluding objects from the migration set based on selected metadata values, and the | Advanced Filters | page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators.

The resulting set of objects can be previewed by clicking the [Preview objects] button on the right side to open the -Preview- window. The -Preview- window offers an overview of the objects resulting from the selection, and some basic functionality for analyzing these such as viewing attributes for individual objects, selecting the attribute to display or exporting this list to a CSV file.

Common controls

The buttons on the right side of the –Migration Set Properties- window are available on all tabbed pages of this window.

The [OK], [Apply] and [Cancel] buttons are self-explanatory and work as expected.

Trying to save a migration set using the [OK] or [Apply] buttons without having assigned any objects to it will display a warning message to the user. Although usually it doesn’t make sense to save a migration-set without having objects assigned to it, this can be used to define migration sets which will be used as templates, and create new migration sets by copying these. In such a case ignore the warning and save the “empty” migration set.

Otherwise define a selection of objects as described in the following chapters before saving the migration set.

The [Preview objects] button and preview functionality is available on all pages of the –Migration Set Properties- window.

If the selection is ok, the selected scans can be assigned to the migration set using the [Select objects] button in the upper right corner.

Note: Assigning objects to a migration set triggers various status changes and updates to the object’s internal metadata, as well as evaluating any additional filter criteria the user may have specified. This can take some time, especially with large numbers of objects. Therefore, the user will be informed that the selection process is about to run in the background, and the Properties window will close. The selection progress can be monitored in the main -Migration Sets- window. While a selection process is running, the user can continue working with other migration sets or other parts of the applications. Selection is a one-time-only operation and does not need to be performed anytime afterwards.

For a migration set that already has a selection of objects assigned instead of the [Select objects] button there will be a [Deselect objects] button. This will free any objects assigned to the migration set but also remove any metadata generated through transformation rules and status information for objects. Recovery of the information lost if deselecting objects from a migration set is not possible. Use with care!

Deselecting objects form a migration sets also involves various updates to the object’s status and internal metadata, which can be a relatively time-consuming operation for large numbers of objects. The process will therefore be run in the background, similar to the procedure described above for selecting objects.

The |Properties| page

A migration set’s basic properties, including user specified properties like the migration set’s name, the type of object it contains, and an optional description can be set here. Other information managed by the system, such as creation or modification dates or the last operation performed on the migration set are also displayed.

The available object types depend on the adapters available with the respective migration center installation and may vary. Always make sure to select the correct object type since the selection here determines what is displayed on the following pages of the properties window.

The |Filescan selection| page

This page allows the user to select and assign any available scan to the current migration set.

The upper list shows all scans with available objects and some relevant information (i.e. scans that haven’t been fully assigned to other migration sets). Move selected scans to the bottom list to assign them to the migration set by clicking on the up/down arrow buttons or double-clicking a scan run. Multiple scans can be selected and assigned to the current migration set.

Only scans matching the object type set on the previous page (|Properties|) will be listed here. Check the object type selection if the desired scans are not displayed here.

The |Exclude objects by values| page

This page offers a simple way of excluding some objects from the selection based on their values for one or more attributes.

In previous versions of migration-center this page was called |Filters|. The functionality provided is the same in all product versions.

The list on the left displays all available attributes. Selecting an attribute displays a list of its distinct values in the Include list to the right. Moving any of these values to the Exclude list on the far right will exclude all objects with that value from the migration set. Move items between the Include/Exclude lists by double clicking or using the arrow buttons between the lists.

Attributes that have been used to exclude some of their values will turn bold to indicate their status. Remove any excluded values for such an attribute to revert it to its default (no exclusion) state.

The |Advanced Filters| page

This page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators. Use advanced filter expressions to specify exactly which objects should be assigned to the current migration set.

For objects matching a filter expression means these objects will be added to the migration set, not excluded!

The feature works by adding filter expressions and grouping them with AND/OR operators. The list of expressions will be evaluated from top to bottom to determine the remaining subset of objects which can be assigned to the migration set.

An expression always consists of a source attribute, a logical operator or condition, and a user specified value. Individual expressions can be created, added to the list below (click the [Add] button to the right), and then moved up or down in the list or deleted using the respective [Up], [Down] and [Delete] buttons. Any number of expressions can be added.

Conditional operators for working with string, numeric, and date/time type values are provided.

Migration Set Views

One, two, or all three of these three views can be opened at any time for the current migration set.

The filter bar at the bottom of the window is common to all views and offers following functionalities:

Other features common to all views are the following toolbar items:

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Click [Refresh] to reload the content of the current view

Source Objects View

The Source Objects view always displays the unmodified state of the objects in the migration set, i.e. the state of the objects as they were scanned from their respective source system. This view does not provide any editing possibilities for the objects displayed, it serves as a permanent reference for checking the original metadata of these objects.

The Source Objects view also allows identifying and removing duplicate objects. Pressing the [View Duplicate Objects] button will display only the duplicates; duplicates are identified by means of a content checksum and the objects’ creation date, e.g. from all the objects with an identical checksum the one with the oldest creation date will be considered the original, and all others will be considered duplicates. This can be used just to display the duplicates or select and remove them from the migration set using the <Remove documents from migration set> entry on the context menu.

The checksum is computed during scan and must be activated in the scanner. See the respective adapter documentation for more information about using the checksum option in the scanner.

Toolbar items

[View Attributes] displays all available (scanned) attributes of one object only. Previous and following object can also be browsed by using the appropriate buttons.

[Edit Transformation Rules] opens the -Transformation Rules- window.

Items in columns can be arranged (increasing/decreasing) by clicking on the column labels.

[View Duplicate Objects] filters identifies and displays duplicate objects

Context menu items

<View Source Attributes> will display all attributes of that object in a small new window and allows browsing through the list objects by object (back and forward.

<View Relations> will display the relations (if any) between the selected object and other objects. It is also possible to see the full details for each relation, including the objects’ IDs it related to, the status and type of that relation, etc.

<View Value> displays the selected cell’s value in a new window. This is particularly helpful when viewing details of a long value (like folder path for example), or a multi-value attribute with a large number of individual values. It is also possible to copy the value to the clipboard, or toggle wrapping the displayed value in the -View Value- window

Copying the currently selected value to the clipboard function can also be performed directly using the <Copy Value> context menu entry or using the {Ctrl+C} keyboard shortcut

<Filter by Value> will immediately filter the view using the selected attribute value. Use the [Clear Filter] button in the filter bar at the bottom to remove a filter and return to the full list.

Objects can be selected and removed from the migration set by selecting <Remove documents from migration set> from the context menu. Objects removed from a migration set become unassigned and are available to be added to another migration set or ignored if these objects are not supposed to be migrated.

Processed Objects View

Opposed to the Source Objects view, the Processed Objects view displays the processed state of the objects in the selected migration set. An object is considered as having been processed if it underwent transformation (and any other operation following transformation, such as validation or import). This view will display the newly generated attribute values for any objects that have been successfully transformed, and the status the processed objects are currently in (i.e. Transformed, Validated, or (Partially) Imported)

If no transformed objects exist in the currently selected migration set (which implies no validated or imported objects can exist either), this view will be empty. Also, if transformation rules are applied to objects but all objects fail transformation this view will be empty as well (in this particular case all objects would be displayed in the third view, Errors)

In addition to the -Source Documents- view, the -Processed Documents- window features a few more functions.

Transforming, validating, and the resetting objects can be performed directly from this window using the appropriate toolbar buttons.

[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Apply Transformation & Validation] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.

For quick access to the source attributes of any object the -Source Attributes- window can be also accessed from the context menu.

Individual objects can also be reset to their default, unprocessed state using the <Reset selected documents> item from the context menu. Resetting an object will essentially remove any metadata generated through transformation rules, processing and status information from such objects. This context menu entry is slightly different from the [Reset all documents] button on the toolbar, which, as the name implies, operated on the entire set of objects instead of an individual selection of objects.

Resetting imported objects would typically be used during testing phases of a project and not during production.

The Processed Objects view offers some new quick filter options below the common filter bar. A processed object includes any objects with the Transformed, Validated, Importer or Partially imported status; objects can be filtered quickly for any of these states using the appropriate checkboxes. Pressing the [Apply Filter] button is not needed just (de)select the checkboxes corresponding to the required status. These quick filter options also work together with the common filter bar, acting as an additional filter criterion.

In the Processed Objects view it is also possible to edit individual objects’ metadata. The [Edit Attributes] item from the context opens a new window with the currently selected objects attributes ordered as a list instead of columns. In the -Edit Attributes- window it is possible to edit any of the current objects’ attributes manually. The -Edit Attributes- window also allows applying the changes, skipping the current document, or moving to the next document in the list without closing the window.

Any manually edited object will be reset to the Transformed status. This makes a difference for objects which have been validated already; since the user has edited these objects manually, the information the object has been validated with has changed and the object will be reset to the Transformed status and require validation using the new attribute values.

Error Objects View

The Error Objects view displays all objects that failed at some point during the migration. All states an object goes through during migration have a corresponding error state, such as Transformation Error, Validation Error, etc. The Error Objects view collects all objects in any state considered as an error and provides a centralized location for tracking such documents, identifying and correcting such objects. In addition to listing Error Objects and their associated attributes, the error message for causing the object to fail be listed in addition for every object; if the error is related to any particular attributes, those attributes will be colored red and hovering over such an attribute will display an additional message about the cause of the error.

All this information makes it easy to identify and correct the error and transfer the object into the proper states allowing it to be migrated.

The Error Objects view provides the same functionalities as the Processed Objects window, since both views serve the same goal: to prepare objects to be migrated and successfully meet the required conditions.

Errors are not a final state in migration-center. If the error can be fixed, it is also possible to correct the objects affected by that error as well and start a new attempt for migrating affected objects. This procedure can be repeated until all objects with errors have been fixed one way or another:

There are five basic ways of handling errors; one of them should always be able to solve typical errors encountered during migration

If the error is caused by a transformation rule (objects in Transformation Error status), adjust the transformation rule to cover the faulty objects and re-apply it to those objects. Any objects which have already been transformed, validated or transformed successfully will not be affected and will maintain their current status and metadata
If the error cannot be fixed by adjusting transformation rules and the object is in the Transformation Error status, it may be necessary to edit objects manually. Use the <Edit Attributes> window to browse through the objects and edit the attributes manually, skipping any objects where manual editing does not apply.
If the error occurs during import due to external factors (objects in Import Error status) like software or hardware issues, network issues, target system configuration problems, proper access permission, etc.) try to remedy the error first and after that just add the migration set back to the importer and simply try to import it again. Migration-center will attempt to import all objects in the Import Error state automatically and will succeed if the cause for the error has been fixed.
If objects have failed Validation (objects in Validation Error status) check the Object Type definitions and Associations in the Transformation Rules window, make the necessary corrections and re-run the Validation process to validate these objects successfully.
Finally, if none of the above helps, or the objects having failed are not even needed to be migrated they can either be removed from the migration set or left in their respective error states (temporarily or permanently). Unless other objects depend on them, leaving unneeded objects in an error state or removing them from the migration set will not affect other objects.

Transformation rules

The transformation rules are a set of rules that apply to all objects of a migration set, transforming the attributes of the objects inside. Transformation rules determine how the metadata provided with the source objects is going to be altered to generate new or modified attribute values, and how attributes from the source and target object models will be connected. A transformation rule consists of one or several transformation steps, where each step references one transformation function. The final transformation step will return the result for a rule.

The -Transformation rules- window can be opened by pressing the [Edit transformation] button in -Migration Sets- window, or the [Edit rules] button in both -View Processed documents- and -View Error documents- windows. Also, the -Transformation rules- window can be opened by context menu <Edit Transformation rules> the three windows where the button is present.

The -Transformation rules- window has three tabs for managing different aspects of transformation and validation, and a set of common buttons accessible from all tabs. These buttons are:

[Generate attributes] will automatically generate the simplest transformation rules based on all source attributes found for all object in the migration set. These simple rules would enable a 1:1 migration of the objects in the current migration set. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected. This function will be further discussed in the next chapter (Attributes).

[Export] will export the current migration set’s transformation rules to a XML file for the purpose of backing up or transferring the model to a migration set located on another DB.

[Import] imports transformation rules generated by the XML files generated with the previous export function. Warning! Importing rules from a file will replace all rules configured previously!

[Ok] saves changes and closes window.

[Cancel] cancels all changes that have not been applied yet and closes window.

[Apply] saves changes without closing the window.

Rules

The |Rules| page consists of four areas, namely Rules, Rules for system attributes, Rule properties and Transformation methods. The Rules and the Rules for system attributes work the same way: selecting a rule will allow its properties and transformation methods to be edited using the controls on the right side of the window. While the former is meant to allow the user to create, edit, and delete any rule, the latter does not allow rules to be added or deleted by the user, it only allows editing the existing rules, which are predefined system rules defined by the object type of the migration set (the object type selected when creating the migration set). Attributes initially obtained by scanning (DOCUMENTUM repository or file system) are called source attributes. They are used as source for direct transformation or for generating more complex transformation rules from them. Source attributes are accessible in the Transformation methods dynamic area.

Rules list

Rules are not immediately connected to any attribute or property of the target system. In the most general sense, rules have arbitrary names and use various source attributes as input, process these using the specified transformation functions and finally output the resulting value. An actual connection between these rules and the object types and their respective attributes is created on the Associations page, which will be discussed later. This generic approach allows migration-center’s Transformation engine to adapt to a variety of content management systems which use generic concepts such as type of object and attributes/properties.

The [Generate rules] button can be useful for starting work on transformation rules. This feature will generate rules based on any source attributes it can find among the objects from the current migration set. The rules will be named using the respective attribute names and will contain only a basic transformation function. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected.

These simple transformation rules will extract the original attribute values without performing any processing on them and would be suitable for a 1:1 migration. For most real-world migration projects at least some of these transformation rules will require changes to make them suit the requirements though. It is of course possible to ignore this feature altogether and add rules one by one, according to the requirements and possibilities of the project.

Rules for system attributes

The system attribute rules work the same way as user defined transformation rules. System rules are predefined rules and usually refer to key information that needs to be specified for the objects to be migrated and work correctly in the respective target system. System attributes depend on the type of object used in the migration set (specified during creation of the migration set and displayed in the -Transformation Rules- windows’ title bar). System rules cannot be renamed, deleted and new system rules cannot be created by the user; only the transformation rules for the predefined system rules can be edited. Here the full functionality of transformation functions is available, as for the user defined rules.

The migration-centers adapter targeted at Documentum provide following system attribute rules:

“dctm_obj_link” specifies the path(s) where a document should be linked to in the target repository. “dctm_obj_link” is also an example for a system attribute which is defined by and used internally by migration-center and does not require association with an actual target attribute. This means there is no need for an actual attribute named “dctm_obj_link” to exist in either the source or target system.
The system attribute “dctm_obj_rendition” specifies file system location to files which represent renditions. Zero or more renditions can be specified for any object and will be added by migration-center during import. This attribute is also an internal attribute used by migration-center similar to “dctm_obj_link”
“r_folder_path” and “r_object_type” can get their default information automatically (see [Generate attributes]) only inside a Documentum-to-Documentum migration set. In the migration set of a file system scan, this information must be put in by user.

Rule properties

The Rule properties allows specifying the name for a newly created rule or renaming an existing rule. Whether a rule can work with multiple values can also be set using the Multi-Value checkbox. These controls are disabled for editing if a system attribute is selected since system attributes cannot have their properties changed by the user.

Transformation methods

The Transformation methods manages the transformation steps and functions used to perform the actual processing for the selected rule. Each step corresponds to a function. Any number of steps can be added to obtain the desired result for the current rule. At least on step must be added for a transformation rule to work.

The Function drop-down list all available transformation functions. Select a function and add it as a step to by clicking the [Insert] button. Functions will always be inserted last in the list. After inserting a function, it is possible to edit, reorder, or delete steps using the respective buttons.

Inserting a new function as a transformation step will immediately open the function’s properties for editing. Configure the function’s parameters accordingly and click [OK] to save the function and add it to the list.

Function properties can be edited at any time by selecting the corresponding transformation step and clicking the [Edit] button above. This will open the same properties window mentioned above and allows the user to make the required changes.

Selecting a function from the “Function” drop-down menu and hovering mouse pointer over the help-icon (“?”) next to [Insert], a windows tooltip will give the user more information about the selected function.

Error and Warning indicators for rules:

Migration-center always checks rules when saving and looks for invalid references, functions, or unused transformation steps. Rules will be colored red for “Error” or yellow for “Warning” as a result of this verification. Migration-center cannot correct such inconsistencies because the intended purpose of the affected rules is not known to migration-center. It is up to the user to check and correct any issues with rules marked red or yellow.

In the example below, the rule called “extension” has been marked yellow by migration-center’s rule auto verification feature.

The user can select that rule to check the cause of the warning. Once a rule marked as an Error or Warning has been selected the transformation rules will also be displayed in the rule’s properties. The actual transformation step causing the error/warning will also be marked red/yellow. In the example below, migration-center warns because there is an unused function in the list. This information is shown in a tooltip when hovering the cursor over the respective transformation step.

Warnings can usually be ignored and will not affect the result of the transformation rule. However, warnings should only be ignored if it really makes sense; e.g. in the example above the user may want to leave the unused step in there, and switch between the two steps for testing purposes. For production use though it is not advised to leave any unused transformation steps because these will be computed, needlessly adding to the time required for transforming the objects.

Rules marked red, indicating errors, are rarely encountered during normal usage. The highest chance for getting Errors is when importing or copying transformation rules into the current migration set; since the newly imported transformation rules may have been created based on another migration set, they may reference source attributes which do not exist among the current objects and this may raise an error because of the invalid reference. Errors can not only result from references to invalid source attributes, but also from invalid references to previously deleted transformation steps, or wrong parameter combinations for some functions.

Marking rules and transformation steps as errors works exactly the same as described above for warnings.

Errors should never be ignored because they will definitely affect, or even prevent transformation rules from working correctly.

Working with transformation functions

Transformation functions provide the functionality behind transformation rules.

Custom functions can be implemented and used along the standard functions provided by migration-center.

Functions typically have multiple input parameters, with one of them usually being the value that needs processing, and output the processed value which may be entirely different from the source value, depending on what the function is designed to do and what parameters it has specified.

Input parameters can range from referencing a source attribute to referencing the output of a previous transformation step, a value typed by the user to various function specific parameters which determine how that function should process its input our output. The number of parameters varies by function. Some functions (like Sysdate) do not have any parameters at all, while other functions have three or four parameters.

A function’s parameter always has two properties: a type and a value. The parameter’s value is determined by the parameter’s type. Parameters can be of the following types:

Source Attribute: the value will be extracted for each object's attribute individually during the transformation process. For objects which do not have the specified source attribute no transformation will take place.

User Value: acts as a constant value and will be set for all objects to which the current rule applies.

Previous step: references the value returned by a previous transformation function from the current transformation rule and uses it as input. If an input value is multi-value, the index of the value to be processed can be selected or typed into the drop-down list on the far right. By default, all values are processed for a multi-value input

The Index selection (the small dropdown list to the far right) is for working with multi-value (repeating) attributes. By default, any function will operate on all values of a multi-value attribute, but it is possible to specify a particular value by setting its index (i.e. the n-th value of a multi-value attribute) in the Index dropdown list.

In case the parameter in question is referring to a single value attribute selecting an Index will have no effect.

The Index selection list offers the following options:

All is the default option and can be used only for the first parameter and not the subsequent parameters.

A specific index ranging from 1 to n (where n is the maximum number of values for the currently referenced multi-value attribute) can also be specified. In this case the current transformation function will only process the requested value and ignore all other values, effectively returning a single value output from a multi-value input.

The Index selection offers some more options when used with the If transformation function:

It is also possible to specify All as an option for the Return if true or Return if false parameters.

When All is used in first parameter, the If is called once for each value of the first parameter (as long as it has multiple values). The corresponding index value from Return if true or Return if false will be returned after every call. This means the result will have the same number of results as the number of values of first parameter. This is the same behavior as in all other transformation functions when used with index All.

The value Any is exclusive to the If transformation function and can be used to define a condition based on a value whose index is not fixed or is simply unknown to the user within a multi-value attribute and therefore cannot be specified using the All or 1 – n settings

When Any or a specific index 1-n is used for the first parameter, the function is called only once and will return all values specified in Return if true or Return if false. If All is used in a return parameter all values will be returned, otherwise only the specific index will be returned.

Transformation function reference

All standard transformation functions provided with migration-center are described below. Full context-sensitive information is also available in the migration-center Client when inserting or editing any transformation function.

Mapping Lists

Mapping lists can be used as an alternative means of transformation in case the desired result cannot be obtained using transformation functions. Usually mapping user names or permissions falls under this type of data, since there is no way to create a transformation rule which maps a list of several hundred users or permissions onto their corresponding new values individually; mapping lists can of course be applied to any attribute, not just the ones mentioned in the example.

Mapping lists are specific to one migration set and are only available for use within the migration set where they are defined. It is also possible to define so called global mapping lists which are available in all migration sets.

Mapping lists are simple 2-column lists and work based on a key-value concept, where the key is specified in the first column, and the associated value in the second column. If the MapValue transformation function is used in a transformation rule with a given mapping list and matches a key from the mapping list to its input value, it will output the corresponding value from the mapping list as the result.

A mapping list must be referenced by a “MapValue()” transformation function in at least one transformation rule on the |Rules| page to have any effect on the objects of the migration set. A mapping list will not do anything by itself.

“Exact Match” will require an exact match with the specified key if checked and will accept any string as a match as long as it contains the key if unchecked. It is recommended to check this option; uncheck it only if needed.

“Case sensitive” will consider a valid match only if the case of the input value matches the key and its case. Usually this would not be used, enable if needed.

Mapping list can be edited in-place if needed, but this is only for short lists or making minor corrections to an existing list. Usually a mapping list would be prepared using an appropriate application such as a spreadsheet, then copied and pasted into migration-center via the context menu available when right clicking into the list area.

The example above illustrates using a mapping list with the “MapValue()” function. Here the mapping list “opened by” is used through the “MapValue()” function in the transformation rule “extension to OPENED BY”. The window below shows the properties for this function.

Migration-center will interpret this transformation rule as follows:

In all objects of the migration set look up the source attribute “extension” and try to match its value to any of the following “mp3, doc, pdf, ppt”; if a match is found, replace the source value with the value corresponding to the matching key specified in the mapping list named “opened by” and output this value as the result.

Global Mapping Lists

Global Mapping Lists are mapping lists independent of any particular migration set. They are thus globally available to all migration sets across the entire application. Use global mapping lists for information which will be used by multiple/all migration sets. Global Mapping Lists can be managed from the main menu <Manage>, <Global Mapping Lists>. The functionalities and properties for creating, copying, deleting, and editing global mapping lists are identical to the ones provided for regular mapping lists, described above.

When using the “MapValue()” function, both the “local” (migration set specific) and the “global” mapping lists are available for use. Global mapping lists can be identified by the word “(global)” appended to their name.

Associations

In the |Associations| page existing transformation rules can be associated with actual object types and attributes from the target system. These associations will provide a means to validate the values generated by the transformation process, and will tell the importer how to assign these values during import

Types

To create a connection from a transformation rule to a target object type there needs to be one transformation rule which outputs the object type’s name. Select this rule from the “Get type name from attribute” drop-down list. For Documentum “r_object_type” is a predefined system rule which server this purpose, but there is no restriction to select a particular rule; any rule will work if it results in something which matches object type names and is not null.

From the “Available type definitions” drop-down list, add the required object type definitions below by pressing [Add]. The object type definitions displayed depend on what is available in the -Object Type Definitions- window. If a required object type definition is not available for creating an association here, Click [OK], add the object type definition, and come back to this page to create an association with it (See chapter 7.8 Object type definitions for details on adding object type definitions to migration-center).

Associations

Selecting an object type definition will populate, the “Target attribute” drop-down list with the attributes of the selected type. The “Rule” drop-down list shows the rules defined on the |Rules| tab. By matching pairs from these two controls and clicking [Add Association] rules can be associated with corresponding attributes. A rule and a target attribute can be associated only once per type.

In case the rules already have names that match attributes of the targeted object types, there is a much quicker way for creating these associations: just click [Auto-Associate]. This button will automatically match rules and attributes by name and add them to the list of associations. Any unneeded rules created this way can be deleted afterwards, and any rules that could not be associated automatically can still be added using the regular method described above.

Note: An association always applies to the currently selected object type in the list, not all of them! If there are multiple object types targeted for a migration, manual/automatic association procedure described above needs to be done for each object type.

Importers

Import is referring the process of transferring content and metadata obtained using the transformation rules to the target system. The importer is the job actually performing this task of outputting data from migration-center or importing data into the target system.

Importers migrate one or more migration sets. Only migration sets with objects that have successfully passed both transformation and validation can be imported.

Similar to a scanner, an importer is a job with a unique name and a set of configuration options, which is started by the user and the corresponding adapter code is executed on the specified Job Server.

Note: for details about the scanners provided with the standard product consult the appropriate documents listed in 1.3 References. For details about customized scanners or scanners provided by third parties please consult the documentation delivered with the respective adapters.

The toolbar and the context menu of the -Importers- window contain the same commands.

[Properties] opens the configuration details of that particular importer. The -Importer- properties window can also be opened by double-clicking an item in the list.
[History] opens the -History- window where more accurate data about all imports performed by the importer in question can be accessed. The log of the scans can be accessed here.
[Run] initiates the run of an importer with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the importers listed
[New] opens a blank -Importer- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected importer, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

The log files generated by an importer are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

All importers, regardless of type share some common parameters

Name: must be a unique name for the current importer

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the importer depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this importer does or add some comments related to the importer

Scheduler

A Scheduler acts as a master process which manages and runs an end-to-end migration fully automated and according to a set schedule.

A scheduler is also a job, and will use preset scanners, transformation rules and importers to run extract, process, and migrate objects to the target system automatically. A full history is available for each run, listing both objects that have been imported successfully as well as any objects which may failed during the process.

There are a number of configuration options regarding the times when such an automated migration should run, ranging from minutes to months, days of the week, or hours of the day. This makes it possible to tailor a schedule according to the needs and possibilities of the project and avoid running performance intensive migration during working hours for example.

The toolbar and the context menu of the -Schedulers- window contain the familiar items: New, Properties, Delete, Run, Refresh.

Double-clicking a scheduler in the list has the same function as the <Properties> menu command.

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

A scheduler can be run immediately if needed, regardless of its actual schedule, through the <Run> command found in the toolbar and the context menu.

Properties

An email report can be scheduled to be sent to a certain email address in case of success or failure of the scheduler processes. For this a valid SMTP-Server and email address must be provided, and at least one of the two checkboxes flagged.

Object Types definitions

Migration-center needs to know the properties of the object types targeted during migration to correctly associate transformation rules with the attributes from these object types and also for validating the values generated by transformation functions. These target object types can be managed in migration-center Client in the -Object Types- window. The window can be opened from the main menu <Manage> - <Object Types…>.

All target object types in use should be defined before attempting to create associations or validate the objects in a migration set. For creating and testing transformation rules it is not necessary to have objects types defined at that point.

Out-of-the-box the product comes with basic object type template definitions for the adapters the customer has acquired licenses for.

Additional user/custom object types can be added to migration-center by the user.

To create a new object type template, click the [New] button in the Object Types toolbar or <New> in the context menu inside the same area. Enter the template’s name in the “Name” field; the name should be exactly the same as the actual name of the object type. The [Delete] button

In the list below the individual attributes and their properties can be defined. Since the process of defining all types and attributes by hand would most likely be too time-consuming it is possible to import an object type directly from a properly formatted CSV file. Object type definitions can be imported using the [Import] button located in the Object templates dynamic area. The imported file needs to be a CSV file (comma separated value) similar to the following example, with each row containing attribute name, data type, minimum length, maximum length, repeating flag, mandatory flag, and an optional regular expression:

id,2,0,0,0,0

first_name,2,0,0,0,0

last_name,2,0,0,0,0

email,2,0,0,0,0

gender,2,0,0,0,0

ip_address,2,0,0,0,0

animal_name,2,0,0,0,0

Valid values for data type are: 0 - Boolean, 1 - Integer, 2 - String, 3 - ID, 4 - Date and Time, 5 – Double.

Some importers provide best practices for extracting the target types from the source systems using common tools. Documentum Importer User Guide and SharePoint Importer User Guide can help you with such suggestions.

The optional regular expression that you can provide for an attribute will be used to validate the attribute values during object transformation and validation. Please provide a rule that matches all valid characters for that attribute. If the attribute value contains any characters that are not matched by the regular expression, its validation will fail. For example, the following regular expression will match all characters that are valid for a SharePoint 2013 file name:

^[^\~\"\#\%\&\*\:\<\>\?\/\\\{\|\}]+$

It will match one or more characters between begin and end of the string that are not in the list of the following characters: ~"#%&*:<>?/\{|}

You can also use the regular expression to enforce a certain data schema for the attribute values. For example, to enforce a schema with three digits followed by a minus followed by five digits you can use the following regex:

^[0-9]{3}-[0-9]{5}$

Object History

Migration-center tracks key stages throughout the lifecycles of object being migrated (such as scan, import, deletion, etc.) and stores information about the exact state of that object at that point in time. The information stored about the objects includes properties such as the status, timestamp, set of metadata, or the ID the object was imported with in the target system. The Object History serves as an internal reference to all objects that have been migrated.

The -History- window can be accessed from migration-center’s main menu <Manage>, menu item <Object History…>

Log files

A complete history is available for any job (scanner, importer, scheduler) from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated for each job can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job

Log files generated by the migration-center Client application in the event of errors can be found in the Client application’s installation folder, e.g. …\fme AG\migration-center Client <version> folder inside the installation path. These logs are not meant for the user but should be sent if requested by our technical product support staff in case there are issues preventing the Client from working normally.

Sizing Guide

Introduction

The present document offers a guideline for sizing and configuring migration-center components in different migration scenarios. Due to the complexity of the migration projects, enterprise environments and applications migration-center interacts with, this guide does not provide exhaustive information, but it offers best practice configurations accumulated in more than 100 migration projects.

The Installation Guide and Database Administrator’s Guide contain a full description of the system requirements and installation steps for all migration-center components and should be consulted first. This document does not substitute the documents mentioned above but completes it with information that might be of importance for getting the best migration-center performance.

Targeted audience

This document is targeted specifically to the technical persons that are responsible with installation and configuration of migration-center components.

Common configurations and common sizing guidelines

This chapter describes the configurations and sizing guidelines that apply to any kind of migration project.

Client

The following hardware and software resources are required by MC client in any of the migration scenarios described in the next chapters.

Server components

Oracle instance

The migration-center database stores all information generated during a migration project. This includes job configurations, object metadata, transformation rules, status and history information, etc.

Small projects (up to 500,000 objects)

Sizing and configuration tips

No special hardware or software requirements
All MC components can be run on the same machine

Deployment overview

For such a small migration projects, where up to 500,000 objects need to be migrated, there are no special hardware or software requirements. Basically, all three MC components can be run on a single desktop computer or on an adequately sized virtual machine. In this case the machine should have enough physical memory for OS, Oracle server, MC components and other applications that may run on that machine. The recommended processor is a dual or quad core having a clock rate of minimum 2.2 GHz.

Oracle instance

Standard database installation can be followed (as it is described in “migration-center installation guide”)
RAM: 4 GB of total memory for the instance.
Use “Automatic memory management”. This can be chosen when creating the Oracle instance.

Medium migration projects (500,000 – 5 million objects)

Sizing and configuration tips

Dedicated Oracle Server machine
Two MC Job Servers for a better scalability

Deployment overview

It is recommended to use a dedicated machine for the Oracle instance needed by MC.

For a better scalability of scanning and importing jobs or when migration timeframe is short, it’s recommended to deploy the Server Components (Job Server) on two machines. In this way you may run multiple scanner and importers in parallel speeding up the entire migration process. The performance of scanners and importers is dependent by the Source and Target System so if one of those systems performs slowly, the migration process can be speeded up by running multiple scanners and importers in parallel. If necessary, the number of deployed Jobservers can be extended.

Oracle instance

The host should not necessarily be a server machine. A well sized desktop machine would be enough.
CPU: Quad Core, min. 2.5 GHz
8 GB of memory allocated for the instance
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible, the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 200 MB.
It is very recommended the instance has at least 3 Redo Log files having in total a minimum size of 1 GB. This is important when multiple big transformation and validation jobs are run by MC because those jobs update a big number of database rows that require enough redo log space. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Large migration projects (5 million – 15 million objects)

Sizing and configuration tips

Dedicated Oracle Instance running on a Server machine.
Three or more MC Job Servers for a better scalability

Deployment overview

It is recommended to use a dedicated Oracle Instance running on server hardware.

For a better scalability of scanning and importing jobs three or more instances of Server Components (Job Server) need to be deployed. The performance of scanners and importers is dependent by the Source and Target System so if one of those systems performs slowly, the migration process can be speeded up by running multiple scanners and importers in parallel. If necessary, the number of deployed Job Servers can be extended.

Oracle instance

A dedicated server machine is required. It is recommended to use a dedicated Oracle instance for running only Migration Center but no other applications.
CPU: 4-8 Cores, min. 2.5 GHz
16-32 GB of memory allocated for the instance
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 400 MB.
Make sure that instance has at least 4 Redo Log files having in total a minimum size of 1.5 GB. This is important when multiple big transformation and validation jobs are run by MC because those jobs update a big number of database rows that require enough redo log space. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Very large migration projects (more than 15 million objects)

Sizing and configuration tips

Multiple Dedicated Oracle Instances running on a Server machine or use an instance on a high-performance Oracle cluster
Four or more Job Servers.
A file server or a storage server used for temporary storing the content.

Deployment overview

The deployment for the migration of a very large number of objects should be planned carefully. In this case it is recommended to use multiple MC databases. In most cases it is not advised to migrate more than 10 million objects with a single Oracle instance even if the hardware is sized accordingly. There are several reasons for using multiple database instances:

The data from different sources is not mixed up in a single instance helping the user to handle easier the errors that appear during migration
The transformations and validations will be better scaled on multiple database instances
Facilitate the work of multiple migration teams

A dedicated file/storage server should be shared by all Job Servers for storing and accessing the content during migration. This will help the migration in the way that set of objects scanned with any scanner can be imported with any importer.

Oracle instance

Several dedicated server machines are required. For each machine it is recommended to use a dedicated Oracle instance for running only Migration Center but no other applications.
CPU: 4-8 Cores, min. 3.0 GHz
Minimum 16 GB of memory allocated for each instance.
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 600 MB.
Is recommended that FMEMC_DATA tablespace to be split on multiple physical data files stored on different physical disks in order to maximize the performance for a specific hardware configuration.
Make sure that instance has at least 5 Redo Log files having in total a minimum size of 2 GB. This is important when multiple big batch jobs (transformations, validations, scanners, importers) jobs are run by MC because those jobs update a big number of database rows in quite a short time and therefore redo log space should be sized accordingly in order to prevent redo log wait events. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Auto Classification Module

Introduction

This document is a guide on the auto classification module for migration-center 3.

Creating transformation rules on vast amounts of documents, can be a laborious task. This module can support the user by automatically classifying documents to predefined categories. Configuring this module might require basic knowledge about machine learning principles.

Installation

The installation process can be performed in a few minutes. But before installing the module, two mandatory prerequisites must be installed.

Prerequisites

The module requires a Java runtime environment of version 8 or higher and Python v3.7. Make sure to configure the environment variables PATH and PYTHONPATH for Python during the installation process in the interactive installation dialog. To do this, you need to check the box “Add Python 3.7 to PATH” at the first installation window. If you are unsure, read the official installation instructions.

You can configure the environment variables later, but it is mandatory for the module. More information can be found in the official Python documentation. If the variables are not configured, the module might not start or work as intended.

Along with the Python language, the installer installs the package manager “pip”. “pip” is a terminal software and needs to be executed in the Windows CMD shell.

If the use of OCR (optical character recognition) is intended, Tesseract with version 4 or 3.05 must be installed (see Installing TessaractOCR). Two other use cases require the installation of the graphical library Graphviz (see Installing Gaphviz) or the Oracle Instant Client for database access.

In addition, it is possible to use a dedicated Apache Tika instance. If you like to do that please refer to section Installing Apache Tika.

The operation of the Auto Classification Module requires configuration files. The samples subdirectory contains a set of example files.

Installation Steps

The installation of the auto classification module can be summarized in seven steps.

Download and extract the software code from the zip archive onto your hard drive. Copy the files to a directory of your choice.
Open a CMD window and navigate to the path where the software was copied to or open a command window from the Windows Explorer.
Install all Python package requirements by executing the following statement in the CMD: pip install -r requirements.txt
(optional) Install Oracle Instant Client (see here)
(optional) Install TesseractOCR (see Installing TesseractOCR)
Create the “config.ini” file, as described in section Deploying the auto classification module.
Provide a classifier configuration file. The configuration schema of the file is specified in Classifier configuration.
Start the module by executing the init file in the CMD: python __init__.py {path to config.ini} The script accepts the file path to the “config.ini” file as the only parameter. If the file path is not provided, the script assumes that the “config.ini” file is in the same directory.
Open the GUI in a web browser: http://{server ip}:{WSGI port}/app

After executing the script, the service boots. Please wait until the service has started successfully, before executing any further operations. The CMD windows needs to stay open.

If additional resources are available or the classification is time critical, it is advised to install the AC-Module and Apache Tika with TesseractOCR on two different servers, although all programs can be executed on the same machine.

Installing TesseractOCR

TesseractOCR is a software for Optical Character Recognition. It extracts text from image files and images embedded in PDF and Office documents. Tesseract should be used if any scanned PDF files are to be classified or documents contain images or diagrams with valuable information.

If scanned PDF files are to be classified, but OCR is not installed, the Auto Classification Module will not be able to extract any text from those files. On the other hand, if it is known that documents contain images with valuable text for the classification process, OCR should be used, as well. Valuable text or valuable words in a document are those words that might be unique for a small size of documents and make the prediction of the correct category easier.

Windows

First, download the Windows installer from the official GitHub repository. When the download is completed, execute the installer. When asked what language packs should be downloaded, select the languages that are going to be classified.

Also, you can add additional languages later on.

Add the installation path into your environment Path variable.

Linux

First, download the Linux installer from the official GitHub repository or from apt. An installation instruction can be found in the project’s wiki.

Installing Apache Tika

By default, the module is shipped with the capability to download Apache Tika from the official download page on first use. Tika will be started on the same machine as the module and no changes of installation procedures are generally required.

If no connection to the internet can be established, the same procedure as starting Apache Tika from a dedicated server can be applied.

To use Apache Tika on a dedicated server, please download the tika-server.jar file from the official repository first.

Afterwards, the tika-server.jar file must be executed and the port must be open for outside communication.

The connection details must be provided to the module via “config.ini”. The configuration file “config.ini” holds three variables that need to be edited. The variables can be found in the Tika section.

“tika_client” needs to be set to true.

“tika_client_port” needs to be set to the correct port number.

“host” needs to be set to the correct name / IP address.

If Tika runs on port 9000 in a dedicated process on the same machine, the configuration looks as follows:

[TIKA] tika_client: true tika_client_port: 9000 host: 127.0.0.1

Installing Graphviz

Graphviz is an additional library for creating charts and plots. It is not mandatory and is only required if the attribute structure of a document repository is investigated and a chart should be created. If Graphviz is properly set up, the script cli_show_distribution.py will automatically create a chart of the attribute structure.

First, Graphviz needs to be downloaded and installed.

Second, the Python wrapper for Graphviz needs to be installed in the Python environment.

On Linux OS, pygraphviz can be installed easily with pip in the CMD:

pip install pygraphviz

For Windows OS, an installation file is provided. Usually, the installation file is already used when installing all library requirements from the “requirements.txt” file. The installation file is a whl file and can be found in the directory “install_packages/”. The whl file can be installed with the CMD command:

pip install pygraphviz-1.5-cp37-cp37m-win_amd64.whl

The supplied whl file is compiled for Windows OS with 64 bit and Python 3.7 installed. Other compiled whl files for Windows can be downloaded from the GitHub repository.

Configuration

The auto classification module offers a powerful functionality to enhance migration project. However, using the module can be tricky. Therefore, this document provides in-depth information for using and configuring the module in a production environment.

Deploying the auto classification module

The module needs to be deployed as a service. Auto classification is a memory consuming task and it is advised to run the module on a dedicated server with access to as much memory as possible. If OCR is performed, it is advised to use a GPU, because character recognition on a CPU is very time consuming.

To deploy the auto classification module, a configuration file must be configured. The file can be placed in any accessible UNC file path. If no configuration file is provided during the initial startup, the module will look for a configuration file in the working directory. In this case, the file must be named config.ini.

The following two chapters provide information on how to set up SSL encryption and a valid path to an algorithm configuration file.

Template files are provided in the sample directory of the module.

Port configuration

The configuration file contains two port properties. One property is defined in the FLASK_SERVER section and the other in the WSGI_SERVER section.

The Flask server is only used for development.

Therefore, if you like to change the port number of the AC module, set the port property under the WSGI_SERVER section. The definition can look like this:

port: 443

By default, the port is set to 3000. This means that every interaction with the module goes through port 3000. This includes API requests and interacting with the web app.

Configuring SSL

The module can provide encryption of HTTP traffic with SSL. It is hardly advised to use this feature to prevent any data interception. Please keep in mind that either HTTPS or HTTP can be used. A dual usage of both modes is not supported.

SSL encryption can be activated by defining the following line in the SSL chapter of the configuration file:

ssl.enabled: true

To use SSL encryption, it is mandatory to provide a valid SSL certificate. The required properties are:

ssl.certificate_file_path ssl.key_file_path

If you do not have access to a valid certificate, you can create one on your own. You can use the program keytool to create a self-signed certificate. The program is shipped with the Java runtime and can be found in the ${JAVA_HOME}/bin folder.

Classifier configuration file

To be able to use the auto classification module, a classifier configuration file must be provided. The file can be specified with the property

file_path

in the config.ini file under the section ALGORITHM_CONFIG

The file’s schema and configuration options are described thoroughly in the section Classifier configuration below.

Starting the module

The module is a Python program. Therefore, starting the module is very easy. Just enter the following command:

python __init__.py {file path to config.ini}

Classifier configuration

A classifier is a combination of an algorithm with a transformation list. Before a classifier can be used, it must be configured and trained.

As a basic understanding, one must know that an algorithm cannot be optimal for every situation, although it might be sophisticated and very powerful. Each algorithm has advantages and disadvantages and one algorithm can solve one problem better than another. Therefore, the most significant boost and most suitable adjustment can be achieved by providing only the most meaningful data to an algorithm. This is what a transformation list can be employed for.

The use of a transformation list is aimed to select the most meaningful words (feature selection) and create additional data from files (feature extraction).

A transformation list can be interpreted as a pipeline for the words of a document before they are submitted to an algorithm.

To explain the value of transformation lists and their relationship to algorithms, a simple example is depicted. The figure below displays the relationship between algorithms and transformation lists. Each entity has a unique id within their type. The list with id 1 enables three transformations:

lower-case
tf-idf
stopwords

The second list (id:2) uses the same transformations and adds the usage of OCR. It is not important in which order the transformations are listed as transformation lists are only used to enable transformations and not to define an ordered sequence.

Next to the transformation lists are three algorithms. Each algorithm points to exactly one transformation list, but one transformation list can be referenced by many algorithms. Based on these relationships, one can reason that three classifiers are defined. If each classifier is tested with the same documents, each of them will compute different results.

A second example displays how transformation lists are applied to documents. During the stages of training, test and prediction, raw documents are supplied. These unprocessed documents need to be transformed by the pipeline of the transformation list.

First, the stop words of the documents are removed. Followed by the replacement of all upper-case letters by their lower-case equivalents. Third, the tf-idf values are computed. In the end, the documents are processed and can be delivered to the SVM. The order of pipeline transformations may change on which transformations are selected.

All classifiers of the auto classification module are configured in a single XML file. The main XML elements are

algorithms
transformation-lists
common-config

The following chapters aim to explain the configuration possibilities in detail.

Also, a template and sample XML file are provided.

Element “algorithms”

Inside the “algorithms” element the individual algorithms are specified. The supported algorithms are:

SVM (element name: “svm”)
Naïve Bayes Multinomial (element name: “naive-bayes-multinomial”)
Logistic regression (element-name: “logistic-regression”)

All algorithms must have an id attribute (attribute name : “id”) with a unique value and an attribute with the correspondent transformation list (attribute name: “transformationListId”). Some algorithms support the prediction of multiple attributes (called multi-output). If multi-output behaviour is desired, the attribute “multi-output” must be set to true.

Additionally, every algorithm offers attributes to configure its behavior.

The SVM supports the following attributes:

kernel (value of “linear”, “poly” or “rbf”)
degree (positive integer, affects only poly-kernel)
gamma (positive double, affects only rbf-kernel)
c (positive float)

Naïve Bayes Multinomial offers the following attributes:

alpha (positive float, affects only Naïve Bayes)

Element “transformation-lists”

A transformation list is a group of transformation functions. It must have a unique id value. Each transformation function operates on the content of documents and transforms the content to a specific output. They can be used to optimize the process of feature selection. The order in which they are specified in the XML file does not matter. The functions are applied before an algorithm is trained. Not every function is supported by every algorithm:

On the following pages every transformation function is described.

Element “common config”

With the “common config” attribute, global properties can be configured. These properties are accessed by all classifiers. Currently it does not offer any configuration parameters.

Usage

The module offers the functionality to analyze a dataset and perform classification tasks with migration-center. The functionality can be consumed through a web service through a HTTP/HTTPS interface with a range of provided Python scripts or within selected adaptors.

The API interface will always respond with a body encoded in JSON format.

In addition, a Web-UI is provided with the Auto Classification module. The UI offers a clean and simple interface to interact with the module and it offers the same functionality as the scripts. The UI can be reached with the following URL:

https://{server-ip}:{WSGI port}/app

The following chapters explain the usage of the functionality to analyze a dataset and to train, validate, test and reinforce a classifier. Furthermore it describes how to make predictions for unclassified documents and to query a status of a training or reinforcement processes.

This document only describes how to use the API and not the UI. However, all parameters described in this chapter, can be used in the UI.

Using a metadata file

The following chapters explain the module usage by using scan run ids from the migration-center database. It is not always a desired practice to access the database. Therefore, the alternative method is to use a unified metadata file. A unified metadata file can be created when using the Filesystem Importer of the migration-center. The metadata file contains all necessary information for the Auto-Classification module.

To use a unified metadata file, the user needs to change the following things on a HTTP request:

Replace the “scan-run” part of the URL with “unified-metadata-file”.
Send the request with the mimetype “form-data”
Send the metadata file with the form key “file”.
(optional) Remove the properties of database connection and scan-run-id.

Analyzing a dataset

Before validating and training a classifier, the training dataset needs to be investigated. The module offers two different reports.

Distribution report

The distribution report creates a barplot for every attribute with the frequencies of the values on the y axis.

A distribution report can be created with a HTTP POST request to the URL:

https://{server-ip}:{port}/api/distribution-report/scan-run

It is required to supply three parameters with the request:

id
dbCon
classAttrs

The id is the scan run id, the database connection must be a JDBC connection string and the class attributes must be string of attributes, separated with a comma.

It returns a pdf report.

Hierarchy report

Much more detailed is the hierarchy report. It allows to specify an attribute hierarchy. It creates a graph which gives an insight into the frequency of every attribute value per hierarchy layer.

A hierarchy report can be created with a HTTP POST request to the URL:

https://{server-ip}:{port}/api/hierarchy-report/scan-run

It is required to supply three parameters with the request:

id
dbCon
classAttrs

The id and database connection parameters are the same as with the distribution report. The class attributes parameter needs to be a json object dumped as a string. It can look like the following example:

{

“type”: {

“subtype”: {}

}

In the example, type is the most general attribute and subtype is dependent on the individual values of the type attribute.

The hierarchy report is returned as a png file.

Modifying a dataset

After analyzing a dataset, one might want to remove certain documents or attribute combinations from the dataset. The dataset splitter functionality supports this process.

Dataset Splitter

The dataset splitter can split a single unified metadata file into two separate files. Furthermore, he can filter documents by attribute values.

It can either be used through the web service API or the web app.

The API function is available through a HTTP Post request to the following URL:

https://{server-ip}:{port}/api/dataset-splitter/unified-metadata-file

With the request, a metadata file (key: file) must be supplied.

Additionally, the user must define a splitting percentage by which the file is split into two files. The key for the payload property is “trainingPercentage” and it must be between 0 and 100. If the value is 0, all data will be stored in the test file. If the value is 50, the data is split in half and stored in both files. If the value is 100, all data is stored in the training file.

An optional third parameter is “exclusions”. This parameter allows to define attribute combinations that are prohibited in the newly created metadata files. The dataset splitter excludes documents based on the parameter. The parameter must be a json object with the following structure:

[ { ‘attribute’: ‘document_type’, ‘values’: [‘concept’, ‘procotol’] } ]

“document_type” is an example of an attribute name, as are “concept” and “protocol” examples of values.

The ampersand (*) is a wildcard value for the values property. If the ampersand is defined, all documents are ignored that hold the attribute, no matter what attribute value they actually have.

The response object is a zip archive which contains two xml files. The two xml files are valid unified metadata files. One file starts with the keyword “TRAINING”, the other with “TEST”.

A short example explains when the exclusion parameter can be used:

A user analyzed a dataset and found out that for the attribute “document_type”, the values “concept” and “protocol” have a very low frequency and they are not classifiable. He wants to remove them from the dataset as he aims to build a classifier on the “document_type”, but the concept and protocol values increase the noise.

From there on, he uses the dataset splitter and supplies the raw metadata file. As the “trainingPercentage”, he sets a value of 100. This will save all document’s metadata into the training file and leave the test file empty. The “exclusions” parameter is defined just as seen above. The configuration leads to the result that every document that either has “concept” or “protocol” as a “document_type” value, is ignored. These documents will not be saved in the new training file.

Therefore the returned zip archive contains a full training file and an empty test file. The theoretical outcome is that the new metadata training file has less noise and the classification of the “document_type” performs better.

As the exclusion parameter is a list, the user can specify more than one exclusion entry. The entries are not connected. This means that not all entries must be true. If one entry is true for a document, the document is automatically ignored.

Ubiquitous process parameters

The following chapters only mention specific parameters of the processes.

Two parameters are supported for the training, validation, testing and reinforcement process without being mentioned explicitly:

use-serialized-documents
description

Parameter “use-serialized-documents”

This parameter takes a Boolean value.

If it is true, it will serialize and store documents after their content has been extracted. This happens before any transformation functions are applied. If a document has already been stored on the filesystem, the stored file is loaded, instead of reextracting the content. This is helpful if OCR needs to be performed and the classifier parameter are being tuned.

One must define the “document_storage_path” setting in the config.ini file under the TIKA section.

Parameter “description”

Every process can have a description. This is helpful to identify a process later.

Training

The training of a classifier is an essential part of auto classification. The module can be trained by performing a HTTP POST request to the URL

https://{server-ip}:{port}/api/train

It is required to supply five parameters with the request:

scan-run-id
db-con
class-attr
algorithm-id
model-target-path

The scan run id is the id of a completed scan run inside of migration center. The documents within the scan run are used to train the classifier. If the scan run does not contain all documents for the training process, the classifier can later be reinforced (see chapter 4.9).

The parameter “db-con” is a connection string to the Oracle database of migration center. The string consists of the following structure:

{account name}/{password}@{host}:{port}/{instance}

The third parameter specifies the class attribute of the documents. If the classifier should predict multiple attributes, the attribute names must be separated by a semicolon.

The algorithm id references the configured algorithm inside the classification configuration file from section Classifier configuration.

The last attribute is a file path to the location of the trained classifier. This location can be any UNC accessible path. For the uses of testing, reinforcement and prediction this file path is consumed.

Depending on the number of documents, the length of the documents and the embedded images, the training requires a significant amount of time. Because of that the initial request returns a process id, after validating the parameters. The actual training process starts after the process id is received. The id can be used to query the module for a status of the training process. Please refer to section Status on how to query the process status.

It is advised to use the provided batch script train.cmd to train a classifier. The script requires six parameters:

Scan run id
Connection string to migration center database
Class attribute name (of the documents in the scan run id)
Algorithm id (as defined in the XML file)
Model path (the trained model will be saved in the location)
IP of the auto classification module service (optional)

If the IP is not defined, the script assumes that the service runs on localhost.

An example usage of the script can look like this:

train.cmd 432 fmemc/fmemc@localhost:1521/xe ac_class 1 C:\Temp\ac-module-dev.model

The module will use the documents within scan run “432” to train the classifier with algorithm id “1” to be able to predict the categories from the attribute name “ac_class”.

A second example shows how to train a classifier to predict two attributes:

train.cmd 432 fmemc/fmemc@localhost:1521/xe ac_class;department 1 C:\Temp\ac-module-dev.model

This classifier will predict the attribute "ac_class" and the "department".

Testing

Testing a classifier requires an already trained classifier.

A test process can be started by performing a POST request to the URL

https://{server-ip}:{port}/api/test

It is mandatory to supply four parameters:

scan-run-id
db-con
class-attr
model-path

The scan run id is the identifier of a completed scan run inside of migration center. The documents within the scan run are used to test the classifier.

The parameter “db-con” is a connection string to the Oracle database of migration center.

The third parameter specifies the class attributes of the documents, separated by a semicolon.

The last parameter specifies a UNC file path to a trained model file.

The module responds with an overall precision value and a list of tested documents. For every document, the actual and predicted classes and confidence values are provided.

Validation

A validation process combines a training and test process. The process splits the provided datasets into k packages and performs k training batches. The user defines the value for the k parameter (common values are 10 or 5).

In every training batch, one package is used as a validation package and the remaining packages are bundled into a training dataset. The validation package works like a test dataset. Because the number of packages and batches is equal. Each package is used as a testing package exactly once.

This process allows to tune algorithm parameters without using the actual test dataset.

You can start a validation process with a POST request to

https://{server-ip}:{port}/api/validate

It is mandatory to provide six parameters

scan-run-id
db-con
class-attr
model-path
algorithm-id
k

After the parameters are validated, the module responds with a process id before starting the validation. The process id can be used to query the current state of the process. Please refer to section Status on how to query a process state.

Grid-Search-Validation

The grid search validation process is a process to automate the tuning of algorithm parameters. It works similar to the general validation process, but uses a default and non-changeable k value of 5 and will not create a process report.

In exchange, the user can define multiple values for each algorithm parameter. The module will apply each cross product of the values on a validation process and return the precision and standard deviation of the predictions.

A grid search validation process can be started with a POST request to

https://{server-ip}:{port}/api/validate-grid-search-cv

It is mandatory to provide six parameters

scan-run-id
db-con
class-attr
model-path
algorithm-id
grid-search-config

The “grid-search-config” parameter is a string and must have the following schema:

Parameters must be separated by an ampersand (&).
Parameter values must be separated by a semicolon (;).

Therefore a sample parameter looks like the following:

C=0.5;1;2;5&degree=1;2;3;4

Because the process will apply every element of the cross product of the parameter values, a total of 16 validation processes are started.

Reinforcement

The process of reinforcement is similar to the train process. The only difference is that it uses an already trained model file and retrains the algorithm with the newly provided documents.

You can start reinforcement with a POST request to

https://{server-ip}:{port}/api/reinforce

It is mandatory to provide four parameters

scan-run-id
db-con
class-attr
model-path

The scan run id is the id of a completed scan run inside of migration center. The documents within the scan run are used to reinforce the classifier.

The parameter “db-con” is a connection string to the Oracle database of migration center.

The third parameter specifies the class attribute of the documents.

The last parameter specifies a UNC file path to a trained model file.

After the parameters are validated, the module responds with a process id before starting the reinforcement. The process id can be used to query the current state of the process. Please refer to section Status on how to query a process state.

Prediction

The prediction of a document’s class can easily be done with two options: with a HTTP API request or via the File System Scanner in migration-center.

HTTP API request

A unclassified document can be predicted with a GET request to

https://{server-ip}:{port}/api/predict

It is mandatory to provide two parameters

document-path
model-path

The module response with a classification and a confidence value:

File System Scanner

To be able to use the file system scanner for document prediction, it is necessary to train a classifier beforehand. Also, three file system scanner parameters must be configured.

The parameters are:

acModelFilePath
acServiceLocation
acUseClassification

The model file path points to a UNC file path of the trained classifier model file. The service location is the URI address of the deployed auto classification module. At last, the usage of classification must be enabled by ticking the parameter “acUseClassification”.

Now, a file system can be scanned by the scanner and predictions are automatically performed. The results can be accessed by viewing the attributes of the scanned objects (see figure below). In this case, the attribute “ai_class” holds the value of the predicted class. The classifier expresses his confidence in his own prediction with a value between 0 and 100. The higher the value, the greater the confidence.

If one is unsure which attribute has been predicted and where its confidence value column is, he should know the easy naming schema: The predicted values are displayed in columns starting with “ai_”, followed by the original attribute name. The confidence values column has the name of “confidence_”, again followed by the original attribute name.

Status

By initializing a training or reinforcement process, a process id is replied. The id can be used to query the status of the process at any time until the module is turned off.

The current state of a process can be retrieved by performing a GET request to the URL

https://{server-ip}:{port}/api/processes/{process_id}

It requires to supply one parameter

process-id

The response contains a status, message, number of processed documents, number of provided documents and list of document results.

The status can have any of the following options:

STARTED
READING_DOCUMENTS
TRAINING
REINFORCING
FINISHED
BAD_REQUEST_ERROR
INTERNAL_SERVER_ERROR
ABORTED

If the status is an error, the message property always offers an explanation.

The “number of processed documents” indicates how many documents have already been transformed by the functions in the transformation list. As soon as all documents have been transformed, the module changes the status to TRAINING.

The “number of provided documents” shows how many documents are in the scan run.

If the process is of type reinforcement, the “number of provided documents” is the sum of documents from the already trained model and the provided scan run.

The document results indicate whether an error occurred while processing a particular document.

Please take in mind that a process status is not stored forever. As soon as the module is turned off or is rebooted, the status of every process is lost.

Common and Known Issues

This chapter explains common and known issues that one might run into.

Images are not extracted from PDF files

If the module does not extract embedded images from PDF files, multiple points can be the reason:

Check if the PdfParser property “extract-inline-images” is set to true.
Check if the PdfParser properties “ocr-strategy” is set to “ocr_only” or “ocr_and_text”. If the module has been configured through the WebApp, the strategy is set automatically to “ocr_only” if “extract-inline-images” is true.
Check if the OCR property “enable-image-processing” is set to 1.

If only a subset of the embedded images are extracted, the issue can be the missing file type support for JPG files. Apache TIka uses the software “PdfBox” to process PDF files. The JPG file type is not automatically supported by PdfBox, because the required code is not applicable to the Apache license and prohibits the usage in commercial software projects.

Apache Tika does not start

In some circumstances Apache Tika does not start although it is properly configured in the config.ini file. Start an investigation as follows:

Check if the used ports are not occupied.
Check if the tika-server.jar file can be downloaded from the script over the internet (internet connection required). If not, download the newest tika-server.jar file and save it in a directory of your choice (i.e: C:\Temp). If applicable, you must rename the file name to “tika-server.jar” explicitly. Now, you must define the Apache Tika environment variable TIKA_PATH with the path to the directory of the tika-server.jar file:
SET TIKA_PATH=C:\Temp\ python __init__.py samples\config
Start a dedicated Tika server from the Windows CMD terminal. Download the newest tika-server.jar file from the official website and execute this command: java -jar tika-server.jar --config={path to tika config xml fil} --port=9000 If java cannot be started in general, try to use the absolute path to the java.jar. If this fixes the issue, you must set the Apache Tika environment variable TIKA_JAVA with the absolute path to the java.exe file before starting the module like so: SET TIKA_JAVA=C:\Program Files\Java\{jre version}\bin python __init__.py samples\config
If you are confronted with an exception that is not listed here, please get in contact with our technical product support at support@migration-center.com.

Documentum security enhancements

Introduction

The present document offers a detailed description of the Documentum security enhancements introduced with version 3.2.4 of migration-center.

The additional security features provide means for system administrators to define an additional layer of security between migration-center and Documentum repositories.

Some of the features offered are:

delegating access to users without having to give them superuser privileges and credentials directly on the respective repositories
controlling the type of operation a user is allowed to perform on a given repository (import and/or export)
controlling which paths a user can access to perform the above actions on a given repository
controlling where a user is allowed to export scanned documents to
limit the validity or duration of a security configuration by setting a date until which the configuration is considered valid
encrypting any passwords saved in the security configuration file

The implementation and configuration of the security enhancement features is transparent to end-users working with migration-center. End-users working with migration-center will be notified through on-screen messages if they trigger actions or use configurations which conflict with any enhanced security settings.

Usage of the enhanced security features is also optional. Removing or renaming the main security configuration file will disable the feature and will revert migration-center to its standard behavior when working with Documentum repositories (as described in the and user guides).

Targeted audience

The security features as well as this document is targeted specifically at system administrators managing Documentum Content Servers and access to the respective repositories, especially through migration-center.

Technical Information

System requirements

Implementation and integration with migration-center

The Documentum security enhancements features are implemented as an additional, optional module which integrates with migration-center’s Documentum Scanner and Documentum Importer.

The presence of the Documentum security enhancements module will be detected by migration-center automatically, and if a valid configuration exists the settings within will apply every time a Documentum Scanner or Documentum Importer job is executed.

The Documentum security enhancements module is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder. This folder contains the code package, sample configuration file, and tools used by the feature.

There is one tool for encrypting passwords to be stored in the configuration file, and another tool for validating the configuration file’s structure. The configuration file itself is a human-readable and editable XML file. The tools and configuration file will be described in detail in the following chapters.

Disabling Documentum Security Enhancements

The Documentum security enhancements feature is disabled by default and will become active only after a correct configuration file has been created and provided by the system administrator.

Should it be required to disable this feature after it has been configured and is in use, this can be achieved easily using either one of the below approaches:

Rename, move, or delete the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder
Rename, move, or delete the mc-dctm-security-config.xml file located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder

As always, consider backing up any files before making changes to them.

Changes to the security configuration file, deleting, renaming or moving it, will not affect currently running Documentum Scanner or Documentum Importer jobs. Changes will only take effect for jobs started afterwards

General Security Considerations

Since the Documentum security enhancements feature’s folder (<migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config) can contain sensitive information, it is advised to secure this folder using the file system and network security and access policies applicable to such information for the environment where migration-center is deployed in order to prevent unauthorized access.

At the same time the user account used for running the migration-center Jobserver service must be allowed access to this folder, as any Documentum Scanner or Documentum Importer will run in the context of the migration-center Jobserver, thus requiring access to the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and its contents.

Configuration

Configuration file

The configuration file controls all aspects of the Documentum security enhancement features. A valid configuration file needs to be provided by the system administrator configuring migration-center for the feature to work. As long as no valid configuration file exists, the feature will be disabled and will have no effect.

The configuration file must be named mc-dctm-security-config.xml and must be located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder.

Since the exact configuration depends on the customer’s environment, a preconfigured file cannot be delivered with migration-center. Instead a sample file illustrating the structure and parameters available for configuration is delivered and can be used as a starting point for creating a valid configuration. The sample file is named mc-dctm-security-config-sample.xml and is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder.

The sample configuration file can be copied and renamed to mc-dctm-security-config.xml to create the base for the actual configuration file, which will be edited by a system administrator.

General Information regarding XML

The configuration file is a human-readable XML file and can be edited with any text editor, or application providing XML editing capabilities. In order for the configuration file to be valid it will have to conform to XML standards in terms of syntax and encoding, something that should be considered when editing the file using a text editor. Migration-center will validate the file and refuse operation if the file structure or its contents are found to be invalid.

Special characters

Certain special characters need to be escaped according to the XML syntax in order to be entered and interpreted correctly as values. See below a list of these characters:

Encoding

The XML standard allows many different types of encodings to be used XML files. Encodings affect how international characters are interpreted and displayed. It is recommended to use the UTF-8 encoding, which can represent all international characters available today in the UTF character set.

Note that the (mandatory) XML header tag <?xml version="1.0" encoding="UTF-8"?> does not set the encoding, it merely specifies that the current file is supposed to be encoded using UTF-8; the actual encoding used is not directly visible to the user and depends on how the file is saved by the application used to save it after editing. Please consult your text editor’s documentation about saving files using specific encodings, such as UTF-8.

Configuration file parameters

This paragraph describes the structure of the Documentum Enhanced Security configuration file. It will list all available configuration elements, their allowed values and whether the respective parameter is mandatory to be set or not.

The <configuration> element

A <configuration> block defines an entire configuration block controlling access of one user to a repository. All other parameters which apply to that repository and user must be contained in the same <configuration> block.

Multiple repositories and users can be configured by creating multiple <configuration> blocks in the configuration file.

Attribute repository_name

repository_name is an attribute of a <configuration> element and defines the Documentum repository the current configuration block applies to.

Attribute migration_user

migration_user is an attribute of a <configuration> element and defines the migration user who will be granted access to the repository. This user will need to be configured in any Documentum job (scanner or importer) meant to access the repository defined in repository_name.

Attribute action_allowed

action_allowed is an attribute of a <configuration> element and defines what type of action a user is allowed to perform on the repository defined in repository_name. A user may be allowed to scan, import, or perform both actions on the repository.

Element effective_date

effective_date is a child element of configuration. effective_date sets a date until which the current configuration is allowed to be executed. If the set date is exceeded when a job is started, the job will not be permitted to execute and the user will be notified about the configuration having expired.

Element super_user_name

super_user_name is a child element of configuration. super_user_name defines a Documentum superuser which will be used to connect to the repository defined in repository_name. The migration-center user running the job will not need to know the Documentum superuser’s name or password, this will only be known and used internally by migration-center.

Element super_user_password

super_user_password is a child element of configuration. super_user_password defines the password for the Documentum superuser which will be used to connect to the repository defined in repository_name. The migration-center user running the job will not need to know the Documentum superuser’s name or password, this will only be known and used internally by migration-center.

Element migration_user_password

migration_user_password is a child element of configuration. migration_user_password defines the password for the migration user who will be granted access to the repository. This password will need to be configured in any Documentum job (scanner or importer) meant to access the repository defined in repository_name.

Element allowed_paths

allowed_paths is a child element of configuration. allowed_paths defines paths the migration user will be allowed to access in the specified Documentum repositories. The allowed_paths element is optional; if this element is omitted, access to all folders will be granted (whether access will actually succeed depends on the respective Documentum repositories permissions of course).

Element allowed_export_locations

allowed_export_locations is a child element of configuration. allowed_export_locations defines local paths or UNC paths (network shares) where the migration user will be allowed to export content from Documentum repositories. This (these) path(s) will need to be used in the Documentum scanner’s configuration as exportLocation (please see the Documentum Scanner User Guide for more information about the Documentum Scanner and the exportLocation parameter).

Element path

path is a child element of allowed_paths or allowed_export_location. It specifies individual Documentum paths (when used with allowed_paths) or local file system paths or UNC paths (network shares) when used with allowed_export_locations.

Supporting Tools

Password encryption tool

Passwords must be encrypted when saved to the configuration file. Encryption is performed using the supplied password encryption tool. The password encryption tool is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and can be executed using encrypt-tool.

Usage:

Enter password
Confirm password
Press [Encrypt] button (password and confirmed password must match in order to proceed)
Password is encrypted and displayed
Copy encrypted password
Paste encrypted password in migration_user_password or super_user_password element of configuration file

Configuration file verification tool

The configuration file must be a valid XML file. For convenience a verification tool is provided to check the validity of the configuration file against an XML Schema Definition file.

Migration-center will also validate the file internally and refuse operation if the file structure or its contents are found to be invalid.

The configuration file verification tool is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and can be executed using check-configuration.

Sample output for a valid configuration file

C:\Program Files (x86)\fme AG\migration-center Server Components 3.13\lib\mc-dctm-adaptor\security-config>check-configuration.cmd

******************************** * Configuration OK * ********************************

Sample output for an invalid (corrupted) configuration file

C:\Program Files (x86)\fme AG\migration-center Server Components 3.13\lib\mc-dctm-adaptor\security-config>check-configuration.cmd

******************************** * Configuration FAILED * ******************************** de.fme.mc.common.MCException: An error has occured when loading security configuration: org.xml.sax.SAXParseException; systemId: file:///C:/Program%20Files%20(x86)/fme%20AG/migration-center%20Server%20Components%203.13/lib/mc-dctm-adaptor/security-config/mc-dctm-security-config.xml; lineNumber: 5; columnNumber: 29; The element type "effective_date" must be terminated by the matching end-tag "</effective_date>".

Sample configuration file

A sample configuration file is provided in <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config/mc-dctm-security-config-sample.xml.

The contents of this file are also listed below for reference.

Glossary

Migration Set A migration set comprises a selection of objects (documents, folders) and set of rules for migrating these objects. A migration set represents the work unit of migration-center. Objects can be added or excluded based on various filtering criteria. Individual transformation rules, mapping lists and validations can be defined for a migration set. Transformation rules generate values for attributes, which are in turn validated by validation rules. Objects failing to pass either transformation or validation rules will be reported as errors, requiring the user to review and fix these errors before being allowed to import such objects.

Attribute A piece of metadata belonging to an object (e.g. name, author, creation date, etc.). Can also refer to the attribute’s value, depending on context.

Transformation Rules A set of rules used for editing, transforming and generating attribute values. A set of transformation rules is always unique to a migration set. A single transformation rules is comprised of one or several different steps, where each step calls exactly one transformation function. Transformation rules can be exported/imported to/from files or copied between migration sets containing the same type of objects.

Transformation Function Transformation functions compute attribute values for a transformation rule. Multiple transformation functions can be used in a single transformation rule.

Job Server The migration-center component listening to incoming job requests, and running jobs by executing the code behind the adapters referred by those jobs. Starting a scanner or importer which uses the Documentum adapter will send a request to the Jobserver set for that scanner, and tell that Jobserver to execute the specified job with its parameters and the corresponding adapter code.

Transformation The transformation process transforms a set of objects according to the set of rules to generate or extract.

Validation Validation checks the attribute values resulting from the Transformation step against the definitions of the object types these attributes are associated with. It checks to make sure the values meet basic properties such as data type, length, repeating or mandatory properties of the attributes they are associated with. Only if an object passes validation for every one of its attributes will it be allowed for import. Objects which do not pass validation are not permitted for import since they would fail anyway.

Mapping list A mapping list is a key-value pair used to match a value from the source data (the key) directly to the specified value.

Abbreviations

AIP - Archival Information Package

AIU - Archival Information Unit

BLOB - Binary large object

DB - Database

CSV - Comma separated values

CLOB - Character large object

DCTM - Documentum

DFC - Documentum Foundation Classes

DMS - Document Management System

DSS - Data Submission Session

GB - Gigabyte

GHz - Gigahertz

IA - InfoArchive

JDBC - Java database connectivity

JRE - Java Runtime Environment

JVM - Java Virtual Machine

KB - Kilobyte

MB - Megabyte

MS - Microsoft

MHz - Megahertz

RAM - Random Access Memory

regex - Regular expression

SIP - Submission Information Package

SPx - Service Pack x

SQL - Structured Query Language

XML - Extensible Markup Language

XSD - File that contains a XML Schema Definition

XSL - File that contains Extensible Stylesheet Language Transformation rules

Scanners

Alfresco Scanner

Introduction

The Alfresco Scanner allows extracting object such as document, folders and lists and saves this data to migration-center for further processing. The key features of Alfresco Scanner are:

Extract documents, folders, custom lists and list items
Extract content, metadata
Extract documents versions

Scanner is the term used in migration-center for an input adapter. Using a scanner such as the Alfresco Scanner to extract data that needs processing in migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

Scanners and importers work as jobs that can be run at any time, and can even be executed repeatedly. For every run a detailed history and log file are created. Multiple scanner and import jobs can be created or run at a time, each being defined by a unique name, a set of configuration parameters and a description (optional).

Install/Uninstall Alfresco Scanner

Install Alfresco Scanner

The Alfresco Scanner it’s not included in the standard installer of migration-center Server Components but it is delivered packaged as Alfresco Module Package (amp). This is because the Alfresco Scanner has to be installed within the Alfresco Repository Server. The following versions of Alfresco are supported (on Windows or Linux): 4.0, 4.1, 4.2, 5.2, 6.1.1, 6.2.0.

Java 1.8 is required for the installation of Alfresco Scanner.

For installing the other adapters you need during your migration process please install the Server Components as it is described in the . It is recommended to install the Server Components on another machine but it is also possible to install it on the Alfresco Server. In case you use the Alfresco Scanner in combination with an Importer running on another machine then the scanner should export the files on a network share that is accessible from the Server Components.

The first step of the installation is to copy mc-alfresco-adaptor-<version>.amp file in the “amps-folder” of the alfresco installation.

The last step is to finish the installation by installing the mc-alfresco-adaptor-<version>.amp file as it is described by the wiki guide of Alfresco under

Before doing this, please backup your original alfresco.war and share.war files to ensure that you can uninstall the migration-center Jobserver after successful migration. This is the only way at the moment as long the Module Management Tool of Alfresco does not support to remove a module from an existing WAR-file.

The Alfresco-Server should be stopped when applying the amp-files. Please notice that Alfresco provides files for installing the amp files, e.g.:

C:\Alfresco34\apply_amps.bat (Windows)

/opt/alfresco/commands/apply_amps.sh (Linux)

Due to a bug of the Alfresco installer under Windows, please be careful if the amp installer via apply_amps.sh works correctly! Under Alfresco 3.4, the file apply_amps.bat must be location in the alfresco location and not in the subfolder bin!

Uninstall Alfresco Scanner

The Alfresco Scanner can be uninstalled by following steps:

Stop the Alfresco Server.
Restore the original alfresco.war and share.war which have been backed up before Alfresco Scanner installation
Remove the file mc-alfresco-adaptor-<version>.amp from the “amps-folder”

Alfresco Scanner Properties

To create a new Alfresco Scanner, create a new scanner and select Alfresco from the Adapter Type drop-down. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type. Mandatory parameters are marked with an *.

The Properties of an existing scanner can be accessed after creating the scanner by double-clicking the scanner in the list, or selecting the Properties button/menu item from the toolbar/context menu. A description is always displayed at the bottom of the window for the selected parameter.

Multiple scanners can be created for scanning different locations, provided each scanner has a unique name.

Common scanner parameters

Alfresco Scanner parameters

The configuration parameters available for the Alfresco Scanner are described below:

History, Reports, Logs

A complete history is available for any Alfresco Scanner job from the respective items’ –History- window. It is accessible through the [History] button/menu entry on the toolbar/context menu. The -History- window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Alfresco Scanner can be found in the Alfresco log folder on the machine where the job was run, e.g. C:\Alfresco\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

CSV & Excel Scanner

Introduction

The CSV & Excel Scanner is one of the source adapters available in migration-center starting with version 3.9. It scans data from CSV and MS Excel files and creates corresponding objects in the migration-center database. If the scanned data also contains links to content files, the CSV & Excel scanner can link those files to the created objects as well.

Scanner is the term used in migration-center for an input adapter. Using a scanner module to read the data that needs processing into migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

CSV & Excel Scanners can be created, configured, started, and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Prerequisites

When scanning Excel files the “Write Attributes” permission is required otherwise the scanner will throw an “Access is denied” error.

Limitations

When attempting to scan Excel files with a large number of rows and/or columns the UI might freeze until the following error is thrown:

ERROR executing job: Error was: java.lang.OutOfMemoryError: GC overhead limit exceeded

This is a limitation of the Apache POI API, and the recommendation is to convert the excel file into a CSV file.

CSV & Excel Scanner Properties

To create a new CSV & Excel Scanner job click on the New Scanner button and select “CSV/Excel” from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the CSV & Excel Scanner parameters.

The Properties window of a scanner can be accessed by double-clicking the scanner in the list or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

CSV & Excel scanner parameters

Using the CSV and Excel Scanner

The CSV & Excel scanner supports two operation modes: normal and enhancement mode. In normal mode, you can scan objects from a CSV or Excel file and those objects are saved in the corresponding scan run as new, distinct objects in the MC database. In enhancement mode, you can scan data from a CSV or Excel file that is added to an existing scan run, i.e. it enhances an existing scan run with additional data.

Normal operation mode: Scanning CSV and Excel files

In normal operation mode, you can scan objects without content, objects with content, and objects with versions from a CSV or Excel file.

Scan objects without content

In order to scan objects without content, you just specify the path to the CSV or Excel file and the name of the column that contains the unique source ID of the objects. For example:

The screenshot below shows an excerpt of CSV file you would like to scan.

You would then enter the path to the CSV file in the “filePath” parameter and enter “id” as value in the “sourceIdColumn” parameter (because the column named “id” contains the unique IDs of the objects in the CSV file).

Scan objects with content

If your source file contained a column with a file path, as seen in the next screenshot, you can enter that column name (“profile_picture” in the example) in the “contentPathColumn” in order to scan that content file along with the metadata of the object.

Scan objects with versions

In order to scan objects with versions, your source file needs to contain two additional columns: one column that identifies all objects that belong to a specific version tree (“versionIdentifierColumn”) and another column that specifies the position of an object in the version tree (“versionLevelColumn”).

In the example below, the source file contains metadata of documents that consist of several versions each. Each document has a “title” and a “document_id”. Each version has a “version_id” and a “content_file”. The combination of “document_id” and “version_id” must be unique. Since the content file path is unique for each version in this example, you could use that column as “sourceIdColumn”.

A valid configuration to scan the example source file could look like this:

Enhancement operation mode: Adding data to existing scan runs

There are many document management applications, which reference data in third-party-systems. If you want, for example, to archive documents from such an application, it might be useful to store that referenced data also in the archive records, in order to have independent, self-contained archive records. To achieve this, you could (1) scan your document management system with the appropriate scanner, (2) export the data from the third-party-system into a CSV file, and (3) enhance the scan run from step (1) with the data from the CSV file using the CSV & Excel scanner in enhancement mode.

The source file for an enhancement mode scan needs a column that stores the source ID of the corresponding object, i.e. the object that will be enhanced with the data in the source file, in the specified scan run.

For example, the following table shows an excerpt of an existing scan run. The source ID of the objects is stored in the column “Id_in_source_system”. In this case, the ID is the full path of the files that were scanned by the scan run.

The source CSV or Excel file for the enhancement scan should contain all the data that you would like to add to the previously scanned objects and a column with the source ID of the corresponding object, as shown in the following table:

In order to run the scanner in enhancement mode, you need to enter a valid scan run ID in the parameter “enrichMetadataForScanRun”. You can find that ID in the -History- view of a scanner:

The “sourceIdColumn” parameter should contain the name of the column with the source ID. That would be “source_id” in the example above.

You can provide an optional prefix for the columns that will be added to the scan run in the “enrichMetadataPrefix” parameter. This is helpful in the case when your source file contains columns with names that already exist in the specified scan run.

Of course you can enhance a certain scan run several times with different source files.

Log files

A complete history for any CSV & Excel Scanner job is available from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with

The parameters the job was run with

Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Documentum Adapter can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Database Scanner

Introduction

The Database scanner can access SQL compliant databases and extract information via user specified SQL SELECT statements.

Note that the Database Scanner can extract content from a database or use the content files which are specified by the user during the transformation process via the mc_content_location system attribute.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

Database Scanners can be created, configured, started and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Exporting objects from a database

The Database scanner can access SQL compliant databases and extract information via user specified SQL SELECT statements. Since many content management systems rely on some type of SQL compliant database to manage their data, the Database scanner can also be used as a generic interface for extracting information from unsupported/obsolete/custom built content management systems. The types of database management systems supported are not limited to any particular vendor or brand, as access happens via JDBC, hence any SQL compliant database having a (compatible) JDBC adapter available can be accessed and queried.

Some common content management system features supported by migration-center Database scanner are metadata, including system metadata such as permission information, owner, creator, content path, etc, as well as version information, as long as these types of information can be obtained from the respective system’s SQL database. The information extracted by the database scanner is stored in the migration-center database and can be processed, transformed, validated and imported just like any other type of scanned information.

Note that the Database Scanner can extract content from a database stored in BLOB/CLOB fields. Alternatively, the content files corresponding to the objects can be specified by the user during the transformation process via the mc_content_location system attribute.

Depending on the way the content is stored, it may be necessary to extract the content to the filesystem first by other means before migration-center can process it. For the mc_content_location system attribute any of the available transformation functions can be used, so it is easy to generate a value resembling a path pointing to the location of the object’s content file, which a migration-center importer can use to import the content. A good practice would be to export content files to the filesystem using the object’s unique identifier as the filename, and then build the path information based on the known path and the objects unique identifier. This location would need to be accessible to the Job Server running the import which will migrate the content to the new target system.

Database Scanner Configuration

Configuring the JDBC driver

The types of database management systems supported are not limited to any particular vendors or brands, as access happens via JDBC, hence any SQL compliant database having a (compatible) JDBC adapter available can be accessed and queried.

Before using the database scanner, the appropriate JDBC driver needs to be installed and configured for the Job Server. To configure a JDBC driver all required Java jar files and Java library paths need to be added to the "... \migration-center Server Components \jdbc.conf" file. The file can be edited with any text editor and contains all information needed for configuration as each setting is described in the file itself.

The JDBC driver for Oracle 11g is delivered with the Database scanner already preconfigured and ready to use.

Scanner configuration

To create a new database Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters, “Database” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type, in this case the Database Scanner.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

Database scanner parameters

Configurations of queries in the “queryFile”

The “queryFile” is an xml file containing a series of SQL queries that will be run sequentially by the scanner for extracting information from the database.

At least two of the queries are mandatory to be configured in this file: one is for selecting the unique identifier of the rows/objects which need to be exported as, and the second query to extract the metadata for each row/object returned by the previous query.

If objects have multiple versions and the related information also needs to be extracted (i.e. a row/object is a version of another row/object), some additional queries are required.

The number of queries for extracting metadata for objects or versions is not limited. Multiple queries can be specified to gather and extract various information related to objects.

Example 1: a simple query file

The xml file is composed for <query> elements that contain the queries that will be run by scanner for extracting objects and metadata. The <query> element may have the following attributes (some of them mandatory):

Example 2: a more complex query file

The scanner will validate the queries configuration file against the following xml schema:

Delta migration

If the parameter “scanUpdates” is not checked, the scanner will only scan new objects or new versions of existing objects and it will ignore all objects that were previously scanned, i.e. that already exist in the MC database (based on their ID in source system).

If the parameter “scanUpdates” is checked, the scanner will scan new objects or new versions of existing objects and it will detect if it should scan existing objects as updates. If an object will be scanned as an update depends on several factors:

If there are no attributes specified in the “deltaFields” parameter, the scanner will scan every object that already exists in the MC database as an update.
If one or multiple attributes are specified in the “deltaFields” parameter, the scanner will scan an object that was previously scanned as an update only if a value of a delta field in the source database is different than the corresponding value in the MC database. If all values (of all fields defined in “deltaFields”) in the source database match the values in the MC database, then the object will not be scanned as an update, it will just be ignored.

The field names in the “deltaFields” are case sensitive so you should define them as they are scanned by the scanner.

Scanning BLOB/CLOB content

Since version 3.2.8 Update 2 of migration-center the database adapter supports extracting BLOB or CLOB content from databases. In order for this feature to work the main-content or version-content query type must be specified in the query file.

The structure of the queries is as follows:

The column for BLOB/CLOB is mandatory and it must have the alias BLOB_CONTENT or CLOB_CONTENT so the scanner is able to process the value correctly and extract the content.

The columns [nameOfFile] and [nameOfExtenstion] are optional. In case they are set, the scanner will use the values to build the name of the file where the content will be exported. The scanner avoids overwriting exiting files by adapting the file name to be unique in the export folder. Also, the characters that area not allowed in the file name will be replaced. Filename and extensions are also saved as source attributes so they can be used in the transformation engine. For avoiding name conflicts with the other attributes, the user can set aliases for these columns. If the columns [nameOfFile] and [nameOfExtenstion] are missing the scanner will use the value of the id in source system as a filename.

Multiple contents for a single object are allowed. If the query returns multiple rows the content returned by every row is exported. All the paths will be stored in the source attribute “mc_content_location” and the path from the last returned row is set as primary content in the column “content_location”.

Here is an example query file:

Scanning Excel files

Although it is still possible to use the database scanner to scan Excel files, we recommend to use our CSV & Excel scanner instead. The CSV & Excel scanner is faster and there are no tweaks necessary to get the JDBC-ODBC bridge working in Java later 1.7.

Prerequisites and Configuration

To scan excel files MS Office or at least Excel ODBC drivers need to be installed on the Job Server machine. To install the Excel ODBC driver one of the easiest ways is to download and install the 32bit version of the Microsoft Access Database Engine 2010 Redistributable that includes the necessary drivers and which can be downloaded from Microsoft’s website.

The connectionURL for should have the following format:

jdbc:odbc:DRIVER={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ=PATH-TO-EXCEL-FILE;ReadOnly=1

Note that all extensions (*.xls, *.xlsx, *.xlsm, *.xlsb) are required.

Here is an example XML Query File:

Solving Null values for some fields

When a column in the sheet contains mixed data (numbers and strings) the ODBC driver uses the first few rows to detect the column type. If the first values are numbers, the ODBC driver tries to handle all the subsequent values as numbers as well and therefore the values that are strings (not numbers) will be ignored, resulting in a “null” value being scanned by migration-center.

To avoid this problem, the parameter “IMEX=1” must be set in the metadata query:

Log files

A complete history is available for any Database Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Database Scanner can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Documentum Scanner

Introduction

The Documentum Scanner extracts objects such as files, folders, relations, etc. from a source Documentum repository and saves this data to migration-center for further processing. As a change in migration-center 3.2, the Documentum Scanner and Importer, are no longer tied to one another – data scanned with the Documentum Scanner can now be imported by any other importer, including of course the Documentum. Starting with version 3.2.9 objects derived from dm_sysobject are supported.

Scanner is the term used in migration-center for an input adapter. It is used to read the data that needs processing into migration-center and is the first step in a migration project, thus scan also refers to the process used to input data to migration-center in general.

A scanner works as a job that can be run at any time and can be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

Documentum Scanners can be created, configured, started and monitored through migration-center Client but the corresponding processes are executed by migration-center Job Server.

Supported Documentum Content Server versions

The Documentum Scanner currently supports Documentum Content Server versions 4i, 5.2.5 to 20.2, including service packs.

For accessing a Documentum repository Documentum Foundation Classes 5.3 or newer is required. Any combinations of DFC versions and Content Server versions supported by EMC Documentum are also supported by migration-center’s Documentum Scanner, but it is recommended to use the DFC version matching the version of the Content Server being scanned. The DFC must be installed and configured on every machine where migration-center Server Components is deployed.

For scanning a Documentum 4i or 5.2.5 source repository, DFC version 5.3 must be used since newer DFC versions do not support accessing older Documentum repositories properly. At the same time, migration-center does not support DFC versions older than 5.3, therefore DFC 5.3 is the only option in this case.

DFC (Documentum Foundation Classes) configuration

Starting from version 3.9 of migration-center additional configurations need to be made for the Documentum adapter to be able to locate Documentum Foundation Classes. This is done by modifying the dfc.conf file, located in the Job Server installation folder.

There are two settings inside the file that by default match the paths of a standard DFC install. One needs to have the path for the config folder of DFC and the other needs the path to the dctm.jar.

See below example:

wrapper.java.classpath.dfcConfig=C:/Documentum/config wrapper.java.classpath.dfcDctmJar=C:/Program Files/Documentum/dctm.jar

The dfcConfig parameter must point to the configuration folder. The dfcDctmJar parameter must point to the dctm.jar file!

Information on folder migration

When scanning Documentum documents, their folder paths are also scanned, and the folder structure can be automatically re-created by migration-center in the target system. This procedure will not keep any of the metadata attached to the folder objects, such as owners, permissions, specific object types, or any custom attributes. Depending on project requirements, it may be required to do a “folder-only migration” first, e.g. for migrating a complete folder structure including custom folder object types, permissions and other attributes first, and then populate this folder structure with documents afterwards. In order to execute a folder-only migration the following steps should be performed to configure the migration process accordingly:

Scanner: on the scanner’s |Parameters| tab check the exportFolderStructure option. Set scanFolderPaths (mandatory in that case) and excludeFolderPaths (if any, optional) leave the parameter documentTypes empty to scan only the folder structure without the documents; list document types as well if both folders and documents should be scanned; now only folders will be scanned without any documents they may contain. Note: Scanning folders is not possible via the dqlString option in the scanner.
Migration set: When creating a new migration set choose a <source type to target type>(folder) object type. Now only the scanner runs containing folder objects will be displayed on the |Filescan Selection| tab. Note that the number of objects contained in the displayed scanner runs now indicates folders and not documents, which is why the number on display (folders) can be different from the total number of objects processed by the scan (if it contains other types of objects besides folders, such as documents).

Folder migration is important. It is necessary to take the approach described above when migrating folder structures with complex folder objects containing custom object types, permissions, attributes, relations, etc. This information will be lost if exportFolderStructure is not selected during scan. If the exportFolderStructure parameter was not set during a scan, it is of course possible to re-run the scan again after setting this option, or to copy/create a new scanner and scan the missing folder information with that one.

Supported Documentum features

Versions

Versions (and branches) are supported by the Documentum Scanner, including custom version labels. The exportVersions parameter in the scanner’s configuration parameters determines if all versions (checked) or only current versions of documents (not checked, default setting) are scanned.

It is important to know that a consistency check of the version tree is performed by migration-center before scanning. A version tree containing invalid or missing references will not be exported at all and the operation will be reported as an error in the scanner’s log. It is not possible for migration-center to process or repair such a version structure because of the missing references.

For documents with versions, the version label extracted by the scanner from the Documentum attribute r_version_label can be changed by the means of the transformation rules during processing. The version structure (i.e. the ordering of the objects relative to their antecedents) cannot be changed using migration-center.

If objects are scanned with the exportVersions option checked, all versions must be imported as well since each object references its antecedent, going back to the very first version. Therefore, it is advised not to drop the versions of an object between the scan and the import processes since this will most likely generate inconsistencies and errors. If an object is intended to be migrated without versions (i.e. only the current version of the object needs to be migrated), then the affected objects should be scanned without enabling the exportVersions option.

Scanning large version trees

Processing a version tree is based on a recursive algorithm, which implies that all objects which are part of a version tree must be loaded into memory together. This can be problematic with very large version trees (several thousand versions). By default, the Documentum Scanner can load and process version trees up to around 2,000 versions in size. For even larger version trees to be processed the Java heap size for the Job Server must be increased according to the following steps:

Stop the Job Server
Open the wrapper.conf file located in the migration-center Server Components installation folder (by default it is %programfiles%\fme AG\migration-center Server Components <Version>)
Search for # Java Additional Parameters # Increase the value of this parameter it if your documentum scanner needs # to scan a large number of versions per document. Alocate 256k for every # 1000 versions/document.
Edit the line wrapper.java.additional.1=-Xss512k, incrementing the default 512k by 256k for every additional 1,000 versions mc should be able to process. E.g. for enabling processing of version trees containing up to 4,000 versions (2,000+1,000+1,000 versions), set the value to 1024k (512k+256k+256k)
Save the file
Start the Job Server

Primary content and renditions

The scanner exports the primary content of all documents unless the skipContent is set to true. The locations where the content was exported can be seen in the column Content location and in the source attribute mc_content_location,If a primary content has multiple pages, the column Content location stores the location where the page 0 was exported since mc_content_location stores all locations of all pages.

Renditions are supported by the Documentum Scanner. The “exportRenditions” parameter in the scanner’s configuration parameters determines if renditions are scanned. Renditions of an object will not count as individual objects, since they are different instances of content belonging to one and the same object. The scanner extracts rendition’s contents, format, page modifiers page numberand storage location used. This information is exposed to the user via migration-center source objects attributes starting with dctm_obj_rendition* in any documents migration set that has Documentum or FirstDoc as source system.

Documentum 4i does not have the page modifier attribute/feature for renditions, therefore such information will not be extracted from a Documentum 4i repository.

Relations

Relations are supported by the Documentum Scanner. The option named exportRelations in the scanner’s configuration determines if they are scanned and added to the migration-center database. Information about them cannot be altered using transformation. Migration-center will manage relations automatically if the appropriate options in the scanner and importer have been selected. They will always be connected to their parent object and can be viewed in migration-center by right-clicking on an object in any view of a migration set and selecting <View Relations> from the context menu. The resulting dialog will list all relations of the selected object with their associated metadata, such as relation name, child object, etc.

IMPORTANT: The children of the scanned relations are not scanned automatically if they are not in the scope of the scanner. The user must ensure the documents and folders that are children in the scanned relations are included in the scope of the scanner (they are linked under the scanned path or they are returned by dqlString).

migration-center’s Documentum Scanner supports relations between folders and/or documents only (i.e. “dm_folder” and “dm_document” objects, as well as their respective subtypes). “dm_subscription” type objects, for example, although supports relations from a technical point of view, will be ignored by the scanner because they are relations involving a “dm_user” object. Custom relation objects (i.e. relation-type objects which are subtypes of “dm_relation”) are also supported, including any custom attributes they may have. The restrictions mentioned above regarding the types of objects connected by a relation also apply to custom relation objects.

Export relations as renditions

As an alternative to scanning relations as they are in Documentum, the scanner offers the possibility to scan the child related documents as renditions of the parent document. For that, the parameter “exportRelationsAsRendtions” should be checked. This requires “scanRelations” to be checked as well. You can filter the relations that will be scanned as renditions by setting the relation names in the parameter “relationsAsRenditionNames”. If this is not set, all relations to documents will be processed as renditions.

Virtual Documents

Documentum Virtual Documents are supported by the Documentum Importer. The option named exportVirtualDocs in the configuration of the scanner determines if virtual documents are scanned and exported to migration-center.

There is a second parameter related to virtual documents, named maintainVirtualDocsIntegrity. This option will allow the scanner to include children of VDs which may be outside the scope of the scanner (paths to scan or dqlString) in order to maintain the integrity of the VD. If this parameter is disabled, any children in the VD that are out of scope (they are not linked under the scanned path or they are not returned by dqlString) will not be scanned and the VD may be incomplete.

The VD binding information (dmr_containment objects) are always scanned and attached to the root object of a VD regardless of the maintainVirtualDocsIntegrity option. In this way it is possible to scan any missing child objects later on and still be able to restore the correct VD structure based on the information stored with the root object.

The exportVDVersions options allows exporting only the latest version of the VD documents. This option applies only to virtual documents since the exportVersions option applies only to normal documents.

The exportVersions option needs to be checked for scanning Virtual Documents (i.e. if the exportVirtualDocuments option is checked) even if the virtual documents themselves do not have multiple versions, otherwise the virtual documents export might produce unexpected results. This is because the VD parents may still reference child objects that are not current versions of those respective objects. This is not an actual product limitation, but rather an issue caused by this particular combination of scanner options and Documentum’s VD features, which rely on information related to versioning.

The Snapshot feature of virtual documents is not supported by migration-center.

Audit Trails

The Documentum Scanner also supports audit trail entries for documents and folders. To enable scanning audit trails, the scanner parameter exportAuditTrail must be checked; in this case the audit trail entries of all documents and folders within the scope of the scan will be scanned as Documentum(audittrail) type objects, similarly to Documentum(document) or Documentum(folder) type objects.

There are some additional parameters used for fine tuning the selection and type of audit trail entries the scanner should consider:

auditTrailType – is the Documentum audit trail object type. By default, this is dm_audittrail but custom audit trail types (derived from dm_audittrail) are supported as well
auditTrailSelection – is used for narrowing down the selection of audit trail records since the number of audit trails can grow large especially in old systems, but not necessarily all audit trail entries may be relevant for a migration to a new system. This option accepts a DQL conformant WHERE clause as would be used in a SELECT statement. If this returns no results, all audit trail objects of scanned documents and folders will be scanned. Example 1: event_name in ('dm_save', 'dm_checkin') Example 2: event_name = 'dm_checkin' and time_stamp >= DATE('01.01.2012', 'DD.MM.YYYY')
auditTrailIgnoreAttributes – contains a comma separated list of dm_audittrail attributes the scanner should ignore. Again, this option can be used to eliminate audit trail information that is not needed for migration right from the scan.

Export audit trail as renditions

Because there are target systems that don’t allow importing audit trail objects, Documentum scanner allows exporting audit trail objects to PDF renditions of the scanned documents. Exporting audit trails objects as PDF renditions applies only to documents.

The following scanner parameters are used for applying this feature:

exportAuditTrailAsRendition – when checked the audit trail entries are written in a PDF files that are saved as renditions for the documents. This parameter can be checked only when exportAuditTrail is checked and skipContent is not checked. If not checked the audit trail entries are exported as Documentum(audittrail).
auditTrailPerVersionTree – this apply only when exportAuditTrailsAsRendition is checked. When it is checked one PDF is generated for all audit trail entries of all versions of the document. The audit trails entries related to the deleted versions are exported as well. The rendition is assigned to the latest version in the tree. When not checked, one PDF rendition is generated for every version in the tree. In this case the audit trails entries related to the deleted versions are not exported because those versions are not exported by the scanner since they don’t exist anymore in the repository.

Exporting audit trail per version tree may have a big impact on the scanner performance. That’s because audit trail entries for documents are queried by the attribute dm_audittrail.chronicle_id. The performance might be dramatically improved by adding an index in the underlying table DM_AUDITTRAIL_S for the column CHRONICLE_ID.

Aspects

Scanning aspects is supported with the latest update of the migration-center Documentum Scanner. Attributes resulting from aspects are scanned automatically for any document or folder type object within scope of the scan.

The notation used by the Documentum Scanner to identify attributes which result from aspects appended to the objects being scanned is the same as used by Documentum, namely <aspect_name>.<aspect_attribute>.

Any number of aspects per document/folder, as well as any number of attributes per aspect are supported.

After a scan has finished, attributes scanned from aspects are available for further processing just like any other source attribute and can be used normally in any transformation rule.

Aspects are supported only for document and folder type objects!

PDF Annotations

Starting with version 3.2.8 Update 2 of migration-center the possibility of scanning PDF annotations has been added to the Documentum Scanner. When activating “exportAnnotations” the scanner will scan the related “dm_note” objects together with DM_ANNOTATE relations. The “dm_note” objects are scanned as normal objects since the DM_ANNOTATE relations are exported as MC relation having the relation_type = “DctmAnnotationRelation”.

During delta migration, the scanner is able to identify the annotation changes and scan them accordingly.

Comments

Scanning the documents and folders related comments is possible and can be activated (default is deactivated) by changing the scanner parameter “exportComments” to true.

The best-known use case for documents and folders comments is within xCP (xCelerated Composition Platform) application, but it can also be used in custom WDK Documentum solutions.

The comment objects will be scanned as MC relation objects and can be seen in the MC client by opening the relations view of a scanned object. They will have the value of RELATION_TYPE as “CommentRelation”. All comment related attributes that have values will be scanned as attributes of these relations.

For performance reason, when a document has more versions, the comment relations will be attached only to the first document version that had comments (since all document versions share the same comments).

During delta migration, the scanner is able to identify comment changes, based on modifications of “i_vstamp” so it will rescan the corresponding document with all its comments (the first version document that had comments – see paragraph above) even if the document did not change.

To be able to scan the document’s comments it is necessary that the DFC used to have a valid Global Registry configured, because the adapter is using the “CommentManager” BOF service to read them.

Migrating Updates (“Update” or “Delta” Migration)

Objects that have changed in the source system since the last scan are scanned as update objects. Whether an object in a migration set is an update or not can be seen by checking the value of the Is_update column – if it’s 1, the current object is an update to a previously scanned object (the base object). There are some things to consider when working with the update migration feature:

Updated objects are detected based on the r_modify_date and i_vstamp attributes. If one of these attributes has changed, the object itself is considered to have changed and will be scanned and added as an update. Typically any action performed in Documentum changes at least one if not both of these attributes, offering a reliable way to detect whether an object has changed since the last scan or not; on the other hand, objects changed by third party code/applications without touching these attributes might not be detected by migration-center as having changed.
Objects deleted from the source after having been migrated are not detected and will not be deleted in the target system. This is by design (due to the added overhead, complexity and risk involved in deleting customer data).
Updates/changes to primary content, renditions, metadata, VD structures, and relations of objects will be detected and updated accordingly.

Documentum Scanner Properties

To create a new Documentum Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters “Documentum” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list or selecting the Properties button/menu item from the toolbar/context menu.

A detailed description is always displayed at the bottom of the window for the currently selected parameter.

Common scanner parameters

Documentum scanner parameters

When editing a field in the |Parameters| tab (like loggingLevel i.e.), a description/help/hint appears in the lower part of the window.

Log files

A complete history is available for any Documentum Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The list of all runs for the selected job together with additional information, such as the number of processed objects, the starting time, the ending time and the status are displayed in a grid format.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Documentum Scanner can be found in the chosen “logs” folder at the installation of the Server Components of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\Dctm-Scanner

the amount of information written to the log files depends on the setting specified in the loggingLevel start parameter for the respective job.

eRoom Scanner

Introduction

The eRoom Adapter consists of one module: the eRoom Scanner, which scans and exports objects from an eRoom site and inputs them into migration-center.

The eRoom scanner supports eRoom 7 or higher. It uses the eRoom XML Query Language (EXQL) to access eRoom data and methods through Simple Object Access Protocol (SOAP) requests, which is only available starting with eRoom 7.

Scanner is the term used in migration-center for an input adapter. Using the eRoom Scanner Module to read the data that needs processing into migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A Scanner is defined by a unique name, a set of configuration parameters and an optional description.

eRoom Scanners can be created, configured, started and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Exporting objects from eRoom using the eRoom Scanner

The eRoom scanner is designed to scan and export objects from eRoom. Currently the eRoom scanner support scanning items of type file (including versions); folder and room information is also appended to the files, thus making it possible to restore an existing hierarchy later.

The eRoom scanner can be targeted at a facility and/ or a room. From there it will start scanning for objects recursively, meaning that in case of a facility it will scan all rooms and from each room it will scan all objects contained in a certain room (folders or files).

The eRoom scanner adheres to migration-center standard behavior, meaning it runs as a job configured by the user via parameters, and saves all extracted objects/items’ content on disk and the corresponding metadata to the migration-center database in the process. After a scan has completed, the newly scanned objects are available for further processing in migration-center.

Known issues

The eRoom scanner does not work with OpenJDK 8, it only works with the Oracle version of Java SE 8.

eRoom Scanner Properties

To create a new eRoom Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters, “eRoom” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type, in this case the eRoom adapter.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list, or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

eRoom scanner parameters

Log files

A complete history is available for any eRoom Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the eRoom Adapter can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Importers

Previous versions

migration-center 3.14

Starting with version 3.13 Update 2, the SharePoint Online Importer does only support app-only principal authentication and no user name / password authentication. Please ensure that this will work for you before you upgrade your existing installation!

Features & Enhancements

Alfresco Scanner
- New feature: Scan only last "n" versions (#54289)
- New feature: Scan multiple sites (#54291)
D2 Importer
- Support for D2 20.2
Documentum Scanner / Importer
- Support for Documentum Server 20.2
OpenText Importer
- Support for OpenText Content Server 20.2
OpenText Scanner
- New feature: Scan shortcuts as distinct MC objects (#53710)
- New feature: Scan any folder subtype objects (#53711)
SharePoint Online Importer (Classic)
- New feature: Add user-agent traffic decoration (#52837)
SharePoint Online Importer (Batch)
- New feature: Importing role assignments (#54440)
Tools
- New tool to export document type definitions from Veeva available (#51539)

Fixes

OpenText Scanner
- NPE occurs using excludeFolderPaths (#54326)
- Wildcards do not work in the root folder (#54329)
- Set attributes are not scanned if the name is longer than 100 chars (#54589)
SharePoint Online Importer (Batch)
- SPO batch importer checks wrong content location (#54568)
- SPO batch importer throws error when importing large folder hierarchy (#54574)
- NPE for objects with NULL value in levelInVersionTree in SPO batch importer (#54637)
SharePoint Importer
- Wrong max length limit for parentFolder system attribute when importing to SP 2019 (#54592)

Known Issues & Limitations

General / logging
- The installer does not update the location of the Job Server's log files. So if you do not want to use the default location for log files, which is <Job Server Home>/logs, you need to manually update the log file location in the <Job Server Home>/lib/mc-core/logback.xml configuration file (#54732)
D2 Importer
- Support for importing to D2 4.1, 4.5, and 4.6 was removed
OpenText Scanner
- Using binders in the parameters "scanFolderPaths" and "excludeFolderPaths" is not supported (#54963)

migration-center 3.13 Update 2

Features & Enhancements

PowerShell tools:
- Support multiple scan run IDs when creating migration sets (#54263)
SharePoint Online importer:
- Support for app-only principal authentication with SharePoint permissions (#54277)
SharePoint Online Bulk importer:
- Add mc_content_location in migset rules (#54182)
- Support for app-only principal authentication with SharePoint permissions (#54277)
SharePoint Online scanner:
- Add scanLatestVersionOnly feature (#5615)
- Add includeFolders and excludeFolders parameters (#53706)

Fixes

OpenText importer: Wrong values for date attribute during physical objects import (#54255)
OpenText scanner:
- NPE occurs using excludeFolderPaths (#54326)
- Wildcards do not work in the root folder (#54329)
SharePoint (Online) importer: Automatically added content type not added to cache (#54420)
SharePoint scanner: Permissions & User fields contain ambiguous username instead of login name (#54240)

migration-center 3.13

Features & Enhancements

New SharePoint Online Batch Importer (#52665)
Add “removeDuplicate” transformation function (#53528)
Add support for Oracle 19c (#52930)
Add support for Oracle JDK 11 & OpenJDK 11 (#53313)
Add support for Oracle JDK 13 & OpenJDK 13 (#52492)
Add support IA 16.7 (#52910)
Added support for Alfresco 6.2.0 for Alfresco scanner and importer (#54108)
Alfresco scanner and importer now require java 1.8 or later
Documentum Scanner
- Scan complete document audit trail and save it as rendition to the current document version (#52846)
OpenText Importer
- Assign objects to Physical Object Box with a dedicate system rule (#52748)
SharePoint importer
- Support valid filename characters on SharePoint on-prem (#53304)
Documentum In-Place
- Implement move content feature in Documentum In-Place adapter (#53518)

Fixes

Fix typo in transformation function “ConverDateTimezones” (#53136)
“GetDateFromString” transformation function returns null in some cases (#53164)
Job server installation on Linux outdated in Installation Guide (#53714)
OpenText Importer became laggy during impersonation after a certain number of requests (#53180)
No error or warning when importing two renditions of the same type with OpenText Importer (#53124)
The underlying connection was closed: An unexpected error occurred on a send in SharePoint Importer (#53099)
Could not get SharePoint X-RequestDigest error message when space character in site collection name (#53190)
SharePoint Importer should log error immediately if required fields are missing (#53844)
HTTP 503 Server Unavailable Error when downloading large content files with SharePoint Online Scanner (#53673)
Fix “Member ID is not valid” in OpenText scanner (#53482)
Fix setting empty Vault Object reference in Veeva importer (#53573)

migration-center 3.12

Features and enhancements

OTCS Scanner
- Scanning Project objects as OTCS(container) with metadata (#52932)
- Save version information as source attributes (#52292)
- Support wildcards in folder paths (#52360)
- Show number of warnings in job run report summary (#52845)
- Scan user field with username instead of user id (#52847)
- Objects scanned from inside projects now contain full parentFolder path (#52960)
- Revised OpenText Adapter User Manuals (#52550)
Veeva Importer
- Re-authentication mechanism in case of session timeout (#52822)
- Pause and retry mechanism to handle burst and daily API limits (#52823)
- Support version binding when importing Veeva Vault binders (#52091)
- Delta Migration of objects in Veeva Importer (#53037)
Added support for Alfresco 6.1.1 in Alfresco Importer (#53008)
Added support for annotations in D2 Importer (#53018)
Documentum scanner now supports dqlExtendedString returning repeating values (#52587)
Added Support for SP 2019 in SP Importer (#52870)
Auto-Classification Module
- Functionality to split dataset into training and test data (#52660)
- Removing of library under GPL license (#52853)
- Classification Process Logging (#52855)

Fixes

Unresolved document file path in ScanRunExtractor (#52876)
Filesystem importer writes invalid XML characters to metadata XML file (#52909)
Missing dependency packages in AC module installer (#52935)
Error validating classification when subtypes not unique (#52945)

Known Issues

In Veeva Importer setting Binder version labels as major/minor does not work (#53160)
In Veeva Importer document updates with empty values for attributes that were not empty does not delete existing value (#53229)
In OTCS Scanner user fields that were deleted from a category are scanned as IDs (#52982)

migration-center 3.11

Features and enhancements

New SharePoint Online Scanner (#52409)
Made OTCS scanner more robust to errors (#52361)
Imptoved OTCS scanner job run log output (#52365)
FileSystem Importer now applies XSL transformation on whole unified metadata XML file (#52394)
Veeva Importer now supports importing Documentum documents as renditions of a specific type to a Veeva Vault document (#52090)

Fixes

Fixed bug in Filesystem Importer where attributes with no value were not being exported in XML metadata file (#52479)
Fixed OTCS Scanner not scanning documents under folder inside nested projects (#52809)
Fixed SPOnline importer bug when importing folders with a “%” in the name (#52857)
Fixed SPOnline importer not refreshing token when all objects of a large import fail (#52775)

Known Issues

SharePoint Online Scanner might receive timeout error from SharePoint Online when scanning libraries with more than 5000 documents (#52865)

migration-center 3.10

Features and enhancements

Added support for Oracle database version 18c (#51451)
Added support for migrating Veeva Vault Objects (#52136)
Added support for version 16.x for the OpenText scanner (#50340)
Added support for D2 versions 16.4 and 16.5 (#50972, #51530)
Added switch for lifecycle error handling in D2 Importer (#50991)
Updated the IBM Domino Scanner (#49173)
Improved database security by changing default password (#51112)
Improved performance of filesystem importer copy operation (#52320)
Enhanced the RepeatingToSingleValue function to not throw exception when length exceeds 4000 bytes (#51798)
Added missing file and folder name checks in the SP and SPO importer user guides (#52076)
Updated linux Jobserver with new YAJSW wrapper (#52347)
Box.net Importer, added the following features:
- Import custom metadata (#50312)
- Import tags (#50314)
- Import comments (#50315)
- Import tasks (#50316)
- Add collaborators to the imported documents (#50602)

Fixes

Fixed bug in DCTM adapters when calculating multi page content (#50442)
Fixed bug in SharePoint scanner when scanning SP 2013+ on Windows Server (#51088)
Fixed error messages bigger than 2000 characters not being saved in the database (#52357)
SharePoint Importer, fixed the following bugs:
- import taking a long time when content_location points to folder (#52115)
- Importer not starting if adfsBaseURL contains trailing slash (#51941)
- DEBUG messages getting logged despite log level ERROR in log4j (#52082)
- Added debug log messages for queue building missing in SP importer (#52084)
- Exception not caught when reverting import action (#52216)
- Importer hanging when encountering an exception in RefreshDigest method (#52238)
- Improved Retry handling (#52270)
- Error when importing into site with %20 in the name (#52272)
- autoCreateFolders always enabled (#52305)
- NullPointerException in file upload when SPOnline throttling occurs (#52325)

migration-center 3.9 – update 3

Features and enhancements

Some new features and enhancements in Domino Scanner
- New scanner parameter "excludedAttributeTypes"
- Default for scanner parameter "exportCompositeItems" now "0"
- Scanner parameter "selectionFormula" set to be editable by default
- 64bit support (based on IBM Domino); 32bit support remains unchanged (based on IBM Notes)
InfoArchive Importer: provide a better way for importing email attachments scanned from IBM Domino

Fixes

Domino Scanner: Extracted floating point numbers truncated/rounded to integer value
Domino Scanner: Attributes of type "NumberRange" were not exported
OpenText CS scanner: some logging improvements
OpenText CS scanner: Fix scanning subprojects
Veeva Vault Importer: Fix setting status_v for the root version
Veeva Vault Importer: Fix importing VD relations that have order_no with decimals.
Veeva Vault Importer: Fix a null pointer exception when logging some results
CSV/Excel Scanner: Fix setting the internal processing type of the scanner
Sharepoint Importer: Fix importing documents having # and % in the name
Sharepoint Importer: Fix importing folder with two consecutive dots (..) in the name
Sharepoint Importer: Fix token expiration after 24 hours.

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

Migration-center 3.9 update 2

Features and enhancements

Add support for importing attachments to Veeva Vault.

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

Migration-center 3.9 update 1

Features and enhancements

Added support for scanning nested compound documents in OTCS Scanner
Added support for scanning the Nickname attribute for folders and documents in OTCS Scanner
Added Proxy support for ADFS Authentication in SharePoint Importer
Added support for filtering Scanners and Importers in MC Client

Fixes

Fixed OTCS Scanner bug not scanning dates properly sometimes
Fixed OTCS Scanner "Member Type not valid" error message
Fixed OTCS Scanner nullPointerException when trying to get the owner name in specific cases
Fixed OTCS Scanner nullPointerException when trying to scan certain Workspaces
Fixed SharePoint Importer bug autoCreateFolders not working for values with leading slash
Fixed SharePoint failing authentication with invalid XML chars in use name or password

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

migration-center 3.9

Features and enhancements

Replaced Tanuki service wrapper with YAJSW (#51159)
Added support for Java 64-bit (#51163)
Added support for OpenJDK 8 (#51156)
New CSV / Excel Scanner (#50343)
Added Content Hash Calculation feature to OTCS scanner (#50736)
Added support for java web service in OTCS Scanner (#51389)
Added support for importing RIM submissions to Veeva Vault (#51208)
Added support for importing Documentum Virtual Documents as Binders in Veeva Vault (#51188)
Added support for setting references to existing master data objects in Veeva Vault (#51151)
Added support for ADFS authentication in SPO Importer (#51414)
Added new Transformation function for time-zone conversions (#51480)

Fixes

Fixed bug when pulling content for updates with the Database scanner (#51142)
Fixed error message in OTCS log (#51619)
Fixed SharePoint Online importer too short path length limitation (#51491)
Fixed SharePoint not resuming properly after pausing job for long time (#51492)
Changed behavior of setting the file extension for the SharePoint Importer (#51503)

migration-center 3.8

Features and enhancements

Documentum Scanner: Added option to scan only the latest version of a VD (#51019)
OTCS Importer: Added support for importing Virtual Documents as Compound Documents (#51008)
New transformation functions: GetDataFromSQL() and Length() (#50997, #50999)
Database Scanner: New Delta migration feature (#50477)
Documentum Importer: Extended waiting time for operations queue to finish (#50995)

Fixes

Fixed error in D2 Importer when property pages contained tabs with visibility conditions (#50987)
Removed unneeded jar file from D2 importer folder (#51073)

migration-center 3.7

Features and enhancements

The Jobserver now requires Java 8. Java 7 is not anymore supported.
New Veeva Vault importer
Extended OpenText importer to import physical items (#50282)
Changed the way of setting RM Classifications in OpenText importer (#50842).

Fixes

Delta migration issue after upgrading from 3.2.5 or older (#50571)

Known issues

Importing not allowed items in the physical item container is permitted (#50978)
RM classifications for physical objects are not removed during delta (#50979)
Physical objects properties of type date are not updated during delta migration (#50980)
Veeva importer may fail to start, if DFC is installed on the same machine (#50981)
No error reported when setting values to Inactive Fields in Veeva Importer (#50880)
No error reported when setting values to fields that are not in the Type / Subtype / Classification used in Veeva Importer (#50871)
No error reported when setting permissions not in the selected lifecycle in Veeva Importer (#50939)

migration-center 3.6

Features and enhancements

New OpenText In-Place Adapter (#49994)
Updated Tika library to version 1.17 for Filesystem scanner (#49258)
OTCS scanner is now able to scan Business Workspaces (#49994)
SharePoint Importer is now able to change the extension of files (#50142)
D2 Importer validates the Enabled, Visible and Mandatory conditions in a Property Page (#50235)
D2 Importer marks documents as Partially Imported when failing to apply a lifecycle action (#50274)

Fixes

Fixed issue with saving migsets when plsql_optimize_level was set to 3 (#49160)
Fixed issue with exporting BLOB content from a Firebird database (#49197)
Fixed error in Documentum In-Place adapter with linking into inexistent folderpath (#49217)
Fixed nullPointerException in SharePoint Importer when using proxy (#49232)
Fixed bug in Documentum audit trail rollback scripts (#49943)
Fixed nullPointerException in Alfresco Importer (#50027)
Fixed error when scanning documents in “Projects” with OTCS Scanner (#50130)
Fixed individual version creator not being scanned in OTCS scanner (#50273)
Fixed missing supported versions for Oracle and DFC in the Installation Guide (#50275)
Fixed DQL Validation for $repeatingvalue placeholder in D2 Importer (#50278)

Known issues

D2 Importer does not validate values for Editable ComboBox if they are not in the dictionary (#49115)
D2 Importer does not validate Enabled and Visibility properties if they have no condition attached to them (#50327)
Running IA Importer with values “16.x” for the InfoArchiveVersion parameter throws error (#50388)

migration-center 3.5

Features and enhancements

Alfresco scanner now supports Alfresco 5.2 (#49977)
Database scanner now supports multi-content Blob/Clob (#49260)
Added support for scanning and importing xCP comments for folders (#49245)
Added support for Documentum Content Server 16.4 (#49243)
Added support for InfoArchive 16.3 (#49244)
OTCS importer now supports folders of type email_folder types in OpenText Content Server (#49257)
OTCS scanner now supports scanning documents and emails from Email folders (#49995)
OTCS scanner now supports OTDS authentication (#50134)
Added checksum verification feature for OpenText Content Server Importer (#49250)
Added support for Oracle 12c Release 2 (#49259)

Fixes

Fixed bug in the Alfresco importer rollback scripts (#49251)
Fixed delta scan issue when only main object metadata changed (#50052)
Fixed issue with description length in OTCS importer (#50118)
Fixed error when setting subterm under non available term in SharePoint Online taxonomies (#49978)
Fixed job getting stuck when connection to SharePoint server is lost during import (#50026)
Fixed error when importing filename with apostrophe or plus sign in SharePoint (#49087, #49151)
Fixed error when importing SharePoint lists with certain settings (#49975)
Fixed rollback not being performed when a importing certain versions fails in SharePoint (#49974)
Fixed bug where document gets overwritten when old version moved to folder with a different document of the same name in SharePoint (#4661)

migration-center 3.4

Features and enhancements

Added comprehensive content hashing with different hashing algorithms and encodings for the following Scanners Documentum, SharePoint, Database and Filesystem (#11636)
Filesystem Importer: XML Metadata file now contains elements for all null and blank attributes (#11606)
InfoArchive Importer: migrating Documentum AuditTrails is now supported (#11610)
InfoArchive Importer: migrating Documentum Virtual Documents with children relations is now supported (#11683)
OpenText Importer: migrating opentext_email objects is now supported (#11670)
SharePoint Online Importer: added content integrity checking feature for imported Documents (#8296)
SharePoint Scanner: scanning entire version trees as a single migration-center object is now supported (#11641)
New system attributes are now available in transformation rules for all adapters (#11941)

Fixes

Fixed objects not being scanned when reading permissions failed in Filesystem Scanner (#11779)
Fixed table key lookup query being case sensitive in OpenText Importer (#11772)
Fixed length limitation for Table Key Lookup attributes in OpenText Importer (#11777)
Fixed setting attribute values from a different object to a category that has multiple rows when using multiple threads in OpenText Importer (#11797)
Improved logging for OpenText Importer (#11781)
Fixed issue where the initializer thread was still running in the background when a job was being stopped manually (#11739)
Fixed not being able to delete certain Scheduler runs that took more than 2 seconds to delete (#11889)

migration-center 3.3

Features and enhancements

Sharepoint Scanner:
- Added support for SharePoint 2016 (#10999)
- Added support for CAML queries (#11629)
- Added support for scanning SharePoint internal attributes (#11653)
Added support for scanning only current version in eRoom scanner (#11649)
Filesystem Scanner:
- mc_content_location and original_location are now available in transformation rules when using moveFilesToFolder (#11474)
- Added parameter for specifying the date format for extracting extended metadata with unusual format (#11673)
- Added possibility of scanning permissions (#11720)
- Upgraded to latest version of Tika Libraries (#11693)
Added support for InfoArchive 4.1 and 4.2 (#10325)
Added support for migrating SharePoint folders and List Items into InfoArchive (#11639)
OpenText importer:
- now support inheriting folder categories when importing folders (#11674)
- now support inheriting permissions (#11631)
- now supports setting null values for popup / dropdown attributes (#11550)

Fixes

Fixed applyD2RulesByOwner error message when using multiple threads in D2 (#11597)
Fixed some renditions being scanned twice in Documentum scanner (#11740)
Fixed wrong file path in log for Filesystem importer (#11469)
Fixed date values not being properly converted to XML format in some cases for InfoArchive importer (#11672)
Fixed contentless documents failing when ‘nocontent’ is specified in InfoArchive importer (#11656)
Fixed missing parent_object_id when scanning delta versions in OTCS Scanner (#11750)
Fixed importing documents that have date table key lookup attribute failing in OTCS Importer (#11576)
Fixed error when scanning uncommon managed metadata in SharePoint scanner (#11621)
Fixed only latest content exported for each version of a document in SharePoint scanner (#11657)
Fixed exportLocation parameter description when using local path in SharePoint scanner (#11568)
Fixed error when scanning documents with non-mandatory but empty taxonomy field in SharePoint scanner (#11652)
Fixed lookup attributes not being scanned at all in SharePoint scanner (#11655)
Improved OTCS scanner documentation (#11667)
Corrected version labels in Installation Guide (#11622)
Fixed Linux Jobserver classpath values (#11690)

migration-center 3.2.9

Features and enhancements

New Documentum In-Place Adapter (#11490)
Alfresco Importer now supports Alfresco 5.2 (#10772)
Documentum Importer - added support for CS 7.3 (#9927)
D2 Importer
- Added support for D2 version 4.7 (#10777)
- Changed D2 DQL and Taxonomy validation to be done by the D2 API (#10317)
- Added information regarding folder migrating options in the D2 Importer User Guide (#10754)
InfoArchive Importer
- Added support for multiple content files per AIU (#8877)
- Added support for multiple Object Types (#8877)
- Added support for Custom Attributes inside the eas_sip.xml file (#9405)
- Added support for calculating the PDI file checksum with SHA-256 and base64 encoding (#11418)
OTCS Scanner is now able to export rendition types (#11419)
OTCS Importer now supports Extended Attributes Model (#11424)
SharePoint Importer now does extra check to avoid overwriting existing content with the same List Item ID when trying to move content to different libraries (#11465)
Added warning regarding using multiple Jobservers with Sharepoint Importer (#11459)

Fixes

OpenText Importer: fixed java heap space error when importing large size renditions. (#11426)
OpenText Importer: fixed the authentication cookie refreshment for impersonated user (#11396)
OpenText Scanner: fixed out of memory error when scanning objects with size larger than 20MB (#11393)
OpenText Scanner: fixed out of memory error when scanning objects with size larger than 8GB (#11434)
Sharepoint Importer: fixed updates to intermediate version being applied to the latest version (#11458)
Sharepoint Importer: fixed a case that caused the importer to not finish (#10338)
D2 Importer: Fixed validation for certain DQLs containing placeholders (#10318)
D2 Importer: Fixed validation of certain taxonomies (#11549)
Fixed a jar conflict between OpenText Importer and Sharepoint Scanner (#11487)

Known issues

Multi-threaded import using applyD2RulesByOwner fails for some objects in D2 Importer
Remaining issues from the main release 3.2 to 3.2.8 Update 2 also apply to release 3.2.8 Update 3 unless noted otherwise.

migration-center 3.2.8 Update 3

Features and enhancements

Add support for migrating to DCM 6.7 (Documentum Compliance Manager).

Known issues

Remaining issues from the main release 3.2 to 3.2.8 Update 2 also apply to release 3.2.8 Update 3 unless noted otherwise.

migration-center 3.2.8 Update 2

Features and enhancements

Added support for xCP comments in Documentum Scanner and Documentum Importer (#10886)
Added support for “dm_note” objects in Documentum Scanner and Documentum Importer (#10887)
Added support for multi-page content in Documentum Scanner, Documentum Importer and D2 Importer (#11271)
Added support for extracting content from BLOB / CLOB column types in the Database Scanner (#10899)

Fixes

Fixed logging the version of the OpenText Importer in the run log (#11322)
Fixed setting the version description attribute in OpenText Importer (#11270)

Known issues

Documentum scanner/importer: The scanned/imported “dm_note” objects are not counted in the job run report (#11391)
DQLs that have enable(row_based) are not processed correctly in the D2 Importer (#11390)
Remaining issues from the main release 3.2 to 3.2.8 Update 1 also apply to release 3.2.8 Update 2 unless noted otherwise

migration-center 3.2.8 Update 1

Features and enhancements

New OpenText scanner: supports scanning documents, compound documents and folders from OpenText repositories version 9.7.1, 10.0 and 10.5. Specific OpenText features are supported like scanning categories, classification, permissions and shortcuts.

Fixes

Several bug fixes regarding validating DQL queries and Taxonomies when using property pages in D2 Importer.

migration-center 3.2.8

Features and enhancements

Added support for D2 LSS (Life Sciences Solution) version 4.1 (#9990)
Added multi-threading capabilities to the D2 importer (#10107)
Added support for Alfresco version 5.1 (#10354)
Added support for OpenText Content Server version 16.0 (#9937)
Added support for SharePoint 2016 (#9925)
Improved SharePoint Online Importer performance when applying internal attributes and taxonomies (#10314)
Added support for InfoArchive 4.0 (#9926)
Added support for Documentum ECS (Elastic Cloud Storage) (#9783)

Fixes

Alfresco Scanner: fixed scanning duplicates on delta scanning a new version for documents created directly in the Alfresco Share interface (#10013)
InfoArchvie Importer: fixed nullPointer exception when importing objects without content (#10746)
Filesystem Importer: fixed logging message when setting owner fails (#10326)
SharePoint Importer: fixed setting author and editor attributes in a multi-domain environment (#10171)
SharePoint Scanner: fixed job finishing successfully with incorrect credentials (#9939)
MC Client: fixed description for RepeatingToSingle transformation function (#10291)

Known Issues

D2 Importer:
- importing version updates with CURRENT not set to latest version fails (#10784)
- the CURRENT label is always set to the latest version when applying Lifecycle (#10686)
- importing branched versions with multiple and no r_version_label associated fails (#10460)
- D2 importer does not work with D2 4.5 or 4.1 when Jobserver runs on Java version 1.8 (#10384)

migration-center 3.2.7 Update 4

Features and enhancements

Improved Sharepoint and Sharepoint Online Importers performance when setting Taxonomies and internal attributes. (#10423)
Improved Sharepoint Importer Error Reporting in certain cases. (#10299)

Known Issues

Remaining issues from the main release 3.2 to 3.2.7 Update 3 also apply to release 3.2.7 Update 4 unless noted otherwise

migration-center 3.2.7 Update 3

Features and enhancements

Documentum Importer can attach now documents to a given lifecycle.

Fixes

FirstDoc importer: Fix a java.lang.ClassCastException error when importing to FDQM module
FirstDoc importer: Fix importing versions when there is an attribute count difference in the repeating group between create SDL and document definition SDL.

Known Issues

Remaining issues from the main release 3.2 to 3.2.7 Update 2 also apply to release 3.2.7 Update 3 unless noted otherwise.

migration-center 3.2.7 Update 2

Features and enhancements

D2 Importer now supports only D2 version 4.6. The older D2 versions are only supported by the previous versions of migration-center
OpenText Importer now supports creating OpenText Compound Documents
OpenText Importer now supports importing renditions
IBM Domino Scanner:
- Large attribute values can now be split into chunks of maximum 4000 bytes
- Added custom attributes: "attachmentItemName" - name of item that an attachments was attached to in the original Domino document, "attachmentName" - name of the attachment in the original document and "$dominoNoteID$" - NoteID of the document in the database that it originates from.

Fixes

Documentum Scanner: Scanning version trees having versions of multiple object types
OpenText Importer: Error messages concerting a single attribute do not have the attribute name
OpenText Importer: Categories not being set if no associations are made for it
Sharepoint Scanner: Documents with paths longer than 260 characters failed to be scanned

Known Issues

Running a SP scanner with incorrect credentials on a Sharepoint with anonymous authentication enabled, does not show errors.
Remaining issues from the main release 3.2 to 3.2.7 Update 1 also apply to release 3.2.7 Update 2 unless noted otherwise.

migration-center 3.2.7 Update 1

Features and enhancements

Alfrecso scanner now supports scanning version check-in comments
Alfresco importer now supports importing version check-in comments
Documentum adapter supports scanning/importing version trees with different object types in them
InfoArchive importer officially supported on the Linux jobserver
InfoArchive importer now supports ingestion via InfoArchive Webservices
OpenText Importer now supports inheriting categories from folders
OpenText Importer now supports setting values for attributes grouped within a Set attribute

Fixes

InfoArchive Importer:
- fixed text inside SIP xml files generated without indent
- fixed ZIP files not being rolled back when there is not enough memory on disk
OpenText Importer:
- fixed impersonate user not working as expected
- fixed importer not working when DFC not installed

Known Issues

DocumentumNCC scanner: null pointer exception when scanning documents without content
InfoArchive importer: SIPs are imported successfully but marked as failed when moveFilesToFolder does not have write permissions
OpenText Importer: Error messages concerting a single attribute do not have the attribute name
OpenText Importer: Categories are not set if no associations are made for it
Sharepoint Scanner: Documents with paths longer than 260 characters fail to be scanned
Remaining issues from the main release 3.2 to 3.2.7 also apply to release 3.2.7 Update 1 unless noted otherwise.

migration-center 3.2.7

Features and enhancements

New OpenText importer: supports importing documents and folders to OpenText repositories version 10.0 and 10.5. Specific OpenText features are supported like setting categories, classification, permissions, shortcuts to the imported documents and folders.
New SharePoint Scanner: supports the scan of SharePoint 2007, 2010 and 2013. Supports scanning documents, folders, listitems, lists and document libraries. Replaces the old scanner which has been retired.
New SharePoint Importer: supports SharePoint 2010 and 2013 and uses REST API. Support import of documents, folders, listitems, lists and document libraries.
Retired the previous SharePoint Importer as SharePoint Legacy: new installations will not have it. Updated installations will have it under the name of SharePoint Legacy.
SharePoint Online Importer supports now documents, folders, listitems, lists and document libraries. Relations can be imported now as attachments for listitems.
D2 importer supports now creating folders based on the paths provided to documents.
Alfresco importer supports now the Alfresco 5.0.
Alfresco scanner and importer: categories and tags are exported and imported using the values displayed in the interface instead of using internal ids.

Fixes

Alfresco scanner and importer:
- Fixed a NullPointerException when scanning documents with aspects from Alfresco.
- Fixed the value of the content location attribute when scanning documents with skipping content.
- The content location is now available in the attribute “mc_content_location” that is accessible to the transformation engine.
- Fixed importing a new document version when the first link was modified compared with the previous version.
- Improve the message description of several errors that may happen when importing updates and versions.
Documentum scanner:
- Fixed a query error when scanning a repository having DB2 as database.
- Fixed scanning version tree having multiple types.
- Fixed the content location for the documents without format.
Box importer:
- Fixed setting “content_created_at” and “content_modified_at” to the imported files.
Filesystem scanner:
- Fixed out of memory when scanning more than 1 million of files
- Fixed the case when scanning multiple folders like: <some folder path>\FName|<some folder path>\FName with Space. In the previous version the second folder was ignored without any error message.
- Report properly the invalid paths when scanning multiple folders.
- Fixed a possible performance issue when scanning extended metadata.
Filesystem importer:
- Fixed a NullPointerException when setting “modified_date” without setting the “creation_date”
- Fixed the case when the target file was overwritten by the metadata file when they were having the same name.
SharePoint Online importer:
- Fixed updates imported as new versions.
- Fixed import fails only with major version enabled.
- Fixed null pointer exception when setting empty “Group or Person” or “Hyperlink” column.
- Fixed items created for failed jobs are now rolled-back where possible.
- Fixed running with invalid credentials does not set the error status.
- Fixed proper log messages are set now for multiple attribute setting errors.
Eroom scanner
- The “er:Path” attribute does not contains the file name anymore.
Database
- Allow installing migration-center on a non-CDB Oracle 12c instance.
- Fixed the throwing two errors when upgrading from version 3.2.5 or older. This was the case when filesystem importer was not licensed.
Client
- Fixed installing the client on Window 8 and Windows 2012 server. No manual registry configuration is required.

Known issues

OpenText Importer: The modified date of created folders (set by the importer) is overwritten by the content server when new documents are added to the folder.
Remaining issues from the main release 3.2 to 3.2.6 also apply to release 3.2.7 unless noted otherwise.
SharePoint Legacy Importer: The Logging system on the SharePoint side is not reusable after a forced restart. The SharePoint Importer uses much more memory then the previous version. Document sets cannot be created in a library with versioning disabled.
SharePoint and SharePoint Online Importer: Null reference not handled in case of the big document import. The author does not get set unless editor attribute is set as well.

migration-center 3.2.6

Features

Added support for Oracle Database 12c.
Added support for Documentum 7.2.
Added support for Windows Server 2012, Windows 8 / 8.1.
New InfoArchive Importer: supports the import of data generated by any scanner to InfoArchive enterprise archiving platform.
New Alfresco Scanner: supports scanning documents, folders, custom lists and list items from an Alfresco repository.
New Microsoft Exchange scanner: supports scanning emails from different accounts using an account that has delegate access to the other ones.
New Exchange Removal adapter: supports deleting emails in the source system that have been imported from Microsoft Exchange.
Documentum Importer supports multithreaded processing for improved performance.
Filesystem Importer supports setting the creator name, creation date and modify date to the imported file.
Enhance the flexibility of generating metadata files in Filesystem Importer.
Support of Jobserver components installation on Linux. Note that not all adapters are available on Linux
The Oracle package SYS.SQLJUTL is not needed anymore by migration-center.

Fixes

Sharepoint importer:
- Fixed updating minor versions of documents.
- Fixed updating documents when “ForceCheckout” flag is set to true in the library.
- Fixed dummy versions not deleted after large file failed import.
- Fixed importer trying to delete inexistent dummy versions.
- Fixed the reporting of some incorrect error messages.
Sharepoint Online importer:
- Fixed setting the attribute “Title” for contact items.
- Fixed the reporting of some incorrect error messages.
Filesystem Importer:
- Fix a null pointer exception when setting invalid path to “unifiedMetadataPath”.
- Fix moving renditions when “moveFiles” is enabled in the importer.

Limitations

migration-center 3.2.6 requires now Java Runtime Environment 1.7.0_09 or later. It will not work with older java versions.
In Documentum 7.x the max allowed length for owner_name, r_modifier, r_creator_name was changed from 32 to 255 char. In MC the default provided types (dm_document, dm_folder, dm_audit_trail) still have the old limit of 32 char.
Filesystem Importer does not allow more than 255 characters for the “content_target_file_path” system rule.

Known Issues

SharePoint Online Importer: When importing large files into a SharePoint Online document library the import might fail due to a 30 minutes timeout defined by Microsoft. (Bug# 8043)
SharePoint Online Importer: Documents that have the attribute is update set to true are not overwritten during import, but a new version is created on import. Document Sets cannot be created in a library with versioning disabled. (Bug #7904)
Sharepoint Online Importer: The importer does not set the “Author” field of a document, if only the “Author” attribute has a rule associated to it. The user must associate rules to both the “Author” and “Editor” attribute for them to be set. (Bug #8629)
Sharepoint on Premise Importer: Importing documents without any specified version label results in an error. (Bug #8186)
Documentum Importer: Importing multiple updates of the same document that is part of a version tree containing branches may fail if using multiple threads. That is an extreme case since in the process of delta migration the object updates are migrated incrementally. (Bug #8175)
InfoArchive Importer: When the “targetDirectory” runs out of disk space during the zip creation, the importer fails as expected, but the zip file is not rolled back (deleted). (Bug #8758)
Alfresco Importer: Certain errors that may occur during import are not descriptive enough. (Bug #8715)
Alfresco Importer: Updating a document will result in an error if the first link of the base object needs to be removed by the update. (Bug #8717)
Alfresco Scanner: Running an Alfresco Scanner with the parameter “exportContent” unchecked results in invalid values set for the “content_location” attribute. (Bug# 8603)
Alfresco Scanner: Running two scanners in parallel on the same location can result in duplicate objects being created. (Bug #8365)
Filesystem Scanner: Folders will be ignored if multiple folder paths that begin with the same letters are used as a multivalue for the scanFolderPath (Bug #8601)
Remaining issues from the main release 3.2 to 3.2.5 also apply to release 3.2.6 unless noted otherwise.

migration-center 3.2.5 Update 1

Features

SharePoint on Premise supports: new object types Sites, Document Sets; custom sites for individual imported objects; non-consecutive versioning for documents.

Fixes

SharePoint Online Importer: internal properties like ‘Modified’, ‘Modified by’, ‘Created’ and ‘Created by’ are properly updating now.
SharePoint Online Importer: removed limitations of the number of items supported during import in the target library.
SharePoint Online Importer: internal name of the taxonomy terms are correctly used.

Known issues

IBM Domino Scanner:

The scanner requires that the temporary directory for the user running MC Job Server Service exists and that the user can write to this directory. If the directory does either not exist or the user does not have write permission to the directory, the creation of temporary files during document and attachment extraction will fail. The logfile will show error messages like

„INFO | jvm 1 | 2014/10/02 12:06:26 | 12:06:26,850 ERROR [Job 1351] com.think_e_solutions.application.documentdirectory… - java.io.IOException: The system cannot find the path specified“.

To work around this issue, make sure the temporary folder exists and the user has write permission for this folder. If the MC Job Server is started manually as a normal user then the “Temp” folder should be C:\Users\Username\AppData\Local\Temp. Therefore, if the MC Job Server is run as a service by the Local System account, the folder is one of the following:

For the 32bits version of Windows: C:\Windows\System32\config\systemprofile\AppData\Local\Temp

For the 64bits version of Windows:

C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Temp

migration-center 3.2.5

Features

New IBM Domino Scanner: supports scanning emails and documents from IBM Domino/Notes
New Sharepoint Online Importer: supports importing Lists, List Items, Documents and folders in Sharepoint Online
Eroom scanner: Multiple erooms can be scanned now with a single scanner configuration

Fixes

Scheduler: Fix a special case when the scheduler run hangs in status “Scanner running”
Documentum Importer: Fix a “NullPointerException” when importing VDRelation updates

Known issues

IBM Domino Scanner:

The scanner requires that the temporary directory for the user running MC Job Server Service exists and that the user can write to this directory. If the directory does either not exist or the user does not have write permission to the directory, the creation of temporary files during document and attachment extraction will fail. The logfile will show error messages like

„INFO | jvm 1 | 2014/10/02 12:06:26 | 12:06:26,850 ERROR [Job 1351] com.think_e_solutions.application.documentdirectory… - java.io.IOException: The system cannot find the path specified“.

For the 32bits version of Windows: C:\Windows\System32\config\systemprofile\AppData\Local\Temp

For the 64bits version of Windows:

C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Temp

migration-center 3.2.4 Update 4

Features

Outlook scanner:
- Extract the email address of the following recipients: To, CC and BCC
- Extract the number of attachments in the source attribute: AttachmentsCount

migration-center 3.2.4 Update 3

Features

Documentum Scanner and Importer: now support the migration of aspects attached to documents and/or folders. Documentum CS and DFC 6.x or higher required.
New Documentum No Content Copy (“NCC”) Scanner and Importer: these customized variants of the regular Documentum adapters allow for fast metadata-only migration between Documentum, while preserving the references to content files. You can move or copy the content files separately at any point during or after the migration.

Fixes

Documentation: fixed several errata in SharePoint Scanner and Importer User Guides in filenames, paths and URLs used throughout the respective documents
eRoom Scanner: fixed the mc_content_location system rule not appearing for documents scanned from eRoom, thus preventing changes to the content file’s location to be made for import if needed
migration-center Client: fixed issue with import of mapping lists containing keys differing only in character case. Now these are imported and treated as distinct values correctly.
Jobserver: fixed issue with establishing SSL connections from Jobserver with certain combination of adapters
Database Scanner: fixed Unexpected error occurred during content integrity checking message when migrating data scanned with the Database Scanner to target systems with importers supporting the content integrity check feature, which the Database Scanner itself does not support.
File System Scanner: Extended metadata containing values longer than 4000 bytes no longer errors out. Instead, the values are truncated to the max 4000 bytes supported by mc.
File System Scanner: fixed extraction of creation date and owner for files with paths longer than 260 characters
Scheduler: fixed Oracle error when interval for a scheduler was set to “Month”

Changes

SharePoint Importer: The ID that is stored after a successful import of an object in id_in_target_system changed from target list specific ID to global unique ID (GUID). This change maintains compatibility with data scanned using mc 3.2.4 U1 and U2 that still uses the list specific ID

Known Issues

MC Client: Importing a mapping list containing identical keys will silently skip the existing key. Solution: verify mapping lists before importing them to migration-center in order to remove or rename identical keys, mapping lists must always have unique keys.

migration-center 3.2.4 Update 2

Features

File System Scanner: added new option to move successfully scanned files to a local path or UNC network path specified by the user. See the migration-center File System Scanner User Guide for more information on the new moveFilesToFolder parameter.
Installer: added option to specify location of log files during installation of migration-center Server Components. Log files were stored in the installation folder’s logs subfolder by default, now any valid local or UNC network path can be specified for storing the log files generated by migration-center during runtime. See the migration-center Installation Guide for more information about installing migration-center Server Components.

Fixes

Database: removed dependency on Oracle SYS.UTL_TCP and SYS.UTL_SMTP packages for performing manual/regular migration activities. Note the scheduler still requires these packages in order to function, but it is now possible to use all other features of migration-center except the scheduler without the need to have access granted to these Oracle system packages.

migration-center 3.2.4 Update 1

Features

SharePoint Importer: Support for importing folder objects (content type Folder and custom subtypes inheriting from Folder)
SharePoint Importer: Support for creating link objects (content type Link to a Document and custom subtypes inheriting from Link to a Document) (see the SharePoint Importer User Guide for more information)
Documentum Scanner: added “exportLatestVersions” to control the number of latest versions to scan (see the Documentum Scanner User Guide for more information)

Fixes

SharePoint Scanner: fixed an issue causing a "Object reference not set to an instance of an object." error during scan with certain configurations of Date and Time and Yes/No columns

migration-center 3.2.4

Features

New Documentum D2 Importer*: currently supports D2 4.1, including D2 specific features such as autonaming, autolinking, rule based security, applying rules based on the document’s owner. etc.
New SharePoint Scanner*: currently supports SharePoint versions 2007/2010/2013, extraction of Document Libraries, exclusion of selected Content Types or file types, checksum generation for verifying content integrity after import, etc.
New Documentum DCM Importer*: currently supports DCM 5.3, including DCM specific features such as document classes, creating controlled documents, applying autonaming rules, setting specific lifecycle states and associated attributes for imported documents. etc.
Updated SharePoint Importer: now implemented as a SharePoint Solution (just like the new SharePoint Scanner), simplifying deployment and improving performance, reliability and error resilience
Updated SharePoint Importer with content integrity check feature (comparison of checksums computed during scan and import)
Updated Documentum Importer: now supports importing to Documentum 7 repositories
Updated box Importer: now works with the box API version 2.0 for improved performance and reliability; verification of content integrity can now be performed during import based on checksums computed by the importer and box respectively
Updated Documentum Scanner: now supports scanning Documentum 4i repositories (via DFC 5.3)
New parameter for Documentum Importer to toggle whether errors affecting individual renditions should be treated as warnings or errors affecting the entire document, thus preventing it from being imported (set to ignore errors and treat as warnings by default for compatibility with previous versions)
New parameter for Filesystem Scanner to ignore hidden files (false by default)
Documentation: added new document “migration-center Database Administrator’s Guide” detailing database installation requirements (privileges, packages, etc.) and procedures for deploying mc in environments where the regular database setup cannot be run, and the database must be prepared manually by a DBA for installing the mc schema

*New adapters must be purchased separately.

Fixes

Scheduler: fixed an issue where a 24-hour time set was converted to 12-hour time
Scheduler: fixed an issue where the hourly interval was not taken into account if the scheduler was configured to run on a minutely or hourly basis
Documentum Scanner: fixed issue where the “scanNullAttributes” parameter’s functionality was reversed with regard to the parameter’s actual setting in the UI
Documentum/D2/DCM Importer: fixed an issue with the importer not properly handling certain null attribute values (in case the null value of an attribute resulted from null values returned by the If transformation function)
Filesystem Importer: fixed an issue causing errors when importing renditions for a selection of documents if some objects had renditions and others didn’t
FirstDoc Importer: fixed an issue with setting version labels in a particular case (Superseded 1.0 – Obsolete 1.1)
FirstDoc Importer: fixed an issue trying to delete already deleted relations
FirstDoc Importer: fixed objects no being rolled back in case errors occur during saving content
FirstDoc Importer: fixed objects not being rolled back in case errors occur during updating of ACLs and links
Documentation: minor corrections and additions in various documents

Changes

SharePoint Importer now integrates with SharePoint as a SharePoint Solution instead of the separate service component it used up to mc 3.2.3. This changes the requirements and deployment procedures for this adapter. These are described in the mc 3.2.4 SharePoint Importer User Guide
Database: Privilege “CREATE ANY JOB” is no longer required, “CREATE JOB” is now sufficient

Limitations

SharePoint Scanner only scans complete Document Library items and cannot scan individual folders currently
SharePoint Importer only supports versioning using contiguous numbering

Known Issues

Scheduler: setting a time interval (the hours during which the scheduler is allowed to run) for a scheduler configured to run on a minutely or hourly basis will not be reflected correctly in the “Next run date” displayed. The scheduler will run as configured, the issue affects only the display of next run date.
Remaining issues from the main release 3.2 to 3.2.3 also apply to release 3.2.4 unless noted otherwise.

migration-center 3.2.3

Features

New Microsoft Outlook Scanner – scan specified folders of a user’s mailbox from Outlook 2007-2010, including messages and their properties, choose to exclude particular subfolders or properties, etc. See the migration-center 3.2.3 – Outlook Scanner User Guide document for more information about this new adapter. As with all adapters, the Microsoft Outlook Scanner is available for purchase separately.
FirstDoc Importer now officially supports FirstDoc versions 6.3 and 6.4 (R&D)
Documentum Importer now supports setting rendition page modifiers and rendition file storage locations through transformation rules.

Fixes

eRoom Scanner: fixed an issue leading to out of memory errors when scanning large files (several hundred megabytes each)
eRoom Scanner: fixed an issue with the skipContent parameter not working properly in some cases
Filesystem Scanner: fixed an issue where the scanner could fail to scan files equal to or larger than 4GB in size
FirstDoc Importer: fixed a memory leak issue related to session management

Changes

Documentum Scanner & Importer: Changed handling of rendition page modifiers (now exposed as a dedicated system attribute that can be set using transformation rules). See the “migration-center 3.2.3 – Documentum Importer User Guide” document for more information about this feature and how it may affects renditions scanned using older versions of the Documentum Scanner.
eRoom Scanner: changed naming of files extracted by the scanner to the mc storage location. This has been done in order to line up the eRoom Scanner’s behavior with the rest of mc’s scanner’s, which all use an object ID rather than actual filenames for the extracted content. This should not affect the regular workflow of a migration from eRoom, as all changes are handled internally by mc and are not exposed to the user (nor does or did the user need to interact with said files at any time).

Limitations

The FirstDoc Importer cannot set Documentum system attributes such as creation and modify dates, creator and modifier user names, the i_is_deleted attribute, and so on. Setting these attributes is only supported using the Documentum Importer.

Known Issues

If a source attribute has the same name as one of migration-center’s internally used columns, the Client will display the value of the migration-center internal attribute for the source attribute as well. This occurs in Source Objects view. The issue also persists in the data exported to a CSV file if the CSV export is performed from Source Objects view. Other views or the View Attributes window will display the correct value of the respective source attribute.
Remaining issues for the main release 3.2 also apply to release 3.2.3 unless noted otherwise.

migration-center 3.2.2

Features

Documentum Content Validation – checks content integrity of documents migrated to a Documentum repository by verifying an MD5 checksum of content files before and after the import. From a Documentum source both primary content and renditions can be checked, for other sources only the primary content is supported. The feature is available as an option in the Documentum Importer. See the migration-center 3.2.2 – Documentum Importer User Guide document for more information about this feature.
Documentum Audit Trails – the Documentum adapters now support the migration of audit trails. The Documentum Scanner can scan the audit trails of any folders and documents within the scanner’s scope. Audit trail objects can then be added to an Audit Trail migset and processed just like documents and folders. Finally, audit trails can be imported using the Documentum Importer. The new options available in the Documentum adapters for controlling and configuring the migration of audit trails are described in the migration-center 3.2.2 – Documentum Scanner User Guide and migration-center 3.2.2 – Documentum Importer User Guide respectively.
The Documentum Scanner has a new option, allowing an additional DQL statement to be executed for every object within the scope of the scan. This allows additional information to be collected for each object from sources other than the document’s properties. Other database tables (registered tables) for example can be queried using this option. See the migration-center 3.2.2 – Documentum Scanner User Guide for more information.
The Filesystem Scanner can now build versions from additional attributes (as long as these attributes can supply the required information. Two attributes per object are required for the feature to work, which can be named arbitrarily and can be defined in the fme metadata files associated with each content file. The attributes to be used as version identifier and version number by the scanner can be specified through additional options available in the Filesystem Scanner. See the migration-center 3.2.2 – Filesystem Scanner User Guide for more information.
The SharePoint Importer can now update previously imported objects (if the objects had been modified in the source and scanned as updates)

Fixes

Documentum Importer: Fixed an issue where importing an update to a Microsoft Word 8.0-2003 document would set the wrong format for the updated document (a_content_type was set to “doc” instead of “msw8” previously)
Documentum Importer: Fixed an issue with importing Filesystem objects to Documentum when user defined version labels were set in transformation rules. Depending on how the labels were set, documents could end up all having CURRENT labels set, or having no CURRENT labels at all. Now the most recently imported version is always set as CURRENT, regardless of the CURRENT label set in Transformation Rules.
Transformation Engine: Fixed an issue where Validating a migration set would validate objects that have been already validated again, which didn’t make sense.
Scheduler: fixed issues where a Scheduler would stop working after several hundred runs and required to be stopped and restarted manually, or report a run as having finished with errors although there were none.
SharePoint Importer: Fixed an issue with the SharePoint Importer where it would set values of multiline columns as literal HTML code sometimes.
SharePoint Importer: Fixed an issue where during import the Comment column was set to value “Version checked in by migration-center” automatically.
Box Importer: improved performance and reliability when importing large files (several hundred MB)
Box Importer: fixed issue where the progress would indicate 100% for the entire duration of a large file being uploaded, with no actual progress being visible to the user
Box Importer: fixed an issue where the max number of threads allowed is set for the importer and wrong credentials are entered – this caused connections to the database to be opened but not closed, exceeding the maximum number of connections allowed by the database after some time.

Changes

FirstDoc Importer: Changed handling of application number from regulatory dictionary to work around issue where duplicate application numbers are used. See the migration-center 3.2.2 – FirstDoc Importer User Guide document for more information about this feature.

Limitations

Documentum Content Validation relies on a checksum computed during scan to compare the checksum it computes after import against. Currently only the Documentum and Filesystem Scanner have the option of computing the required checksum during scan (Documentum Scanner: main content and renditions; Filesystem Scanner: only main content). Scanners for other source systems could of course be extended to also compute a checksum during scan time which the Documentum Content Validator could use.

Known Issues

The SharePoint Importer and the FirstDoc Importer cannot function together due to some conflicting Java methods. Workaround: use only one of the adapters on a given machine by deleting the other adapter’s folder from the migration-center Server Components libs subfolder.
Same issues as for the main release 3.2 also apply to release 3.2.2

migration-center 3.2

Features

As a major feature, migration-center 3.2 provides several new adapters for various source and target systems, listed below. The interaction between scanners and importers has also been reworked, so now scanners and importers are no longer required to exist in pairs in order to work together. Instead, any scanner/importer exists as a generic, standalone adapter that can be used in combination with any other available scanner/importer, allowing for any combination of source and target systems possible using the available adapters.

The adapters available for migration-center 3.2 are:

Documentum Scanner
eRoom Scanner
Filesystem Scanner
Database Scanner
Documentum Importer
Filesystem Importer
Box Importer
Alfresco Importer
SharePoint Importer
FirstDoc Importer

Filesystem-Documentum adapters have been discontinued and replaced by generic Filesystem and Documentum adapters respectively. The new adapters perform the same while at the same time allowing interaction between any scanner and importer.

Consult the individual adapters’ User Guide documents for detailed information regarding supported functionalities and systems.

Changes and Improvements

migration-center 3.2 now requires the Java Runtime Environment 1.6 It will not work with older versions.
Filesystem-Documentum Scanner has been replaced by generic Filesystem Scanner which generates content able to be migrated using Documentum Importer or other importers to the respective target systems
Documentum Scanner is no longer connected to the Documentum Importer and generates content able to be migrated using other importers as well
Filesystem-Documentum Importer has been replaced by generic Documentum Importer which can accept and import content from any scanner, including filesystem or Documentum sources
Documentum Importer: improved performance for importing large numbers of Virtual Documents by up to 15% by skipping some unneeded processing
Documentum Scanner: content files are now written to the export location by using the Windows/DOS extension corresponding to the respective formats instead of the Documentum format name used previously. This is necessary to facilitate interaction between the Documentum Scanner and various importers targeted at other content management systems that rely on the Windows/DOS extension to work properly.
Documentum Scanner has a new “skipContent” parameter - enabling it skips extraction of the actual content, thus saving time during scan. The feature is intended to be used for testing or other special purposes; not recommended to enable for production use.
Filesystem Scanner can now detect changes to external metadata files as well instead of just the main content file and can add the updated metadata as an update to the object. This is useful for scenarios where the content is expected to change less frequently than the external metadata files.
Filesystem Scanner now also supports reading external metadata for folders, providing the same functionalities as for file metadata.
Filesystem Scanner has a new “ignoreAttributes” parameter – this allows defining a list of unwanted attributes that should be ignored during scan. This mostly applies to attributes coming from external metadata files of from extended metadata from the files’ contents
Filesystem Scanner has a new “ignoreWarnings” parameter – this allows scanning documents even if some of their metadata (owner or creation date) cannot be extracted or if the external metadata file is missing. Use only if that kind of information is not critical for the migration, and/or would otherwise cause too many files to be skipped.
Client/Transformation Engine: system attributes are no longer available to be associated on the “Associations” page of the “Transformation Rules” window. These are used automatically by the respective importers and did not need to be associated anyway
Client/Transformation Engine: set default rule to be pre-selected in the “Get type name from” drop-down list on the “Associations” page for all the different object types (user can of course change the selection if desired)

Fixes

Documentum Scanner: fixed an isolated issue where a particular combination of scanner parameters, source attribute values and objects without content would prevent such objects from being scanned
Documentum Importer: fixed an issue where adding a new version of a document with less values for a repeating attribute than the previous version had for the same repeating attribute would keep the previous version’s repeating attribute values which exceeded the number of attribute values present in the currently added version
Documentum Importer: fixed an issue where an underscore character (“_”) in a rendition file’s path was interpreted as a page modifier separator, leading to unexpected errors. Now the “_” character’s role as a page modifier separator correctly applies to a rendition’s filename only, not its entire path
Documentum Importer: fixed an issue where the CURRENT version label was not updated correctly in case an object and updates of that object were imported together
Documentum Importer: fixed an issue where updating folder objects during import would not clear empty attributes in case those attributes had values different from null before the update.
Filesystem Scanner: fixed an issue where the modification date for folders would not be scanned
Filesystem Scanner: removed “content location” value for folder objects added previously by scanner since folders do not have content of their own. This does not influence functionality.
Filesystem Scanner: fixed error extracting extended metadata from PDF files
Filesystem Scanner: fixed some discrepancies with object counts in log files when scanning updated objects
Transformation engine: a multi-value rule is now validated against the multi-value setting of the attribute it is associated with, so trying to set multiple values for an attribute defined as single value for example will now throw a validation error.
Client: fixed an issue where the Object History feature did not return results at all under certain circumstances
Client: fixed multiple minor issues related to focusing and scrolling in various grids/lists

Limitations

Database: installation of the Oracle database schema can only be performed with the “sys” user
Database: multiple mc database schemata are not supported on the same database instance
Database installer: avoid running the database installer from folder locations containing “(“ or “)” characters in the path
Documentum Features: Cabinets: dm_cabinet objects currently cannot be represented as individual objects in mc, hence mc cannot store or set specific attributes for cabinets eitheri
Documentum Features: Cabinets: mc cannot migrate empty cabinets (i.e. dm_cabinet objects with no other objects linked to it) for reasons stated abovei
Documentum Features: Relations: support for migrating dm_relation type objects is limited to relations between dm_document and dm_folder type objects (and their respective subtypes)i
Documentum Features: Virtual documents: the snapshot feature of virtual documents is not supportedi
Documentum Scanner: for the DQL query to be accepted by the scanner it must conform to the following template: “select r_object_id from <dm_document|subtype of dm_document> (where …)”. Other data returned by the query or object types other than dm_document (or its subtypes) is not supported.
Documentum Scanner: the ExportVersions option needs to be checked for scanning Virtual Documents (i.e. if the ExportVirtualDocuments option is checked) even if the virtual documents themselves do not have multiple versions, otherwise the virtual documents export might produce unexpected results. This is because the VD parents may still reference child objects which are not current versions of those respective objects. This is not an actual product limitation, but rather an issue caused by this particular combination of Scanner options and Documentum’s VD features
Documentum Scanner: scanning dm_folder type objects using the DQL option is not supported currently.
Documentum Importer: the importer does not support importing multiple updates of the same object in the same run (or the base object and of its updates). Every update of an object must be imported in different runs (this is how it would happen during normal conditions anyway)
Update migration: Objects deleted from the source will not be detected and the corresponding target object will not be deleted (if already imported)
Update migration: whether a source object has been updated is determined by checking the i_vstamp and r_modify_date attributes; source objects changed by third party code/applications which do not touch these attributes might not be detected by mc
Client: the client application does not support multi-byte characters, such as characters encoded with Unicode. This does not affect the migration of such information, but merely limits the Client’s ability to display such values on screen; multi-byte characters are supported and processed accordingly by migration-center’s Database and Server components, which are the components involved in the actual data processingi
eRoom Scanner: scanning updates of eRoom objects is not supported currently; this also applies to newly added versions to a previously migrated object. Only newly created objects will be detected and migrated. This is due to the way eRoom handles object IDs internally, which prevents mc from correctly detecting changes to versioned objects. The full functionality may be implemented in a future release.
Filesystem Importer: does not import folders as standalone objects. Folders will be created as a result of the path information attached to the documents though, so folder structures are not lost. The full functionality may be implemented in a future release.
Database Scanner: the database adapter does not extract content from a database, only metadata. Content can be specified by the user during the transformation process via the mc_content_location system attribute. It may be necessary to extract the content to the filesystem first by other means before migration-center can process it. Content extraction functionality may be implemented in a future release.
Attributes: The maximum length of an attribute name is 100 bytesi
Attributes: The maximum length of an attribute value is 4000 bytesi
Attributes: The maximum length of a path for a file system object is 512 bytesi Note: all max supported string lengths are specified in bytes. This equals characters as long as the characters are single-byte characters (i.e. Latin characters). For multi-byte characters (as used by most languages and scripts having other than the basic Latin characters) it might result in less than the equivalent number of characters, depending on the number and byte length of multi-byte characters within the string (as used in UTF-8 encoding)

Known Issues

Documentum Importer: During runtime, the importer creates a registered table for internal use; this table will be not deleted after the import process has finished because it might be used by other importers running concurrently on other Jobservers. Since an importer job running on one Jobserver does not know about any importers that may be running on other Jobservers, it cannot tell whether it is safe to delete the table, which is why it is left in place. This registered table does not store actual data; it acts as a view to data already stored by Documentum. It is safe to remove this registered table once the migration project is finished. The following query is used to create the registered table: register table dm_dbo.dm_sysobject_s (r_object_id char(16), r_modify_date DATE, r_modifier char(32), r_creator_name char(32))
Documentum Scanner: if several scanners are running concurrently and are scanning overlapping locations (e.g. due to objects being linked into multiple locations), a scanner might detect an object scanned earlier by another scanner as an update, although nothing has changed about that object. This has been observed in combination with relations attached to the objects. This will result in some redundant object appearing as updates while they are in fact the same, but apart from this the final result of the migration is not affected in any way. The redundant objects will be handled like regular objects/updates and imported accordingly.
Documentum: If the group_permit, world_permit, owner_permit AND acl_domain, acl_name attributes are configured to be migrated together the *_permit attributes will override the permissions set by the acl_* attributes. This is due to Documentum’s inner workings and not migration-center. Also, Documentum will not throw an error in such a case, which makes it impossible for migration-center to tell that the acl_* attributes have been overridden and as such it will not report an error either, considering that all attributes have been set correctly. It is advised to use either the *_permit attributes OR the acl_* attributes in the same rule set in order to set permissions.
Transformation engine: Transformation Rules window/Associations page: if object types are added and no attributes are associated for these object types all objects matching the respective types will be migrated with no attribute values set (except for system attributes handled automatically by Documentum). Avoid adding object types without associating all required attributes for the respective types.
Client: on rare occasions, some windows or parts of a window will flicker or fail to refresh their contents. To work around issues like these, use the window’s Refresh button, scroll the window left/right, or close and re-open the affected window if nothing else helps.
Client: when editing an object type’s definition and trying to change an attribute’s type the drop-down list will not appear on Microsoft Windows 7 systems unless miglient.exe is configured to run with “Disable visual themes” (option can set on the Compatibility page in the executable’s properties)
Client: trying to import a mapping list from a file will not work on Microsoft Windows 7 systems because the context menu containing the command will not appear unless miglient.exe is configured to run in Windows XP compatibility mode (option can set on the Compatibility page in the executable’s properties)
Client: After using "Export to CSV" the folder where the CSV file has been saved is still in use by migration-center Client
Scheduler: the scheduler may report runs that finished with warnings during import as having finished with errors instead. Make sure to check any scheduler history entries listed as “Finished with errors” to determine whether the cause is an actual error condition or merely a warning, which is not a critical condition.
Installer: The migration-client installer does not work with User Account Control enabled in Windows 7. Please either disable UAC for the duration of the installation, or if installation needs to be performed using UAC enabled, manually grant full access permissions for the required users on the installation folders afterwards

Database update from previous versions: as a result of the update process increasing the default size for attribute fields to 4000 bytes (up from the previous 2000 bytes), Oracle’s internal data alignment structures may fragment. This can result in a performance drop of up to 20% when working with updated data (apparent when transforming/validating/resetting). A clean installation is not affected, neither is new data which is added to an updated database, because in these cases the new data will be properly aligned to the new 4000 byte sized fields as it is added to the database.

Client User Guide

Introduction

Documentum Scanner
eRoom Scanner
Filesystem Scanner
Database Scanner
Outlook Scanner
SharePoint Scanner
Documentum NCC Scanner
Alfresco Scanner
Domino Scanner
Exchange Scanner
OpenText Scanner
Documentum Importer
DCTM Audit trail Importer
Alfresco Importer
Filesystem Importer
FirstDoc Importer
D2 Importer
DCM Importer
box Importer
Alfresco Importer
SharePoint Legacy Importer
SharePoint Online Importer
SharePoint 2013 Importer
Documentum NCC Importer
InfoArchive Importer
OpenText Importer

migration-center client 3.13 cannot be used to connect to a migration-center database 3.4 as well as migration-center client 3.4 cannot be used to connect to migration-center database 3.13.

Other adapters can of course be developed according to customer or project requirements.

Using migration-center will tell why migration-center is “first in migration technology”.

Product features

The transformation of metadata through intelligent, rule-based mapping of objects
Incremental delta migrations for newly created and changed documents and folders
Identification of duplicates
one-time or regular batch migration of large volumes of documents and folders
Complete audit ability, error reporting and error handling
1:1 migration of folder structures and mapping of objects within, including metadata
1:1 migration of relations (i.e. Virtual Documents, etc.)
The possibility to export and import type templates, transformation rules, object grids (export only) to CSV files.

Features that were never/rarely used have been removed, resulting in a cleaner, more user-friendly and productivity-oriented GUI.

Compatibility

Architecture Diagram

All of the aforementioned information is defined and controlled via the migration-center Client, a GUI application which acts as the command and control center for migration-center.

Multiple instances of the above components can be deployed across the network, even across geographical boundaries, and can work together on the same or multiple projects.

The actual content moved during a migration (files, documents, emails) is stored on the file system (local or remote) and does not require a specific migration-center component for managing.

Not all adapters illustrated in the above diagram are part of the standard product package.

Client

During installation a default user is created. If no other users have been created, use the default user account shown below for logging on to the migration-center database.

Username: fmemc

Password: migration123

Job Server

Database

The migration-center database contains most of the data processing logic and stores all the migration sets, including their objects, metadata and transformation rules.

As with the other components, at least one database needs to be installed for migration-center to work, but several databases can be deployed in the same environment, depending on the requirements.

Both the Client and the Job Server need to connect to a migration-center database to function.

Client Functionalities

This chapter details the features and functionalities available to the user through migration-center Client.

Main application window

migration-center’s main application window adheres to typical windows applications’ user interface, offering a main menu bar and toolbar for accessing general, frequently used features.

Major features and functionalities each open in their own child windows, providing toolbars and context menus that apply to the items displayed in that window.

Any combination of child windows can be opened at the same time; child windows can be tiled or cascaded.

<migration-center> <Connect to…> Calls up a dialogue window where the user can connect to a (another) migration- center specific database.
<migration-center> <Reconnect> Reconnects client to currently used migration-center DB
<migration-center> <Renew License> Allows entering a new license key
<migration-center> <About> Displays information about licensing and product versions
<migration-center> <Exit> Exits migration-center
<Manage> <Scanners…> opens the -Scanners- window for creating and managing scanner jobs
<Manage> <Migration Sets…> open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
<Manage> <Importers…> opens the -Importers- window for creating and managing importer jobs
<Manage> <Jobs…> open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
<Manage> <Jobservers…> open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
<Manage> <Schedulers…> opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
<Manage> <Object Types…> opens the -Object Types- window, where the object type definitions can be imported and edited (these are required for associating transformation rules with for the attribute model transformation are defined)
<Manage> <Global Mapping Lists> opens the -Global Mapping Lists- window, where global mapping lists can be created or imported. A global mapping is automatically available for use in any migration set.
<Manage> <Object History…> opens the -History- window where it is possible to look up any object that underwent processing in migration-center. Key steps of the objects lifecycle during the migration can be viewed, such as the objects status, metadata and the exact timestamp at that moment.

The main toolbar provides quick access to frequently used functionalities. These correspond with the same functionalities available from the main menu.

[Scanners] opens the -Scanners- window for creating and managing scanner jobs
[Migration Sets] open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
[Importers] opens the -Importers- window for creating and managing importer jobs
[Jobs] open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
[Jobservers] open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
[Schedulers] opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
[Cascade] Arranges all open child windows in a cascade
[Tile Horizontally] Tiles all open child windows horizontally
[Tile Vertically] Tiles all open child windows vertically

Job Servers

To input/output any data from/to migration-center there needs to be at least one location with a running Job Server defined.

The location for a Job Server can be specified as the machine’s hostname or fully qualified domain name, if required, or simply as an IP address.

Note: Do not use loopback addresses or the “localhost” name.

The toolbar and the context menu of the -Jobservers- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[Refresh] is useful for rechecking the status of the locations listed
[New] opens a blank -Jobserver- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a location from list

Properties window

To create a new Job Server location

select <New> from the context menu or from the toolbar
Enter the IP address or hostname of the machine the Job Server is running on and the port number it is listening on.
Enter the appropriate port on which the Job Server on that machine was configured

The default port number is 9700, unless changed during setup.

Creating a scanner and trying to save it without having any Job Servers defined will prompt the user to define one Job Server location and save the scanner with the new created Job Server location.

Scanners

The toolbar and the context menu of the -Scanners- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[History] opens the -History- window where more information about all scans performed by the selected scanner can be viewed. The log files from these scans can also be accessed here.
[View Scanned Objects] will display a grid with the scanned objects in a new window, where these can be viewed, filtered, and exported to a CSV (comma separated values) file. This functionality does not allow any editing to be performed on the objects.
[Run] initiates the run of a scanner with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the scanners listed
[New] opens a blank -Scanner- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected scanner, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

All scanners, regardless of type share some common parameters

Name: must be a unique name for the current scanner

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the scanner depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this scanner does or add some comments related to the scanner

In the lower part of the windows the view can be filtered based on the available source attribute. Select an attribute from the list, enter Filter value, and press ENTER or [Apply filter]. To remove any applied filters, press the [Clear filter] button. Enter “(null)” to list attributes for which the selected filter attribute has no value

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Filtering attributes and (de)selecting columns is neither permanent, nor does it affect the actual data in any way. It only affects the current display and serves to analyze the data on display.

Migration Sets

It is possible to add multiple scans to one migration set, as it is possible to split a scan containing a large number of objects onto several migration sets.

Buttons and context menu items:

[Properties] is used to change the configuration of an item in the list. The -Migration Set- properties window can also be opened by double-clicking an item in the list
[Edit Transformation Rules] opens the window where transformation rules are managed and rules are applied
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Transform and Validate] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
[Reset All] resets all objects of in the selected migration set to their initial attribute values, essentially removing any processing performed during the transformation process and resetting status changes resulting from validation and import. The [Reset All] button is available in all views (-Source Objects-, -Processed Objects-, and -Error Objects-).

Resetting imported objects would typically be used during testing phases of a project and not during production.

[Stop] appears as a button only when the selected migration set is busy being transformed or validated, and only if these processes take longer than 3 seconds for the selected migration set.
[Refresh] refreshes data in all info-columns
[View All Objects] (press drop-down list/button combo) opens 3 windows: Source Documents, Processed Documents, and Error Documents.
[Source Objects] (select from drop-down list/button combo) displays all scanned objects attributed to the respective migration set. Original attributes and attribute values are displayed.
[Processed Objects] (select from drop-down list/button combo) displays the objects after they have been processed by applying the transformation rules.
[Error Objects] (select from drop-down list/button combo) displays objects that failed transformation/validation criteria, or failed to import, indicating the cause
[New] opens a blank -Migration Set- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured migration set from list

Items in columns can be sorted by clicking on the column headers. Clicking the same column header repeatedly switches between ascending/descending sorting order.

Migration Set Properties window

Common controls

The buttons on the right side of the –Migration Set Properties- window are available on all tabbed pages of this window.

The [OK], [Apply] and [Cancel] buttons are self-explanatory and work as expected.

Otherwise define a selection of objects as described in the following chapters before saving the migration set.

The [Preview objects] button and preview functionality is available on all pages of the –Migration Set Properties- window.

If the selection is ok, the selected scans can be assigned to the migration set using the [Select objects] button in the upper right corner.

The |Properties| page

The |Filescan selection| page

This page allows the user to select and assign any available scan to the current migration set.

Only scans matching the object type set on the previous page (|Properties|) will be listed here. Check the object type selection if the desired scans are not displayed here.

The |Exclude objects by values| page

This page offers a simple way of excluding some objects from the selection based on their values for one or more attributes.

In previous versions of migration-center this page was called |Filters|. The functionality provided is the same in all product versions.

The |Advanced Filters| page

For objects matching a filter expression means these objects will be added to the migration set, not excluded!

Conditional operators for working with string, numeric, and date/time type values are provided.

Migration Set Views

Three views are available for a migration set: Source Objects, Processed Objects, and Errors. These views can be accessed in the -Migration Sets- window by context menu or toolbar. Each window displays a different view of the current migration set objects, including their metadata, current status, associated migration-center internal information, messages related to the various processing steps, etc.

One, two, or all three of these three views can be opened at any time for the current migration set.

The filter bar at the bottom of the window is common to all views and offers following functionalities:

Select an attribute from the list, enter Filter value, and press ENTER or [Apply filter]. To remove any applied filters press the [Clear filter] button. Enter “(null)” to list objects for which the selected filter attribute has no value Use “_” (underscore) as a placeholder for one character, and “ % “ (percent) for zero or more characters.

Other features common to all views are the following toolbar items:

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Click [Refresh] to reload the content of the current view

Source Objects View

The checksum is computed during scan and must be activated in the scanner. See the respective adapter documentation for more information about using the checksum option in the scanner.

Toolbar items

[View Attributes] displays all available (scanned) attributes of one object only. Previous and following object can also be browsed by using the appropriate buttons.

[Edit Transformation Rules] opens the -Transformation Rules- window.

Items in columns can be arranged (increasing/decreasing) by clicking on the column labels.

[View Duplicate Objects] filters identifies and displays duplicate objects

Context menu items

<View Source Attributes> will display all attributes of that object in a small new window and allows browsing through the list objects by object (back and forward.

Copying the currently selected value to the clipboard function can also be performed directly using the <Copy Value> context menu entry or using the {Ctrl+C} keyboard shortcut

<Filter by Value> will immediately filter the view using the selected attribute value. Use the [Clear Filter] button in the filter bar at the bottom to remove a filter and return to the full list.

Processed Objects View

In addition to the -Source Documents- view, the -Processed Documents- window features a few more functions.

Transforming, validating, and the resetting objects can be performed directly from this window using the appropriate toolbar buttons.

[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Apply Transformation & Validation] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.

For quick access to the source attributes of any object the -Source Attributes- window can be also accessed from the context menu.

Resetting imported objects would typically be used during testing phases of a project and not during production.

Error Objects View

All this information makes it easy to identify and correct the error and transfer the object into the proper states allowing it to be migrated.

There are five basic ways of handling errors; one of them should always be able to solve typical errors encountered during migration

If the error is caused by a transformation rule (objects in Transformation Error status), adjust the transformation rule to cover the faulty objects and re-apply it to those objects. Any objects which have already been transformed, validated or transformed successfully will not be affected and will maintain their current status and metadata
If the error cannot be fixed by adjusting transformation rules and the object is in the Transformation Error status, it may be necessary to edit objects manually. Use the <Edit Attributes> window to browse through the objects and edit the attributes manually, skipping any objects where manual editing does not apply.
If the error occurs during import due to external factors (objects in Import Error status) like software or hardware issues, network issues, target system configuration problems, proper access permission, etc.) try to remedy the error first and after that just add the migration set back to the importer and simply try to import it again. Migration-center will attempt to import all objects in the Import Error state automatically and will succeed if the cause for the error has been fixed.
If objects have failed Validation (objects in Validation Error status) check the Object Type definitions and Associations in the Transformation Rules window, make the necessary corrections and re-run the Validation process to validate these objects successfully.
Finally, if none of the above helps, or the objects having failed are not even needed to be migrated they can either be removed from the migration set or left in their respective error states (temporarily or permanently). Unless other objects depend on them, leaving unneeded objects in an error state or removing them from the migration set will not affect other objects.

Transformation rules

The -Transformation rules- window has three tabs for managing different aspects of transformation and validation, and a set of common buttons accessible from all tabs. These buttons are:

[Copy Rules from...] allows copying a set of transformation rules from another migration set and applies it to the current migration set, thereby replacing any existing rules. Rules can only be copied from migration sets of the same type as the current migration set. If no migration sets of the same type exist, the list will be empty. Any migration sets of the same type will be displayed in a list, allowing the user to select one and click [Ok] to copy and apply its transformation rules to the current migration set.

[Export] will export the current migration set’s transformation rules to a XML file for the purpose of backing up or transferring the model to a migration set located on another DB.

[Import] imports transformation rules generated by the XML files generated with the previous export function. Warning! Importing rules from a file will replace all rules configured previously!

[Ok] saves changes and closes window.

[Cancel] cancels all changes that have not been applied yet and closes window.

[Apply] saves changes without closing the window.

Rules

Rules list

Rules for system attributes

The migration-centers adapter targeted at Documentum provide following system attribute rules:

“dctm_obj_link” specifies the path(s) where a document should be linked to in the target repository. “dctm_obj_link” is also an example for a system attribute which is defined by and used internally by migration-center and does not require association with an actual target attribute. This means there is no need for an actual attribute named “dctm_obj_link” to exist in either the source or target system.
The system attribute “dctm_obj_rendition” specifies file system location to files which represent renditions. Zero or more renditions can be specified for any object and will be added by migration-center during import. This attribute is also an internal attribute used by migration-center similar to “dctm_obj_link”
“r_folder_path” and “r_object_type” can get their default information automatically (see [Generate attributes]) only inside a Documentum-to-Documentum migration set. In the migration set of a file system scan, this information must be put in by user.

Rule properties

Transformation methods

Error and Warning indicators for rules:

In the example below, the rule called “extension” has been marked yellow by migration-center’s rule auto verification feature.

Marking rules and transformation steps as errors works exactly the same as described above for warnings.

Errors should never be ignored because they will definitely affect, or even prevent transformation rules from working correctly.

Working with transformation functions

Transformation functions provide the functionality behind transformation rules.

Custom functions can be implemented and used along the standard functions provided by migration-center.

A function’s parameter always has two properties: a type and a value. The parameter’s value is determined by the parameter’s type. Parameters can be of the following types:

User Value: acts as a constant value and will be set for all objects to which the current rule applies.

In case the parameter in question is referring to a single value attribute selecting an Index will have no effect.

The Index selection list offers the following options:

All is the default option and can be used only for the first parameter and not the subsequent parameters.

The Index selection offers some more options when used with the If transformation function:

It is also possible to specify All as an option for the Return if true or Return if false parameters.

Transformation function reference

Mapping Lists

The |Mapping List| page allows the user to create, duplicate, delete and edit entries for mapping lists using the buttons provided in the toolbar. Mapping lists also have two properties:

“Case sensitive” will consider a valid match only if the case of the input value matches the key and its case. Usually this would not be used, enable if needed.

Migration-center will interpret this transformation rule as follows:

Global Mapping Lists

Associations

Types

Associations

Importers

Importers migrate one or more migration sets. Only migration sets with objects that have successfully passed both transformation and validation can be imported.

The toolbar and the context menu of the -Importers- window contain the same commands.

[Properties] opens the configuration details of that particular importer. The -Importer- properties window can also be opened by double-clicking an item in the list.
[History] opens the -History- window where more accurate data about all imports performed by the importer in question can be accessed. The log of the scans can be accessed here.
[Run] initiates the run of an importer with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the importers listed
[New] opens a blank -Importer- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected importer, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

All importers, regardless of type share some common parameters

Name: must be a unique name for the current importer

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the importer depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this importer does or add some comments related to the importer

Scheduler

A Scheduler acts as a master process which manages and runs an end-to-end migration fully automated and according to a set schedule.

The toolbar and the context menu of the -Schedulers- window contain the familiar items: New, Properties, Delete, Run, Refresh.

Double-clicking a scheduler in the list has the same function as the <Properties> menu command.

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

A scheduler can be run immediately if needed, regardless of its actual schedule, through the <Run> command found in the toolbar and the context menu.

Properties

The |General settings| tab contains only three directly editable fields: “Name”, “Description” and a checkbox called “Active”. The checkbox changes the status of (Active: true/false) of a configured scheduler. If no flag is set (status: false) a configured scheduler will not run at the configured time and date. The Info dynamic area displays various information about the scheduler.

In the |Configuration| tab, the appropriate (previously configured) scanners, importers and attribute models are selected. Caution is to be executed when choosing an attribute model, so that an attribute model of similar purpose and type to scanner and importer shall be chosen.

The |Interval| tab allows the configuration of the time and date the scheduler will begin to run and stop to function. Also, the frequency of the runs can be set up here.

The |History| tab provides information about various configuration aspects and run dates of the scheduler, of the current state, last operation undertaken, as well as giving access to the logs.

Object Types definitions

Out-of-the-box the product comes with basic object type template definitions for the adapters the customer has acquired licenses for.

Additional user/custom object types can be added to migration-center by the user.

id,2,0,0,0,0

first_name,2,0,0,0,0

last_name,2,0,0,0,0

email,2,0,0,0,0

gender,2,0,0,0,0

ip_address,2,0,0,0,0

animal_name,2,0,0,0,0

Valid values for data type are: 0 - Boolean, 1 - Integer, 2 - String, 3 - ID, 4 - Date and Time, 5 – Double.

^[^\~\"\#\%\&\*\:\<\>\?\/\\\{\|\}]+$

It will match one or more characters between begin and end of the string that are not in the list of the following characters: ~"#%&*:<>?/\{|}

^[0-9]{3}-[0-9]{5}$

Object History

The -History- window can be accessed from migration-center’s main menu <Manage>, menu item <Object History…>

It is possible to search Object History by type of object or attribute values, including internal system attributes used by migration-center. The search parameters also support wildcards such as “_” (underline) for one character, and “ % “ (percent) for zero or more characters. The button displays the full set of attributes associated with the selected object at that respective point in time. This can prove useful in case some attributes are wrong or missing after import to the target system – by verifying the attributes assigned to the object by migration-center at the time of import it is possible to determine whether any of the inconsistencies encountered in the target system are due to migration-center or not.

Log files

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated for each job can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job

Auto Classification Module

Introduction

This document is a guide on the auto classification module for migration-center 3.

Installation

The installation process can be performed in a few minutes. But before installing the module, two mandatory prerequisites must be installed.

Prerequisites

Along with the Python language, the installer installs the package manager “pip”. “pip” is a terminal software and needs to be executed in the Windows CMD shell.

In addition, it is possible to use a dedicated Apache Tika instance. If you like to do that please refer to section Installing Apache Tika.

The operation of the Auto Classification Module requires configuration files. The samples subdirectory contains a set of example files.

Installation Steps

The installation of the auto classification module can be summarized in seven steps.

Download and extract the software code from the zip archive onto your hard drive. Copy the files to a directory of your choice.
Open a CMD window and navigate to the path where the software was copied to or open a command window from the Windows Explorer.
Install all Python package requirements by executing the following statement in the CMD: pip install -r requirements.txt
(optional) Install Oracle Instant Client (see here)
(optional) Install TesseractOCR (see Installing TesseractOCR)
Create the “config.ini” file, as described in section Deploying the auto classification module.
Provide a classifier configuration file. The configuration schema of the file is specified in Classifier configuration.
Start the module by executing the init file in the CMD: python __init__.py {path to config.ini} The script accepts the file path to the “config.ini” file as the only parameter. If the file path is not provided, the script assumes that the “config.ini” file is in the same directory.
Open the GUI in a web browser: http://{server ip}:{WSGI port}/app

After executing the script, the service boots. Please wait until the service has started successfully, before executing any further operations. The CMD windows needs to stay open.

Installing TesseractOCR

Windows

Also, you can add additional languages later on.

Add the installation path into your environment Path variable.

Linux

First, download the Linux installer from the official GitHub repository or from apt. An installation instruction can be found in the project’s wiki.

Installing Apache Tika

If no connection to the internet can be established, the same procedure as starting Apache Tika from a dedicated server can be applied.

To use Apache Tika on a dedicated server, please download the tika-server.jar file from the official repository first.

Afterwards, the tika-server.jar file must be executed and the port must be open for outside communication.

“tika_client” needs to be set to true.

“tika_client_port” needs to be set to the correct port number.

“host” needs to be set to the correct name / IP address.

If Tika runs on port 9000 in a dedicated process on the same machine, the configuration looks as follows:

[TIKA] tika_client: true tika_client_port: 9000 host: 127.0.0.1

Installing Graphviz

First, Graphviz needs to be downloaded and installed.

Second, the Python wrapper for Graphviz needs to be installed in the Python environment.

On Linux OS, pygraphviz can be installed easily with pip in the CMD:

pip install pygraphviz

pip install pygraphviz-1.5-cp37-cp37m-win_amd64.whl

The supplied whl file is compiled for Windows OS with 64 bit and Python 3.7 installed. Other compiled whl files for Windows can be downloaded from the GitHub repository.

Configuration

Deploying the auto classification module

The following two chapters provide information on how to set up SSL encryption and a valid path to an algorithm configuration file.

Template files are provided in the sample directory of the module.

Port configuration

The configuration file contains two port properties. One property is defined in the FLASK_SERVER section and the other in the WSGI_SERVER section.

The Flask server is only used for development.

Therefore, if you like to change the port number of the AC module, set the port property under the WSGI_SERVER section. The definition can look like this:

port: 443

By default, the port is set to 3000. This means that every interaction with the module goes through port 3000. This includes API requests and interacting with the web app.

Configuring SSL

SSL encryption can be activated by defining the following line in the SSL chapter of the configuration file:

ssl.enabled: true

To use SSL encryption, it is mandatory to provide a valid SSL certificate. The required properties are:

ssl.certificate_file_path ssl.key_file_path

Classifier configuration file

To be able to use the auto classification module, a classifier configuration file must be provided. The file can be specified with the property

file_path

in the config.ini file under the section ALGORITHM_CONFIG

The file’s schema and configuration options are described thoroughly in the section Classifier configuration below.

Starting the module

The module is a Python program. Therefore, starting the module is very easy. Just enter the following command:

python __init__.py {file path to config.ini}

Classifier configuration

A classifier is a combination of an algorithm with a transformation list. Before a classifier can be used, it must be configured and trained.

The use of a transformation list is aimed to select the most meaningful words (feature selection) and create additional data from files (feature extraction).

A transformation list can be interpreted as a pipeline for the words of a document before they are submitted to an algorithm.

lower-case
tf-idf
stopwords

All classifiers of the auto classification module are configured in a single XML file. The main XML elements are

algorithms
transformation-lists
common-config

The following chapters aim to explain the configuration possibilities in detail.

Also, a template and sample XML file are provided.

Element “algorithms”

Inside the “algorithms” element the individual algorithms are specified. The supported algorithms are:

SVM (element name: “svm”)
Naïve Bayes Multinomial (element name: “naive-bayes-multinomial”)
Logistic regression (element-name: “logistic-regression”)

Additionally, every algorithm offers attributes to configure its behavior.

The SVM supports the following attributes:

kernel (value of “linear”, “poly” or “rbf”)
degree (positive integer, affects only poly-kernel)
gamma (positive double, affects only rbf-kernel)
c (positive float)

Naïve Bayes Multinomial offers the following attributes:

alpha (positive float, affects only Naïve Bayes)

Element “transformation-lists”

On the following pages every transformation function is described.

Element “common config”

With the “common config” attribute, global properties can be configured. These properties are accessed by all classifiers. Currently it does not offer any configuration parameters.

Usage

The API interface will always respond with a body encoded in JSON format.

https://{server-ip}:{WSGI port}/app

This document only describes how to use the API and not the UI. However, all parameters described in this chapter, can be used in the UI.

Using a metadata file

To use a unified metadata file, the user needs to change the following things on a HTTP request:

Replace the “scan-run” part of the URL with “unified-metadata-file”.
Send the request with the mimetype “form-data”
Send the metadata file with the form key “file”.
(optional) Remove the properties of database connection and scan-run-id.

Analyzing a dataset

Before validating and training a classifier, the training dataset needs to be investigated. The module offers two different reports.

Distribution report

The distribution report creates a barplot for every attribute with the frequencies of the values on the y axis.

A distribution report can be created with a HTTP POST request to the URL:

https://{server-ip}:{port}/api/distribution-report/scan-run

It is required to supply three parameters with the request:

id
dbCon
classAttrs

The id is the scan run id, the database connection must be a JDBC connection string and the class attributes must be string of attributes, separated with a comma.

It returns a pdf report.

Hierarchy report

Much more detailed is the hierarchy report. It allows to specify an attribute hierarchy. It creates a graph which gives an insight into the frequency of every attribute value per hierarchy layer.

A hierarchy report can be created with a HTTP POST request to the URL:

https://{server-ip}:{port}/api/hierarchy-report/scan-run

It is required to supply three parameters with the request:

id
dbCon
classAttrs