1 of 50

3.17 Overview

This online documentation describes how to use the migration-center software in order to migrate content from a source system into a target system (or into the same system in case of an in-place migration).

Please see the Release Notes for a summary of new and changed features as well as known issues in the migration-center releases.

Also, please make sure that you have read System Requirements before you install and use migration-center in order to achieve the best performance and results for your migration projects.

The supported source and target systems and their versions can be found in our knowledge base article List of supported versions.

The migration-center is a modular system that uses so called adapters in order to connect to the various source and target systems. The adapters to connect to a source system are called scanners and the adapters to connect to a target system are called importers. The capabilities and the configuration parameters of each adapter are described in the corresponding manual.

You can find the explanations of the terms and concepts used in this manual in the Glossary.

And last but not least, Legal Notice contains some important legal information.

Customized versions of adapters or entirely new ones (for other types of source and target systems) can be created and added to migration-center thanks to the open, API-based and documented architecture of the product. If such customized adapters are deployed in your migration project, the documentation provided with them should be consulted instead, since the features, behavior and parameters are likely to be different from the adapters described in these guides.

Further Resources

For additional tips and tricks, latest blog posts, FAQs, and helpful how-to articles, please visit our user forum.

In case you have technical questions or suggestions for improving migration-center, please send an email to our product support at support@migration-center.com.

Legal Notice

fme AG believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." FME AG MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any fme AG software described in this publication requires an applicable software license.

All trademarks used herein are the property of their respective owners.

Release Notes

migration-center 3.17

The Auto Classification Module has been removed and replaced with the Amazon Comprehend Enrich Scanner.

Existing configurations of the Auto Classification Module will no longer work with the 3.17 Jobserver.

Features & Enhancements

General:
- Transformation function getDataFromSql can return multiple values (#535420)
Hyland OnBase Importer
- Import documents
- Set custom metadata
- Import document revisions
- Delta migration (only for metadata)
Amazon Comprehend Enrich Scanner
- Enrich documents metadata based on the content
- Language detection
- Extract entities (standard and custom)
- Support custom classifiers
Filesystem Scanner:
- Make versioning case insensitive (#58656)
Documentum Scanner:
- Allow more flexible queries for selecting documents to be scanned (#58345)
- Create counter for already scanned objects in the execution summary (#58907)
- Export documents with content_location pointing to the path in the storage (#54317)
Cara Importer:
- Import audit trails from Documentum as audit trails objects (#58797)
- Index the imported documents (#59513)

Fixes

General:
- Scheduler next run date not calculated correctly (#58845)
Cara Importer:
- Fix reporting errors for missing/incorrect system attributes (#58488)
- Fix nonintuitive error message when required content or rendition attributes are not set (#58608)
CSV-Excel Scanner:
- Fix scanning CSV with UTF8 - BOM with first column used as ID (#54010)
Veeva Importer:
- Fix max length for system rules (#59046)
- Attributes mapping does not work properly when the object_type__v is not the same as the object that is imported (#57383)
Veeva Scanner:
- Binder relations are scanned even if the child documents are not (#56230)
SharePoint Online Importer:
- Temp file is deleted early if HTML error is thrown (#58869)
OpenText Content Server Importer:
- Importing compound documents doesn't fail when folder with same name already exists (#52038)
D2 Importer:
- Repeating default values are not set correctly (#58692)

Known Issues & Limitations

Cara Importer:
- Asynchronous Indexing not setting index on documents (#59574)
SharePoint Online Scanner:
- CAML query not working on SPO Scanner (#59597)

Glossary

Migration Set A migration set comprises a selection of objects (documents, folders) and set of rules for migrating these objects. A migration set represents the work unit of migration-center. Objects can be added or excluded based on various filtering criteria. Individual transformation rules, mapping lists and validations can be defined for a migration set. Transformation rules generate values for attributes, which are in turn validated by validation rules. Objects failing to pass either transformation or validation rules will be reported as errors, requiring the user to review and fix these errors before being allowed to import such objects.

Attribute A piece of metadata belonging to an object (e.g. name, author, creation date, etc.). Can also refer to the attribute’s value, depending on context.

Transformation Rules A set of rules used for editing, transforming and generating attribute values. A set of transformation rules is always unique to a migration set. A single transformation rules is comprised of one or several different steps, where each step calls exactly one transformation function. Transformation rules can be exported/imported to/from files or copied between migration sets containing the same type of objects.

Transformation Function Transformation functions compute attribute values for a transformation rule. Multiple transformation functions can be used in a single transformation rule.

Job Server The migration-center component listening to incoming job requests, and running jobs by executing the code behind the adapters referred by those jobs. Starting a scanner or importer which uses the Documentum adapter will send a request to the Jobserver set for that scanner, and tell that Jobserver to execute the specified job with its parameters and the corresponding adapter code.

Transformation The transformation process transforms a set of objects according to the set of rules to generate or extract.

Validation Validation checks the attribute values resulting from the Transformation step against the definitions of the object types these attributes are associated with. It checks to make sure the values meet basic properties such as data type, length, repeating or mandatory properties of the attributes they are associated with. Only if an object passes validation for every one of its attributes will it be allowed for import. Objects which do not pass validation are not permitted for import since they would fail anyway.

Mapping list A mapping list is a key-value pair used to match a value from the source data (the key) directly to the specified value.

Getting Started

Quick Start Guide

Component Overview

Migration-center has 3 main components: Database component, Windows Client and Jobserver component.

The Database stores all the migration-center configurations and all scanned and imported objects with their source and target metadata.

The MC Client is the UI where you can manage all the scanner, importer and migset configurations and you can trigger scan, import or transformation jobs. Multiple Clients can be installed.

The Jobserver component receives the scan or import configuration from the Client and connects to source/target systems to extract/import objects, processes metadata into/from the Database and the content into/from the file system staging area. Multiple Jobservers can be installed.

For the following systems there are additional steps or components: Alfresco, Documentum, Domino/Lotus Notes, SharePoint, SharePoint Online.

Please refer to their respective user guides for the actual instructions.

Component installation

This is just a quick overview of the installation process. For specific details please see System Requirements and Installation Guide.

All main migration-center components can be installed either on the same machine or on separate machines, by using the windows installers.

The Database installer needs a 32-bit Oracle Client to connect to the Oracle Database and install the migration-center schema. If the installer is run directly on the Oracle Server, the Oracle Database needs to be 32-bit for the installer to see it, otherwise an 32-bit Oracle Client needs to be installed as well.

The Jobserver installer can be run on any windows machine, but to actually start the Jobserver you need Java 8 or 11 installed. The Linux Jobserver does not have an installer, it has scripts to install the service.

The MC Client installer can be run on any Windows machine as well, but to connect the Client to the Oracle Database it also needs a 32-bit Oracle Client configured or a 32-bit Oracle Database if it is installed on the Oracle server. The Oracle Client must have the connection information of the Database in its tnsnames.ora file.

Component port connections

The Client connects as follows: - to the Database via Oracle Client through the Oracle port, default is 1521 - to the Jobserver via port 9700 by default (this can be changed in the ...\lib\mc-core\server-config.properties file)

The Jobserver connects as follows: - to the Database via JDBC connection on the same Oracle port as the Client - to each source or target system differently based on the system itself

When using the scheduler feature the job will be triggered by the Database instance itself, therefore the Database machine will also connect to the Jobserver via port 9700 by default.

Here is a diagram of the mentioned port connections:

Migration Process

The general process of performing migrations with migration-center is done in several steps:

The Analyze phase involves configuring a Scanner to connect to a Source System, that will extract the metadata and save it as objects in the migration-center database, and export the content to a defined Filesystem location, which acts as a staging area.

The Organize phase involves assigning Scan Runs into Migration Sets, or migsets for short.

The next 3 phases, Transform, Validate and Correct, work together and involve creating Transformation Rules by which your Source Metadata is Transformed into Target Metadata. The metadata is assigned to Target Type definitions, which can be defined in advance. And based on those definitions the metadata is Validated. If the resulting target metadata is not the desired one, you can reset the objects and repeat the process by correcting or adding transformation rules.

The Import phase, is the last step, and involves creating an Importer, which will connect to the Target System, assigning a migset with Validated Objects to it and starting the import run. You can monitor the progress in the importer run history or directly in the migsets view.

All the migration phases can be done in parallel for different batches of documents that are migrated.

Starting the Client

When you first start the migration-center Client you will see a login window.

The default login user is fmemc and the default password is migration123. From the Database dropdown select the your database instance where you installed the migration-center database component.

If you are not seeing anything in the Database dropdown, either your Oracle Client installation is not on 32-bits or the installation is not correctly done.

Defining a Jobserver

Before you are able to run a scanner you first need to define a Jobserver.

Open to the Jobservers Definition window and enter the hostname of the machine, where the Jobserver component is installed, and the port (default is 9700).

Avoid using localhost or 127.0.0.1 for the Jobserver location. That will cause issues with Clients installed on other machines and when using the scheduler feature.

Creating a Scanner

Open the Scanners window and click on the new button on the left:

Fill in the Name of this scanner, select a type and select a Jobserver where you want this scan to run.

After you have completed at least all the mandatory parameters of the scanner configuration you can save it by clicking on OK.

You can run it by hitting Run in the toolbar, or in the Right Click context menu.

Clicking on History allows you to see a list of all Scan Runs performed by the this scanner configuration.

Clicking on View Scanned Objects shows you the Source Objects grid for this scanner or scan run.

Creating a Migset

Open the Migsets window and click on the new button on the left. Enter a name for this migset and select the Migration Path from the Type dropdown.

From the next tab, Filescan Selection, you will see the scan runs corresponding to your chosen migration-path. Select which scan runs you want to include in this migset by double clicking them or by using the down arrow button.

To actually get the objects locked in this migset you need to click on the Select Objects button

Filtering the objects (optional)

As an optional step before clicking on the Select Objects button, there are 2 filtering options:

Simple Filtering, where you can exclude objects based on their values:

Advanced Filtering, where you can create rules by which to select the objects:

Creating Transformation Rules

With the Migset you have just created selected, click on the Edit Transformation Rules from the toolbar or from the right click context menu.

Regular Transformation Rules

The Rules section in the top left corner lists the Transformation Rules created so far in this migset. You can create a new one using the New button. In the middle there's the Rule Properties section where you can specify the rule Name and if it should return multi-value results.

The Transformation methods section contains a dropdown of all the Functions that you can use inside your transformation rule. These functions take Source Attributes and static values to generate a new value as a Target Attribute. Multiple functions can be linked together by processing the result of a Previous Step:

As you can imaging this is the main strength of migration-center as it allows you to create very complex rules to ensure you get the required results in your Target System.

System rules

On the bottom left corner we have the Rules for system attributes. These function the same as the regular transformation rules, except they have a fixed name and therefore serve a specific purpose during the import. Please refer to the user guide of the Importer you are using for details on what each system rule does.

Associations

On the Associations tab of the transformation rules window you can associate the previously defined Transformation rules to a Target Attribute of an object type.

First you must select a rule which will determine what object type each object will have. This is done from the Get type name from rule dropdown and in most cases there already is a system rule for this purpose. Based on the value this rule returns, each object will be associated with a target type from the list below.

The Available Type Definitions shows you a list of all Object Type Definitions in migration-center and you can select which ones are to be used in this migset. In the screenshot above we have selected opentext_document and opentext_rm_classification.

Clicking through the list of select types will show, on the right side, the current associations between Rules and Target attributes.

Rules that were not associated to any target attribute will NOT be migrated during the import

Object Type Definitions

To mange the available object types in migration-center go to Manage -> Object Types...

This will open Object Type definition window where you can create new Types and either manually adding each of the type's attributes or importing them from a CSV file. More details are in the Client User Guide.

Transforming and Validating objects

After you have finished writing the necessary transformation rules you can save the migset, and trigger Transformation and then Validation on the migset.

Transformation will apply the rules in the migset on each object individually and will generate the target objects. The objects will move to the Transformed state.

Validation will compare the generated values against the object type definition restrictions to ensure the values fit. If successful the objects will move to the Validated state.

The source metadata remains unchanged during the entire migration process and can be viewed in the Source Objects view

After doing the transformation, the Target Objects can be viewed in the Processed Objects view.

If any issue occurred during transformation or validation, the objects will move to the Error state and can be checked in the Error Objects view.

Only objects that reached the Validated State will be processed by the importer. This includes objects which were validated and then marked as error during import.

Creating an Importer

The Importers section is very similar to the Scanners one so we will only cover the differences.

After setting all necessary parameters for your import to function correctly you must go to the Migsets tab which will allow you to select one or more migsets for import. After that you can save the importer and run it.

You can monitor the status of the import either in the the Jobs view, in the importer History or in the Migsets view. Each view will show the progress of the migration in varying levels of detail, with the migsets one showing the most details of the objects being imported.

When the import has finished you can see the status changed to Finished and all the objects marked as Imported. If any errors or warnings occurred you can view them either in the Error Objects view of the migset or by opening the import run log from the history of the importer.

Important Features

Creating a Scheduled Job

You can automate the migration process from Scan to Import by using the Scheduler feature. This is very useful in setting up a continuous migration of active systems where the users are still modifying and creating documents.

How it works: You create a scheduled job and select existing valid scanner, migset and importer configurations. If the Scheduler is set to active and depending on what interval settings are set, the scheduler will automatically perform the following actions: - Start the scanner - Create a copy of your migset and assign the scanned objects to it - Run Transformation and Validation - Assign the migset to the importer - Start the importer - (optional) send an email report if configured

The types of the scanner, migset and importer selected in a scheduler must belong to the same migration path.

Create a new Scheduler and enter the Name and Description.

Next, on the Configuration tab, you can select your Scanner, Migset and Importer. You can also configure the email reporting feature.

Next, on the Interval tab, you have all the options to configure when your scheduled job will trigger and when it will stop triggering.

You can click on Ok or Apply to save the Scheduler and it will start at the scheduled time.

If you open the History tab you can see the list of all runs of the scheduler along with some information about the status of each run.

Mapping Lists

The mapping lists are a simple and very powerful feature of migration-center. They can be used for a wide variety of scenarios.

A mapping list is just a collection of key - value pairs. You can either create one globally using the Manage -> Global Mapping Lists... menu. This will make the mapping list available for use in any migset.

Or you can create it directly in a migset for use only in that migset.

To use your created mapping list you must use the mapValue() function inside a migset's transformation rules.

The value to be transformed is matched with the values in the Key column of the mapping list. If a value is found then the function returns the equivalent value from the Value column. If the value cannot be matched or the source value is null, the function will trigger a Transformation error on the object or return an empty value (depending if Report missing values is set to 1 or 0).

Client User Guide

Introduction

The focus with migration-center 3.x is on providing a lightweight, yet highly functional, usable and extensible migration solution which can grow to match the requirements of present and future migration projects through customization and connectivity to other document and content management systems.

The current release is primarily a release that consolidates the features and adapters developed in the versions 3.2.x, which itself is a significant update to the initial version 3.0 with major new features, usability, performance, and reliability improvements. Besides the incremental improvements and fixes, migration-center also brings along an entire suite of new adapters available directly from fme; currently the full list of adapters for migration-center available from fme is:

Documentum Scanner
Filesystem Scanner
Database Scanner
Outlook Scanner
SharePoint Scanner
Documentum NCC Scanner
Alfresco Scanner
Domino Scanner
Exchange Scanner
OpenText Scanner
Documentum Importer
Alfresco Importer
Filesystem Importer
D2 Importer
box Importer
Alfresco Importer
SharePoint Online Importer
SharePoint On-Premises Importer
Documentum NCC Importer
InfoArchive Importer
OpenText Importer
Veeva Importer
Trackwise Digital Importer
Cara 5.3 Importer

migration-center client 3.13 cannot be used to connect to a migration-center database 3.4 as well as migration-center client 3.4 cannot be used to connect to migration-center database 3.13.

Other adapters can of course be developed according to customer or project requirements.

Product features

As stated in the introduction, migration-center 3.x is a complete rebuild. The general concept and useful features from the previous version have been retained functionality-wise and improved in the current version. migration-center’s strengths remain:

The transformation of metadata through intelligent, rule-based mapping of objects
Incremental delta migrations for newly created and changed documents and folders
Identification of duplicates
one-time or regular batch migration of large volumes of documents and folders
Complete audit ability, error reporting and error handling
1:1 migration of folder structures and mapping of objects within, including metadata
1:1 migration of relations (i.e. Virtual Documents, etc.)
The possibility to export and import type templates, transformation rules, object grids (export only) to CSV files.

Features that were never/rarely used have been removed, resulting in a cleaner, more user-friendly and productivity-oriented GUI.

Compatibility

Migration-center 3.x is not compatible with migration-center 2.x. Migrations initially started with migration-center 2.x should be completed before upgrading to the new version; a version upgrade halfway through a project is not possible.

Architecture Diagram

From a technical point of view, migration-center is a distributed, client-server type application using a database management system as a backend. All these components interact to allow for a highly configurable, automated mass import of very large amounts of data from a source document/content management system into another content management system.

The solution centers on an Oracle database used for storing object related information like metadata, migration rules, and all configuration and status information necessary for managing and tracking the steps involved in the migration process.

All of the aforementioned information is defined and controlled via the migration-center Client, a GUI application which acts as the command and control center for migration-center.

Finally, there are the adapters which connect migration-center to various source and target systems via migration-center’s own API on one end and the respective system’s API(s) on the other end. These adapters which interact directly with the source and target systems are controlled by a service, which is also managed by the migration-center Client application (configuring and launching the appropriate jobs). There can be any number of adapters installed and managed with migration-center.

Multiple instances of the above components can be deployed across the network, even across geographical boundaries, and can work together on the same or multiple projects.

The actual content moved during a migration (files, documents, emails) is stored on the file system (local or remote) and does not require a specific migration-center component for managing.

Depending on the combination of adapters deployed in a particular migration project, the source and target systems supported can be same/different versions of the same solution or entirely heterogeneous systems sharing nothing but basic concepts of content management, like a content object with associated metadata.

Not all adapters illustrated in the above diagram are part of the standard product package.

Client

Migration-center Client is a GUI application and represents the interface between the user and the migration-center core features and functionalities. All migration-center functionalities like creating, monitoring and managing jobs, tasks, and migration sets are accessible through the client.

The client does not perform any tasks by itself, it’s merely a management interface; therefore the Client application can be disconnected or shut down safely at any time, without affecting any tasks that might be running in migration-center at that time.

The client is a standard Windows application that can be installed anywhere, provided it has a network connection to migration-center’s central database, and the prerequisites installed (as described in the Installation Guide). Any number of Clients can be deployed across a network, and connect to the same, or different migration-center databases.

After starting the migration-center Client application, the database login window opens. The Client application is not operational by itself, since all information required to operate migration-center is stored in a migration-center database, which is why a connection to such a database is required at all times to use the Client. Type the username and the password, select the migration database from the list of available database connections and click [OK]. migration-center Client will remember the last database it connected to and will automatically reconnect to this database the next time it is started.

During installation a default user is created. If no other users have been created, use the default user account shown below for logging on to the migration-center database.

Username: fmemc

Password: migration123

Job Server

The Job Server is a migration-center component installed with the migration-center Server Components; it runs as a Windows service and does not interact with the user. Its purpose is to execute various tasks in the background, like scanning or importing documents. The tasks running on a given Job Server can be monitored and managed by the user through the Client. A migration-center Job Server can be installed anywhere as long as there is a network connection to the migration-center database; several Job Servers can be deployed across the network and managed using migration-center Clients. Since the Job Server is the process actually accessing the source and target systems, the user account this service runs with should have the required permissions for proper access to the respective source and target systems.

Database

The migration-center database contains most of the data processing logic and stores all the migration sets, including their objects, metadata and transformation rules.

As with the other components, at least one database needs to be installed for migration-center to work, but several databases can be deployed in the same environment, depending on the requirements.

Both the Client and the Job Server need to connect to a migration-center database to function.

Client Known Issues

Changes done in the Object Types window will not be saved in the following scenario:

Opening the MC Client and making changes to the object types before opening any of the following screens: scanner properties, migset properties, transformation rules, importer properties. Until the screens are opened the changes are saved only in the Client and not in the MC Database.

Clicking on Cancel in any of the screens will revert the object type changes. Clicking on Ok on any of the mentioned screens will save the object type changes.

This only occurs when the client was freshly opened. After going to any of the mentioned screens the object type screen will behave as expected.

Client Functionalities

This chapter details the features and functionalities available to the user through migration-center Client.

Main application window

migration-center’s main application window adheres to typical windows applications’ user interface, offering a main menu bar and toolbar for accessing general, frequently used features.

Major features and functionalities each open in their own child windows, providing toolbars and context menus that apply to the items displayed in that window.

Any combination of child windows can be opened at the same time; child windows can be tiled or cascaded.

Some windows (e.g. windows displaying properties for items, or message boxes informing the user or requesting input from the user) always display on top of other windows and stay there until they are closed by the user.

All the important main menu commands are represented in the toolbar of the main window, are self-explanatory and will be described in the next chapter - Main window toolbar. The menu commands which do not have a correspondent in the toolbar menu will be described here.

<migration-center> <Connect to…> Calls up a dialogue window where the user can connect to a (another) migration- center specific database.
<migration-center> <Reconnect> Reconnects client to currently used migration-center DB
<migration-center> <Renew License> Allows entering a new license key
<migration-center> <About> Displays information about licensing and product versions
<migration-center> <Exit> Exits migration-center
<Manage> <Scanners…> opens the -Scanners- window for creating and managing scanner jobs
<Manage> <Migration Sets…> open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
<Manage> <Importers…> opens the -Importers- window for creating and managing importer jobs
<Manage> <Jobs…> open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
<Manage> <Jobservers…> open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
<Manage> <Schedulers…> opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
<Manage> <Object Types…> opens the -Object Types- window, where the object type definitions can be imported and edited (these are required for associating transformation rules with for the attribute model transformation are defined)
<Manage> <Global Mapping Lists> opens the -Global Mapping Lists- window, where global mapping lists can be created or imported. A global mapping is automatically available for use in any migration set.
<Manage> <Object History…> opens the -History- window where it is possible to look up any object that underwent processing in migration-center. Key steps of the objects lifecycle during the migration can be viewed, such as the objects status, metadata and the exact timestamp at that moment.

The main toolbar provides quick access to frequently used functionalities. These correspond with the same functionalities available from the main menu.

[Scanners] opens the -Scanners- window for creating and managing scanner jobs
[Migration Sets] open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
[Importers] opens the -Importers- window for creating and managing importer jobs
[Jobs] open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
[Jobservers] open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
[Schedulers] opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
[Cascade] Arranges all open child windows in a cascade
[Tile Horizontally] Tiles all open child windows horizontally
[Tile Vertically] Tiles all open child windows vertically

Job Servers

To input/output any data from/to migration-center there needs to be at least one location with a running Job Server defined.

The -Jobservers- window has its own toolbar and right-click context menu containing items that allow the user to create, edit, delete, and monitor the availability of the Job Servers running on the specified Job Server locations.

The location for a Job Server can be specified as the machine’s hostname or fully qualified domain name, if required, or simply as an IP address.

Note: Do not use loopback addresses or the “localhost” name.

The toolbar and the context menu of the -Jobservers- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[Refresh] is useful for rechecking the status of the locations listed
[New] opens a blank -Jobserver- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a location from list

Properties window

To create a new Job Server location

select <New> from the context menu or from the toolbar
Enter the IP address or hostname of the machine the Job Server is running on and the port number it is listening on.
Enter the appropriate port on which the Job Server on that machine was configured

The new location is displayed in the -Jobservers- window. Check the Status column to make sure the required Job Server is listed as Available and thus capable of accepting incoming connections. The Job Server on that particular location is now ready to be used by any input/output adapter.

The default port number is 9700, unless changed during setup.

Creating a scanner and trying to save it without having any Job Servers defined will prompt the user to define one Job Server location and save the scanner with the new created Job Server location.

Scanners

Scanners are responsible for filtering and extracting metadata and content from the source system and inputting this information into the migration-center database. Different types of scanners can be available in a migration-center installation; different types of scanners also offer different configuration parameters, depending on the source system they connect with migration-center. All scanners provide the same basic functionalities, such as extracting documents and folders form the source system and extracting the metadata associated with these objects and storing it in the migration-center database, as well as adding system specific information allowing migration-center to understand and process these objects correctly.

The toolbar and the context menu of the -Scanners- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[History] opens the -History- window where more information about all scans performed by the selected scanner can be viewed. The log files from these scans can also be accessed here.
[View Scanned Objects] will display a grid with the scanned objects in a new window, where these can be viewed, filtered, and exported to a CSV (comma separated values) file. This functionality does not allow any editing to be performed on the objects.
[Run] initiates the run of a scanner with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the scanners listed
[New] opens a blank -Scanner- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected scanner, such as the number of processed objects, the start and ending time and the status of each run.

Double clicking a history entry or clicking the [Open] button on the toolbar with a selected history entry opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

The log files generated by a scanner are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components 3.13\logs

All scanners, regardless of type share some common parameters

Name: must be a unique name for the current scanner

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the scanner depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this scanner does or add some comments related to the scanner

In the History of a scan, or after the objects have been assigned to a Migration set the metadata of these objects can be viewed in the -Scanned Objects- window which can be opened with the [View Scanned Objects] button.

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

The [Export to CSV] button allows the user to export the entire grid in its current configuration (i.e. including any applied filters and visible columns) into a comma separated file. The CSV field separator, and the repeating attribute separator can be specified by the user before exporting the file.

Filtering attributes and (de)selecting columns is neither permanent, nor does it affect the actual data in any way. It only affects the current display and serves to analyze the data on display.

Migration Sets

Migration sets are sets of individual objects but with a common set of transformation and validation rules applied to them. Operations like transformation, validation and import are always performed on such a set of objects and cannot be performed on individual objects that are not assigned to any migration set. Assigning the objects resulting from a scan to a migration set is therefore the first step towards making these objects operable for migration-center’s features.

It is possible to add multiple scans to one migration set, as it is possible to split a scan containing a large number of objects onto several migration sets.

It is also possible to filter the selection of objects before adding it to a migration set based on user specified selection criteria and conditions; objects which do not meet these criteria will not be added to the current migration set and will continue to be available and can be filtered further, added to another migration set, or left unassigned if they are not supposed to be migrated.

Buttons and context menu items:

[Edit Transformation Rules] opens the window where transformation rules are managed and rules are applied
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Transform and Validate] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
[Reset All] resets all objects of in the selected migration set to their initial attribute values, essentially removing any processing performed during the transformation process and resetting status changes resulting from validation and import. The [Reset All] button is available in all views (-Source Objects-, -Processed Objects-, and -Error Objects-).

Imported objects will NOT be reset by default and require an extra checkbox to be checked prior to executing the reset command. This is for security reasons, because resetting imported objects will erase all information linking the source objects to the objects created by migration-center during import, therefore rendering migration-center unaware of having imported the affected objects at all.

Resetting imported objects would typically be used during testing phases of a project and not during production.

It is strongly advised NOT to reset imported objects in cases where dependencies to these objects may exist in other, even future migration sets, such as newer versions of these objects or relations between these and other object which haven’t been migrated yet; this also applies if it is intended to update these objects during future migrations.

[Stop] appears as a button only when the selected migration set is busy being transformed or validated, and only if these processes take longer than 3 seconds for the selected migration set.
[Refresh] refreshes data in all info-columns
[View All Objects] (press drop-down list/button combo) opens 3 windows: Source Documents, Processed Documents, and Error Documents.
[Source Objects] (select from drop-down list/button combo) displays all scanned objects attributed to the respective migration set. Original attributes and attribute values are displayed.
[Processed Objects] (select from drop-down list/button combo) displays the objects after they have been processed by applying the transformation rules.
[Error Objects] (select from drop-down list/button combo) displays objects that failed transformation/validation criteria, or failed to import, indicating the cause
[New] opens a blank -Migration Set- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured migration set from list

Items in columns can be sorted by clicking on the column headers. Clicking the same column header repeatedly switches between ascending/descending sorting order.

In the lower part of the window, migration sets can be filtered by name. This can prove useful if there are a large number of migration sets, such as the migration sets generated automatically by repeated scheduled migration runs.

Migration Set Properties window

The migration set’s properties window opens when creating a new migration set and can be accessed for an existing migration set by double-clicking it or selecting [Properties] button or menu item from the -Migration Sets- windows toolbar or context menu.

The -Migration Set Properties- window also determines the selection of objects assigned to the migration set. The |Filescan Selection | allows scans to be selected and added to the migration set, | Exclude objects by values | offers a simple method of removing excluding objects from the migration set based on selected metadata values, and the | Advanced Filters | page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators.

The resulting set of objects can be previewed by clicking the [Preview objects] button on the right side to open the -Preview- window. The -Preview- window offers an overview of the objects resulting from the selection, and some basic functionality for analyzing these such as viewing attributes for individual objects, selecting the attribute to display or exporting this list to a CSV file.

Common controls

The buttons on the right side of the –Migration Set Properties- window are available on all tabbed pages of this window.

The [OK], [Apply] and [Cancel] buttons are self-explanatory and work as expected.

Trying to save a migration set using the [OK] or [Apply] buttons without having assigned any objects to it will display a warning message to the user. Although usually it doesn’t make sense to save a migration-set without having objects assigned to it, this can be used to define migration sets which will be used as templates, and create new migration sets by copying these. In such a case ignore the warning and save the “empty” migration set.

Otherwise define a selection of objects as described in the following chapters before saving the migration set.

The [Preview objects] button and preview functionality is available on all pages of the –Migration Set Properties- window.

If the selection is ok, the selected scans can be assigned to the migration set using the [Select objects] button in the upper right corner.

Note: Assigning objects to a migration set triggers various status changes and updates to the object’s internal metadata, as well as evaluating any additional filter criteria the user may have specified. This can take some time, especially with large numbers of objects. Therefore, the user will be informed that the selection process is about to run in the background, and the Properties window will close. The selection progress can be monitored in the main -Migration Sets- window. While a selection process is running, the user can continue working with other migration sets or other parts of the applications. Selection is a one-time-only operation and does not need to be performed anytime afterwards.

For a migration set that already has a selection of objects assigned instead of the [Select objects] button there will be a [Deselect objects] button. This will free any objects assigned to the migration set but also remove any metadata generated through transformation rules and status information for objects. Recovery of the information lost if deselecting objects from a migration set is not possible. Use with care!

Deselecting objects form a migration sets also involves various updates to the object’s status and internal metadata, which can be a relatively time-consuming operation for large numbers of objects. The process will therefore be run in the background, similar to the procedure described above for selecting objects.

The |Properties| page

A migration set’s basic properties, including user specified properties like the migration set’s name, the type of object it contains, and an optional description can be set here. Other information managed by the system, such as creation or modification dates or the last operation performed on the migration set are also displayed.

The available object types depend on the adapters available with the respective migration center installation and may vary. Always make sure to select the correct object type since the selection here determines what is displayed on the following pages of the properties window.

The |Filescan selection| page

This page allows the user to select and assign any available scan to the current migration set.

The upper list shows all scans with available objects and some relevant information (i.e. scans that haven’t been fully assigned to other migration sets). Move selected scans to the bottom list to assign them to the migration set by clicking on the up/down arrow buttons or double-clicking a scan run. Multiple scans can be selected and assigned to the current migration set.

Only scans matching the object type set on the previous page (|Properties|) will be listed here. Check the object type selection if the desired scans are not displayed here.

The |Exclude objects by values| page

This page offers a simple way of excluding some objects from the selection based on their values for one or more attributes.

In previous versions of migration-center this page was called |Filters|. The functionality provided is the same in all product versions.

The list on the left displays all available attributes. Selecting an attribute displays a list of its distinct values in the Include list to the right. Moving any of these values to the Exclude list on the far right will exclude all objects with that value from the migration set. Move items between the Include/Exclude lists by double clicking or using the arrow buttons between the lists.

Attributes that have been used to exclude some of their values will turn bold to indicate their status. Remove any excluded values for such an attribute to revert it to its default (no exclusion) state.

The |Advanced Filters| page

This page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators. Use advanced filter expressions to specify exactly which objects should be assigned to the current migration set.

For objects matching a filter expression means these objects will be added to the migration set, not excluded!

The feature works by adding filter expressions and grouping them with AND/OR operators. The list of expressions will be evaluated from top to bottom to determine the remaining subset of objects which can be assigned to the migration set.

An expression always consists of a source attribute, a logical operator or condition, and a user specified value. Individual expressions can be created, added to the list below (click the [Add] button to the right), and then moved up or down in the list or deleted using the respective [Up], [Down] and [Delete] buttons. Any number of expressions can be added.

Conditional operators for working with string, numeric, and date/time type values are provided.

Migration Set Views

One, two, or all three of these three views can be opened at any time for the current migration set.

The filter bar at the bottom of the window is common to all views and offers following functionalities:

Other features common to all views are the following toolbar items:

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Click [Refresh] to reload the content of the current view

Source Objects View

The Source Objects view always displays the unmodified state of the objects in the migration set, i.e. the state of the objects as they were scanned from their respective source system. This view does not provide any editing possibilities for the objects displayed, it serves as a permanent reference for checking the original metadata of these objects.

The Source Objects view also allows identifying and removing duplicate objects. Pressing the [View Duplicate Objects] button will display only the duplicates; duplicates are identified by means of a content checksum and the objects’ creation date, e.g. from all the objects with an identical checksum the one with the oldest creation date will be considered the original, and all others will be considered duplicates. This can be used just to display the duplicates or select and remove them from the migration set using the <Remove documents from migration set> entry on the context menu.

The checksum is computed during scan and must be activated in the scanner. See the respective adapter documentation for more information about using the checksum option in the scanner.

Toolbar items

[View Attributes] displays all available (scanned) attributes of one object only. Previous and following object can also be browsed by using the appropriate buttons.

[Edit Transformation Rules] opens the -Transformation Rules- window.

Items in columns can be arranged (increasing/decreasing) by clicking on the column labels.

[View Duplicate Objects] filters identifies and displays duplicate objects

Context menu items

<View Source Attributes> will display all attributes of that object in a small new window and allows browsing through the list objects by object (back and forward.

<View Relations> will display the relations (if any) between the selected object and other objects. It is also possible to see the full details for each relation, including the objects’ IDs it related to, the status and type of that relation, etc.

<View Value> displays the selected cell’s value in a new window. This is particularly helpful when viewing details of a long value (like folder path for example), or a multi-value attribute with a large number of individual values. It is also possible to copy the value to the clipboard, or toggle wrapping the displayed value in the -View Value- window

Copying the currently selected value to the clipboard function can also be performed directly using the <Copy Value> context menu entry or using the {Ctrl+C} keyboard shortcut

<Filter by Value> will immediately filter the view using the selected attribute value. Use the [Clear Filter] button in the filter bar at the bottom to remove a filter and return to the full list.

Objects can be selected and removed from the migration set by selecting <Remove documents from migration set> from the context menu. Objects removed from a migration set become unassigned and are available to be added to another migration set or ignored if these objects are not supposed to be migrated.

Processed Objects View

Opposed to the Source Objects view, the Processed Objects view displays the processed state of the objects in the selected migration set. An object is considered as having been processed if it underwent transformation (and any other operation following transformation, such as validation or import). This view will display the newly generated attribute values for any objects that have been successfully transformed, and the status the processed objects are currently in (i.e. Transformed, Validated, or (Partially) Imported)

If no transformed objects exist in the currently selected migration set (which implies no validated or imported objects can exist either), this view will be empty. Also, if transformation rules are applied to objects but all objects fail transformation this view will be empty as well (in this particular case all objects would be displayed in the third view, Errors)

In addition to the -Source Documents- view, the -Processed Documents- window features a few more functions.

Transforming, validating, and the resetting objects can be performed directly from this window using the appropriate toolbar buttons.

[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Apply Transformation & Validation] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.

For quick access to the source attributes of any object the -Source Attributes- window can be also accessed from the context menu.

Individual objects can also be reset to their default, unprocessed state using the <Reset selected documents> item from the context menu. Resetting an object will essentially remove any metadata generated through transformation rules, processing and status information from such objects. This context menu entry is slightly different from the [Reset all documents] button on the toolbar, which, as the name implies, operated on the entire set of objects instead of an individual selection of objects.

Resetting imported objects would typically be used during testing phases of a project and not during production.

The Processed Objects view offers some new quick filter options below the common filter bar. A processed object includes any objects with the Transformed, Validated, Importer or Partially imported status; objects can be filtered quickly for any of these states using the appropriate checkboxes. Pressing the [Apply Filter] button is not needed just (de)select the checkboxes corresponding to the required status. These quick filter options also work together with the common filter bar, acting as an additional filter criterion.

In the Processed Objects view it is also possible to edit individual objects’ metadata. The [Edit Attributes] item from the context opens a new window with the currently selected objects attributes ordered as a list instead of columns. In the -Edit Attributes- window it is possible to edit any of the current objects’ attributes manually. The -Edit Attributes- window also allows applying the changes, skipping the current document, or moving to the next document in the list without closing the window.

Any manually edited object will be reset to the Transformed status. This makes a difference for objects which have been validated already; since the user has edited these objects manually, the information the object has been validated with has changed and the object will be reset to the Transformed status and require validation using the new attribute values.

Error Objects View

The Error Objects view displays all objects that failed at some point during the migration. All states an object goes through during migration have a corresponding error state, such as Transformation Error, Validation Error, etc. The Error Objects view collects all objects in any state considered as an error and provides a centralized location for tracking such documents, identifying and correcting such objects. In addition to listing Error Objects and their associated attributes, the error message for causing the object to fail be listed in addition for every object; if the error is related to any particular attributes, those attributes will be colored red and hovering over such an attribute will display an additional message about the cause of the error.

All this information makes it easy to identify and correct the error and transfer the object into the proper states allowing it to be migrated.

The Error Objects view provides the same functionalities as the Processed Objects window, since both views serve the same goal: to prepare objects to be migrated and successfully meet the required conditions.

Errors are not a final state in migration-center. If the error can be fixed, it is also possible to correct the objects affected by that error as well and start a new attempt for migrating affected objects. This procedure can be repeated until all objects with errors have been fixed one way or another:

There are five basic ways of handling errors; one of them should always be able to solve typical errors encountered during migration

If the error is caused by a transformation rule (objects in Transformation Error status), adjust the transformation rule to cover the faulty objects and re-apply it to those objects. Any objects which have already been transformed, validated or transformed successfully will not be affected and will maintain their current status and metadata
If the error cannot be fixed by adjusting transformation rules and the object is in the Transformation Error status, it may be necessary to edit objects manually. Use the <Edit Attributes> window to browse through the objects and edit the attributes manually, skipping any objects where manual editing does not apply.
If the error occurs during import due to external factors (objects in Import Error status) like software or hardware issues, network issues, target system configuration problems, proper access permission, etc.) try to remedy the error first and after that just add the migration set back to the importer and simply try to import it again. Migration-center will attempt to import all objects in the Import Error state automatically and will succeed if the cause for the error has been fixed.
If objects have failed Validation (objects in Validation Error status) check the Object Type definitions and Associations in the Transformation Rules window, make the necessary corrections and re-run the Validation process to validate these objects successfully.
Finally, if none of the above helps, or the objects having failed are not even needed to be migrated they can either be removed from the migration set or left in their respective error states (temporarily or permanently). Unless other objects depend on them, leaving unneeded objects in an error state or removing them from the migration set will not affect other objects.

Transformation rules

The transformation rules are a set of rules that apply to all objects of a migration set, transforming the attributes of the objects inside. Transformation rules determine how the metadata provided with the source objects is going to be altered to generate new or modified attribute values, and how attributes from the source and target object models will be connected. A transformation rule consists of one or several transformation steps, where each step references one transformation function. The final transformation step will return the result for a rule.

The -Transformation rules- window can be opened by pressing the [Edit transformation] button in -Migration Sets- window, or the [Edit rules] button in both -View Processed documents- and -View Error documents- windows. Also, the -Transformation rules- window can be opened by context menu <Edit Transformation rules> the three windows where the button is present.

The -Transformation rules- window has three tabs for managing different aspects of transformation and validation, and a set of common buttons accessible from all tabs. These buttons are:

[Generate attributes] will automatically generate the simplest transformation rules based on all source attributes found for all object in the migration set. These simple rules would enable a 1:1 migration of the objects in the current migration set. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected. This function will be further discussed in the next chapter (Attributes).

[Export] will export the current migration set’s transformation rules to a XML file for the purpose of backing up or transferring the model to a migration set located on another DB.

[Import] imports transformation rules generated by the XML files generated with the previous export function. Warning! Importing rules from a file will replace all rules configured previously!

[Ok] saves changes and closes window.

[Cancel] cancels all changes that have not been applied yet and closes window.

[Apply] saves changes without closing the window.

Rules

The |Rules| page consists of four areas, namely Rules, Rules for system attributes, Rule properties and Transformation methods. The Rules and the Rules for system attributes work the same way: selecting a rule will allow its properties and transformation methods to be edited using the controls on the right side of the window. While the former is meant to allow the user to create, edit, and delete any rule, the latter does not allow rules to be added or deleted by the user, it only allows editing the existing rules, which are predefined system rules defined by the object type of the migration set (the object type selected when creating the migration set). Attributes initially obtained by scanning (DOCUMENTUM repository or file system) are called source attributes. They are used as source for direct transformation or for generating more complex transformation rules from them. Source attributes are accessible in the Transformation methods dynamic area.

Rules list

Rules are not immediately connected to any attribute or property of the target system. In the most general sense, rules have arbitrary names and use various source attributes as input, process these using the specified transformation functions and finally output the resulting value. An actual connection between these rules and the object types and their respective attributes is created on the Associations page, which will be discussed later. This generic approach allows migration-center’s Transformation engine to adapt to a variety of content management systems which use generic concepts such as type of object and attributes/properties.

The [Generate rules] button can be useful for starting work on transformation rules. This feature will generate rules based on any source attributes it can find among the objects from the current migration set. The rules will be named using the respective attribute names and will contain only a basic transformation function. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected.

These simple transformation rules will extract the original attribute values without performing any processing on them and would be suitable for a 1:1 migration. For most real-world migration projects at least some of these transformation rules will require changes to make them suit the requirements though. It is of course possible to ignore this feature altogether and add rules one by one, according to the requirements and possibilities of the project.

Rules for system attributes

The system attribute rules work the same way as user defined transformation rules. System rules are predefined rules and usually refer to key information that needs to be specified for the objects to be migrated and work correctly in the respective target system. System attributes depend on the type of object used in the migration set (specified during creation of the migration set and displayed in the -Transformation Rules- windows’ title bar). System rules cannot be renamed, deleted and new system rules cannot be created by the user; only the transformation rules for the predefined system rules can be edited. Here the full functionality of transformation functions is available, as for the user defined rules.

The migration-centers adapter targeted at Documentum provide following system attribute rules:

“dctm_obj_link” specifies the path(s) where a document should be linked to in the target repository. “dctm_obj_link” is also an example for a system attribute which is defined by and used internally by migration-center and does not require association with an actual target attribute. This means there is no need for an actual attribute named “dctm_obj_link” to exist in either the source or target system.
The system attribute “dctm_obj_rendition” specifies file system location to files which represent renditions. Zero or more renditions can be specified for any object and will be added by migration-center during import. This attribute is also an internal attribute used by migration-center similar to “dctm_obj_link”
“r_folder_path” and “r_object_type” can get their default information automatically (see [Generate attributes]) only inside a Documentum-to-Documentum migration set. In the migration set of a file system scan, this information must be put in by user.

Rule properties

The Rule properties allows specifying the name for a newly created rule or renaming an existing rule. Whether a rule can work with multiple values can also be set using the Multi-Value checkbox. These controls are disabled for editing if a system attribute is selected since system attributes cannot have their properties changed by the user.

Transformation methods

The Transformation methods manages the transformation steps and functions used to perform the actual processing for the selected rule. Each step corresponds to a function. Any number of steps can be added to obtain the desired result for the current rule. At least on step must be added for a transformation rule to work.

The Function drop-down list all available transformation functions. Select a function and add it as a step to by clicking the [Insert] button. Functions will always be inserted last in the list. After inserting a function, it is possible to edit, reorder, or delete steps using the respective buttons.

Inserting a new function as a transformation step will immediately open the function’s properties for editing. Configure the function’s parameters accordingly and click [OK] to save the function and add it to the list.

Function properties can be edited at any time by selecting the corresponding transformation step and clicking the [Edit] button above. This will open the same properties window mentioned above and allows the user to make the required changes.

Selecting a function from the “Function” drop-down menu and hovering mouse pointer over the help-icon (“?”) next to [Insert], a windows tooltip will give the user more information about the selected function.

Error and Warning indicators for rules:

Migration-center always checks rules when saving and looks for invalid references, functions, or unused transformation steps. Rules will be colored red for “Error” or yellow for “Warning” as a result of this verification. Migration-center cannot correct such inconsistencies because the intended purpose of the affected rules is not known to migration-center. It is up to the user to check and correct any issues with rules marked red or yellow.

In the example below, the rule called “extension” has been marked yellow by migration-center’s rule auto verification feature.

The user can select that rule to check the cause of the warning. Once a rule marked as an Error or Warning has been selected the transformation rules will also be displayed in the rule’s properties. The actual transformation step causing the error/warning will also be marked red/yellow. In the example below, migration-center warns because there is an unused function in the list. This information is shown in a tooltip when hovering the cursor over the respective transformation step.

Warnings can usually be ignored and will not affect the result of the transformation rule. However, warnings should only be ignored if it really makes sense; e.g. in the example above the user may want to leave the unused step in there, and switch between the two steps for testing purposes. For production use though it is not advised to leave any unused transformation steps because these will be computed, needlessly adding to the time required for transforming the objects.

Rules marked red, indicating errors, are rarely encountered during normal usage. The highest chance for getting Errors is when importing or copying transformation rules into the current migration set; since the newly imported transformation rules may have been created based on another migration set, they may reference source attributes which do not exist among the current objects and this may raise an error because of the invalid reference. Errors can not only result from references to invalid source attributes, but also from invalid references to previously deleted transformation steps, or wrong parameter combinations for some functions.

Marking rules and transformation steps as errors works exactly the same as described above for warnings.

Errors should never be ignored because they will definitely affect, or even prevent transformation rules from working correctly.

Working with transformation functions

Transformation functions provide the functionality behind transformation rules.

Custom functions can be implemented and used along the standard functions provided by migration-center.

Functions typically have multiple input parameters, with one of them usually being the value that needs processing, and output the processed value which may be entirely different from the source value, depending on what the function is designed to do and what parameters it has specified.

Input parameters can range from referencing a source attribute to referencing the output of a previous transformation step, a value typed by the user to various function specific parameters which determine how that function should process its input our output. The number of parameters varies by function. Some functions (like Sysdate) do not have any parameters at all, while other functions have three or four parameters.

A function’s parameter always has two properties: a type and a value. The parameter’s value is determined by the parameter’s type. Parameters can be of the following types:

Source Attribute: the value will be extracted for each object's attribute individually during the transformation process. For objects which do not have the specified source attribute no transformation will take place.

User Value: acts as a constant value and will be set for all objects to which the current rule applies.

Previous step: references the value returned by a previous transformation function from the current transformation rule and uses it as input. If an input value is multi-value, the index of the value to be processed can be selected or typed into the drop-down list on the far right. By default, all values are processed for a multi-value input

The Index selection (the small dropdown list to the far right) is for working with multi-value (repeating) attributes. By default, any function will operate on all values of a multi-value attribute, but it is possible to specify a particular value by setting its index (i.e. the n-th value of a multi-value attribute) in the Index dropdown list.

In case the parameter in question is referring to a single value attribute selecting an Index will have no effect.

The Index selection list offers the following options:

All is the default option and can be used only for the first parameter and not the subsequent parameters.

A specific index ranging from 1 to n (where n is the maximum number of values for the currently referenced multi-value attribute) can also be specified. In this case the current transformation function will only process the requested value and ignore all other values, effectively returning a single value output from a multi-value input.

The Index selection offers some more options when used with the If transformation function:

It is also possible to specify All as an option for the Return if true or Return if false parameters.

When All is used in first parameter, the If is called once for each value of the first parameter (as long as it has multiple values). The corresponding index value from Return if true or Return if false will be returned after every call. This means the result will have the same number of results as the number of values of first parameter. This is the same behavior as in all other transformation functions when used with index All.

The value Any is exclusive to the If transformation function and can be used to define a condition based on a value whose index is not fixed or is simply unknown to the user within a multi-value attribute and therefore cannot be specified using the All or 1 – n settings

When Any or a specific index 1-n is used for the first parameter, the function is called only once and will return all values specified in Return if true or Return if false. If All is used in a return parameter all values will be returned, otherwise only the specific index will be returned.

Transformation function reference

All standard transformation functions provided with migration-center are described below. Full context-sensitive information is also available in the migration-center Client when inserting or editing any transformation function.

Mapping Lists

Mapping lists can be used as an alternative means of transformation in case the desired result cannot be obtained using transformation functions. Usually mapping user names or permissions falls under this type of data, since there is no way to create a transformation rule which maps a list of several hundred users or permissions onto their corresponding new values individually; mapping lists can of course be applied to any attribute, not just the ones mentioned in the example.

Mapping lists are specific to one migration set and are only available for use within the migration set where they are defined. It is also possible to define so called global mapping lists which are available in all migration sets.

Mapping lists are simple 2-column lists and work based on a key-value concept, where the key is specified in the first column, and the associated value in the second column. If the MapValue transformation function is used in a transformation rule with a given mapping list and matches a key from the mapping list to its input value, it will output the corresponding value from the mapping list as the result.

A mapping list must be referenced by a “MapValue()” transformation function in at least one transformation rule on the |Rules| page to have any effect on the objects of the migration set. A mapping list will not do anything by itself.

“Exact Match” will require an exact match with the specified key if checked and will accept any string as a match as long as the mapped value is included in the key if unchecked. It is recommended to check this option; uncheck it only if needed.

“Case sensitive” will consider a valid match only if the case of the input value matches the key and its case. Usually this would not be used, enable if needed.

Mapping list can be edited in-place if needed, but this is only for short lists or making minor corrections to an existing list. Usually a mapping list would be prepared using an appropriate application such as a spreadsheet, then copied and pasted into migration-center via the context menu available when right clicking into the list area.

The example above illustrates using a mapping list with the “MapValue()” function. Here the mapping list “opened by” is used through the “MapValue()” function in the transformation rule “extension to OPENED BY”. The window below shows the properties for this function.

Migration-center will interpret this transformation rule as follows:

In all objects of the migration set look up the source attribute “extension” and try to match its value to any of the following “mp3, doc, pdf, ppt”; if a match is found, replace the source value with the value corresponding to the matching key specified in the mapping list named “opened by” and output this value as the result.

Global Mapping Lists

Global Mapping Lists are mapping lists independent of any particular migration set. They are thus globally available to all migration sets across the entire application. Use global mapping lists for information which will be used by multiple/all migration sets. Global Mapping Lists can be managed from the main menu <Manage>, <Global Mapping Lists>. The functionalities and properties for creating, copying, deleting, and editing global mapping lists are identical to the ones provided for regular mapping lists, described above.

When using the “MapValue()” function, both the “local” (migration set specific) and the “global” mapping lists are available for use. Global mapping lists can be identified by the word “(global)” appended to their name.

Associations

In the |Associations| page existing transformation rules can be associated with actual object types and attributes from the target system. These associations will provide a means to validate the values generated by the transformation process, and will tell the importer how to assign these values during import

Types

To create a connection from a transformation rule to a target object type there needs to be one transformation rule which outputs the object type’s name. Select this rule from the “Get type name from attribute” drop-down list. For Documentum “r_object_type” is a predefined system rule which server this purpose, but there is no restriction to select a particular rule; any rule will work if it results in something which matches object type names and is not null.

From the “Available type definitions” drop-down list, add the required object type definitions below by pressing [Add]. The object type definitions displayed depend on what is available in the -Object Type Definitions- window. If a required object type definition is not available for creating an association here, Click [OK], add the object type definition, and come back to this page to create an association with it (See chapter 7.8 Object type definitions for details on adding object type definitions to migration-center).

Associations

Selecting an object type definition will populate, the “Target attribute” drop-down list with the attributes of the selected type. The “Rule” drop-down list shows the rules defined on the |Rules| tab. By matching pairs from these two controls and clicking [Add Association] rules can be associated with corresponding attributes. A rule and a target attribute can be associated only once per type.

In case the rules already have names that match attributes of the targeted object types, there is a much quicker way for creating these associations: just click [Auto-Associate]. This button will automatically match rules and attributes by name and add them to the list of associations. Any unneeded rules created this way can be deleted afterwards, and any rules that could not be associated automatically can still be added using the regular method described above.

Note: An association always applies to the currently selected object type in the list, not all of them! If there are multiple object types targeted for a migration, manual/automatic association procedure described above needs to be done for each object type.

Importers

Import is referring the process of transferring content and metadata obtained using the transformation rules to the target system. The importer is the job actually performing this task of outputting data from migration-center or importing data into the target system.

Importers migrate one or more migration sets. Only migration sets with objects that have successfully passed both transformation and validation can be imported.

Similar to a scanner, an importer is a job with a unique name and a set of configuration options, which is started by the user and the corresponding adapter code is executed on the specified Job Server.

Note: for details about the scanners provided with the standard product consult the appropriate documents listed in 1.3 References. For details about customized scanners or scanners provided by third parties please consult the documentation delivered with the respective adapters.

The toolbar and the context menu of the -Importers- window contain the same commands.

[Properties] opens the configuration details of that particular importer. The -Importer- properties window can also be opened by double-clicking an item in the list.
[History] opens the -History- window where more accurate data about all imports performed by the importer in question can be accessed. The log of the scans can be accessed here.
[Run] initiates the run of an importer with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the importers listed
[New] opens a blank -Importer- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected importer, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

The log files generated by an importer are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

All importers, regardless of type share some common parameters

Name: must be a unique name for the current importer

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the importer depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this importer does or add some comments related to the importer

Scheduler

A Scheduler acts as a master process which manages and runs an end-to-end migration fully automated and according to a set schedule.

A scheduler is also a job, and will use preset scanners, transformation rules and importers to run extract, process, and migrate objects to the target system automatically. A full history is available for each run, listing both objects that have been imported successfully as well as any objects which may failed during the process.

There are a number of configuration options regarding the times when such an automated migration should run, ranging from minutes to months, days of the week, or hours of the day. This makes it possible to tailor a schedule according to the needs and possibilities of the project and avoid running performance intensive migration during working hours for example.

The toolbar and the context menu of the -Schedulers- window contain the familiar items: New, Properties, Delete, Run, Refresh.

Double-clicking a scheduler in the list has the same function as the <Properties> menu command.

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

A scheduler can be run immediately if needed, regardless of its actual schedule, through the <Run> command found in the toolbar and the context menu.

Properties

An email report can be scheduled to be sent to a certain email address in case of success or failure of the scheduler processes. For this a valid SMTP-Server and email address must be provided, and at least one of the two checkboxes flagged.

Object Types definitions

Migration-center needs to know the properties of the object types targeted during migration to correctly associate transformation rules with the attributes from these object types and also for validating the values generated by transformation functions. These target object types can be managed in migration-center Client in the -Object Types- window. The window can be opened from the main menu <Manage> - <Object Types…>.

All target object types in use should be defined before attempting to create associations or validate the objects in a migration set. For creating and testing transformation rules it is not necessary to have objects types defined at that point.

Out-of-the-box the product comes with basic object type template definitions for the adapters the customer has acquired licenses for.

Additional user/custom object types can be added to migration-center by the user.

To create a new object type template, click the [New] button in the Object Types toolbar or <New> in the context menu inside the same area. Enter the template’s name in the “Name” field; the name should be exactly the same as the actual name of the object type. The [Delete] button

In the list below the individual attributes and their properties can be defined. Since the process of defining all types and attributes by hand would most likely be too time-consuming it is possible to import an object type directly from a properly formatted CSV file. Object type definitions can be imported using the [Import] button located in the Object templates dynamic area. The imported file needs to be a CSV file (comma separated value) similar to the following example, with each row containing attribute name, data type, minimum length, maximum length, repeating flag, mandatory flag, and an optional regular expression:

id,2,0,0,0,0

first_name,2,0,0,0,0

last_name,2,0,0,0,0

email,2,0,0,0,0

gender,2,0,0,0,0

ip_address,2,0,0,0,0

animal_name,2,0,0,0,0

Valid values for data type are: 0 - Boolean, 1 - Integer, 2 - String, 3 - ID, 4 - Date and Time, 5 – Double.

Some importers provide best practices for extracting the target types from the source systems using common tools. Documentum Importer User Guide and SharePoint Importer User Guide can help you with such suggestions.

The optional regular expression that you can provide for an attribute will be used to validate the attribute values during object transformation and validation. Please provide a rule that matches all valid characters for that attribute. If the attribute value contains any characters that are not matched by the regular expression, its validation will fail. For example, the following regular expression will match all characters that are valid for a SharePoint 2013 file name:

^[^\~\"\#\%\&\*\:\<\>\?\/\\\{\|\}]+$

It will match one or more characters between begin and end of the string that are not in the list of the following characters: ~"#%&*:<>?/\{|}

You can also use the regular expression to enforce a certain data schema for the attribute values. For example, to enforce a schema with three digits followed by a minus followed by five digits you can use the following regex:

^[0-9]{3}-[0-9]{5}$

Object History

Migration-center tracks key stages throughout the lifecycles of object being migrated (such as scan, import, deletion, etc.) and stores information about the exact state of that object at that point in time. The information stored about the objects includes properties such as the status, timestamp, set of metadata, or the ID the object was imported with in the target system. The Object History serves as an internal reference to all objects that have been migrated.

The -History- window can be accessed from migration-center’s main menu <Manage>, menu item <Object History…>

Log files

A complete history is available for any job (scanner, importer, scheduler) from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated for each job can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job

Log files generated by the migration-center Client application in the event of errors can be found in the Client application’s installation folder, e.g. …\fme AG\migration-center Client <version> folder inside the installation path. These logs are not meant for the user but should be sent if requested by our technical product support staff in case there are issues preventing the Client from working normally.

Administration

Oracle Database Creation

Migration-center uses an Oracle database to store information about documents, migration sets, users, jobs, settings and others.

This guide describes the process of creating an Oracle 19c (v19.3.0.0.0) database instance on Microsoft Windows. Even though the description is very detailed, the installing administrator is expected to have a good degree of experience in configuring databases.

Oracle Express Edition can be used for small migration projects that don't requires more resources than available in Express Edition. Check the for more details regarding the required resources.

Database configuration assistant

Open the Database Configuration Assistant from the Oracle program group in the Start menu. The assistant will guide you through the creation steps.

Step 1: Database Operation

Select the Create Database option and click Next.

Step 2: Creation Mode

It is recommended to go with the Advanced configuration.

Step 3: Deployment Type

For the purpose of most installations Oracle Singla Instance Database and General Purpose is sufficient.

Step 4: Database identification

Define the database name here. The name must be unique, meaning that there should not be another database with this name on the network. The SID is the name of a database instance and can differ – if desired - from the global database name. In order to make an easy identification of the database possible, a descriptive name should be selected, for instance „migration“.

You can either create it as a Container DB with a Pluggable DB inside or not, depending on preference.

Step 5: Storage Options

For use with migration-center the “File System” option is sufficient. For more information on the “ASM” method, please refer to the Oracle documentation.

Step 6: Recovery Options

This step describes the database's archiving and recovery modes. The default settings are fine here.

Step 7: Network Options

If a default listener is not already configured, you can select one here or choose to create a new one:

Step 8: Configuration Options

This step contains several tabs on which the configuration options for the new database must be specified.

Memory

For most projects Automatic Memory Management is a good option.

When processing mass data, it may be necessary to change the parameters after some time. This can be done retrospectively in the Oracle Enterprise Manager. To do so, select <Administration / Memory Parameters> in the menu. In the dialog box displayed the settings for the "SGA” and “PGA” values can be changed and saved.

Sizing

At this point the block sizes and maximum number of user processes are defined. The default block size and maximum number of processes can be applied.

Character Sets

The Unicode character set AL32UTF8 must be selected here. This character set will ensure that all special characters of any language will be correctly saved in the database.

Connection Mode

The option Dedicated Server Mode connection type is appropriate for migration-center.

Sample Schemas

Sample Schemas are not needed so leave the option unchecked.

Step 9: Management options

To manage the database, Oracle Enterprise Manager is recommended. In Oracle 19c, databases can be managed centrally using the Grid Control or locally using the Database Control. Since the choice of database management has no influence on the operability of migration-center, this can be configured according to personal preferences or company policies. For the purposes of this guide the local Database Control will not be selected here. E-mail notification and backup can be configured separately at a later time, if necessary.

Step 10: User Credentials

Now the passwords for the Oracle system accounts SYS, SYSTEM and PDBADMIN must be entered and confirmed. These accounts with their passwords are needed for the installation of the migration-center database schema.

These accounts/passwords grant full rights for the new Oracle database and should therefore be chosen and safeguarded carefully. The allocation of different passwords is recommended.

Step 11: Creation Option

Optionally a script can be generated which contains all the settings for the creation of this database. With this script, the setup can be repeated later if necessary. Moreover this script can be used as documentation for the settings used.

Step 12: Summary

The assistant shows a summary of the previous settings. This summary can be stored for documentation purposes in an RSP file.

Clicking on Finish will start the database creation process. This process can take some time.

Additional Information

The Oracle database Configuration Assistant generates the following tablespaces (we assume that unused tablespace is marked to be deleted):

INDX
SYSTEM
TEMP
TOOLS
UNDOTBS
USERS

Summary of the most important Oracle 19c installation parameters

Oracle components:

Oracle Database
Oracle XML DB
Oracle Net Listener
Oracle Call Interface

General options:

Oracle JVM
Enterprise Manager Repository (optional)

Initialization parameters:

Dependent on server

Character sets:

Database character set (AL32UTF8)

Data files:

Use standard parameters

Control files:

Use standard parameters

Redo log groups:

Use standard parameters

Database Administrator’s Guide

Introduction

The present document offers a detailed description of migration-center requirements specifically related to the Oracle database management system it uses as the backend.

The Installation Guide contains a full description of the system requirements and installation steps for all migration-center components and should be consulted first. This document does not substitute the Installation Guide, but completes it with information that might be of importance for Oracle database administrators, such as the various packages, privileges and other Oracle specific features required and/or used by migration-center.

Targeted audience

This document is targeted specifically at Oracle database administrators about to deploy migration-center in corporate environments where databases need to adhere to strict internal guidelines and policies, and where the default database installation procedure used by migration-center may need to be reviewed first, or possibly adapted to meet specific requirements before deploying the solution.

The migration-center Database

The migration-center database stores all information generated during a migration project. This includes job configurations, object metadata, transformation rules, status and history information, etc.

One exception here is the actual document’s/file’s content, which is not stored in the database and is of no relevance to database deployment or configuration.

Requirements

Please consult the migration-center Installation Guide for the complete information regarding installation steps, recommended settings and configuration options for Oracle. Below is just an excerpt of the essential requirements, not a full overview of the installation process.

Hardware Requirements

2-4GB of RAM should be assigned to the Oracle database instance (not the entire Oracle server) where the migration-center schema will be deployed. The Oracle specific memory allocation settings can be left at their defaults.
For storage requirements and sizing of Oracle datafiles see section Oracle Tablespaces below.
There is no differentiation between physical or virtual hardware in terms of requirements; faster physical hosts may be required to compensate for losses in I/O performance or latency if running the database server on virtual machines.

Software Requirements

Oracle RDBMS version: 11g R2 - 19c Architecture: both 32bit or 64bit are supported Edition: Developer or Enterprise Edition Oracle Express Edition (Oracle XE) is not supported in production use due to its limitations!
Ideally a separate Oracle database instance should be designated for use by migration-center alone, rather than an instance shared with other database applications. This would help performance and related troubleshooting if necessary as there wouldn’t be multiple applications and thus multiple potential problem sources to investigate. From a purely technical perspective sharing a database instance with other database applications poses no problems.
The operating system the Oracle Server is running on is of no relevance to migration-center, as all communication with the database happens at a higher level, independently of the underlying OS or hardware architecture. All operating systems officially supported by the Oracle versions mentioned above are also supported by migration-center.
Please consult the appropriate Oracle documents for more information on operating systems supported by Oracle databases.

Oracle Schema

The migration-center database schema can be deployed on any Oracle database instance meeting the requirements and setting described in the migration-center Installation Guide

The schema is named FMEMC. The schema name is fixed cannot be changed; as a consequence it is also not possible to have more than one schema installed on the same database instance.
The schema is home to all database objects created and used by migration-center, such as tables, stored procedures, Java code, etc. As part of the schema the user FMEMC is created and granted access to all necessary packages, objects and privileges (also see section Oracle Privileges). For storing data and indices two tablespaces are also created for use by FMEMC (also see section Oracle Tablespaces).

Oracle Tablespaces

The tablespace meant for storing user data is called FMEMC_DATA and will store information such as job configurations, object metadata, transformation rules, status and history information, etc.
A separate tablespace called FMEMC_INDEX is used for storing indices of indexed fields

Much like the schema name, the tablespace names are fixed and cannot be changed.

Oracle Datafiles

By default each of the two tablespaces mentioned above store information in 2 data files
The data files are set to autoextend by default
The data files can be customized in terms of count, size, storage location, and whether autoextend is allowed or not. With the autoextend option disabled the free space within the data files needs to be monitored and extended accordingly during the migration process to prevent the tables from filling up and stalling the migration process.

The above must be changed in the installation scripts and cannot be set through the regular setup program’s GUI.

The primary factor for sizing the Oracle data files is the number of objects planned to be migrated using the respective database instance, as well as the number of attributes stored per object, and the length of the respective attribute values; these factors can (and do) of course vary from case to case.

For more comprehensive information on typical migration project sizes (from small to very large) and the requirements in terms of storage space resulting from there, as well as general recommendations for sizing the database in preparation of an upcoming migration project please consult the Sizing Guide.

Oracle Privileges

This chapter details the privileges required by migration-center for accessing the Oracle packages and various functionalities it needs to work properly. By default these privileges are granted to the FMEMC user during installation. After database installation has completed successfully, it is possible to log in with the user FMEMC and connect to the migration-center database in order to start using the solution.

Privileges Required For Installation

The regular installation process using the setup program with GUI requires logging on as SYS. Since this might not be possible in some environments, there is the alternative of customizing the critical part of the setup and having an administrator execute the appropriate scripts.

This would generally involve having an administrator customize and runs the scripts for creating the user FMEMC, tablespaces, data files, etc., and then run the rest of the setup by running the setup program using the previously created user FMEMC for connecting to the database instead of SYS. In this case the below permissions must be granted to user FMEMC in order for the setup program to determine whether migration-center tablespaces and/or data files already exist on the instance where the database is supposed to be installed.

GRANT SELECT ON SYS.DBA_DATA_FILES TO FMEMC;

GRANT SELECT ON SYS.DBA_TABLESPACES TO FMEMC;

Privileges for Various Functionalities

Encrypting passwords saved by migration-center
Read current database configuration from view v$instance (host name and database instance name)
Execute core migration-center job-based functionalities such as transformation, validation, or scheduled migration as native Oracle Jobs
migration-center Transformation Engine (process migration-center data by applying transformation rules and altering object metadata)

GRANT CONNECT, RESOURCE TO FMEMC;

GRANT CREATE JOB TO FMEMC;

GRANT SELECT ON SYS.V_$INSTANCE TO FMEMC;

GRANT CREATE VIEW TO FMEMC;

GRANT SYS.DBMS_LOCK TO FMEMC;

GRANT SYS.UTL_RAW TO FMEMC;

GRANT SYS.UTL_ENCODE TO FMEMC;

GRANT SYS.DBMS_OBFUSCATION_TOOLKIT;

These privileges must be granted for migration-center to operate normally. It is not possible for the solution to offer even the core functionalities without having been granted the above privileges!

Privileges for migration-center Scheduler

Start scheduled migration jobs at the specified time using Oracle’s built-in scheduler GRANT SYS.DBMS_SCHEDULER TO FMEMC;
Establish network communication from scheduler to Job Server over TCP GRANT SYS.UTL_TCP TO FMEMC;
Allow the scheduler to send notification emails to configured users about outcome of scheduled migration jobs GRANT SYS.UTL_SMTP TO FMEMC;

Privileges on packages SYS.UTL_TCP and SYS.DBMS_SCHEDULER must be granted for the scheduler functionality to work. The SYS.UTL_SMTP package is required for sending notification emails to configured users summarizing the outcome of scheduled migration jobs. Since this can also be checked directly in the mc Client application at any time, sending notification emails is an optional feature, therefore access to the SYS.UTL_SMTP package is not mandatory.

Preparing the Database for Setup Using Scripts

As mentioned above, parts of the installation process can be controlled via installation scripts. These can be used for preparing the database manually by an administrator and finally running the regular setup program connected with the migration-center user FMEMC instead of SYS which may not be allowed to be used.

Database scripts

The script files used during setup are located in the Database\util folder of the migration-center installation package. These files can be used to adapt the setup procedure to some degree to specific requirements of the customer’s environment. The individual files and their role in the migration-center database installation process are described in the table below.

Instructions for preparing the database manually and performing the installation

Decide whether to use a custom or the default location for the data files. Pick create_tablespaces_custom_location.sql or create_tablespaces_default_location.sql accordingly.
Review/adapt the selected tablespace creation script
Execute the tablespace creation script
Review/adapt the user creation script (do not change user name!)
Execute the user creation script
Run the migration-center database installation program (Database\InstallDataBase.exe) and proceed as described in the installation guide. Exception to the Installation Guide: log in as the previously created user FMEMC instead of user SYS!
Verify the database installation log after installation
Log on to migration-center using the migration-center Client and verify basic functionalities
Use the drop_fmemc_schema script to drop the entire schema in case of an unsuccessful installation, or if the schema needs to be removed for other reasons. WARNING! This will delete all migration center data from the database instance! Make sure the schema has been backed up before dropping a schema that already contains user data created by migration-center.

Sizing Guide

Introduction

The present document offers a guideline for sizing and configuring migration-center components in different migration scenarios. Due to the complexity of the migration projects, enterprise environments and applications migration-center interacts with, this guide does not provide exhaustive information, but it offers best practice configurations accumulated in more than 100 migration projects.

The Installation Guide and Database Administrator’s Guide contain a full description of the system requirements and installation steps for all migration-center components and should be consulted first. This document does not substitute the documents mentioned above but completes it with information that might be of importance for getting the best migration-center performance.

Targeted audience

This document is targeted specifically to the technical persons that are responsible with installation and configuration of migration-center components.

Common configurations and common sizing guidelines

This chapter describes the configurations and sizing guidelines that apply to any kind of migration project.

Client

The following hardware and software resources are required by MC client in any of the migration scenarios described in the next chapters.

Server components (Job Server)

Database

Small projects (up to 500,000 objects)

Sizing and configuration tips

No special hardware or software requirements
All MC components can be run on the same machine

Deployment overview

For such a small migration projects, where up to 500,000 objects need to be migrated, there are no special hardware or software requirements. Basically, all three MC components can be run on a single desktop computer or on an adequately sized virtual machine. In this case the machine should have enough physical memory for OS, Oracle server, MC components and other applications that may run on that machine. The recommended processor is a dual or quad core having a clock rate of minimum 2.2 GHz.

Oracle instance

Standard database installation can be followed (as it is described in “migration-center installation guide”)
RAM: 4 GB of total memory for the instance.
Use “Automatic memory management”. This can be chosen when creating the Oracle instance.

Medium migration projects (500,000 – 5 million objects)

Sizing and configuration tips

Dedicated Oracle Server machine
Two MC Job Servers for a better scalability

Deployment overview

It is recommended to use a dedicated machine for the Oracle instance needed by MC.

For a better scalability of scanning and importing jobs or when migration timeframe is short, it’s recommended to deploy the Server Components (Job Server) on two machines. In this way you may run multiple scanner and importers in parallel speeding up the entire migration process. The performance of scanners and importers is dependent by the Source and Target System so if one of those systems performs slowly, the migration process can be speeded up by running multiple scanners and importers in parallel. If necessary, the number of deployed Jobservers can be extended.

Oracle instance

The host should not necessarily be a server machine. A well sized desktop machine would be enough.
CPU: Quad Core, min. 2.5 GHz
8 GB of memory allocated for the instance
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible, the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 200 MB.
It is very recommended the instance has at least 3 Redo Log files having in total a minimum size of 1 GB. This is important when multiple big transformation and validation jobs are run by MC because those jobs update a big number of database rows that require enough redo log space. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Large migration projects (5 million – 15 million objects)

Sizing and configuration tips

Dedicated Oracle Instance running on a Server machine.
Three or more MC Job Servers for a better scalability

Deployment overview

It is recommended to use a dedicated Oracle Instance running on server hardware.

For a better scalability of scanning and importing jobs three or more instances of Server Components (Job Server) need to be deployed. The performance of scanners and importers is dependent by the Source and Target System so if one of those systems performs slowly, the migration process can be speeded up by running multiple scanners and importers in parallel. If necessary, the number of deployed Job Servers can be extended.

Oracle instance

A dedicated server machine is required. It is recommended to use a dedicated Oracle instance for running only Migration Center but no other applications.
CPU: 4-8 Cores, min. 2.5 GHz
16-32 GB of memory allocated for the instance
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 400 MB.
Make sure that instance has at least 4 Redo Log files having in total a minimum size of 1.5 GB. This is important when multiple big transformation and validation jobs are run by MC because those jobs update a big number of database rows that require enough redo log space. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Very large migration projects (more than 15 million objects)

Sizing and configuration tips

Multiple Dedicated Oracle Instances running on a Server machine or use an instance on a high-performance Oracle cluster
Four or more Job Servers.
A file server or a storage server used for temporary storing the content.

Deployment overview

The deployment for the migration of a very large number of objects should be planned carefully. In this case it is recommended to use multiple MC databases. In most cases it is not advised to migrate more than 10 million objects with a single Oracle instance even if the hardware is sized accordingly. There are several reasons for using multiple database instances:

The data from different sources is not mixed up in a single instance helping the user to handle easier the errors that appear during migration
The transformations and validations will be better scaled on multiple database instances
Facilitate the work of multiple migration teams

A dedicated file/storage server should be shared by all Job Servers for storing and accessing the content during migration. This will help the migration in the way that set of objects scanned with any scanner can be imported with any importer.

Oracle instance

Several dedicated server machines are required. For each machine it is recommended to use a dedicated Oracle instance for running only Migration Center but no other applications.
CPU: 4-8 Cores, min. 3.0 GHz
Minimum 16 GB of memory allocated for each instance.
Use “Automatic memory management” in order for the instance to tune its target memory size, redistributing memory as needed between the system global area (SGA) and the instance program global area (instance PGA). If that is not possible the recommendation is to allocate as much as possible for SGA memory (especially for buffer cache) and keep PGA memory in the range of 600 MB.
Is recommended that FMEMC_DATA tablespace to be split on multiple physical data files stored on different physical disks in order to maximize the performance for a specific hardware configuration.
Make sure that instance has at least 5 Redo Log files having in total a minimum size of 2 GB. This is important when multiple big batch jobs (transformations, validations, scanners, importers) jobs are run by MC because those jobs update a big number of database rows in quite a short time and therefore redo log space should be sized accordingly in order to prevent redo log wait events. You may get the information about existing redo log files by running the query: SELECT * FROM v$log;

Documentum security enhancements

Introduction

The present document offers a detailed description of the Documentum security enhancements introduced with version 3.2.4 of migration-center.

The additional security features provide means for system administrators to define an additional layer of security between migration-center and Documentum repositories.

Some of the features offered are:

delegating access to users without having to give them superuser privileges and credentials directly on the respective repositories
controlling the type of operation a user is allowed to perform on a given repository (import and/or export)
controlling which paths a user can access to perform the above actions on a given repository
controlling where a user is allowed to export scanned documents to
limit the validity or duration of a security configuration by setting a date until which the configuration is considered valid
encrypting any passwords saved in the security configuration file

The implementation and configuration of the security enhancement features is transparent to end-users working with migration-center. End-users working with migration-center will be notified through on-screen messages if they trigger actions or use configurations which conflict with any enhanced security settings.

Usage of the enhanced security features is also optional. Removing or renaming the main security configuration file will disable the feature and will revert migration-center to its standard behavior when working with Documentum repositories (as described in the and user guides).

Targeted audience

The security features as well as this document is targeted specifically at system administrators managing Documentum Content Servers and access to the respective repositories, especially through migration-center.

Technical Information

System requirements

Implementation and integration with migration-center

The Documentum security enhancements features are implemented as an additional, optional module which integrates with migration-center’s Documentum Scanner and Documentum Importer.

The presence of the Documentum security enhancements module will be detected by migration-center automatically, and if a valid configuration exists the settings within will apply every time a Documentum Scanner or Documentum Importer job is executed.

The Documentum security enhancements module is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder. This folder contains the code package, sample configuration file, and tools used by the feature.

There is one tool for encrypting passwords to be stored in the configuration file, and another tool for validating the configuration file’s structure. The configuration file itself is a human-readable and editable XML file. The tools and configuration file will be described in detail in the following chapters.

Disabling Documentum Security Enhancements

The Documentum security enhancements feature is disabled by default and will become active only after a correct configuration file has been created and provided by the system administrator.

Should it be required to disable this feature after it has been configured and is in use, this can be achieved easily using either one of the below approaches:

Rename, move, or delete the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder
Rename, move, or delete the mc-dctm-security-config.xml file located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder

As always, consider backing up any files before making changes to them.

Changes to the security configuration file, deleting, renaming or moving it, will not affect currently running Documentum Scanner or Documentum Importer jobs. Changes will only take effect for jobs started afterwards

General Security Considerations

Since the Documentum security enhancements feature’s folder (<migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config) can contain sensitive information, it is advised to secure this folder using the file system and network security and access policies applicable to such information for the environment where migration-center is deployed in order to prevent unauthorized access.

At the same time the user account used for running the migration-center Jobserver service must be allowed access to this folder, as any Documentum Scanner or Documentum Importer will run in the context of the migration-center Jobserver, thus requiring access to the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and its contents.

Configuration

Configuration file

The configuration file controls all aspects of the Documentum security enhancement features. A valid configuration file needs to be provided by the system administrator configuring migration-center for the feature to work. As long as no valid configuration file exists, the feature will be disabled and will have no effect.

The configuration file must be named mc-dctm-security-config.xml and must be located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder.

Since the exact configuration depends on the customer’s environment, a preconfigured file cannot be delivered with migration-center. Instead a sample file illustrating the structure and parameters available for configuration is delivered and can be used as a starting point for creating a valid configuration. The sample file is named mc-dctm-security-config-sample.xml and is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder.

The sample configuration file can be copied and renamed to mc-dctm-security-config.xml to create the base for the actual configuration file, which will be edited by a system administrator.

General Information regarding XML

The configuration file is a human-readable XML file and can be edited with any text editor, or application providing XML editing capabilities. In order for the configuration file to be valid it will have to conform to XML standards in terms of syntax and encoding, something that should be considered when editing the file using a text editor. Migration-center will validate the file and refuse operation if the file structure or its contents are found to be invalid.

Special characters

Certain special characters need to be escaped according to the XML syntax in order to be entered and interpreted correctly as values. See below a list of these characters:

Encoding

The XML standard allows many different types of encodings to be used XML files. Encodings affect how international characters are interpreted and displayed. It is recommended to use the UTF-8 encoding, which can represent all international characters available today in the UTF character set.

Note that the (mandatory) XML header tag <?xml version="1.0" encoding="UTF-8"?> does not set the encoding, it merely specifies that the current file is supposed to be encoded using UTF-8; the actual encoding used is not directly visible to the user and depends on how the file is saved by the application used to save it after editing. Please consult your text editor’s documentation about saving files using specific encodings, such as UTF-8.

Configuration file parameters

This paragraph describes the structure of the Documentum Enhanced Security configuration file. It will list all available configuration elements, their allowed values and whether the respective parameter is mandatory to be set or not.

The <configuration> element

A <configuration> block defines an entire configuration block controlling access of one user to a repository. All other parameters which apply to that repository and user must be contained in the same <configuration> block.

Multiple repositories and users can be configured by creating multiple <configuration> blocks in the configuration file.

Attribute repository_name

repository_name is an attribute of a <configuration> element and defines the Documentum repository the current configuration block applies to.

Attribute migration_user

migration_user is an attribute of a <configuration> element and defines the migration user who will be granted access to the repository. This user will need to be configured in any Documentum job (scanner or importer) meant to access the repository defined in repository_name.

Attribute action_allowed

action_allowed is an attribute of a <configuration> element and defines what type of action a user is allowed to perform on the repository defined in repository_name. A user may be allowed to scan, import, or perform both actions on the repository.

Element effective_date

effective_date is a child element of configuration. effective_date sets a date until which the current configuration is allowed to be executed. If the set date is exceeded when a job is started, the job will not be permitted to execute and the user will be notified about the configuration having expired.

Element super_user_name

super_user_name is a child element of configuration. super_user_name defines a Documentum superuser which will be used to connect to the repository defined in repository_name. The migration-center user running the job will not need to know the Documentum superuser’s name or password, this will only be known and used internally by migration-center.

Element super_user_password

super_user_password is a child element of configuration. super_user_password defines the password for the Documentum superuser which will be used to connect to the repository defined in repository_name. The migration-center user running the job will not need to know the Documentum superuser’s name or password, this will only be known and used internally by migration-center.

Element migration_user_password

migration_user_password is a child element of configuration. migration_user_password defines the password for the migration user who will be granted access to the repository. This password will need to be configured in any Documentum job (scanner or importer) meant to access the repository defined in repository_name.

Element allowed_paths

allowed_paths is a child element of configuration. allowed_paths defines paths the migration user will be allowed to access in the specified Documentum repositories. The allowed_paths element is optional; if this element is omitted, access to all folders will be granted (whether access will actually succeed depends on the respective Documentum repositories permissions of course).

Element allowed_export_locations

allowed_export_locations is a child element of configuration. allowed_export_locations defines local paths or UNC paths (network shares) where the migration user will be allowed to export content from Documentum repositories. This (these) path(s) will need to be used in the Documentum scanner’s configuration as exportLocation (please see the Documentum Scanner User Guide for more information about the Documentum Scanner and the exportLocation parameter).

Element path

path is a child element of allowed_paths or allowed_export_location. It specifies individual Documentum paths (when used with allowed_paths) or local file system paths or UNC paths (network shares) when used with allowed_export_locations.

Supporting Tools

Password encryption tool

Passwords must be encrypted when saved to the configuration file. Encryption is performed using the supplied password encryption tool. The password encryption tool is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and can be executed using encrypt-tool.

Usage:

Enter password
Confirm password
Press [Encrypt] button (password and confirmed password must match in order to proceed)
Password is encrypted and displayed
Copy encrypted password
Paste encrypted password in migration_user_password or super_user_password element of configuration file

Configuration file verification tool

The configuration file must be a valid XML file. For convenience a verification tool is provided to check the validity of the configuration file against an XML Schema Definition file.

Migration-center will also validate the file internally and refuse operation if the file structure or its contents are found to be invalid.

The configuration file verification tool is located in the <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config folder and can be executed using check-configuration.

Sample output for a valid configuration file

C:\Program Files (x86)\fme AG\migration-center Server Components 3.13\lib\mc-dctm-adaptor\security-config>check-configuration.cmd

******************************** * Configuration OK * ********************************

Sample output for an invalid (corrupted) configuration file

C:\Program Files (x86)\fme AG\migration-center Server Components 3.13\lib\mc-dctm-adaptor\security-config>check-configuration.cmd

******************************** * Configuration FAILED * ******************************** de.fme.mc.common.MCException: An error has occured when loading security configuration: org.xml.sax.SAXParseException; systemId: file:///C:/Program%20Files%20(x86)/fme%20AG/migration-center%20Server%20Components%203.13/lib/mc-dctm-adaptor/security-config/mc-dctm-security-config.xml; lineNumber: 5; columnNumber: 29; The element type "effective_date" must be terminated by the matching end-tag "</effective_date>".

Sample configuration file

A sample configuration file is provided in <migration-center Server Components installation folder>/lib/mc-dctm-adaptor/security-config/mc-dctm-security-config-sample.xml.

The contents of this file are also listed below for reference.

Scanners

Amazon Comprehend Enrich Scanner

Introduction

The Amazon Comprehend Enrich Scanner is one of the source adapters available in migration-center starting with version 3.17. It is a special adapter which enhances the objects scanned by another source adapters with some data computing using the Amazon Comprehend. The supported Comprehend classifiers: Dominant Language Classifier, Entities Classifier and Custom Classifier.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run, a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

Amazon Comprehend Enrich Scanners can be created, configured, started, and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Known Issues and Limitations

The scanner will extract text in enrich mode even if the scanner was run in simulation mode before.
The entities and language classifiers can be run in the same scan run, but the entities classifier will not take into consideration the language extracted by the dominant language classifier. The attribute generated by the language classifier can be used by the entities classifier if it will be run before the entities classifier. More information about the way to use the source attribute as entities language attribute is presented in .

Scanner Properties

To create a new Amazon Comprehend Enrich Scanner job click on New Scanner button and select "AmazonComprehendEnrich" from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the Amazon Comprehend Enrich Scanner parameters.

The Properties window of a scanner can be accessed by double-clicking the scanner in the list or selecting the Properties button or entry from the toolbar or context menu.

Common Scanner Parameters

Amazon Comprehend Enrich Scanner Parameters

Classifiers Configuration

The classifiers are configured using an XML file. The structure of this file is a predefined one and allows the user to configure the classifiers as much as possible.

The supported classifiers are divided into two types: standard classifiers and custom classifiers. There is a predefined XML structure for each classifier type. An example of this configuration file can be found in \fme AG\migration-center Server Components <Version>\lib\mc-aws-comprehend-scanner\classifiers-config.xml.

For every classifier, you can specify if the score should be displayed by using the XML attribute "dispayScore". You need to specify this attribute just if you want to have the score as an attribute in migration-center otherwise, the attribute can be omitted because the default value is false.

Standard Classifier

The standard classifiers are split into two supported classifiers and the difference between them is made using an XML attribute named "type".

Dominant Language Classifier

The structure of this classifier is presented in the following block. The "threshold" XML element is mandatory and is used to filter the values. If the score for a specific language is lower than the threshold value then the language is not saved on database.

Entities Classifier

The structure for the entities classifier is presented in the following block.

The XML sub-elements are:

threshold - is used to filter the entities. If the entity score is less than the threshold value then the entity will not be saved in the database.
language - is a mandatory parameter used to specify the language of the documents. If the user has documents with different languages then the user is allowed to use a source attribute to specify the language. The attribute name should be prefixed with $ character, eg. $aws_language.
entityRecognizerArn - is used to specify the custom entity classifier instead of the standard one.
entities - specify the entities that will be saved on the database. If the entity is not present in the entities list, then the attribute will be ignored by the scanner.

Custom Classifier

The Custom Classifier is used to classify documents using custom created categories. The scanner allows users to use multiple custom classifiers in the same scan run.

The XML sub-element "classifierEndpointArn" is mandatory and specifies the Amazon Resource Names of the custom classifier.

The "threshold" sub-element is to filter the classes. If the class score is lower than the provided value for the threshold, then the attribute will not be saved on the database. The attribute name on the database will be "aws_className_awsJobId".

Using the Amazon Comprehend Enrich Scanner

The Amazon Comprehend Enrich Scanner can be run in two modes: simulation mode and enrich mode.

We recommend you to run the scanner in simulation mode to analyze the cost before running it to extract the Comprehend attributes.

Simulation Mode

The parameter "executeClassifiers" should be not checked when you want to run the scanner in simulation mode. To see the information generated by the scanner, the parameter "loggingLevel" should be set to 3 or 4.

The scanner extracts the text from documents locally and computes the number of characters and units to help the user to estimate the cost of classifiers execution.

The information generated during execution is present in the report log. An example of a report log is present in the following image.

Enrich Mode

To run the scanner in enrich mode you need to check the parameter "executeClassifiers".

The first step that the Amazon Comprehend Enrich Scanner does is to extract locally the text from documents. After that, the text files are uploaded to S3 on inputS3Uri. The classifiers jobs are fired and the result of those are saved in S3 on outputS3Uri. The scanner downloads the files and saves the results on database.

The following image presents the attributes in the database after one scan run on enrich mode with standard entities classifier and dominant language classifier.

Tika Configuration

The Tika library is used by the scanner to extract the text from documents.

The scanner provides a tika configuration file that contains all necessary parsers to extract the text from all office documents. The user can modify the configuration file if more tunings are wanted. The file is located on \fme AG\migration-center Server Components <Version>\lib\mc-aws-comprehend-scanner\tika-config.xml.

The "OOXMLParser" is used for office documents like docx and the "PDFParser" is used for pdf documents. The default configuration provided by the Tika library will be used for other documents type.

Tesseract OCR Configuration

After you installed the Tesseract you need to complete the TesseractOCRConfig.properties file with tesseractPath and tessdataPath. Example:

By default, the Tesseract is disabled. If the user wants to enable the Tesseract, the following steps should be followed:

Open tika-config.xml and remove from DefaultParser the line <parser-eexclude class="org.apache.tika.parser.ocr.TesseractOCRParser"/>

Change the value of the ocrStrategy XML element of PDFParser with ocr_and_text.

Additional Configuration

For configuring some additional parameters that will apply to all scanner runs, a configuration file (internal-configuration.properties) provided in the folder …\lib\mc-aws-comprehend-scanner. The following settings are available:

Logs Files

A complete history is available for any Amazon Comprehend Enrich Scanner job is available from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed for every classifier and errors that occurred during runtime.

Log files generated by the Amazon Comprehend Enrich Scanner can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Alfresco Scanner

Introduction

The Alfresco Scanner allows extracting object such as document, folders and lists and saves this data to migration-center for further processing. The key features of Alfresco Scanner are:

Extract documents, folders, custom lists and list items
Extract content, metadata
Extract documents versions

Scanner is the term used in migration-center for an input adapter. Using a scanner such as the Alfresco Scanner to extract data that needs processing in migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

Scanners and importers work as jobs that can be run at any time, and can even be executed repeatedly. For every run a detailed history and log file are created. Multiple scanner and import jobs can be created or run at a time, each being defined by a unique name, a set of configuration parameters and a description (optional).

Install/Uninstall Alfresco Scanner

Alfresco Jobserver

The Alfresco adapters are not included in the standard migration-center Jobserver but it is delivered packaged as Alfresco Module Package (.amp) which has to be installed in the Alfresco Repository Server. This amp file contains an entire Jobserver that will run under the Alfresco's Tomcat, and contains only the Alfresco adapters in it. For using other adapters please install the regular Server Components as it is described in the and use that one.

The following versions of Alfresco are supported (on Windows or Linux): 4.0, 4.1, 4.2, 5.2, 6.1.1, 6.2.0. Java 1.8 is required for the installation of Alfresco Scanner.

To use the Alfresco Scanner, your scanner configuration must use the Alfresco Server as a Jobserver, with port 9701 by default.

Install Alfresco Scanner

The first step of the installation is to copy mc-alfresco-adaptor-<version>.amp file in the “amps-folder” of the alfresco installation.

Before doing this, please backup your original alfresco.war and share.war files to ensure that you can uninstall the migration-center Jobserver after successful migration. This is the only way at the moment as long the Module Management Tool of Alfresco does not support to remove a module from an existing WAR-file.

The Alfresco-Server should be stopped when applying the amp-files. Please notice that Alfresco provides files for installing the amp files, e.g.:

C:\Alfresco34\apply_amps.bat (Windows)

/opt/alfresco/commands/apply_amps.sh (Linux)

Due to a bug of the Alfresco installer under Windows, please be careful if the amp installer via apply_amps.sh works correctly! Under Alfresco 3.4, the file apply_amps.bat must be location in the alfresco location and not in the subfolder bin!

Uninstall Alfresco Scanner

The Alfresco Scanner can be uninstalled by following steps:

Stop the Alfresco Server.
Restore the original alfresco.war and share.war which have been backed up before Alfresco Scanner installation
Remove the file mc-alfresco-adaptor-<version>.amp from the “amps-folder”

Alfresco Scanner Properties

To create a new Alfresco Scanner, create a new scanner and select Alfresco from the Adapter Type drop-down. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type. Mandatory parameters are marked with an *.

The Properties of an existing scanner can be accessed after creating the scanner by double-clicking the scanner in the list, or selecting the Properties button/menu item from the toolbar/context menu. A description is always displayed at the bottom of the window for the selected parameter.

Multiple scanners can be created for scanning different locations, provided each scanner has a unique name.

Common scanner parameters

Alfresco Scanner parameters

The configuration parameters available for the Alfresco Scanner are described below:

History, Reports, Logs

A complete history is available for any Alfresco Scanner job from the respective items’ –History- window. It is accessible through the [History] button/menu entry on the toolbar/context menu. The -History- window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Alfresco Scanner can be found in the Alfresco log folder on the machine where the job was run, e.g. C:\Alfresco\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

CSV & Excel Scanner

Introduction

The CSV & Excel Scanner is one of the source adapters available in migration-center starting with version 3.9. It scans data from CSV and MS Excel files and creates corresponding objects in the migration-center database. If the scanned data also contains links to content files, the CSV & Excel scanner can link those files to the created objects as well.

Scanner is the term used in migration-center for an input adapter. Using a scanner module to read the data that needs processing into migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

CSV & Excel Scanners can be created, configured, started, and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Prerequisites

When scanning Excel files the “Write Attributes” permission is required otherwise the scanner will throw an “Access is denied” error.

Limitations

When attempting to scan Excel files with a large number of rows and/or columns the UI might freeze until the following error is thrown:

ERROR executing job: Error was: java.lang.OutOfMemoryError: GC overhead limit exceeded

This is a limitation of the Apache POI API, and the recommendation is to convert the excel file into a CSV file.

CSV & Excel Scanner Properties

To create a new CSV & Excel Scanner job click on the New Scanner button and select “CSV/Excel” from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the CSV & Excel Scanner parameters.

The Properties window of a scanner can be accessed by double-clicking the scanner in the list or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

CSV & Excel scanner parameters

Using the CSV and Excel Scanner

The CSV & Excel scanner supports two operation modes: normal and enhancement mode. In normal mode, you can scan objects from a CSV or Excel file and those objects are saved in the corresponding scan run as new, distinct objects in the MC database. In enhancement mode, you can scan data from a CSV or Excel file that is added to an existing scan run, i.e. it enhances an existing scan run with additional data.

Normal operation mode: Scanning CSV and Excel files

In normal operation mode, you can scan objects without content, objects with content, and objects with versions from a CSV or Excel file.

Scan objects without content

In order to scan objects without content, you just specify the path to the CSV or Excel file and the name of the column that contains the unique source ID of the objects. For example:

The screenshot below shows an excerpt of CSV file you would like to scan.

You would then enter the path to the CSV file in the “filePath” parameter and enter “id” as value in the “sourceIdColumn” parameter (because the column named “id” contains the unique IDs of the objects in the CSV file).

Scan objects with content

If your source file contained a column with a file path, as seen in the next screenshot, you can enter that column name (“profile_picture” in the example) in the “contentPathColumn” in order to scan that content file along with the metadata of the object.

Scan objects with versions

In order to scan objects with versions, your source file needs to contain two additional columns: one column that identifies all objects that belong to a specific version tree (“versionIdentifierColumn”) and another column that specifies the position of an object in the version tree (“versionLevelColumn”).

Starting with version 3.16, the scanner allows adding new versions to an existing version tree (scanned in a previous run) just by setting the specific root id.

Note: If the last version was updated, the new scanned versions will be related to the updated object.

In the example below, the source file contains metadata of documents that consist of several versions each. Each document has a “title” and a “document_id”. Each version has a “version_id” and a “content_file”. The combination of “document_id” and “version_id” must be unique. Since the content file path is unique for each version in this example, you could use that column as “sourceIdColumn”.

A valid configuration to scan the example source file could look like this:

Enhancement operation mode: Adding data to existing scan runs

There are many document management applications, which reference data in third-party-systems. If you want, for example, to archive documents from such an application, it might be useful to store that referenced data also in the archive records, in order to have independent, self-contained archive records. To achieve this, you could (1) scan your document management system with the appropriate scanner, (2) export the data from the third-party-system into a CSV file, and (3) enhance the scan run from step (1) with the data from the CSV file using the CSV & Excel scanner in enhancement mode.

The source file for an enhancement mode scan needs a column that stores the source ID of the corresponding object, i.e. the object that will be enhanced with the data in the source file, in the specified scan run.

For example, the following table shows an excerpt of an existing scan run. The source ID of the objects is stored in the column “Id_in_source_system”. In this case, the ID is the full path of the files that were scanned by the scan run.

The source CSV or Excel file for the enhancement scan should contain all the data that you would like to add to the previously scanned objects and a column with the source ID of the corresponding object, as shown in the following table:

In order to run the scanner in enhancement mode, you need to enter a valid scan run ID in the parameter “enrichMetadataForScanRun”. You can find that ID in the -History- view of a scanner:

The “sourceIdColumn” parameter should contain the name of the column with the source ID. That would be “source_id” in the example above.

You can provide an optional prefix for the columns that will be added to the scan run in the “enrichMetadataPrefix” parameter. This is helpful in the case when your source file contains columns with names that already exist in the specified scan run.

Of course you can enhance a certain scan run several times with different source files.

Scan multi-value fields

The scanner allows splitting values from one or multiple columns in multiple distinct values based on a delimiter.

For example, in this particular case, if you want to split the values of columns Record_Series and Classification, all you need to do is to add the column names in the multivalueFields parameter and set the "|" delimiter in multivalueDelimiter parameter:

If the delimiter between the values will not correspond with the one that was insered in the multivalueDelimiter attribute, the cell value will be no longer considered a multi-value so it will be stored as single-value.

Limitations

The user cannot set as input for multivalueFields the value from fields sourceIdColumn, versionIdentifierColumn, or versionLevelColumn because those are single-value fields.
If the parameter multivalueFields is set then parameter multivalueDelimiter must be set as well.

Delta Migration

Starting with 3.16 of migration-center, the CSV/Excel scanner is able to scan existing objects as updates. When the scanUpdates parameter is checked, the scanner compares the attribute values of the previously scanned object with the ones in CSV/Excel. If they are not exactly the same then the object will be scanned as an update. If they are the same, the object is ignored and an info message will be added in the report log. When the scanUpdates parameter is not checked, the modified objects will be ignored and an info message will be logged.

Log files

A complete history for any CSV & Excel Scanner job is available from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with

The parameters the job was run with

Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Documentum Adapter can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Database Scanner

Introduction

The Database scanner can access SQL compliant databases and extract information via user specified SQL SELECT statements.

Note that the Database Scanner can extract content from a database or use the content files which are specified by the user during the transformation process via the mc_content_location system attribute.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

Database Scanners can be created, configured, started and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Exporting objects from a database

The Database scanner can access SQL compliant databases and extract information via user specified SQL SELECT statements. Since many content management systems rely on some type of SQL compliant database to manage their data, the Database scanner can also be used as a generic interface for extracting information from unsupported/obsolete/custom built content management systems. The types of database management systems supported are not limited to any particular vendor or brand, as access happens via JDBC, hence any SQL compliant database having a (compatible) JDBC adapter available can be accessed and queried.

Some common content management system features supported by migration-center Database scanner are metadata, including system metadata such as permission information, owner, creator, content path, etc, as well as version information, as long as these types of information can be obtained from the respective system’s SQL database. The information extracted by the database scanner is stored in the migration-center database and can be processed, transformed, validated and imported just like any other type of scanned information.

Note that the Database Scanner can extract content from a database stored in BLOB/CLOB fields. Alternatively, the content files corresponding to the objects can be specified by the user during the transformation process via the mc_content_location system attribute.

Depending on the way the content is stored, it may be necessary to extract the content to the filesystem first by other means before migration-center can process it. For the mc_content_location system attribute any of the available transformation functions can be used, so it is easy to generate a value resembling a path pointing to the location of the object’s content file, which a migration-center importer can use to import the content. A good practice would be to export content files to the filesystem using the object’s unique identifier as the filename, and then build the path information based on the known path and the objects unique identifier. This location would need to be accessible to the Job Server running the import which will migrate the content to the new target system.

Database Scanner Configuration

Configuring the JDBC driver

The types of database management systems supported are not limited to any particular vendors or brands, as access happens via JDBC, hence any SQL compliant database having a (compatible) JDBC adapter available can be accessed and queried.

Before using the database scanner, the appropriate JDBC driver needs to be installed and configured for the Job Server. To configure a JDBC driver all required Java jar files and Java library paths need to be added to the "... \migration-center Server Components \jdbc.conf" file. The file can be edited with any text editor and contains all information needed for configuration as each setting is described in the file itself.

The JDBC driver for Oracle 11g is delivered with the Database scanner already preconfigured and ready to use.

Scanner configuration

To create a new database Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters, “Database” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type, in this case the Database Scanner.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

Database scanner parameters

Configurations of queries in the “queryFile”

The “queryFile” is an xml file containing a series of SQL queries that will be run sequentially by the scanner for extracting information from the database.

There are two ways to scan objects with metadata:

Using one query that contains both the unique identifier of the rows/objects which need to be exported and the names of the columns that represent the object's metadata.
Using at least two of the queries: one is for selecting the unique identifier of the rows/objects which need to be exported as, and the second query to extract the metadata for each row/object returned by the previous query. Note: If the main query has metadata, those won't be taken into consideration.

If objects have multiple versions and the related information that need to be extracted (i.e. a row/object is a version of another row/object), some additional queries are required.

The number of queries for extracting metadata for objects or versions is not limited. Multiple queries can be specified to gather and extract various information related to objects.

Example 1.1: a simple file with one query

Example 1.2: a simple file with two queries

The xml file is composed for <query> elements that contain the queries that will be run by scanner for extracting objects and metadata. The <query> element may have the following attributes (some of them mandatory):

Example 2: a more complex query file

The scanner will validate the queries configuration file against the following xml schema:

Delta migration

If the parameter “scanUpdates” is not checked, the scanner will only scan new objects or new versions of existing objects and it will ignore all objects that were previously scanned, i.e. that already exist in the MC database (based on their ID in source system).

If the parameter “scanUpdates” is checked, the scanner will scan new objects or new versions of existing objects and it will detect if it should scan existing objects as updates. If an object will be scanned as an update depends on several factors:

If there are no attributes specified in the “deltaFields” parameter, the scanner will scan every object that already exists in the MC database as an update.
If one or multiple attributes are specified in the “deltaFields” parameter, the scanner will scan an object that was previously scanned as an update only if a value of a delta field in the source database is different than the corresponding value in the MC database. If all values (of all fields defined in “deltaFields”) in the source database match the values in the MC database, then the object will not be scanned as an update, it will just be ignored.

The field names in the “deltaFields” are case sensitive so you should define them as they are scanned by the scanner.

Scanning BLOB/CLOB content

Since version 3.2.8 Update 2 of migration-center the database adapter supports extracting BLOB or CLOB content from databases. In order for this feature to work the main-content or version-content query type must be specified in the query file.

The structure of the queries is as follows:

The column for BLOB/CLOB is mandatory and it must have the alias BLOB_CONTENT or CLOB_CONTENT so the scanner is able to process the value correctly and extract the content.

The columns [nameOfFile] and [nameOfExtenstion] are optional. In case they are set, the scanner will use the values to build the name of the file where the content will be exported. The scanner avoids overwriting exiting files by adapting the file name to be unique in the export folder. Also, the characters that area not allowed in the file name will be replaced. Filename and extensions are also saved as source attributes so they can be used in the transformation engine. For avoiding name conflicts with the other attributes, the user can set aliases for these columns. If the columns [nameOfFile] and [nameOfExtenstion] are missing the scanner will use the value of the id in source system as a filename.

Multiple contents for a single object are allowed. If the query returns multiple rows the content returned by every row is exported. All the paths will be stored in the source attribute “mc_content_location” and the path from the last returned row is set as primary content in the column “content_location”.

Example 3: scanning BLOB content

Scanning Excel files

Although it is still possible to use the database scanner to scan Excel files, we recommend to use our CSV & Excel scanner instead. The CSV & Excel scanner is faster and there are no tweaks necessary to get the JDBC-ODBC bridge working in Java later 1.7.

Prerequisites and Configuration

To scan excel files MS Office or at least Excel ODBC drivers need to be installed on the Job Server machine. To install the Excel ODBC driver one of the easiest ways is to download and install the 32bit version of the Microsoft Access Database Engine 2010 Redistributable that includes the necessary drivers and which can be downloaded from Microsoft’s website.

The connectionURL for should have the following format:

jdbc:odbc:DRIVER={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};DBQ=PATH-TO-EXCEL-FILE;ReadOnly=1

Note that all extensions (*.xls, *.xlsx, *.xlsm, *.xlsb) are required.

Example 4: XML Queries for scanning Excel files

Solving Null values for some fields

When a column in the sheet contains mixed data (numbers and strings) the ODBC driver uses the first few rows to detect the column type. If the first values are numbers, the ODBC driver tries to handle all the subsequent values as numbers as well and therefore the values that are strings (not numbers) will be ignored, resulting in a “null” value being scanned by migration-center.

To avoid this problem, the parameter “IMEX=1” must be set in the metadata query:

Log files

A complete history is available for any Database Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Database Scanner can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Documentum Scanner

Introduction

The Documentum Scanner extracts objects such as files, folders, relations, etc. from a source Documentum repository and saves this data to migration-center for further processing. As a change in migration-center 3.2, the Documentum Scanner and Importer, are no longer tied to one another – data scanned with the Documentum Scanner can now be imported by any other importer, including of course the Documentum. Starting with version 3.2.9 objects derived from dm_sysobject are supported.

Scanner is the term used in migration-center for an input adapter. It is used to read the data that needs processing into migration-center and is the first step in a migration project, thus scan also refers to the process used to input data to migration-center in general.

A scanner works as a job that can be run at any time and can be executed repeatedly. For every run a detailed history and log file are created.

A scanner is defined by a unique name, a set of configuration parameters and an optional description.

Documentum Scanners can be created, configured, started and monitored through migration-center Client but the corresponding processes are executed by migration-center Job Server.

Supported Documentum Content Server versions

The Documentum Scanner currently supports Documentum Content Server versions 4i, 5.2.5 to 20.4, including service packs.

For accessing a Documentum repository Documentum Foundation Classes 5.3 or newer is required. Any combinations of DFC versions and Content Server versions supported by EMC Documentum are also supported by migration-center’s Documentum Scanner, but it is recommended to use the DFC version matching the version of the Content Server being scanned. The DFC must be installed and configured on every machine where migration-center Server Components is deployed.

For scanning a Documentum 4i or 5.2.5 source repository, DFC version 5.3 must be used since newer DFC versions do not support accessing older Documentum repositories properly. At the same time, migration-center does not support DFC versions older than 5.3, therefore DFC 5.3 is the only option in this case.

DFC (Documentum Foundation Classes) configuration

Starting from version 3.9 of migration-center additional configurations need to be made for the Documentum adapter to be able to locate Documentum Foundation Classes. This is done by modifying the dfc.conf file, located in the Job Server installation folder.

There are two settings inside the file that by default match the paths of a standard DFC install. One needs to have the path for the config folder of DFC and the other needs the path to the dctm.jar.

See below example:

wrapper.java.classpath.dfcConfig=C:/Documentum/config wrapper.java.classpath.dfcDctmJar=C:/Program Files/Documentum/dctm.jar

The dfcConfig parameter must point to the configuration folder. The dfcDctmJar parameter must point to the dctm.jar file!

Information on folder migration

When scanning Documentum documents, their folder paths are also scanned, and the folder structure can be automatically re-created by migration-center in the target system. This procedure will not keep any of the metadata attached to the folder objects, such as owners, permissions, specific object types, or any custom attributes. Depending on project requirements, it may be required to do a “folder-only migration” first, e.g. for migrating a complete folder structure including custom folder object types, permissions and other attributes first, and then populate this folder structure with documents afterwards. In order to execute a folder-only migration the following steps should be performed to configure the migration process accordingly:

Scanner: on the scanner’s |Parameters| tab check the exportFolderStructure option. Set scanFolderPaths (mandatory in that case) and excludeFolderPaths (if any, optional) leave the parameter documentTypes empty to scan only the folder structure without the documents; list document types as well if both folders and documents should be scanned; now only folders will be scanned without any documents they may contain. Note: Scanning folders is not possible via the dqlString option in the scanner.
Migration set: When creating a new migration set choose a <source type to target type>(folder) object type. Now only the scanner runs containing folder objects will be displayed on the |Filescan Selection| tab. Note that the number of objects contained in the displayed scanner runs now indicates folders and not documents, which is why the number on display (folders) can be different from the total number of objects processed by the scan (if it contains other types of objects besides folders, such as documents).

Folder migration is important. It is necessary to take the approach described above when migrating folder structures with complex folder objects containing custom object types, permissions, attributes, relations, etc. This information will be lost if exportFolderStructure is not selected during scan. If the exportFolderStructure parameter was not set during a scan, it is of course possible to re-run the scan again after setting this option, or to copy/create a new scanner and scan the missing folder information with that one.

Supported Documentum features

Versions

Versions (and branches) are supported by the Documentum Scanner, including custom version labels. The exportVersions parameter in the scanner’s configuration parameters determines if all versions (checked) or only current versions of documents (not checked, default setting) are scanned.

It is important to know that a consistency check of the version tree is performed by migration-center before scanning. A version tree containing invalid or missing references will not be exported at all and the operation will be reported as an error in the scanner’s log. It is not possible for migration-center to process or repair such a version structure because of the missing references.

For documents with versions, the version label extracted by the scanner from the Documentum attribute r_version_label can be changed by the means of the transformation rules during processing. The version structure (i.e. the ordering of the objects relative to their antecedents) cannot be changed using migration-center.

If objects are scanned with the exportVersions option checked, all versions must be imported as well since each object references its antecedent, going back to the very first version. Therefore, it is advised not to drop the versions of an object between the scan and the import processes since this will most likely generate inconsistencies and errors. If an object is intended to be migrated without versions (i.e. only the current version of the object needs to be migrated), then the affected objects should be scanned without enabling the exportVersions option.

Scanning large version trees

Processing a version tree is based on a recursive algorithm, which implies that all objects which are part of a version tree must be loaded into memory together. This can be problematic with very large version trees (several thousand versions). By default, the Documentum Scanner can load and process version trees up to around 2,000 versions in size. For even larger version trees to be processed the Java heap size for the Job Server must be increased according to the following steps:

Stop the Job Server
Open the wrapper.conf file located in the migration-center Server Components installation folder (by default it is %programfiles%\fme AG\migration-center Server Components <Version>)
Search for # Java Additional Parameters # Increase the value of this parameter it if your documentum scanner needs # to scan a large number of versions per document. Alocate 256k for every # 1000 versions/document.
Edit the line wrapper.java.additional.1=-Xss512k, incrementing the default 512k by 256k for every additional 1,000 versions mc should be able to process. E.g. for enabling processing of version trees containing up to 4,000 versions (2,000+1,000+1,000 versions), set the value to 1024k (512k+256k+256k)
Save the file
Start the Job Server

Primary content and renditions

The scanner exports the primary content of all documents unless the skipContent or exportStoragePath are checked. The locations where the content was exported can be seen in the column Content location and in the source attribute mc_content_location,If a primary content has multiple pages, the column Content location stores the location where the page 0 was exported since mc_content_location stores all locations of all pages.

If skipContent is checked the primary content and the renditions of the document will not be exported. So, the documents will be exported as content less objects.

If exportStoragePath is checked the primary content and the renditions of the document will not be exported to staging area. Instead the content related attributes (content_location, mc_content_location, dctm_obj_rendition) will be set with the full path of the content in the repository file store.

Renditions are supported by the Documentum Scanner. The “exportRenditions” parameter in the scanner’s configuration parameters determines if renditions are scanned. Renditions of an object will not count as individual objects, since they are different instances of content belonging to one and the same object. The scanner extracts rendition’s contents, format, page modifiers page numberand storage location used. This information is exposed to the user via migration-center source objects attributes starting with dctm_obj_rendition* in any documents migration set that has Documentum or FirstDoc as source system.

Documentum 4i does not have the page modifier attribute/feature for renditions, therefore such information will not be extracted from a Documentum 4i repository.

Document paths

The scanner collects the full folder paths where the document is linked and add them to the following source attributes:

dctm_obj_link - stores the first path of every folder where the document is linked. If the folder itself is linked on multiple other folders, only the first path of the folder is extracted.
dctm_obj_al_links - stores all paths (r_folder_path) of all folders where the document is linked.

Relations

Relations are supported by the Documentum Scanner. The option named exportRelations in the scanner’s configuration determines if they are scanned and added to the migration-center database. Information about them cannot be altered using transformation. Migration-center will manage relations automatically if the appropriate options in the scanner and importer have been selected. They will always be connected to their parent object and can be viewed in migration-center by right-clicking on an object in any view of a migration set and selecting <View Relations> from the context menu. The resulting dialog will list all relations of the selected object with their associated metadata, such as relation name, child object, etc.

IMPORTANT: The children of the scanned relations are not scanned automatically if they are not in the scope of the scanner. The user must ensure the documents and folders that are children in the scanned relations are included in the scope of the scanner (they are linked under the scanned path or they are returned by dqlString).

migration-center’s Documentum Scanner supports relations between folders and/or documents only (i.e. “dm_folder” and “dm_document” objects, as well as their respective subtypes). “dm_subscription” type objects, for example, although supports relations from a technical point of view, will be ignored by the scanner because they are relations involving a “dm_user” object. Custom relation objects (i.e. relation-type objects which are subtypes of “dm_relation”) are also supported, including any custom attributes they may have. The restrictions mentioned above regarding the types of objects connected by a relation also apply to custom relation objects.

Export relations as renditions

As an alternative to scanning relations as they are in Documentum, the scanner offers the possibility to scan the child related documents as renditions of the parent document. For that, the parameter “exportRelationsAsRendtions” should be checked. This requires “scanRelations” to be checked as well. You can filter the relations that will be scanned as renditions by setting the relation names in the parameter “relationsAsRenditionNames”. If this is not set, all relations to documents will be processed as renditions.

Virtual Documents

Documentum Virtual Documents are supported by the Documentum Importer. The option named exportVirtualDocs in the configuration of the scanner determines if virtual documents are scanned and exported to migration-center.

There is a second parameter related to virtual documents, named maintainVirtualDocsIntegrity. This option will allow the scanner to include children of VDs which may be outside the scope of the scanner (paths to scan or dqlString) in order to maintain the integrity of the VD. If this parameter is disabled, any children in the VD that are out of scope (they are not linked under the scanned path or they are not returned by dqlString) will not be scanned and the VD may be incomplete.

The VD binding information (dmr_containment objects) are always scanned and attached to the root object of a VD regardless of the maintainVirtualDocsIntegrity option. In this way it is possible to scan any missing child objects later on and still be able to restore the correct VD structure based on the information stored with the root object.

The exportVDVersions options allows exporting only the latest version of the VD documents. This option applies only to virtual documents since the exportVersions option applies only to normal documents.

The exportVersions option needs to be checked for scanning Virtual Documents (i.e. if the exportVirtualDocuments option is checked) even if the virtual documents themselves do not have multiple versions, otherwise the virtual documents export might produce unexpected results. This is because the VD parents may still reference child objects that are not current versions of those respective objects. This is not an actual product limitation, but rather an issue caused by this particular combination of scanner options and Documentum’s VD features, which rely on information related to versioning.

The Snapshot feature of virtual documents is not supported by migration-center.

Audit Trails

The Documentum Scanner also supports audit trail entries for documents and folders. To enable scanning audit trails, the scanner parameter exportAuditTrail must be checked; in this case the audit trail entries of all documents and folders within the scope of the scan will be scanned as Documentum(audittrail) type objects, similarly to Documentum(document) or Documentum(folder) type objects.

There are some additional parameters used for fine tuning the selection and type of audit trail entries the scanner should consider:

auditTrailType – is the Documentum audit trail object type. By default, this is dm_audittrail but custom audit trail types (derived from dm_audittrail) are supported as well
auditTrailSelection – is used for narrowing down the selection of audit trail records since the number of audit trails can grow large especially in old systems, but not necessarily all audit trail entries may be relevant for a migration to a new system. This option accepts a DQL conformant WHERE clause as would be used in a SELECT statement. If this returns no results, all audit trail objects of scanned documents and folders will be scanned. Example 1: event_name in ('dm_save', 'dm_checkin') Example 2: event_name = 'dm_checkin' and time_stamp >= DATE('01.01.2012', 'DD.MM.YYYY')
auditTrailIgnoreAttributes – contains a comma separated list of dm_audittrail attributes the scanner should ignore. Again, this option can be used to eliminate audit trail information that is not needed for migration right from the scan.

Export audit trail as renditions

Because there are target systems that don’t allow importing audit trail objects, Documentum scanner allows exporting audit trail objects to PDF renditions of the scanned documents. Exporting audit trails objects as PDF renditions applies only to documents.

The following scanner parameters are used for applying this feature:

exportAuditTrailAsRendition – when checked the audit trail entries are written in a PDF files that are saved as renditions for the documents. This parameter can be checked only when exportAuditTrail is checked and skipContent is not checked. If not checked the audit trail entries are exported as Documentum(audittrail).
auditTrailPerVersionTree – this apply only when exportAuditTrailsAsRendition is checked. When it is checked one PDF is generated for all audit trail entries of all versions of the document. The audit trails entries related to the deleted versions are exported as well. The rendition is assigned to the latest version in the tree. When not checked, one PDF rendition is generated for every version in the tree. In this case the audit trails entries related to the deleted versions are not exported because those versions are not exported by the scanner since they don’t exist anymore in the repository.

Exporting audit trail per version tree may have a big impact on the scanner performance. That’s because audit trail entries for documents are queried by the attribute dm_audittrail.chronicle_id. The performance might be dramatically improved by adding an index in the underlying table DM_AUDITTRAIL_S for the column CHRONICLE_ID.

Aspects

Scanning aspects is supported with the latest update of the migration-center Documentum Scanner. Attributes resulting from aspects are scanned automatically for any document or folder type object within scope of the scan.

The notation used by the Documentum Scanner to identify attributes which result from aspects appended to the objects being scanned is the same as used by Documentum, namely <aspect_name>.<aspect_attribute>.

Any number of aspects per document/folder, as well as any number of attributes per aspect are supported.

After a scan has finished, attributes scanned from aspects are available for further processing just like any other source attribute and can be used normally in any transformation rule.

Aspects are supported only for document and folder type objects!

PDF Annotations

Starting with version 3.2.8 Update 2 of migration-center the possibility of scanning PDF annotations has been added to the Documentum Scanner. When activating “exportAnnotations” the scanner will scan the related “dm_note” objects together with DM_ANNOTATE relations. The “dm_note” objects are scanned as normal objects since the DM_ANNOTATE relations are exported as MC relation having the relation_type = “DctmAnnotationRelation”.

During delta migration, the scanner is able to identify the annotation changes and scan them accordingly.

Comments

Scanning the documents and folders related comments is possible and can be activated (default is deactivated) by changing the scanner parameter “exportComments” to true.

The best-known use case for documents and folders comments is within xCP (xCelerated Composition Platform) application, but it can also be used in custom WDK Documentum solutions.

The comment objects will be scanned as MC relation objects and can be seen in the MC client by opening the relations view of a scanned object. They will have the value of RELATION_TYPE as “CommentRelation”. All comment related attributes that have values will be scanned as attributes of these relations.

For performance reason, when a document has more versions, the comment relations will be attached only to the first document version that had comments (since all document versions share the same comments).

During delta migration, the scanner is able to identify comment changes, based on modifications of “i_vstamp” so it will rescan the corresponding document with all its comments (the first version document that had comments – see paragraph above) even if the document did not change.

To be able to scan the document’s comments it is necessary that the DFC used to have a valid Global Registry configured, because the adapter is using the “CommentManager” BOF service to read them.

Migrating Updates (“Update” or “Delta” Migration)

Objects that have changed in the source system since the last scan are scanned as update objects. Whether an object in a migration set is an update or not can be seen by checking the value of the Is_update column – if it’s 1, the current object is an update to a previously scanned object (the base object). There are some things to consider when working with the update migration feature:

Updated objects are detected based on the r_modify_date and i_vstamp attributes. If one of these attributes has changed, the object itself is considered to have changed and will be scanned and added as an update. Typically any action performed in Documentum changes at least one if not both of these attributes, offering a reliable way to detect whether an object has changed since the last scan or not; on the other hand, objects changed by third party code/applications without touching these attributes might not be detected by migration-center as having changed.
Objects deleted from the source after having been migrated are not detected and will not be deleted in the target system. This is by design (due to the added overhead, complexity and risk involved in deleting customer data).
Updates/changes to primary content, renditions, metadata, VD structures, and relations of objects will be detected and updated accordingly.

Documentum Scanner Properties

To create a new Documentum Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters “Documentum” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list or selecting the Properties button/menu item from the toolbar/context menu.

A detailed description is always displayed at the bottom of the window for the currently selected parameter.

Common scanner parameters

Documentum scanner parameters

When editing a field in the |Parameters| tab (like loggingLevel i.e.), a description/help/hint appears in the lower part of the window.

DQLString Limitations

There are some limitations and best practices regarding dqlString parameter:

The query must return only r_object_id and therefore it must start with "select r_object_id" or "select distinct r_object_id"
"order by" clause and "union" is not allowed in the query
The query should return only ids of documents (strings starting with "09..")
The query should return only the current version of the document (check "exportVersions" for exporting all versions)
For better performance, the query should return distinct values for r_object_id

Log files

A complete history is available for any Documentum Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The list of all runs for the selected job together with additional information, such as the number of processed objects, the starting time, the ending time and the status are displayed in a grid format.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Documentum Scanner can be found in the chosen “logs” folder at the installation of the Server Components of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\Dctm-Scanner

the amount of information written to the log files depends on the setting specified in the loggingLevel start parameter for the respective job.

Exchange Scanner

Introduction

The Microsoft Exchange Scanner is a new adapter available since migration-center 3.2.6. The Exchange scanner can extract messages (emails) from Exchange mailboxes and use it as input into migration-center, from where it can be processed and migrated to other systems supported by the various mc importers.

The Microsoft Exchange Scanner currently supports Microsoft Exchange 2010. It uses the Independentsoft JWebServices for Exchange Java API to access an Exchange mailbox and extract emails including attachments and properties.

Scanner is the term used in migration-center for an input adapter. Using the Exchange Scanner Module to read the data that needs processing into migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A Scanner is defined by a unique name, a set of configuration parameters and an optional description.

Exchange Scanners can be created, configured, started and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.

Exporting objects from an Exchange mailbox

The Exchange scanner connects to the Exchange Server with a specified Exchange mail account and can extract messages from one (or multiple) folder(s) within the current user’s or other user’s mailboxes. The account used to connect to exchange must have delegate access permission to the other accounts from which mails will be scanned. All subfolders of the specified folder(s) will automatically be processed as well; an option for excluding select subfolders from scanning is also available. See chapter below for more information about the features and configuration parameters available in the Exchange scanner.

In addition to the emails themselves, attachments and properties of the respective messages are also extracted. The messages and included attachments are stored as .eml files on disk, while the properties are written to the mc database, as is the standard with all migration-center scanners.

After a scan has completed, the newly scanned email messages and their properties are available for further processing in migration-center.

Exchange Scanner Properties

To create a new Exchange Scanner job, specify the respective adapter type in the Scanner Properties window – from the list of available adapters, “Exchange” must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type, in this case the Exchange adapter’s parameters.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list, or selecting the Properties button or entry from the toolbar or context menu.

Common scanner parameters

Exchange scanner parameters

Note: Configuration parameters ending with (*) are mandatory.

Log files

A complete history is available for any Exchange Scanner job from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the Exchange Adapter can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Domino Scanner

Introduction

Scanner is the term used in migration-center for an input adapter. Using the IBM Domino scanner module to read the data that needs processing into migration-center is the first step in a migration project, thus “scan” also refers to the process used to input data to migration-center.

The IBM Domino Scanner is available since migration-center 3.2.5. It extracts documents, metadata and attachments from IBM Domino/Notes applications and use them as input for migration-center. After the scan the generated data can be processed and migrated to other systems supported by the various migration-center importers.

The currently supported formats of the documents export are Domino XML (dxl), Hypertext Markup Language (html), ARPA Internet Text Message (rfc 822/eml) and HTML from the EML. In addition, the scanner is capable of generating a Portable Document Format (pdf) rendition based on a DXL file of that document.

The IBM Domino Scanner currently supports all IBM Notes/Domino versions 6.x and above. Documents from applications that have been built with older IBM Notes/Domino versions can be extracted without any limitation.

The module works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.

A Scanner is defined by a unique name, a set of configuration parameters and an optional description.

IBM Domino scanners can be created, configured, started and monitored through migration-center client, but the corresponding processes are executed by migration-center Job Server.

Installation

Prerequisites

To be able to use the IBM Domino scanner additional software must be installed on the migration-center job server that executes IBM Domino scanner operations.

The scanner is available in a 64-bit and 32-bit version, each of which has different requirements.

The 32-bit version of the scanner relies on IBM Notes whereas the 64-bit version of the scanner uses IBM Domino. Because IBM Domino lacks some libraries used by the scanner to generate specific document formats (e.g. “EML”, “HTML” and “RTF”) the 64-bit version of the scanner can currently not generate any other formats than “DXL” and “PDF”.

The 32-bit scanner requires:

Microsoft Windows based 32-bit/64-bit Operating System.
Java Runtime Environment (JRE) 1.8.x (32-bit)
IBM Notes 9.0.1 or later
IBM Notes 9.0.1 or later must be installed on the migration-center job server. Please refer to chapter “9. IBM Notes/Domino installation and configuration” for detailed instructions about installing and configuring IBM Notes.
Microsoft Visual C++ 2017 Redistributable Package (x86) The Microsoft Visual C++ 2017 Redistributable Package can be downloaded from: https://aka.ms/vs/16/release/vc_redist.x86.exe

The 64-bit scanner requires:

Microsoft Windows based 64-bit Operating System.
Java Runtime Environment (JRE) 1.8.x (64-bit)
IBM Domino 9.0.1 or later IBM Domino version 9.0.1 must be installed on the migration-center job server. The community version may be used. Please refer to chapter “9. IBM Notes/Domino installation and configuration” for detailed instructions about installing and configuring IBM Domino.
Microsoft Visual C++ 2017 Redistributable Package (x64) The Microsoft Visual C++ 2017 Redistributable Package can be downloaded from: https://aka.ms/vs/16/release/vc_redist.x86.exe

If the scanner shall be used for Domino documents containing Object Linking and Embedding (OLE) objects, Apache OpenOffice 4.1.5 or later must be installed. Please refer to the section “Exporting OLE objects” for details.

If documents extracted from IBM Domino/Notes applications should be transformed into PDF (PDF, PDF/a-1a, PDF/a-1b) by the scanner, a second system, a “rendition server” is required. The rendition server must have the optional PDF Generation Module installed. For details about setting up a rendition server based on PDF Generation Module refer to the PDF Generation Module manual.

Timezone settings

IBM Domino stores all date and time information based on GMT/UTC internally.

If a date and time value is converted into a text value for display purposes in an IBM Domino API based software solution, the date and time value is always displayed using the client’s current timezone settings.

As the scanner is an IBM Domino API based software product, the migration-center job server’s timezone setting will be used to extract all date and time values from IBM Domino documents, i.e. they will be available in the migration-center database and always be related to the migration-center Job Server’s timezone.

If you require date and time values to be shown based on a specific timezone inside the migration-center database, set migration-center Job Server’s timezone accordingly.

If you require “normalized” date and time values in migration-center, set the migration-center Job Server’s timezone to GMT/UTC.

Running the installer

If the installer is run separately from the migration-center installer, it must be run with administrative privileges. If it’s run as a normal user, the installer cannot update configuration files and set environment variables as required.

Exporting objects from an IBM Domino/Notes application

After a scan has completed, the newly scanned documents along with their metadata, attachments and the content of the richtext fields they contain are available for further processing in migration-center.

IBM Domino Scanner properties

To create a new IBM Domino Scanner job, specify the respective adapter type in the scanner properties window – from the list of available adapters, “Domino” must be selected. Once the adapter type has been selected, the list of parameters will be populated with the parameters specific to the selected adapter type.

The properties window of a scanner can be accessed by double-clicking a scanner in the list or by selecting the [Properties] button for the corresponding selected entry on the toolbar or context menu.

Common scanner parameters

IBM Domino Scanner parameters

Document Formats

The IBM Domino Scanner for fme migration-center supports the generation of different output formats for a Domino document. Each of the formats has its advantages and disadvantages. Which one best suits your needs can be determined by closely looking at your requirements, i.e. how users should work with the documents once migration into the new target system has been completed.

The formats currently supported will be described in detail in the following sections.

The .MSG and eml2HTML formats require an additional license for creating.

Domino XML (DXL)

The Domino XML (DXL) format is an XML format that has been defined by IBM/Lotus. It has been around for a while (at least since Domino version 6). A DXL file represents an entire Domino document including all its metadata, richtext elements and attachments.

The generation of DXL files from Domino documents relies on core functionality of Domino’s C-API as provided by IBM.

DXL files can be used to extract any document information from Domino applications. Based on special helper applications that are not part of Domino/Notes, a DXL file can be re-imported back into the original Domino application in order to read its content or otherwise work with the document at a later point in time.

DXL is especially useful whenever Domino documents should be transformed into PDF. The “PDF Generation Module” which is available as an add-on product for the IBM Domino Scanner makes use of the DXL format for PDF generation.

ARPA Internet Text Message (RFC 822/EML)

The ARPA Internet Text Message format (RFC 822) describes the syntax for messages that are sent among computer users (e-mail). The EML file format adheres to RFC 822.

Any Domino document – not only e-mails – can be transformed into EML format based on core functionality of Domino’s C-API as provided by IBM. An EML file contains the document’s content, its metadata as well as its attachments.

The EML format does not guarantee preservation of the Domino document’s integrity. Information from the document maybe lost or changed during conversion into EML (see Domino C-API documentation).

The major benefit of EML is that – since version 8 of Notes – an EML file can be opened in Notes again without the need for special helper applications.

Hypertext Markup Language (HTML)

Hypertext Markup Language (HTML) files can be generated for Domino documents based on two different approaches both of which will now be described.

Hypertext Markup Language (HTML) – direct approach

The Domino C-API offers the ability to directly transform a domino document into an HTML file.

As with the EML file format, the direct HTML generation based on the Domino C-API has some issues regarding completeness of information. One example are images that had been embedded into richtext fields. Those images will not be visible in the HTML file created.

EML to Hypertext Markup Language (EML2HTML) – indirect approach

Besides the direct approach described in the previous section, HTML can also be created from the EML format.

In most scenarios that the Domino scanner has been tested on, the result of the indirect approach had a much higher quality than that of the direct approach.

Generating EML2HTML requires a third party library that needs to be purchased separately. Please contact your fme sales representative for details.

Microsoft Message Format (MSG)

The MSG format is the format that is used by Microsoft Outlook to store e-mails on the filesystem. It’s a container format that includes the e-mail and all its attachments.

Generating MSG requires a third-party library that needs to be purchased separately. Please contact your fme sales representative for details.

Richtext format (RTF)

The Domino scanner can extract the entire Domino document (not just the document’s richtext fields) as a single RTF file. This functionality is provided by the Domino C-API.

Portable Document Format (PDF/PDF/a-1a/PDF/a-1b)

All the PDF formats preserve the Domino document in a read-only form that looks like the document opened in Notes.

The PDF generation module takes care of collapsible sections, fixed-width images and tables and other Domino specific features that PDF printing might interfere with.

If required, all the Domino document’s attachments can be re-attached to the PDF file that was generated (see parameter “embedAttachmentsIntoPDF”). Thereby, the entire e-mail will be preserved in a read-only format that can be viewed anywhere at any time requiring a standard PDF reader only.

Exporting OLE objects

If the IBM Domino documents contain OLE embedded objects, Apache OpenOffice 4.1.5 or later must be installed and configured on the migration-center job server in order to properly extract the OLE objects.

Install Apache OpenOffice 4.1.5 on the migration-center job server.

Add the folder containing the “soffice.exe” file to the system’s search path. This folder is typically:

<Apache OpenOffice installation folder>/program

Add the following entry to the file “wrapper.conf” inside the migration-center server components installation folder and replace <#> with an appropriate value for your installation:

wrapper.java.classpath.<#>=<Apache OpenOffice installation folder>/program/classes/*.jar

Open the configuration file „documentDirectoryRuntimeConfiguration.xml“ located in subfolder „lib/mc-domino-scanner/conf“ of your migration-center server components‘ installation folder in your favorite editor for XML files.

Go to line 83 of the file which looks like:

<parameter name="exportOLEObjects">false</parameter>

and replace “false” with “true”.

The entry inside the configuration file should look like:

If you want to use a different port for the Apache OpenOffice server than the default port (8100), go to line 84 of the file:

Uncomment it and and replace “8100” with the portnumber to use, e.g “1234”.

The entry inside the configuration file should look like:

Save the configuration file.

Generating PDF renditions

While PDF generation can be activated in the scanner’s configuration (parameters “primaryDocumentFormat”, “secondaryDocumentFormats” and “embedAttachmentsIntoPDF”), the setup of PDF generation requires and the additional “PDF Generation Module”.

The “PDF Generation Module” is licensed separately.

From a technical perspective, the “PDF Generation Module” requires an additional system (“rendition server”). This system will be used to print any IBM Notes document using a PDF printer driver based on IBM Notes’ standard print functionality. The process for PDF generation is as follows:

The scanner submits a request to create a PDF rendition for an existing Domino document or a DXL file to PDF Generation Module on the rendition server.
PDF Generation Module creates a PDF rendition of the document.
If PDF generation was successful, PDF Generation Module will save the PDF to a shared network drive.
PDF Generation Module will signal success or failure to the scanner.

Setting up the rendition server requires additional configurative actions. For each IBM Domino application/database template that was used to create documents, an empty database needs to be created based on this template and either made available locally on the rendition server or on the IBM Domino server.

Each of these empty databases needs to be prepared for PDF printing. As necessary configuration steps vary depending on the application that is being worked on, they cannot be described here.

Please contact your fme representative should you wish to implement PDF generation for migration of an IBM Domino application/database.

Log files

A complete history is available for any IBM Domino Scanner job from the respective item’s history window. It is accessible through the [History] button/menu entry on the toolbar/context menu.

The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the [Open] button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

version information of the migration-center Server Components the job was run with
the parameters the job was run with
the execution summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime

Log files generated by the IBM Domino Scanner can be found in the server components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Known Issues

The following issues with the MC Domino Scanner are known to exist and will be fixed in later releases:

The scanner requires that the temporary directory for the user running MC Job Server Service exists and that the user can write to this directory. If the directory does either not exist or the user does not have write permission to the directory, the creation of temporary files during document and attachment extraction will fail. The logfile will show error messages like

„INFO | jvm 1 | 2014/10/02 12:06:26 | 12:06:26,850 ERROR [Job 1351] com.think_e_solutions.application.documentdirectory… - java.io.IOException: The system cannot find the path specified“.

To work around this issue, make sure the temporary folder exists and the user has write permission for this folder. If the MC Job Server is started manually as a normal user then the Temp folder should be C:\Users\Username\AppData\Local\Temp. Therefore, if the MC Job Server is run as a service by the Local System account, the folder is one of the following:

For the 32-bit version of Windows:

C:\Windows\System32\config\systemprofile\AppData\Local\Temp

For the 64-bit version of Windows:

C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Temp

If a document is exported from IBM Domino but the related entries in the mc database cannot be created (e.g. because an attribute’s value exceeds the maximum number of characters allowed for a field in the mc database), the related files can be found in the filesystem (inside the export directory). If this document is scanned again, it will be treated as a new document, not as an update.
If the scanner parameter “relationType” is set to “relation”, relations will be automatically deleted by migration-center if they do not exist anymore. If the scanner parameter “relationType” is set to “object”, objects representing relationships cannot be deleted if the relation is invalidated.

Example: If a document had one attachment when scanned in scanner run #1 and that attachment was removed from the document before scanner run #2, the scanner cannot remove the object representing the “attachment” relation between document and attachment (created in scanner run #1) in scanner run #2.

If a PDF rendition is requested and DXLUtility receives the request to generate the rendition but isn’t able to import the DXL file into the appropriate IBM Domino database on the rendition server, it’s likely that the shared folder used to transfer DXL and PDF files between the scanner and PDF Generation Module cannot be read by the user running PDF Generation Module on the rendition server.
The scanner will crash the Java VM if the parameter “exportCompositeItems” is set to “true” and the log level in log4j.xml (located in subdirectory “conf” of the scanner installation directory) is set to “ERROR”.
The 64-bit version of the scanner relies on IBM Domino. As Domino lacks the required libraries to export “EML”, “HTML” or “RTF”, the 64-bit version of the scanner cannot export documents in any other format than “DXL” or “PDF”. If other formats are required, the scanner’s 32-bit version needs to be run based on IBM Notes instead.

Domino attribute types

The following table lists all (relevant) Domino attribute types.

The scanner parameter “excludedAttributeTypes” is a logical “OR” of all types that should be excluded from the scan.

OpenText Scanner

Introduction

The OpenText Scanner allows extracting objects such as documents, folders compound documents and saves this data to migration-center for further processing. The key features of OpenText Scanner are:

Supports OTCS versions: 9.7.1, 10.0, 10.5, 16.0, 16.2, 16.2.4, 20.4
Export documents content and their metadata (versions, ACLs, categories, attribute sets, classifications, renditions)
Export folders and their metadata (categories, attribute sets, classifications)
Export shortcuts of documents and folders
Scale the performance by using multiple threads for scanning data

Scanner is the term used in migration-center for an input adapter. Using a scanner such as the OpenText Scanner to extract data that needs processing in migration-center is the first step in a migration project, thus scan also refers to the process used to input data to migration-center.

Scanners and importers work as jobs that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created. Multiple scanner and import jobs can be created or run at a time, each being defined by a unique name, a set of configuration parameters and a description (optional).

Exporting from OpenText

Exporting objects

The OpenText scanner connects to the OpenText Content Server via the specified “webserviceURL” set in the scanner properties and can export folders, documents or compound documents. The account used to connect to the content server must have System Administration Rights to extract the content. All subfolders of the specified folder(s) will automatically be processed as well; an option for excluding select subfolders from scanning is also available. See below for more information about the features and configuration parameters available in the OpenText scanner. Email folders are supported so the documents and emails located in email folders are scanned as well.

In addition to the documents themselves, renditions and properties can also be extracted. The content and renditions are exported as files on disk, while the properties are stored to the mc database, as it is the standard with all migration-center scanners.

After a scan has completed, the newly scanned documents and their properties are available for further processing in migration-center.

Below is a list of object types, which are currently supported by the OpenText scanner:

Any type of container nodes (we have successfully tested so far the following types: Folder, Email Folder, Project, Binder, EcmWorkspace)
Document (ID: 144)
Compound Document (ID: 136)
Email (ID: 749)
Shortcut
Generation
CAD Document (ID: 736) - scanned as a regular document

Any objects with different types will be ignored during initialization. A list of all node types of scanned containers is provided in the run log as well as all node types in the scope of scanning that were not exported because they are not supported.

Exporting the ACL metadata for folders and documents

The ACLs of the scanned folders and documents will be exported automatically as source attribute ACLs.

ACLs attribute can have multiple values, so that each value has the following format:

<ACLType#RightName#Permission-1|Permission-2|Permission-n>

The following table describes all valid values for defining a correct ACLs value:

Ex:

ACL#csuser#See|SeeContents Owner#manager1#See|SeeContents|Modify

OpenText Scanner Properties

To create a new OpenText Scanner job, specify the respective adapter type in the Scanner Properties window from the list of available adapters, OpenText must be selected. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type, in this case the OpenText parameters.

The Properties window of a scanner can be accessed by double-clicking a scanner in the list or selecting the [Properties] button or entry from the toolbar or context menu.

Common scanner parameters

OpenText scanner parameters

Known issues

When extracting the Owner, Version Created By, Created By and any other attribute the references a user it may happen the internal id of the user cannot be resolved by the Content Server and therefore instead of the username the scanner will extract its id. When this happens, a warning is logged in the report log.

Email folder can only be extracted from the Content Server 16 and later.

Wildcards feature is working properly in most cases, but it has some limitations:

Folders which contain the character “?” in the name will not be scanned using “*” wildcard. Example: Ex: "/*" will not scan "/test?name"
The wildcards in the middle of the path do not work: “/*/Level2” will scan no documents under the “Level2” folder
“/Level1/*exactname*” will scan no documents which are located in “exactname” folder

When scanning documents by using “scanFolderPaths”, if two or more paths overlap, the documents which are located under them will be scanned twice or more. However, this issue will not affect the content integrity of the documents. Each duplicate document will have a separate content location.

Example:

/folder1/folder2/doc1

/folder1/folder3/doc2

/folder1/folder3/doc3

scanFolderPaths: “/folder1 | /folder1/folder3”

Considering the above scenario, the documents “doc2” & “doc3” will be scanned twice.

When at least two ids overlap with their children when setting rootFolderIds the documents will be scanned twice or more.

Example:

rootFolderIds* : 2000, 2001

If the root folder with id 2000 contains as children root folder 2001, all the objects under the root folder 2001 will be scanned twice.

Importers

Previous versions

migration-center 3.16

Since 3.16, the following adapters will no longer be supported:

eRoom Scanner
Exchange Removal Scanner
DCM Importer
FirstDoc Importer
Legacy SharePoint Importer

These adapters will no longer work with the 3.16 Jobserver after upgrading existing migration-center environments to 3.16.

Features & Enhancements

General
- Add support for Java 11 for all adapters (#53313)
- Add new transformation function: GetInternalAttributeValue (#57376)
- Add new transformation function: CalculateNewNumber (#58237)
Cara Importer (NEW)
- Add support for checksum verification (#57744)
- Add support for importing VDs (#57756)
- Add support for proxy communication (#58057)
- Add support for uploading content using REST (#58169)
D2 InPlace Adapter (NEW)
- D2 Auto-Naming functionality (#57662)
- D2 Auto-Linking functionality (#57663)
- D2 Auto-Security functionality (#57664)
- Validate attributes values based on D2 dictionaries (#57666)
- Apply D2 rules based on a document’s owner (#57667)
CSV/Excel Scanner
- Add support for delta migration on CSV/Excel scanner (#56911)
D2 Importer
- Add support for D2 20.4 (#55909)
Database Scanner
- Improve performance when scanning data from a single query (#56714)
Documentum Importer/Scanner
- Add support for DCTM server 20.4 (#55904)
Filesystem Scanner
- Add support for scanning paths provided in a text file (#57073)
- Specify folders to exclude as relative paths (#57074)
- Enhance explicit versioning to new versions to existing ones during delta scan(#58563)
InfoArchive Importer
- Add support for InfoArchive 20.4 (#55912)
- Add support for generating huge PDI files (#56912)
Opentext Content Server Scanner/Importer
- Add support for OpenText Content Server 20.4 (#55901)
SharePoint Online Batch Importer
- Set retention policy labels (#56114)
- Set sensitivity labels (#57086)
- Add support for QuickXorHash (needed for files > 15 GB) (#57226)

Fixes

CSV/Excel Scanner
- Cells that have nothing but are scanned as null (#57228)
Documentum Importer
- Repeating attributes are not rewritten on new versions (#58231)
Opentext Content Server Importer
- The job status is not properly set when occurring OutOfMemory error (#57022)
SharePoint Online Batch Importer
- MIP Service fails to load DLL (#58038)
Veeva Scanner
- The temporary location is not cleared up after the scan run (#57853)
Veeva Importer
- Validation errors are not cleared after each validated document (#57206)
- Submissions cannot be imported when having more than 200 applications (#58062)

Known Issues & Limitations

Veeva Scanner
- The binder relations are scanned even if the children documents are not (#56230)
SharePoint Online Batch Importer
- When trying to import a broken version tree, the import fails for all documents (even the ones that had a correct version tree) with an error status (#57099)
Database Scanner
- The summary is not completely accurate when using a single query (#58228)
InfoArchive Importer
- The first version of child doc remains validated when includeChildrenVersions is set to false and includeChildrenOnce is set to true (#57766)
Veeva Importer
- Attributes mapping does not work properly when the object_type__v is not the same as the object that is imported (#57383)
Opentext Content Server Importer
- NPE when importing documents with no content (#58288)

migration-center 3.15 Update 2

Features & Enhancements

General
- Migrate OT shortcuts and URLs to SPO link documents (#55974)
- New TrackWise Digital importer available (#53977)
- Improve the performance of the mapping list transformations (#56913)
SharePoint Online Batch Importer
- Improve robustness of verification step (#55898)
- Allow import into multiple sites (#56043)
- Save imported object URL in object_info1 column (#56113)
SharePoint Online Scanner
- App-Only authentication in SPO scanner (#54259)
Veeva Scanner
- Automatic resume after daily API limit was exceeded (#56898)

Fixes

File System Scanner
- content_location and mc_content_location not updated when using moveFilesToFolder (#56471)
- Metadata files are not moved when moveFileToFolder is checked (#56488)
SharePoint Online Classic Importer
- 'null' added to path when importing to root folder (#56603)
- Move folder not working in SPO importer (#56703)
Veeva Importer
- Document with auto-generated name __v cannot be imported (#56684)
- Error when importing doc with empty string in attachments and skipUploadToFTP (#56826)
Veeva Scanner
- The rendition for already scanned objects is exported to disk in some cases (#56283)
- Proxy configuration ignored for HTTP requests (#56566)
- Running scanners in parallel fails with error (#56763)

Known Issues & Limitations

Veeva Scanner
- The binder relations are scanned even if the children documents are not (#56230)
InfoArchive Importer
- OutOfMemory exception might be thrown by InforArchive importer when generating very large SIP file (#55698)
SharePoint Online Batch Importer
- When trying to import a broken version tree, the import fails for all documents (even the ones that had a correct version tree) with an error status (#57099)
SharePoint On-Prem Importer
- Moving folders fails with 'File not found' error (#57111)

migration-center 3.15 Update 1

Features & Enhancements

General
- Add Veeva Scanner that exports documents and binders from Veeva Vault (#54569)
- Migrate OpenText shortcuts and URLs to SPO link documents (#55974)
Documentum Scanner
- Export all folder paths of all folders where a document is linked (#55486)
OpenText Scanner
- Add the URL/path of the original object to the scanned shortcut (#56206)
- Allow users to configure a list of workspaces/root nodes to be scanned (#55899)
- Scan URLs as OpenText(object) type so they can be imported to other systems (ex: SharePoint Online) (#56001)
SharePoint Online Importer
- Add support for setting sensitivity label for importing documents (#55645)
Veeva Importer
- Enable Migration Mode programmatically for the imported objects so Migration Mode is not necessary to be enabled on the vault level (#55493)

Fixes

Veeva Importer
- Duplicate values are not supported for object_type__v attribute in RIM Vault (#55664)

Known Issues & Limitations

Veeva Scanner
- The renditions for already scanned objects is exported to disk in some cases (#56283)
- The binder relations are scanned even if the children documents are not (#56230)
InfoArchive Importer
- OutOfMemory exception might be thrown by InforArchive importer when generating very large SIP file (#55698)

migration-center 3.15

Features & Enhancements

CSV - Excel Scanner:
- Added support for scanning repeating attributes (#54993)
InfoArchive Importer:
- Added support for multiple references to the same content (#55529)
OTCS Scanner:
- Added support for scanning CAD documents as regular documents (#55900)
SharePoint Online Batch Importer:
- Added support for setting approval status on documents (#54990)
- Added support for setting version numbering (#55505)
- Added support for setting Lookup field type (#55521)
Veeva Importer:
- Added support for delta migration of Veeva objects (#54928)
- Added support for importing attachments for Veeva Objects (#55535)
Core Database:
- Added additional multi-value transformation functions (#55424)

Fixes

OTCS Importer:
- Fixed record date not being set in opentext_rm_classification (#55431)
SharePoint Online Batch Importer:
- Fixed import failing with invalid XML characters in attributes (#55007)
- Fixed import failing verification (#55574)
- Fixed import failing into library with deep path (#55745)
SharePoint Online Importer:
- Fixed import failing into site collection URL with spaces (#55571)
- Fixed not being able to assign AD group (#55796)
- Fixed error on folder update (#55870)

Known Issues & Limitations

Alfresco Scanner:
- Scan doc with versions edited online while autoVersion false, but had autoVersion switched to true afterwards has wrong version content (#55983)
CSV - Excel Scanner:
- Content_location is not scanned as multi-value (#55635)
Veeva Importer:
- Objects are not rolled back if attachment was set on objects that do not support attachments (#55939)
Documentum NCC Importer:
- Delta migration for the multi-page content does not work properly when a new page is added to the primary content (#55739)

migration-center 3.14 Update 1

Features & Enhancements

Veeva Importer
- New feature: Import documents using existing content from FTP server (#55303)
- New feature: Add support for importing relations (#54991)
SharePoint Online Importer (Batch)
- New feature: Allow assignment of any valid SP user (#55219)
- Support for OneDrive (#54262)
Filesystem Scanner
- New feature: Perform transformation of source object XML files before processing (#54776)

Fixes

SharePoint Online Importer (Batch)
- Update objects not marked as processed/imported (#55058)
Veeva Importer
- Import version tree with renditions fails (#55396)
SharePoint Importer
- Folder and document get created although a column cannot be set (#55001)
- SPO importer fails setting lookup column with a NULL value (#55140)
- CSOM Processor fails with 401 Unauthorized error if a job ran longer than 24 hours (#55323)
Opentext Scanner
- Keywords system rule is limited to 255 characters for Physical Objects migset (#55341)

Known Issues & Limitations

General / logging
- The installer does not update the location of the Job Server's log files. So if you do not want to use the default location for log files, which is <Job Server Home>/logs, you need to manually update the log file location in the <Job Server Home>/lib/mc-core/logback.xml configuration file (#54732)

migration-center 3.14

Starting with version 3.13 Update 2, the SharePoint Online Importer does only support app-only principal authentication and no user name / password authentication. Please ensure that this will work for you before you upgrade your existing installation!

Features & Enhancements

Alfresco Scanner
- New feature: Scan only last "n" versions (#54289)
- New feature: Scan multiple sites (#54291)
D2 Importer
- Support for D2 20.2
Documentum Scanner / Importer
- Support for Documentum Server 20.2
OpenText Importer
- Support for OpenText Content Server 20.2
OpenText Scanner
- New feature: Scan shortcuts as distinct MC objects (#53710)
- New feature: Scan any folder subtype objects (#53711)
SharePoint Online Importer (Classic)
- New feature: Add user-agent traffic decoration (#52837)
SharePoint Online Importer (Batch)
- New feature: Importing role assignments (#54440)
Tools
- New tool to export document type definitions from Veeva available (#51539)

Fixes

OpenText Scanner
- NPE occurs using excludeFolderPaths (#54326)
- Wildcards do not work in the root folder (#54329)
- Set attributes are not scanned if the name is longer than 100 chars (#54589)
SharePoint Online Importer (Batch)
- SPO batch importer checks wrong content location (#54568)
- SPO batch importer throws error when importing large folder hierarchy (#54574)
- NPE for objects with NULL value in levelInVersionTree in SPO batch importer (#54637)
SharePoint Importer
- Wrong max length limit for parentFolder system attribute when importing to SP 2019 (#54592)

Known Issues & Limitations

General / logging
- The installer does not update the location of the Job Server's log files. So if you do not want to use the default location for log files, which is <Job Server Home>/logs, you need to manually update the log file location in the <Job Server Home>/lib/mc-core/logback.xml configuration file (#54732)
D2 Importer
- Support for importing to D2 4.1, 4.5, and 4.6 was removed
OpenText Scanner
- Using binders in the parameters "scanFolderPaths" and "excludeFolderPaths" is not supported (#54963)

migration-center 3.13 Update 2

Features & Enhancements

PowerShell tools:
- Support multiple scan run IDs when creating migration sets (#54263)
SharePoint Online importer:
- Support for app-only principal authentication with SharePoint permissions (#54277)
SharePoint Online Bulk importer:
- Add mc_content_location in migset rules (#54182)
- Support for app-only principal authentication with SharePoint permissions (#54277)
SharePoint Online scanner:
- Add scanLatestVersionOnly feature (#5615)
- Add includeFolders and excludeFolders parameters (#53706)

Fixes

OpenText importer: Wrong values for date attribute during physical objects import (#54255)
OpenText scanner:
- NPE occurs using excludeFolderPaths (#54326)
- Wildcards do not work in the root folder (#54329)
SharePoint (Online) importer: Automatically added content type not added to cache (#54420)
SharePoint scanner: Permissions & User fields contain ambiguous username instead of login name (#54240)

migration-center 3.13

Features & Enhancements

New SharePoint Online Batch Importer (#52665)
Add “removeDuplicate” transformation function (#53528)
Add support for Oracle 19c (#52930)
Add support for Oracle JDK 11 & OpenJDK 11 (#53313)
Add support for Oracle JDK 13 & OpenJDK 13 (#52492)
Add support IA 16.7 (#52910)
Added support for Alfresco 6.2.0 for Alfresco scanner and importer (#54108)
Alfresco scanner and importer now require java 1.8 or later
Documentum Scanner
- Scan complete document audit trail and save it as rendition to the current document version (#52846)
OpenText Importer
- Assign objects to Physical Object Box with a dedicate system rule (#52748)
SharePoint importer
- Support valid filename characters on SharePoint on-prem (#53304)
Documentum In-Place
- Implement move content feature in Documentum In-Place adapter (#53518)

Fixes

Fix typo in transformation function “ConverDateTimezones” (#53136)
“GetDateFromString” transformation function returns null in some cases (#53164)
Job server installation on Linux outdated in Installation Guide (#53714)
OpenText Importer became laggy during impersonation after a certain number of requests (#53180)
No error or warning when importing two renditions of the same type with OpenText Importer (#53124)
The underlying connection was closed: An unexpected error occurred on a send in SharePoint Importer (#53099)
Could not get SharePoint X-RequestDigest error message when space character in site collection name (#53190)
SharePoint Importer should log error immediately if required fields are missing (#53844)
HTTP 503 Server Unavailable Error when downloading large content files with SharePoint Online Scanner (#53673)
Fix “Member ID is not valid” in OpenText scanner (#53482)
Fix setting empty Vault Object reference in Veeva importer (#53573)

migration-center 3.12

Features and enhancements

OTCS Scanner
- Scanning Project objects as OTCS(container) with metadata (#52932)
- Save version information as source attributes (#52292)
- Support wildcards in folder paths (#52360)
- Show number of warnings in job run report summary (#52845)
- Scan user field with username instead of user id (#52847)
- Objects scanned from inside projects now contain full parentFolder path (#52960)
- Revised OpenText Adapter User Manuals (#52550)
Veeva Importer
- Re-authentication mechanism in case of session timeout (#52822)
- Pause and retry mechanism to handle burst and daily API limits (#52823)
- Support version binding when importing Veeva Vault binders (#52091)
- Delta Migration of objects in Veeva Importer (#53037)
Added support for Alfresco 6.1.1 in Alfresco Importer (#53008)
Added support for annotations in D2 Importer (#53018)
Documentum scanner now supports dqlExtendedString returning repeating values (#52587)
Added Support for SP 2019 in SP Importer (#52870)
Auto-Classification Module
- Functionality to split dataset into training and test data (#52660)
- Removing of library under GPL license (#52853)
- Classification Process Logging (#52855)

Fixes

Unresolved document file path in ScanRunExtractor (#52876)
Filesystem importer writes invalid XML characters to metadata XML file (#52909)
Missing dependency packages in AC module installer (#52935)
Error validating classification when subtypes not unique (#52945)

Known Issues

In Veeva Importer setting Binder version labels as major/minor does not work (#53160)
In Veeva Importer document updates with empty values for attributes that were not empty does not delete existing value (#53229)
In OTCS Scanner user fields that were deleted from a category are scanned as IDs (#52982)

migration-center 3.11

Features and enhancements

New SharePoint Online Scanner (#52409)
Made OTCS scanner more robust to errors (#52361)
Imptoved OTCS scanner job run log output (#52365)
FileSystem Importer now applies XSL transformation on whole unified metadata XML file (#52394)
Veeva Importer now supports importing Documentum documents as renditions of a specific type to a Veeva Vault document (#52090)

Fixes

Fixed bug in Filesystem Importer where attributes with no value were not being exported in XML metadata file (#52479)
Fixed OTCS Scanner not scanning documents under folder inside nested projects (#52809)
Fixed SPOnline importer bug when importing folders with a “%” in the name (#52857)
Fixed SPOnline importer not refreshing token when all objects of a large import fail (#52775)

Known Issues

SharePoint Online Scanner might receive timeout error from SharePoint Online when scanning libraries with more than 5000 documents (#52865)

migration-center 3.10

Features and enhancements

Added support for Oracle database version 18c (#51451)
Added support for migrating Veeva Vault Objects (#52136)
Added support for version 16.x for the OpenText scanner (#50340)
Added support for D2 versions 16.4 and 16.5 (#50972, #51530)
Added switch for lifecycle error handling in D2 Importer (#50991)
Updated the IBM Domino Scanner (#49173)
Improved database security by changing default password (#51112)
Improved performance of filesystem importer copy operation (#52320)
Enhanced the RepeatingToSingleValue function to not throw exception when length exceeds 4000 bytes (#51798)
Added missing file and folder name checks in the SP and SPO importer user guides (#52076)
Updated linux Jobserver with new YAJSW wrapper (#52347)
Box.net Importer, added the following features:
- Import custom metadata (#50312)
- Import tags (#50314)
- Import comments (#50315)
- Import tasks (#50316)
- Add collaborators to the imported documents (#50602)

Fixes

Fixed bug in DCTM adapters when calculating multi page content (#50442)
Fixed bug in SharePoint scanner when scanning SP 2013+ on Windows Server (#51088)
Fixed error messages bigger than 2000 characters not being saved in the database (#52357)
SharePoint Importer, fixed the following bugs:
- import taking a long time when content_location points to folder (#52115)
- Importer not starting if adfsBaseURL contains trailing slash (#51941)
- DEBUG messages getting logged despite log level ERROR in log4j (#52082)
- Added debug log messages for queue building missing in SP importer (#52084)
- Exception not caught when reverting import action (#52216)
- Importer hanging when encountering an exception in RefreshDigest method (#52238)
- Improved Retry handling (#52270)
- Error when importing into site with %20 in the name (#52272)
- autoCreateFolders always enabled (#52305)
- NullPointerException in file upload when SPOnline throttling occurs (#52325)

migration-center 3.9 – update 3

Features and enhancements

Some new features and enhancements in Domino Scanner
- New scanner parameter "excludedAttributeTypes"
- Default for scanner parameter "exportCompositeItems" now "0"
- Scanner parameter "selectionFormula" set to be editable by default
- 64bit support (based on IBM Domino); 32bit support remains unchanged (based on IBM Notes)
InfoArchive Importer: provide a better way for importing email attachments scanned from IBM Domino

Fixes

Domino Scanner: Extracted floating point numbers truncated/rounded to integer value
Domino Scanner: Attributes of type "NumberRange" were not exported
OpenText CS scanner: some logging improvements
OpenText CS scanner: Fix scanning subprojects
Veeva Vault Importer: Fix setting status_v for the root version
Veeva Vault Importer: Fix importing VD relations that have order_no with decimals.
Veeva Vault Importer: Fix a null pointer exception when logging some results
CSV/Excel Scanner: Fix setting the internal processing type of the scanner
Sharepoint Importer: Fix importing documents having # and % in the name
Sharepoint Importer: Fix importing folder with two consecutive dots (..) in the name
Sharepoint Importer: Fix token expiration after 24 hours.

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

Migration-center 3.9 update 2

Features and enhancements

Add support for importing attachments to Veeva Vault.

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

Migration-center 3.9 update 1

Features and enhancements

Added support for scanning nested compound documents in OTCS Scanner
Added support for scanning the Nickname attribute for folders and documents in OTCS Scanner
Added Proxy support for ADFS Authentication in SharePoint Importer
Added support for filtering Scanners and Importers in MC Client

Fixes

Fixed OTCS Scanner bug not scanning dates properly sometimes
Fixed OTCS Scanner "Member Type not valid" error message
Fixed OTCS Scanner nullPointerException when trying to get the owner name in specific cases
Fixed OTCS Scanner nullPointerException when trying to scan certain Workspaces
Fixed SharePoint Importer bug autoCreateFolders not working for values with leading slash
Fixed SharePoint failing authentication with invalid XML chars in use name or password

This hotfix requires a reinstall of the Jobserver and Client components, as well as an update of the Database component. Please refer to the Installation Guide for details regarding the update.

migration-center 3.9

Features and enhancements

Replaced Tanuki service wrapper with YAJSW (#51159)
Added support for Java 64-bit (#51163)
Added support for OpenJDK 8 (#51156)
New CSV / Excel Scanner (#50343)
Added Content Hash Calculation feature to OTCS scanner (#50736)
Added support for java web service in OTCS Scanner (#51389)
Added support for importing RIM submissions to Veeva Vault (#51208)
Added support for importing Documentum Virtual Documents as Binders in Veeva Vault (#51188)
Added support for setting references to existing master data objects in Veeva Vault (#51151)
Added support for ADFS authentication in SPO Importer (#51414)
Added new Transformation function for time-zone conversions (#51480)

Fixes

Fixed bug when pulling content for updates with the Database scanner (#51142)
Fixed error message in OTCS log (#51619)
Fixed SharePoint Online importer too short path length limitation (#51491)
Fixed SharePoint not resuming properly after pausing job for long time (#51492)
Changed behavior of setting the file extension for the SharePoint Importer (#51503)

migration-center 3.8

Features and enhancements

Documentum Scanner: Added option to scan only the latest version of a VD (#51019)
OTCS Importer: Added support for importing Virtual Documents as Compound Documents (#51008)
New transformation functions: GetDataFromSQL() and Length() (#50997, #50999)
Database Scanner: New Delta migration feature (#50477)
Documentum Importer: Extended waiting time for operations queue to finish (#50995)

Fixes

Fixed error in D2 Importer when property pages contained tabs with visibility conditions (#50987)
Removed unneeded jar file from D2 importer folder (#51073)

migration-center 3.7

Features and enhancements

The Jobserver now requires Java 8. Java 7 is not anymore supported.
New Veeva Vault importer
Extended OpenText importer to import physical items (#50282)
Changed the way of setting RM Classifications in OpenText importer (#50842).

Fixes

Delta migration issue after upgrading from 3.2.5 or older (#50571)

Known issues

Importing not allowed items in the physical item container is permitted (#50978)
RM classifications for physical objects are not removed during delta (#50979)
Physical objects properties of type date are not updated during delta migration (#50980)
Veeva importer may fail to start, if DFC is installed on the same machine (#50981)
No error reported when setting values to Inactive Fields in Veeva Importer (#50880)
No error reported when setting values to fields that are not in the Type / Subtype / Classification used in Veeva Importer (#50871)
No error reported when setting permissions not in the selected lifecycle in Veeva Importer (#50939)

migration-center 3.6

Features and enhancements

New OpenText In-Place Adapter (#49994)
Updated Tika library to version 1.17 for Filesystem scanner (#49258)
OTCS scanner is now able to scan Business Workspaces (#49994)
SharePoint Importer is now able to change the extension of files (#50142)
D2 Importer validates the Enabled, Visible and Mandatory conditions in a Property Page (#50235)
D2 Importer marks documents as Partially Imported when failing to apply a lifecycle action (#50274)

Fixes

Fixed issue with saving migsets when plsql_optimize_level was set to 3 (#49160)
Fixed issue with exporting BLOB content from a Firebird database (#49197)
Fixed error in Documentum In-Place adapter with linking into inexistent folderpath (#49217)
Fixed nullPointerException in SharePoint Importer when using proxy (#49232)
Fixed bug in Documentum audit trail rollback scripts (#49943)
Fixed nullPointerException in Alfresco Importer (#50027)
Fixed error when scanning documents in “Projects” with OTCS Scanner (#50130)
Fixed individual version creator not being scanned in OTCS scanner (#50273)
Fixed missing supported versions for Oracle and DFC in the Installation Guide (#50275)
Fixed DQL Validation for $repeatingvalue placeholder in D2 Importer (#50278)

Known issues

D2 Importer does not validate values for Editable ComboBox if they are not in the dictionary (#49115)
D2 Importer does not validate Enabled and Visibility properties if they have no condition attached to them (#50327)
Running IA Importer with values “16.x” for the InfoArchiveVersion parameter throws error (#50388)

migration-center 3.5

Features and enhancements

Alfresco scanner now supports Alfresco 5.2 (#49977)
Database scanner now supports multi-content Blob/Clob (#49260)
Added support for scanning and importing xCP comments for folders (#49245)
Added support for Documentum Content Server 16.4 (#49243)
Added support for InfoArchive 16.3 (#49244)
OTCS importer now supports folders of type email_folder types in OpenText Content Server (#49257)
OTCS scanner now supports scanning documents and emails from Email folders (#49995)
OTCS scanner now supports OTDS authentication (#50134)
Added checksum verification feature for OpenText Content Server Importer (#49250)
Added support for Oracle 12c Release 2 (#49259)

Fixes

Fixed bug in the Alfresco importer rollback scripts (#49251)
Fixed delta scan issue when only main object metadata changed (#50052)
Fixed issue with description length in OTCS importer (#50118)
Fixed error when setting subterm under non available term in SharePoint Online taxonomies (#49978)
Fixed job getting stuck when connection to SharePoint server is lost during import (#50026)
Fixed error when importing filename with apostrophe or plus sign in SharePoint (#49087, #49151)
Fixed error when importing SharePoint lists with certain settings (#49975)
Fixed rollback not being performed when a importing certain versions fails in SharePoint (#49974)
Fixed bug where document gets overwritten when old version moved to folder with a different document of the same name in SharePoint (#4661)

migration-center 3.4

Features and enhancements

Added comprehensive content hashing with different hashing algorithms and encodings for the following Scanners Documentum, SharePoint, Database and Filesystem (#11636)
Filesystem Importer: XML Metadata file now contains elements for all null and blank attributes (#11606)
InfoArchive Importer: migrating Documentum AuditTrails is now supported (#11610)
InfoArchive Importer: migrating Documentum Virtual Documents with children relations is now supported (#11683)
OpenText Importer: migrating opentext_email objects is now supported (#11670)
SharePoint Online Importer: added content integrity checking feature for imported Documents (#8296)
SharePoint Scanner: scanning entire version trees as a single migration-center object is now supported (#11641)
New system attributes are now available in transformation rules for all adapters (#11941)

Fixes

Fixed objects not being scanned when reading permissions failed in Filesystem Scanner (#11779)
Fixed table key lookup query being case sensitive in OpenText Importer (#11772)
Fixed length limitation for Table Key Lookup attributes in OpenText Importer (#11777)
Fixed setting attribute values from a different object to a category that has multiple rows when using multiple threads in OpenText Importer (#11797)
Improved logging for OpenText Importer (#11781)
Fixed issue where the initializer thread was still running in the background when a job was being stopped manually (#11739)
Fixed not being able to delete certain Scheduler runs that took more than 2 seconds to delete (#11889)

migration-center 3.3

Features and enhancements

Sharepoint Scanner:
- Added support for SharePoint 2016 (#10999)
- Added support for CAML queries (#11629)
- Added support for scanning SharePoint internal attributes (#11653)
Added support for scanning only current version in eRoom scanner (#11649)
Filesystem Scanner:
- mc_content_location and original_location are now available in transformation rules when using moveFilesToFolder (#11474)
- Added parameter for specifying the date format for extracting extended metadata with unusual format (#11673)
- Added possibility of scanning permissions (#11720)
- Upgraded to latest version of Tika Libraries (#11693)
Added support for InfoArchive 4.1 and 4.2 (#10325)
Added support for migrating SharePoint folders and List Items into InfoArchive (#11639)
OpenText importer:
- now support inheriting folder categories when importing folders (#11674)
- now support inheriting permissions (#11631)
- now supports setting null values for popup / dropdown attributes (#11550)

Fixes

Fixed applyD2RulesByOwner error message when using multiple threads in D2 (#11597)
Fixed some renditions being scanned twice in Documentum scanner (#11740)
Fixed wrong file path in log for Filesystem importer (#11469)
Fixed date values not being properly converted to XML format in some cases for InfoArchive importer (#11672)
Fixed contentless documents failing when ‘nocontent’ is specified in InfoArchive importer (#11656)
Fixed missing parent_object_id when scanning delta versions in OTCS Scanner (#11750)
Fixed importing documents that have date table key lookup attribute failing in OTCS Importer (#11576)
Fixed error when scanning uncommon managed metadata in SharePoint scanner (#11621)
Fixed only latest content exported for each version of a document in SharePoint scanner (#11657)
Fixed exportLocation parameter description when using local path in SharePoint scanner (#11568)
Fixed error when scanning documents with non-mandatory but empty taxonomy field in SharePoint scanner (#11652)
Fixed lookup attributes not being scanned at all in SharePoint scanner (#11655)
Improved OTCS scanner documentation (#11667)
Corrected version labels in Installation Guide (#11622)
Fixed Linux Jobserver classpath values (#11690)

migration-center 3.2.9

Features and enhancements

New Documentum In-Place Adapter (#11490)
Alfresco Importer now supports Alfresco 5.2 (#10772)
Documentum Importer - added support for CS 7.3 (#9927)
D2 Importer
- Added support for D2 version 4.7 (#10777)
- Changed D2 DQL and Taxonomy validation to be done by the D2 API (#10317)
- Added information regarding folder migrating options in the D2 Importer User Guide (#10754)
InfoArchive Importer
- Added support for multiple content files per AIU (#8877)
- Added support for multiple Object Types (#8877)
- Added support for Custom Attributes inside the eas_sip.xml file (#9405)
- Added support for calculating the PDI file checksum with SHA-256 and base64 encoding (#11418)
OTCS Scanner is now able to export rendition types (#11419)
OTCS Importer now supports Extended Attributes Model (#11424)
SharePoint Importer now does extra check to avoid overwriting existing content with the same List Item ID when trying to move content to different libraries (#11465)
Added warning regarding using multiple Jobservers with Sharepoint Importer (#11459)

Fixes

OpenText Importer: fixed java heap space error when importing large size renditions. (#11426)
OpenText Importer: fixed the authentication cookie refreshment for impersonated user (#11396)
OpenText Scanner: fixed out of memory error when scanning objects with size larger than 20MB (#11393)
OpenText Scanner: fixed out of memory error when scanning objects with size larger than 8GB (#11434)
Sharepoint Importer: fixed updates to intermediate version being applied to the latest version (#11458)
Sharepoint Importer: fixed a case that caused the importer to not finish (#10338)
D2 Importer: Fixed validation for certain DQLs containing placeholders (#10318)
D2 Importer: Fixed validation of certain taxonomies (#11549)
Fixed a jar conflict between OpenText Importer and Sharepoint Scanner (#11487)

Known issues

Multi-threaded import using applyD2RulesByOwner fails for some objects in D2 Importer
Remaining issues from the main release 3.2 to 3.2.8 Update 2 also apply to release 3.2.8 Update 3 unless noted otherwise.

migration-center 3.2.8 Update 3

Features and enhancements

Add support for migrating to DCM 6.7 (Documentum Compliance Manager).

Known issues

Remaining issues from the main release 3.2 to 3.2.8 Update 2 also apply to release 3.2.8 Update 3 unless noted otherwise.

migration-center 3.2.8 Update 2

Features and enhancements

Added support for xCP comments in Documentum Scanner and Documentum Importer (#10886)
Added support for “dm_note” objects in Documentum Scanner and Documentum Importer (#10887)
Added support for multi-page content in Documentum Scanner, Documentum Importer and D2 Importer (#11271)
Added support for extracting content from BLOB / CLOB column types in the Database Scanner (#10899)

Fixes

Fixed logging the version of the OpenText Importer in the run log (#11322)
Fixed setting the version description attribute in OpenText Importer (#11270)

Known issues

Documentum scanner/importer: The scanned/imported “dm_note” objects are not counted in the job run report (#11391)
DQLs that have enable(row_based) are not processed correctly in the D2 Importer (#11390)
Remaining issues from the main release 3.2 to 3.2.8 Update 1 also apply to release 3.2.8 Update 2 unless noted otherwise

migration-center 3.2.8 Update 1

Features and enhancements

New OpenText scanner: supports scanning documents, compound documents and folders from OpenText repositories version 9.7.1, 10.0 and 10.5. Specific OpenText features are supported like scanning categories, classification, permissions and shortcuts.

Fixes

Several bug fixes regarding validating DQL queries and Taxonomies when using property pages in D2 Importer.

migration-center 3.2.8

Features and enhancements

Added support for D2 LSS (Life Sciences Solution) version 4.1 (#9990)
Added multi-threading capabilities to the D2 importer (#10107)
Added support for Alfresco version 5.1 (#10354)
Added support for OpenText Content Server version 16.0 (#9937)
Added support for SharePoint 2016 (#9925)
Improved SharePoint Online Importer performance when applying internal attributes and taxonomies (#10314)
Added support for InfoArchive 4.0 (#9926)
Added support for Documentum ECS (Elastic Cloud Storage) (#9783)

Fixes

Alfresco Scanner: fixed scanning duplicates on delta scanning a new version for documents created directly in the Alfresco Share interface (#10013)
InfoArchvie Importer: fixed nullPointer exception when importing objects without content (#10746)
Filesystem Importer: fixed logging message when setting owner fails (#10326)
SharePoint Importer: fixed setting author and editor attributes in a multi-domain environment (#10171)
SharePoint Scanner: fixed job finishing successfully with incorrect credentials (#9939)
MC Client: fixed description for RepeatingToSingle transformation function (#10291)

Known Issues

D2 Importer:
- importing version updates with CURRENT not set to latest version fails (#10784)
- the CURRENT label is always set to the latest version when applying Lifecycle (#10686)
- importing branched versions with multiple and no r_version_label associated fails (#10460)
- D2 importer does not work with D2 4.5 or 4.1 when Jobserver runs on Java version 1.8 (#10384)

migration-center 3.2.7 Update 4

Features and enhancements

Improved Sharepoint and Sharepoint Online Importers performance when setting Taxonomies and internal attributes. (#10423)
Improved Sharepoint Importer Error Reporting in certain cases. (#10299)

Known Issues

Remaining issues from the main release 3.2 to 3.2.7 Update 3 also apply to release 3.2.7 Update 4 unless noted otherwise

migration-center 3.2.7 Update 3

Features and enhancements

Documentum Importer can attach now documents to a given lifecycle.

Fixes

FirstDoc importer: Fix a java.lang.ClassCastException error when importing to FDQM module
FirstDoc importer: Fix importing versions when there is an attribute count difference in the repeating group between create SDL and document definition SDL.

Known Issues

Remaining issues from the main release 3.2 to 3.2.7 Update 2 also apply to release 3.2.7 Update 3 unless noted otherwise.

migration-center 3.2.7 Update 2

Features and enhancements

D2 Importer now supports only D2 version 4.6. The older D2 versions are only supported by the previous versions of migration-center
OpenText Importer now supports creating OpenText Compound Documents
OpenText Importer now supports importing renditions
IBM Domino Scanner:
- Large attribute values can now be split into chunks of maximum 4000 bytes
- Added custom attributes: "attachmentItemName" - name of item that an attachments was attached to in the original Domino document, "attachmentName" - name of the attachment in the original document and "$dominoNoteID$" - NoteID of the document in the database that it originates from.

Fixes

Documentum Scanner: Scanning version trees having versions of multiple object types
OpenText Importer: Error messages concerting a single attribute do not have the attribute name
OpenText Importer: Categories not being set if no associations are made for it
Sharepoint Scanner: Documents with paths longer than 260 characters failed to be scanned

Known Issues

Running a SP scanner with incorrect credentials on a Sharepoint with anonymous authentication enabled, does not show errors.
Remaining issues from the main release 3.2 to 3.2.7 Update 1 also apply to release 3.2.7 Update 2 unless noted otherwise.

migration-center 3.2.7 Update 1

Features and enhancements

Alfrecso scanner now supports scanning version check-in comments
Alfresco importer now supports importing version check-in comments
Documentum adapter supports scanning/importing version trees with different object types in them
InfoArchive importer officially supported on the Linux jobserver
InfoArchive importer now supports ingestion via InfoArchive Webservices
OpenText Importer now supports inheriting categories from folders
OpenText Importer now supports setting values for attributes grouped within a Set attribute

Fixes

InfoArchive Importer:
- fixed text inside SIP xml files generated without indent
- fixed ZIP files not being rolled back when there is not enough memory on disk
OpenText Importer:
- fixed impersonate user not working as expected
- fixed importer not working when DFC not installed

Known Issues

DocumentumNCC scanner: null pointer exception when scanning documents without content
InfoArchive importer: SIPs are imported successfully but marked as failed when moveFilesToFolder does not have write permissions
OpenText Importer: Error messages concerting a single attribute do not have the attribute name
OpenText Importer: Categories are not set if no associations are made for it
Sharepoint Scanner: Documents with paths longer than 260 characters fail to be scanned
Remaining issues from the main release 3.2 to 3.2.7 also apply to release 3.2.7 Update 1 unless noted otherwise.

migration-center 3.2.7

Features and enhancements

New OpenText importer: supports importing documents and folders to OpenText repositories version 10.0 and 10.5. Specific OpenText features are supported like setting categories, classification, permissions, shortcuts to the imported documents and folders.
New SharePoint Scanner: supports the scan of SharePoint 2007, 2010 and 2013. Supports scanning documents, folders, listitems, lists and document libraries. Replaces the old scanner which has been retired.
New SharePoint Importer: supports SharePoint 2010 and 2013 and uses REST API. Support import of documents, folders, listitems, lists and document libraries.
Retired the previous SharePoint Importer as SharePoint Legacy: new installations will not have it. Updated installations will have it under the name of SharePoint Legacy.
SharePoint Online Importer supports now documents, folders, listitems, lists and document libraries. Relations can be imported now as attachments for listitems.
D2 importer supports now creating folders based on the paths provided to documents.
Alfresco importer supports now the Alfresco 5.0.
Alfresco scanner and importer: categories and tags are exported and imported using the values displayed in the interface instead of using internal ids.

Fixes

Alfresco scanner and importer:
- Fixed a NullPointerException when scanning documents with aspects from Alfresco.
- Fixed the value of the content location attribute when scanning documents with skipping content.
- The content location is now available in the attribute “mc_content_location” that is accessible to the transformation engine.
- Fixed importing a new document version when the first link was modified compared with the previous version.
- Improve the message description of several errors that may happen when importing updates and versions.
Documentum scanner:
- Fixed a query error when scanning a repository having DB2 as database.
- Fixed scanning version tree having multiple types.
- Fixed the content location for the documents without format.
Box importer:
- Fixed setting “content_created_at” and “content_modified_at” to the imported files.
Filesystem scanner:
- Fixed out of memory when scanning more than 1 million of files
- Fixed the case when scanning multiple folders like: <some folder path>\FName|<some folder path>\FName with Space. In the previous version the second folder was ignored without any error message.
- Report properly the invalid paths when scanning multiple folders.
- Fixed a possible performance issue when scanning extended metadata.
Filesystem importer:
- Fixed a NullPointerException when setting “modified_date” without setting the “creation_date”
- Fixed the case when the target file was overwritten by the metadata file when they were having the same name.
SharePoint Online importer:
- Fixed updates imported as new versions.
- Fixed import fails only with major version enabled.
- Fixed null pointer exception when setting empty “Group or Person” or “Hyperlink” column.
- Fixed items created for failed jobs are now rolled-back where possible.
- Fixed running with invalid credentials does not set the error status.
- Fixed proper log messages are set now for multiple attribute setting errors.
Eroom scanner
- The “er:Path” attribute does not contains the file name anymore.
Database
- Allow installing migration-center on a non-CDB Oracle 12c instance.
- Fixed the throwing two errors when upgrading from version 3.2.5 or older. This was the case when filesystem importer was not licensed.
Client
- Fixed installing the client on Window 8 and Windows 2012 server. No manual registry configuration is required.

Known issues

OpenText Importer: The modified date of created folders (set by the importer) is overwritten by the content server when new documents are added to the folder.
Remaining issues from the main release 3.2 to 3.2.6 also apply to release 3.2.7 unless noted otherwise.
SharePoint Legacy Importer: The Logging system on the SharePoint side is not reusable after a forced restart. The SharePoint Importer uses much more memory then the previous version. Document sets cannot be created in a library with versioning disabled.
SharePoint and SharePoint Online Importer: Null reference not handled in case of the big document import. The author does not get set unless editor attribute is set as well.

migration-center 3.2.6

Features

Added support for Oracle Database 12c.
Added support for Documentum 7.2.
Added support for Windows Server 2012, Windows 8 / 8.1.
New InfoArchive Importer: supports the import of data generated by any scanner to InfoArchive enterprise archiving platform.
New Alfresco Scanner: supports scanning documents, folders, custom lists and list items from an Alfresco repository.
New Microsoft Exchange scanner: supports scanning emails from different accounts using an account that has delegate access to the other ones.
New Exchange Removal adapter: supports deleting emails in the source system that have been imported from Microsoft Exchange.
Documentum Importer supports multithreaded processing for improved performance.
Filesystem Importer supports setting the creator name, creation date and modify date to the imported file.
Enhance the flexibility of generating metadata files in Filesystem Importer.
Support of Jobserver components installation on Linux. Note that not all adapters are available on Linux
The Oracle package SYS.SQLJUTL is not needed anymore by migration-center.

Fixes

Sharepoint importer:
- Fixed updating minor versions of documents.
- Fixed updating documents when “ForceCheckout” flag is set to true in the library.
- Fixed dummy versions not deleted after large file failed import.
- Fixed importer trying to delete inexistent dummy versions.
- Fixed the reporting of some incorrect error messages.
Sharepoint Online importer:
- Fixed setting the attribute “Title” for contact items.
- Fixed the reporting of some incorrect error messages.
Filesystem Importer:
- Fix a null pointer exception when setting invalid path to “unifiedMetadataPath”.
- Fix moving renditions when “moveFiles” is enabled in the importer.

Limitations

migration-center 3.2.6 requires now Java Runtime Environment 1.7.0_09 or later. It will not work with older java versions.
In Documentum 7.x the max allowed length for owner_name, r_modifier, r_creator_name was changed from 32 to 255 char. In MC the default provided types (dm_document, dm_folder, dm_audit_trail) still have the old limit of 32 char.
Filesystem Importer does not allow more than 255 characters for the “content_target_file_path” system rule.

Known Issues

SharePoint Online Importer: When importing large files into a SharePoint Online document library the import might fail due to a 30 minutes timeout defined by Microsoft. (Bug# 8043)
SharePoint Online Importer: Documents that have the attribute is update set to true are not overwritten during import, but a new version is created on import. Document Sets cannot be created in a library with versioning disabled. (Bug #7904)
Sharepoint Online Importer: The importer does not set the “Author” field of a document, if only the “Author” attribute has a rule associated to it. The user must associate rules to both the “Author” and “Editor” attribute for them to be set. (Bug #8629)
Sharepoint on Premise Importer: Importing documents without any specified version label results in an error. (Bug #8186)
Documentum Importer: Importing multiple updates of the same document that is part of a version tree containing branches may fail if using multiple threads. That is an extreme case since in the process of delta migration the object updates are migrated incrementally. (Bug #8175)
InfoArchive Importer: When the “targetDirectory” runs out of disk space during the zip creation, the importer fails as expected, but the zip file is not rolled back (deleted). (Bug #8758)
Alfresco Importer: Certain errors that may occur during import are not descriptive enough. (Bug #8715)
Alfresco Importer: Updating a document will result in an error if the first link of the base object needs to be removed by the update. (Bug #8717)
Alfresco Scanner: Running an Alfresco Scanner with the parameter “exportContent” unchecked results in invalid values set for the “content_location” attribute. (Bug# 8603)
Alfresco Scanner: Running two scanners in parallel on the same location can result in duplicate objects being created. (Bug #8365)
Filesystem Scanner: Folders will be ignored if multiple folder paths that begin with the same letters are used as a multivalue for the scanFolderPath (Bug #8601)
Remaining issues from the main release 3.2 to 3.2.5 also apply to release 3.2.6 unless noted otherwise.

migration-center 3.2.5 Update 1

Features

SharePoint on Premise supports: new object types Sites, Document Sets; custom sites for individual imported objects; non-consecutive versioning for documents.

Fixes

SharePoint Online Importer: internal properties like ‘Modified’, ‘Modified by’, ‘Created’ and ‘Created by’ are properly updating now.
SharePoint Online Importer: removed limitations of the number of items supported during import in the target library.
SharePoint Online Importer: internal name of the taxonomy terms are correctly used.

Known issues

IBM Domino Scanner:

The scanner requires that the temporary directory for the user running MC Job Server Service exists and that the user can write to this directory. If the directory does either not exist or the user does not have write permission to the directory, the creation of temporary files during document and attachment extraction will fail. The logfile will show error messages like

„INFO | jvm 1 | 2014/10/02 12:06:26 | 12:06:26,850 ERROR [Job 1351] com.think_e_solutions.application.documentdirectory… - java.io.IOException: The system cannot find the path specified“.

To work around this issue, make sure the temporary folder exists and the user has write permission for this folder. If the MC Job Server is started manually as a normal user then the “Temp” folder should be C:\Users\Username\AppData\Local\Temp. Therefore, if the MC Job Server is run as a service by the Local System account, the folder is one of the following:

For the 32bits version of Windows: C:\Windows\System32\config\systemprofile\AppData\Local\Temp

For the 64bits version of Windows:

C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Temp

migration-center 3.2.5

Features

New IBM Domino Scanner: supports scanning emails and documents from IBM Domino/Notes
New Sharepoint Online Importer: supports importing Lists, List Items, Documents and folders in Sharepoint Online
Eroom scanner: Multiple erooms can be scanned now with a single scanner configuration

Fixes

Scheduler: Fix a special case when the scheduler run hangs in status “Scanner running”
Documentum Importer: Fix a “NullPointerException” when importing VDRelation updates

Known issues

IBM Domino Scanner:

The scanner requires that the temporary directory for the user running MC Job Server Service exists and that the user can write to this directory. If the directory does either not exist or the user does not have write permission to the directory, the creation of temporary files during document and attachment extraction will fail. The logfile will show error messages like

„INFO | jvm 1 | 2014/10/02 12:06:26 | 12:06:26,850 ERROR [Job 1351] com.think_e_solutions.application.documentdirectory… - java.io.IOException: The system cannot find the path specified“.

For the 32bits version of Windows: C:\Windows\System32\config\systemprofile\AppData\Local\Temp

For the 64bits version of Windows:

C:\Windows\SysWOW64\config\systemprofile\AppData\Local\Temp

migration-center 3.2.4 Update 4

Features

Outlook scanner:
- Extract the email address of the following recipients: To, CC and BCC
- Extract the number of attachments in the source attribute: AttachmentsCount

migration-center 3.2.4 Update 3

Features

Documentum Scanner and Importer: now support the migration of aspects attached to documents and/or folders. Documentum CS and DFC 6.x or higher required.
New Documentum No Content Copy (“NCC”) Scanner and Importer: these customized variants of the regular Documentum adapters allow for fast metadata-only migration between Documentum, while preserving the references to content files. You can move or copy the content files separately at any point during or after the migration.

Fixes

Documentation: fixed several errata in SharePoint Scanner and Importer User Guides in filenames, paths and URLs used throughout the respective documents
eRoom Scanner: fixed the mc_content_location system rule not appearing for documents scanned from eRoom, thus preventing changes to the content file’s location to be made for import if needed
migration-center Client: fixed issue with import of mapping lists containing keys differing only in character case. Now these are imported and treated as distinct values correctly.
Jobserver: fixed issue with establishing SSL connections from Jobserver with certain combination of adapters
Database Scanner: fixed Unexpected error occurred during content integrity checking message when migrating data scanned with the Database Scanner to target systems with importers supporting the content integrity check feature, which the Database Scanner itself does not support.
File System Scanner: Extended metadata containing values longer than 4000 bytes no longer errors out. Instead, the values are truncated to the max 4000 bytes supported by mc.
File System Scanner: fixed extraction of creation date and owner for files with paths longer than 260 characters
Scheduler: fixed Oracle error when interval for a scheduler was set to “Month”

Changes

SharePoint Importer: The ID that is stored after a successful import of an object in id_in_target_system changed from target list specific ID to global unique ID (GUID). This change maintains compatibility with data scanned using mc 3.2.4 U1 and U2 that still uses the list specific ID

Known Issues

MC Client: Importing a mapping list containing identical keys will silently skip the existing key. Solution: verify mapping lists before importing them to migration-center in order to remove or rename identical keys, mapping lists must always have unique keys.

migration-center 3.2.4 Update 2

Features

File System Scanner: added new option to move successfully scanned files to a local path or UNC network path specified by the user. See the migration-center File System Scanner User Guide for more information on the new moveFilesToFolder parameter.
Installer: added option to specify location of log files during installation of migration-center Server Components. Log files were stored in the installation folder’s logs subfolder by default, now any valid local or UNC network path can be specified for storing the log files generated by migration-center during runtime. See the migration-center Installation Guide for more information about installing migration-center Server Components.

Fixes

Database: removed dependency on Oracle SYS.UTL_TCP and SYS.UTL_SMTP packages for performing manual/regular migration activities. Note the scheduler still requires these packages in order to function, but it is now possible to use all other features of migration-center except the scheduler without the need to have access granted to these Oracle system packages.

migration-center 3.2.4 Update 1

Features

SharePoint Importer: Support for importing folder objects (content type Folder and custom subtypes inheriting from Folder)
SharePoint Importer: Support for creating link objects (content type Link to a Document and custom subtypes inheriting from Link to a Document) (see the SharePoint Importer User Guide for more information)
Documentum Scanner: added “exportLatestVersions” to control the number of latest versions to scan (see the Documentum Scanner User Guide for more information)

Fixes

SharePoint Scanner: fixed an issue causing a "Object reference not set to an instance of an object." error during scan with certain configurations of Date and Time and Yes/No columns

migration-center 3.2.4

Features

New Documentum D2 Importer*: currently supports D2 4.1, including D2 specific features such as autonaming, autolinking, rule based security, applying rules based on the document’s owner. etc.
New SharePoint Scanner*: currently supports SharePoint versions 2007/2010/2013, extraction of Document Libraries, exclusion of selected Content Types or file types, checksum generation for verifying content integrity after import, etc.
New Documentum DCM Importer*: currently supports DCM 5.3, including DCM specific features such as document classes, creating controlled documents, applying autonaming rules, setting specific lifecycle states and associated attributes for imported documents. etc.
Updated SharePoint Importer: now implemented as a SharePoint Solution (just like the new SharePoint Scanner), simplifying deployment and improving performance, reliability and error resilience
Updated SharePoint Importer with content integrity check feature (comparison of checksums computed during scan and import)
Updated Documentum Importer: now supports importing to Documentum 7 repositories
Updated box Importer: now works with the box API version 2.0 for improved performance and reliability; verification of content integrity can now be performed during import based on checksums computed by the importer and box respectively
Updated Documentum Scanner: now supports scanning Documentum 4i repositories (via DFC 5.3)
New parameter for Documentum Importer to toggle whether errors affecting individual renditions should be treated as warnings or errors affecting the entire document, thus preventing it from being imported (set to ignore errors and treat as warnings by default for compatibility with previous versions)
New parameter for Filesystem Scanner to ignore hidden files (false by default)
Documentation: added new document “migration-center Database Administrator’s Guide” detailing database installation requirements (privileges, packages, etc.) and procedures for deploying mc in environments where the regular database setup cannot be run, and the database must be prepared manually by a DBA for installing the mc schema

*New adapters must be purchased separately.

Fixes

Scheduler: fixed an issue where a 24-hour time set was converted to 12-hour time
Scheduler: fixed an issue where the hourly interval was not taken into account if the scheduler was configured to run on a minutely or hourly basis
Documentum Scanner: fixed issue where the “scanNullAttributes” parameter’s functionality was reversed with regard to the parameter’s actual setting in the UI
Documentum/D2/DCM Importer: fixed an issue with the importer not properly handling certain null attribute values (in case the null value of an attribute resulted from null values returned by the If transformation function)
Filesystem Importer: fixed an issue causing errors when importing renditions for a selection of documents if some objects had renditions and others didn’t
FirstDoc Importer: fixed an issue with setting version labels in a particular case (Superseded 1.0 – Obsolete 1.1)
FirstDoc Importer: fixed an issue trying to delete already deleted relations
FirstDoc Importer: fixed objects no being rolled back in case errors occur during saving content
FirstDoc Importer: fixed objects not being rolled back in case errors occur during updating of ACLs and links
Documentation: minor corrections and additions in various documents

Changes

SharePoint Importer now integrates with SharePoint as a SharePoint Solution instead of the separate service component it used up to mc 3.2.3. This changes the requirements and deployment procedures for this adapter. These are described in the mc 3.2.4 SharePoint Importer User Guide
Database: Privilege “CREATE ANY JOB” is no longer required, “CREATE JOB” is now sufficient

Limitations

SharePoint Scanner only scans complete Document Library items and cannot scan individual folders currently
SharePoint Importer only supports versioning using contiguous numbering

Known Issues

Scheduler: setting a time interval (the hours during which the scheduler is allowed to run) for a scheduler configured to run on a minutely or hourly basis will not be reflected correctly in the “Next run date” displayed. The scheduler will run as configured, the issue affects only the display of next run date.
Remaining issues from the main release 3.2 to 3.2.3 also apply to release 3.2.4 unless noted otherwise.

migration-center 3.2.3

Features

New Microsoft Outlook Scanner – scan specified folders of a user’s mailbox from Outlook 2007-2010, including messages and their properties, choose to exclude particular subfolders or properties, etc. See the migration-center 3.2.3 – Outlook Scanner User Guide document for more information about this new adapter. As with all adapters, the Microsoft Outlook Scanner is available for purchase separately.
FirstDoc Importer now officially supports FirstDoc versions 6.3 and 6.4 (R&D)
Documentum Importer now supports setting rendition page modifiers and rendition file storage locations through transformation rules.

Fixes

eRoom Scanner: fixed an issue leading to out of memory errors when scanning large files (several hundred megabytes each)
eRoom Scanner: fixed an issue with the skipContent parameter not working properly in some cases
Filesystem Scanner: fixed an issue where the scanner could fail to scan files equal to or larger than 4GB in size
FirstDoc Importer: fixed a memory leak issue related to session management

Changes

Documentum Scanner & Importer: Changed handling of rendition page modifiers (now exposed as a dedicated system attribute that can be set using transformation rules). See the “migration-center 3.2.3 – Documentum Importer User Guide” document for more information about this feature and how it may affects renditions scanned using older versions of the Documentum Scanner.
eRoom Scanner: changed naming of files extracted by the scanner to the mc storage location. This has been done in order to line up the eRoom Scanner’s behavior with the rest of mc’s scanner’s, which all use an object ID rather than actual filenames for the extracted content. This should not affect the regular workflow of a migration from eRoom, as all changes are handled internally by mc and are not exposed to the user (nor does or did the user need to interact with said files at any time).

Limitations

The FirstDoc Importer cannot set Documentum system attributes such as creation and modify dates, creator and modifier user names, the i_is_deleted attribute, and so on. Setting these attributes is only supported using the Documentum Importer.

Known Issues

If a source attribute has the same name as one of migration-center’s internally used columns, the Client will display the value of the migration-center internal attribute for the source attribute as well. This occurs in Source Objects view. The issue also persists in the data exported to a CSV file if the CSV export is performed from Source Objects view. Other views or the View Attributes window will display the correct value of the respective source attribute.
Remaining issues for the main release 3.2 also apply to release 3.2.3 unless noted otherwise.

migration-center 3.2.2

Features

Documentum Content Validation – checks content integrity of documents migrated to a Documentum repository by verifying an MD5 checksum of content files before and after the import. From a Documentum source both primary content and renditions can be checked, for other sources only the primary content is supported. The feature is available as an option in the Documentum Importer. See the migration-center 3.2.2 – Documentum Importer User Guide document for more information about this feature.
Documentum Audit Trails – the Documentum adapters now support the migration of audit trails. The Documentum Scanner can scan the audit trails of any folders and documents within the scanner’s scope. Audit trail objects can then be added to an Audit Trail migset and processed just like documents and folders. Finally, audit trails can be imported using the Documentum Importer. The new options available in the Documentum adapters for controlling and configuring the migration of audit trails are described in the migration-center 3.2.2 – Documentum Scanner User Guide and migration-center 3.2.2 – Documentum Importer User Guide respectively.
The Documentum Scanner has a new option, allowing an additional DQL statement to be executed for every object within the scope of the scan. This allows additional information to be collected for each object from sources other than the document’s properties. Other database tables (registered tables) for example can be queried using this option. See the migration-center 3.2.2 – Documentum Scanner User Guide for more information.
The Filesystem Scanner can now build versions from additional attributes (as long as these attributes can supply the required information. Two attributes per object are required for the feature to work, which can be named arbitrarily and can be defined in the fme metadata files associated with each content file. The attributes to be used as version identifier and version number by the scanner can be specified through additional options available in the Filesystem Scanner. See the migration-center 3.2.2 – Filesystem Scanner User Guide for more information.
The SharePoint Importer can now update previously imported objects (if the objects had been modified in the source and scanned as updates)

Fixes

Documentum Importer: Fixed an issue where importing an update to a Microsoft Word 8.0-2003 document would set the wrong format for the updated document (a_content_type was set to “doc” instead of “msw8” previously)
Documentum Importer: Fixed an issue with importing Filesystem objects to Documentum when user defined version labels were set in transformation rules. Depending on how the labels were set, documents could end up all having CURRENT labels set, or having no CURRENT labels at all. Now the most recently imported version is always set as CURRENT, regardless of the CURRENT label set in Transformation Rules.
Transformation Engine: Fixed an issue where Validating a migration set would validate objects that have been already validated again, which didn’t make sense.
Scheduler: fixed issues where a Scheduler would stop working after several hundred runs and required to be stopped and restarted manually, or report a run as having finished with errors although there were none.
SharePoint Importer: Fixed an issue with the SharePoint Importer where it would set values of multiline columns as literal HTML code sometimes.
SharePoint Importer: Fixed an issue where during import the Comment column was set to value “Version checked in by migration-center” automatically.
Box Importer: improved performance and reliability when importing large files (several hundred MB)
Box Importer: fixed issue where the progress would indicate 100% for the entire duration of a large file being uploaded, with no actual progress being visible to the user
Box Importer: fixed an issue where the max number of threads allowed is set for the importer and wrong credentials are entered – this caused connections to the database to be opened but not closed, exceeding the maximum number of connections allowed by the database after some time.

Changes

FirstDoc Importer: Changed handling of application number from regulatory dictionary to work around issue where duplicate application numbers are used. See the migration-center 3.2.2 – FirstDoc Importer User Guide document for more information about this feature.

Limitations

Documentum Content Validation relies on a checksum computed during scan to compare the checksum it computes after import against. Currently only the Documentum and Filesystem Scanner have the option of computing the required checksum during scan (Documentum Scanner: main content and renditions; Filesystem Scanner: only main content). Scanners for other source systems could of course be extended to also compute a checksum during scan time which the Documentum Content Validator could use.

Known Issues

The SharePoint Importer and the FirstDoc Importer cannot function together due to some conflicting Java methods. Workaround: use only one of the adapters on a given machine by deleting the other adapter’s folder from the migration-center Server Components libs subfolder.
Same issues as for the main release 3.2 also apply to release 3.2.2

migration-center 3.2

Features

As a major feature, migration-center 3.2 provides several new adapters for various source and target systems, listed below. The interaction between scanners and importers has also been reworked, so now scanners and importers are no longer required to exist in pairs in order to work together. Instead, any scanner/importer exists as a generic, standalone adapter that can be used in combination with any other available scanner/importer, allowing for any combination of source and target systems possible using the available adapters.

The adapters available for migration-center 3.2 are:

Documentum Scanner
eRoom Scanner
Filesystem Scanner
Database Scanner
Documentum Importer
Filesystem Importer
Box Importer
Alfresco Importer
SharePoint Importer
FirstDoc Importer

Filesystem-Documentum adapters have been discontinued and replaced by generic Filesystem and Documentum adapters respectively. The new adapters perform the same while at the same time allowing interaction between any scanner and importer.

Consult the individual adapters’ User Guide documents for detailed information regarding supported functionalities and systems.

Changes and Improvements

migration-center 3.2 now requires the Java Runtime Environment 1.6 It will not work with older versions.
Filesystem-Documentum Scanner has been replaced by generic Filesystem Scanner which generates content able to be migrated using Documentum Importer or other importers to the respective target systems
Documentum Scanner is no longer connected to the Documentum Importer and generates content able to be migrated using other importers as well
Filesystem-Documentum Importer has been replaced by generic Documentum Importer which can accept and import content from any scanner, including filesystem or Documentum sources
Documentum Importer: improved performance for importing large numbers of Virtual Documents by up to 15% by skipping some unneeded processing
Documentum Scanner: content files are now written to the export location by using the Windows/DOS extension corresponding to the respective formats instead of the Documentum format name used previously. This is necessary to facilitate interaction between the Documentum Scanner and various importers targeted at other content management systems that rely on the Windows/DOS extension to work properly.
Documentum Scanner has a new “skipContent” parameter - enabling it skips extraction of the actual content, thus saving time during scan. The feature is intended to be used for testing or other special purposes; not recommended to enable for production use.
Filesystem Scanner can now detect changes to external metadata files as well instead of just the main content file and can add the updated metadata as an update to the object. This is useful for scenarios where the content is expected to change less frequently than the external metadata files.
Filesystem Scanner now also supports reading external metadata for folders, providing the same functionalities as for file metadata.
Filesystem Scanner has a new “ignoreAttributes” parameter – this allows defining a list of unwanted attributes that should be ignored during scan. This mostly applies to attributes coming from external metadata files of from extended metadata from the files’ contents
Filesystem Scanner has a new “ignoreWarnings” parameter – this allows scanning documents even if some of their metadata (owner or creation date) cannot be extracted or if the external metadata file is missing. Use only if that kind of information is not critical for the migration, and/or would otherwise cause too many files to be skipped.
Client/Transformation Engine: system attributes are no longer available to be associated on the “Associations” page of the “Transformation Rules” window. These are used automatically by the respective importers and did not need to be associated anyway
Client/Transformation Engine: set default rule to be pre-selected in the “Get type name from” drop-down list on the “Associations” page for all the different object types (user can of course change the selection if desired)

Fixes

Documentum Scanner: fixed an isolated issue where a particular combination of scanner parameters, source attribute values and objects without content would prevent such objects from being scanned
Documentum Importer: fixed an issue where adding a new version of a document with less values for a repeating attribute than the previous version had for the same repeating attribute would keep the previous version’s repeating attribute values which exceeded the number of attribute values present in the currently added version
Documentum Importer: fixed an issue where an underscore character (“_”) in a rendition file’s path was interpreted as a page modifier separator, leading to unexpected errors. Now the “_” character’s role as a page modifier separator correctly applies to a rendition’s filename only, not its entire path
Documentum Importer: fixed an issue where the CURRENT version label was not updated correctly in case an object and updates of that object were imported together
Documentum Importer: fixed an issue where updating folder objects during import would not clear empty attributes in case those attributes had values different from null before the update.
Filesystem Scanner: fixed an issue where the modification date for folders would not be scanned
Filesystem Scanner: removed “content location” value for folder objects added previously by scanner since folders do not have content of their own. This does not influence functionality.
Filesystem Scanner: fixed error extracting extended metadata from PDF files
Filesystem Scanner: fixed some discrepancies with object counts in log files when scanning updated objects
Transformation engine: a multi-value rule is now validated against the multi-value setting of the attribute it is associated with, so trying to set multiple values for an attribute defined as single value for example will now throw a validation error.
Client: fixed an issue where the Object History feature did not return results at all under certain circumstances
Client: fixed multiple minor issues related to focusing and scrolling in various grids/lists

Limitations

Database: installation of the Oracle database schema can only be performed with the “sys” user
Database: multiple mc database schemata are not supported on the same database instance
Database installer: avoid running the database installer from folder locations containing “(“ or “)” characters in the path
Documentum Features: Cabinets: dm_cabinet objects currently cannot be represented as individual objects in mc, hence mc cannot store or set specific attributes for cabinets eitheri
Documentum Features: Cabinets: mc cannot migrate empty cabinets (i.e. dm_cabinet objects with no other objects linked to it) for reasons stated abovei
Documentum Features: Relations: support for migrating dm_relation type objects is limited to relations between dm_document and dm_folder type objects (and their respective subtypes)i
Documentum Features: Virtual documents: the snapshot feature of virtual documents is not supportedi
Documentum Scanner: for the DQL query to be accepted by the scanner it must conform to the following template: “select r_object_id from <dm_document|subtype of dm_document> (where …)”. Other data returned by the query or object types other than dm_document (or its subtypes) is not supported.
Documentum Scanner: the ExportVersions option needs to be checked for scanning Virtual Documents (i.e. if the ExportVirtualDocuments option is checked) even if the virtual documents themselves do not have multiple versions, otherwise the virtual documents export might produce unexpected results. This is because the VD parents may still reference child objects which are not current versions of those respective objects. This is not an actual product limitation, but rather an issue caused by this particular combination of Scanner options and Documentum’s VD features
Documentum Scanner: scanning dm_folder type objects using the DQL option is not supported currently.
Documentum Importer: the importer does not support importing multiple updates of the same object in the same run (or the base object and of its updates). Every update of an object must be imported in different runs (this is how it would happen during normal conditions anyway)
Update migration: Objects deleted from the source will not be detected and the corresponding target object will not be deleted (if already imported)
Update migration: whether a source object has been updated is determined by checking the i_vstamp and r_modify_date attributes; source objects changed by third party code/applications which do not touch these attributes might not be detected by mc
Client: the client application does not support multi-byte characters, such as characters encoded with Unicode. This does not affect the migration of such information, but merely limits the Client’s ability to display such values on screen; multi-byte characters are supported and processed accordingly by migration-center’s Database and Server components, which are the components involved in the actual data processingi
eRoom Scanner: scanning updates of eRoom objects is not supported currently; this also applies to newly added versions to a previously migrated object. Only newly created objects will be detected and migrated. This is due to the way eRoom handles object IDs internally, which prevents mc from correctly detecting changes to versioned objects. The full functionality may be implemented in a future release.
Filesystem Importer: does not import folders as standalone objects. Folders will be created as a result of the path information attached to the documents though, so folder structures are not lost. The full functionality may be implemented in a future release.
Database Scanner: the database adapter does not extract content from a database, only metadata. Content can be specified by the user during the transformation process via the mc_content_location system attribute. It may be necessary to extract the content to the filesystem first by other means before migration-center can process it. Content extraction functionality may be implemented in a future release.
Attributes: The maximum length of an attribute name is 100 bytesi
Attributes: The maximum length of an attribute value is 4000 bytesi
Attributes: The maximum length of a path for a file system object is 512 bytesi Note: all max supported string lengths are specified in bytes. This equals characters as long as the characters are single-byte characters (i.e. Latin characters). For multi-byte characters (as used by most languages and scripts having other than the basic Latin characters) it might result in less than the equivalent number of characters, depending on the number and byte length of multi-byte characters within the string (as used in UTF-8 encoding)

Known Issues

Documentum Importer: During runtime, the importer creates a registered table for internal use; this table will be not deleted after the import process has finished because it might be used by other importers running concurrently on other Jobservers. Since an importer job running on one Jobserver does not know about any importers that may be running on other Jobservers, it cannot tell whether it is safe to delete the table, which is why it is left in place. This registered table does not store actual data; it acts as a view to data already stored by Documentum. It is safe to remove this registered table once the migration project is finished. The following query is used to create the registered table: register table dm_dbo.dm_sysobject_s (r_object_id char(16), r_modify_date DATE, r_modifier char(32), r_creator_name char(32))
Documentum Scanner: if several scanners are running concurrently and are scanning overlapping locations (e.g. due to objects being linked into multiple locations), a scanner might detect an object scanned earlier by another scanner as an update, although nothing has changed about that object. This has been observed in combination with relations attached to the objects. This will result in some redundant object appearing as updates while they are in fact the same, but apart from this the final result of the migration is not affected in any way. The redundant objects will be handled like regular objects/updates and imported accordingly.
Documentum: If the group_permit, world_permit, owner_permit AND acl_domain, acl_name attributes are configured to be migrated together the *_permit attributes will override the permissions set by the acl_* attributes. This is due to Documentum’s inner workings and not migration-center. Also, Documentum will not throw an error in such a case, which makes it impossible for migration-center to tell that the acl_* attributes have been overridden and as such it will not report an error either, considering that all attributes have been set correctly. It is advised to use either the *_permit attributes OR the acl_* attributes in the same rule set in order to set permissions.
Transformation engine: Transformation Rules window/Associations page: if object types are added and no attributes are associated for these object types all objects matching the respective types will be migrated with no attribute values set (except for system attributes handled automatically by Documentum). Avoid adding object types without associating all required attributes for the respective types.
Client: on rare occasions, some windows or parts of a window will flicker or fail to refresh their contents. To work around issues like these, use the window’s Refresh button, scroll the window left/right, or close and re-open the affected window if nothing else helps.
Client: when editing an object type’s definition and trying to change an attribute’s type the drop-down list will not appear on Microsoft Windows 7 systems unless miglient.exe is configured to run with “Disable visual themes” (option can set on the Compatibility page in the executable’s properties)
Client: trying to import a mapping list from a file will not work on Microsoft Windows 7 systems because the context menu containing the command will not appear unless miglient.exe is configured to run in Windows XP compatibility mode (option can set on the Compatibility page in the executable’s properties)
Client: After using "Export to CSV" the folder where the CSV file has been saved is still in use by migration-center Client
Scheduler: the scheduler may report runs that finished with warnings during import as having finished with errors instead. Make sure to check any scheduler history entries listed as “Finished with errors” to determine whether the cause is an actual error condition or merely a warning, which is not a critical condition.
Installer: The migration-client installer does not work with User Account Control enabled in Windows 7. Please either disable UAC for the duration of the installation, or if installation needs to be performed using UAC enabled, manually grant full access permissions for the required users on the installation folders afterwards

Database update from previous versions: as a result of the update process increasing the default size for attribute fields to 4000 bytes (up from the previous 2000 bytes), Oracle’s internal data alignment structures may fragment. This can result in a performance drop of up to 20% when working with updated data (apparent when transforming/validating/resetting). A clean installation is not affected, neither is new data which is added to an updated database, because in these cases the new data will be properly aligned to the new 4000 byte sized fields as it is added to the database.

Client User Guide

Introduction

Documentum Scanner
Filesystem Scanner
Database Scanner
Outlook Scanner
SharePoint Scanner
Documentum NCC Scanner
Alfresco Scanner
Domino Scanner
Exchange Scanner
OpenText Scanner
Documentum Importer
Alfresco Importer
Filesystem Importer
D2 Importer
box Importer
Alfresco Importer
SharePoint Online Importer
SharePoint On-Premises Importer
Documentum NCC Importer
InfoArchive Importer
OpenText Importer
Veeva Importer
Trackwise Digital Importer
Cara 5.3 Importer

migration-center client 3.13 cannot be used to connect to a migration-center database 3.4 as well as migration-center client 3.4 cannot be used to connect to migration-center database 3.13.

Other adapters can of course be developed according to customer or project requirements.

Product features

The transformation of metadata through intelligent, rule-based mapping of objects
Incremental delta migrations for newly created and changed documents and folders
Identification of duplicates
one-time or regular batch migration of large volumes of documents and folders
Complete audit ability, error reporting and error handling
1:1 migration of folder structures and mapping of objects within, including metadata
1:1 migration of relations (i.e. Virtual Documents, etc.)
The possibility to export and import type templates, transformation rules, object grids (export only) to CSV files.

Features that were never/rarely used have been removed, resulting in a cleaner, more user-friendly and productivity-oriented GUI.

Compatibility

Architecture Diagram

All of the aforementioned information is defined and controlled via the migration-center Client, a GUI application which acts as the command and control center for migration-center.

Multiple instances of the above components can be deployed across the network, even across geographical boundaries, and can work together on the same or multiple projects.

The actual content moved during a migration (files, documents, emails) is stored on the file system (local or remote) and does not require a specific migration-center component for managing.

Not all adapters illustrated in the above diagram are part of the standard product package.

Client

During installation a default user is created. If no other users have been created, use the default user account shown below for logging on to the migration-center database.

Username: fmemc

Password: migration123

Job Server

Database

The migration-center database contains most of the data processing logic and stores all the migration sets, including their objects, metadata and transformation rules.

As with the other components, at least one database needs to be installed for migration-center to work, but several databases can be deployed in the same environment, depending on the requirements.

Both the Client and the Job Server need to connect to a migration-center database to function.

Client Known Issues

Changes done in the Object Types window will not be saved in the following scenario:

Clicking on Cancel in any of the screens will revert the object type changes. Clicking on Ok on any of the mentioned screens will save the object type changes.

This only occurs when the client was freshly opened. After going to any of the mentioned screens the object type screen will behave as expected.

Client Functionalities

This chapter details the features and functionalities available to the user through migration-center Client.

Main application window

migration-center’s main application window adheres to typical windows applications’ user interface, offering a main menu bar and toolbar for accessing general, frequently used features.

Major features and functionalities each open in their own child windows, providing toolbars and context menus that apply to the items displayed in that window.

Any combination of child windows can be opened at the same time; child windows can be tiled or cascaded.

<migration-center> <Connect to…> Calls up a dialogue window where the user can connect to a (another) migration- center specific database.
<migration-center> <Reconnect> Reconnects client to currently used migration-center DB
<migration-center> <Renew License> Allows entering a new license key
<migration-center> <About> Displays information about licensing and product versions
<migration-center> <Exit> Exits migration-center
<Manage> <Scanners…> opens the -Scanners- window for creating and managing scanner jobs
<Manage> <Migration Sets…> open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
<Manage> <Importers…> opens the -Importers- window for creating and managing importer jobs
<Manage> <Jobs…> open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
<Manage> <Jobservers…> open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
<Manage> <Schedulers…> opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
<Manage> <Object Types…> opens the -Object Types- window, where the object type definitions can be imported and edited (these are required for associating transformation rules with for the attribute model transformation are defined)
<Manage> <Global Mapping Lists> opens the -Global Mapping Lists- window, where global mapping lists can be created or imported. A global mapping is automatically available for use in any migration set.
<Manage> <Object History…> opens the -History- window where it is possible to look up any object that underwent processing in migration-center. Key steps of the objects lifecycle during the migration can be viewed, such as the objects status, metadata and the exact timestamp at that moment.

The main toolbar provides quick access to frequently used functionalities. These correspond with the same functionalities available from the main menu.

[Scanners] opens the -Scanners- window for creating and managing scanner jobs
[Migration Sets] open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
[Importers] opens the -Importers- window for creating and managing importer jobs
[Jobs] open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
[Jobservers] open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
[Schedulers] opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
[Cascade] Arranges all open child windows in a cascade
[Tile Horizontally] Tiles all open child windows horizontally
[Tile Vertically] Tiles all open child windows vertically

Job Servers

To input/output any data from/to migration-center there needs to be at least one location with a running Job Server defined.

The location for a Job Server can be specified as the machine’s hostname or fully qualified domain name, if required, or simply as an IP address.

Note: Do not use loopback addresses or the “localhost” name.

The toolbar and the context menu of the -Jobservers- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[Refresh] is useful for rechecking the status of the locations listed
[New] opens a blank -Jobserver- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a location from list

Properties window

To create a new Job Server location

select <New> from the context menu or from the toolbar
Enter the IP address or hostname of the machine the Job Server is running on and the port number it is listening on.
Enter the appropriate port on which the Job Server on that machine was configured

The default port number is 9700, unless changed during setup.

Creating a scanner and trying to save it without having any Job Servers defined will prompt the user to define one Job Server location and save the scanner with the new created Job Server location.

Scanners

The toolbar and the context menu of the -Scanners- window contain the same commands.

[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[History] opens the -History- window where more information about all scans performed by the selected scanner can be viewed. The log files from these scans can also be accessed here.
[View Scanned Objects] will display a grid with the scanned objects in a new window, where these can be viewed, filtered, and exported to a CSV (comma separated values) file. This functionality does not allow any editing to be performed on the objects.
[Run] initiates the run of a scanner with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the scanners listed
[New] opens a blank -Scanner- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected scanner, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

All scanners, regardless of type share some common parameters

Name: must be a unique name for the current scanner

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the scanner depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this scanner does or add some comments related to the scanner

In the lower part of the windows the view can be filtered based on the available source attribute. Select an attribute from the list, enter Filter value, and press ENTER or [Apply filter]. To remove any applied filters, press the [Clear filter] button. Enter “(null)” to list attributes for which the selected filter attribute has no value

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Filtering attributes and (de)selecting columns is neither permanent, nor does it affect the actual data in any way. It only affects the current display and serves to analyze the data on display.

Migration Sets

It is possible to add multiple scans to one migration set, as it is possible to split a scan containing a large number of objects onto several migration sets.

Buttons and context menu items:

[Properties] is used to change the configuration of an item in the list. The -Migration Set- properties window can also be opened by double-clicking an item in the list
[Edit Transformation Rules] opens the window where transformation rules are managed and rules are applied
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Transform and Validate] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
[Reset All] resets all objects of in the selected migration set to their initial attribute values, essentially removing any processing performed during the transformation process and resetting status changes resulting from validation and import. The [Reset All] button is available in all views (-Source Objects-, -Processed Objects-, and -Error Objects-).

Resetting imported objects would typically be used during testing phases of a project and not during production.

[Stop] appears as a button only when the selected migration set is busy being transformed or validated, and only if these processes take longer than 3 seconds for the selected migration set.
[Refresh] refreshes data in all info-columns
[View All Objects] (press drop-down list/button combo) opens 3 windows: Source Documents, Processed Documents, and Error Documents.
[Source Objects] (select from drop-down list/button combo) displays all scanned objects attributed to the respective migration set. Original attributes and attribute values are displayed.
[Processed Objects] (select from drop-down list/button combo) displays the objects after they have been processed by applying the transformation rules.
[Error Objects] (select from drop-down list/button combo) displays objects that failed transformation/validation criteria, or failed to import, indicating the cause
[New] opens a blank -Migration Set- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured migration set from list

Items in columns can be sorted by clicking on the column headers. Clicking the same column header repeatedly switches between ascending/descending sorting order.

Migration Set Properties window

Common controls

The buttons on the right side of the –Migration Set Properties- window are available on all tabbed pages of this window.

The [OK], [Apply] and [Cancel] buttons are self-explanatory and work as expected.

Otherwise define a selection of objects as described in the following chapters before saving the migration set.

The [Preview objects] button and preview functionality is available on all pages of the –Migration Set Properties- window.

If the selection is ok, the selected scans can be assigned to the migration set using the [Select objects] button in the upper right corner.

The |Properties| page

The |Filescan selection| page

This page allows the user to select and assign any available scan to the current migration set.

Only scans matching the object type set on the previous page (|Properties|) will be listed here. Check the object type selection if the desired scans are not displayed here.

The |Exclude objects by values| page

This page offers a simple way of excluding some objects from the selection based on their values for one or more attributes.

In previous versions of migration-center this page was called |Filters|. The functionality provided is the same in all product versions.

The |Advanced Filters| page

For objects matching a filter expression means these objects will be added to the migration set, not excluded!

Conditional operators for working with string, numeric, and date/time type values are provided.

Migration Set Views

Three views are available for a migration set: Source Objects, Processed Objects, and Errors. These views can be accessed in the -Migration Sets- window by context menu or toolbar. Each window displays a different view of the current migration set objects, including their metadata, current status, associated migration-center internal information, messages related to the various processing steps, etc.

One, two, or all three of these three views can be opened at any time for the current migration set.

The filter bar at the bottom of the window is common to all views and offers following functionalities:

Select an attribute from the list, enter Filter value, and press ENTER or [Apply filter]. To remove any applied filters press the [Clear filter] button. Enter “(null)” to list objects for which the selected filter attribute has no value Use “_” (underscore) as a placeholder for one character, and “ % “ (percent) for zero or more characters.

Other features common to all views are the following toolbar items:

Choose which columns to display using the [Show/Hide Columns] button in the toolbar.

Click [Refresh] to reload the content of the current view

Source Objects View

The checksum is computed during scan and must be activated in the scanner. See the respective adapter documentation for more information about using the checksum option in the scanner.

Toolbar items

[View Attributes] displays all available (scanned) attributes of one object only. Previous and following object can also be browsed by using the appropriate buttons.

[Edit Transformation Rules] opens the -Transformation Rules- window.

Items in columns can be arranged (increasing/decreasing) by clicking on the column labels.

[View Duplicate Objects] filters identifies and displays duplicate objects

Context menu items

<View Source Attributes> will display all attributes of that object in a small new window and allows browsing through the list objects by object (back and forward.

Copying the currently selected value to the clipboard function can also be performed directly using the <Copy Value> context menu entry or using the {Ctrl+C} keyboard shortcut

<Filter by Value> will immediately filter the view using the selected attribute value. Use the [Clear Filter] button in the filter bar at the bottom to remove a filter and return to the full list.

Processed Objects View

In addition to the -Source Documents- view, the -Processed Documents- window features a few more functions.

Transforming, validating, and the resetting objects can be performed directly from this window using the appropriate toolbar buttons.

[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Apply Transformation & Validation] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.

For quick access to the source attributes of any object the -Source Attributes- window can be also accessed from the context menu.

Resetting imported objects would typically be used during testing phases of a project and not during production.

Error Objects View

All this information makes it easy to identify and correct the error and transfer the object into the proper states allowing it to be migrated.

There are five basic ways of handling errors; one of them should always be able to solve typical errors encountered during migration

If the error is caused by a transformation rule (objects in Transformation Error status), adjust the transformation rule to cover the faulty objects and re-apply it to those objects. Any objects which have already been transformed, validated or transformed successfully will not be affected and will maintain their current status and metadata
If the error cannot be fixed by adjusting transformation rules and the object is in the Transformation Error status, it may be necessary to edit objects manually. Use the <Edit Attributes> window to browse through the objects and edit the attributes manually, skipping any objects where manual editing does not apply.
If the error occurs during import due to external factors (objects in Import Error status) like software or hardware issues, network issues, target system configuration problems, proper access permission, etc.) try to remedy the error first and after that just add the migration set back to the importer and simply try to import it again. Migration-center will attempt to import all objects in the Import Error state automatically and will succeed if the cause for the error has been fixed.
If objects have failed Validation (objects in Validation Error status) check the Object Type definitions and Associations in the Transformation Rules window, make the necessary corrections and re-run the Validation process to validate these objects successfully.
Finally, if none of the above helps, or the objects having failed are not even needed to be migrated they can either be removed from the migration set or left in their respective error states (temporarily or permanently). Unless other objects depend on them, leaving unneeded objects in an error state or removing them from the migration set will not affect other objects.

Transformation rules

The -Transformation rules- window has three tabs for managing different aspects of transformation and validation, and a set of common buttons accessible from all tabs. These buttons are:

[Copy Rules from...] allows copying a set of transformation rules from another migration set and applies it to the current migration set, thereby replacing any existing rules. Rules can only be copied from migration sets of the same type as the current migration set. If no migration sets of the same type exist, the list will be empty. Any migration sets of the same type will be displayed in a list, allowing the user to select one and click [Ok] to copy and apply its transformation rules to the current migration set.

[Export] will export the current migration set’s transformation rules to a XML file for the purpose of backing up or transferring the model to a migration set located on another DB.

[Import] imports transformation rules generated by the XML files generated with the previous export function. Warning! Importing rules from a file will replace all rules configured previously!

[Ok] saves changes and closes window.

[Cancel] cancels all changes that have not been applied yet and closes window.

[Apply] saves changes without closing the window.

Rules

Rules list

Rules for system attributes

The migration-centers adapter targeted at Documentum provide following system attribute rules:

“dctm_obj_link” specifies the path(s) where a document should be linked to in the target repository. “dctm_obj_link” is also an example for a system attribute which is defined by and used internally by migration-center and does not require association with an actual target attribute. This means there is no need for an actual attribute named “dctm_obj_link” to exist in either the source or target system.
The system attribute “dctm_obj_rendition” specifies file system location to files which represent renditions. Zero or more renditions can be specified for any object and will be added by migration-center during import. This attribute is also an internal attribute used by migration-center similar to “dctm_obj_link”
“r_folder_path” and “r_object_type” can get their default information automatically (see [Generate attributes]) only inside a Documentum-to-Documentum migration set. In the migration set of a file system scan, this information must be put in by user.

Rule properties

Transformation methods

Error and Warning indicators for rules:

In the example below, the rule called “extension” has been marked yellow by migration-center’s rule auto verification feature.

Marking rules and transformation steps as errors works exactly the same as described above for warnings.

Errors should never be ignored because they will definitely affect, or even prevent transformation rules from working correctly.

Working with transformation functions

Transformation functions provide the functionality behind transformation rules.

Custom functions can be implemented and used along the standard functions provided by migration-center.

A function’s parameter always has two properties: a type and a value. The parameter’s value is determined by the parameter’s type. Parameters can be of the following types:

User Value: acts as a constant value and will be set for all objects to which the current rule applies.

In case the parameter in question is referring to a single value attribute selecting an Index will have no effect.

The Index selection list offers the following options:

All is the default option and can be used only for the first parameter and not the subsequent parameters.

The Index selection offers some more options when used with the If transformation function:

It is also possible to specify All as an option for the Return if true or Return if false parameters.

Transformation function reference

Mapping Lists

The |Mapping List| page allows the user to create, duplicate, delete and edit entries for mapping lists using the buttons provided in the toolbar. Mapping lists also have two properties:

“Case sensitive” will consider a valid match only if the case of the input value matches the key and its case. Usually this would not be used, enable if needed.

Migration-center will interpret this transformation rule as follows:

Global Mapping Lists

Associations

Types

Associations

Importers

Importers migrate one or more migration sets. Only migration sets with objects that have successfully passed both transformation and validation can be imported.

The toolbar and the context menu of the -Importers- window contain the same commands.

[Properties] opens the configuration details of that particular importer. The -Importer- properties window can also be opened by double-clicking an item in the list.
[History] opens the -History- window where more accurate data about all imports performed by the importer in question can be accessed. The log of the scans can be accessed here.
[Run] initiates the run of an importer with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the importers listed
[New] opens a blank -Importer- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

The -History- windows displays various information about the selected importer, such as the number of processed objects, the start and ending time and the status of each run.

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

All importers, regardless of type share some common parameters

Name: must be a unique name for the current importer

Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the importer depend on the adapter selected here.

Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list

Description: provides space to enter an optional description of what this importer does or add some comments related to the importer

Scheduler

A Scheduler acts as a master process which manages and runs an end-to-end migration fully automated and according to a set schedule.

The toolbar and the context menu of the -Schedulers- window contain the familiar items: New, Properties, Delete, Run, Refresh.

Double-clicking a scheduler in the list has the same function as the <Properties> menu command.

Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.

A scheduler can be run immediately if needed, regardless of its actual schedule, through the <Run> command found in the toolbar and the context menu.

Properties

The |General settings| tab contains only three directly editable fields: “Name”, “Description” and a checkbox called “Active”. The checkbox changes the status of (Active: true/false) of a configured scheduler. If no flag is set (status: false) a configured scheduler will not run at the configured time and date. The Info dynamic area displays various information about the scheduler.

In the |Configuration| tab, the appropriate (previously configured) scanners, importers and attribute models are selected. Caution is to be executed when choosing an attribute model, so that an attribute model of similar purpose and type to scanner and importer shall be chosen.

The |Interval| tab allows the configuration of the time and date the scheduler will begin to run and stop to function. Also, the frequency of the runs can be set up here.

The |History| tab provides information about various configuration aspects and run dates of the scheduler, of the current state, last operation undertaken, as well as giving access to the logs.

Object Types definitions

Out-of-the-box the product comes with basic object type template definitions for the adapters the customer has acquired licenses for.

Additional user/custom object types can be added to migration-center by the user.

id,2,0,0,0,0

first_name,2,0,0,0,0

last_name,2,0,0,0,0

email,2,0,0,0,0

gender,2,0,0,0,0

ip_address,2,0,0,0,0

animal_name,2,0,0,0,0

Valid values for data type are: 0 - Boolean, 1 - Integer, 2 - String, 3 - ID, 4 - Date and Time, 5 – Double.

^[^\~\"\#\%\&\*\:\<\>\?\/\\\{\|\}]+$

It will match one or more characters between begin and end of the string that are not in the list of the following characters: ~"#%&*:<>?/\{|}

^[0-9]{3}-[0-9]{5}$

Object History

The -History- window can be accessed from migration-center’s main menu <Manage>, menu item <Object History…>

It is possible to search Object History by type of object or attribute values, including internal system attributes used by migration-center. The search parameters also support wildcards such as “_” (underline) for one character, and “ % “ (percent) for zero or more characters. The button displays the full set of attributes associated with the selected object at that respective point in time. This can prove useful in case some attributes are wrong or missing after import to the target system – by verifying the attributes assigned to the object by migration-center at the time of import it is possible to determine whether any of the inconsistencies encountered in the target system are due to migration-center or not.

Log files

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated for each job can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job

SharePoint Online Importer

Introduction

The Microsoft SharePoint Online Importer allows migrating documents, folders, list items and lists/libraries to SharePoint Online offering following features:

Import documents to Microsoft SharePoint Online Document Library items
Import URL / Link documents
Import folders to Microsoft SharePoint Online Document Library items
Import list items to Microsoft SharePoint Online Document Library items
Import lists/libraries to Microsoft SharePoint Online Sites
Set values for any columns in SharePoint Online, including user defined columns
Set values for SharePoint Online specific internal field values
Create documents, folders and list items using standard SharePoint Online content types or custom content types
Set permissions for individual documents, folders, list items and lists/libraries
Set the folder path and create folders if they don’t exist
Apply Content Types automatically to lists/libraries if they are not applied yet
Delta migration
Import versions (minor or major, automatic enabling of versioning in the targeted document library)
Import files with a size up to 15 GB
Set Microsoft Information Protection (MIP) Sensitivity Labels on content files prior to uploading them to SPO

The SharePoint Importer is implemented mainly as a Job Server component but comes with a separate component for setting SharePoint Online specific internal field values, which can be installed optionally and if necessary.

Starting with migration-center version 3.13 Update 2 the SharePoint Online importer only supports app-only principal authentication. It is not possible to use user name / password authentication with this and later versions. Please consider this before you upgrade your existing installation.

SharePoint Online Importers can be created, configured, started and monitored through migration-center Client, while the corresponding processes are executed by migration-center Job Server and the migration-center SharePoint Online Importer respectively.

The term importer is used in migration-center for describing a component which imports processed data from migration-center to another system.

Installation

To install the main product components, consult the migration-center Installation Guide document.

The migration-center SharePoint Importer requires installing an additional, separate component besides the main product components. This additional component is able to set system values (such as creation date, modified date, author and editor) as well as taxonomy values for your objects. It is designed to run as a Windows service and needs the .NET Framework 4.7.2 installed on your computer, which is running this service as well as the migration-center Job Server.

The app-only principal authentication used by the importer calls the following HTTPS endpoints. Please ensure that the job server machine has access to those endpoints:

<tenant name>.sharepoint.com:443
accounts.accesscontrol.windows.net:443

This component must be installed on all machines where the migration-center server components is installed.

To install this additional component, it is necessary to run an installation file, which is located within the

SharePoint component folder of your migration-center Job Server installation location, which is by default C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-sharepoint-online-importer\CSOM_Service\install. This folder contains the file install.bat, which must be executed with administrative privileges.

After the service is installed you will need to start it manually for the first time, after that the service is configured to start automatically as soon as the computers operating system is loaded.

In case it is necessary to uninstall this component, the file uninstall.bat must be executed.

Configuration

The migration-center SharePoint Online Importer can import objects generated by any of the available (and compatible) scanners. Most scanners can store the data they extract from the source systems they access in either a local path, or a UNC network path.

As is the case with all importers, they need to able to access the files extracted by a scanner in order to import.

See the respective scanner’s user guide for more information on configuration parameters if necessary.

Preparation for app-only principal authentication

The importer supports only app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors: Azure AD app-only principal authentication and SharePoint app-only principal authentication.

Azure AD app-only authentication requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.

If you want to restrict the access of the migration-center application to certain site collections or sites, you can use SharePoint app-only authentication.

Azure AD app-only principal authentication

Running multiple Job Servers for importing into SharePoint must be done with great care, and each of the Job Servers must NOT import in the same library at the same time. If this occurs, because the job will change library settings concurrently, the versioning of the objects being imported in that library will not be correct.

The migration-center SharePoint Online Batch Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.

Please follow these steps in order to setup your migration-center application in your Azure AD.

The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Configure necessary permissions for the migration-center application

Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:

Microsoft APIs
- SharePoint
  - Application permissions
    Sites
    Sites.FullControl.All
    TermStore
    TermStore.Read.All
    User
    User.Read.All
- Graph
  - Application permissions
    Sites
    Sites.FullControl.All

Click on the blue "Add permissions" button at the bottom to add the permissions to your application. The "Application permissions" are those granted to the migration-center application when running as App Only.

Step 4: Uploading the self-signed certificate

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.

Final step: Setting the necessary parameters in the importer

In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:

SharePoint app-only principal authentication

SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.

The information in this chapter is based on the following guidelines from Microsoft:

https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint

Step 1: Create a self-signed certificate for your migration-center Azure AD application

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Uploading the self-signed certificate and generate secret key

After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.

Store the retrieved information (client id and client secret) since you'll need this later! Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Step 4: Granting permissions to the app-only principal

Next step is granting permissions to the newly created principal in SharePoint Online.

If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.

If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.

Once the page is loaded add your client id and look up the created principal by pressing the "Lookup" button:

Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.

To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

Use this permission XML for granting site collection scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:

Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Final step: Setting the necessary parameters in the importer

In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Preparation for applying Microsoft Information Protection sensitivity labels

Starting with v3.15 Update 1, our SharePoint Online classic importer supports applying Microsoft Information Protection (MIP) sensitivity labels on the content files before they get uploaded to SharePoint Online. To learn more about sensitivity labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide.

Register an Azure AD app

The MIP software development kit provided by Microsoft that we use to apply the sensitivity labels on the content files requires you to register an application in your Azure AD. You could use the same app as in Azure AD app-only principal authentication or setup a separate application as described here: https://docs.microsoft.com/en-us/information-protection/develop/setup-configure-mip#register-a-client-application-with-azure-active-directory

You need to create a client secret for your registered application and configure the following access permissions:

Install VC++ Runtime

The Microsoft MIP SDK also needs the VC++ runtime installed on the job server machine to work properly. You might get a LoadLibrary failed for: [C:\fme AG\migration-center Server Components 3.15\lib\mc-sharepointonline-importer\MIPService\x86\mip_dotnet.dll] error if the VC++ runtime is missing and you want to apply sensitivity labels on your documents.

For your convenience, we provide the VC++ runtime installers in the lib\mc-sharepointonline-importer\MIPService\VC_Redistributables folder of your MC job server installation. Which variant to install (i.e. x86 or x64) depends on your .NET configuration. Usually you should install the x86 VC++ runtime (even on a x64 system). If that does not work, you should install the x64 variant.

Create the configuration file for the importer

The SharePoint Online classic importer requires you to provide several configuration parameters in a XML configuration file. You can find a template configuration file (template-service.config) in the lib\mc-sharepointonline-importer folder of your migration-center job server installation.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="<YOUR AZURE APP ID>" />
		<add key="Azure.App.Name" value="<YOUR AZURE APP NAME>"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="<YOUR ENCRYPTED AZURE APP SECRET>"/>
		<add key="Azure.App.RedirectUri" value="<YOUR AZURE APP REDIRECT URL>"/>
		<add key="Azure.Tenant.Domain" value="<YOUR AZURE TENANT DOMAIN>"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="<OPTIONAL PATH TO LABELED CONTENT FOLDER>"/>

		<!-- File name prefix for the labeled files. -->
		<add key="Labeling.File.Prefix" value="lbld__"/>
	</appSettings>
</configuration>

Create a copy of the template file, fill in the required information from your Azure AD app that you had created in the previous step, and enter the full path to your config file in the mipServiceConfigFile parameter of your SharePoint Online importer. The result should look similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="91583618-eb95-4ae9-8f9e-c5259491f66d" />
		<add key="Azure.App.Name" value="miptest"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="0;#75+wbxpdQ0c=;#pp39p7v4QA57VKiP0NNcnuX/FEk2FiE8iD4PrJnysfLuDbNra2xsFHA2K9wAKG8z"/>
		<add key="Azure.App.RedirectUri" value="miptest://authorize"/>
		<add key="Azure.Tenant.Domain" value="migrationcenter.onmicrosoft.com"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="C:\Temp\sensitivity labeled docs"/>

		<!-- File name prefix for the labeled files. -->
		<add key="Labeling.File.Prefix" value="lbld__"/>
	</appSettings>
</configuration>

We provide a Java based command line tool for encryption, which you must use to encrypt the Azure.App.Secret value before you put it in the configuration file. To encrypt the secret, proceed as follows:

Open a command line window on the job server machine.
Switch to the migration-center Server Components <version>\lib\mc-core folder, e.g. migration-center Server Components 3.16\lib\mc-core
Run the following command: java -cp mc-common-<version>.jar de.fme.mc.common.encryption.CmdTool
The tool will ask for the text to encrypt. Enter or paste the secret value and press enter.
The tool will output the encrypted text. Copy and paste it into the configuration file.

Limitations and known issues

Due to restrictions in SharePoint, documents cannot be moved from one Library to another using migration-center once they have been imported. This applies to Version and Update objects.

Moving folders is only supported within the same site, i.e. the importer parameter "siteName" and the system attribute "site" must have the same values for the initial and any update import runs.

Even though some other systems such as Documentum allow editing of older versions, either by replacing metadata, or creating branches, this is not supported by SharePoint. If you have updates to intermediate versions to a version tree that is already imported, the importer will return an error upon trying to import them. The only way to import them is to reset the original version tree and re-import it in the same run with the updates.

The SharePoint Online system has some limitations regarding file names, folder names, and file size. Our SharePoint Online importer will perform the following validations before a file gets imported to SharePoint Online (in order to fail fast and avoid unnecessary uploads):

Max. length of a file name: 400 characters
Max. length of a folder name: 400 characters
Invalid leading chars for file or folder name: SPACE
Invalid trailing chars for file or folder name: SPACE, PERIOD
Invalid file or folder names: "AUX", "PRN", "NUL", "CON", "COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9"
The following characters are not allowed in file or folder names: " * : < > ? / \ |
Max. length of a file path: 400 characters
Max. size of a file: 15 GB
Max. size of an attachment: 250 MB

SharePoint Importer Properties

To create a new SharePoint Online Importer, create a new importer and select SharePoint Online from the Adapter Type drop-down. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type. Mandatory parameters are marked with an *.

The Properties of an existing importer can be accessed after creating the importer by double-clicking the importer in the list or selecting the Properties button/menu item from the toolbar/context menu. A description is always displayed at the bottom of the window for the selected parameter.

Multiple importers can be created for importing to different target locations, provided each importer has a unique name.

Common importer parameters

SharePoint Importer parameters

The configuration parameters available for the SharePoint Importer are described below:

Content Integrity Check

Starting from migration-center 3.5 the SharePoint Online importer has the option of checking the integrity of each document’s content after it has been imported. This will be done if the “checkContentIntegrity” parameter is checked and it will verify only documents that have a generated checksum in the Source Attributes.

Currently the only supported checksum algorithm is MD5 with HEX encoding.

For certain types of documents such as Office documents and also .MSG documents, “Document Information Panel” is activated and SharePoint changes the content slightly upon upload. This will cause the integrity check to fail for those documents and there is no workaround so far, other than importing without the content integrity check or finding a way to disable this feature directly in SharePoint.

Working with the migration-center SharePoint object type

Migration Sets

Documents

Documents targeted at a Microsoft SharePoint Online Document library will have to be added to a migration set. This migration set must be configured to accept objects of type <source object type>ToSPOnline(document).

Create a new migration set and set the <source object type>ToSPOnline(document) in the Type drop-down. This is set in the -Migration Set Properties- window which appears when creating a new migration set. The type of object can no longer be changed after a migration set has been created.

The migration set is now set to work with SharePoint Online documents.

Links

The same procedure as for documents also applies to links about to be imported to SharePoint Online. For links the object type to select for a migration set would be <source object type>ToSPOnline(link).

Folders

The same procedure as for documents also applies to folders about to be imported to SharePoint Online. For folders the object type to select for a migration set would be <source object type>ToSPOnline(folder).

Lists

The same procedure as for documents also applies to lists or libraries about to be imported to SharePoint Online. For lists or libraries, the object type to select for a migration set would be <source object type>ToSPOnline(list).

The migration-center SharePoint Online Importer supports only the creation of lists or library!

List Items

The same procedure as for documents also applies to list items about to be imported to SharePoint Online. For list items the object type to select for a migration set would be <source object type>ToSPOnline(listItem).

Transformation rules

Documents

<source object type>ToSPOnline(document) type objects have a number of predefined rules listed under Rules for system attributes in the -Transformation Rules- window. These rules are described in the table below.

Other rules can be defined by the user to set various SharePoint column values, such as Name, Title, checkin_comment to name a few of the more frequently used. Any SharePoint column which exists in the document library targeted by the contentType rule can be set by creating a rule and associating it with the corresponding column in the associations tab.

Links

SharePoint Online supports documents with the file format ".url". We call these documents "link documents" since they do not have real content but just point to another document in SharePoint Online or to an external web site. The importer will create the necessary content of the link documents on the fly and save them in the default temporary files folder of the OS. Thus, the importer will ignore the content of any source object that is imported as a link document because it will import the generated link content instead.

Links have slightly different rules than documents. The exact system rules applicable to links are listed below.

Folders

Folders have different rules than documents. The exact system rules applicable to folders are listed below.

List Items

List items have different rules as documents. The exact system rules applicable to list items are listed below.

Lists

Lists have different rules as documents. The exact system rules applicable to lists are listed below.

Object Type definitions

Documents, Folders and List Items

In order to associate transformation rules with SharePoint Online columns, a migration-center object type definition for the respective content type needs to be created. Object type definitions are created in the migration-center client. To create an object type definition, go to Manage/Object Types and create or import a new object type definition. In case your content type contains two or more columns with the same display name, you need to specify the columns internal names as attribute names.

Working with object type definitions and defining attributes is a core product functionality and is described in detail in the Client User Guide.

A Microsoft SharePoint Online content type corresponds to a migration-center object type definition. For the Microsoft SharePoint Online adapter, an object type definition can be specified in four ways, depending on whether a particular SharePoint Online content type is to be used or not or multiple but different content types using the same name across site collections:

My Content Type describes explicitly a SharePoint Online content Type named My Content Type defined either in the SharePoint document library or at Site Collection level
@My Document Library describes the SharePoint Online document library named My Document Library using only columns, which are defined explicitly in the SharePoint Online document library My Document Library
My Content Type;#Any Custom Value describes a SharePoint Online content type named My Content Type. Everything after the delimiter ;# will be cut off on importing
@My Document Library;#Any Custom Value describes the SharePoint Online document library named My Document library using only columns, which are defined explicitly in the SharePoint Online document library My Document Library. Everything after the delimiter ;# will be cut off on importing.

The migration-center SharePoint Online importer is able to modify the following SharePoint Online specific internal field values for documents, list items and folders: Created By, Modified By, Created and Modified. To set these internal field values it is necessary to create attributes named Author, Editor, Created and Modified and associate them in the transformation rules appropriately.

Lists

Importing lists or libraries into SharePoint Online is a little bit different than for documents, folders or list items, since the migration-center SharePoint Online Importer sets Microsoft defined attributes, which are not visible to the user. In order to set those attributes, it is necessary to create object type definitions for each type of list template (see List of possible list templates). The SharePoint Online Adapter is able to set any possible attribute of a list. See List of possible list attributes for more information about possible attributes.

Object Values

File Reader

The SharePoint Online Importer is able to read values from files. This might be necessary if the length of a string might exceed the maximum length of an oracle database column, which is 4,096 bytes.

To tell the SharePoint Online Importer reading Strings from a text file, the filepath of the file, containing the value must be placed within the markup <@MCFILE>filepath</@MCFILE>.

The File Reader does not read values from files for the following attributes: Check-In Comment, Author, Editor, Creation Date, Modified Date.

Example:

Assuming you have a file at path \\scanlocation\temp\1\4a9daccf-5ace-4237-856c-76f3bd3e3165.txt you must type the following String in a rule:

<@MCFILE>\\scanlocation\temp\1\4a9daccf-5ace-4237-856c-76f3bd3e3165.txt</@MCFILE>

On Import the SharePoint Online Importer extracts the contents of this file and adds them to the associated target attribute.

Filename

The SharePoint Online Importer is able to modify the filename of a file, which is imported to SharePoint Online. To set the filename, you must create a target attribute named Name in the object type definition. You can associate any filename (without an extension) for a document. The importer automatically picks the extension of the original file or the file specified in rule mc_content_location. If the extension needs to be changed as well, use the system rule fileExtension in order to specify a new file extension. You can also change only the extension of a filename by setting fileExtension and not setting a new filename in the Name attribute.

Example:

Original content location: \\Fileshare\Migration\Scan\566789\content.dat

Check-In Comment

The SharePoint Online Importer is able to set the check-in comment for documents. To set the check-in comment, you must create a target attribute named checkin_comment in the object type definition. You can associate any string to set the check-in comment

Author

The SharePoint Online Importer is able to set the author for list items, folders and documents. To set the author, you must create a target attribute named author in the object type definition. The value of this attribute must be the loginname of the user, who shall be set as author.

Editor

The SharePoint Online Importer is able to set the editor for list items, folders and documents. To set the editor, you must create a target attribute named editor in the object type definition. The value of this attribute must be the loginname of the user, who shall be set as editor.

Creation Date

The SharePoint Online Importer is able to set the creation date for list items, folders and documents. To set the creation date, you must create a target attribute named created in the object type definition. The value of this attribute must be any valid date.

Modified Date

The SharePoint Online Importer can set the modified date for list items, folders and documents. To set the modified date, you must create a target attribute named modified in the object type definition. The value of this attribute must be any valid date.

Lookup values

The SharePoint Online Importer can set lookup values. If you want to set a lookup value, you can either set the ID of the related item or the title. If you set the title of the document and there is more than one item with the same title in the lookup list, the SharePoint Importer marks your import object as an error, because the lookup item could not be identified unequivocally.

The importer will treat any numeric value provided as the ID of the lookup value. In case you want to look up the value by title and the title is numeric, please surround the numeric value with " characters. The importer will automatically remove any " characters in the provided value before trying to match it with the title of the lookup field.

URL values

The SharePoint Online Importer can set URL values. You can define a friendly name as well as the actual URL of the link by concatenating the friendly name with ;# and the actual URL.

Example: migration-center;#http://www.migration-center.com

Taxonomy values

The SharePoint Importer can set taxonomy values and Migration Center provides two ways to do it. The first possibility is to use the name of the Taxonomy Service Application, a term group name, a term set name and the actual term. Those four values must be concatenated like the following representation:

TaxonomyServiceApplicationName>TermGroupName>TermSetName>TermName

Example: Taxonomy_+poj2l53kslma4==>Group1>TermSet 1>Value

The second possibility is to use the unique identifier of the term within curly brackets {}.

Example: {2e9b53f9-04fe-4abd-bc33-e1410b1a062a}

Multiple values can be set also but the attribute has to be set as a Multi-value Repeating attribute.

In case you receive an error on setting taxonomy values make sure the TaxonomyServiceApplicationName is up to date and valid since Microsoft changes the identifier at certain times.

Performance

The performance of the import process is heavily impacted by some specific features, namely autoAdjustVersioning, autoAdjustAttachments, autoAddContentTypes, autoCreateFolders, setting taxonomy values, and setting system attributes like Author, Editor, Created, and Modified.

In case you are importing an update for a major version the increase can get up to three times compared to a normal document import. Combining all the above-mentioned features over the same document can increase the time for up to four times. Take this into consideration when planning an import as the time might vary based on the above described features.

You will achieve the highest import performance if you perform the appropriate configuration of your SharePoint Online system before you start the import and disabled the above-mentioned features in the SharePoint Online importer.

History, Reports, Logs

A complete history is available for any SharePoint Online Importer job from the respective items’ –History- window. It is accessible through the [History] button/menu entry on the toolbar/context menu. The -History- window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:

Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Log files generated by the SharePoint Online Importer can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs

The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.