Client User Guide
Introduction
The focus with migration-center 3.x is on providing a lightweight, yet highly functional, usable and extensible migration solution which can grow to match the requirements of present and future migration projects through customization and connectivity to other document and content management systems.
The current release is primarily a release that consolidates the features and adapters developed in the versions 3.2.x, which itself is a significant update to the initial version 3.0 with major new features, usability, performance, and reliability improvements. Besides the incremental improvements and fixes, migration-center also brings along an entire suite of new adapters available directly from fme; currently the full list of adapters for migration-center available from fme is:
Documentum Scanner
Filesystem Scanner
Database Scanner
Outlook Scanner
SharePoint Scanner
Documentum NCC Scanner
Alfresco Scanner
Domino Scanner
Exchange Scanner
OpenText Scanner
Documentum Importer
Alfresco Importer
Filesystem Importer
D2 Importer
box Importer
Alfresco Importer
SharePoint Online Importer
SharePoint On-Premises Importer
Documentum NCC Importer
InfoArchive Importer
OpenText Importer
Veeva Importer
Trackwise Digital Importer
Cara 5.3 Importer
migration-center client 3.13 cannot be used to connect to a migration-center database 3.4 as well as migration-center client 3.4 cannot be used to connect to migration-center database 3.13.
Other adapters can of course be developed according to customer or project requirements.
Product features
As stated in the introduction, migration-center 3.x is a complete rebuild. The general concept and useful features from the previous version have been retained functionality-wise and improved in the current version. migration-center’s strengths remain:
The transformation of metadata through intelligent, rule-based mapping of objects
Incremental delta migrations for newly created and changed documents and folders
Identification of duplicates
one-time or regular batch migration of large volumes of documents and folders
Complete audit ability, error reporting and error handling
1:1 migration of folder structures and mapping of objects within, including metadata
1:1 migration of relations (i.e. Virtual Documents, etc.)
The possibility to export and import type templates, transformation rules, object grids (export only) to CSV files.
Features that were never/rarely used have been removed, resulting in a cleaner, more user-friendly and productivity-oriented GUI.
Compatibility
Migration-center 3.x is not compatible with migration-center 2.x. Migrations initially started with migration-center 2.x should be completed before upgrading to the new version; a version upgrade halfway through a project is not possible.
Architecture Diagram
From a technical point of view, migration-center is a distributed, client-server type application using a database management system as a backend. All these components interact to allow for a highly configurable, automated mass import of very large amounts of data from a source document/content management system into another content management system.
The solution centers on an Oracle database used for storing object related information like metadata, migration rules, and all configuration and status information necessary for managing and tracking the steps involved in the migration process.
All of the aforementioned information is defined and controlled via the migration-center Client, a GUI application which acts as the command and control center for migration-center.
Finally, there are the adapters which connect migration-center to various source and target systems via migration-center’s own API on one end and the respective system’s API(s) on the other end. These adapters which interact directly with the source and target systems are controlled by a service, which is also managed by the migration-center Client application (configuring and launching the appropriate jobs). There can be any number of adapters installed and managed with migration-center.
Multiple instances of the above components can be deployed across the network, even across geographical boundaries, and can work together on the same or multiple projects.
The actual content moved during a migration (files, documents, emails) is stored on the file system (local or remote) and does not require a specific migration-center component for managing.
Depending on the combination of adapters deployed in a particular migration project, the source and target systems supported can be same/different versions of the same solution or entirely heterogeneous systems sharing nothing but basic concepts of content management, like a content object with associated metadata.
Not all adapters illustrated in the above diagram are part of the standard product package.
Client
Migration-center Client is a GUI application and represents the interface between the user and the migration-center core features and functionalities. All migration-center functionalities like creating, monitoring and managing jobs, tasks, and migration sets are accessible through the client.
The client does not perform any tasks by itself, it’s merely a management interface; therefore the Client application can be disconnected or shut down safely at any time, without affecting any tasks that might be running in migration-center at that time.
The client is a standard Windows application that can be installed anywhere, provided it has a network connection to migration-center’s central database, and the prerequisites installed (as described in the Installation Guide). Any number of Clients can be deployed across a network, and connect to the same, or different migration-center databases.
After starting the migration-center Client application, the database login window opens. The Client application is not operational by itself, since all information required to operate migration-center is stored in a migration-center database, which is why a connection to such a database is required at all times to use the Client. Type the username and the password, select the migration database from the list of available database connections and click [OK]. migration-center Client will remember the last database it connected to and will automatically reconnect to this database the next time it is started.
During installation a default user is created. If no other users have been created, use the default user account shown below for logging on to the migration-center database.
Username: fmemc
Password: migration123
Job Server
The Job Server is a migration-center component installed with the migration-center Server Components; it runs as a Windows service and does not interact with the user. Its purpose is to execute various tasks in the background, like scanning or importing documents. The tasks running on a given Job Server can be monitored and managed by the user through the Client. A migration-center Job Server can be installed anywhere as long as there is a network connection to the migration-center database; several Job Servers can be deployed across the network and managed using migration-center Clients. Since the Job Server is the process actually accessing the source and target systems, the user account this service runs with should have the required permissions for proper access to the respective source and target systems.
Database
The migration-center database contains most of the data processing logic and stores all the migration sets, including their objects, metadata and transformation rules.
As with the other components, at least one database needs to be installed for migration-center to work, but several databases can be deployed in the same environment, depending on the requirements.
Both the Client and the Job Server need to connect to a migration-center database to function.
Client Known Issues
Changes done in the Object Types window will not be saved in the following scenario:
Opening the MC Client and making changes to the object types before opening any of the following screens: scanner properties, migset properties, transformation rules, importer properties. Until the screens are opened the changes are saved only in the Client and not in the MC Database.
Clicking on Cancel in any of the screens will revert the object type changes. Clicking on Ok on any of the mentioned screens will save the object type changes.
This only occurs when the client was freshly opened. After going to any of the mentioned screens the object type screen will behave as expected.
Client Functionalities
This chapter details the features and functionalities available to the user through migration-center Client.
Main application window
migration-center’s main application window adheres to typical windows applications’ user interface, offering a main menu bar and toolbar for accessing general, frequently used features.
Major features and functionalities each open in their own child windows, providing toolbars and context menus that apply to the items displayed in that window.
Any combination of child windows can be opened at the same time; child windows can be tiled or cascaded.
Some windows (e.g. windows displaying properties for items, or message boxes informing the user or requesting input from the user) always display on top of other windows and stay there until they are closed by the user.
All the important main menu commands are represented in the toolbar of the main window, are self-explanatory and will be described in the next chapter - Main window toolbar. The menu commands which do not have a correspondent in the toolbar menu will be described here.
<migration-center> <Connect to…> Calls up a dialogue window where the user can connect to a (another) migration- center specific database.
<migration-center> <Reconnect> Reconnects client to currently used migration-center DB
<migration-center> <Renew License> Allows entering a new license key
<migration-center> <About> Displays information about licensing and product versions
<migration-center> <Exit> Exits migration-center
<Manage> <Scanners…> opens the -Scanners- window for creating and managing scanner jobs
<Manage> <Migration Sets…> open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
<Manage> <Importers…> opens the -Importers- window for creating and managing importer jobs
<Manage> <Jobs…> open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
<Manage> <Jobservers…> open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
<Manage> <Schedulers…> opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
<Manage> <Object Types…> opens the -Object Types- window, where the object type definitions can be imported and edited (these are required for associating transformation rules with for the attribute model transformation are defined)
<Manage> <Global Mapping Lists> opens the -Global Mapping Lists- window, where global mapping lists can be created or imported. A global mapping is automatically available for use in any migration set.
<Manage> <Object History…> opens the -History- window where it is possible to look up any object that underwent processing in migration-center. Key steps of the objects lifecycle during the migration can be viewed, such as the objects status, metadata and the exact timestamp at that moment.
Main window toolbar
The main toolbar provides quick access to frequently used functionalities. These correspond with the same functionalities available from the main menu.
[Scanners] opens the -Scanners- window for creating and managing scanner jobs
[Migration Sets] open the –Migration Sets- windows for creating and managing migration sets, setting up and testing transformation rules and validating the transformed objects
[Importers] opens the -Importers- window for creating and managing importer jobs
[Jobs] open the –Jobs- window which displays currently running jobs and allows the user to pause or stop running jobs if needed
[Jobservers] open the –Jobservers- window which displays currently running jobs and allows the user to pause or stop any running jobs if needed
[Schedulers] opens the -Schedulers- window, where fully automated, time scheduled migration runs can be configured and managed
[Cascade] Arranges all open child windows in a cascade
[Tile Horizontally] Tiles all open child windows horizontally
[Tile Vertically] Tiles all open child windows vertically
Job Servers
To input/output any data from/to migration-center there needs to be at least one location with a running Job Server defined.
The -Jobservers- window has its own toolbar and right-click context menu containing items that allow the user to create, edit, delete, and monitor the availability of the Job Servers running on the specified Job Server locations.
The location for a Job Server can be specified as the machine’s hostname or fully qualified domain name, if required, or simply as an IP address.
Note: Do not use loopback addresses or the “localhost” name.
Toolbar and Context Menu
The toolbar and the context menu of the -Jobservers- window contain the same commands.
[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[Refresh] is useful for rechecking the status of the locations listed
[New] opens a blank -Jobserver- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a location from list
Properties window
To create a new Job Server location
select <New> from the context menu or from the toolbar
Enter the IP address or hostname of the machine the Job Server is running on and the port number it is listening on.
Enter the appropriate port on which the Job Server on that machine was configured
The new location is displayed in the -Jobservers- window. Check the Status column to make sure the required Job Server is listed as Available and thus capable of accepting incoming connections. The Job Server on that particular location is now ready to be used by any input/output adapter.
The default port number is 9700, unless changed during setup.
Creating a scanner and trying to save it without having any Job Servers defined will prompt the user to define one Job Server location and save the scanner with the new created Job Server location.
Scanners
Scanners are responsible for filtering and extracting metadata and content from the source system and inputting this information into the migration-center database. Different types of scanners can be available in a migration-center installation; different types of scanners also offer different configuration parameters, depending on the source system they connect with migration-center. All scanners provide the same basic functionalities, such as extracting documents and folders form the source system and extracting the metadata associated with these objects and storing it in the migration-center database, as well as adding system specific information allowing migration-center to understand and process these objects correctly.
Toolbar and Context Menu
The toolbar and the context menu of the -Scanners- window contain the same commands.
[Properties] is used to change the configuration of an item in the list. -Properties- window can also be opened by double-clicking an item in the list
[History] opens the -History- window where more information about all scans performed by the selected scanner can be viewed. The log files from these scans can also be accessed here.
[View Scanned Objects] will display a grid with the scanned objects in a new window, where these can be viewed, filtered, and exported to a CSV (comma separated values) file. This functionality does not allow any editing to be performed on the objects.
[Run] initiates the run of a scanner with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the scanners listed
[New] opens a blank -Scanner- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list
Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.
The -History- windows displays various information about the selected scanner, such as the number of processed objects, the start and ending time and the status of each run.
Double clicking a history entry or clicking the [Open] button on the toolbar with a selected history entry opens the log file created by that run. The log file contains more information about the run of the selected job:
Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.
The log files generated by a scanner are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components 3.13\logs
All scanners, regardless of type share some common parameters
Name: must be a unique name for the current scanner
Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the scanner depend on the adapter selected here.
Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list
Description: provides space to enter an optional description of what this scanner does or add some comments related to the scanner
In the History of a scan, or after the objects have been assigned to a Migration set the metadata of these objects can be viewed in the -Scanned Objects- window which can be opened with the [View Scanned Objects] button.
Choose which columns to display using the [Show/Hide Columns] button in the toolbar.
The [Export to CSV] button allows the user to export the entire grid in its current configuration (i.e. including any applied filters and visible columns) into a comma separated file. The CSV field separator, and the repeating attribute separator can be specified by the user before exporting the file.
Filtering attributes and (de)selecting columns is neither permanent, nor does it affect the actual data in any way. It only affects the current display and serves to analyze the data on display.
Migration Sets
Migration sets are sets of individual objects but with a common set of transformation and validation rules applied to them. Operations like transformation, validation and import are always performed on such a set of objects and cannot be performed on individual objects that are not assigned to any migration set. Assigning the objects resulting from a scan to a migration set is therefore the first step towards making these objects operable for migration-center’s features.
It is possible to add multiple scans to one migration set, as it is possible to split a scan containing a large number of objects onto several migration sets.
It is also possible to filter the selection of objects before adding it to a migration set based on user specified selection criteria and conditions; objects which do not meet these criteria will not be added to the current migration set and will continue to be available and can be filtered further, added to another migration set, or left unassigned if they are not supposed to be migrated.
Buttons and context menu items:
[Edit Transformation Rules] opens the window where transformation rules are managed and rules are applied
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Transform and Validate] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
[Reset All] resets all objects of in the selected migration set to their initial attribute values, essentially removing any processing performed during the transformation process and resetting status changes resulting from validation and import. The [Reset All] button is available in all views (-Source Objects-, -Processed Objects-, and -Error Objects-).
Imported objects will NOT be reset by default and require an extra checkbox to be checked prior to executing the reset command. This is for security reasons, because resetting imported objects will erase all information linking the source objects to the objects created by migration-center during import, therefore rendering migration-center unaware of having imported the affected objects at all.
Resetting imported objects would typically be used during testing phases of a project and not during production.
It is strongly advised NOT to reset imported objects in cases where dependencies to these objects may exist in other, even future migration sets, such as newer versions of these objects or relations between these and other object which haven’t been migrated yet; this also applies if it is intended to update these objects during future migrations.
[Stop] appears as a button only when the selected migration set is busy being transformed or validated, and only if these processes take longer than 3 seconds for the selected migration set.
[Refresh] refreshes data in all info-columns
[View All Objects] (press drop-down list/button combo) opens 3 windows: Source Documents, Processed Documents, and Error Documents.
[Source Objects] (select from drop-down list/button combo) displays all scanned objects attributed to the respective migration set. Original attributes and attribute values are displayed.
[Processed Objects] (select from drop-down list/button combo) displays the objects after they have been processed by applying the transformation rules.
[Error Objects] (select from drop-down list/button combo) displays objects that failed transformation/validation criteria, or failed to import, indicating the cause
[New] opens a blank -Migration Set- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured migration set from list
Items in columns can be sorted by clicking on the column headers. Clicking the same column header repeatedly switches between ascending/descending sorting order.
In the lower part of the window, migration sets can be filtered by name. This can prove useful if there are a large number of migration sets, such as the migration sets generated automatically by repeated scheduled migration runs.
Migration Set Properties window
The migration set’s properties window opens when creating a new migration set and can be accessed for an existing migration set by double-clicking it or selecting [Properties] button or menu item from the -Migration Sets- windows toolbar or context menu.
The -Migration Set Properties- window also determines the selection of objects assigned to the migration set. The |Filescan Selection | allows scans to be selected and added to the migration set, | Exclude objects by values | offers a simple method of removing excluding objects from the migration set based on selected metadata values, and the | Advanced Filters | page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators.
The resulting set of objects can be previewed by clicking the [Preview objects] button on the right side to open the -Preview- window. The -Preview- window offers an overview of the objects resulting from the selection, and some basic functionality for analyzing these such as viewing attributes for individual objects, selecting the attribute to display or exporting this list to a CSV file.
Common controls
The buttons on the right side of the –Migration Set Properties- window are available on all tabbed pages of this window.
The [OK], [Apply] and [Cancel] buttons are self-explanatory and work as expected.
Trying to save a migration set using the [OK] or [Apply] buttons without having assigned any objects to it will display a warning message to the user. Although usually it doesn’t make sense to save a migration-set without having objects assigned to it, this can be used to define migration sets which will be used as templates, and create new migration sets by copying these. In such a case ignore the warning and save the “empty” migration set.
Otherwise define a selection of objects as described in the following chapters before saving the migration set.
The [Preview objects] button and preview functionality is available on all pages of the –Migration Set Properties- window.
If the selection is ok, the selected scans can be assigned to the migration set using the [Select objects] button in the upper right corner.
Note: Assigning objects to a migration set triggers various status changes and updates to the object’s internal metadata, as well as evaluating any additional filter criteria the user may have specified. This can take some time, especially with large numbers of objects. Therefore, the user will be informed that the selection process is about to run in the background, and the Properties window will close. The selection progress can be monitored in the main -Migration Sets- window. While a selection process is running, the user can continue working with other migration sets or other parts of the applications. Selection is a one-time-only operation and does not need to be performed anytime afterwards.
For a migration set that already has a selection of objects assigned instead of the [Select objects] button there will be a [Deselect objects] button. This will free any objects assigned to the migration set but also remove any metadata generated through transformation rules and status information for objects. Recovery of the information lost if deselecting objects from a migration set is not possible. Use with care!
Deselecting objects form a migration sets also involves various updates to the object’s status and internal metadata, which can be a relatively time-consuming operation for large numbers of objects. The process will therefore be run in the background, similar to the procedure described above for selecting objects.
The |Properties| page
A migration set’s basic properties, including user specified properties like the migration set’s name, the type of object it contains, and an optional description can be set here. Other information managed by the system, such as creation or modification dates or the last operation performed on the migration set are also displayed.
The available object types depend on the adapters available with the respective migration center installation and may vary. Always make sure to select the correct object type since the selection here determines what is displayed on the following pages of the properties window.
The |Filescan selection| page
This page allows the user to select and assign any available scan to the current migration set.
The upper list shows all scans with available objects and some relevant information (i.e. scans that haven’t been fully assigned to other migration sets). Move selected scans to the bottom list to assign them to the migration set by clicking on the up/down arrow buttons or double-clicking a scan run. Multiple scans can be selected and assigned to the current migration set.
Only scans matching the object type set on the previous page (|Properties|) will be listed here. Check the object type selection if the desired scans are not displayed here.
The |Exclude objects by values| page
This page offers a simple way of excluding some objects from the selection based on their values for one or more attributes.
In previous versions of migration-center this page was called |Filters|. The functionality provided is the same in all product versions.
The list on the left displays all available attributes. Selecting an attribute displays a list of its distinct values in the Include list to the right. Moving any of these values to the Exclude list on the far right will exclude all objects with that value from the migration set. Move items between the Include/Exclude lists by double clicking or using the arrow buttons between the lists.
Attributes that have been used to exclude some of their values will turn bold to indicate their status. Remove any excluded values for such an attribute to revert it to its default (no exclusion) state.
The |Advanced Filters| page
This page allows creating complex, linked filter expressions to filter the selection of objects using any attribute, value and various conditional operators. Use advanced filter expressions to specify exactly which objects should be assigned to the current migration set.
For objects matching a filter expression means these objects will be added to the migration set, not excluded!
The feature works by adding filter expressions and grouping them with AND/OR operators. The list of expressions will be evaluated from top to bottom to determine the remaining subset of objects which can be assigned to the migration set.
An expression always consists of a source attribute, a logical operator or condition, and a user specified value. Individual expressions can be created, added to the list below (click the [Add] button to the right), and then moved up or down in the list or deleted using the respective [Up], [Down] and [Delete] buttons. Any number of expressions can be added.
Conditional operators for working with string, numeric, and date/time type values are provided.
Migration Set Views
One, two, or all three of these three views can be opened at any time for the current migration set.
The filter bar at the bottom of the window is common to all views and offers following functionalities:
Other features common to all views are the following toolbar items:
Choose which columns to display using the [Show/Hide Columns] button in the toolbar.
The [Export to CSV] button allows the user to export the entire grid in its current configuration (i.e. including any applied filters and visible columns) into a comma separated file. The CSV field separator, and the repeating attribute separator can be specified by the user before exporting the file.
Click [Refresh] to reload the content of the current view
Source Objects View
The Source Objects view always displays the unmodified state of the objects in the migration set, i.e. the state of the objects as they were scanned from their respective source system. This view does not provide any editing possibilities for the objects displayed, it serves as a permanent reference for checking the original metadata of these objects.
The Source Objects view also allows identifying and removing duplicate objects. Pressing the [View Duplicate Objects] button will display only the duplicates; duplicates are identified by means of a content checksum and the objects’ creation date, e.g. from all the objects with an identical checksum the one with the oldest creation date will be considered the original, and all others will be considered duplicates. This can be used just to display the duplicates or select and remove them from the migration set using the <Remove documents from migration set> entry on the context menu.
The checksum is computed during scan and must be activated in the scanner. See the respective adapter documentation for more information about using the checksum option in the scanner.
Toolbar items
[View Attributes] displays all available (scanned) attributes of one object only. Previous and following object can also be browsed by using the appropriate buttons.
[Edit Transformation Rules] opens the -Transformation Rules- window.
Items in columns can be arranged (increasing/decreasing) by clicking on the column labels.
[View Duplicate Objects] filters identifies and displays duplicate objects
Context menu items
<View Source Attributes> will display all attributes of that object in a small new window and allows browsing through the list objects by object (back and forward.
<View Relations> will display the relations (if any) between the selected object and other objects. It is also possible to see the full details for each relation, including the objects’ IDs it related to, the status and type of that relation, etc.
<View Value> displays the selected cell’s value in a new window. This is particularly helpful when viewing details of a long value (like folder path for example), or a multi-value attribute with a large number of individual values. It is also possible to copy the value to the clipboard, or toggle wrapping the displayed value in the -View Value- window
Copying the currently selected value to the clipboard function can also be performed directly using the <Copy Value> context menu entry or using the {Ctrl+C} keyboard shortcut
<Filter by Value> will immediately filter the view using the selected attribute value. Use the [Clear Filter] button in the filter bar at the bottom to remove a filter and return to the full list.
Objects can be selected and removed from the migration set by selecting <Remove documents from migration set> from the context menu. Objects removed from a migration set become unassigned and are available to be added to another migration set or ignored if these objects are not supposed to be migrated.
Processed Objects View
Opposed to the Source Objects view, the Processed Objects view displays the processed state of the objects in the selected migration set. An object is considered as having been processed if it underwent transformation (and any other operation following transformation, such as validation or import). This view will display the newly generated attribute values for any objects that have been successfully transformed, and the status the processed objects are currently in (i.e. Transformed, Validated, or (Partially) Imported)
If no transformed objects exist in the currently selected migration set (which implies no validated or imported objects can exist either), this view will be empty. Also, if transformation rules are applied to objects but all objects fail transformation this view will be empty as well (in this particular case all objects would be displayed in the third view, Errors)
In addition to the -Source Documents- view, the -Processed Documents- window features a few more functions.
Transforming, validating, and the resetting objects can be performed directly from this window using the appropriate toolbar buttons.
[Apply Transformation] (press drop-down list/button combo) checks attribute model rules and applies the transformation rules to objects. After this step, misconfigured attributes and transformation rules will be pointed out.
[Apply Validation] (select from drop-down list/button combo) when validation is applied, all the attributes of all objects individually are compared against the limitations of the object type they belong to. For details on object type definitions, please see chapter Object Type definitions.
[Apply Transformation & Validation] (select from drop-down list/button combo) will perform both actions in sequence, automatically. Use this option once the transformation and validation rules are finalized and should be applied to migration-sets without requiring user intervention to launch the validation process once transformation has completed.
For quick access to the source attributes of any object the -Source Attributes- window can be also accessed from the context menu.
Individual objects can also be reset to their default, unprocessed state using the <Reset selected documents> item from the context menu. Resetting an object will essentially remove any metadata generated through transformation rules, processing and status information from such objects. This context menu entry is slightly different from the [Reset all documents] button on the toolbar, which, as the name implies, operated on the entire set of objects instead of an individual selection of objects.
Imported objects will NOT be reset by default and require an extra checkbox to be checked prior to executing the reset command. This is for security reasons, because resetting imported objects will erase all information linking the source objects to the objects created by migration-center during import, therefore rendering migration-center unaware of having imported the affected objects at all.
Resetting imported objects would typically be used during testing phases of a project and not during production.
It is strongly advised NOT to reset imported objects in cases where dependencies to these objects may exist in other, even future migration sets, such as newer versions of these objects or relations between these and other object which haven’t been migrated yet; this also applies if it is intended to update these objects during future migrations.
The Processed Objects view offers some new quick filter options below the common filter bar. A processed object includes any objects with the Transformed, Validated, Importer or Partially imported status; objects can be filtered quickly for any of these states using the appropriate checkboxes. Pressing the [Apply Filter] button is not needed just (de)select the checkboxes corresponding to the required status. These quick filter options also work together with the common filter bar, acting as an additional filter criterion.
In the Processed Objects view it is also possible to edit individual objects’ metadata. The [Edit Attributes] item from the context opens a new window with the currently selected objects attributes ordered as a list instead of columns. In the -Edit Attributes- window it is possible to edit any of the current objects’ attributes manually. The -Edit Attributes- window also allows applying the changes, skipping the current document, or moving to the next document in the list without closing the window.
Any manually edited object will be reset to the Transformed status. This makes a difference for objects which have been validated already; since the user has edited these objects manually, the information the object has been validated with has changed and the object will be reset to the Transformed status and require validation using the new attribute values.
Error Objects View
The Error Objects view displays all objects that failed at some point during the migration. All states an object goes through during migration have a corresponding error state, such as Transformation Error, Validation Error, etc. The Error Objects view collects all objects in any state considered as an error and provides a centralized location for tracking such documents, identifying and correcting such objects. In addition to listing Error Objects and their associated attributes, the error message for causing the object to fail be listed in addition for every object; if the error is related to any particular attributes, those attributes will be colored red and hovering over such an attribute will display an additional message about the cause of the error.
All this information makes it easy to identify and correct the error and transfer the object into the proper states allowing it to be migrated.
The Error Objects view provides the same functionalities as the Processed Objects window, since both views serve the same goal: to prepare objects to be migrated and successfully meet the required conditions.
Errors are not a final state in migration-center. If the error can be fixed, it is also possible to correct the objects affected by that error as well and start a new attempt for migrating affected objects. This procedure can be repeated until all objects with errors have been fixed one way or another:
There are five basic ways of handling errors; one of them should always be able to solve typical errors encountered during migration
If the error is caused by a transformation rule (objects in Transformation Error status), adjust the transformation rule to cover the faulty objects and re-apply it to those objects. Any objects which have already been transformed, validated or transformed successfully will not be affected and will maintain their current status and metadata
If the error cannot be fixed by adjusting transformation rules and the object is in the Transformation Error status, it may be necessary to edit objects manually. Use the <Edit Attributes> window to browse through the objects and edit the attributes manually, skipping any objects where manual editing does not apply.
If the error occurs during import due to external factors (objects in Import Error status) like software or hardware issues, network issues, target system configuration problems, proper access permission, etc.) try to remedy the error first and after that just add the migration set back to the importer and simply try to import it again. Migration-center will attempt to import all objects in the Import Error state automatically and will succeed if the cause for the error has been fixed.
If objects have failed Validation (objects in Validation Error status) check the Object Type definitions and Associations in the Transformation Rules window, make the necessary corrections and re-run the Validation process to validate these objects successfully.
Finally, if none of the above helps, or the objects having failed are not even needed to be migrated they can either be removed from the migration set or left in their respective error states (temporarily or permanently). Unless other objects depend on them, leaving unneeded objects in an error state or removing them from the migration set will not affect other objects.
Transformation rules
The transformation rules are a set of rules that apply to all objects of a migration set, transforming the attributes of the objects inside. Transformation rules determine how the metadata provided with the source objects is going to be altered to generate new or modified attribute values, and how attributes from the source and target object models will be connected. A transformation rule consists of one or several transformation steps, where each step references one transformation function. The final transformation step will return the result for a rule.
The -Transformation rules- window can be opened by pressing the [Edit transformation] button in -Migration Sets- window, or the [Edit rules] button in both -View Processed documents- and -View Error documents- windows. Also, the -Transformation rules- window can be opened by context menu <Edit Transformation rules> the three windows where the button is present.
The -Transformation rules- window has three tabs for managing different aspects of transformation and validation, and a set of common buttons accessible from all tabs. These buttons are:
[Generate attributes] will automatically generate the simplest transformation rules based on all source attributes found for all object in the migration set. These simple rules would enable a 1:1 migration of the objects in the current migration set. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected. This function will be further discussed in the next chapter (Attributes).
[Export] will export the current migration set’s transformation rules to a XML file for the purpose of backing up or transferring the model to a migration set located on another DB.
[Import] imports transformation rules generated by the XML files generated with the previous export function. Warning! Importing rules from a file will replace all rules configured previously!
[Ok] saves changes and closes window.
[Cancel] cancels all changes that have not been applied yet and closes window.
[Apply] saves changes without closing the window.
Rules
The |Rules| page consists of four areas, namely Rules, Rules for system attributes, Rule properties and Transformation methods. The Rules and the Rules for system attributes work the same way: selecting a rule will allow its properties and transformation methods to be edited using the controls on the right side of the window. While the former is meant to allow the user to create, edit, and delete any rule, the latter does not allow rules to be added or deleted by the user, it only allows editing the existing rules, which are predefined system rules defined by the object type of the migration set (the object type selected when creating the migration set). Attributes initially obtained by scanning (DOCUMENTUM repository or file system) are called source attributes. They are used as source for direct transformation or for generating more complex transformation rules from them. Source attributes are accessible in the Transformation methods dynamic area.
Rules list
Rules are not immediately connected to any attribute or property of the target system. In the most general sense, rules have arbitrary names and use various source attributes as input, process these using the specified transformation functions and finally output the resulting value. An actual connection between these rules and the object types and their respective attributes is created on the Associations page, which will be discussed later. This generic approach allows migration-center’s Transformation engine to adapt to a variety of content management systems which use generic concepts such as type of object and attributes/properties.
The [Generate rules] button can be useful for starting work on transformation rules. This feature will generate rules based on any source attributes it can find among the objects from the current migration set. The rules will be named using the respective attribute names and will contain only a basic transformation function. Rules generated this way will behave just like any user created rule afterwards, meaning they can be edited, renamed or deleted as needed. If rules with the same name already exist, they will not be affected.
These simple transformation rules will extract the original attribute values without performing any processing on them and would be suitable for a 1:1 migration. For most real-world migration projects at least some of these transformation rules will require changes to make them suit the requirements though. It is of course possible to ignore this feature altogether and add rules one by one, according to the requirements and possibilities of the project.
Rules for system attributes
The system attribute rules work the same way as user defined transformation rules. System rules are predefined rules and usually refer to key information that needs to be specified for the objects to be migrated and work correctly in the respective target system. System attributes depend on the type of object used in the migration set (specified during creation of the migration set and displayed in the -Transformation Rules- windows’ title bar). System rules cannot be renamed, deleted and new system rules cannot be created by the user; only the transformation rules for the predefined system rules can be edited. Here the full functionality of transformation functions is available, as for the user defined rules.
The migration-centers adapter targeted at Documentum provide following system attribute rules:
“dctm_obj_link” specifies the path(s) where a document should be linked to in the target repository. “dctm_obj_link” is also an example for a system attribute which is defined by and used internally by migration-center and does not require association with an actual target attribute. This means there is no need for an actual attribute named “dctm_obj_link” to exist in either the source or target system.
The system attribute “dctm_obj_rendition” specifies file system location to files which represent renditions. Zero or more renditions can be specified for any object and will be added by migration-center during import. This attribute is also an internal attribute used by migration-center similar to “dctm_obj_link”
“r_folder_path” and “r_object_type” can get their default information automatically (see [Generate attributes]) only inside a Documentum-to-Documentum migration set. In the migration set of a file system scan, this information must be put in by user.
Rule properties
The Rule properties allows specifying the name for a newly created rule or renaming an existing rule. Whether a rule can work with multiple values can also be set using the Multi-Value checkbox. These controls are disabled for editing if a system attribute is selected since system attributes cannot have their properties changed by the user.
Transformation methods
The Transformation methods manages the transformation steps and functions used to perform the actual processing for the selected rule. Each step corresponds to a function. Any number of steps can be added to obtain the desired result for the current rule. At least on step must be added for a transformation rule to work.
The Function drop-down list all available transformation functions. Select a function and add it as a step to by clicking the [Insert] button. Functions will always be inserted last in the list. After inserting a function, it is possible to edit, reorder, or delete steps using the respective buttons.
Inserting a new function as a transformation step will immediately open the function’s properties for editing. Configure the function’s parameters accordingly and click [OK] to save the function and add it to the list.
Function properties can be edited at any time by selecting the corresponding transformation step and clicking the [Edit] button above. This will open the same properties window mentioned above and allows the user to make the required changes.
Selecting a function from the “Function” drop-down menu and hovering mouse pointer over the help-icon (“?”) next to [Insert], a windows tooltip will give the user more information about the selected function.
Error and Warning indicators for rules:
Migration-center always checks rules when saving and looks for invalid references, functions, or unused transformation steps. Rules will be colored red for “Error” or yellow for “Warning” as a result of this verification. Migration-center cannot correct such inconsistencies because the intended purpose of the affected rules is not known to migration-center. It is up to the user to check and correct any issues with rules marked red or yellow.
In the example below, the rule called “extension” has been marked yellow by migration-center’s rule auto verification feature.
The user can select that rule to check the cause of the warning. Once a rule marked as an Error or Warning has been selected the transformation rules will also be displayed in the rule’s properties. The actual transformation step causing the error/warning will also be marked red/yellow. In the example below, migration-center warns because there is an unused function in the list. This information is shown in a tooltip when hovering the cursor over the respective transformation step.
Warnings can usually be ignored and will not affect the result of the transformation rule. However, warnings should only be ignored if it really makes sense; e.g. in the example above the user may want to leave the unused step in there, and switch between the two steps for testing purposes. For production use though it is not advised to leave any unused transformation steps because these will be computed, needlessly adding to the time required for transforming the objects.
Rules marked red, indicating errors, are rarely encountered during normal usage. The highest chance for getting Errors is when importing or copying transformation rules into the current migration set; since the newly imported transformation rules may have been created based on another migration set, they may reference source attributes which do not exist among the current objects and this may raise an error because of the invalid reference. Errors can not only result from references to invalid source attributes, but also from invalid references to previously deleted transformation steps, or wrong parameter combinations for some functions.
Marking rules and transformation steps as errors works exactly the same as described above for warnings.
Errors should never be ignored because they will definitely affect, or even prevent transformation rules from working correctly.
Working with transformation functions
Transformation functions provide the functionality behind transformation rules.
Custom functions can be implemented and used along the standard functions provided by migration-center.
Functions typically have multiple input parameters, with one of them usually being the value that needs processing, and output the processed value which may be entirely different from the source value, depending on what the function is designed to do and what parameters it has specified.
Input parameters can range from referencing a source attribute to referencing the output of a previous transformation step, a value typed by the user to various function specific parameters which determine how that function should process its input our output. The number of parameters varies by function. Some functions (like Sysdate) do not have any parameters at all, while other functions have three or four parameters.
A function’s parameter always has two properties: a type and a value. The parameter’s value is determined by the parameter’s type. Parameters can be of the following types:
Source Attribute: the value will be extracted for each object's attribute individually during the transformation process. For objects which do not have the specified source attribute no transformation will take place.
User Value: acts as a constant value and will be set for all objects to which the current rule applies.
Previous step: references the value returned by a previous transformation function from the current transformation rule and uses it as input. If an input value is multi-value, the index of the value to be processed can be selected or typed into the drop-down list on the far right. By default, all values are processed for a multi-value input
The Index selection (the small dropdown list to the far right) is for working with multi-value (repeating) attributes. By default, any function will operate on all values of a multi-value attribute, but it is possible to specify a particular value by setting its index (i.e. the n-th value of a multi-value attribute) in the Index dropdown list.
In case the parameter in question is referring to a single value attribute selecting an Index will have no effect.
The Index selection list offers the following options:
All is the default option and can be used only for the first parameter and not the subsequent parameters.
A specific index ranging from 1 to n (where n is the maximum number of values for the currently referenced multi-value attribute) can also be specified. In this case the current transformation function will only process the requested value and ignore all other values, effectively returning a single value output from a multi-value input.
The Index selection offers some more options when used with the If transformation function:
It is also possible to specify All as an option for the Return if true or Return if false parameters.
When All is used in first parameter, the If is called once for each value of the first parameter (as long as it has multiple values). The corresponding index value from Return if true or Return if false will be returned after every call. This means the result will have the same number of results as the number of values of first parameter. This is the same behavior as in all other transformation functions when used with index All.
The value Any is exclusive to the If transformation function and can be used to define a condition based on a value whose index is not fixed or is simply unknown to the user within a multi-value attribute and therefore cannot be specified using the All or 1 – n settings
When Any or a specific index 1-n is used for the first parameter, the function is called only once and will return all values specified in Return if true or Return if false. If All is used in a return parameter all values will be returned, otherwise only the specific index will be returned.
Transformation function reference
All standard transformation functions provided with migration-center are described below. Full context-sensitive information is also available in the migration-center Client when inserting or editing any transformation function.
Function | Description |
CalculateNewDate | CalculateNewDate() computes a new date by adding Years, Months and Days to a valid source date. The Database’s Datetime pattern must be followed by the input value. Negative integer parameters are accepted. Example: CalculateNewDate('01.01.2001 12:32:05', 1,2,3) returns '04.03.2002 12:32:05' |
CalculateNewNumber | It adds or substract a number to another number. Decimal and negative numbers are allowed. If a provide value cannot be converted to a number then an error is reported during the transformation. Ex: CalculateNewNumber(10, 5.2) returns 15.2 CalculateNewNumber(1000, -100) returns 900 |
Concatenate | Concatenate() concatenates up to three strings values into one and returns it as a single value Concatenate('AAA',' and ','BBB') returns "AAA and BBB". IMPORTANT: The function returns only the first 4000 bytes of the resulted string since this is the maximum length allowed for an attribute value. |
ConcatenateEach | Concatenates every value of the first input parameter with every value of a second parameter, resulting in a list with Count(Source Values 1) x Count(Source Values 2) elements. Example: If Source Values 1 has the values "A", "B", "C" and Source Values 2 has the values "X", "Y" the result will be "AX", "AY", "BX", "BY", "CX", "CY". If the result of any concatenation exceeds 4000 bytes, the function will throw an error. |
ConvertDateTimezones | ConvertDateTimezones converts a given date from one timezone to another. The accepted timezones are the ones provided by Oracle, so you can see them with the following query: SELECT DISTINCT tzname FROM v$timezone_names; |
Count Values | It counts the number of values of a repeating attribute. If null or no values provided it returns 0. |
GetDataFromSql | Map one or multiple source attributes with the data extracted from an external table. The external table must be located on the migration-center database in any schema that is accessible by FMEMC user. The user FMEMC must have "select" permission on that table. The query must return a single column and must have at least one parameter and maximum 3 that will be replaced at runtime with the values of the parameter1, parameter2 and parameter3. The parameter name in the query is any string the start with : (colon). The number of parameters in the query must match the number of parameters set in the function. If the query returns multiple values only the first one will be taken in consideration by the transformation engine. Example:
Important Note: Since the SQL query is executed for each object in the migset, you should ensure that it is executed fast, i.e. the columns used in the where condition should be indexed. |
GetDateFromString | GetDateFromString() extracts a date type expression from any string if it matches the date expression specified by the user. Use this function to extract non-standard or partial dates from source attributes like a filename. Example: GetDateFromString('filename 2007-Dec 14. 16:11','YYYY- MON DD. HH24:MI') will identify and extract the non-standard date format contained within the input string |
GetInternalAttributeValue | It is used to get the value of an internal attribute of the source objects. An internal attribute is a fix column, in the source objects view that is used internally by migration-center for different purposes. The available internal attributes that can be used with this function are: Id, Is_update, Content_location, Content_hash, Id_in_source_system, Parent_version_object_id, Level_in_version_tree, Scanned_date The name of the internal attributes are not case sensitive. If other internal attribute name is provided an error is reported during the transformation. |
GetPathLevel | GetPathLevel() understands strings representing a path and can extract specific levels from that path. The path separator character, the path level to start from and the path level up to which the function should extract the subpath can be specified as input parameters. The function will also strip leading and ending path separators from the result. Example: GetPathLevel('/this/is/the/folder/structure','/','2','4') will parse the input string looking for "/" as the path separator, and return path levels 2-4, i.e. "is/the/folder" |
GetValue | GetValue() is used to migrate attributes where the attribute's value is not supposed to be changed or to generate and set user defined values which are not present in the source data and cannot or have not been generated by other transformation functions. The GetValue() function always returns the exact same value it gets as input without altering it in any way. Examples: GetValue('user') outputs the string value "user" for all objects to which the current rule applies GetValue(filename[1]) outputs the value of the source attribute filename for all objects to which the current rule applies. For each object, the value will vary according to the actual value from that objects' source attribute named filename. |
GetValueAt | It gets the value at specific index from a multi-value attribute. Index counting starts with 1. If the provided index is out of range the function returns null. Example: GetValuesAt('a,b,c', 2) reruns 'b' and GetValuesAt('a,b,c', 4) returns null. |
GetValueIndex | Get the first index number of a given value for a multi-value attribute. If no value was found 0 is returned. The parameter "ExactMatch" specifies if exact match will be used for comparing the values. Use '1' or 'T' for exact match and '0' or 'F' for "contains" search. In any case the search is case sensitive. Example: GetValueIndex('abc,def,ghi', 'de', 'F') returns 0 GetValueIndex('abc,def,ghi', 'de', 'T') returns 2 GetValueIndex('abc,def,ghi', 'DE', 'T') returns 0 GetValueIndex('a,b,c,b','b' ', 'F') returns 2 |
If | If() evaluates a logical condition and returns different outputs depending on whether the condition is found to be true or false. A previous transformation step from the current rule, a source attribute or a user specified string are all valid arguments for both input and output values as well as for the logical condition. The If() function can correctly evaluate conditions based on various types of data such as strings, numbers, dates, null values, etc. and offers a number of predefined conditional operators. |
Length | Calculates the length of the string using Unicode characters. |
Ltrim | The Ltrim function removes characters from the left of the given Source String, with all the leftmost characters that appear in the Characters to trim removed. The function begins scanning the Source String value from its first character and removes all characters that appear in the Characters to trim until reaching a character that is not in the trim expression and then returns the result. If second parameter is empty the leading spaces will be removed, i.e. Ltrim('babcde','ab') will remove the first 3 characters so the result will be 'cde'". |
MapValue | MapValue() considers the input value a key, looks for a row with a matching key in a specified mapping list and returns the value corresponding to that key if a suitable match is found. Keys with no match can be reported as transformations errors (optional). A mapping list must be defined before using a MapValue function. Mapping lists can be defined either on the Mapping lists tab in the Transformation Rules window of a migration set (case in which they would be available only to that particular migration set), or as a global mapping list (available to all migration sets) from the Manage menu in the main application window. Use the MapValue() function to define direct mappings of source attribute values to target attribute values based on simple key-value lists. |
MultiValue_ReplaceNulls | MultiValue_ReplaceNulls() can replace null values in a multi-value attribute with a user defined value. Example: MultiValue_ReplaceNulls(multi_value_input[all],'default') will replace all null values from the multi-value source_attribute named "multi_value_input" with "default" The function can also remove null values from a multi-value attribute if no replacement string is defined. To use this function in a rule, the rule must be a multi-value rule. Example: MultiValue_ReplaceNulls(multi_value_input[all],'') will remove all null values from the multi-value source_attribute named "multi_value_input", thereby reducing the total number of values for the multi-value attribute by the number of null values found |
Multivalue_RemoveDuplicates | Remove duplicates from a multi-value attribute. If the input values are 'a', 'b', 'b', 'c', 'a' the result will be 'a', 'b', 'c'. The function is available only for multi-value transformation rules. |
RemoveDuplicates | It is provided for removing duplicates from a given string. Example: RemoveDuplicates('DE|RO|IT|DE|P','|') will remove duplicates form the first string by using the delimiter ‘|’ so it will return "DE|RO|IT|P. The function can be used in combination with RepeatingToSingleValue and SingleToRepeatingValues for removing duplicated values from a repeating source attribute. Example: #1 RepeatingToSingleValue (countries[all], ‘|’) #2 RemoveDuplicates(#1, ‘|’) #3 SingleToRepeatingValues(#2,’|’) |
RepeatingToSingleValue | RepeatingToSingleValue() concatenates all values of the source string value into one single string. Optional parameters include the delimiter to be used (can be zero, one or multiple characters), the range of values which should be concatenated and a replacement string to be used in place of any NULL values the source may contain. It is recommended to use a multi-value (repeating) attribute or previous step as source for this function Example: SingleToRepeatingValues(keywords[all],'|') will return value1|value2|value3. SingleToRepeatingValues(keywords[all],'|', 2, 3) will return value2|value3. IMPORTANT: The function returns only the first 4000 bytes of the resulted string since this is the maximum length allowed for an attribute value. |
ReplaceStringRegex | ReplaceStringRegex() Replaces the parts of the input value that match the regular expression specified by the user with a user defined value Example: ReplaceStringRegex('AAAAA-CX-9234-BBBBB','\w{2}-\d{4}','AB-0000') will parse the input string looking for a match; according to the regex this would be a sequence of 2 letters followed by a dash and four numbers. Since the input does contain a matching part, it will be replaced with "AB-0000", and the final output of the function will be "AAAAA-AB-0000-BBBBB" |
Rtrim | The Rtrim function removes characters from the right of the given Source String, with all the rightmost characters that appear in the Characters to trim removed. The function begins scanning the Source String value from its last character and removes all characters that appear in the Characters to trim until reaching a character that is not in the trim expression and then returns the result. If second parameter is empty the trailing spaces will be removed, i.e. Rtrim('cdebab','ab') will remove the last 3 characters so the result will be 'cde'. |
SingleToRepeatingValues | SingleToRepeatingValues() separates a string value based on a user specified separator character and returns all resulting values as a multi-value result Use this function to transform a string of comma separated values into a multi-value attribute with multiple individual values. To use this function in a rule, the rule must be a multi-value rule Example: SingleToRepeatingValues(comma_separated[all],',') will parse the source attribute named "comma_separated" looking for commas (","), strip the commas and create a multi-value list from the resulting values. |
SplitStringRegex | SplitStringRegex() is an advanced function for splitting up a string value by specifying the separator as a regular expression rather than a single character (the regex can represent a single character as well). Depending on the number of matches for the specified separator, multiple substrings can result from the function; which one of the resulting substrings the function should return can also be specified by the user. Example: SplitStringRegex('one-(two)-three','(-\()|(\)-)','2') will split the string into substrings based on the separators described by the regex and return the second substring which is "two" |
Substring | Substring() returns part of the input string. Which part of the string should be returned can be specified as a number of characters starting from a given index within the input string. Example: Substring('teststring','3','5') returns 5 characters starting with the 3rd character, which is "ststr" |
SubStringRegex | SubstringRegex() is an advanced transformation function for extracting a substring from the input value. A regular expression can be used to extract a complex substring from the input string, such as a particular name, a formatted number sequence, a custom date expression, an email address, etc. SubstringRegex('0123abc 4567 ',' \d{4} ') will return " 4567 " according to the regex defining the substring. |
Sysdate | Sysdate() outputs the current system date as a value. Use this function to track the date when a document underwent transformation in migration-center. This function does not have any properties. |
ToLowerCase | ToLowercase() transforms all characters from the input string value to lowercase characters |
ToUpperCase | ToUpperCase() transforms all characters from the input string value to uppercase characters |
Mapping Lists
Mapping lists can be used as an alternative means of transformation in case the desired result cannot be obtained using transformation functions. Usually mapping user names or permissions falls under this type of data, since there is no way to create a transformation rule which maps a list of several hundred users or permissions onto their corresponding new values individually; mapping lists can of course be applied to any attribute, not just the ones mentioned in the example.
Mapping lists are specific to one migration set and are only available for use within the migration set where they are defined. It is also possible to define so called global mapping lists which are available in all migration sets.
Mapping lists are simple 2-column lists and work based on a key-value concept, where the key is specified in the first column, and the associated value in the second column. If the MapValue transformation function is used in a transformation rule with a given mapping list and matches a key from the mapping list to its input value, it will output the corresponding value from the mapping list as the result.
A mapping list must be referenced by a “MapValue()” transformation function in at least one transformation rule on the |Rules| page to have any effect on the objects of the migration set. A mapping list will not do anything by itself.
“Exact Match” will require an exact match with the specified key if checked and will accept any string as a match as long as it contains the key if unchecked. It is recommended to check this option; uncheck it only if needed.
“Case sensitive” will consider a valid match only if the case of the input value matches the key and its case. Usually this would not be used, enable if needed.
Mapping list can be edited in-place if needed, but this is only for short lists or making minor corrections to an existing list. Usually a mapping list would be prepared using an appropriate application such as a spreadsheet, then copied and pasted into migration-center via the context menu available when right clicking into the list area.
The example above illustrates using a mapping list with the “MapValue()” function. Here the mapping list “opened by” is used through the “MapValue()” function in the transformation rule “extension to OPENED BY”. The window below shows the properties for this function.
Migration-center will interpret this transformation rule as follows:
In all objects of the migration set look up the source attribute “extension” and try to match its value to any of the following “mp3, doc, pdf, ppt”; if a match is found, replace the source value with the value corresponding to the matching key specified in the mapping list named “opened by” and output this value as the result.
Global Mapping Lists
Global Mapping Lists are mapping lists independent of any particular migration set. They are thus globally available to all migration sets across the entire application. Use global mapping lists for information which will be used by multiple/all migration sets. Global Mapping Lists can be managed from the main menu <Manage>, <Global Mapping Lists>. The functionalities and properties for creating, copying, deleting, and editing global mapping lists are identical to the ones provided for regular mapping lists, described above.
When using the “MapValue()” function, both the “local” (migration set specific) and the “global” mapping lists are available for use. Global mapping lists can be identified by the word “(global)” appended to their name.
Associations
In the |Associations| page existing transformation rules can be associated with actual object types and attributes from the target system. These associations will provide a means to validate the values generated by the transformation process, and will tell the importer how to assign these values during import
Types
To create a connection from a transformation rule to a target object type there needs to be one transformation rule which outputs the object type’s name. Select this rule from the “Get type name from attribute” drop-down list. For Documentum “r_object_type” is a predefined system rule which server this purpose, but there is no restriction to select a particular rule; any rule will work if it results in something which matches object type names and is not null.
From the “Available type definitions” drop-down list, add the required object type definitions below by pressing [Add]. The object type definitions displayed depend on what is available in the -Object Type Definitions- window. If a required object type definition is not available for creating an association here, Click [OK], add the object type definition, and come back to this page to create an association with it (See chapter 7.8 Object type definitions for details on adding object type definitions to migration-center).
Associations
Selecting an object type definition will populate, the “Target attribute” drop-down list with the attributes of the selected type. The “Rule” drop-down list shows the rules defined on the |Rules| tab. By matching pairs from these two controls and clicking [Add Association] rules can be associated with corresponding attributes. A rule and a target attribute can be associated only once per type.
In case the rules already have names that match attributes of the targeted object types, there is a much quicker way for creating these associations: just click [Auto-Associate]. This button will automatically match rules and attributes by name and add them to the list of associations. Any unneeded rules created this way can be deleted afterwards, and any rules that could not be associated automatically can still be added using the regular method described above.
Note: An association always applies to the currently selected object type in the list, not all of them! If there are multiple object types targeted for a migration, manual/automatic association procedure described above needs to be done for each object type.
Importers
Import is referring the process of transferring content and metadata obtained using the transformation rules to the target system. The importer is the job actually performing this task of outputting data from migration-center or importing data into the target system.
Importers migrate one or more migration sets. Only migration sets with objects that have successfully passed both transformation and validation can be imported.
Similar to a scanner, an importer is a job with a unique name and a set of configuration options, which is started by the user and the corresponding adapter code is executed on the specified Job Server.
Note: for details about the scanners provided with the standard product consult the appropriate documents listed in 1.3 References. For details about customized scanners or scanners provided by third parties please consult the documentation delivered with the respective adapters.
Toolbar and Context Menu
The toolbar and the context menu of the -Importers- window contain the same commands.
[Properties] opens the configuration details of that particular importer. The -Importer- properties window can also be opened by double-clicking an item in the list.
[History] opens the -History- window where more accurate data about all imports performed by the importer in question can be accessed. The log of the scans can be accessed here.
[Run] initiates the run of an importer with the currently saved configuration
[Refresh] is useful for rechecking the “last run status” of the importers listed
[New] opens a blank -Importer- properties window
[Copy] makes a copy (duplicate) of an item in the list
[Delete] deletes a configured scanner from list
Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.
The -History- windows displays various information about the selected importer, such as the number of processed objects, the start and ending time and the status of each run.
Double clicking a history entry or clicking the [Open] button on the toolbar with a selected history entry opens the log file created by that run. The log file contains more information about the run of the selected job:
Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.
The log files generated by an importer are always saved to disk and can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs
All importers, regardless of type share some common parameters
Name: must be a unique name for the current importer
Adapter Type: displays all adapters available with the current migration-center installation. The configuration parameters for the importer depend on the adapter selected here.
Location: Job Server locations defined in the -Jobservers- window can be selected from in this drop-down list
Description: provides space to enter an optional description of what this importer does or add some comments related to the importer
Scheduler
A Scheduler acts as a master process which manages and runs an end-to-end migration fully automated and according to a set schedule.
A scheduler is also a job, and will use preset scanners, transformation rules and importers to run extract, process, and migrate objects to the target system automatically. A full history is available for each run, listing both objects that have been imported successfully as well as any objects which may failed during the process.
There are a number of configuration options regarding the times when such an automated migration should run, ranging from minutes to months, days of the week, or hours of the day. This makes it possible to tailor a schedule according to the needs and possibilities of the project and avoid running performance intensive migration during working hours for example.
Toolbar and Context Menu
The toolbar and the context menu of the -Schedulers- window contain the familiar items: New, Properties, Delete, Run, Refresh.
Double-clicking a scheduler in the list has the same function as the <Properties> menu command.
Items in columns can be sorted (increasing/decreasing) by clicking on the column labels.
A scheduler can be run immediately if needed, regardless of its actual schedule, through the <Run> command found in the toolbar and the context menu.
Properties
An email report can be scheduled to be sent to a certain email address in case of success or failure of the scheduler processes. For this a valid SMTP-Server and email address must be provided, and at least one of the two checkboxes flagged.
Object Types definitions
Migration-center needs to know the properties of the object types targeted during migration to correctly associate transformation rules with the attributes from these object types and also for validating the values generated by transformation functions. These target object types can be managed in migration-center Client in the -Object Types- window. The window can be opened from the main menu <Manage> - <Object Types…>.
All target object types in use should be defined before attempting to create associations or validate the objects in a migration set. For creating and testing transformation rules it is not necessary to have objects types defined at that point.
Out-of-the-box the product comes with basic object type template definitions for the adapters the customer has acquired licenses for.
Additional user/custom object types can be added to migration-center by the user.
To create a new object type template, click the [New] button in the Object Types toolbar or <New> in the context menu inside the same area. Enter the template’s name in the “Name” field; the name should be exactly the same as the actual name of the object type. The [Delete] button
In the list below the individual attributes and their properties can be defined. Since the process of defining all types and attributes by hand would most likely be too time-consuming it is possible to import an object type directly from a properly formatted CSV file. Object type definitions can be imported using the [Import] button located in the Object templates dynamic area. The imported file needs to be a CSV file (comma separated value) similar to the following example, with each row containing attribute name, data type, minimum length, maximum length, repeating flag, mandatory flag, and an optional regular expression:
id,2,0,0,0,0
first_name,2,0,0,0,0
last_name,2,0,0,0,0
email,2,0,0,0,0
gender,2,0,0,0,0
ip_address,2,0,0,0,0
animal_name,2,0,0,0,0
Valid values for data type are: 0 - Boolean, 1 - Integer, 2 - String, 3 - ID, 4 - Date and Time, 5 – Double.
Some importers provide best practices for extracting the target types from the source systems using common tools. Documentum Importer User Guide and SharePoint Importer User Guide can help you with such suggestions.
The optional regular expression that you can provide for an attribute will be used to validate the attribute values during object transformation and validation. Please provide a rule that matches all valid characters for that attribute. If the attribute value contains any characters that are not matched by the regular expression, its validation will fail. For example, the following regular expression will match all characters that are valid for a SharePoint 2013 file name:
^[^\~\"\#\%\&\*\:\<\>\?\/\\\{\|\}]+$
It will match one or more characters between begin and end of the string that are not in the list of the following characters: ~"#%&*:<>?/\{|}
You can also use the regular expression to enforce a certain data schema for the attribute values. For example, to enforce a schema with three digits followed by a minus followed by five digits you can use the following regex:
^[0-9]{3}-[0-9]{5}$
Object History
Migration-center tracks key stages throughout the lifecycles of object being migrated (such as scan, import, deletion, etc.) and stores information about the exact state of that object at that point in time. The information stored about the objects includes properties such as the status, timestamp, set of metadata, or the ID the object was imported with in the target system. The Object History serves as an internal reference to all objects that have been migrated.
The -History- window can be accessed from migration-center’s main menu <Manage>, menu item <Object History…>
Log files
A complete history is available for any job (scanner, importer, scheduler) from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.
Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:
Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.
Log files generated for each job can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <version>\logs
The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job
Log files generated by the migration-center Client application in the event of errors can be found in the Client application’s installation folder, e.g. …\fme AG\migration-center Client <version> folder inside the installation path. These logs are not meant for the user but should be sent if requested by our technical product support staff in case there are issues preventing the Client from working normally.
Last updated
