Veeva Importer
Introduction
The Veeva Importer is one of the target adapters available in migration-center starting with version 3.7. It takes the objects processed in migration-center and imports them into a Veeva Vault. Currently, for every supported Veeva Vault, there is a specific importer type:
Clinical: ClinicalImporter
Quality: QualityImporter
RIM: RIMImporter
Importer is the term used for an output adapter which is most likely used at the last step of the migration process. An Importer (such as the Veeva Importer) takes care of importing the objects processed in migration-center into the target system (such as the Veeva Vault).
An importer works as a job that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created.
An importer is defined by a unique name, a set of configuration parameters and an optional description. Veeva Importers can be created, configured, started and monitored through migration-center Client, while the corresponding processes are executed in the background by migration-center Job Server.
When the term "Veeva Importer" is used in the following, it refers to all three Veeva Importers. The name of the specific Veeva Importer will be used if there is a feature applicable just for that adapter.
Prerequisites
Network Prerequisites
The importer uses FTP to upload the content of the documents and their versions to the Veeva Vault environment. This means the content will be uploaded first to the Veeva FTP server, so the necessary outbound ports (i.e. 21 and 56000 – 56100) should be opened in your firewalls as described here:
http://vaulthelp.vod309.com/wordpress/admin-user-help/admin-basics/accessing-your-vaults-ftp-server/
Planning the Migration
Before starting the ingestion into Veeva Vault you should inform Vault Product Support about this by filling this form:
https://docs.google.com/forms/d/e/1FAIpQLSf5ef7zgyG-d3-r5ZYxFRnfNRNdxRptXUReDebwDI7P0B-_SA/viewform
You should use the form 3 business days in advance of any migration and at least one week in advance of any significant migration. A significant migration includes over 10,000 office documents, 1,000 large documents (like video) or 500,000 object records.
In addition, if Migration is being run during “off hours” for the Vault’s location, or weekends, you should ensure that Vault Product Support is aware of requested activity.
Veeva Importer Properties
To create a new Veeva Clinical Importer job click on the New Importer button and select “Veeva Clinical” from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the Veeva parameters. The other types of Veeva Importer can be created in a similar way by selecting the corresponding adapter type.
The Properties window of an importer can be accessed by double-clicking an importer in the list or selecting the Properties button or entry from the toolbar or context menu.
Common Importer Parameters
Configuration parameters | Values |
Name | Enter a unique name for this importer Mandatory |
Adapter type | Select the “Veeva Clinical” adapter from the list of available adapters Mandatory |
Location | Select the Job Server location where this job should be run. Job Servers are defined in the Jobserver window. If no Job Server was selected, migration-center will prompt the user to define a Job Server Location when saving the importer. Mandatory |
Description | Enter a description for this importer (optional) |
Veeva Importer Parameters
Configuration parameters | Values |
username* | Veeva username. It must be a Vault Owner. Mandatory |
password* | The user’s password. Mandatory |
server* | Veeva Vault name. Ex: fme-clinical.veevavault.com Mandatory |
proxyServer | The name or IP of the proxy server if there is any. |
proxyPort | The port for the proxy server. |
proxyUser | The username if required by the proxy server. |
proxyPassword | The password if required by the proxy server. |
attributeMappingLocation | The path of the configuration file that will be used for setting the references to the existing master data objects when importing documents. See Setting References to Existing... for more details. |
preserveVersionBinding | Indicates if the version binding will be preserved when importing virtual documents from Documentum as binders in Veeva Vault. See Working with Binders for more details. |
importRelations | Indicates if relations between documents will imported. If checked, the relations between the documents being imported will be imported as well. If not checked the relations will not be imported. They can be imported in another run by assigning the same migsets to the importer. |
batchSize* | The number of documents or versions that will be imported in a single bulk operation. Mandatory |
skipUploadToFTP | If not checked, the importer will copy the documents content to the FTP server during import from the location where they have been exported by the scanner. If checked, the importer presumes the documents content was copied to the FTP server prior starting the import. In this case all paths of the content in the system rules mc_content_location, mc_rendition_paths, attachments or submissionFilePath must be set with the corresponding paths on the FTP server. |
loggingLevel* | Sets the verbosity of the log file. Values: 1 - logs only errors during scan 2 - is the default value reporting all warnings and errors 3 - logs all successfully performed operations in addition to any warnings or errors 4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production) Mandatory |
On the | Migsets | tab, the user can select the migration sets to be imported with this importer. Depending on the chosen Adapter Type only the migration sets compatible with this type of importer will be displayed and can be selected for import. Also, only migration sets containing at least one object in a validated state will be displayed (since objects which haven’t been validated cannot be imported).
Available migration sets can be moved between the two columns by double clicking or using the arrows buttons.
Additional Importer Configuration
For configuring some additional parameters that will apply to all importers runs, a configuration file (config.properties) provided in the folder …\lib\mc-veeva-importer. The following settings are available:
Configuration name | Description |
threshold_burst_limit_remaining | This is set with the number of API calls remaining for the current 5-minutes burst limit before the importer enters in sleep mode and waits for the next burst interval. Default value = 500 |
threshold_daily_limit_remaining | This is set with the number of API calls remaining for the current 24-hours daily window before the importer stops importing objects. An appropriate message is logged in the report log when daily limit is reached. Note: Set this value to 0 to enable the automatic pause and resume feature of the importer. When set to 0, the importer will pause for "waiting_time" hours when there are less than 1,500 API calls left before it resumes processing. Default value = 50000 |
client_id_prefix | This is the prefix of the Client ID that is passed to every Vault API call. The Client ID is composed from this prefix followed by the id of the job run. The Client ID is always logged in the report log. Default value = fme-mc-importer |
request_timeout | The time in milliseconds the importer waits for response after every API call. Default value = 60000 |
wait_time | Waiting time when reaching the daily limit. This parameter is used just when the threshold_daily_limit_remaining is set to 0. The value is represented in hours. Default value = 4 |
no_of_request_attempts | Represents the number of request attempts when the first call fails due to a server error(5xx error code). The REST API call will be executed at least once independent of the parameter value. Default: 2 |
For applying the changes in the config.properties file the Job Server needs to be restarted.
Working with Objects
Vault Objects are part of the application data model in Veeva Vault. Vault Objects are usually configured to be referenced in the document fields. For example, the Product document field corresponds to the Product object and is associated with all document types.
Veeva Importer allows importing object records from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeeva<type>(object)” must be created. Ex: Veeva Quality Importer works with migsets of type “<Source>toVeevaQuality(object)”.
Rules for System Attributes
The following rules for system attributes are available when importing objects.
System Rule | Description |
name__v | It must be set with the name of the object being imported. For some objects the “name__v” field is configured to be generated automatically by Veeva Vault when object is created. Usually, “name__v” is configured to be unique so if duplicated values are provided to the importer the affected objects fail to import. Optional |
object_name | The name of the object. Ex: study__v, study_country__v Mandatory |
target_type | The name of the migration-center internal object type that is used in the association. Ex: veeva-quality-object Mandatory |
attachments | Can be used to set file attachments to the object just in case it allows that (The "Allow attachments" option has to be set on true). You must provide the full path (UNC or local filesystem on the job server) to the attachment file. Multiple files can be specified. If the parameter skipUploadToFTP is checked, you must provide the FTP paths to the files. Optional |
Any editable object fields can be set using the normal transformation rules.
On Delta Migration, the existing attachments of an object will be replaced with those that are set on the "attachments" system attribute otherwise, the object will keep its attachments.
Working with Documents
Veeva importer allows importing documents from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeevaClinical(document)” / “<Source>toVeevaQuality(document)” / “<Source>toVeevaRIM(document)” must be created.
Importing documents with multiple versions is supported by Veeva importer. The structure of the versions tree is generated by the scanners of the systems that support this feature and provide means to extract it. The order of the versions in Veeva Vault is set through the system rules: major_version_number_v and minor_version_number__v.
All objects from a version structure must be imported since each of them references its antecedents, going back to the very first version. Therefore, it is advised to not drop the versions of an object between the scan and the import processes as this will generate inconsistencies and errors. If an object is intended to be migrated without versions (i.e. only the current version of the object needs to be migrated) than the affected objects should be scanned without enabling the respective scanner’s versioning option.
Rules for System Attributes
The following rules for system attributes are available when importing documents.
System Rule | Description |
type__v* | The Type from the Veeva Document Types tree Mandatory |
subtype__v | The SubType from the Veeva Document Types tree |
classification__v | The Classification from the Veeva Document Types tree |
file_extension | Optional rule for: - setting the extension of the imported document in case the source content does not have any extension in the filesystem. - overriding the existing extension from the source content on the filesystem |
lifecycle__v* | The name of the lifecycle that should be attached to the document. Mandatory |
status__v* | The name of the lifecycle status Mandatory |
major_version_number__v | Rule for indicating the major version number |
minor_version_number__v | Rule for indicating the minor version number |
mc_content_location | Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the adapter definition then the adapter will suppose that this parameter contains the FTP path for every document. |
mc_rendition_paths | Path on the filesystem or FTP where the renditions are located. Must be the same number of values as mc_rendition_types. See Renditions for more details. |
mc_rendition_types | The rendition type from the available Veeva types for the specified document type. Must be the same number of values as mc_rendition_paths. |
name__v | The name of the document |
parent_section | Optional rule used for specifying the section names in the binders where the documents will be imported. |
permissions | Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com mc_custom_role__c.groups=approver_group__c |
attachments | Should be set with the file paths of the attachments to set for the current document. If the parameter skipUploadToFTP is not checked then the values should contain the paths on the filesystem to the files otherwise, the rule will contain the FTP paths of the attachment files. |
target_type* | The name of the object type from the migration-center’s object type definitions. Ex: veeva-rim-document Mandatory |
mc_content_location | This rule offers the possibility to overwrite the location of the objects content. |
Renditions
When importing documents using the Veeva importer you have the option of migrating existing renditions adding different content as renditions. This is done using the two system rules available in the migset: ”mc_rendition_paths” and ”mc_rendition_types”. Each of these two rules can have multiple values and you need to set the same number of values for both rules. Doing otherwise will cause the importer to throw an error.
If the parameter skipUploadToFTP is not checked then the ”mc_rendition_paths” rule will need to contain a path on the filesystem to the content just like for the ”mc_content_location” attribute otherwise, the rule will contain the FTP path of the desired content.
The ”mc_rendition_type” will need to specify the name of the rendition type from Veeva. The specific types of renditions available are specified in the document type definition in Veeva.
”mc_rendition_type” needs the internal name of the rendition type (e.g. large_size_asset__v).
Relations
Veeva Importer allows importing "dm_relation" objects scanned by Documentum Scanner as Veeva Vault relations. The internal name of Veeva Vault relations that will be created should be configured in the configurations file "relations-mapping.properties" located in the jobserver folder ./lib/mc-veeva-importer.
The internal name of available Veeva Vault relations can be found in the Vault configuration menu. The internal name must be used instead of the label.
The relations will be imported after all documents and versions are imported. If a document part of a relation could not be imported, the relation will not be imported so the document being parent in the relation will be moved in status "Partially Imported". The documents in the status "Partially Imported" can be submitted to the importer again but their relations will be imported only after the documents being children in the relations are imported.
More information about how relations are working in the Veeva Vault can be found here: About document relationships.
Working with Binders
Binders allow users to organize and group documents in a flexible structure. Binders are comprised of sections, which can nest to build a hierarchical structure, and links to documents in Vault.
Veeva importer allows you to import Virtual Documents scanned from Documentum as Binders in any supported Veeva Vault. For that a migset of type “DctmtoVeeva<type>(binder)” must be created.
The following features are currently supported:
Create binders from scratch by preserving the hierarchical structure of the Virtual Document
Create binders based on a template so the sections are generated as they are defined in the template
Importing Virtual Documents version as Binder versions
Import the content of the Virtual Document as the first child of the Binder
Preserving version binding (implicit and symbolic) for documents and binders
When importing Virtual Documents as Binders all normal documents that are referenced by the Virtual Documents should be imported before or together with the binders (in the same importer run).
Setting version labels as minor or major does not work for binders!
Rules for System Attributes
The following rules for system attributes are available when importing binders.
System Rule | Description |
type__v* | The Type from the Veeva Document Types tree Mandatory |
subtype__v | The SubType from the Veeva Document Types tree |
classification__v | The Classification from the Veeva Document Types tree |
lifecycle__v* | Should be set with the name of the lifecycle where the document will be attached. Mandatory |
status__v* | The name of the lifecycle status Mandatory |
major_version_number__v | Rule for indicating the major version number (does not work for binders) |
minor_version_number__v | Rule for indicating the minor version number (does not work for binders) |
mc_content_location | Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the adapter definition then the adapter will suppose that this parameter contains the FTP path of the file. |
name__v | The name of the document |
permissions | Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com mc_custom_role__c.groups=approver_group__c |
template_name | The name of the template the Binder will be created from. Leave it empty to create the binder from scratch. |
target_type* | The name of the object type from the migration-center’s object type definitions. Ex: veeva-clinical-document Mandatory |
Version Binding
When importing Documentum Virtual Documents as Veeva Vault Binders, the binder components can be bound to a specific version. When checking “preserveVersionBinding” in the importer configuration, the binder components are bound to the same version as they are bound in Virtual Documents in Documentum. The components will be bound to the latest version in Veeva Vault in the following cases:
The version binding is not specified in the “version_label” attribute of the “dmr_containment” object in the Documentum
The version binding is specified in Documentum but “preserveVersionBinding” was not enabled in Documentum
Circular references between binders are not allowed in Veeva Vault and therefore the importer will throw an error when trying to import circular references. To avoid that, the circular references should be fixed in the source system before scanning the Virtual Documents.
Importing Documents to Binder Sections
Creating binders with sections is possible by specifying a template name that contains sections. The documents can be imported to sections by setting the “parent_section” system rule in the migset containing the normal documents. In case of hierarchical sections, the sections will be defined separated by slash “/”. Ex: Section1/Section1.1/Section1.1.1
Working with RIM Submissions
Veeva RIM Importer allows you to import Submission Archives in RIM Vault. For that, a migset of type “DctmtoVeevaRIM(submission)” must be created. The importer does not create the submission definition but import submission archives to existing submission. The zip file to be imported must contain the “Submission” folder and “Application” folder.
The following rules for system attributes are available when importing binders.
System Rule | Description |
application_number | Application number the submission belongs to |
submission__v | The submission name |
submission_file_path | The path to the zip file containing the submission archive. If the skipUploadToFTP parameter is set in the adapter definition then the adapter will suppose that this parameter contains the FTP path of the zip file. |
target_type | The name of the object type from the migration-center’s object type definitions: veeva-rim-submission |
Working with Veeva Vault Specific Features
Setting the Document type/subtype/classification
There is a known limitation regarding setting classification elements when the Veeva Vault configuration contains elements with the same label on the same level. In other words, Veeva platform allows you to have, for example, 2 types with the same label and different names (My Type - my_type__c, My Type - my_type1__c). In order to prevent any issue, we strongly recommend the type, subtype and classification elements to be unique.
When performing a job with the Veeva importer you must specify the “target_type” and the type/subtype/classification for the documents. Due to how the Veeva document types are structured, the object type definition in migration-center will act as a container for all the necessary target attributes from Veeva. Therefore the object type definition will be set for the ”target_type” system rule, but the actual type of the document will be defined by the values set for the type__v / subtype__v / classification__v system rules.
Required Attributes Based on Vault Type
Depending on which type of Veeva Vault you are importing into (Clinical, Quality or RIM) there are several attributes which are mandatory, and you will need to associate transformation rules with valid values, as follows:
Clinical: product__v, study__v, blinding__v
Quality: country__v, owning_department__v, owning_facility__v
RIM: none
The values for these attributes need to be the actual label value and not the internal name or id (i.e. “Base Doc Lifecycle” instead of “clinical_doc_lifecycle__c”. This applies to all of the attributes mentioned above.
Lifecycles
Each document type in Veeva can have one or multiple Lifecycles associated with it. When importing a document of a certain type you need to specify, using the ”lifecycle__v” attribute, which of the available lifecycles must be set for the document.
To find out or modify the available lifecycles for a specific type you need to navigate in Veeva to the Admin section -> Configuration tab -> Document Types and open the specific type you are interested in.
From the same Configuration section you will find Document Lifecycles, here you can select a lifecycle and see the specific lifecycle states which you can set for the ”status__v” attribute.
Permissions
The available permissions for a specific document are specified by the lifecycle which is attached to that document. To see what available permissions there are, you need to go to the Admin interface -> Configuration -> Document Lifecycles then select the lifecycle and go to the tab Roles. These are the permissions you will be able to set with the Veeva Importer if you attach this particular lifecycle to the document you are importing.
The format is the following:
{Role_name}.{USERS or GROUPS}=username1,username2,…
Example:
reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com
mc_custom_role__c.groups=approver_group__c
Setting References to Existing Master Data Objects in Veeva
Veeva supports document/object attributes that reference master data objects by their internal ID. Usually these internal IDs are unknown when migrating documents/objects from a source system to Veeva. Thus, the migration-center let the users set other values than “ID” for the referenced objects. By default, the user should provide the value of the field name__v for the referenced objects. Setting the value of other fields is possible through a configuration file that is specified in the importer parameter: attributeMappingLocation.
The configuration file should have the following structure for document attributes:
For example, when setting the field “country__v” the user may want to set in migration-center the country abbreviation value. In this case, the following line should be added in the configuration file:
The configuration file should have the following structure for object attributes:
If you want to use the "name__v" instead of "id" when you set the "country__v" field from "location__v" object then the following line should be added in the configuration file:
The field of the object (abbreviation__c, country__v) that will be set in the configuration file should be defined as unique in the Object definition in Veeva Vault otherwise the lookup functionality may not work properly.
Setting an object field that refers to a specific Veeva document version can be done by setting the document id so it shouldn't be specified in the configuration file.
Filtering the Values of the Master Data Objects
The migration-center allows the user to filter the possible values that can be set to a document or object field. It is possible to use multiple fields to get the desired value for the attribute that references a master data object.
In addition to the structure for document or object attributes, the following structure should be added:
The "and" keyword is added in case of usage of multiple fields.
For example, when setting the “country__v” document field, the user may want to set in migration-center the country abbreviation value and he wants to enforce using only "country__v" active objects. In this case, the following line should be added in the configuration file:
If you want to set the "product_type__v" field which refers to a Controlled Vocabulary object when importing Submission objects and you want to ensure that only the controlled vocabulary objects of "Submission Type" are taken into consideration then the following line should be added in the configuration file:
If a picklist attribute is used to filter the reference object records, then the Picklist Value Name must be used for specifying the attribute value.
Attachments
The system rule “attachments” allows the user to import attachments for the documents. It should be set with the fully qualified paths of the attachments.
If a document has two or more attachments with the same name but different content, the importer will create multiple versions for the same attachment.
Delta Migration
Objects that have changed in the source system since the last scan are scanned as update objects. Whether an object in a migration set is an update or not can be seen by checking the value of the Is_update column – if it’s 1, the current object is an update to a previously scanned object (the base object). An update object cannot be imported unless its base object has been imported previously.
Currently, update objects are processed by Veeva importer with the following limitations:
Only document and object updates can be imported. The updates for Binders are reported as errors by the importer
Only metadata is updated for the documents. The content, renditions, permissions and classifications are not updated.
New versions of documents are fully supported
New fields can be added by an update to existing Veeva Objects
Beside the delta mechanism described above, the importer allows importing new versions to documents based on “external_id__v” system rule. If that’s set, the importer will behave in the following way:
A document with the same external id exists in the vault The importer adds the document being imported as a new version to the existing document. The new version can only be imported if a version with the same major and minor value doesn’t already exists.
A document with the same external id cannot be found in the vault
If the document being imported is a root version (level_in_version_tree = 0) then a new document is created.
If the document being imported is not a root version (level_in_version_tree > 0) then the documents fail to import, and an appropriate error message is logged.
Object names must be unique. Thus, the importer will report an error if an update tries to change the name of the object to an existing one.
Log Files
A complete history is available for any Veeva Importer job is available from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.
Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:
Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.
Log files generated by the Documentum Adapter can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs
The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.
Last updated
