Veeva Importer

Introduction

The Veeva Importer is one of the target connectors available in migration-center starting with version 3.7. It takes the objects processed in migration-center and imports them into a Veeva Vault. Currently, for every supported Veeva Vault, there is a specific importer type:

• Clinical: ClinicalImporter

• Quality: QualityImporter

• RIM: RIMImporter

Currently, the Veeva Connector uses version 21.2 of the Veeva REST API.

When the term "Veeva Importer" is used in the following, it refers to all three Veeva Importers. The name of the specific Veeva Importer will be used if there is a feature applicable just for that connector.

Known issues & limitations

• The objectsOrder feature requires that all the object_name values present in the migset are set in the parameter, otherwise the order will be decided by the importer. The values must also be present in Veeva Vault. If a value that is not present in Veeva Vault is set, the importer will not throw an error. (#59981, #59980)

• Setting version labels directly as minor or major does not work for binders. (#53160) (see User Actions for workaround)

• No error reported when setting values to Inactive Fields in Veeva Importer (#50880)

• No error reported when setting permissions not in the selected lifecycle in Veeva Importer (#50939)

• Binders are put in batches differently than documents as follows:

  • without preserveVersionBinding: 1 batch for ALL root binders and then no batching for the rest of binder versions

  • with preserveVersionBinding: All binder objects are imported in a single batch

Prerequisites

Network Prerequisites

The importer uses FTP to upload the content of the documents and their versions to the Veeva Vault environment. This means the content will be uploaded first to the Veeva FTP server, so the necessary outbound ports (i.e. 21 and 56000 – 56100) should be opened in your firewalls as described here:

http://vaulthelp.vod309.com/wordpress/admin-user-help/admin-basics/accessing-your-vaults-ftp-server/

https://support.veeva.com/hc/en-us/articles/216008767-Error-Connection-refused-by-server-Unable-to-Connect-to-FTP-Site-Using-Vault

Planning the Migration

Before starting the ingestion into Veeva Vault you should inform Vault Product Support about this by filling this form:

https://docs.google.com/forms/d/e/1FAIpQLSeIgAcLsdkdlRRtM0mIyKPbTraExiTY13ojAJSl2HMfEVlCuA/viewform

You should use the form 3 business days in advance of any migration and at least one week in advance of any significant migration. A significant migration includes over 10,000 office documents, 1,000 large documents (like video) or 500,000 object records.

In addition, if Migration is being run during “off hours” for the Vault’s location, or weekends, you should ensure that Vault Product Support is aware of requested activity.

Veeva Importer Properties

To create a new Veeva Clinical Importer job click on the New Importer button and select “Veeva Clinical” from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the Veeva parameters. The other types of Veeva Importer can be created in a similar way by selecting the corresponding adapter type.

The Properties window of an importer can be accessed by double-clicking an importer in the list or selecting the Properties button or entry from the toolbar or context menu.

Common Importer Parameters

Configuration parameters	Values
Name	Enter a unique name for this importer Mandatory
Adapter type	Select the “Veeva Clinical” connector from the list of available connectors Mandatory
Location	Select the Job Server location where this job should be run. Job Servers are defined in the Jobserver window. If no Job Server was selected, migration-center will prompt the user to define a Job Server Location when saving the importer. Mandatory
Description	Enter a description for this importer (optional)

Veeva Importer Parameters

Configuration parameters	Values
username*	Veeva username. It must be a Vault Owner. Mandatory
password*	The user’s password. Mandatory
server*	Veeva Vault name. Ex: fme-clinical.veevavault.com Mandatory
proxyServer	The name or IP of the proxy server if there is any.
proxyPort	The port for the proxy server.
proxyUser	The username if required by the proxy server.
proxyPassword	The password if required by the proxy server.
attributeMappingLocation	The path of the configuration file that will be used for setting the references to the existing master data objects when importing documents. See Attribute mapping
preserveVersionBinding	Indicates if the version binding will be preserved when importing virtual documents from Documentum as binders in Veeva Vault. See Working with Binders for more details.
importRelations	Indicates if relations between documents will imported. If checked, the relations between the documents being imported will be imported as well. If not checked the relations will not be imported. They can be imported in another run by assigning the same migsets to the importer.
batchSize*	The number of documents or versions that will be imported in a single bulk operation. Mandatory
useFtpServer	When checked the content is uploaded to Staging Folder by using FTPS protocol. When not checked the content is uploaded to Staging Folder by using REST API calls. The recommended way is to leave it unchecked.
skipUploadToFTP	If not checked, the importer will copy the documents content to the FTP server during import from the location where they have been exported by the scanner. If checked, the importer presumes the documents content was copied to the FTP server prior starting the import. In this case all paths of the content in the system rules mc_content_location, mc_rendition_paths, attachments or submissionFilePath must be set with the corresponding paths on the FTP server.
allowIncompleteBinders	If checked binders with children that are not or cannot be imported, will be imported partially. Please note that incomplete binders might not fixed later because older versions of binders are immutable in Veeva Vault
importBinderContentAsAttachment	If checked, the content of the VD to be imported as binder, will be added as an attachment to the binder version. The attachment name will be <object_name>_versionNumber.
ftpMaxNumberOfThreads*	Maximum number of concurrent threads used to transfer content to the FTPS server. The max value is 20 but according with Veeva migration best practices it is strongly recommended to use maximum 5 threads.
objectsOrder	The names of objects in the import order. If is not filled then the importer will compute the order automatically. Example: test_object_1__c,test_object_2__c The test_object_1__c will be imported first and the objects of type test_object_2__c will be imported only after all objects of type test_object_1__c have been imported.
loggingLevel*	Sets the verbosity of the log file. Values: 1 - logs only errors during scan 2 - is the default value reporting all warnings and errors 3 - logs all successfully performed operations in addition to any warnings or errors 4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production) Mandatory

On the | Migsets | tab, the user can select the migration sets to be imported with this importer. Depending on the chosen Adapter Type only the migration sets compatible with this type of importer will be displayed and can be selected for import. Also, only migration sets containing at least one object in a validated state will be displayed (since objects which haven’t been validated cannot be imported).

Available migration sets can be moved between the two columns by double clicking or using the arrows buttons.

Additional Importer Configuration

For configuring some additional parameters that will apply to all importers runs, a configuration file (config.properties) provided in the folder …\lib\mc-veeva-importer. The following settings are available:

Configuration name	Description
client_id_prefix	This is the prefix of the Client ID that is passed to every Vault API call. The Client ID is always logged in the report log. For a better identification of the requests (if necessary) the default value should be changed to include the name of the company as described here: https://developer.veevavault.com/docs/#client-id Default value = fme-vault-client-migrationcenter
request_timeout	The time in milliseconds the importer waits for response after every API call. Default value = 60000
no_of_request_attempts	Represents the number of request attempts when the first call fails due to a server error(5xx error code). The REST API call will be executed at least once independent of the parameter value. Default: 2

For applying the changes in the config.properties file the Job Server needs to be restarted.

Working with Objects

Vault Objects are part of the application data model in Veeva Vault. Vault Objects are usually configured to be referenced in the document fields. For example, the Product document field corresponds to the Product object and is associated with all document types.

Veeva Importer allows importing object records from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeeva<type>(object)” must be created. Ex: Veeva Quality Importer works with migsets of type “<Source>toVeevaQuality(object)”.

Rules for System Attributes

The following rules for system attributes are available when importing objects.

System Rule	Description
name__v	It must be set with the name of the object being imported. For some objects the “name__v” field is configured to be generated automatically by Veeva Vault when object is created. Usually, “name__v” is configured to be unique so if duplicated values are provided to the importer the affected objects fail to import. Optional
object_name	The name of the object. Ex: study__v, study_country__v Mandatory
target_type	The name of the migration-center internal object type that is used in the association. Ex: veeva-quality-object Mandatory
attachments	Can be used to set file attachments to the object just in case it allows that (The "Allow attachments" option has to be set on true). You must provide the full path (UNC or local filesystem on the job server) to the attachment file. Multiple files can be specified. If the parameter skipUploadToFTP is checked, you must provide the FTP paths to the files. Optional

Any editable object fields can be set using the normal transformation rules.

On Delta Migration, the existing attachments of an object will be replaced with those that are set on the "attachments" system attribute otherwise, the object will keep its attachments.

Working with Documents

Veeva importer allows importing documents from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeevaClinical(document)” / “<Source>toVeevaQuality(document)” / “<Source>toVeevaRIM(document)” must be created.

Importing documents with multiple versions is supported by Veeva importer. The structure of the versions tree is generated by the scanners of the systems that support this feature and provide means to extract it. The order of the versions in Veeva Vault is set through the system rules: major_version_number_v and minor_version_number__v.

All objects from a version structure must be imported since each of them references its antecedents, going back to the very first version. Therefore, it is advised to not drop the versions of an object between the scan and the import processes as this will generate inconsistencies and errors. If an object is intended to be migrated without versions (i.e. only the current version of the object needs to be migrated) than the affected objects should be scanned without enabling the respective scanner’s versioning option.

Rules for System Attributes

The following rules for system attributes are available when importing documents.

System Rule	Description
type__v*	The Type from the Veeva Document Types tree Mandatory
subtype__v	The SubType from the Veeva Document Types tree
classification__v	The Classification from the Veeva Document Types tree
external_id__v	Rule used for sepcial Delta Migration behavior. See External ID
file_extension	Optional rule for: - setting the extension of the imported document in case the source content does not have any extension in the filesystem. - overriding the existing extension from the source content on the filesystem
is_deleted	When set to True the importer will delete the that specific version after import. This is useful when importing from source systems that with obsolete or broken versions.
lifecycle__v*	The name of the lifecycle that should be attached to the document. Mandatory
status__v*	The name of the lifecycle status Mandatory
major_version_number__v	Rule for indicating the major version number Mandatory
minor_version_number__v	Rule for indicating the minor version number Mandatory
mc_content_location	Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the connector definition then the connector will suppose that this parameter contains the FTP path for every document.
mc_rendition_paths	Path on the filesystem or FTP where the renditions are located. Must be the same number of values as mc_rendition_types. See Renditions for more details.
mc_rendition_types	The rendition type from the available Veeva types for the specified document type. Must be the same number of values as mc_rendition_paths.
name__v	The name of the document
parent_section	Optional rule used for specifying the section names in the binders where the documents will be imported.
permissions	Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=[e-mail],[e-mail] mc_custom_role__c.groups=approver_group__c
attachments	Should be set with the file paths of the attachments to set for the current document. If the parameter skipUploadToFTP is not checked then the values should contain the paths on the filesystem to the files otherwise, the rule will contain the FTP paths of the attachment files.
target_type*	The name of the object type from the migration-center’s object type definitions. Ex: veeva-rim-document Mandatory
mc_content_location	This rule offers the possibility to overwrite the location of the objects content.

Renditions

When importing documents using the Veeva importer you have the option of migrating existing renditions adding different content as renditions. This is done using the two system rules available in the migset: ”mc_rendition_paths” and ”mc_rendition_types”. Each of these two rules can have multiple values and you need to set the same number of values for both rules. Doing otherwise will cause the importer to throw an error.

If the parameter skipUploadToFTP is not checked then the ”mc_rendition_paths” rule will need to contain a path on the filesystem to the content just like for the ”mc_content_location” attribute otherwise, the rule will contain the FTP path of the desired content.

The ”mc_rendition_type” will need to specify the name of the rendition type from Veeva. The specific types of renditions available are specified in the document type definition in Veeva.

”mc_rendition_type” needs the internal name of the rendition type (e.g. large_size_asset__v).

Relations

Veeva Importer allows importing "dm_relation" objects scanned by Documentum Scanner as Veeva Vault relations. The internal name of Veeva Vault relations that will be created should be configured in the configurations file "relations-mapping.properties" located in the jobserver folder ./lib/mc-veeva-importer.

relations-mapping.properties

DctmRelation=supporting_documents__c

If the source relation name contains space characters they will need to be escaped with a forward slash ("\ ") or by using the unicode character for space ("\u0020").

The internal name of available Veeva Vault relations can be found in the Vault configuration menu. The internal name must be used instead of the label.

The relations will be imported after all documents and versions are imported. If a document part of a relation could not be imported, the relation will not be imported so the document being parent in the relation will be moved in status "Partially Imported". The documents in the status "Partially Imported" can be submitted to the importer again but their relations will be imported only after the documents being children in the relations are imported.

More information about how relations are working in the Veeva Vault can be found here: About document relationships.

Working with Binders

Binders allow users to organize and group documents in a flexible structure. Binders are comprised of sections, which can nest to build a hierarchical structure, and links to documents in Vault.

Veeva importer allows you to import Virtual Documents scanned from Documentum as Binders in any supported Veeva Vault. For that a migset of type “DctmtoVeeva<type>(binder)” must be created.

The following features are currently supported:

• Create binders by preserving the hierarchical structure of the Virtual Document

• Importing Virtual Documents version as Binder versions

• Preserving version binding (implicit and symbolic) for documents and binders

• Import the content of the Virtual Document as the first child of the Binder OR import the content as an attachment if importBinderContentAsAttachment is checked.

• Create binders based on a template so the sections are generated as they are defined in the template

• Setting binder permissions

When importing Virtual Documents as Binders all normal documents that are referenced by the Virtual Documents should be imported before or together with the binders (in the same importer run). Unless allowIncompleteBinders is used, in which case binders can be imported partially without the child documents.

Setting version labels as minor or major does not work for binders!

Rules for System Attributes

The following rules for system attributes are available when importing binders.

System Rule	Description
type__v*	The Type from the Veeva Document Types tree Mandatory
subtype__v	The SubType from the Veeva Document Types tree
classification__v	The Classification from the Veeva Document Types tree
lifecycle__v*	Should be set with the name of the lifecycle where the document will be attached. Mandatory
status__v*	The name of the lifecycle status Mandatory
major_version_number__v	Rule for indicating the major version number (does not work for binders)
minor_version_number__v	Rule for indicating the minor version number (does not work for binders)
mc_content_location	Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the connector definition then the connector will suppose that this parameter contains the FTP path of the file.
name__v	The name of the document
permissions	Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=[e-mail],[e-mail] mc_custom_role__c.groups=approver_group__c
template_name	The name of the template the Binder will be created from. Leave it empty to create the binder from scratch.
target_type*	The name of the object type from the migration-center’s object type definitions. Ex: veeva-clinical-document Mandatory
user_actions	Allows settings a user action or a chain of user actions to change the version number or the status. The name of the user action will be used.

Version Binding

When importing Documentum Virtual Documents as Veeva Vault Binders, the binder components can be bound to a specific version. When checking “preserveVersionBinding” in the importer configuration, the binder components are bound to the same version as they are bound in Virtual Documents in Documentum. The components will be bound to the latest version in Veeva Vault in the following cases:

• The version binding is not specified in the “version_label” attribute of the “dmr_containment” object in the Documentum

• The version binding is specified in Documentum but “preserveVersionBinding” was not enabled in the importer.

Circular references between binders are not allowed in Veeva Vault and therefore the importer will throw an error when trying to import circular references. To avoid that, the circular references should be fixed in the source system before scanning the Virtual Documents.

User actions

You can set the binder version status or the version number by specifying user actions that will perform the necessary changes.

A Document Lifecycle in Veeva (which is also used on binders) has certain States. On each state there can be configured one or several User Actions and Entry Actions.

The User Actions can be used to change the document to another state (among other things). When a document or binder enters this new Lifecycle state, the Entry Actions defined for that state will trigger. These entry actions can be configured to also change the version of the binder to a major or minor one.

Therefore to change set a binder version to major or minor you need to set a User Action that changes to a new state which has the Entry Action that sets the version:

User Action with change to new state

State with Entry Action to set new major version

A user action or a chain of user actions can be specified by using the system rule "user_actions". The value from name__v field will be used in the migset configuration. Example: change_to_third__c

The system rules "status__v", "major_version_number__v" and "minor_version_number__v" will be used just for the root version of the binder. Otherwise, the status will be set to the default one configured in the document lifecycle and the binder number will be computed based on the configuration made on the Vault.

The user actions will be executed in sequence. The binder will be created having the initial state of the document lifecycle, in this case, it is the Draft state. This state has a change_to_second__c user action that will be executed first and the binder will be moved to the status Second State. The new state now has a user action named change_to_third__c which is executed next and binder is moved to Third State.

If a user action fails due to any error, the entire binder version will fail to import and be reverted from Veeva.

Importing Documents to Binder Sections

Creating binders with sections is possible by specifying a template name that contains sections. The documents can be imported to sections by setting the “parent_section” system rule in the migset containing the normal documents. In case of hierarchical sections, the sections will be defined separated by slash “/”. Ex: Section1/Section1.1/Section1.1.1

Working with RIM Submissions

Veeva RIM Importer allows you to import Submission Archives in RIM Vault. For that, a migset of type “DctmtoVeevaRIM(submission)” must be created. The importer does not create the submission definition but import submission archives to existing submission. The zip file to be imported must contain the “Submission” folder and “Application” folder.

The following rules for system attributes are available when importing binders.

System Rule	Description
application_number	Application number the submission belongs to
submission__v	The submission name
submission_file_path	The path to the zip file containing the submission archive. If the skipUploadToFTP parameter is set in the connector definition then the connector will suppose that this parameter contains the FTP path of the zip file.
target_type	The name of the object type from the migration-center’s object type definitions: veeva-rim-submission

Working with Veeva Vault Specific Features

Setting the Document type/subtype/classification

There is a known limitation regarding setting classification elements when the Veeva Vault configuration contains elements with the same label on the same level. In other words, Veeva platform allows you to have, for example, 2 types with the same label and different names (My Type - my_type__c, My Type - my_type1__c). In order to prevent any issue, we strongly recommend the type, subtype and classification elements to be unique.

When performing a job with the Veeva importer you must specify the “target_type” and the type/subtype/classification for the documents. Due to how the Veeva document types are structured, the object type definition in migration-center will act as a container for all the necessary target attributes from Veeva. Therefore the object type definition will be set for the ”target_type” system rule, but the actual type of the document will be defined by the values set for the type__v / subtype__v / classification__v system rules.

Required Attributes Based on Vault Type

Depending on which type of Veeva Vault you are importing into (Clinical, Quality or RIM) there are several attributes which are mandatory, and you will need to associate transformation rules with valid values, as follows:

• Clinical: product__v, study__v, blinding__v

• Quality: country__v, owning_department__v, owning_facility__v

• RIM: none

The values for these attributes need to be the actual label value and not the internal name or id (i.e. “Base Doc Lifecycle” instead of “clinical_doc_lifecycle__c”. This applies to all of the attributes mentioned above.

Lifecycles

Each document type in Veeva can have one or multiple Lifecycles associated with it. When importing a document of a certain type you need to specify, using the ”lifecycle__v” attribute, which of the available lifecycles must be set for the document.

To find out or modify the available lifecycles for a specific type you need to navigate in Veeva to the Admin section -> Configuration tab -> Document Types and open the specific type you are interested in.

From the same Configuration section you will find Document Lifecycles, here you can select a lifecycle and see the specific lifecycle states which you can set for the ”status__v” attribute.

Permissions

The available permissions for a specific document are specified by the lifecycle which is attached to that document. To see what available permissions there are, you need to go to the Admin interface -> Configuration -> Document Lifecycles then select the lifecycle and go to the tab Roles. These are the permissions you will be able to set with the Veeva Importer if you attach this particular lifecycle to the document you are importing.

The format is the following:

{Role_name}.{USERS or GROUPS}=username1,username2,…

Example:

reviewer__v.users=[e-mail],[e-mail]

mc_custom_role__c.groups=approver_group__c

Attribute Mapping for objects fields

In Veeva a Document or an Object can have an attribute (field) that references other Veeva objects.

In Veeva these references are made by using the ID of the referenced object. But when you import with migration-center you need to use the name__v value to find the reference object by default. This is because, in most cases, the ID of the object is unknown during a migration.

To change this behavior and use other attribute values to reference objects, you can configure an attribute mapping file and set it in the attributeMappingLocation parameter.

The format of the mapping in the file depends on whether you are specifying Document or Object fields:

Document fields

Consider you are importing Documents in Veeva that have a field named owning_department__v that references objects of type department__v.

To make the Veeva Importer find Department objects based on the Department Number/Code (number__v) instead of the default name__v, the attribute mapping file must contain the following line:

owning_department__v=department__v,number__v

Now, in migration-center, for the owning_department__v attribute you can assign values that represent the number__v of a Department object and the importer will properly find and reference the object.

The general rule looks like this:

<document_field_name>=<referenced_object_name>,<object_lookup_field>

Object fields

When importing Objects instead of Documents, the format is similar, but you need to also specify the object type name in the left side.

Consider you are importing an Object of type example_object__c which has a field named object_field__c that references objects of type ref_example_object.

To make the Veeva find these ref_example_object by their global_id__sys instead of the name, the attribute mapping file needs to contain the following line:

object_field__c.example_object__c=ref_example_object,global_id__sys

The general rule looks like this:

<object_field_name>.<object_name>=<ref_object_name>,<object_lookup_field>

Note that for the objects you use a dot in the lefthand side as opposed to a comma in the righthand

The new fields used to reference the objects need to be defined as unique in the Object definition otherwise the lookup functionality may not work properly.

Setting an object field that refers to a specific Veeva document version can be done by setting the document id so it shouldn't be specified in the configuration file.

Filtering the Values of the Master Data Objects

The migration-center allows the user to filter the possible values that can be set to a document or object field. It is possible to use multiple fields to get the desired value for the attribute that references a master data object.

In addition to the structure for document or object attributes, the following structure should be added:

where <object_field_1>=<field_value_1>

The "and" keyword is added in case of usage of multiple fields.

where <object_field_1>=<field_value_1> and <object_field_2>=<field_value_2> and <obejct_field_3>=<object_field_3>

For example, when setting the “country__v” document field, the user may want to set in migration-center the country abbreviation value and he wants to enforce using only "country__v" active objects. In this case, the following line should be added in the configuration file:

country__v=country__v,abbreviation__c where status__v='active__v'

If you want to set the "product_type__v" field which refers to a Controlled Vocabulary object when importing Submission objects and you want to ensure that only the controlled vocabulary objects of "Submission Type" are taken into consideration then the following line should be added in the configuration file:

product_type__v.submission__v=controlled_vocabulary__rim,name__v where controlled_vocabulary_type__rim='submission_type__rim'

If a picklist attribute is used to filter the reference object records, then the Picklist Value Name must be used for specifying the attribute value.

Attachments

The system rule “attachments” allows the user to import attachments for the documents. It should be set with the fully qualified paths of the attachments.

If a document has two or more attachments with the same name but different content, the importer will create multiple versions for the same attachment.

Delta Migration

Objects that have changed in the source system since the last scan are scanned as update objects. Whether an object in a migration set is an update or not can be seen by checking the value of the Is_update column – if it’s 1, the current object is an update to a previously scanned object (the base object). An update object cannot be imported unless its base object has been imported previously.

Currently, update objects are processed by Veeva importer with the following limitations:

• Only document and object updates can be imported. The updates for Binders are reported as errors by the importer

• Only metadata is updated for the documents. The content, renditions, permissions and classifications are not updated.

• New versions of documents are fully supported

• New fields can be added by an update to existing Veeva Objects

External ID

Beside the delta mechanism described above, the importer allows importing new versions to documents based on “external_id__v” system rule. If that’s set, the importer will behave in the following way:

1. A document with the same external id exists in the vault The importer adds the document being imported as a new version to the existing document. The new version can only be imported if a version with the same major and minor value doesn’t already exists.

2. A document with the same external id cannot be found in the vault

   1. If the document being imported is a root version (level_in_version_tree = 0) then a new document is created.

   2. If the document being imported is not a root version (level_in_version_tree > 0) then the documents fail to import, and an appropriate error message is logged.

3. Object names must be unique. Thus, the importer will report an error if an update tries to change the name of the object to an existing one.