Veeva Importer
Introduction
The Veeva Importer is one of the target connectors available in migration-center starting with version 3.7. It takes the objects processed in migration-center and imports them into a Veeva Vault. Currently, for every supported Veeva Vault, there is a specific importer type:
Clinical: ClinicalImporter
Quality: QualityImporter
RIM: RIMImporter
Currently, the Veeva Connector uses version 22.3 of the Veeva REST API.
When the term "Veeva Importer" is used in the following, it refers to all three Veeva Importers. The name of the specific Veeva Importer will be used if there is a feature applicable just for that connector.
Known issues & limitations
The objectsOrder feature requires that all the object_name values present in the migset are set in the parameter, otherwise the order will be decided by the importer. The values must also be present in Veeva Vault. If a value that is not present in Veeva Vault is set, the importer will not throw an error. (#59981, #59980)
Setting version labels directly as minor or major does not work for binders. (#53160) (see User Actions for workaround)
No error reported when setting values to Inactive Fields in Veeva Importer (#50880)
No error reported when setting permissions not in the selected lifecycle in Veeva Importer (#50939)
Binders are put in batches differently than documents as follows:
without preserveVersionBinding: 1 batch for ALL root binders and then no batching for the rest of binder versions
with preserveVersionBinding: All binder objects are imported in a single batch
In a multi-value picklist attribute, you can set values that contain commas (,) only by enclosing the value in quotes
i.e. value,1 -> "value,1"https://hawq.apache.org/docs/userguide/2.3.0.0-incubating/datamgmt/load/g-escaping-in-csv-formatted-files.html
Prerequisites
Network Prerequisites
The importer uses FTP to upload the content of the documents and their versions to the Veeva Vault environment. This means the content will be uploaded first to the Veeva FTP server, so the necessary outbound ports (i.e. 21 and 56000 – 56100) should be opened in your firewalls as described here:
http://vaulthelp.vod309.com/wordpress/admin-user-help/admin-basics/accessing-your-vaults-ftp-server/
Planning the Migration
Before starting the ingestion into Veeva Vault you should inform Vault Product Support about this by filling this form:
https://docs.google.com/forms/d/e/1FAIpQLSeIgAcLsdkdlRRtM0mIyKPbTraExiTY13ojAJSl2HMfEVlCuA/viewform
You should use the form 3 business days in advance of any migration and at least one week in advance of any significant migration. A significant migration includes over 10,000 office documents, 1,000 large documents (like video) or 500,000 object records.
In addition, if Migration is being run during “off hours” for the Vault’s location, or weekends, you should ensure that Vault Product Support is aware of requested activity.
Veeva Importer Properties
To create a new Veeva Clinical Importer job click on the New Importer button and select “Veeva Clinical” from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the Veeva parameters. The other types of Veeva Importer can be created in a similar way by selecting the corresponding adapter type.
The Properties window of an importer can be accessed by double-clicking an importer in the list or selecting the Properties button or entry from the toolbar or context menu.
Common Importer Parameters
Configuration parameters | Values |
Name | Enter a unique name for this importer Mandatory |
Adapter type | Select the “Veeva Clinical” connector from the list of available connectors Mandatory |
Location | Select the Job Server location where this job should be run. Job Servers are defined in the Jobserver window. If no Job Server was selected, migration-center will prompt the user to define a Job Server Location when saving the importer. Mandatory |
Description | Enter a description for this importer (optional) |
Veeva Importer Parameters
Configuration parameters | Values |
username* | Veeva username. It must be a Vault Owner. Mandatory |
password* | The user’s password. Mandatory |
server* | Veeva Vault name. Ex: fme-clinical.veevavault.com Mandatory |
proxyServer | The name or IP of the proxy server if there is any. |
proxyPort | The port for the proxy server. |
proxyUser | The username if required by the proxy server. |
proxyPassword | The password if required by the proxy server. |
attributeMappingLocation | The path of the configuration file that will be used for setting the references to the existing master data objects when importing documents. See Attribute mapping |
preserveVersionBinding | Indicates if the version binding will be preserved when importing virtual documents from Documentum as binders in Veeva Vault. See Working with Binders for more details. |
importRelations | Indicates if relations between documents will imported. If checked, the relations between the documents being imported will be imported as well. If not checked the relations will not be imported. They can be imported in another run by assigning the same migsets to the importer. |
batchSize* | The number of documents or versions that will be imported in a single bulk operation. Mandatory |
useFtpServer | When checked the content is uploaded to Staging Folder by using FTPS protocol. When not checked the content is uploaded to Staging Folder by using REST API calls. The recommended way is to leave it unchecked. |
skipUploadToFTP | If not checked, the importer will copy the documents content to the FTP server during import from the location where they have been exported by the scanner. If checked, the importer presumes the documents content was copied to the FTP server prior starting the import. In this case all paths of the content in the system rules mc_content_location, mc_rendition_paths, attachments or submissionFilePath must be set with the corresponding paths on the FTP server. |
bypassObjectValidationRules | If checked it allows importing invalid values for fields or missing mandatory values |
allowIncompleteBinders | If checked binders with children that are not or cannot be imported, will be imported partially. Please note that incomplete binders might not fixed later because older versions of binders are immutable in Veeva Vault |
importBinderContentAsAttachment | If checked, the content of the VD to be imported as binder, will be added as an attachment to the binder version. The attachment name will be <object_name>_versionNumber. |
ftpMaxNumberOfThreads* | Maximum number of concurrent threads used to transfer content to the FTPS server. The max value is 20 but according with Veeva migration best practices it is strongly recommended to use maximum 5 threads. |
objectsOrder | The names of objects in the import order. If is not filled then the importer will compute the order automatically. Example: test_object_1__c,test_object_2__c The test_object_1__c will be imported first and the objects of type test_object_2__c will be imported only after all objects of type test_object_1__c have been imported. |
loggingLevel* | Sets the verbosity of the log file. Values: 1 - logs only errors during scan 2 - is the default value reporting all warnings and errors 3 - logs all successfully performed operations in addition to any warnings or errors 4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production) Mandatory |
On the | Migsets | tab, the user can select the migration sets to be imported with this importer. Depending on the chosen Adapter Type only the migration sets compatible with this type of importer will be displayed and can be selected for import. Also, only migration sets containing at least one object in a validated state will be displayed (since objects which haven’t been validated cannot be imported).
Available migration sets can be moved between the two columns by double clicking or using the arrows buttons.
Additional Importer Configuration
For configuring some additional parameters that will apply to all importers runs, a configuration file (config.properties) provided in the folder …\lib\mc-veeva-importer. The following settings are available:
Configuration name | Description |
client_id_prefix | This is the prefix of the Client ID that is passed to every Vault API call. The Client ID is always logged in the report log. For a better identification of the requests (if necessary) the default value should be changed to include the name of the company as described here: https://developer.veevavault.com/docs/#client-id Default value = fme-vault-client-migrationcenter |
request_timeout | The time in milliseconds the importer waits for response after every API call. Default value = 60000 |
no_of_request_attempts | Represents the number of request attempts when the first call fails due to a server error(5xx error code). The REST API call will be executed at least once independent of the parameter value. Default: 2 |
For applying the changes in the config.properties file the Job Server needs to be restarted.
Working with Objects
Vault Objects are part of the application data model in Veeva Vault. Vault Objects are usually configured to be referenced in the document fields. For example, the Product document field corresponds to the Product object and is associated with all document types.
Veeva Importer allows importing object records from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeeva<type>(object)” must be created. Ex: Veeva Quality Importer works with migsets of type “<Source>toVeevaQuality(object)”.
Rules for System Attributes
The following rules for system attributes are available when importing objects.
System Rule | Description |
name__v | It must be set with the name of the object being imported. For some objects the “name__v” field is configured to be generated automatically by Veeva Vault when object is created. Usually, “name__v” is configured to be unique so if duplicated values are provided to the importer the affected objects fail to import. Optional |
object_name | The name of the object. Ex: study__v, study_country__v Mandatory |
target_type | The name of the migration-center internal object type that is used in the association. Ex: veeva-quality-object Mandatory |
attachments | Can be used to set file attachments to the object just in case it allows that (The "Allow attachments" option has to be set on true). You must provide the full path (UNC or local filesystem on the job server) to the attachment file. Multiple files can be specified. If the parameter skipUploadToFTP is checked, you must provide the FTP paths to the files. Optional |
Any editable object fields can be set using the normal transformation rules.
On Delta Migration, the existing attachments of an object will be replaced with those that are set on the "attachments" system attribute otherwise, the object will keep its attachments.
Working with Documents
Veeva importer allows importing documents from any supported source system to any supported Veeva Vault. For that, a migset of type “<Source>toVeevaClinical(document)” / “<Source>toVeevaQuality(document)” / “<Source>toVeevaRIM(document)” must be created.
Importing documents with multiple versions is supported by Veeva importer. The structure of the versions tree is generated by the scanners of the systems that support this feature and provide means to extract it. The order of the versions in Veeva Vault is set through the system rules: major_version_number_v and minor_version_number__v.
All objects from a version structure must be imported since each of them references its antecedents, going back to the very first version. Therefore, it is advised to not drop the versions of an object between the scan and the import processes as this will generate inconsistencies and errors. If an object is intended to be migrated without versions (i.e. only the current version of the object needs to be migrated) than the affected objects should be scanned without enabling the respective scanner’s versioning option.
Rules for System Attributes
The following rules for system attributes are available when importing documents.
System Rule | Description |
type__v* | The Type from the Veeva Document Types tree Mandatory |
subtype__v | The SubType from the Veeva Document Types tree |
classification__v | The Classification from the Veeva Document Types tree |
external_id__v | Rule used for sepcial Delta Migration behavior. See External ID |
file_extension | Optional rule for: - setting the extension of the imported document in case the source content does not have any extension in the filesystem. - overriding the existing extension from the source content on the filesystem |
is_deleted | When set to True the importer will delete the that specific version after import. This is useful when importing from source systems that with obsolete or broken versions. |
lifecycle__v* | The name of the lifecycle that should be attached to the document. Mandatory |
status__v* | The name of the lifecycle status Mandatory |
major_version_number__v | Rule for indicating the major version number Mandatory |
minor_version_number__v | Rule for indicating the minor version number Mandatory |
mc_content_location | Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the connector definition then the connector will suppose that this parameter contains the FTP path for every document. |
mc_rendition_paths | Path on the filesystem or FTP where the renditions are located. Must be the same number of values as mc_rendition_types. See Renditions for more details. |
mc_rendition_types | The rendition type from the available Veeva types for the specified document type. Must be the same number of values as mc_rendition_paths. |
name__v | The name of the document |
parent_section | Optional rule used for specifying the section names in the binders where the documents will be imported. |
permissions | Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com mc_custom_role__c.groups=approver_group__c |
attachments | Should be set with the file paths of the attachments to set for the current document. If the parameter skipUploadToFTP is not checked then the values should contain the paths on the filesystem to the files otherwise, the rule will contain the FTP paths of the attachment files. |
target_type* | The name of the object type from the migration-center’s object type definitions. Ex: veeva-rim-document Mandatory |
mc_content_location | This rule offers the possibility to overwrite the location of the objects content. |
Renditions
When importing documents using the Veeva importer you have the option of migrating existing renditions adding different content as renditions. This is done using the two system rules available in the migset: ”mc_rendition_paths” and ”mc_rendition_types”. Each of these two rules can have multiple values and you need to set the same number of values for both rules. Doing otherwise will cause the importer to throw an error.
If the parameter skipUploadToFTP is not checked then the ”mc_rendition_paths” rule will need to contain a path on the filesystem to the content just like for the ”mc_content_location” attribute otherwise, the rule will contain the FTP path of the desired content.
The ”mc_rendition_type” will need to specify the name of the rendition type from Veeva. The specific types of renditions available are specified in the document type definition in Veeva.
”mc_rendition_type” needs the internal name of the rendition type (e.g. large_size_asset__v).
Relations
Veeva Importer allows importing "dm_relation" objects scanned by Documentum Scanner as Veeva Vault relations. The internal name of Veeva Vault relations that will be created should be configured in the configurations file "relations-mapping.properties" located in the jobserver folder ./lib/mc-veeva-importer.
If the source relation name contains space characters they will need to be escaped with a forward slash ("\ ") or by using the unicode character for space ("\u0020").
The internal name of available Veeva Vault relations can be found in the Vault configuration menu. The internal name must be used instead of the label.
The relations will be imported after all documents and versions are imported. If a document part of a relation could not be imported, the relation will not be imported so the document being parent in the relation will be moved in status "Partially Imported". The documents in the status "Partially Imported" can be submitted to the importer again but their relations will be imported only after the documents being children in the relations are imported.
More information about how relations are working in the Veeva Vault can be found here: About document relationships.
Working with Binders
Binders allow users to organize and group documents in a flexible structure. Binders are comprised of sections, which can nest to build a hierarchical structure, and links to documents in Vault.
Veeva importer allows you to import Virtual Documents scanned from Documentum as Binders in any supported Veeva Vault. For that a migset of type “DctmtoVeeva<type>(binder)” must be created.
The following features are currently supported:
Create binders by preserving the hierarchical structure of the Virtual Document
Importing Virtual Documents version as Binder versions
Preserving version binding (implicit and symbolic) for documents and binders
Import the content of the Virtual Document as the first child of the Binder OR import the content as an attachment if importBinderContentAsAttachment is checked.
Create binders based on a template so the sections are generated as they are defined in the template
Setting binder permissions
When importing Virtual Documents as Binders all normal documents that are referenced by the Virtual Documents should be imported before or together with the binders (in the same importer run). Unless allowIncompleteBinders is used, in which case binders can be imported partially without the child documents.
Setting version labels as minor or major does not work for binders!
Rules for System Attributes
The following rules for system attributes are available when importing binders.
System Rule | Description |
type__v* | The Type from the Veeva Document Types tree Mandatory |
subtype__v | The SubType from the Veeva Document Types tree |
classification__v | The Classification from the Veeva Document Types tree |
lifecycle__v* | Should be set with the name of the lifecycle where the document will be attached. Mandatory |
status__v* | The name of the lifecycle status Mandatory |
major_version_number__v | Rule for indicating the major version number (does not work for binders) |
minor_version_number__v | Rule for indicating the minor version number (does not work for binders) |
mc_content_location | Optional rule for importing the content from another location than the one exported by the scanner. If the skipUploadToFTP parameter is checked in the connector definition then the connector will suppose that this parameter contains the FTP path of the file. |
name__v | The name of the document |
permissions | Allows setting permissions for the imported document. The format is the following: {Role_name}.{USERS or GROUPS}=username1,username2,… Example: reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com mc_custom_role__c.groups=approver_group__c |
template_name | The name of the template the Binder will be created from. Leave it empty to create the binder from scratch. |
target_type* | The name of the object type from the migration-center’s object type definitions. Ex: veeva-clinical-document Mandatory |
user_actions | Allows settings a user action or a chain of user actions to change the version number or the status. The name of the user action will be used. |
Version Binding
When importing Documentum Virtual Documents as Veeva Vault Binders, the binder components can be bound to a specific version. When checking “preserveVersionBinding” in the importer configuration, the binder components are bound to the same version as they are bound in Virtual Documents in Documentum. The components will be bound to the latest version in Veeva Vault in the following cases:
The version binding is not specified in the “version_label” attribute of the “dmr_containment” object in the Documentum
The version binding is specified in Documentum but “preserveVersionBinding” was not enabled in the importer.
Circular references between binders are not allowed in Veeva Vault and therefore the importer will throw an error when trying to import circular references. To avoid that, the circular references should be fixed in the source system before scanning the Virtual Documents.
User actions
You can set the binder version status or the version number by specifying user actions that will perform the necessary changes.
A Document Lifecycle in Veeva (which is also used on binders) has certain States. On each state there can be configured one or several User Actions and Entry Actions.
The User Actions can be used to change the document to another state (among other things). When a document or binder enters this new Lifecycle state, the Entry Actions defined for that state will trigger. These entry actions can be configured to also change the version of the binder to a major or minor one.
Therefore to change set a binder version to major or minor you need to set a User Action that changes to a new state which has the Entry Action that sets the version:
A user action or a chain of user actions can be specified by using the system rule "user_actions". The value from name__v field will be used in the migset configuration. Example: change_to_third__c
The system rules "status__v", "major_version_number__v" and "minor_version_number__v" will be used just for the root version of the binder. Otherwise, the status will be set to the default one configured in the document lifecycle and the binder number will be computed based on the configuration made on the Vault.
The user actions will be executed in sequence. The binder will be created having the initial state of the document lifecycle, in this case, it is the Draft state. This state has a change_to_second__c user action that will be executed first and the binder will be moved to the status Second State. The new state now has a user action named change_to_third__c which is executed next and binder is moved to Third State.
If a user action fails due to any error, the entire binder version will fail to import and be reverted from Veeva.
Importing Documents to Binder Sections
Creating binders with sections is possible by specifying a template name that contains sections. The documents can be imported to sections by setting the “parent_section” system rule in the migset containing the normal documents. In case of hierarchical sections, the sections will be defined separated by slash “/”. Ex: Section1/Section1.1/Section1.1.1
Working with RIM Submissions
Veeva RIM Importer allows you to import Submission Archives in RIM Vault. For that, a migset of type “DctmtoVeevaRIM(submission)” must be created. The importer does not create the submission definition but import submission archives to existing submission. The zip file to be imported must contain the “Submission” folder and “Application” folder.
The following rules for system attributes are available when importing binders.
System Rule | Description |
application_number | Application number the submission belongs to |
submission__v | The submission name |
submission_file_path | The path to the zip file containing the submission archive. If the skipUploadToFTP parameter is set in the connector definition then the connector will suppose that this parameter contains the FTP path of the zip file. |
target_type | The name of the object type from the migration-center’s object type definitions: veeva-rim-submission |
Working with Veeva Vault Specific Features
Setting the Document type/subtype/classification
There is a known limitation regarding setting classification elements when the Veeva Vault configuration contains elements with the same label on the same level. In other words, Veeva platform allows you to have, for example, 2 types with the same label and different names (My Type - my_type__c, My Type - my_type1__c). In order to prevent any issue, we strongly recommend the type, subtype and classification elements to be unique.
When performing a job with the Veeva importer you must specify the “target_type” and the type/subtype/classification for the documents. Due to how the Veeva document types are structured, the object type definition in migration-center will act as a container for all the necessary target attributes from Veeva. Therefore the object type definition will be set for the ”target_type” system rule, but the actual type of the document will be defined by the values set for the type__v / subtype__v / classification__v system rules.
Required Attributes Based on Vault Type
Depending on which type of Veeva Vault you are importing into (Clinical, Quality or RIM) there are several attributes which are mandatory, and you will need to associate transformation rules with valid values, as follows:
Clinical: product__v, study__v, blinding__v
Quality: country__v, owning_department__v, owning_facility__v
RIM: none
The values for these attributes need to be the actual label value and not the internal name or id (i.e. “Base Doc Lifecycle” instead of “clinical_doc_lifecycle__c”. This applies to all of the attributes mentioned above.
Lifecycles
Each document type in Veeva can have one or multiple Lifecycles associated with it. When importing a document of a certain type you need to specify, using the ”lifecycle__v” attribute, which of the available lifecycles must be set for the document.
To find out or modify the available lifecycles for a specific type you need to navigate in Veeva to the Admin section -> Configuration tab -> Document Types and open the specific type you are interested in.
From the same Configuration section you will find Document Lifecycles, here you can select a lifecycle and see the specific lifecycle states which you can set for the ”status__v” attribute.
Permissions
The available permissions for a specific document are specified by the lifecycle which is attached to that document. To see what available permissions there are, you need to go to the Admin interface -> Configuration -> Document Lifecycles then select the lifecycle and go to the tab Roles. These are the permissions you will be able to set with the Veeva Importer if you attach this particular lifecycle to the document you are importing.
The format is the following:
{Role_name}.{USERS or GROUPS}=username1,username2,…
Example:
reviewer__v.users=user1@vvtechpartner.com,user2@vvtechpartner.com
mc_custom_role__c.groups=approver_group__c
Attribute Mapping for objects fields
In Veeva a Document or an Object can have an attribute (field) that references other Veeva objects.
In Veeva these references are made by using the ID of the referenced object. But when you import with migration-center you need to use the name__v value to find the reference object by default. This is because, in most cases, the ID of the object is unknown during a migration.
To change this behavior and use other attribute values to reference objects, you can configure an attribute mapping file and set it in the attributeMappingLocation parameter.
The format of the mapping in the file depends on whether you are specifying Document or Object fields:
Document fields
Consider you are importing Documents in Veeva that have a field named owning_department__v that references objects of type department__v.
To make the Veeva Importer find Department objects based on the Department Number/Code (number__v) instead of the default name__v, the attribute mapping file must contain the following line:
Now, in migration-center, for the owning_department__v attribute you can assign values that represent the number__v of a Department object and the importer will properly find and reference the object.
The general rule looks like this:
Object fields
When importing Objects instead of Documents, the format is similar, but you need to also specify the object type name in the left side.
Consider you are importing an Object of type example_object__c which has a field named object_field__c that references objects of type ref_example_object.
To make the Veeva find these ref_example_object by their global_id__sys instead of the name, the attribute mapping file needs to contain the following line:
The general rule looks like this:
Note that for the objects you use a dot in the lefthand side as opposed to a comma in the righthand
The new fields used to reference the objects need to be defined as unique in the Object definition otherwise the lookup functionality may not work properly.
Setting an object field that refers to a specific Veeva document version can be done by setting the document id so it shouldn't be specified in the configuration file.
Filtering the Values of the Master Data Objects
The migration-center allows the user to filter the possible values that can be set to a document or object field. It is possible to use multiple fields to get the desired value for the attribute that references a master data object.
In addition to the structure for document or object attributes, the following structure should be added:
The "and" keyword is added in case of usage of multiple fields.
For example, when setting the “country__v” document field, the user may want to set in migration-center the country abbreviation value and he wants to enforce using only "country__v" active objects. In this case, the following line should be added in the configuration file:
If you want to set the "product_type__v" field which refers to a Controlled Vocabulary object when importing Submission objects and you want to ensure that only the controlled vocabulary objects of "Submission Type" are taken into consideration then the following line should be added in the configuration file:
If a picklist attribute is used to filter the reference object records, then the Picklist Value Name must be used for specifying the attribute value.
Attachments
The system rule “attachments” allows the user to import attachments for the documents. It should be set with the fully qualified paths of the attachments.
If a document has two or more attachments with the same name but different content, the importer will create multiple versions for the same attachment.
Delta Migration
Objects that have changed in the source system since the last scan are scanned as update objects. Whether an object in a migration set is an update or not can be seen by checking the value of the Is_update column – if it’s 1, the current object is an update to a previously scanned object (the base object). An update object cannot be imported unless its base object has been imported previously.
Currently, update objects are processed by Veeva importer with the following limitations:
Only document and object updates can be imported. The updates for Binders are reported as errors by the importer
Only metadata is updated for the documents. The content, renditions, permissions and classifications are not updated.
New versions of documents are fully supported
New fields can be added by an update to existing Veeva Objects
External ID
Beside the delta mechanism described above, the importer allows importing new versions to documents based on “external_id__v” system rule. If that’s set, the importer will behave in the following way:
A document with the same external id exists in the vault The importer adds the document being imported as a new version to the existing document. The new version can only be imported if a version with the same major and minor value doesn’t already exists.
A document with the same external id cannot be found in the vault
If the document being imported is a root version (level_in_version_tree = 0) then a new document is created.
If the document being imported is not a root version (level_in_version_tree > 0) then the documents fail to import, and an appropriate error message is logged.
Object names must be unique. Thus, the importer will report an error if an update tries to change the name of the object to an existing one.
