SharePoint Online Batch Importer

Introduction

The SharePoint Online Batch importer allows migrating documents and folders to SharePoint Online and OneDrive. Since OneDrive is based on SharePoint Online, you can use the SharePoint Online Batch importer to import documents and folders to OneDrive as well. If we refer to SharePoint Online in the following, this does apply to OneDrive as well in most cases. We will write appropriate notes in case the behavior of the importers differs for OneDrive.

The SharePoint Online Batch importer offers the following features:

Leverages Microsoft SharePoint Online Import Migration API (bulk import API)
Import documents to Microsoft SharePoint Online Document Library items
Import folders to Microsoft SharePoint Online Document Library items
Import link documents (.url files) to Microsoft SharePoint Online Document Library items
Set values for any columns in SharePoint Online, including user defined columns
Set values for SharePoint Online specific internal field values, i.e. author, editor, time created, time last modified
Set the folder path and create folders if they don’t exist
Set role assignments (permissions) on documents and folders
Import versions
Automatic or manual/custom version numbering
Set moderation/approval status on documents
Set Microsoft Information Protection (MIP) Sensitivity Labels on content files prior to uploading them to SPO
Set compliance (retention) labels on folders and documents

The SharePoint Online Batch importer is implemented mainly as a Job Server component but comes with a separate component for communicating with SharePoint Online (we refer to this component as CSOM Service because it uses the Microsoft CSOM API).

Features

The SharePoint Online Batch importer comes with the following features, limitations, and known issues:

The importer uses the Azure storage containers provided by SharePoint Online.
The importer only supports
- import of documents with metadata (incl. role assignments) and versions,
- import of folders with metadata (incl. role assignments) and
- import of link documents.
The following column / field types are currently supported by the importer:
- User
- Text
- Integer
- Number
- Choice and MultiChoice
- Lookup
- Note
- DateTime
- Boolean
- TaxonomyFieldType
When importing documents or folders, the importer will automatically create any subfolders (of default type) needed below the configured base folder if parameter "autoCreateFolder" is set to "true".
The importer only supports setting managed metadata terms (taxonomies) by their ID in the format "{9ca5bcac-b5b5-445c-8dab-3d1c2181bcf2}".
Import into OneDrive was tested with Azure AD app-only principal authentication.
The importer supports importing files with long paths (i.e. longer than 260 characters), but you must enable this feature in the Windows operating system first (see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation).

Limitations and Known Issues

You can only assign site collection groups to user/group fields and role assignments.
Any invalid XML characters in Text and Note field values will be automatically replaced by a '_' character during import. Otherwise the import would fail with an error.
The target of a migration job must be a SharePoint Site Collection. All objects in an import job must be imported into sites / libraries in the site collection specified in the importer definition.
Delta migration is not supported by the SharePoint Online Migration API and thus this feature is not available with the SharePoint Online Batch Importer. If you need the delta migration feature, please use our SharePoint Online Importer, which supports this feature.
If you set the system rule “declareAsRecord” to “true” for an object, the importer will declare the object as a record if the target library is a Records Library - no matter how the records management is configured for that library. If you set "declareAsRecord" to "false" for an object, the import will not declare the object as record, even if automatic records declaration is enabled for that library.
Due to the asynchronous nature of the SharePoint Migration API
- The progress indicator in the migration-center client works currently only on a per batch level, i.e. if you import only one batch, the progress jumps from 0% to 100% when that batch was imported. Future versions of the importer may provide a more granular progress indicator.
- The migration-center database might not reflect the actual import state of the objects in the migset. For example, it might happen that objects get imported but are still marked as Validated or Error in the migration-center database. For technical details please see Technical Details on the Importer
Due to the per batch functionality of the importer, if a batch contains one or more errors, the entire batch will not be imported and all documents will be set to the error state, even those that do not contain actual errors. In order to identify the objects causing the error please consult the import run logs and the additional logs. Once the problem objects are fixed or removed from the migration set the import will be successful.
If a target site has the Document ID feature enabled: The document ID will be populated for the files imported via the migration API as well – just that it will happen asynchronously. So the DocId might be missing right after the import, but it will be populated within 24hrs (when the backend job runs).
Due to a limitation in the SPO Bulk Import API, the importer allows you to set values for choice/multi-choice fields that are not in the list of available values - even if the "fill-in" option is disabled. If you want to ensure that only allowed values are set, please use a mapping list in the transformation rules for the choice/multi-choice fields.
When trying to import a broken version tree, the import fails for all documents (even the ones that had a correct version tree) with an error status (#57099).
When setting a compliance label on a folder, all children of that folder will get the same label if no compliance label is specified for them.

Install and Uninstall

To install the main product components, consult the migration-center Installation Guide document.

The migration-center SharePoint Online Batch Importer requires installing an additional, separate component besides the main product components. In the following we refer to this component as “SPOnline Batch CSOM Service”.

The SPOnline Batch CSOM Service is necessary to use the SharePoint Online Migration API with migration-center. Its installation and uninstallation are described below.

The app-only principal authentication used by the importer calls the following HTTPS endpoints. Please ensure that the job server machine has access to those endpoints:

<tenant name>.sharepoint.com:443
accounts.accesscontrol.windows.net:443

Prerequisites

The SPOnline Batch CSOM Service is designed to run as a Windows service and needs the .NET Framework 4.7.2 (or later) installed on the migration-center Job Server machine.

Installation Steps

You must install the SPOnline Batch CSOM Service on all machines where you had installed the migration-center server components.

To install SPOnline Batch CSOM Service, it is necessary to run an installation file, which is located within the SharePoint Online Batch Importer component folder of your migration-center Job Server installation location, which is by default C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install. This folder contains the file install.bat, which must be executed with administrative privileges (i.e. “Run as Administrator”).
After the service is installed you will need to start it manually for the first time, after that the service is configured to start automatically as soon as the computer’s operating system is loaded.

When running the CSOM service with a domain account you might need to grant access to the account by running the following command: netsh http add urlacl url=http://+:57097/ user=<your user>

<your user> might be in the format domain\username or username@domain.com

Uninstallation

In case you need to uninstall the SPOnline Batch CSOM Service, please run the file uninstall.bat in C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install with administrative privileges (i.e. “Run as Administrator”).

Before uninstalling the Jobserver component, the CSOM service must be uninstalled as described here.

Working with the Importer

In this chapter you will learn working with the SharePoint Online Batch importer. Specifically, you will see how to import documents with metadata and versions and how to import folders with metadata.

The SharePoint Online Batch importer lets you import documents and folders into a folder of a document library, i.e. you have to configure the SharePoint Online site, document library and the base folder of your import in the importer configuration (see Configuring the Importer). All documents and folders that you import go into the specified base folder or into a folder below.

Preparation for app-only principal authentication

The importer supports app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors: Azure AD app-only principal authentication and SharePoint app-only principal authentication.

Azure AD app-only authentication requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.

If you want to restrict the access of the migration-center application to certain site collections or sites, you can use SharePoint app-only authentication.

Azure AD app-only principal authentication

The migration-center SharePoint Online Batch Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.

Please follow these steps in order to setup your migration-center application in your Azure AD.

The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Configure necessary permissions for the migration-center application

Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:

Microsoft APIs
- SharePoint
  - Application permissions
    Sites
    Sites.FullControl.All
    TermStore
    TermStore.Read.All
    User
    User.Read.All
- Graph
  - Application permissions
    Sites
    Sites.FullControl.All

Click on the blue "Add permissions" button at the bottom to add the permissions to your application. The "Application permissions" are those granted to the migration-center application when running as App Only.

Step 4: Uploading the self-signed certificate

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

Step 5: Grand admin consent

The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.

Final Step: Setting the necessary parameters in the importer

In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the migration-center Azure AD application.

appCertificatePath

The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application.

appCertificatePassword

The password to read the certificate specified in appCertificatePath.

SharePoint app-only principal authentication

SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.

The information in this chapter is based on the following guidelines from Microsoft:

https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint

Step 1: Create a self-signed certificate for your migration-center Azure AD application

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Uploading the self-signed certificate and generate secret key

After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.

Store the retrieved information (client id and client secret) since you'll need this later! Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Step 4: Granting permissions to the app-only principal

Next step is granting permissions to the newly created principal in SharePoint Online.

If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.

If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.

Once the page is loaded add your client id and look up the created principal:

Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.

To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

Use this permission XML for granting site collection scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:

Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Final Step: Setting the necessary parameters in the importer

In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the SharePoint application you have created.

appClientSecret

The client secret, which you have generated when setting up the SharePoint application.

Preparation for applying Microsoft Information Protection sensitivity labels

Starting with v3.16, our SharePoint Online Batch importer supports applying Microsoft Information Protection (MIP) sensitivity labels on the content files before they get uploaded to SharePoint Online. To learn more about sensitivity labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide.

Register an Azure AD app

The MIP software development kit provided by Microsoft that we use to apply the sensitivity labels on the content files requires you to register an application in your Azure AD. You could use the same app as in Azure AD app-only principal authentication or setup a separate application as described here: https://docs.microsoft.com/en-us/information-protection/develop/setup-configure-mip#register-a-client-application-with-azure-active-directory

You need to create a client secret for your registered application and configure the following access permissions:

Install VC++ Runtime

The Microsoft MIP SDK also needs the VC++ runtime installed on the job server machine to work properly. You might get a LoadLibrary failed for: [C:\fme AG\migration-center Server Components 3.15\lib\mc-sharepointonline-importer\MIPService\x86\mip_dotnet.dll] error if the VC++ runtime is missing and you want to apply sensitivity labels on your documents.

For your convenience, we provide the VC++ runtime installers in the lib\mc-sharepointonline-importer\MIPService\VC_Redistributables folder of your MC job server installation. Which variant to install (i.e. x86 or x64) depends on your .NET configuration. Usually you should install the x86 VC++ runtime (even on a x64 system). If that does not work, you should install the x64 variant.

Create the configuration file for the importer

The SharePoint Online Batch importer requires you to provide several configuration parameters in a XML configuration file. You can find a template configuration file (template-mip.config) in the lib\mc-spo-batch-importer folder of your migration-center job server installation.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="<YOUR AZURE APP ID>" />
		<add key="Azure.App.Name" value="<YOUR AZURE APP NAME>"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="<YOUR ENCRYPTED AZURE APP SECRET>"/>
		<add key="Azure.App.RedirectUri" value="<YOUR AZURE APP REDIRECT URL>"/>
		<add key="Azure.Tenant.Domain" value="<YOUR AZURE TENANT DOMAIN>"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="<OPTIONAL PATH TO LABELED CONTENT FOLDER>"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

Create a copy of the template file, fill in the required information from your Azure AD app that you had created in the previous step, and enter the full path to your config file in the mipConfigFile parameter of your SharePoint Online Batch importer. The result should look similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="91583618-eb95-4ae9-8f9e-c5259491f66d" />
		<add key="Azure.App.Name" value="miptest"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="0;#75+wbxpdQ0c=;#pp39p7v4QA57VKiP0NNcnuX/FEk2FiE8iD4PrJnysfLuDbNra2xsFHA2K9wAKG8z"/>
		<add key="Azure.App.RedirectUri" value="miptest://authorize"/>
		<add key="Azure.Tenant.Domain" value="migrationcenter.onmicrosoft.com"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="C:\Temp\sensitivity labeled docs"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

We provide a Java based command line tool for encryption, which you must use to encrypt the Azure.App.Secret value before you put it in the configuration file. To encrypt the secret, proceed as follows:

Open a command line window on the job server machine.
Switch to the migration-center Server Components <version>\lib\mc-core folder, e.g. migration-center Server Components 3.16\lib\mc-core
Run the following command: java -cp mc-common-<version>.jar de.fme.mc.common.encryption.CmdTool (of course replace <version> with the appropriate version number of your migration-center)
The tool will ask for the text to encrypt. Enter or paste the secret value and press enter.
The tool will output the encrypted text. Copy and paste it into the configuration file.

Importing Documents

In order to import documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(document)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(document)” has the following system attributes, which primarily determine the content type, name and location of the documents.

You can find a detailed description of the available system rules in the table below:

Configuration parameters

Values

mc_content_location

Rule for setting alternative content of the document, i.e. if you want to upload a different content file than the scanned one.

Example: D:\Some\Alternative\Content\Path\Content.pdf

Optional

If you leave this rule empty, the importer will import the scanned content along with the document. In case you want to upload a different content, e.g. maybe you converted a PDF file into PDF-A format, use this rule to specify the location of the new content file.

s__complianceTag

Rule for setting the compliance tag for the folder.

Example: INVOICE

Optional

See Setting Compliance Labels for details

s__contentType*

Rule for setting the SharePoint content type of the document.

Example: Document

Mandatory

This value must match an existing migration-center object type definition and of course a content type in the target SharePoint document library.

s__createdBy

Name of the user who created the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__createdDate

Date and time when the document was created.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__declareAsRecord

Flag indicating if the document should be declared as a record.

Example: True

Optional

Record declaration will only work if

SharePoint Online is the target (and not OneDrive)
the target library is a Records Library and
manual record declaration is enabled in the Records Library.

s__library*

Specify the title of the target library. Example: Documents

Mandatory

Note: For import into a personal OneDrive use Documents as library

s__moderationStatus

Sets the moderation / approval status of the document. Must be one of the following values (you can either use the status names or their numerical value): Approved (or 0), Denied (or 1), Draft (or 3), Pending (or 2), Scheduled (or 4)

Optional

s__moderationStatusComment

Sets the moderation / approval status comment of the document.

Optional

s__modifiedBy

Name of the user who last modified the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__modifiedDate

Date and time when the document was last modified.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__name*

The name of the document including its file extension.

Example: My Document.docx

Mandatory

s__parentFolder*

The parent folder of the document.

Example: /My subfolder

Mandatory

The parent folder is relative to the base folder configured in the importer definition, e.g. if the base folder is /Shared Documents and the parent folder is /My subfolder, the effective folder for the document will be /Shared Documents/My subfolder

s__roleAssignments

Specify role assignments for the current object. If a role definition is assigned to the current object, the Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint group exists in your SharePoint site. To set role assignments for single users, the value must match the following pattern: <loginname>;#<roledefinitionname> To set role assignments for SharePoint groups, the value must match the following pattern: @<groupname>;#<roledefinitionname> Example: user;#Read @Contributors;#Contribute

Optional

s__sensitivityLabel

Name of the sensitivity label to apply on the content file of the document.

Example: Confidential

Optional

Note: When applying sensitivity labels, you also need to specify the path to an additional MIP configuration file. See Preparation for applying Microsoft Information Protection sensitivity labels for more details.

s__site*

Specify the target (sub-)site for the document.

Example: My Site or My Site/My SubSite

Mandatory

Note: For import into a personal OneDrive use /personal/<your personal site> as site, e.g. /personal/donald_duck_onmicrosoft_com

s__versionNumber

Sets the version number of the document. Must be in the format: x.y Example: 1.0

Mandatory if importer parameter "autoVersioning" is set to "false".

Specified value will be ignored if parameter "autoVersioning" is set to "true".

Usually you want to set additional custom attributes on the documents. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Importing Link Documents

SharePoint Online supports documents with the file format ".url". We call these documents "link documents" since they do not have real content but just point to another document in SharePoint Online or to an external web site. The importer will create the necessary content of the link documents on the fly and save them in the following folder: <mc log folder>/SPOnline Batch Importer/<job run id>/link_files. Thus, the importer will ignore the content of any source object that is imported as a link document because it will import the generated link content instead.

In order to import link documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(link)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(link)” has the following system attributes, which primarily determine the content type, name and URL of the linked documents or web site respectively.

You can find a detailed description of the available system rules in the table below:

Configuration parameters

Values

s__contentType*

Rule for setting the SharePoint content type of the link document. This should be the default "Document" content type.

Example: Document

Mandatory

This value must match an existing migration-center object type definition and of course a content type in the target SharePoint document library.

s__createdBy

Name of the user who created the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__createdDate

Date and time when the document was created.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__library*

Specify the title of the target library. Example: Documents

Mandatory

Note: For import into a personal OneDrive use Documents as library

s__modifiedBy

Name of the user who last modified the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__modifiedDate

Date and time when the document was last modified.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__name*

The name of the link document including its ".url" file extension. It is important that you add the ".url" file extension to the file name. Otherwise SharePoint Online will not handle the document as a link.

Example: My Document.docx.url

Mandatory

s__parentFolder*

The parent folder of the link document.

Example: /My subfolder

Mandatory

s__roleAssignments

Optional

s__site*

Specify the target (sub-)site for the document.

Example: My Site or My Site/My SubSite

Mandatory

Note: For import into a personal OneDrive use /personal/<your personal site> as site, e.g. /personal/donald_duck_onmicrosoft_com

s__url*

The target URL of this link document. Can be a URL to another SharePoint Online document or a URL to an external web site.

Example:

https://contoso.sharepoint.com/sites/MySite/Documents/Test.docx

Mandatory

s__versionNumber

Sets the version number of the link document. Must be in the format: x.y Example: 1.0

Mandatory if importer parameter "autoVersioning" is set to "false".

Specified value will be ignored if parameter "autoVersioning" is set to "true".

You can set additional custom attributes on the link documents. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Importing Folders

In order to import folders, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(folder)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules.

A migration set with a target type of “SPOnlineBatch(folder)” has the following system attributes, which primarily determine the content type, name and location of the folders.

You can find a detailed description of the available system rules in the table below:

Configuration parameters

Values

s__complianceTag

Rule for setting the compliance tag for the folder.

Example: INVOICE

Optional

See Setting Compliance Labels for details

s__contentType*

Rule for setting the SharePoint content type of the folder.

Example: Folder

Mandatory

This value must match an existing migration-center object type definition and of course a content type in the target SharePoint document library.

s__createdBy

Name of the user who created the folder. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__createdDate

Date and time when the folder was created.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__library*

Specify the title of the target library. Example: Documents

Mandatory

Note: For import into a personal OneDrive use Documents as library

s__modifiedBy

Name of the user who last modified the folder. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__modifiedDate

Date and time when the folder was last modified.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__name*

The name of the folder.

Example: My Folder

Mandatory

s__parentFolder*

The parent folder of the folder.

Example: /My subfolder

Mandatory

The parent folder is relative to the base folder configured in the importer definition, e.g. if the base folder is /Shared Documents and the parent folder is /My subfolder, the effective folder path of the folder will be /Shared Documents/My subfolder/My Folder

s__roleAssignments

Optional

s__site*

Specify the target (sub-)site for the document.

Example: My Site or My Site/My SubSite

Mandatory

Note: For import into a personal OneDrive use /personal/<your personal site> as site, e.g. /personal/donald_duck_onmicrosoft_com

Usually you want to set additional custom attributes on the folders. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Configuring the Importer

To create a new SharePoint Online Batch Importer, go to -Importers- and press the [New] button. Then select as

An importer of type SPOnline Batch has the following parameters, which you need to set with the appropriate values:

Configuration parameters

Values

tenantName*

The name of your SharePoint Online Tenant

Example: Contoso

Mandatory

There are several web site that explain how to determine a SharePoint Online tenant name, e.g. https://morgantechspace.com/2019/07/how-to-find-your-tenant-name-in-office-365.html

siteCollectionUrl*

The URL of your target site collection.

Examples: https://contoso.sharepoint.com/sites/MySite https://contoso-my.sharepoint.com (for OneDrive)

Mandatory

appClientId*

The ID of either the migration-center Azure AD application or the SharePoint application.

Example: ab187da0-c04d-4f82-9f43-51f41c0a3bf0

Mandatory

See Preparation for app-only principal authentication

appCertificatePath

The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application.

Example: D:\migration-center\config\azure-ad-app-cert.pfx

Mandatory

See Preparation for app-only principal authentication

appCertificatePassword

The password to read the certificate specified in appCertificatePath.

Mandatory

See Preparation for app-only principal authentication

appClientSecret

The client secret, which you have generated when setting up the SharePoint application (SharePoint app-only principal authentication).

Mandatory

See Preparation for app-only principal authentication

proxyServer

This is the URL, which defines the location of your proxy to connect to the Internet.

Optional

Just leave blank if no proxy is used.

proxyUsername

The login of the user, who connects and authenticates on the proxy, which was specified in parameter proxyServer.

Example: corporatedomain\username

Optional

Just leave blank if no proxy is used.

proxyPassword

Password of the proxy user specified above.

Optional

Just leave blank if no proxy is used.

autoCreateFolder*

Enable/disable whether folders, which do not exist, should be created during import.

Enabled: The importer creates any specified folders automatically.

Disabled: No new folders will be created, but any existing folders (same path and name) are used. References to non-existing folders will throw an error.

autoVersioning*

Enable/disable whether version numbers should be generated automatically by the importer.

Enabled: The importer will generate consecutive version numbers for the versions of a document based on the versioning setting of the target library.

Disabled: The user must provide appropriate version numbers for the documents using the "s__versionNumber" system attribute.

mipConfigFile

Full path to the Microsoft Information Protection (MIP) configuration file, which contains additional settings for setting sensitivity labels on the content files to import.

Example: C:\migration-center\mip.config

Optional

See Preparation for applying Microsoft Information Protection sensitivity labels for more details about the MIP configuration file.

numberOfThreads*

Number of batches that shall be imported in parallel.

Default: 8

Mandatory

You may increase this value for large imports, depending on the configuration of the machine that runs the job server.

loggingLevel*

Sets the verbosity of the log file.

Allowed values:

1 - logs only errors during scan

2 - is the default value reporting all warnings and errors

3- logs all successfully performed operations in addition to any warnings or errors

4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production)

Mandatory

Tracing imported objects

The SharePoint Online Batch Importer will store the following information for imported objects in the MC database, which will allow you to trace an imported object back to the corresponding source object.

Attribute in MC database

Value

ID_IN_TARGET_SYSTEM

String that consists of the following components:

<list item id>|<library guid>[|<version number>]

The version number component is optional. These information will allow you to uniquely identify the imported object in the target system.

Example values:

387088|d69e991b-d488-49bb-b739-858669e60b19|1.0

384956|d69e991b-d488-49bb-b739-858669e60b19

OBJECT_INFO1

Server relative URL to the object.

Example values:

/teams/bu6/TestLibrary/Test

/teams/bu6/TestLibrary/Test/Document.txt

Note: OBJECT_INFO1 is an internal column in the SOURCE_OBJECTS table and not visible in the MC client.

Please note that the URLs in the OBJECT_INFO1 attribute will only be valid for the latest version in a version tree because SharePoint uses different URLs for latest versions and old versions. For example, if the document from the example above had two versions 1.0 and 2.0, both values in OBJECT_INFO1 would be: /teams/bu6/TestLibrary/Test/Document.txt But the correct URL for version 1.0 would be: /teams/bu6/TestLibrary/_vti_history/512/Test/Document.txt

Setting Compliance Labels

Compliance labels are managed in the MS 365 compliance center, which you can access with the following URL: https://compliance.microsoft.com/

For more details about creating and publishing compliance labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/create-apply-retention-labels?view=o365-worldwide

You can assign any label that is published for the target SharePoint site to the folders and documents in a migration set. The migration-center provides a new system rule for setting compliance labels:

s__complianceTag

If you want to set a label on a document or folder, it is mandatory to specify the label name in the "s__complianceTag" system rule.

The importer will check before importing the object if the specified label is available in the target site and throw an error if it is not in the list of available labels.

Technical Details on the Importer

This chapter will give you an overview of the importer’s operating principle.

Overview

Each SharePoint Online Batch Importer job will go through the following steps:

Split the list of objects to import into batches of approximately 250 objects.
For each batch repeat
1. Generate the XML files (e.g. Manifest.xml etc.) necessary for submitting the batch to the SharePoint Migration API.
2. Upload XML files and content files of the batch to an Azure BLOB container storage.
3. Submit the import batch to the SharePoint platform as a migration job.
4. Monitor the progress of the migration job.
5. When the migration job has finished, verify that all objects submitted for import were successfully imported by retrieving them by ID and save the result of the verification back to the migration-center database.

Depending on your job configuration, several batches are imported in parallel (see “numberOfThreads” parameter in the import job configuration).

Splitting into batches

The importer will split the list of documents or folders to import into batches of approximately 250 items using the following rules

All objects must have the same target site & library.
All versions of an item must be contained in the same batch.
A batch should contain a maximum of 250 items.
The content size of the items in a batch should not exceed 250 MB

The second rule can lead to batches with more than 250 objects. The first or last rule can lead to batches with less than 250 objects.

XML files

The SharePoint Import Migration API requires the following XML files for each import batch:

ExportSettings.XML
LookupListMap.XML
Manifest.XML
Requirements.XML
RootObjectMap.XML
SystemData.XML
UserGroupMap.XML
ViewFormsList.XML

For more details on those files, please see https://docs.microsoft.com/en-us/sharepoint/dev/apis/migration-api-overview#import-package-structure

Additional Logs

Additional log files generated by the SharePoint Online Batch Importer can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\SPOnline Batch Importer\<job run id>

You can find the following files in the <job run id> folder:

The import-job.log contains detailed information about the job run and can be used to investigate import issues.
The generated manifest files for each batch in the <batch number>/manifest sub-folder.
The log files generated by SharePoint Online for each batch in the <batch-number>/spo-logs sub-folder.
The content files generated for link documents in the <batch number>/link_files sub-folder.

Last updated 2 years ago