SharePoint Online Batch Importer

Introduction

The SharePoint Online Batch importer allows migrating documents and folders to SharePoint Online and OneDrive. Since OneDrive is based on SharePoint Online, you can use the SharePoint Online Batch importer to import documents and folders to OneDrive as well. If we refer to SharePoint Online in the following, this does apply to OneDrive as well in most cases. We will write appropriate notes in case the behavior of the importers differs for OneDrive.

The SharePoint Online Batch importer offers the following features:

Leverages Microsoft SharePoint Online Import Migration API (bulk import API)
Import documents to Microsoft SharePoint Online Document Library items
Import folders to Microsoft SharePoint Online Document Library items
Import link documents (.url files) to Microsoft SharePoint Online Document Library items
Set values for any columns in SharePoint Online, including user defined columns
Set values for SharePoint Online specific internal field values, i.e. author, editor, time created, time last modified
Set the folder path and create folders if they don’t exist
Set role assignments (permissions) on documents and folders
Import versions
Automatic or manual/custom version numbering
Set moderation/approval status on documents
Set Microsoft Information Protection (MIP) Sensitivity Labels on content files prior to uploading them to SPO
Set compliance (retention) labels on folders and documents

The SharePoint Online Batch importer is implemented mainly as a Job Server component but comes with a separate component for communicating with SharePoint Online (we refer to this component as CSOM Service because it uses the Microsoft CSOM API).

Features

The SharePoint Online Batch importer comes with the following features

The importer uses the Azure storage containers provided by SharePoint Online.
The importer only supports
- import of documents with metadata (incl. role assignments) and versions,
- import of folders with metadata (incl. role assignments) and
- import of link documents.
The following column / field types are currently supported by the importer:
- User
- Text
- Integer
- Number
- Choice and MultiChoice
- Lookup
- Note
- DateTime
- Boolean
- TaxonomyFieldType
When importing documents or folders, the importer will automatically create any subfolders (of default type) needed below the configured base folder if parameter "autoCreateFolder" is set to "true".
The importer only supports setting managed metadata terms (taxonomies) by their ID in the format "{9ca5bcac-b5b5-445c-8dab-3d1c2181bcf2}".
Import into OneDrive was tested with Azure AD app-only principal authentication.
The importer supports importing files with long paths (i.e. longer than 260 characters), but you must enable this feature in the Windows operating system first (see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation).

Known Issues and Limitations

Batch limitations

The importer will split the list of documents or folders to import into batches of approximately 250 items using the following rules

All objects must have the same target site & library.
All versions of an item must be contained in the same batch.
A batch should contain a maximum of 250 items.
The content size of the items in a batch should not exceed 250 MB

The second rule can lead to batches with more than 250 objects. The first or last rule can lead to batches with less than 250 objects.

Other issues and limitations

You can only assign site collection groups to user/group fields and role assignments.
Any invalid XML characters in Text and Note field values will be automatically replaced by a '_' character during import. Otherwise the import would fail with an error.
The target of a migration job must be a SharePoint Site Collection. All objects in an import job must be imported into sites / libraries in the site collection specified in the importer definition.
Delta migration is not supported by the SharePoint Online Migration API and thus this feature is not available with the SharePoint Online Batch Importer. If you need the delta migration feature, please use our SharePoint Online Importer, which supports this feature.
If you set the system rule “declareAsRecord” to “true” for an object, the importer will declare the object as a record if the target library is a Records Library - no matter how the records management is configured for that library. If you set "declareAsRecord" to "false" for an object, the import will not declare the object as record, even if automatic records declaration is enabled for that library.
Due to the asynchronous nature of the SharePoint Migration API
- The progress indicator in the migration-center client works currently only on a per batch level, i.e. if you import only one batch, the progress jumps from 0% to 100% when that batch was imported. Future versions of the importer may provide a more granular progress indicator.
- The migration-center database might not reflect the actual import state of the objects in the migset. For example, it might happen that objects get imported but are still marked as Validated or Error in the migration-center database. For technical details please see Technical Details on the Importer
Due to the per batch functionality of the importer, if a batch contains one or more errors, the entire batch will not be imported and all documents will be set to the error state, even those that do not contain actual errors. In order to identify the objects causing the error please consult the import run logs and the additional logs. Once the problem objects are fixed or removed from the migration set the import will be successful.
If a target site has the Document ID feature enabled: The document ID will be populated for the files imported via the migration API as well – just that it will happen asynchronously. So the DocId might be missing right after the import, but it will be populated within 24hrs (when the backend job runs).
Due to a limitation in the SPO Bulk Import API, the importer allows you to set values for choice/multi-choice fields that are not in the list of available values - even if the "fill-in" option is disabled. If you want to ensure that only allowed values are set, please use a mapping list in the transformation rules for the choice/multi-choice fields.
When trying to import a broken version tree, the import fails for all documents (even the ones that had a correct version tree) with an error status (#57099).
When setting a compliance label on a folder, all children of that folder will get the same label if no compliance label is specified for them.
When importing objects with mandatory attributes in SPO the objects will be imported and no error will be thrown even when the mandatory attributes were not set in migration-center. This is important to note as the SharePoint Online Importer, which needs to be used for delta migrations, will throw errors on import if mandatory attributes are not set.
- One way to work around this limitation is to set these attributes as mandatory in the Object Type created inside migration-center. This way any empty mandatory attributes will throw errors during the Validation step.
Users cannot be set to custom "Person or Group" attributes (but Group names can be set). System attributes like Author or Editor can contains Users.
The moderation comment cannot be set, even though it is configured in the corresponding system rule. This is a Microsoft issue in the SharePoint Migration API

Prerequisites

CSOM service

The migration-center SharePoint Online Batch Importer requires installing an additional component.

This additional component needs the .NET Framework 4.7.2 installed and it’s designed to run as a Windows service and must be installed on all machines where the a Job Server is installed.

To install this additional component, it is necessary to run an installation file, which is located within the SharePoint folder in the Jobserver instal: ...\lib\mc-spo-batch-importer\CSOM_Service\install

To install the service run the install.bat file using administrative privileges. You need to start it manually after the install. Afterwards the service is configured to start automatically at system startup.

The CSOM service must be run with the same user as the Job Server service so that it has the same access to the export location.

When running the CSOM service with a domain account you might need to grant access to the account by running the following command: netsh http add urlacl url=http://+:57097/ user=<your user>

<your user> might be in the format domain\username or username@domain.com

To uninstall the service run the uninstall.bat file using administrative privileges.

Before uninstalling the Jobserver component, the CSOM service must be uninstalled as described here.

Port access

The app-only principal authentication used by the importer calls the following HTTPS endpoints. Please ensure that the job server machine has access to those endpoints:

<tenant name>.sharepoint.com:443
accounts.accesscontrol.windows.net:443

Authentication

The importer supports only app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors:

Azure AD app-only principal authentication Requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.
SharePoint app-only principal authentication Can be set to restrict the access of the migration-center application to certain site collections or sites.

Azure AD app-only

The migration-center SharePoint Online Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.

Please follow these steps in order to setup your migration-center application in your Azure AD.

The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Configure necessary permissions for the migration-center application

Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:

Microsoft APIs
- SharePoint
  - Application permissions
    Sites
    Sites.FullControl.All
    TermStore
    TermStore.Read.All
    User
    User.Read.All
- Graph
  - Application permissions
    Sites
    Sites.FullControl.All

Click on the blue "Add permissions" button at the bottom to add the permissions to your application. The "Application permissions" are those granted to the migration-center application when running as App Only.

Step 4: Uploading the self-signed certificate

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.

Step 6: Setting the necessary parameters in the importer

In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the migration-center Azure AD application.

appCertificatePath

The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application.

appCertificatePassword

The password to read the certificate specified in appCertificatePath.

SharePoint app-only

SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.

The information in this chapter is based on the following guidelines from Microsoft: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint

Step 1: Create a self-signed certificate for your migration-center Azure AD application

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Uploading the self-signed certificate and generate secret key

After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.

Store the retrieved information (client id and client secret) since you'll need this later! Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Step 4: Granting permissions to the app-only principal

Next step is granting permissions to the newly created principal in SharePoint Online.

If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.

If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.

Once the page is loaded add your client id and look up the created principal by pressing the "Lookup" button:

Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.

To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true">
 <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
 <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" />
</AppPermissionRequests>

Use this permission XML for granting site collection scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true">
 <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
 <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" />
</AppPermissionRequests>

When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:

Please safeguard the created client id/secret combination as would it be your administrator account. Using these one can read/update all data in your SharePoint Online environment!

Step 5: Setting the necessary parameters in the importer

In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the SharePoint application you have created.

appClientSecret

The client secret, which you have generated when setting up the SharePoint application.

Importer Configuration

To create a new SharePoint Online Batch Importer, go to -Importers- and press the [New] button. Then select the Adapter Type "SPOnline Batch".

Importer parameters

The common adaptor parameters are described in Common Parameters.

The configuration parameters available for the SharePoint Online Batch Importer are described below:

tenantName* The name of your SharePoint Online Tenant Example: Contoso
siteCollectionUrl* The URL of your target site collection. See SubSite vs SiteCollection for details. Examples: https://contoso.sharepoint.com/sites/MySite https://contoso-my.sharepoint.com (for OneDrive)
appClientId* The ID of either the migration-center Azure AD application or the SharePoint application. Example: ab187da0-c04d-4f82-9f43-51f41c0a3bf0 See: Authentication.
appCertificatePath* The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application. Example: D:\migration-center\config\azure-ad-app-cert.pfx See: Authentication.
appCertificatePassword The password to read the certificate specified in appCertificatePath. See: Authentication.
appClientSecret The client secret, which you have generated when setting up the SharePoint application. See: Authentication.
proxyServer This is the URL, which defines the location of your proxy to connect to the Internet. Example. http://proxy.local:3128
proxyUsername The login of the user, who connects and authenticates on the proxy, which was specified in parameter proxyServer. Example: corporatedomain\username
proxyPassword Password of the proxy user specified above.
autoCreateFolder* If set the importer will automatically create the folder strcture needed for importing the object.
autoVersioning* Sets whether version numbers should be generated automatically by the importer. Enabled: The importer will generate consecutive version numbers for the versions of a document based on the versioning setting of the target library. Disabled: The user must provide appropriate version numbers for the documents using the "s__versionNumber" system attribute.
mipConfigFile Full path to the Microsoft Information Protection (MIP) configuration file, which contains additional settings for setting sensitivity labels on the content files to import. Example: C:\migration-center\mip.config See: MIP sensitivity labels
numberOfThreads* Number of batches that shall be imported in parallel. Default: 8
loggingLevel* See: Common Parameters.

Parameters marked with an asterisk (*) are mandatory.

Migset System Rules

Documents

In order to import documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(document)”.

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(document)” has the following system attributes, which primarily determine the content type, name and location of the documents.

mc_content_location Specify the location of a document’s content file. If not set, the default content location (i.e. where the document has been scanned from) will be used automatically. Set a different path (including filename) if the content has moved since it was scanned, or if content should be substituted with another file.
s__complianceTag Rule for setting the compliance tag for the folder. Example: INVOICE See: Compliance Labels.
s__contentType* Rule for setting the SharePoint content type of the document. This value must match a content type in the target SharePoint document library.
s__createdBy Name of the user who created the document. The value must be a unique username. Example: MikeGration@asdf.onmicrosoft.com
s__createdDate Sets the date and time when the document was created.
s__declareAsRecord Flag indicating if the document should be declared as a record. Record declaration will only work if SharePoint Online is the target (and not OneDrive), the target library is a Records Library and manual record declaration is enabled in the library.
s__library* Specify the title of the target library. Note: For import into a personal OneDrive use Documents as library
s__moderationStatus Sets the moderation / approval status of the document. Must be one of the following values (either the status names or their numerical value): 0 or Approved
1 or Denied 2 or Pending 3 or Draft
4 or Scheduled
s__moderationStatusComment Sets the moderation / approval status comment of the document.
s__modifiedBy Name of the user who last modified the document. The value must be a unique username. Example: MikeGration@asdf.onmicrosoft.com
s__modifiedDate Date and time when the document was last modified.
s__name* The name of the document including its file extension. Example: My Document.docx
s__parentFolder* The parent folder of the document. Example: /folder/subfolder
s__roleAssignments Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. Pattern for setting single users: username;#roledefinitionname Pattern for setting groups: @groupName;#roledefinitionname Example: user;#Read @Contributors;#Contribute
s__sensitivityLabel Name of the sensitivity label to apply on the content file of the document.
Example: Confidential
Note: When applying sensitivity labels, you also need to specify the path to an additional MIP configuration file. See: MIP sensitivity labels.
s__site* Specify the target (sub-)site for the document.
Example: My Site or My Site/My SubSite
For OneDrive: use /personal/<your personal site> Example: /personal/donald_duck_onmicrosoft_com
s__versionNumber Sets the version number of the document. Must be in the format: x.y Example: 1.0
Mandatory if importer parameter "autoVersioning" is set to "false".
Specified value will be ignored if parameter "autoVersioning" is set to "true".
target_type* Rule for setting the association with a migration-center Object Type. Validation will be done based on this system rule.

Links

SharePoint Online supports documents with the file format ".url". We call these documents "link documents" since they do not have real content but just point to another document in SharePoint Online or to an external web site. The importer will create the necessary content of the link documents on the fly and save them in the following folder: <mc log folder>/SPOnline Batch Importer/<job run id>/link_files. Thus, the importer will ignore the content of any source object that is imported as a link document because it will import the generated link content instead.

In order to import link documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(link)”.

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(link)” has the following system attributes, which primarily determine the content type, name and URL of the linked documents or web site respectively.

Compared to the Documents migset, the Link migset is missing the mc_content_location, s__complianceTag, s__declareAsRecord, s__moderationStatus, s__moderationStatusComment, s__sensitivityLabel system rules and has the following new rules:

s__url* The target URL of this link document. Can be a URL to another SharePoint document or a URL to an external web site.
Example:
https://contoso.sharepoint.com/sites/MySite/Documents/Test.docx

Folders

In order to import folders, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(folder)”.

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules.

A migration set with a target type of “SPOnlineBatch(folder)” has the following system attributes, which primarily determine the content type, name and location of the folders.

Compared to the Documents migset, the Folder migset is missing the mc_content_location, s__declareAsRecord, s__moderationStatus, s__moderationStatusComment, s__sensitivityLabel and s__versionNumber system rules, and has no additional ones.

MIP sensitivity labels

The SharePoint Online importer supports applying Microsoft Information Protection (MIP) sensitivity labels on the content files before they get uploaded to SharePoint Online. To learn more about sensitivity labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide.

Register an Azure AD app

The MIP software development kit provided by Microsoft that we use to apply the sensitivity labels on the content files requires you to register an application in your Azure AD. You could use the same app as in Azure AD app-only principal authentication or setup a separate application as described here: https://docs.microsoft.com/en-us/information-protection/develop/setup-configure-mip#register-a-client-application-with-azure-active-directory

You need to create a client secret for your registered application and configure the following access permissions:

Install VC++ Runtime

The Microsoft MIP SDK also needs the VC++ runtime installed on the job server machine to work properly. You might get a LoadLibrary failed for: [C:\fme AG\migration-center Server Components 3.15\lib\mc-sharepointonline-importer\MIPService\x86\mip_dotnet.dll] error if the VC++ runtime is missing and you want to apply sensitivity labels on your documents.

For your convenience, we provide the VC++ runtime installers in the lib\mc-sharepointonline-importer\MIPService\VC_Redistributables folder of your MC job server installation. Which variant to install (i.e. x86 or x64) depends on your .NET configuration. Usually you should install the x86 VC++ runtime (even on a x64 system). If that does not work, you should install the x64 variant.

Create the configuration file for the importer

The SharePoint Online Batch importer requires you to provide several configuration parameters in a XML configuration file. You can find a template configuration file (template-mip.config) in the lib\mc-spo-batch-importer folder of your migration-center job server installation.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="<YOUR AZURE APP ID>" />
		<add key="Azure.App.Name" value="<YOUR AZURE APP NAME>"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="<YOUR ENCRYPTED AZURE APP SECRET>"/>
		<add key="Azure.App.RedirectUri" value="<YOUR AZURE APP REDIRECT URL>"/>
		<add key="Azure.Tenant.Domain" value="<YOUR AZURE TENANT DOMAIN>"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="<OPTIONAL PATH TO LABELED CONTENT FOLDER>"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

Create a copy of the template file, fill in the required information from your Azure AD app that you had created in the previous step, and enter the full path to your config file in the mipConfigFile parameter of your SharePoint Online Batch importer. The result should look similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="91583618-eb95-4ae9-8f9e-c5259491f66d" />
		<add key="Azure.App.Name" value="miptest"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="0;#75+wbxpdQ0c=;#pp39p7v4QA57VKiP0NNcnuX/FEk2FiE8iD4PrJnysfLuDbNra2xsFHA2K9wAKG8z"/>
		<add key="Azure.App.RedirectUri" value="miptest://authorize"/>
		<add key="Azure.Tenant.Domain" value="migrationcenter.onmicrosoft.com"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="C:\Temp\sensitivity labeled docs"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

We provide a Java based command line tool for encryption, which you must use to encrypt the Azure.App.Secret value before you put it in the configuration file. To encrypt the secret, proceed as follows:

Open a command line window on the job server machine.
Switch to the migration-center Server Components <version>\lib\mc-core folder, e.g. migration-center Server Components 3.16\lib\mc-core
Run the following command: java -cp mc-common-<version>.jar de.fme.mc.common.encryption.CmdTool
The tool will ask for the text to encrypt. Enter or paste the secret value and press enter.
The tool will output the encrypted text. Copy and paste it into the configuration file.

Compliance Labels

Compliance labels are managed in the MS 365 compliance center, which you can access with the following URL: https://compliance.microsoft.com/

For more details about creating and publishing compliance labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/create-apply-retention-labels?view=o365-worldwide

You can assign any label that is published for the target SharePoint site to the folders and documents in a migration set. The migration-center provides a new system rule for setting compliance labels:

s__complianceTag

If you want to set a label on a document or folder, it is mandatory to specify the label name in the "s__complianceTag" system rule.

The importer will check before importing the object if the specified label is available in the target site and throw an error if it is not in the list of available labels.

Technical Details on the Importer

This chapter will give you an overview of the importer’s operating principle.

Overview

Each SharePoint Online Batch Importer job will go through the following steps:

Split the list of objects to import into batches of approximately 250 objects.
For each batch repeat
1. Generate the XML files (e.g. Manifest.xml etc.) necessary for submitting the batch to the SharePoint Migration API.
2. Upload XML files and content files of the batch to an Azure BLOB container storage.
3. Submit the import batch to the SharePoint platform as a migration job.
4. Monitor the progress of the migration job.
5. When the migration job has finished, verify that all objects submitted for import were successfully imported by retrieving them by ID and save the result of the verification back to the migration-center database.

Depending on your job configuration, several batches are imported in parallel (see “numberOfThreads” parameter in the import job configuration).

Tracing imported objects

The SharePoint Online Batch Importer will store the following information for imported objects in the MC database, which will allow you to trace an imported object back to the corresponding source object.

Attribute in MC database

Value

ID_IN_TARGET_SYSTEM

String that consists of the following components:

<list item id>|<library guid>[|<version number>]

The version number component is optional. These information will allow you to uniquely identify the imported object in the target system.

Example values:

387088|d69e991b-d488-49bb-b739-858669e60b19|1.0

384956|d69e991b-d488-49bb-b739-858669e60b19

OBJECT_INFO1

Server relative URL to the object.

Example values:

/teams/bu6/TestLibrary/Test

/teams/bu6/TestLibrary/Test/Document.txt

Note: OBJECT_INFO1 is an internal column in the SOURCE_OBJECTS table and not visible in the MC client.

Please note that the URLs in the OBJECT_INFO1 attribute will only be valid for the latest version in a version tree because SharePoint uses different URLs for latest versions and old versions. For example, if the document from the example above had two versions 1.0 and 2.0, both values in OBJECT_INFO1 would be: /teams/bu6/TestLibrary/Test/Document.txt But the correct URL for version 1.0 would be: /teams/bu6/TestLibrary/_vti_history/512/Test/Document.txt

XML files

The SharePoint Import Migration API requires the following XML files for each import batch:

ExportSettings.XML
LookupListMap.XML
Manifest.XML
Requirements.XML
RootObjectMap.XML
SystemData.XML
UserGroupMap.XML
ViewFormsList.XML

For more details on those files, please see https://docs.microsoft.com/en-us/sharepoint/dev/apis/migration-api-overview#import-package-structure

SubSite vs SiteCollection

The siteCollectionUrl parameter in the importer requires the URL of a SiteCollection, not a SubSite.

SiteCollections have their own permissions and access. Consider the following two SiteCollections:

https://contoso.sharepoint.com https://contoso.sharepoint.com/sites/MySite

If the appClientID was created for https://contoso.sharepoint.com/sites/MySite but in the importer siteCollectionUrl you are setting https://contoso.sharepoint.com you will receive a 401 Unauthorized error.

How to tell if a site is a SiteCollection or a SubSite

Go to the site you are trying to import into, select Site Contents and then Site Settings. If you see the Site Collection Administration groups with all relevant links as in the image below then it means you are on a site collection.

If you see the Site Collection Administration with a single link saying Go to top level site settings as in the image below then you are on subsite.

Additional Logs

Additional log files generated by the SharePoint Online Batch Importer can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\SPOnline Batch Importer\<job run id>

You can find the following files in the <job run id> folder:

The import-job.log contains detailed information about the job run and can be used to investigate import issues.
The generated manifest files for each batch in the <batch number>/manifest sub-folder.
The log files generated by SharePoint Online for each batch in the <batch-number>/spo-logs sub-folder.
The content files generated for link documents in the <batch number>/link_files sub-folder.