SharePoint Online Batch Importer

Introduction

The SharePoint Online Batch importer allows migrating documents and folders to SharePoint Online and OneDrive. Since OneDrive is based on SharePoint Online, you can use the SharePoint Online Batch importer to import documents and folders to OneDrive as well. If we refer to SharePoint Online in the following, this does apply to OneDrive as well in most cases. We will write appropriate notes in case the behavior of the importers differs for OneDrive.

The SharePoint Online Batch importer offers the following features:

• Leverages Microsoft SharePoint Online Import Migration API (bulk import API)

• Import documents to Microsoft SharePoint Online Document Library items

• Import folders to Microsoft SharePoint Online Document Library items

• Import link documents (.url files) to Microsoft SharePoint Online Document Library items

• Set values for any columns in SharePoint Online, including user defined columns

• Set values for SharePoint Online specific internal field values, i.e. author, editor, time created, time last modified

• Set the folder path and create folders if they don’t exist

• Set role assignments (permissions) on documents and folders

• Import versions

• Automatic or manual/custom version numbering

• Set moderation/approval status on documents

• Set Microsoft Information Protection (MIP) Sensitivity Labels on content files prior to uploading them to SPO

• Set compliance (retention) labels on folders and documents

The SharePoint Online Batch importer is implemented mainly as a Job Server component but comes with a separate component for communicating with SharePoint Online (we refer to this component as CSOM Service because it uses the Microsoft CSOM API).

SharePoint Online Batch importers can be created, configured, started and monitored through migration-center Client, while the corresponding processes are executed by migration-center Job Server and the migration-center SharePoint Online Batch Importer respectively.

The term importer is used in migration-center for describing a component which imports processed data from migration-center to another system.

Scanners and importers work as jobs that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created. Multiple scanner and import jobs can be created or run at a time, each being defined by a unique name, a set of configuration parameters and a description (optional).

Features, Limitations, and Known Issues

The SharePoint Online Batch importer comes with the following features, limitations, and known issues:

• The importer uses the Azure storage containers provided by SharePoint Online.

• The importer only supports

  • import of documents with metadata (incl. role assignments) and versions,

  • import of folders with metadata (incl. role assignments) and

  • import of link documents.

• You can only assign site collection groups to user/group fields and role assignments.

• The following column / field types are currently supported by the importer:

  • User

  • Text

  • Integer

  • Number

  • Choice and MultiChoice

  • Lookup

  • Note

  • DateTime

  • Boolean

  • TaxonomyFieldType

• Any invalid XML characters in Text and Note field values will be automatically replaced by a '_' character during import. Otherwise the import would fail with an error.

• The target of a migration job must be a SharePoint Site Collection. All objects in an import job must be imported into sites / libraries in the site collection specified in the importer definition.

• When importing documents or folders, the importer will automatically create any subfolders (of default type) needed below the configured base folder if parameter "autoCreateFolder" is set to "true".

• The importer only supports setting managed metadata terms (taxonomies) by their ID in the format "{9ca5bcac-b5b5-445c-8dab-3d1c2181bcf2}".

• Delta migration is not supported by the SharePoint Online Migration API and thus this feature is not available with the SharePoint Online Batch Importer. If you need the delta migration feature, please use our SharePoint Online Importer, which supports this feature.

• If you set the system rule “declareAsRecord” to “true” for an object, the importer will declare the object as a record if the target library is a Records Library - no matter how the records management is configured for that library. If you set "declareAsRecord" to "false" for an object, the import will not declare the object as record, even if automatic records declaration is enabled for that library.

• Due to the asynchronous nature of the SharePoint Migration API

  • The progress indicator in the migration-center client works currently only on a per batch level, i.e. if you import only one batch, the progress jumps from 0% to 100% when that batch was imported. Future versions of the importer may provide a more granular progress indicator.

  • The migration-center database might not reflect the actual import state of the objects in the migset. For example, it might happen that objects get imported but are still marked as Validated or Error in the migration-center database. For technical details please see Technical Details on the Importer

• If a target site has the Document ID feature enabled: The document ID will be populated for the files imported via the migration API as well – just that it will happen asynchronously. So the DocId might be missing right after the import, but it will be populated within 24hrs (when the backend job runs).

• Import into OneDrive was only tested with Azure AD app-only principal authentication.

• Due to a limitation in the SPO Bulk Import API, the importer allows you to set values for choice/multi-choice fields that are not in the list of available values - even if the "fill-in" option is disabled. If you want to ensure that only allowed values are set, please use a mapping list in the transformation rules for the choice/multi-choice fields.

• When trying to import a broken version tree, the import fails for all documents (even the ones that had a correct version tree) with an error status.

• When setting a compliance label on a folder, all children of that folder will get the same label if no compliance label is specified for them.

• The importer supports importing files with long paths (i.e. longer than 260 characters), but you must enable this feature in the Windows operating system first (see https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation).

Install and Uninstall

To install the main product components, consult the migration-center Installation Guide document.

The migration-center SharePoint Online Batch Importer requires installing an additional, separate component besides the main product components. In the following we refer to this component as “SPOnline Batch CSOM Service”.

The SPOnline Batch CSOM Service is necessary to use the SharePoint Online Migration API with migration-center. Its installation and uninstallation are described below.

The app-only principal authentication used by the importer calls the following HTTPS endpoints. Please ensure that the job server machine has access to those endpoints:

• <tenant name>.sharepoint.com:443

• accounts.accesscontrol.windows.net:443

Prerequisites

The SPOnline Batch CSOM Service is designed to run as a Windows service and needs the .NET Framework 4.7.2 (or later) installed on the migration-center Job Server machine.

Installation Steps

You must install the SPOnline Batch CSOM Service on all machines where you had installed the migration-center server components.

1. To install SPOnline Batch CSOM Service, it is necessary to run an installation file, which is located within the SharePoint Online Batch Importer component folder of your migration-center Job Server installation location, which is by default C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install. This folder contains the file install.bat, which must be executed with administrative privileges (i.e. “Run as Administrator”).

2. After the service is installed you will need to start it manually for the first time, after that the service is configured to start automatically as soon as the computer’s operating system is loaded.

Uninstallation

In case you need to uninstall the SPOnline Batch CSOM Service, please run the file uninstall.bat in C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install with administrative privileges (i.e. “Run as Administrator”).

Working with the Importer

In this chapter you will learn working with the SharePoint Online Batch importer. Specifically, you will see how to import documents with metadata and versions and how to import folders with metadata.

The SharePoint Online Batch importer lets you import documents and folders into a folder of a document library, i.e. you have to configure the SharePoint Online site, document library and the base folder of your import in the importer configuration (see Configuring the Importer). All documents and folders that you import go into the specified base folder or into a folder below.

Preparation for app-only principal authentication

The importer supports app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors: Azure AD app-only principal authentication and SharePoint app-only principal authentication.

Azure AD app-only authentication requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.

If you want to restrict the access of the migration-center application to certain site collections or sites, you can use SharePoint app-only authentication.

Azure AD app-only principal authentication

The migration-center SharePoint Online Batch Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.

Please follow these steps in order to setup your migration-center application in your Azure AD.

The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Step 3: Configure necessary permissions for the migration-center application

Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:

• Microsoft APIs

  • SharePoint

    • Application permissions

      • Sites

        • Sites.FullControl.All

      • TermStore

        • TermStore.Read.All

      • User

        • User.Read.All

  • Graph

    • Application permissions

      • Sites

        • Sites.FullControl.All

Click on the blue "Add permissions" button at the bottom to add the permissions to your application. The "Application permissions" are those granted to the migration-center application when running as App Only.

Step 4: Uploading the self-signed certificate

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

Step 5: Grand admin consent

The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.

Final Step: Setting the necessary parameters in the importer

In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:

SharePoint app-only principal authentication

SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.

The information in this chapter is based on the following guidelines from Microsoft:

https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Step 3: Uploading the self-signed certificate and generate secret key

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.

Step 4: Granting permissions to the app-only principal

Next step is granting permissions to the newly created principal in SharePoint Online.

If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.

If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.

Once the page is loaded add your client id and look up the created principal:

Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.

To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

Use this permission XML for granting site collection scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:

Final Step: Setting the necessary parameters in the importer

In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Preparation for applying Microsoft Information Protection sensitivity labels

Starting with v3.16, our SharePoint Online Batch importer supports applying Microsoft Information Protection (MIP) sensitivity labels on the content files before they get uploaded to SharePoint Online. To learn more about sensitivity labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide.

Register an Azure AD app

The MIP software development kit provided by Microsoft that we use to apply the sensitivity labels on the content files requires you to register an application in your Azure AD. You could use the same app as in Azure AD app-only principal authentication or setup a separate application as described here: https://docs.microsoft.com/en-us/information-protection/develop/setup-configure-mip#register-a-client-application-with-azure-active-directory

You need to create a client secret for your registered application and configure the following access permissions:

Install VC++ Runtime

The Microsoft MIP SDK also needs the VC++ runtime installed on the job server machine to work properly. You might get a LoadLibrary failed for: [C:\fme AG\migration-center Server Components 3.15\lib\mc-sharepointonline-importer\MIPService\x86\mip_dotnet.dll] error if the VC++ runtime is missing and you want to apply sensitivity labels on your documents.

For your convenience, we provide the VC++ runtime installers in the lib\mc-sharepointonline-importer\MIPService\VC_Redistributables folder of your MC job server installation. Which variant to install (i.e. x86 or x64) depends on your .NET configuration. Usually you should install the x86 VC++ runtime (even on a x64 system). If that does not work, you should install the x64 variant.

Create the configuration file for the importer

The SharePoint Online Batch importer requires you to provide several configuration parameters in a XML configuration file. You can find a template configuration file (template-mip.config) in the lib\mc-spo-batch-importer folder of your migration-center job server installation.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="<YOUR AZURE APP ID>" />
		<add key="Azure.App.Name" value="<YOUR AZURE APP NAME>"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="<YOUR ENCRYPTED AZURE APP SECRET>"/>
		<add key="Azure.App.RedirectUri" value="<YOUR AZURE APP REDIRECT URL>"/>
		<add key="Azure.Tenant.Domain" value="<YOUR AZURE TENANT DOMAIN>"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="<OPTIONAL PATH TO LABELED CONTENT FOLDER>"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

Create a copy of the template file, fill in the required information from your Azure AD app that you had created in the previous step, and enter the full path to your config file in the mipConfigFile parameter of your SharePoint Online Batch importer. The result should look similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
	<appSettings>
		<!-- Azure App Settings  -->
		<add key="Azure.App.Id" value="91583618-eb95-4ae9-8f9e-c5259491f66d" />
		<add key="Azure.App.Name" value="miptest"/>
		<add key="Azure.App.Version" value="1.0.0"/>
		<add key="Azure.App.Secret" value="0;#75+wbxpdQ0c=;#pp39p7v4QA57VKiP0NNcnuX/FEk2FiE8iD4PrJnysfLuDbNra2xsFHA2K9wAKG8z"/>
		<add key="Azure.App.RedirectUri" value="miptest://authorize"/>
		<add key="Azure.Tenant.Domain" value="migrationcenter.onmicrosoft.com"/>
		
		<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
		<add key="Labeling.Temp.Folder" value="C:\Temp\sensitivity labeled docs"/>

		<!-- File name format for the labeled files. The placeholders {id}, {cnt}, and {ext} will be replaced -->
		<!-- by job run id, document counter, and original file extension at runtime. -->
		<add key="Labeling.Filename.Format" value="lbld_{id}_{cnt}{ext}"/>
	</appSettings>
</configuration>

We provide a Java based command line tool for encryption, which you must use to encrypt the Azure.App.Secret value before you put it in the configuration file. To encrypt the secret, proceed as follows:

1. Open a command line window on the job server machine.

2. Switch to the migration-center Server Components <version>\lib\mc-core folder, e.g. migration-center Server Components 3.16\lib\mc-core

3. Run the following command: java -cp mc-common-<version>.jar de.fme.mc.common.encryption.CmdTool (of course replace <version> with the appropriate version number of your migration-center)

4. The tool will ask for the text to encrypt. Enter or paste the secret value and press enter.

5. The tool will output the encrypted text. Copy and paste it into the configuration file.

Importing Documents

In order to import documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(document)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(document)” has the following system attributes, which primarily determine the content type, name and location of the documents.

You can find a detailed description of the available system rules in the table below:

Usually you want to set additional custom attributes on the documents. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Importing Link Documents

SharePoint Online supports documents with the file format ".url". We call these documents "link documents" since they do not have real content but just point to another document in SharePoint Online or to an external web site. The importer will create the necessary content of the link documents on the fly and save them in the following folder: <mc log folder>/SPOnline Batch Importer/<job run id>/link_files. Thus, the importer will ignore the content of any source object that is imported as a link document because it will import the generated link content instead.

In order to import link documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(link)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(link)” has the following system attributes, which primarily determine the content type, name and URL of the linked documents or web site respectively.

You can find a detailed description of the available system rules in the table below:

You can set additional custom attributes on the link documents. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Importing Folders

In order to import folders, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(folder)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules.

A migration set with a target type of “SPOnlineBatch(folder)” has the following system attributes, which primarily determine the content type, name and location of the folders.

You can find a detailed description of the available system rules in the table below:

Usually you want to set additional custom attributes on the folders. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Configuring the Importer

To create a new SharePoint Online Batch Importer, go to -Importers- and press the [New] button. Then select as

An importer of type SPOnline Batch has the following parameters, which you need to set with the appropriate values:

Tracing imported objects

The SharePoint Online Batch Importer will store the following information for imported objects in the MC database, which will allow you to trace an imported object back to the corresponding source object.

Setting Compliance Labels

Compliance labels are managed in the MS 365 compliance center, which you can access with the following URL: https://compliance.microsoft.com/

For more details about creating and publishing compliance labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/create-apply-retention-labels?view=o365-worldwide

You can assign any label that is published for the target SharePoint site to the folders and documents in a migration set. The migration-center provides a new system rule for setting compliance labels:

• s__complianceTag

If you want to set a label on a document or folder, it is mandatory to specify the label name in the "s__complianceTag" system rule.

The importer will check before importing the object if the specified label is available in the target site and throw an error if it is not in the list of available labels.

Technical Details on the Importer

This chapter will give you an overview of the importer’s operating principle.

Overview

Each SharePoint Online Batch Importer job will go through the following steps:

1. Split the list of objects to import into batches of approximately 250 objects.

2. For each batch repeat

   1. Generate the XML files (e.g. Manifest.xml etc.) necessary for submitting the batch to the SharePoint Migration API.

   2. Upload XML files and content files of the batch to an Azure BLOB container storage.

   3. Submit the import batch to the SharePoint platform as a migration job.

   4. Monitor the progress of the migration job.

   5. When the migration job has finished, verify that all objects submitted for import were successfully imported by retrieving them by ID and save the result of the verification back to the migration-center database.

Depending on your job configuration, several batches are imported in parallel (see “numberOfThreads” parameter in the import job configuration).

Splitting into batches

The importer will split the list of documents or folders to import into batches of approximately 250 items using the following rules

• All objects must have the same target site & library.

• All versions of an item must be contained in the same batch.

• A batch should contain a maximum of 250 items.

• The content size of the items in a batch should not exceed 250 MB

The second rule can lead to batches with more than 250 objects. The first or last rule can lead to batches with less than 250 objects.

XML files

The SharePoint Import Migration API requires the following XML files for each import batch:

• ExportSettings.XML

• LookupListMap.XML

• Manifest.XML

• Requirements.XML

• RootObjectMap.XML

• SystemData.XML

• UserGroupMap.XML

• ViewFormsList.XML

For more details on those files, please see https://docs.microsoft.com/en-us/sharepoint/dev/apis/migration-api-overview#import-package-structure

History, Reports, Logs

History

A complete history is available for any SharePoint Online Batch Importer job from the respective items’ –History- window. It is accessible through the [History] button/menu entry on the toolbar/context menu. The -History- window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Job Report File

Double clicking an entry in the -History-window or clicking the Open button on the toolbar opens the job report file created by that run. The report file contains more information about the run of the selected job:

• Version information of the migration-center Server Components the job was run with

• The parameters the job was run with

• Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Job Log Files

Log files generated by the SharePoint Online Batch Importer can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\SPOnline Batch Importer\<job run id>

You can find the following files in the <job run id> folder:

• The import-job.log contains detailed information about the job run and can be used to investigate import issues.

• The generated manifest files for each batch in the <batch number>/manifest sub-folder.

• The log files generated by SharePoint Online for each batch in the <batch-number>/spo-logs sub-folder.

• The content files generated for link documents in the <batch number>/link_files sub-folder.