SharePoint Online Batch Importer

Introduction

The SharePoint Online Batch importer allows migrating documents and folders to SharePoint Online and OneDrive. Since OneDrive is based on SharePoint Online, you can use the SharePoint Online Batch importer to import documents and folders to OneDrive as well. If we refer to SharePoint Online in the following, this does apply to OneDrive as well in most cases. We will write appropriate notes in case the behavior of the importers differs for OneDrive.

The SharePoint Online Batch importer offers the following features:

  • Leverages Microsoft SharePoint Online Import Migration API (bulk import API)

  • Import documents to Microsoft SharePoint Online Document Library items

  • Import folders to Microsoft SharePoint Online Document Library items

  • Set values for any columns in SharePoint Online, including user defined columns

  • Set values for SharePoint Online specific internal field values, i.e. author, editor, time created, time last modified

  • Set the folder path and create folders if they don’t exist

  • Set role assignments (permissions) on documents and folders

  • Import versions

  • Import files with a size up to 15GB

  • Automatic or manual/custom version numbering

  • Set moderation/approval status on documents

The SharePoint Online Batch importer is implemented mainly as a Job Server component but comes with a separate component for communicating with SharePoint Online (we refer to this component as CSOM Service because it uses the Microsoft CSOM API).

SharePoint Online Batch importers can be created, configured, started and monitored through migration-center Client, while the corresponding processes are executed by migration-center Job Server and the migration-center SharePoint Online Batch Importer respectively.

The term importer is used in migration-center for describing a component which imports processed data from migration-center to another system.

Scanners and importers work as jobs that can be run at any time and can even be executed repeatedly. For every run a detailed history and log file are created. Multiple scanner and import jobs can be created or run at a time, each being defined by a unique name, a set of configuration parameters and a description (optional).

Features, Limitations, and Known Issues

The SharePoint Online Batch importer comes with the following features, limitations, and known issues:

  • The importer uses the Azure storage containers provided by SharePoint Online.

  • The importer only supports

    • import of documents with metadata (incl. role assignments) and versions and

    • import of folders with metadata (incl. role assignments).

  • You can only assign site collection groups to user/group fields and role assignments.

  • The following column / field types are currently supported by the importer:

    • User

    • Text

    • Integer

    • Number

    • Choice

    • Note

    • DateTime

    • Boolean

    • TaxonomyFieldType

  • Any invalid XML characters in Text and Note field values will be automatically replaced by a '_' character during import. Otherwise the import would fail with an error.

  • The target of a migration job must be a Folder in a Document Library of the specified Web (Site). All folders and documents will go inside or below that Folder.

  • When importing documents or folders, the importer will automatically create any subfolders (of default type) needed below the configured base folder if parameter "autoCreateFolder" is set to "true".

  • The importer will not respect the version numbers of the objects. Instead it will create ascending major version numbers, i.e. "1.0" for version 1, "2.0" for version 2 etc.

  • The importer only supports setting managed metadata terms (taxonomies) by their ID in the format "{9ca5bcac-b5b5-445c-8dab-3d1c2181bcf2}". Setting multiple terms on a target attribute is currently not supported.

  • Delta migration is not supported by the SharePoint Online Migration API and thus this feature is not available with the SharePoint Online Batch Importer. If you need the delta migration feature, please use our SharePoint Online Importer, which supports this feature.

  • If you set the system rule “declareAsRecord” to “true” for an object, the importer will declare the object as a record if the target library is a Records Library.

  • Due to the asynchronous nature of the SharePoint Migration API

    • The progress indicator in the migration-center client works currently only on a per batch level, i.e. if you import only one batch, the progress jumps from 0% to 100% when that batch was imported. Future versions of the importer may provide a more granular progress indicator.

    • The migration-center database might not reflect the actual import state of the objects in the migset. For example, it might happen that objects get imported but are still marked as Validated or Error in the migration-center database. For technical details please see Chapter 5 - Technical Details on the Importer

  • If your site has the Document ID feature enabled: The document ID will be populated for the files imported via the migration API as well – just that it will happen asynchronously. So the DocId might be missing right after the import, but it will be populated within 24hrs (when the backend job runs).

  • Import into OneDrive was only tested with Azure AD app-only principal authentication.

Install and Uninstall

To install the main product components, consult the migration-center Installation Guide document.

The migration-center SharePoint Online Batch Importer requires installing an additional, separate component besides the main product components. In the following we refer to this component as “SPOnline Batch CSOM Service”.

The SPOnline Batch CSOM Service is necessary to use the SharePoint Online Migration API with migration-center. Its installation and uninstallation are described below.

Prerequisites

The SPOnline Batch CSOM Service is designed to run as a Windows service and needs the .NET Framework 4.7.2 (or later) installed on the migration-center Job Server machine.

Installation Steps

You must install the SPOnline Batch CSOM Service on all machines where you had installed the migration-center server components.

  1. To install SPOnline Batch CSOM Service, it is necessary to run an installation file, which is located within the SharePoint Online Batch Importer component folder of your migration-center Job Server installation location, which is by default C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install. This folder contains the file install.bat, which must be executed with administrative privileges (i.e. “Run as Administrator”).

  2. After the service is installed you will need to start it manually for the first time, after that the service is configured to start automatically as soon as the computer’s operating system is loaded.

Uninstallation

In case you need to uninstall the SPOnline Batch CSOM Service, please run the file uninstall.bat in C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-spo-batch-importer\CSOM_Service\install with administrative privileges (i.e. “Run as Administrator”).

Working with the Importer

In this chapter you will learn working with the SharePoint Online Batch importer. Specifically, you will see how to import documents with metadata and versions and how to import folders with metadata.

The SharePoint Online Batch importer lets you import documents and folders into a folder of a document library, i.e. you have to configure the SharePoint Online site, document library and the base folder of your import in the importer configuration (see Configuring the Importer). All documents and folders that you import go into the specified base folder or into a folder below.

Preparation for app-only principal authentication

The importer supports app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors: Azure AD app-only principal authentication and SharePoint app-only principal authentication.

Azure AD app-only authentication requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.

If you want to restrict the access of the migration-center application to certain site collections or sites, you can use SharePoint app-only authentication.

Azure AD app-only principal authentication

The migration-center SharePoint Online Batch Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.

Please follow these steps in order to setup your migration-center application in your Azure AD.

The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Configure necessary permissions for the migration-center application

Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:

  • Microsoft APIs

    • SharePoint

      • Application permissions

        • Sites

          • Sites.FullControl.All

        • TermStore

          • TermStore.Read.All

        • User

          • User.Read.All

    • Graph

      • Application permissions

        • Sites

          • Sites.FullControl.All

Step 4: Uploading the self-signed certificate

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

Step 5: Grand admin consent

The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.

Final Step: Setting the necessary parameters in the importer

In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the migration-center Azure AD application.

appCertificatePath

The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application.

appCertificatePassword

The password to read the certificate specified in appCertificatePath.

SharePoint app-only principal authentication

SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.

The information in this chapter is based on the following guidelines from Microsoft:

https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azureacs https://docs.microsoft.com/en-us/sharepoint/dev/sp-add-ins/add-in-permissions-in-sharepoint

Step 1: Create a self-signed certificate for your migration-center Azure AD application

In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.

You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.

It's important that you run the below scripts with Administrator privileges.

To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-sharepointonline -importer\scripts folder:

.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30

The dates are provided in ISO date format: YYYY-MM-dd

You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.

Save the password of the private key as you’ll need it later.

Step 2: Register the migration-center Azure AD application

Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.

In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.

Once the application has been created copy the "Application (client) ID" as you’ll need it later.

Step 3: Uploading the self-signed certificate and generate secret key

Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.

After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.

Store the retrieved information (client id and client secret) since you'll need this later! Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Step 4: Granting permissions to the app-only principal

Next step is granting permissions to the newly created principal in SharePoint Online.

If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.

If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.

Once the page is loaded add your client id and look up the created principal:

Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.

To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

Use this permission XML for granting site collection scoped permissions:

<AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" /> <AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" /> </AppPermissionRequests>

When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:

Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!

Final Step: Setting the necessary parameters in the importer

In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:

Configuration parameters

Values

appClientId

The ID of the SharePoint application you have created.

appClientSecret

The client secret, which you have generated when setting up the SharePoint application.

Importing Documents

In order to import documents, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(document)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules. A migration set with a target type of “SPOnlineBatch(document)” has the following system attributes, which primarily determine the content type, name and location of the documents.

You can find a detailed description of the available system rules in the table below:

Configuration parameters

Values

mc_content_location

Rule for setting alternative content of the document, i.e. if you want to upload a different content file than the scanned one.

Example: D:\Some\Alternative\Content\Path\Content.pdf

Optional

If you leave this rule empty, the importer will import the scanned content along with the document. In case you want to upload a different content, e.g. maybe you converted a PDF file into PDF-A format, use this rule to specify the location of the new content file.

s__contentType*

Rule for setting the SharePoint content type of the document.

Example: Document

Mandatory

This value must match an existing migration-center object type definition and of course a content type in the target SharePoint document library.

s__createdBy

Name of the user who created the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__createdDate

Date and time when the document was created.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__declareAsRecord

Flag indicating if the document should be declared as a record.

Example: True

Optional

Record declaration will only work if

  • SharePoint Online is the target (and not OneDrive)

  • the target library is a Records Library and

  • manual record declaration is enabled in the Records Library.

s__moderationStatus

Sets the moderation / approval status of the document. Must be one of the following values (you can either use the status names or their numerical value: Approved (or 0), Denied (or 1), Draft (or 3), Pending (or 2), Scheduled (or 4)

Optional

s__moderationStatusComment

Sets the moderation / approval status comment of the document.

Optional

s__modifiedBy

Name of the user who last modified the document. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__modifiedDate

Date and time when the document was last modified.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__name*

The name of the document including its file extension.

Example: My Document.docx

Mandatory

s__parentFolder*

The parent folder of the document.

Example: /My subfolder

Mandatory

The parent folder is relative to the base folder configured in the importer definition, e.g. if the base folder is /Shared Documents and the parent folder is /My subfolder, the effective folder for the document will be /Shared Documents/My subfolder

s__roleAssignments

Specify role assignments for the current object. If a role definition is assigned to the current object, the Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint group exists in your SharePoint site. To set role assignments for single users, the value must match the following pattern: <loginname>;#<roledefinitionname> To set role assignments for SharePoint groups, the value must match the following pattern: @<groupname>;#<roledefinitionname> Example: user;#Read @Contributors;#Contribute

Optional

s__versionNumber

Sets the version number of the document. Must be in the format: x.y Example: 1.0

Mandatory if importer parameter "autoVersioning" is set to "false".

Specified value will be ignored if parameter "autoVersioning" is set to "true".

Usually you want to set additional custom attributes on the documents. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Importing Folders

In order to import folders, you need to create a migration set with a processing type of “<Source>ToSPOnlineBatch(document)” as shown in the screenshot below:

After you have selected the objects to import from the file scans, you need to configure the migration set’s transformation rules.

A migration set with a target type of “SPOnlineBatch(folder)” has the following system attributes, which primarily determine the content type, name and location of the folders.

You can find a detailed description of the available system rules in the table below:

Configuration parameters

Values

s__contentType*

Rule for setting the SharePoint content type of the folder.

Example: Folder

Mandatory

This value must match an existing migration-center object type definition and of course a content type in the target SharePoint document library.

s__createdBy

Name of the user who created the folder. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__createdDate

Date and time when the folder was created.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__modifiedBy

Name of the user who last modified the folder. The value must be a unique username.

Example: MikeGration@asdf.onmicrosoft.com

Optional

If not provided, the system account will be set.

s__modifiedDate

Date and time when the folder was last modified.

Example: 2020-03-31 14:14:03

Optional

If not provided, the current date and time will be set.

s__name*

The name of the folder.

Example: My Folder

Mandatory

s__parentFolder*

The parent folder of the folder.

Example: /My subfolder

Mandatory

The parent folder is relative to the base folder configured in the importer definition, e.g. if the base folder is /Shared Documents and the parent folder is /My subfolder, the effective folder path of the folder will be /Shared Documents/My subfolder/My Folder

s__roleAssignments

Specify role assignments for the current object. If a role definition is assigned to the current object, the Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint group exists in your SharePoint site. To set role assignments for single users, the value must match the following pattern: <loginname>;#<roledefinitionname> To set role assignments for SharePoint groups, the value must match the following pattern: @<groupname>;#<roledefinitionname> Example: user;#Read @Contributors;#Contribute

Optional

Usually you want to set additional custom attributes on the folders. Therefore, you need to define the regular transformation rules first, and associate them with the corresponding target type attributes after that. For more details on transformations rules and associations, please see the corresponding chapters in the Client User Guide.

You also need to select the system rule “s__contentType” in the “Get type name from rule” dropdown box on the | Associations | tab and add all required type definitions in the “Types” list box.

Finally, after you have transformed and validated your migration set, you should be ready to import it using the SharePoint Online Batch Importer.

Configuring the Importer

To create a new SharePoint Online Batch Importer, go to -Importers- and press the [New] button. Then select as

An importer of type SPOnline Batch has the following parameters, which you need to set with the appropriate values:

Configuration parameters

Values

tenantName*

The name of your SharePoint Online Tenant

Example: Contoso

Mandatory

There are several web site that explain how to determine a SharePoint Online tenant name, e.g. https://morgantechspace.com/2019/07/how-to-find-your-tenant-name-in-office-365.html

tenantUrl*

The URL of your SharePoint Online Tenant

Examples: https://contoso.sharepoint.com https://contoso-my.sharepoint.com (for OneDrive)

Mandatory

siteName*

The path to your target site for the import.

Examples: /sites/My Site /personal/my_name_company_domain (for OneDrive)

Mandatory

documentLibraryName*

The name of your target document library in the specified site.

Example: Shared Documents

Mandatory

The name of the document library might depend on your site's locale settings, e.g. it might be Dokumente for a German locale.

importLocation*

The base folder for your import in the specified document library.

Example: /The/documents/go/here

Mandatory

If you want to import into the root folder of the document library, just specify “/” as your import location.

appClientId*

The ID of either the migration-center Azure AD application or the SharePoint application.

Example: ab187da0-c04d-4f82-9f43-51f41c0a3bf0

Mandatory

See Preparation for app-only principal authentication

appCertificatePath

The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application.

Example: D:\migration-center\config\azure-ad-app-cert.pfx

Mandatory

See Preparation for app-only principal authentication

appCertificatePassword

The password to read the certificate specified in appCertificatePath.

Mandatory

See Preparation for app-only principal authentication

appClientSecret

The client secret, which you have generated when setting up the SharePoint application (SharePoint app-only principal authentication).

Mandatory

See Preparation for app-only principal authentication

proxyServer

This is the URL, which defines the location of your proxy to connect to the Internet.

Optional

Just leave blank if no proxy is used.

proxyUsername

The login of the user, who connects and authenticates on the proxy, which was specified in parameter proxyServer.

Example: corporatedomain\username

Optional

Just leave blank if no proxy is used.

proxyPassword

Password of the proxy user specified above.

Optional

Just leave blank if no proxy is used.

autoCreateFolder*

Enable/disable whether folders, which do not exist, should be created during import.

Enabled: The importer creates any specified folders automatically.

Disabled: No new folders will be created, but any existing folders (same path and name) are used. References to non-existing folders will throw an error.

autoVersioning*

Enable/disable whether version numbers should be generated automatically by the importer.

Enabled: The importer will generate consecutive version numbers for the versions of a document based on the versioning setting of the target library.

Disabled: The user must provide appropriate version numbers for the documents using the "s__versionNumber" system attribute.

numberOfThreads*

Number of batches that shall be imported in parallel.

Default: 8

Mandatory

You may increase this value for large imports, depending on the configuration of the machine that runs the job server.

loggingLevel*

Sets the verbosity of the log file.

Allowed values:

1 - logs only errors during scan

2 - is the default value reporting all warnings and errors

3- logs all successfully performed operations in addition to any warnings or errors

4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production)

Mandatory

Technical Details on the Importer

This chapter will give you an overview of the importer’s operating principle.

Overview

Each SharePoint Online Batch Importer job will go through the following steps:

  1. Split the list of objects to import into batches of approximately 250 objects.

  2. For each batch repeat

    1. Generate the XML files (e.g. Manifest.xml etc.) necessary for submitting the batch to the SharePoint Migration API.

    2. Upload XML files and content files of the batch to an Azure BLOB container storage.

    3. Submit the import batch to the SharePoint platform as a migration job.

    4. Monitor the progress of the migration job.

    5. When the migration job has finished, verify that all objects submitted for import were successfully imported by retrieving them by ID and save the result of the verification back to the migration-center database.

Depending on your job configuration, several batches are imported in parallel (see “numberOfThreads” parameter in the import job configuration).

Splitting into batches

The importer will split the list of documents or folders to import into batches of approximately 250 items using the following rules

  • All versions of an item must be contained in the same batch.

  • A batch should contain a maximum of 250 items.

  • The content size of the items in a batch should not exceed 250 MB

The first rule can lead to batches with more than 250 objects. The last rule can lead to batches with less than 250 objects.

XML files

The SharePoint Import Migration API requires the following XML files for each import batch:

  • ExportSettings.XML

  • LookupListMap.XML

  • Manifest.XML

  • Requirements.XML

  • RootObjectMap.XML

  • SystemData.XML

  • UserGroupMap.XML

  • ViewFormsList.XML

For more details on those files, please see https://docs.microsoft.com/en-us/sharepoint/dev/apis/migration-api-overview#import-package-structure

History, Reports, Logs

History

A complete history is available for any SharePoint Online Batch Importer job from the respective items’ –History- window. It is accessible through the [History] button/menu entry on the toolbar/context menu. The -History- window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.

Job Report File

Double clicking an entry in the -History-window or clicking the Open button on the toolbar opens the job report file created by that run. The report file contains more information about the run of the selected job:

  • Version information of the migration-center Server Components the job was run with

  • The parameters the job was run with

  • Execution Summary that contains the total number of objects processed, the number of documents and folders scanned or imported, the count of warnings and errors that occurred during runtime.

Job Log Files

Log files generated by the SharePoint Online Batch Importer can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs\SPOnline Batch Importer\<job run id>

You can find the following files in the <job run id> folder:

  • The import-job.log contains detailed information about the job run and can be used to investigate import issues.

  • The generated manifest files for each batch in the <batch number>/manifest sub-folder.

  • The log files generated by SharePoint Online for each batch in the <batch-number>/spo-logs sub-folder.

The amount of information written to the report and log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.

Last updated