SharePoint Online Importer
The Microsoft SharePoint Online Importer allows migrating documents, folders, list items and lists/libraries to SharePoint Online offering following features:
- Import documents to Microsoft SharePoint Online Document Library items
- Import URL / Link documents
- Import folders to Microsoft SharePoint Online Document Library items
- Import list items to Microsoft SharePoint Online Document Library items
- Import lists/libraries to Microsoft SharePoint Online Sites
- Set values for any columns in SharePoint Online, including user defined columns
- Set values for SharePoint Online specific internal field values
- Create documents, folders and list items using standard SharePoint Online content types or custom content types
- Set permissions for individual documents, folders, list items and lists/libraries
- Set the folder path and create folders if they don’t exist
- Apply Content Types automatically to lists/libraries if they are not applied yet
- Delta migration
- Import versions (minor or major, automatic enabling of versioning in the targeted document library)
- Import files with a size up to 250 GB
- Set Microsoft Information Protection (MIP) Sensitivity Labels on content files prior to uploading them to SPO
The SharePoint Importer is implemented mainly as a Job Server component but comes with a separate component for setting SharePoint Online specific internal field values, which can be installed optionally and if necessary.
Starting with migration-center version 3.13 Update 2 the SharePoint Online importer only supports app-only principal authentication. It is not possible to use user name / password authentication with this and later versions. Please consider this before you upgrade your existing installation.
The SharePoint Importer is implemented mainly as a Job Server component but comes with a separate component for setting SharePoint Online specific internal field values, which can be installed optionally and if necessary.
To install the main product components, consult the migration-center Installation Guide document.
The migration-center SharePoint Importer requires installing an additional, separate component besides the main product components. This additional component is able to set system values (such as creation date, modified date, author and editor) as well as taxonomy values for your objects. It is designed to run as a Windows service and needs the .NET Framework 4.7.2 installed on your computer, which is running this service as well as the migration-center Job Server.
The app-only principal authentication used by the importer calls the following HTTPS endpoints. Please ensure that the job server machine has access to those endpoints:
- <tenant name>.sharepoint.com:443
- accounts.accesscontrol.windows.net:443
This component must be installed on all machines where the migration-center server components is installed.
To install this additional component, it is necessary to run an installation file, which is located within the
SharePoint component folder of your migration-center Job Server installation location, which is by default C:\Program Files (x86)\fme AG\migration-center Server Components <Version>\lib\mc-sharepoint-online-importer\CSOM_Service\install. This folder contains the file install.bat, which must be executed with administrative privileges.
After the service is installed you will need to start it manually for the first time, after that the service is configured to start automatically as soon as the computers operating system is loaded.
When running the CSOM service with a domain account you might need to grant access to the account by running the following command:
netsh http add urlacl url=http://+:9999/ user=<your user><your user> might be in the format domain\username or [email protected]
In case it is necessary to uninstall this component, the file uninstall.bat must be executed.
Before uninstalling the Jobserver component, the CSOM service must be uninstalled as described here.
The migration-center SharePoint Online Importer can import objects generated by any of the available (and compatible) scanners. Most scanners can store the data they extract from the source systems they access in either a local path, or a UNC network path.
As is the case with all importers, they need to able to access the files extracted by a scanner in order to import.
See the respective scanner’s user guide for more information on configuration parameters if necessary.
The importer supports only app-principal authentication for connecting to SharePoint Online. The app-principal authentication comes in two flavors: Azure AD app-only principal authentication and SharePoint app-only principal authentication.
Azure AD app-only authentication requires full control access for the migration-center application on your SharePoint Online tenant. This includes full control on ALL site collections of your tenant.
If you want to restrict the access of the migration-center application to certain site collections or sites, you can use SharePoint app-only authentication.
Running multiple Job Servers for importing into SharePoint must be done with great care, and each of the Job Servers must NOT import in the same library at the same time. If this occurs, because the job will change library settings concurrently, the versioning of the objects being imported in that library will not be correct.
The migration-center SharePoint Online Batch Importer supports Azure AD app-only authentication. This is the authentication method for background processes accessing SharePoint Online recommended by Microsoft. When using SharePoint Online you can define applications in Azure AD and these applications can be granted permissions to your SharePoint Online tenant.
Please follow these steps in order to setup your migration-center application in your Azure AD.
The information in this chapter is based on the following Microsoft guidelines: https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread
In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.
You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.
It's important that you run the below scripts with Administrator privileges.
To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:
.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30The dates are provided in ISO date format: YYYY-MM-dd
You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.
Save the password of the private key as you’ll need it later.
Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.
In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.
Once the application has been created copy the "Application (client) ID" as you’ll need it later.
Now click on "API permissions" in the left menu bar and click on the "Add a permission" button. A new blade will appear. Here you choose the permissions that are required by migration-center. Choose i.e.:
- Microsoft APIs
- SharePoint
- Application permissions
- Sites
- Sites.FullControl.All
- TermStore
- TermStore.Read.All
- User
- User.Read.All
- Graph
- Application permissions
- Sites
- Sites.FullControl.All
Click on the blue "Add permissions" button at the bottom to add the permissions to your application. The "Application permissions" are those granted to the migration-center application when running as App Only.
Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.
The “Sites.FullControl.All” application permission requires admin consent in a tenant before it can be used. In order to do this, click on "API permissions" in the left menu again. At the bottom you will see a section "Grand consent". Click on the "Grand admin consent for" button and confirm the action by clicking on the "Yes" button that appears at the top.
In order to use Azure AD app-only principal authentication with the SharePoint Online Batch importer you need to fill in the following importer parameters with the information you gathered in the steps above:
Configuration parameters | Values |
appClientId | The ID of the migration-center Azure AD application. |
appCertificatePath | The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application. |
appCertificatePassword | The password to read the certificate specified in appCertificatePath. |
SharePoint app-only authentication allows you to grant fine granular access permissions on your SharePoint Online tenant for the migration-center application.
The information in this chapter is based on the following guidelines from Microsoft:
In Azure AD when doing App-Only you typically use a certificate to request access: anyone having the certificate and its private key can use the app and the permissions granted to the app. The below steps walk you through the setup of this model.
You are now ready to configure the Azure AD Application for invoking SharePoint Online with an App-Only access token. To do that, you must create and configure a self-signed X.509 certificate, which will be used to authenticate your migration-center Application against Azure AD, while requesting the App-Only access token. First you must create the self-signed X.509 Certificate, which can be created by using the makecert.exe tool that is available in the Windows SDK or through a provided PowerShell script which does not have a dependency to makecert. Using the PowerShell script is the preferred method and is explained in this chapter.
It's important that you run the below scripts with Administrator privileges.
To create a self-signed certificate with this script, which you can find in the <job server folder>\lib\mc-spo-batch-importer\scripts folder:
.\Create-SelfSignedCertificate.ps1 -CommonName "MyCompanyName" -StartDate 2020-07-01 -EndDate 2022-06-30The dates are provided in ISO date format: YYYY-MM-dd
You will be asked to give a password to encrypt your private key, and both the .PFX file and .CER file will be exported to the current folder.
Save the password of the private key as you’ll need it later.
Next step is registering an Azure AD application in the Azure Active Directory tenant that is linked to your Office 365 tenant. To do that, open the Office 365 Admin Center (https://admin.microsoft.com) using the account of a user member of the Tenant Global Admins group. Click on the "Azure Active Directory" link that is available under the "Admin centers" group in the left-side tree view of the Office 365 Admin Center. In the new browser's tab that will be opened you will find the Microsoft Azure portal (https://portal.azure.com/). If it is the first time that you access the Azure portal with your account, you will have to register a new Azure subscription, providing some information and a credit card for any payment need. But don't worry, in order to play with Azure AD and to register an Office 365 Application you will not pay anything. In fact, those are free capabilities. Once having access to the Azure portal, select the "Azure Active Directory" section and choose the option "App registrations". See the next figure for further details.
In the "App registrations" tab you will find the list of Azure AD applications registered in your tenant. Click the "New registration" button in the upper left part of the blade. Next, provide a name for your application, e.g. “migration-center” and click on "Register" at the bottom of the blade.
Once the application has been created copy the "Application (client) ID" as you’ll need it later.
Next step is “connecting” the certificate you created earlier to the application. Click on "Certificates & secrets" in the left menu bar. Click on the "Upload certificate" button, select the .CER file you generated earlier and click on "Add" to upload it.
After that, you need to create a secret key. Click on “New client secret” to generate a new secret key. Give it an appropriate description, e.g. “migration-center” and choose an expiration period that matches your migration project time frame. Click on “Add” to create the key.
Store the retrieved information (client id and client secret) since you'll need this later! Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!
Next step is granting permissions to the newly created principal in SharePoint Online.
If you want to grant tenant scoped permissions this granting can only be done via the “appinv.aspx” page on the tenant administration site. If your tenant URL is https://contoso-admin.sharepoint.com, you can reach this site via https://contoso-admin.sharepoint.com/_layouts/15/appinv.aspx.
If you want to grant site collection scoped permissions, open the “appinv.aspx” on the specific site collection, e.g. https://contoso.sharepoint.com/sites/mysite/_layouts/15/appinv.aspx.
Once the page is loaded add your client id and look up the created principal by pressing the "Lookup" button:
Please enter “www.migration-center.com” in field “App Domain” and “https://www.migration-center.com” in field “Redirect URL”.
To grant permissions, you'll need to provide the permission XML that describes the needed permissions. The migration-center application will always need the “FullControl” permission. Use the following permission XML for granting tenant scoped permissions:
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/tenant" Right="FullControl" />
<AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" />
</AppPermissionRequests>Use this permission XML for granting site collection scoped permissions:
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" />
<AppPermissionRequest Scope="http://sharepoint/taxonomy" Right="Read" />
</AppPermissionRequests>When you click on “Create” you'll be presented with a permission consent dialog. Press “Trust It” to grant the permissions:
Please safeguard the created client id/secret combination as would it be your administrator account. Using this client id/secret one can read/update all data in your SharePoint Online environment!
In order to use SharePoint app-only principal authentication with the SharePoint Online importer you need to fill in the following importer parameters with the information you gathered in the steps above:
Configuration parameters | Values |
appClientId | The ID of the SharePoint application you have created. |
appClientSecret | The client secret, which you have generated when setting up the SharePoint application. |
Starting with v3.15 Update 1, our SharePoint Online classic importer supports applying Microsoft Information Protection (MIP) sensitivity labels on the content files before they get uploaded to SharePoint Online. To learn more about sensitivity labels, please see https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide.
The MIP software development kit provided by Microsoft that we use to apply the sensitivity labels on the content files requires you to register an application in your Azure AD. You could use the same app as in Azure AD app-only principal authentication or setup a separate application as described here: https://docs.microsoft.com/en-us/information-protection/develop/setup-configure-mip#register-a-client-application-with-azure-active-directory
You need to create a client secret for your registered application and configure the following access permissions:
The Microsoft MIP SDK also needs the VC++ runtime installed on the job server machine to work properly. You might get a
LoadLibrary failed for: [C:\fme AG\migration-center Server Components 3.15\lib\mc-sharepointonline-importer\MIPService\x86\mip_dotnet.dll] error if the VC++ runtime is missing and you want to apply sensitivity labels on your documents.For your convenience, we provide the VC++ runtime installers in the
lib\mc-sharepointonline-importer\MIPService\VC_Redistributables folder of your MC job server installation. Which variant to install (i.e. x86 or x64) depends on your .NET configuration. Usually you should install the x86 VC++ runtime (even on a x64 system). If that does not work, you should install the x64 variant.The SharePoint Online classic importer requires you to provide several configuration parameters in a XML configuration file. You can find a template configuration file (
template-service.config) in the lib\mc-sharepointonline-importer folder of your migration-center job server installation.<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<!-- Azure App Settings -->
<add key="Azure.App.Id" value="<YOUR AZURE APP ID>" />
<add key="Azure.App.Name" value="<YOUR AZURE APP NAME>"/>
<add key="Azure.App.Version" value="1.0.0"/>
<add key="Azure.App.Secret" value="<YOUR ENCRYPTED AZURE APP SECRET>"/>
<add key="Azure.App.RedirectUri" value="<YOUR AZURE APP REDIRECT URL>"/>
<add key="Azure.Tenant.Domain" value="<YOUR AZURE TENANT DOMAIN>"/>
<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
<add key="Labeling.Temp.Folder" value="<OPTIONAL PATH TO LABELED CONTENT FOLDER>"/>
<!-- File name prefix for the labeled files. -->
<add key="Labeling.File.Prefix" value="lbld__"/>
</appSettings>
</configuration>
Create a copy of the template file, fill in the required information from your Azure AD app that you had created in the previous step, and enter the full path to your config file in the
mipServiceConfigFile parameter of your SharePoint Online importer. The result should look similar to the following:<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<!-- Azure App Settings -->
<add key="Azure.App.Id" value="91583618-eb95-4ae9-8f9e-c5259491f66d" />
<add key="Azure.App.Name" value="miptest"/>
<add key="Azure.App.Version" value="1.0.0"/>
<add key="Azure.App.Secret" value="0;#75+wbxpdQ0c=;#pp39p7v4QA57VKiP0NNcnuX/FEk2FiE8iD4PrJnysfLuDbNra2xsFHA2K9wAKG8z"/>
<add key="Azure.App.RedirectUri" value="miptest://authorize"/>
<add key="Azure.Tenant.Domain" value="migrationcenter.onmicrosoft.com"/>
<!-- Optional: Use a temporary folder to store the labeled files. If empty, the labeled files will be placed next to their originals. -->
<add key="Labeling.Temp.Folder" value="C:\Temp\sensitivity labeled docs"/>
<!-- File name prefix for the labeled files. -->
<add key="Labeling.File.Prefix" value="lbld__"/>
</appSettings>
</configuration>
We provide a Java based command line tool for encryption, which you must use to encrypt the Azure.App.Secret value before you put it in the configuration file. To encrypt the secret, proceed as follows:
- 1.Open a command line window on the job server machine.
- 2.Switch to the
migration-center Server Components <version>\lib\mc-corefolder, e.g.migration-center Server Components 3.16\lib\mc-core - 3.Run the following command:
java -cp mc-common-<version>.jar de.fme.mc.common.encryption.CmdTool - 4.The tool will ask for the text to encrypt. Enter or paste the secret value and press enter.
- 5.The tool will output the encrypted text. Copy and paste it into the configuration file.
Due to restrictions in SharePoint, documents cannot be moved from one Library to another using migration-center once they have been imported. This applies to Version and Update objects.
Moving folders is only supported within the same site, i.e. the importer parameter "siteName" and the system attribute "site" must have the same values for the initial and any update import runs.
Even though some other systems such as Documentum allow editing of older versions, either by replacing metadata, or creating branches, this is not supported by SharePoint. If you have updates to intermediate versions to a version tree that is already imported, the importer will return an error upon trying to import them. The only way to import them is to reset the original version tree and re-import it in the same run with the updates.
Running multiple Job Servers for importing into SharePoint must be done with great care, and each of the Job Servers must NOT import in the same library at the same time. If this occurs, because the job will change library settings concurrently, the versioning of the objects being imported in that library will not be correct.
The SharePoint Online system has some limitations regarding file names, folder names, and file size. Our SharePoint Online importer will perform the following validations before a file gets imported to SharePoint Online (in order to fail fast and avoid unnecessary uploads):
- Max. length of a file name: 400 characters
- Max. length of a folder name: 400 characters
- Invalid leading chars for file or folder name: SPACE
- Invalid trailing chars for file or folder name: SPACE, PERIOD
- Invalid file or folder names: "AUX", "PRN", "NUL", "CON", "COM0", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9", "LPT0", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9"
- The following characters are not allowed in file or folder names: " * : < > ? / \ |
- Max. length of a file path: 400 characters
- Max. size of a file: 250 GB
- Max. size of an attachment: 250 MB
To create a new SharePoint Online Importer, create a new importer and select SharePoint Online from the Adapter Type drop-down. Once the adapter type has been selected, the Parameters list will be populated with the parameters specific to the selected adapter type. Mandatory parameters are marked with an *.
The Properties of an existing importer can be accessed after creating the importer by double-clicking the importer in the list or selecting the Properties button/menu item from the toolbar/context menu. A description is always displayed at the bottom of the window for the selected parameter.
Multiple importers can be created for importing to different target locations, provided each importer has a unique name.
Configuration parameters | Values |
Name | Enter a unique name for this importer Mandatory |
Adapter type | Select SharePoint Online from the list of available connectors Mandatory |
Location | Select the Job Server location where this job should be run. Job Servers are defined in the Jobserver window. If no Job Server has been created by the user to this point, migration-center will prompt the user to define a Job Server Location when saving the Importer. Mandatory |
Description | Enter a description for this job (optional) |
The configuration parameters available for the SharePoint Importer are described below:
Configuration parameters | Values |
tenantName* | The name of your SharePoint Online Tenant Example: Contoso Mandatory There are several web site that explain how to determine a SharePoint Online tenant name, e.g. https://morgantechspace.com/2019/07/how-to-find-your-tenant-name-in-office-365.html |
tenantUrl* | |
siteName* | The path to your target site collection for the import. Example: /sites/My Site Mandatory |
appClientId* | The ID of either the migration-center Azure AD application or the SharePoint application. Example: ab187da0-c04d-4f82-9f43-51f41c0a3bf0 Mandatory |
appCertificatePath | The full path to the certificate .PFX file, which you have generated when setting up the Azure AD application. Example: D:\migration-center\config\azure-ad-app-cert.pfx Mandatory for Azure AD app |
appCertificatePassword | The password to read the certificate specified in appCertificatePath. Mandatory for Azure AD app |
appClientSecret | The client secret, which you have generated when setting up the SharePoint application (SharePoint app-only principal authentication). Mandatory for SharePoint app |
autoAdjustVersioning | Enable/Disable whether the lists/libraries should be updated to allow versions imports. Enabled: The importer will update the lists/libraries if needed. Disabled: In case the feature is needed in the import process but not allowed by the target lists/libraries and error will be thrown. |
autoAdjustAttachments | Enable/Disable whether the lists should be updated to allow attachments imports. Enabled: The importer will update the lists if needed. Disabled: In case the feature is needed in the import process but not allowed by the target lists/libraries and error will be thrown. |
autoAddContentTypes | Enable/Disable whether the lists/libraries should be updated to allow items with custom content type imports. Enabled: The importer will update the lists if needed. Disabled: In case the feature is needed in the import process but not allowed by the target lists/libraries and error will be thrown. |
autoCreateFolders | Enable/disable whether folders which do not exist should be created during import. Enabled: the importer created any specified folders automatically. Disabled: no new folders are created, but any existing folders (same path and name) are used. References to non-existing folders throw an error. |
proxyURL | This is the URL, which defines the location of your proxy to connect to the Internet. This parameter can be left blank if no proxy is used to connect to the internet. |
proxyUsername | The login of the user, who connects and authenticates on the proxy, which was specified in parameter proxyURL. Example: corporatedomain\username |
proxyPassword | Password of the proxy user specified above |
checkContentIntegrity | If checked the importer will check the integrity of each document that has a checksum generated by the scanner, by downloading a copy of the content after the import and comparing the two checksums. |
mipServiceConfigFile | Full path to the MIP service configuration file. Please see Preparation for applying Microsoft Information Protection sensitivity labels for more details. Example: c:\migration-center\my-mipservice.config |
loggingLevel* | Sets the verbosity of the log file. Values: 1 - logs only errors during scan 2 - is the default value reporting all warnings and errors 3- logs all successfully performed operations in addition to any warnings or errors 4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production) Mandatory |
Starting from migration-center 3.5 the SharePoint Online importer has the option of checking the integrity of each document’s content after it has been imported. This will be done if the “checkContentIntegrity” parameter is checked and it will verify only documents that have a generated checksum in the Source Attributes.
Currently the only supported checksum algorithm is MD5 with HEX encoding.
For certain types of documents such as Office documents and also .MSG documents, “Document Information Panel” is activated and SharePoint changes the content slightly upon upload. This will cause the integrity check to fail for those documents and there is no workaround so far, other than importing without the content integrity check or finding a way to disable this feature directly in SharePoint.
Documents targeted at a Microsoft SharePoint Online Document library will have to be added to a migration set. This migration set must be configured to accept objects of type <source object type>ToSPOnline(document).
Create a new migration set and set the <source object type>ToSPOnline(document) in the Type drop-down. This is set in the -Migration Set Properties- window which appears when creating a new migration set. The type of object can no longer be changed after a migration set has been created.
The migration set is now set to work with SharePoint Online documents.
The same procedure as for documents also applies to links about to be imported to SharePoint Online. For links the object type to select for a migration set would be <source object type>ToSPOnline(link).
The same procedure as for documents also applies to folders about to be imported to SharePoint Online. For folders the object type to select for a migration set would be <source object type>ToSPOnline(folder).
The same procedure as for documents also applies to lists or libraries about to be imported to SharePoint Online. For lists or libraries, the object type to select for a migration set would be <source object type>ToSPOnline(list).
The migration-center SharePoint Online Importer supports only the creation of lists or library!
The same procedure as for documents also applies to list items about to be imported to SharePoint Online. For list items the object type to select for a migration set would be <source object type>ToSPOnline(listItem).
<source object type>ToSPOnline(document) type objects have a number of predefined rules listed under Rules for system attributes in the -Transformation Rules- window. These rules are described in the table below.
Configuration parameters | Values |
contentType | Rule setting the content type. Example: Task Mandatory This value must match existing migration-center object type definitions; see paragraph 0 Object Type definition |
copyRoleAssignments | Specify if roles from the parent object are copied or not. This system rule is necessary only if the system rule roleAssignments is set, otherwise it is not used. By default, this value is set to false. |
fileExtension | Specify the file extension of the file name that is used to upload the content file to SharePoint. |
isMajorVersion | Specify, if the current object will be created as a major or minor version in the SharePoint Online target library. Example: TRUE or YES or 1 (check in as major version) FALSE or NO or 0 (check in as minor version) |
library | Specify the name of the library, where to import the current object Mandatory |
mc_content_location | Specify the location of a document’s content file.
|
parentFolder | Rule which sets the path for the current object inside the targeted Document Library item Example: /username/myfolder/folder Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted library, specify a forward slash “/” |
roleAssignments | Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Online Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. To set role assignments for single users, the value must match the following pattern: <loginname>;#roledefinitionname To set role assignments for SharePoint groups, the value must match the following pattern: @<group name>;#roledefinitionname To set role assignments for AD domain groups, you must specify the group's login name, which is: c:0o.c|federateddirectoryclaimprovider|<group guid> Examples: [email protected];#Read @Contributors;#Contribute c:0o.c|federateddirectoryclaimprovider|f919916d-15d1-4b13-8cbd-1973d4e50671;#Read |
sensitivityLabel | Name of the MIP sensitivity label to apply. See https://docs.microsoft.com/en-us/microsoft-365/compliance/sensitivity-labels?view=o365-worldwide for more details. |
site | Specify the target (sub-) site relative to your site collection, which was specified in the SharePoint Online Importer connector parameters. Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted site collection, specify a forward slash “/” |
Other rules can be defined by the user to set various SharePoint column values, such as Name, Title, checkin_comment to name a few of the more frequently used. Any SharePoint column which exists in the document library targeted by the contentType rule can be set by creating a rule and associating it with the corresponding column in the associations tab.
SharePoint Online supports documents with the file format ".url". We call these documents "link documents" since they do not have real content but just point to another document in SharePoint Online or to an external web site. The importer will create the necessary content of the link documents on the fly and save them in the default temporary files folder of the OS. Thus, the importer will ignore the content of any source object that is imported as a link document because it will import the generated link content instead.
Links have slightly different rules than documents. The exact system rules applicable to links are listed below.
Configuration parameters | Values |
contentType | Rule setting the content type. Example: Document Mandatory This value must match existing migration-center object type definitions. |
copyRoleAssignments | Specify if roles from the parent object are copied or not. This system rule is necessary only if the system rule roleAssignments is set, otherwise it is not used. By default, this value is set to false. |
isMajorVersion | Specify, if the current object will be created as a major or minor version in the SharePoint Online target library. Example: TRUE or YES or 1 (check in as major version) FALSE or NO or 0 (check in as minor version) |
library | Specify the name of the library, where to import the current object Mandatory |
parentFolder | Rule which sets the path for the current object inside the targeted Document Library item Example: /username/myfolder/folder Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted library, specify a forward slash “/” |
roleAssignments | Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Online Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. To set role assignments for single users, the value must match the following pattern: <loginname>;#roledefinitionname To set role assignments for SharePoint groups, the value must match the following pattern: @<group name>;#roledefinitionname To set role assignments for AD domain groups, you must specify the group's login name, which is: c:0o.c|federateddirectoryclaimprovider|<group guid> Examples: [email protected];#Read @Contributors;#Contribute c:0o.c|federateddirectoryclaimprovider|f919916d-15d1-4b13-8cbd-1973d4e50671;#Read |
site | Specify the target (sub-) site relative to your site collection, which was specified in the SharePoint Online Importer connector parameters. Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted site collection, specify a forward slash “/” |
url | The target URL of this link document. Can be a URL to another SharePoint Online document or a URL to an external web site. Example: https://contoso.sharepoint.com/sites/MySite/Documents/Test.docx Mandatory |
Folders have different rules than documents. The exact system rules applicable to folders are listed below.
Configuration parameters | Values |
contentType | Rule setting the content type. Example: Task Mandatory This value must match existing migration-center object type definitions. |
library | Specify the name of the library, where to import the current object. Mandatory |
name | Rule which sets the name of the current folder inside the targeted Document Library item Example: /my first folder Mandatory |
parentFolder | Rule which sets the path for the current object inside the targeted Document Library item. Example: /username/myfolder/folder Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted library, specify a forward slash “/”. |
copyRoleAssignments | Specify if roles from the parent object are copied or not. This system rule is necessary only if the system rule roleAssignments is set, otherwise it is not used. By default, this value is set to false. |
roleAssignments | Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Online Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. To set role assignments for single users, the value must match the following pattern: <loginname>;#roledefinitionname To set role assignments for SharePoint groups, the value must match the following pattern: @<group name>;#roledefinitionname Example: [email protected];#Read @Contributors;#Contribute |
site | Specify the target (sub-) site relative to your site collection, which was specified in the SharePoint Online Importer connector parameters. Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted site collection, specify a forward slash “/”. |
List items have different rules as documents. The exact system rules applicable to list items are listed below.
Configuration parameters | Values |
contentType | Rule setting the content type. Example: Task Mandatory This value must match existing migration-center object type definitions. |
isMajorVersion | Specify, if the current object will be created as a major or minor version in the SharePoint Online target library. Example: TRUE or YES or 1 (check in as major version) FALSE or NO or 0 (check in as minor version) |
library | Specify the name of the library, where to import the current object Mandatory |
parentFolder | Rule which sets the path for the current object inside the targeted Document Library item Example: /username/myfolder/folder Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted library, specify a forward slash “/” |
relationLinkField | Rule which sets a list of column names, where to insert links to AttachmentRelation associated with the current object. This rule can be left blank, if no links shall be inserted. |
copyRoleAssignments | Specify if roles from the parent object are copied or not. This system rule is necessary only if the system rule roleAssignments is set, otherwise it is not used. By default, this value is set to false. |
roleAssignments | Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Online Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. To set role assignments for single users, the value must match the following pattern: <loginname>;#roledefinitionname To set role assignments for SharePoint groups, the value must match the following pattern: @<group name>;#roledefinitionname Example:[email protected];#Read @Contributors;#Contribute |
site | Specify the target (sub-) site relative to your site collection, which was specified in the SharePoint Online Importer connector parameters. Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted site collection, specify a forward slash “/”. |
Lists have different rules as documents. The exact system rules applicable to lists are listed below.
Configuration parameters | Values |
baseTemplate | Rule setting the base template for this list. A list of valid values can be found in the appendix. Mandatory This value must match existing migration-center object type definitions. |
copyRoleAssignments | Specify if roles from the parent object are copied or not. This system rule is necessary only if the system rule roleAssignments is set, otherwise it is not used. By default, this value is set to false. |
roleAssignments | Specify role assignments for the current object. If a role definition is assigned to the current object, the migration-center SharePoint Online Importer breaks the role inheritance. It is possible to specify either a list of users and/or a list of SharePoint groups. If a group is specified, it is necessary, that the targeted SharePoint Group exists in your SharePoint Site. To set role assignments for single users, the value must match the following pattern: <loginname>;#roledefinitionname To set role assignments for SharePoint groups, the value must match the following pattern: @<group name>;#roledefinitionname Example:[email protected];#Read @Contributors;#Contribute |
site | Specify the target (sub-) site relative to your site collection, which was specified in the SharePoint Online Importer connector parameters. Mandatory This rule must contain a value. If the current object shall be imported on root level of the targeted site collection, specify a forward slash “/”. |
title | Specify the title of the list or library, which will be created. |
In order to associate transformation rules with SharePoint Online columns, a migration-center object type definition for the respective content type needs to be created. Object type definitions are created in the migration-center client. To create an object type definition, go to Manage/Object Types and create or import a new object type definition. In case your content type contains two or more columns with the same display name, you need to specify the columns internal names as attribute names.
Working with object type definitions and defining attributes is a core product functionality and is described in detail in the Client User Guide.
A Microsoft SharePoint Online content type corresponds to a migration-center object type definition. For the Microsoft SharePoint Online connector, an object type definition can be specified in four ways, depending on whether a particular SharePoint Online content type is to be used or not or multiple but different content types using the same name across site collections:
- My Content Type describes explicitly a SharePoint Online content Type named My Content Type defined either in the SharePoint document library or at Site Collection level
- @My Document Library describes the SharePoint Online document library named My Document Library using only columns, which are defined explicitly in the SharePoint Online document library My Document Library
- My Content Type;#Any Custom Value describes a SharePoint Online content type named My Content Type. Everything after the delimiter ;# will be cut off on importing
- @My Document Library;#Any Custom Value describes the SharePoint Online document library named My Document library using only columns, which are defined explicitly in the SharePoint Online document library My Document Library. Everything after the delimiter ;# will be cut off on importing.
The migration-center SharePoint Online importer is able to modify the following SharePoint Online specific internal field values for documents, list items and folders: Created By, Modified By, Created and Modified. To set these internal field values it is necessary to create attributes named Author, Editor, Created and Modified and associate them in the transformation rules appropriately.
Importing lists or libraries into SharePoint Online is a little bit different than for documents, folders or list items, since the migration-center SharePoint Online Importer sets Microsoft defined attributes, which are not visible to the user. In order to set those attributes, it is necessary to create object type definitions for each type of list template (see List of possible list templates). The SharePoint Online Connector is able to set any possible attribute of a list. See List of possible list attributes for more information about possible attributes.
The SharePoint Online Importer is able to read values from files. This might be necessary if the length of a string might exceed the maximum length of an oracle database column, which is 4,096 bytes.
To tell the SharePoint Online Importer reading Strings from a text file, the filepath of the file, containing the value must be placed within the markup <@MCFILE>filepath</@MCFILE>.
The File Reader does not read values from files for the following attributes: Check-In Comment, Author, Editor, Creation Date, Modified Date.
Example:
Assuming you have a file at path \\scanlocation\temp\1\4a9daccf-5ace-4237-856c-76f3bd3e3165.txt you must type the following String in a rule:
<@MCFILE>\\scanlocation\temp\1\4a9daccf-5ace-4237-856c-76f3bd3e3165.txt</@MCFILE>
On Import the SharePoint Online Importer extracts the contents of this file and adds them to the associated target attribute.
The SharePoint Online Importer is able to modify the filename of a file, which is imported to SharePoint Online. To set the filename, you must create a target attribute named Name in the object type definition. You can associate any filename (without an extension) for a document. The importer automatically picks the extension of the original file or the file specified in rule mc_content_location. If the extension needs to be changed as well, use the system rule fileExtension in order to specify a new file extension. You can also change only the extension of a filename by setting fileExtension and not setting a new filename in the Name attribute.
Example:
mc_content_location | Name | fileExtension | Result filename |
- | - | - | content.dat |
\\Fileshare\Migration\Conversion\invoice.pdf | - | - | invoice.pdf |
- | MyContent | - | MyContent.dat |
- | MyContent | pdf | MyContent.pdf |
- | - | pdf | content.pdf |
The SharePoint Online Importer is able to set the check-in comment for documents. To set the check-in comment, you must create a target attribute named checkin_comment in the object type definition. You can associate any string to set the check-in comment
The SharePoint Online Importer is able to set the author for list items, folders and documents. To set the author, you must create a target attribute named author in the object type definition. The value of this attribute must be the loginname of the user, who shall be set as author.
The SharePoint Online Importer is able to set the editor for list items, folders and documents. To set the editor, you must create a target attribute named editor in the object type definition. The value of this attribute must be the loginname of the user, who shall be set as editor.
The SharePoint Online Importer is able to set the creation date for list items, folders and documents. To set the creation date, you must create a target attribute named created in the object type definition. The value of this attribute must be any valid date.
The SharePoint Online Importer can set the modified date for list items, folders and documents. To set the modified date, you must create a target attribute named modified in the object type definition. The value of this attribute must be any valid date.
The SharePoint Online Importer can set lookup values. If you want to set a lookup value, you can either set the ID of the related item or the title. If you set the title of the document and there is more than one item with the same title in the lookup list, the SharePoint Importer marks your import object as an error, because the lookup item could not be identified unequivocally.
The importer will treat any numeric value provided as the ID of the lookup value. In case you want to look up the value by title and the title is numeric, please surround the numeric value with " characters. The importer will automatically remove any " characters in the provided value before trying to match it with the title of the lookup field.
The SharePoint Online Importer can set URL values. You can define a friendly name as well as the actual URL of the link by concatenating the friendly name with ;# and the actual URL.
Example: migration-center;#http://www.migration-center.com
The SharePoint Importer can set taxonomy values and Migration Center provides two ways to do it. The first possibility is to use the name of the Taxonomy Service Application, a term group name, a term set name and the actual term. Those four values must be concatenated like the following representation:
TaxonomyServiceApplicationName>TermGroupName>TermSetName>TermName
Example: Taxonomy_+poj2l53kslma4==>Group1>TermSet 1>Value
The second possibility is to use the unique identifier of the term within curly brackets {}.
Example: {2e9b53f9-04fe-4abd-bc33-e1410b1a062a}
Multiple values can be set also but the attribute has to be set as a Multi-value Repeating attribute.
In case you receive an error on setting taxonomy values make sure the TaxonomyServiceApplicationName is up to date and valid since Microsoft changes the identifier at certain times.
The performance of the import process is heavily impacted by some specific features, namely autoAdjustVersioning, autoAdjustAttachments, autoAddContentTypes, autoCreateFolders, setting taxonomy values, and setting system attributes like Author, Editor, Created, and Modified.
In case you are importing an update for a major version the increase can get up to three times compared to a normal document import. Combining all the above-mentioned features over the same document can increase the time for up to four times. Take this into consideration when planning an import as the time might vary based on the above described features.
You will achieve the highest import performance if you perform the appropriate configuration of your SharePoint Online system before you start the import and disabled the above-mentioned features in the SharePoint Online importer.
Template Name | Template Identifier | Description |
GenericList | 100 | A basic list which can be adapted for multiple purposes. |
DocumentLibrary | 101 | Contains a list of documents and other files. |
Survey | 102 | Fields on a survey list represent questions that are asked of survey participants. Items in a list represent a set of responses to a particular survey. |
Links | 103 | Contains a list of hyperlinks and their descriptions. |
Announcements | 104 | Contains a set of simple announcements. |
Contacts | 105 | Contains a list of contacts used for tracking people in a site. |
Events | 106 | Contains a list of single and recurring events. An events list typically has special views for displaying events on a calendar. |
Tasks | 107 | Contains a list of items that represent completed and pending work items. |
DiscussionBoard | 108 | Contains discussions topics and their replies. |
PictureLibrary | 109 | Contains a library adapted for storing and viewing digital pictures. |
DataSources | 110 | Contains data connection description files. - hidden |
XmlForm | 115 | Contains XML documents. An XML form library can also contain templates for displaying and editing XML files via forms, as well as rules for specifying how XML data is converted to and from list items. |
NoCodeWorkflows | 117 | Contains additional workflow definitions that describe new processes that can be used within lists. These workflow definitions do not contain advanced code-based extensions. - hidden |
WorkflowProcess | 118 | Contains a list used to support execution of custom workflow process actions. - hidden |
WebPageLibrary | 119 | Contains a set of editable Web pages. |
CustomGrid | 120 | Contains a set of list items with a grid-editing view. |
WorkflowHistory | 140 | Contains a set of history items for instances of workflows. |
GanttTasks | 150 | Contains a list of tasks with specialized Gantt views of task data. |
IssueTracking | 1100 | Contains a list of items used to track issues. |
Attribute | Type | Description |
ContentTypesEnabled | Boolean | Gets or sets a value that specifies whether content types are enabled for the list. |
DefaultContentApprovalWorkflowId | String | Gets or sets a value that specifies the default workflow identifier for content approval on the list. Returns an empty GUID if there is no default content approval workflow. |
DefaultDisplayFormUrl | String | Gets or sets a value that specifies the location of the default display form for the list. Clients specify a server-relative URL, and the server returns a site-relative URL |
DefaultEditFormUrl | String | Gets or sets a value that specifies the URL of the edit form to use for list items in the list. Clients specify a server-relative URL, and the server returns a site-relative URL. |
DefaultNewFormUrl | String | Gets or sets a value that specifies the location of the default new form for the list. Clients specify a server-relative URL, and the server returns a site-relative URL. |
Description | String |