Amazon Comprehend Enrich Scanner
Last updated
Last updated
The Amazon Comprehend Enrich Scanner is one of the source adapters available in migration-center starting with version 3.17. It is a special adapter which enhances the objects scanned by another source adapters with some data computing using the Amazon Comprehend. The supported Comprehend classifiers: Dominant Language Classifier, Entities Classifier and Custom Classifier.
The scanner module works as a job that can be run at any time and can even be executed repeatedly. For every run, a detailed history and log file are created.
A scanner is defined by a unique name, a set of configuration parameters and an optional description.
Amazon Comprehend Enrich Scanners can be created, configured, started, and monitored through migration-center Client, but the corresponding processes are executed by migration-center Job Server.
The scanner will extract text in enrich mode even if the scanner was run in simulation mode before.
The entities and language classifiers can be run in the same scan run, but the entities classifier will not take into consideration the language extracted by the dominant language classifier. The attribute generated by the language classifier can be used by the entities classifier if it will be run before the entities classifier. More information about the way to use the source attribute as entities language attribute is presented in Classifiers Configuration.
To create a new Amazon Comprehend Enrich Scanner job click on New Scanner button and select "AmazonComprehendEnrich" from the adapter type dropdown list. Once the adapter type has been selected, the parameters list will be populated with the Amazon Comprehend Enrich Scanner parameters.
The Properties window of a scanner can be accessed by double-clicking the scanner in the list or selecting the Properties button or entry from the toolbar or context menu.
Configuration parameters | Description |
Name | Enter a unique name for this scanner Mandatory |
Adapter type | Select the “CSV/Excel” adapter from the list of available adapters Mandatory |
Location | Select the Job Server location where this job should run. Job Servers are defined in the Jobserver window. If no Job Server was selected, migration-center will prompt the user to define a Job Server Location when saving the scanner. Mandatory |
Description | Enter a description for this scanner (optional) |
Configuration parameters | Description |
publicKey | The Amazon public key used to create a connection to AWS. |
privateKey | The Amazon private key. This should be the pair of the public key. |
region | The Amazon region used to create the connection to AWS. |
executeClassifiers | Flag indicating if the classifiers will be executed. If this parameter is not checked then the scanner will run in Simulation mode, otherwise, the classifiers jobs will be fired in Comprehend. |
inputS3Uri | The S3 location where the text files will be uploaded. |
outputS3Uri | The S3 location where the output of the classifier will be located. |
kmsKeyId | The ARN of custom managed key used to encrypt the data in S3. Example: arn:aws:kms:eu-central-1:0908887578777:key/d484ee92-ffff1-444e-bcbb0-7cccceffcffc |
dataAccessRoleArn | The ARN of the role that has Comprehend as trusted entities. Example: MCDEMO_Comprehend |
deleteFiles | Flag indicating if the files from S3 will be deleted. If the parameter is checked then the files from inputS3Uriand outputS3Uri will be deleted. |
jobRunId* | The id of the job which scanned the objects that will be enriched.The jobRunId must exist. Mandatory |
configurationFile | The location of the file where the classifiers are configured. When the executeClassifiers parameter is checked, then this parameter is mandatory. The way to configure the classifiers is detailed in Classifiers Configuration. |
loggingLevel* | Sets the verbosity of the log file. Values: 1 - logs only errors during scan 2 - is the default value reporting all warnings and errors 3 - logs all successfully performed operations in addition to any warnings or errors 4 - logs all events (for debugging only, use only if instructed by fme product support since it generates a very large amount of output. Do not use in production) Mandatory |
The classifiers are configured using an XML file. The structure of this file is a predefined one and allows the user to configure the classifiers as much as possible.
The supported classifiers are divided into two types: standard classifiers and custom classifiers. There is a predefined XML structure for each classifier type. An example of this configuration file can be found in \fme AG\migration-center Server Components <Version>\lib\mc-aws-comprehend-scanner\classifiers-config.xml.
For every classifier, you can specify if the score should be displayed by using the XML attribute "dispayScore". You need to specify this attribute just if you want to have the score as an attribute in migration-center otherwise, the attribute can be omitted because the default value is false.
The standard classifiers are split into two supported classifiers and the difference between them is made using an XML attribute named "type".
Dominant Language Classifier
The structure of this classifier is presented in the following block. The "threshold" XML element is mandatory and is used to filter the values. If the score for a specific language is lower than the threshold value then the language is not saved on database.
Entities Classifier
The structure for the entities classifier is presented in the following block.
The XML sub-elements are:
threshold - is used to filter the entities. If the entity score is less than the threshold value then the entity will not be saved in the database.
language - is a mandatory parameter used to specify the language of the documents. If the user has documents with different languages then the user is allowed to use a source attribute to specify the language. The attribute name should be prefixed with $ character, eg. $aws_language.
entityRecognizerArn - is used to specify the custom entity classifier instead of the standard one.
entities - specify the entities that will be saved on the database. If the entity is not present in the entities list, then the attribute will be ignored by the scanner.
The Custom Classifier is used to classify documents using custom created categories. The scanner allows users to use multiple custom classifiers in the same scan run.
The XML sub-element "classifierEndpointArn" is mandatory and specifies the Amazon Resource Names of the custom classifier.
The "threshold" sub-element is to filter the classes. If the class score is lower than the provided value for the threshold, then the attribute will not be saved on the database. The attribute name on the database will be "aws_className_awsJobId".
The Amazon Comprehend Enrich Scanner can be run in two modes: simulation mode and enrich mode.
To extract the text in both cases the scanner uses Tika and Tesseract for OCR. The OCR is disabled by default, but it can be activated by the user. More information can be found in chapter Tika Configuration and Tesseract OCR Configuration.
We recommend you to run the scanner in simulation mode to analyze the cost before running it to extract the Comprehend attributes.
The parameter "executeClassifiers" should be not checked when you want to run the scanner in simulation mode. To see the information generated by the scanner, the parameter "loggingLevel" should be set to 3 or 4.
The scanner extracts the text from documents locally and computes the number of characters and units to help the user to estimate the cost of classifiers execution.
The information generated during execution is present in the report log. An example of a report log is present in the following image.
To run the scanner in enrich mode you need to check the parameter "executeClassifiers".
The first step that the Amazon Comprehend Enrich Scanner does is to extract locally the text from documents. After that, the text files are uploaded to S3 on inputS3Uri. The classifiers jobs are fired and the result of those are saved in S3 on outputS3Uri. The scanner downloads the files and saves the results on database.
The following image presents the attributes in the database after one scan run on enrich mode with standard entities classifier and dominant language classifier.
The Tika library is used by the scanner to extract the text from documents.
The scanner provides a tika configuration file that contains all necessary parsers to extract the text from all office documents. The user can modify the configuration file if more tunings are wanted. The file is located on \fme AG\migration-center Server Components <Version>\lib\mc-aws-comprehend-scanner\tika-config.xml.
The "OOXMLParser" is used for office documents like docx and the "PDFParser" is used for pdf documents. The default configuration provided by the Tika library will be used for other documents type.
More information about the configuration can be found at https://tika.apache.org/1.26/configuring.html.
The Tesseract OCR is used to extract the text from the embedded images and also from the image file. The documentation of this library is https://tesseract-ocr.github.io/tessdoc/Home.html.
To install this library you can follow the article: https://medium.com/quantrium-tech/installing-and-using-tesseract-4-on-windows-10-4f7930313f82. The executable file can be download from https://sourceforge.net/projects/tesseract-ocr-alt/files/ or https://digi.bib.uni-mannheim.de/tesseract/.
After you installed the Tesseract you need to complete the TesseractOCRConfig.properties file with tesseractPath and tessdataPath. Example:
By default, the Tesseract is disabled. If the user wants to enable the Tesseract, the following steps should be followed:
Open tika-config.xml and remove from DefaultParser the line <parser-eexclude class="org.apache.tika.parser.ocr.TesseractOCRParser"/>
Change the value of the ocrStrategy XML element of PDFParser with ocr_and_text.
For configuring some additional parameters that will apply to all scanner runs, a configuration file (internal-configuration.properties) provided in the folder …\lib\mc-aws-comprehend-scanner. The following settings are available:
Configuration name | Description |
waiting_time_between_requests | The time in seconds that the scanner will wait until it will make a request to Amazon to get the Comprehend Classifier Job status. Example: waiting_time_between_requests=10 means that the scanner will make a request and if the status is "in progress" then the scanner will wait 10 seconds until it will make another request to check the status |
A complete history is available for any Amazon Comprehend Enrich Scanner job is available from the respective items’ History window. It is accessible through the History button/menu entry on the toolbar/context menu. The History window displays a list of all runs for the selected job together with additional information, such as the number of processed objects, the start and ending time and the status.
Double clicking an entry or clicking the Open button on the toolbar opens the log file created by that run. The log file contains more information about the run of the selected job:
Version information of the migration-center Server Components the job was run with
The parameters the job was run with
Execution Summary that contains the total number of objects processed for every classifier and errors that occurred during runtime.
Log files generated by the Amazon Comprehend Enrich Scanner can be found in the Server Components installation folder of the machine where the job was run, e.g. …\fme AG\migration-center Server Components <Version>\logs
The amount of information written to the log files depends on the setting specified in the ‘loggingLevel’ start parameter for the respective job.
