Training the machine learning model

About this task

By exploring the performance metrics, you can identify ways to improve the model's accuracy.

Restriction: Only three machine learning models can be trained at a time per Knowledge Studio instance. If your instance contains multiple workspaces and the number of machine learning models that are being trained in other workspaces totals 3 already, then your request to train the machine learning model in your workspace will be queued until the other training processes are done.

Procedure

To create a machine learning model:

1. Log in as a Knowledge Studio administrator and select your workspace.

2. Select Machine Learning Model > Performance.

3. Verify that all of the document sets have been approved and that all annotation conflicts have been resolved through adjudication. Only documents that have become ground truth through adjudication or approval can be used to train the model.

4. Click Train and evaluate.

5. Optional: To specify how you want to allocate documents from your document sets to be used by the system-level training, test, or blind sets, click Edit settings.

   See Document set management for help determining which ratios to apply.

6. Click Train to train the model, or click Train & Evaluate to train the model, evaluate annotations added by the machine learning model, and analyze the performance statistics.

   Important: Training a machine learning model can take several minutes or several hours, depending on the number of human annotations that exist and the total number of words across all documents.

7. Select the document sets that you want to use for training the model.

   Note: The document sets must contain at least 10 annotated documents.

8. After the model is created, select one of the following actions:

Procedure

To evaluate the annotations added by the model:

1. Select Machine Learning Model > Performance > Train and evaluate. The Training/Test/Blind Sets page is displayed.
2. Click View Ground Truth for the training set or test set to see the annotations that were added through pre-annotation and by human annotators. The ground truth editor opens. Click to open individual documents and see how the mentions, relations, and coreferenced mentions were annotated.
3. On the Performance page, click View Decoding Results to see the annotations that the machine learning model added to documents in the test set. This button is available only after you evaluate the model. By viewing results, you can see how well the machine learning model labeled mentions, relations, and coreferenced mentions in the test data.
4. If you want to change how the documents are divided between training, test, and blind data sets, click Performance > Train and evaluate > Edit Settings. For example, if initial results seem acceptable, you might want to increase the number of documents in the test set to further test the machine learning model's results. You can change the ratio for how documents are automatically divided for different purposes, or you can select specific document sets to use as training data, test data, and blind data.
5. If you made any changes, click Train & Evaluate to retrain the model and re-evaluate the annotations.