Annotation and QA tasks

Annotation Task

Annotation task: A task of labeling contents, such as text, audio, images, video, etc. to efficiently organize the dataset. The machine learning models are used to recognize this annotation and make predictions.

Create an Annotation Task

To create an annotation task, follow the instructions for each section:

1. Open the Data Management from the left side panel.
2. Select and open the dataset that you need to create an annotation task.
3. Click Actions or Down-arrow.
4. Select Annotation Tasks > Create New Task.

Section 1. General

4. Enter or select the required details in the General section:
   1. Task Name: Enter a name for the new task. By default, the project name + (total number of tasks + 1) is displayed.
      For example, if the project name is abc and the total number of tasks you have already is 5, then the new task name is abc-6.
   2. Task Type: Select the Labeling type.
   3. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
   4. Status: By default, the To Do status is displayed, and it cannot be changed.
   5. Priority: Select a priority from the list. By default, Medium is selected.
   6. (Optional) Completion Due Date: Select a task's due date from the calendar.
5. Click Next: Data Source.

Section 2. Data Source

6. Enter or select the required details in the Data Source section.

   1. Select Dataset: By default, the selected dataset name is displayed, click on it to select a different dataset. The Dataset field is disabled, if you select any particular item(s) in the Dataset.

      Note:

      You cannot create a task with a dataset that contains items 80,000 or above. To use this dataset, sampling must be done or replaced with another dataset. You can view the number of total items on the top-right side of the page.

   2. (Optional) Filters: Refine data selection by selecting specific folders, using DQL filters, or sub-sampling (randomly and equally distributed). The Folder or DQL field is Active only if you do not select any items in the Dataset.

      1. Folders: Select a folder from the dataset if available.
      2. Selected Filters / Saved DQL Query: Select a filter or saved DQL query from the list, if available.
      3. Data Sampling: Enter the Percentage or Number of Items for the task. For example, if the percentage is 100% for four items, then 75% is for three items from the selected dataset.

7. (Optional) Webm Conversion: By default, Enforce WEBM conversion of video items for frame-accurate annotations is selected.

8. Click Next: Instructions.

Section 3. Instructions

9. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.

   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. Labeling Instructions (.pdf): The labeling instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction. You can select the page range accordingly.

10. Click Next: Statuses. The Statuses section is displayed.

Section 4. Statuses

11. By default, the Completed status is selected. Click Add New Status to add a new status.
12. Click Next: Assignments.

Section 5. Assignments

13. Enter or select the required details in the Assignments section.
    1. Allocation Method: Select one of the following allocation methods:
       1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if needed: Pulling batch size (items) and Max items in an assignment.
       2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
    2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned users list.
    3. Assigned Users:
       1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
       2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

14. Click Next: Quality.

Section 6. Quality (Optional)

Enable advanced quality monitoring options to ensure data quality and review performance. The Quality task section is Not available for the Pulling Allocation method.

15. Select the quality task type and proceed to customize its properties. By default, None is selected. Select the following types as needed:
    1. Consensus: The Consensus task is to create replicas of the items for simultaneous work by multiple annotators and generate majority-vote datasets. For more information, see Consensus.
    2. Qualification: The Qualification task is used to create a dataset from multiple annotators. For more information, see Qualification.
    3. Honeypot: The Honeypot task is used to create a dataset from multiple annotators. For more information, see Honeypot.
16. Click Create Task. A confirmation message is displayed.

Edit an Annotation Task

1. Open the Annotation Workflow page from the left-side portal menu.
2. Identify the annotation task that is to be edited, and click on the Details.
3. Click Task Action Down Arrow.
4. Select Edit Task from the list.
5. Select the required section and make the changes. For more information about each section, see Create an Annotation Task.
6. Click Save Changes.

Add Items to an Existing Annotation Task

Before adding items to an existing task, you may select the items you wish to add by clicking on them (CTRL+click to select multiple items). If you do not select any items, you can choose to filter the items with a DQL query or add all items that are not already included in the task.

1. Open the Data Management page.
2. Identify and open the Dataset.
3. Select the required items from the dataset.
4. Click Actions or Down-arrow.
5. Select Annotation Tasks > Add to an existing task from the list.
6. In the Select Task section, select the task to which you need to add items.
7. Click Next: Data Source.
8. In the Data Source section, edit the Filters (Optional) details.
9. Click Next: Assignments.
10. In the Assignments section, add contributors from the Available Contributors list to the Assigned Contributors list.
11. Click Add Items.

QA Task

The task of reviewing annotation work is done as an annotation task. It has the option to flag annotations as having an 'issue' and send them for correction by the original annotator.

To learn about the QA process, see QA Process.

Create a QA Task from an Annotation Task

To create a QA task, follow the instructions for each section:

1. Open the Annotation Workflow page from the left-side portal menu.
2. Click Create Task.

Section 1. General

3. Enter or select the required details in the General section:
   1. Task Name: By default, your task name - QA is displayed. Modify, if needed.
   2. Task Type: Select QA as task type.
   3. Owner: By default, the current user's email ID is displayed. Click on it to select a different owner from the list.
   4. Status: By default, the To Do status is displayed, and it cannot be changed.
   5. Priority: Select a priority from the list. By default, Medium is selected.
   6. (Optional) Completion Due Date: Select a task's due date from the calendar.
4. Click Next: Data Source.

Section 2. Data Source

6. Enter or select the required details in the Data Source section.

   1. Select Dataset: By default, the dataset used to create the task is displayed.
   2. (Optional) Filters: Refine data selection by selecting specific folders, using DQL filters, or sub-sampling (randomly and equally distributed). The Folder or DQL field is Active only if you do not select any items in the Dataset.
      1. Folders: Select a folder from the dataset.
      2. Selected Filters / Saved DQL Query: Select a filter or saved DQL query from the list.
      3. Data Sampling: Enter the Percentage or Number of Items for the task. For example, if the percentage is 100% for four items, then 75% is for three items from the selected dataset.

7. Click Next: Instructions.

Section 3. Instructions

8. Enter or select the required details in the Instructions section. The number of Labels and Attributes is displayed on the top-right side of the page.

   1. Recipe: By default, the default recipe is displayed. Select a recipe from the list, if needed.
   2. QA Instructions (.pdf): The QA Instruction document is displayed, if available. Go to the Recipe section to upload a PDF instruction.

9. Click Next: Statuses. The Statuses section is displayed.

Section 4. Statuses

10. By default, the Approved status is selected. Click Add New Status to add a new status.
11. Click Next: Assignments.

Section 5. Assignments

12. Enter or select the required details in the Assignments section.

    1. Allocation Method: Select one of the following allocation methods:
       1. Pulling: The pulling distribution method means that annotators only pull a batch of items at a time and the maximum number of items in an assignment. You can make changes in the following fields if needed: Pulling batch size (items) and Max items in an assignment.
       2. Distribution: The distribution allocation method means that the items will be distributed in advance among users, equally or based on a custom percentage.
          1. The Auto Distribution option distributes the task equally among the users. By default, it is checked.
          2. The Show only unassigned users to any labeling task option allows existing users to complete their task.
    2. Available Users: Search for or select users from the list, and click the Forward arrow icon to add them to the Assigned Users list.
    3. Assigned Users:
       1. Search for or view the assigned users from the list. The allocation percentage is equally distributed if you select Auto Distribution.
       2. Select and click the Backward arrow icon to remove them from the Assigned Distribution list.

13. Click Create Task.

Edit a QA Task

1. Open the Annotation Workflow page from the left-side portal menu.
2. Identify the QA task that is to be edited and click on the Ellipsis icon.
3. Select Edit Task from the list.
4. Select the required section and make the changes. For more information, see the Create a QA task topic.
5. Click Save.