Consensus and Golden Set Tasks

Using Data Versioning for Advanced Workflows

1. Setup your dataset, and clone it multiple times according to the number of copies needed. Read this for instructions on how to clone a dataset.
2. Create 1 task per dataset, and assign it to a contributor / annotator, or upload your model inferences. Read this to learn how to create a task (WEB/SDK)

3. When ready to continue (annotators finished their work or model inferences uploaded), merge together all the relevant datasets. Read here to understand more about merging datasets.

Consensus Workflow

Consensus workflow is about assigning multiple annotators with the same task, and then comparing their work and measuring the differences. Consensus is used to:
Benchmark annotators - by comparing their work with known ‘ground truth’
Benchmarking model performance - comparing model inferences with ‘ground truth’
Validate recipe (ontology and instructions) - Measuring differences between annotators indicates clarity and usability of recipe

When cloning a dataset, tasks can be run simultaneously over different instances, and assigned to different annotators and models. When everyone is done, the Datasets are merged back, and each item copy holds annotations of all sources.
To achieve consensus workflow follow the instructions provided hereinabove, and export the data of the newly merged dataset, containing data from all sources, or review individual images by double-clicking them in the dataset browser, to see them in the Annotation Studio.

Chained Annotation Tasks / Micro-tasking

The smaller the annotation task, the easier it is for an annotator to understand ontology and instruction, focus on specific object types, and the higher success rate.
Micro-tasking is achieved in the Dataloop system by creating multiple versions (cloning) of the same dataset, and running a separate annotation-task on each, with a separate recipe (ontology and instructions).
To achieve micro-tasking or sequenced / gained annotations tasks workflow follow the instructions provided hereinabove, and export the data of the newly merged dataset, containing data from the different annotation tasks.

Additional QA / QC Tasks

Some tasks requires an extra step of QA or QC, for example validation by domain experts, and usually for a subset of the data (for example - 10%).
Additional QA or QC is achieved in the Dataloop system by cloning the completed dataset (completely, or by filtering it in the dataset browser and cloning the filtered Items) and opening a QA task from the dataset browser.

Use our versioning feature to performe several QC (Quality control) tasks from the dataset page.

Versioning Limitations

Same recipe

• Cloned datasets - items, annotations and metadata will be merged. This means that you will see annotations from different datasets on the same item.
• Not cloned datasets - items will be summed up and will cause duplicates of similar items.

Different recipes

You cannot merge datasets with different recipes, switch recipe to match the recipes. Do so by clicking the three dots on the right side of a dataset and clicking "Switch Recipe " to switch the dataset's recipe to a different one.

Concensus 100%

Assign the same dataset & recipe to different annotators to test if the data is "labelable" and the accuracy of the recipe.

1. Clone your tested dataset

Clone the dataset you wish to test, the same recipe will be linked to both the master and all of the cloned datasets.
Repeat this step for every annotator, we recommand to have at least three.

Click the three dots on the right side of a dataset and click "Clone Dataset"

A dialog box will appear. Fill in the cloned dataset name, and de-select annotations and metadata, to clone only the items (binary level).

2. Create Tasks and Redistribute them between annotators

Create a task for each cloned dataset and assign them to the annotators.
Follow "Create a Task" (WEB/SDK) and "Redistribute a Task"(WEB/SDK) pages, to create and redistribure tasks with all of the items.

3. Annotators Assignments

After the annotators will annotate and complete each item on the assignment they will classify it as "complete" (or "discard" for Items that are not relevant for labeling) more on Task Assignment page.

4. Merge Cloned Datasets

Back in the dataset page merge all of the relevant datasets by clicking the box to select all of the datasets, or hover over the empty space and click the shown boxes to select specific ones.

After selecting the datasets and clicking "merge" a dialog box will appear.

Fill in the merged dataset name and select items annotations and metadata to merge with the information filled in by the annotators.

5. Compare Results

See the annotations from the different annotators on the studio.
Enter the annotataion studio, by clicking on the dataset page or project dashboard, and clicking on the “BROWSE” icon of a dataset. On the dataset browser press on any of the items to view the results.

Partial Concensus <100%

This is the same as the full concensus, but on a sample of the data to save time.
The only difference is with step 2, where you create a task with a part of the data by one of two ways:

1. Filter

Filter items you want as a part of your task using the filter tab.
For example, to filter items by folder directory, press directories and select by the dropdown list or by inputting the name of the directory (starting with “/”) and the auto-complete feature will help you find the relevant directory.
Multiple directories can be selected to show aggregation from all selected directories.

2. Limit Items

Arbitrary limit the number of items by the number of items or by precentage when creating the task.

Majority Vote

Use the consensus (or partial consensus) flow to test annotators majority.
Check the annotators agreenent on annotations. If the agreement is low, create another clone task.

Golden Set

Duplicate a "Golden Set" (Ground Truth) dataset without annotations and assign it to an annotator you wish to test. Compare the annotators results with the golden set to check the annotator's accuracy

1. Clone the Golden Set Dataset

Clone the golden datset, the same recipe will be linked to all of the both datasets.
Repeat this step for every annotator, we recommand to have at least three.

Click the three dots on the right side of the dataset and click "Clone Dataset"

A dialog box will appear.Fill in the cloned dataset name, and de-select annotations and metadata, to clone only the items (binary level).

2. Create a Task and Assign it to the Tested Annotator

Create a task from the cloned dataset and assign it to the annotator.
Follow "Create a Task" (WEB/SDK) and "Redistribute a Task"(WEB/SDK) pages, to create and redistribure tasks with all of the items.

3. Annotators Assignments

After the annotators will annotate and complete each item on the assignment they will classify it as "complete" (or "discard" for Items that are not relevant for labeling) more on Task Assignment page.

4. Merge Cloned and "Golden Set" Datasets

Back in the dataset page merge all the cloned and "Golden Set" satasets by hovering over the empty space and clicking the shown boxes to select.

After selecting the datasets and clicking "merge" a dialog box will appear.

Fill in the merged dataset name and select items annotations and metadata to merge with the information filled in by the annotators.

5. Compare Results

See the annotations from the different annotators on the studio.
Enter the annotataion studio, by clicking on the dataset page or project dashboard, and clicking on the “BROWSE” icon of a dataset. On the dataset browser press on any of the items to view the results.