Data Filtering

Filters are part of the Dataset browser, enabling users to filter items based on item attributes, item metadata, and user metadata.

To add a filter to the query, follow the instructions:

1. Click on the required filter option.
2. Select or enter related values, and applying it.
3. Click Submit Filter. Details of the different filters are explained hereunder.

For example: Filtering for items with Annotated status that includes annotations (of any type) with labels Person and Food.

Filters Operands

The following operators are applied in and between filters unless otherwise specified when manually modifying a DQL query.

Cross Filter Operand

A relation between multiple filters in a single query is based on the AND operand.
For example: Filtering by Annotated status AND john@doe.ai user AND annotation type: Box, will return items that are annotated, have Box annotation and have john@doe.ai as an annotator in one or more of the annotations.

Inter-Filter Operand

The operand relation between multiple values in a specific filter is based on OR operand.
For example: filtering by labels filter with values Person and Dog will return all items with annotations of one of these labels, and not necessarily both.

Filter Options

To simplify the user-interface, it is possible to show/hide filters.

1. From the Ellipsis (3-dot actions) icon, select Edit filter options from the listy.
2. Uncheck any unnecessary filters.
3. Click Apply.

Save and Reuse Filters

• Press the DQL button to open the DQL editor and see the current query based on the filters used, if any. It is possible to manually edit the query and execute it.

• To save a query:
  • Press the Ellipsis (3-dot action) icon
  • Select Save as from the menu.
  • Enter a name for the new query.
  • Click Save.
• To reload a previously saved query:
  • Click the DQL drop-down.
  • Select a query by its name.
• To run the reloaded query, press the Execute button at the bottom of the filter side panel.

Filter Types

Item Status

There are two meanings to item status:

1. Changes to its metadata implying a change made to the item (or not):
       1. Annotated: Annotations created and saved to the items' JSON.
       2. Described: The Item description is added.

2. An actual status set on the item during an annotation or QA tasks
       1. Has status/no status - This applies to having any status whatsoever on the item. An item with no status is typically an item that has not been worked on at all.
       2. Discard: The item was discarded at some point.
       3. Complete: The Item was completed on at least one annotation task.
       4. Approved: The Item was approved on at least one QA task.

Labels

Allows filtering items that have one or more labels added to the query criteria.

1. Click a label from the list to add it to the filter.
2. Click it again to remove it.
3. Use the search box to find labels and/or sub-labels.
4. Collapse the filter to apply the selected labels.

Users

Allows filtering items annotated by the selected User(s).
 1. Click on the Users filter to expand it.
 2. Click on the Search bar, and search for and select the user(s) from the list.
 4. Click again on the selected users to exclude them from the search criteria.
 5. Click Remove All to remove all users at once from the search criteria.
 6. Collapse the filter to conclude that users' avatars are added as an indication of selection.
 7. Click Submit filter.
 
 

Item Uploaded Date

Use either the From or To date filter options (to/from) or both to refine search criteria.

• Select a date in the From… field. It returns all items uploaded to the platform from the selected date.
• Select a date in the To… field. It returns all items uploaded to the platform before the selected date.
• Click Apply to apply the filter criteria.
• Click Reset to remove the date selection.

File Name

Allows filtering for items with filenames that include the text entered in the filter.
For example, submitting cat and dog will result in all the items that have either one of the values in the filename.

File Extension

Allows filtering for items based on their file extension.
For example, png and jpg will return all files whose extension is either png or jpg.

Attributes

Allows filtering items that have one or more of the attributes added to the query criteria.

1. Expand the attributes and check an attribute option to add it to the filter.
2. Uncheck an attribute to remove it.
3. Collapse the filter to apply the selected labels.

Filtering by multiple attributes returns items with annotations that have one or more of the attributes in the search criteria. For example, the OR operand between them. Based on the selected Annotations that are saved in the item.

Items With No Attributes

To select all items with no attributes, edit the DQL query to include an empty array.
 "attributes": {}

    "filter": {
        "$and":[{
            "hidden":false
        },
            {
                "type":"file"
            }]},
    "join": {
        "filter": {
            "$and":[{
                "$or":[{
                    "$or":[{
                        "metadata": {
                            "system": {
                                "attributes": {}
         }}}]}]}]},
        "on": {
            "resource":"annotations",
            "local":"itemId",
            "forigen":"id"
        }}}

Annotation Type

Selecting a tool from the list will return all items with annotations from the type of the selected tool(s).
For example, selecting Box and Point filter options will return all items that have Box Annotations and all the items that have Point Annotations.

1. Expand the annotation type filter.
2. Click on the dropdown and select the tools to include in the search criteria.
   1. Classification
   2. Point
   3. Box
   4. Polygon
   5. Polyline
   6. Segmentation
   7. Note
   8. Pose
   9. Ellipse
   10. Cuboid
3. Click again on selected users to exclude from the search criteria.
4. Click Apply.
5. Click Submit filter.

See Annotation Tools for reference.

Item Size

Use either Minimum or Maximum filter options or both to refine the search criteria.

• Enter a minimum value in the Minimum field. It returns all items with size equal or above the given value.
• Enter a maximum value in the Maximum field. It returns all items with size equal or below the give value.
• Click Apply.

Item Metadata

Filter items based on metadata information in the items' JSON file.

1. Click on the Item Metadata filter.
2. Select a Meta Key Type:
   1. System: Keys stored in the reserved 'system' area, created and used by Dataloop
   2. Custom: Keys stored anywhere outside the 'system' area.
3. Select a data type:
   1. String: The filter applies only to string data types.
   2. Number: The filter applies only to numerical data types.
   3. Any: Filter applies all the data where it searches only on key names, not on values.
4. Enter the key name to search.
5. Enter data in the Value field (case sensitive).
6. Click Apply.
7. Click the Submit filter.

The following is an example of the user-metadata field 'details':

{
	"system": {
		"originalname": "name2.png",
		"size": 675834,
		"encoding": "7bit",
		"mimetype": "image/png",
		"annotationStatus": []
	},
	"details": {
		"car-color": "blue"
	}
}

Annotation System Metadata

Filter items based on annotation metadata information in the items.

1. Click on the Annotation Metadata & QA filter.
2. Select between items with annotations that have issues flagged on them, or annotations flagged for review:
   1. Item with issues
   2. For Review (QA)
3. Select a Meta Key Type:
   1. System: Keys stored in the reserved 'system' area, created and used by Dataloop
   2. Custom: Keys stored anywhere outside the 'system' area.
4. Select a data type:
   1. String: The filter applies only to string data types.
   2. Number: The filter applies only to numerical data types.
   3. Any: Filter applies all the data where it searches only on key names, not on values.
5. Enter the key name to search.
6. Enter data in the Value field (case sensitive).
7. Click Apply.
8. Click the Submit filter.

The following is an example of user-metadata field 'details':

To search for annotations that have been Approved following a QA cycle (issue raised, corrected and approved by reviewer),  use:

• Field type = TEXT
• Key= "metadata.system.status"
• Value= "approved*"

Item Media Type

The filter allows searching based on general media types, such as video or image, or specific mime type. For example, searching for video/mp4 will return all mp4 videos.