Data Browser

The dataset browser enables advanced data exploration and visualization.

Accessing The Dataset Browser

Access the dataset browser from the Dataset record in the following options.

• In the Datasets page, click the Browse icon for the respective entry.

• In the Project Overview > Data-Management widget, click on the dataset name.

Highlighted Features

• Thumbnails view with adjustable thumbnail size
• List view with file details
• Filters based on item attributes, item metadata, and user metadata
• Direct DQL queries
• Save and reuse DQL queries
• Folders management, such as create, rename, delete folders, etc.
• File management - move between folders, clone, delete
• Create models from selected data
• Create an annotation/QA task from the selected data
• Trigger selected data to a function (FaaS) or Pipeline
• View item metadata
• Item function execution log
• Export data (Item JSON file)
• Upload data (when using the File system storage)
• Perform bulk operations on selected items or active queries, such as move, delete, invoke, etc.

Data View Options

The Dataset browser allows both thumbnails and list view. For images and video files, the ETL preprocessing step generates the thumbnails when items are added to the Dataset. For other file types, a simple icon represents the file type. Additional thumbnails can be generated using custom preprocessing functions.

• The list/details view provides more information on every image and enables pagination and sorting features by column.
• Switch between the different views using the respective view-mode buttons.

• Adjust the thumbnail size using the slider control at the bottom left of the page. Use smaller thumbnails to view more items on a page.

• In the thumbnails view, use the Show file names option to hide the file names.

Pagination

The Dataset browser supports pagination, and the default items per page is 100. Number of items per page is adjustable from 25 to 1000.

In the thumbnail view, only a certain number of images are displayed per page. You can scroll through the pages to view more images.

View Context

Items in the Dataset browser are displayed in one of three following filter contexts:

• All dataset (default)
• Folder
• Task

All Dataset (default)

The All Dataset context allows applying filters and displays all items in a single root level, regardless of their physical location. It enables using filters for querying the dataset based on:

• Item attributes: Such as file size, file name, and mime type.
• Item metadata: Annotated or not, has status in a task or not, has annotation from specific label/attribute, by specific users, and more.
• User metadata: Search by any metadata key and/or value added to items.

Using the filters results in structuring a DQL query. Users in the project have access to read, store, and reuse DQL queries.

Folders

The dataset browser enables organizing file items in file-system-like folders and subfolders structure. Allows to view Items from multiple folders at once.

• Switch to folders view by clicking on the folder icon.

• Create a new folder at root level by clicking the new folder icon at the top (1),
• Create a folder as a subfolder when clicking the new folder button at any level (2).

• To move items between folders, select one or more items from the current page and click on the move items icon.

Task

Task context allows seeing all items in a specific task, regardless of its folder context in a dataset, since items in a task can come from different locations.
It allows narrowing down the search and filtering to a specific task, for example, when filtering for specific annotations, labels, or attributes.

• Switch to the task context by clicking on the Task icon.

• Select between all tasks to view all items that are part of a task, and a specifically selected one.
• Click the filters icon to further filter the query to items with a specific status in tasks, or even to items with a specific status in specific data.
  
  • No status: Items in a task with no status. For example, still pending work or completion of work
  • Has status: Items that has any status in a task
  • Completed: Items that passed an annotation task
  • Approved: Items that passed a QA task

Export Data

Data export is available from the Dataset browser. A JSON file will be downloaded in a single ZIP archive file.

1. To begin the export process, click Export icon.

2. Select the export Scope:

   • Entire dataset: The ZIP file includes JSON files for all items in the dataset.
   • Active query: If using a filter/query, select this option to download JSON files only for items in the query result.
   • Selected items: Select this option to download JSON files only for manually selected items.

3. For the Active query and Selected items options, you can optionally add PNG files of semantic segmentation annotations (masks).

4. Click Export. A ZIP file will be created and downloaded.

Right-click Menu

Right-click on any item to have a pop-up menu with item-specific actions.
It is not possible to apply actions from the right-click menu to multiple selected items. The menu is opened for the item on which the right-click is made.

Item Right-Side Panel

Select a single file to see its details in the right-side panel and be able to trigger it to a function (FaaS) or a pipeline.

Item Details

The side-panel includes the following details and functionality for every selected item:

1. Item details
   1. Created date
   2. Item description - text description that can be viewed and edited from here. Item description can be added while uploading files, and serves as another easy means to search for items containing some text or description.
      
   3. Item ID
   4. Dataset ID in which it resides
   5. File path - folder path in which the file resides
   6. Item path - a URL link to the item on the Dataloop platform
2. Item metadata - View, edit and save the item metadata.
   1. Click to expand and view the metadata saved in the items' JSON file, including System and User metadata.
   2. Click the Edit icon (1) to open the editor.
   3. Click the Format icon (2) to format as JSON any content edited.
   4. Save the new metadata (3).

4. Function: Click to expand and view details of past executions, including:
   1. App name
   2. Function name
   3. Execution status - Success, Failed, and Running
   4. Date and time of the execution

• Click the link to review the full details on the Executions or Logs page.
• Click the Play icon to rerun the item to the function, regardless of the last execution status.

Trigger Item To Function/Pipeline

Trigger a selected item to a function from a running service (FaaS) or a running pipeline.

1. Select the function/pipeline from the dropdown list. Functions are presented with the Service name and the project in which they are installed.
2. Click Execute to trigger the item.

Automation - Functions & Triggers

The Dataset-browser includes two important automation features

1. Functions: Any function deployed to this Applications can be invoked on selected items, or active query.

2. Triggers: Create a trigger for the selected items, active query, selected folder, or entire dataset. Available triggers are:
   1. Item created: Invoke any new item to the function
   2. Item update: Invoke any item update/saved
   3. Item deleted: Invoke any item deleted

Bulk Operations

The Dataset browser allows bulk operations in the required context. To perform bulk operations:

1. Manually select one or more items using the CTRL+ left mouse key.

2. Click on Select All to select all items on the current page. This does not select all items in the query. Adjust pagination settings to include more items to enhance the scope.

   

3. Invoke the items to one of the following bulk operations:

   • Move: Move all items to another folder
   • Delete: Delete all items
   • Invoke: Select a function from the UI-Slot to invoke all items
   • Clone: Clone all items to another folder or another/new dataset
   • Download: Download JSON files for those items