DQL
  • Print
  • Share
  • Dark
    Light

DQL

  • Print
  • Share
  • Dark
    Light

The DQL - The Dataloop Query Language

While the Dataloop platform gives you different capabilities to organize your data such as datasets, folders and versioning systems, you still need the ability to query your data. This is where our Data index - Dataloop Query Language becomes handy.

Using The Dataloop Query Language, you may navigate through massive amounts of data.

The Query Language

The query language has 3 main functionalities. Together they allow for many different use cases:

  • Read: Read of items operation, allowing to give items a criteria
  • Update: Allowing update of metadata for a collection of items at once
  • Delete: Allows removing items in bulk
📘  Both REST API and SDK (with FaaS) support Dataloop Query Language - DQL.

SDK Usage

For an SDK usage of DQL go to Filters .

DQL Structure

Every DQL query has the following components:

  • Resource : The target resource for the query. Currentlyitems and annotations are supported.
  • Filter : A logical filter which determines which resources will be affected by the query.
  • Sort : The properties and order by which resources are sorted.

With each query type additional properties apply:

Read

  • Page: Which page should be returned by the query.
  • Page size: The paging size being used, blocked by 1000 items per page.

Update

  • Update: The JSON object to update on the query result.

Delete

  • Delete: A Boolean flag, indicating whether to delete the requested items.

Rest API Usage

Basic DQL Definition

DQL Query is defined as follows:

{
    resource: 'items' | 'annotations'
        Specifies to resource to be queried

    filter: {...}
        Specifies the filter to be applied
        A combination of field values and operators

    sort: {...}             [optional]
        Specifies how to sort the results of the query

    update: {...}           [optional]
        Specifies which fields & values should be updated 

    delete: boolean         [optional]
        When true - resources matched by the filter will be deleted
        Default: false

    page: number            [optional]
        Specify page offset of the results
        Default: 0

    pageSize: number        [optional]
        Specify how many items should be in each page
        Default: 100
}

A typical response will look like the following:

{
    totalItemsCount: number
    items: Array<QueryResource>
    totalPagesCount: number
    hasNextPage: boolean
}

#A possible result:

{
    "totalItemsCount": 2,
    "totalPagesCount": 1,
    "hasNextPage": false,
    "items": [
        {
            "id": "5d0783852dbc15306a59ef6c",
            "createdAt": "2019-06-18T23:29:15.775Z",
            "filename": "/5546670769_8df950c6b6.jpg",
            "type": "file"
            // ...
        },
        {
            "id": "5d0783852dbc15306a59ef6d",
            "createdAt": "2019-06-19T23:29:15.775Z",
            "filename": "/5551018983_3ce908ac98.jpg",
            "type": "file"
            // ...
        }
    ]
}

Sorting

Sorting can be performed on any field that belongs to a resource. A field can be sorted in ascending or descending order.

Multiple fields sorting is also supported.

"sort": {
        "<field>": "ascending|descending"
    }

#Example(the full query), Sort directories by name, Acending  
{
    "resource": "items",
    "filter": { "type": "dir" },
    "sort": {
        "filename": "ascending"
    }
}

#Sorting on multiple fields is possible, but recommended to avoid such operation
{
    "resource": "annotations",
    "filter": { "creator": "assaf@dataloop.ai" },
    "sort": {
        "label": "ascending"
        "createdAt": "descending"
    }
}

Filters

Filters are essentially a list of fields and their respected operators. For example, to get all items named foo.bar (run this items query):

{ 
    "filename": { 
        "$eq": "foo.bar" 
    }
}

Here's another example: Delete all jpg files (notice the wildcard):

{
    "resource": "items",
    "filter": { "filename": "*.jpg" },
    "delete": true
}

DQL Filter Operators

Equal / Not equal

$eq specifies equality condition.
$ne operator returns all the items that did not match the specified value.

{ <field>: { $eq: <value> } }
{ <field>: { $ne: <value> } }
 Example:
 {
  "resource": "items",
  "filter": {
    "type": "file", // Match only items of type 'file'
    "createdAt": { "$eq": "2019-06-17T23:29:15.775Z" } // Match items that were created after the specified date
  }
}

Greater / Lower Than

$gt operator returns all items where the field is greater than the specified value.
$lt operator returns all items where the field is less than the specified value.

{<field>:  {$gt:  <value>}  }
{<field>:  {$lt:  <value>}  }

Logical AND / OR

$or operator specifies a list of expressions and matches all items that satisfy at least one of the expressions that may contain operators

$and operator specifies a list of expressions and matches of all items that satisfy all of the expressions that may contain operators

{ $or: [ { <expression1> }, { <expression2> }, ... , { <expressionN> } ] }
{ $and: [ { <expression1> }, { <expression2> }, ... , { <expressionN> } ] } 
{
    "resource": "items",
    "filter": {
        "$or": [
            { ... },
            { ... },
            "$and": [
                { ... }
            ]
        ],
        "$and": [
            { ... },
            { ... },
            "$or": [
                { ... },
                { ... }
            ]
        ]
    }
}

Field List Matching

$in operator specifies a list of values and matches all items where field matches one of the values.

{ field: { $in: [<value1>, <value2>, ... <valueN> ] } }

Example:

{ 
    "field": { 
        "$in": ["option-1", "option-2"] 
    }
}

Wildcards

Wildcards * and ? are supported on field value

  • Will match zero or more characters.
    ? Will match one character at most.

    Example:

#Match all filenames that contains magic

{ "filename": "*magic*" }

Match mimetypes image/jpg and image/jpeg

{ "filename": "*jp?g" }

Update Query

Add the JSON you wish to apply to the metadata update.

#This will update all polygon annotations metadata with some exciting information
{
    "resource": "annotations",
    "filter": {
        "type": "segment"
    },
    "update": {
        "metadata": {
            "some": {
                "exciting": "information"
            }
        }
    }
}

Delete Query

Add the delete flag and the query will delete matched resources.

#The query below will delete all items that has a .jpg suffix Delete operations does not return any results

{
    "resource": "items",
    "filter": { "filename": "*.jpg" },
    "delete": true
}
Was This Article Helpful?