FaaS Introduction
  • Dark
    Light
  • PDF

FaaS Introduction

  • Dark
    Light
  • PDF

The Dataloop Function As A Service enables users to deploy packages and run services from them, with access to computing resources and data from the Dataloop system. Altogether, it serves as an amazingly flexible platform that provides outstanding flexibility, increasing the platform’s capabilities, and essentially allows achieving any needs and requirements, and automating many processes.

The FaaS guideline uses Dataloop SDK, Click here for the FaaS section in the SDK cheat sheet.

How Does it Work

Compute

FaaS uses the Kubernetes cluster, located near our backend for very low latency requests with minor operational costs.

CPU Environment
OS: ubuntu:18.04

Python version:
Python 3.6.3

Python packages:

Package             Version
------------------- -----------
absl-py             0.9.0
aiohttp             3.6.2
asn1crypto          0.24.0
astor               0.8.1
async-timeout       3.0.1
attr                0.3.1
attrs               19.3.0
certifi             2020.6.20
chardet             3.0.4
cryptography        2.1.4
cycler              0.10.0
decorator           4.4.2
dictdiffer          0.8.1
docopt              0.6.2
dtlpy               1.19.18
dtlpy-agent         1.19.18.0
ffmpeg-python       0.2.0
future              0.18.2
fuzzyfinder         2.1.0
gast                0.3.2
grpcio              1.26.0
h5py                2.10.0
hdf5storage         0.1.15
idna                2.10
idna-ssl            1.1.0
imageio             2.6.1
imgaug              0.2.9
Jinja2              2.11.2
Keras               2.1.6
keyring             10.6.0
keyrings.alt        3.0
kiwisolver          1.1.0
Markdown            3.1.1
MarkupSafe          1.1.1
matplotlib          3.1.2
multidict           4.7.6
networkx            2.4
nms                 0.1.6
numpy               1.19.2
opencv-python       3.4.2.17
pandas              1.1.2
pathspec            0.8.0
pika                1.0.1
Pillow              7.2.0
pip                 19.3.1
GPU Environment
OS: ubuntu:16.04

Python version:
Python 3.6.3

Python packages:

Package                            Version             
---------------------------------- --------------------
absl-py                            0.9.0               
aiohttp                            3.6.2               
alabaster                          0.7.10              
anaconda-client                    1.6.5               
anaconda-navigator                 1.6.9               
anaconda-project                   0.8.0               
asn1crypto                         0.22.0              
astor                              0.8.1               
astroid                            1.5.3               
astropy                            2.0.2               
async-timeout                      3.0.1               
attr                               0.3.1               
attrs                              19.3.0              
Babel                              2.5.0               
backports.shutil-get-terminal-size 1.0.0               
beautifulsoup4                     4.6.0               
bitarray                           0.8.1               
bkcharts                           0.2                 
blaze                              0.11.3              
bleach                             2.0.0               
bokeh                              0.12.10             
boto                               2.48.0              
Bottleneck                         1.2.1               
certifi                            2017.7.27.1         
cffi                               1.10.0              
chardet                            3.0.4               
click                              6.7                 
cloudpickle                        0.4.0               
clyent                             1.2.2               
colorama                           0.3.9               
conda                              4.8.2               
conda-build                        3.0.27              
conda-package-handling             1.6.0               
conda-verify                       2.0.0               
contextlib2                        0.5.5               
cryptography                       2.8                 
cycler                             0.10.0              
Cython                             0.26.1              
cytoolz                            0.8.2               
dask                               2.12.0              
datashape                          0.5.4               
decorator                          4.1.2               
dictdiffer                         0.8.1               
distributed                        1.19.1              
docopt                             0.6.2               
docutils                           0.14                
dtlpy                              1.19.12             
dtlpy-agent                        1.19.12.0           
entrypoints                        0.2.3               
et-xmlfile                         1.0.1               
fastcache                          1.0.2               
ffmpeg-python                      0.2.0               
filelock                           2.0.12              
Flask                              0.12.2              
Flask-Cors                         3.0.3               
future                             0.18.2              
fuzzyfinder                        2.1.0               
gast                               0.3.3               
gevent                             1.2.2               
glob2                              0.5                 
gmpy2                              2.0.8               
greenlet                           0.4.12              
grpcio                             1.27.2              
h5py                               2.7.0               
hdf5storage                        0.1.15              
heapdict                           1.0.0               
html5lib                           0.999999999         
idna                               2.6                 
idna-ssl                           1.1.0               
imageio                            2.2.0               
imagesize                          0.7.1               
imgaug                             0.2.9               
ipykernel                          4.6.1               
ipython                            6.1.0               
ipython-genutils                   0.2.0               
ipywidgets                         7.0.0               
isort                              4.2.15              
itsdangerous                       0.24                
jdcal                              1.3                 
jedi                               0.10.2              
Jinja2                             2.11.2              
joblib                             0.14.1              
jsonschema                         2.6.0               
jupyter-client                     5.1.0               
jupyter-console                    5.2.0               
jupyter-core                       4.3.0               
jupyterlab                         0.27.0              
jupyterlab-launcher                0.4.0               
jwt                                0.6.1               
Keras                              2.1.6               
lazy-object-proxy                  1.3.1               
llvmlite                           0.20.0              
locket                             0.2.0               
lxml                               4.1.0               
Markdown                           3.2.1               
MarkupSafe                         1.0                 
matplotlib                         2.1.0               
mccabe                             0.6.1               
mistune                            0.7.4               
mpmath                             0.19                
msgpack-python                     0.4.8               
multidict                          4.7.6               
multipledispatch                   0.4.9               
navigator-updater                  0.1.0               
nbconvert                          5.3.1               
nbformat                           4.4.0               
networkx                           2.0                 
nltk                               3.2.4               
nms                                0.1.6               
nose                               1.3.7               
notebook                           5.0.0               
numba                              0.35.0+10.g143f70e90
numexpr                            2.6.2               
numpy                              1.18.0              
numpydoc                           0.7.0               
odo                                0.5.1               
olefile                            0.44                
opencv-python                      3.4.2.17            
openpyxl                           2.4.8               
packaging                          16.8                
pandas                             1.1.2               
pandocfilters                      1.4.2               
partd                              0.3.8               
path.py                            10.3.1              
pathlib2                           2.3.0               
pathspec                           0.8.0               
patsy                              0.4.1               
pep8                               1.7.0               
pexpect                            4.2.1               
pickleshare                        0.7.4               
Pillow                             7.2.0               
pip                                20.0.2              
pkginfo                            1.4.1               
ply                                3.10                
prompt-toolkit                     3.0.7               
protobuf                           3.11.3              
psutil                             5.6.7               
ptyprocess                         0.5.2               
py                                 1.4.34              
py3nvml                            0.2.5               
pycodestyle                        2.3.1               
pycosat                            0.6.3               
pycparser                          2.18                
pycrypto                           2.6.1               
pycurl                             7.43.0              
pyflakes                           1.6.0               
Pygments                           2.2.0               
PyJWT                              1.7.1               
pylint                             1.7.4               
pyodbc                             4.0.17              
pyOpenSSL                          17.2.0              
pyparsing                          2.2.0               
PySocks                            1.6.7               
pytest                             3.2.1               
python-dateutil                    2.8.1               
pytz                               2017.2              
PyWavelets                         0.5.2               
PyYAML                             3.12                
pyzmq                              16.0.2              
QtAwesome                          0.4.4               
qtconsole                          4.3.1               
QtPy                               1.3.1               
requests                           2.24.0              
requests-toolbelt                  0.9.1               
rope                               0.10.5              
ruamel-yaml                        0.11.14             
scikit-image                       0.14.2              
scikit-learn                       0.21.3              
scipy                              1.1.0               
seaborn                            0.8                 
setuptools                         36.5.0.post20170921 
Shapely                            1.7.0               
simplegeneric                      0.8.1               
singledispatch                     3.4.0.3             
six                                1.11.0              
snowballstemmer                    1.2.1               
sortedcollections                  0.5.3               
sortedcontainers                   1.5.7               
Sphinx                             1.6.3               
sphinxcontrib-websupport           1.0.1               
spyder                             3.2.4               
SQLAlchemy                         1.1.13              
statsmodels                        0.8.0               
sympy                              1.1.1               
tables                             3.4.2               
tabulate                           0.8.7               
tblib                              1.3.2               
tensorboard                        1.9.0               
tensorflow-gpu                     1.9.0               
termcolor                          1.1.0               
terminado                          0.6                 
testpath                           0.3.1               
toolz                              0.8.2               
torch                              1.1.0               
tornado                            6.0.2               
tqdm                               4.42.1              
traitlets                          4.3.2               
typing                             3.6.2               
typing-extensions                  3.7.4.3             
unicodecsv                         0.14.1              
urllib3                            1.22                
validators                         0.18.1              
wcwidth                            0.1.7               
webencodings                       0.5.1               
webvtt-py                          0.4.3               
Werkzeug                           0.12.2              
wheel                              0.29.0              
widgetsnbextension                 3.0.2               
wrapt                              1.10.11             
xlrd                               1.1.0               
XlsxWriter                         1.0.2               
xlwt                               1.3.0               
xmltodict                          0.12.0              
yarl                               1.5.1               
zict                               0.1.3 

Packages

recipe

Packages are installed into the Dataloop FaaS system via SDK or directly from the platform UI:

  1. Define classes.
  2. Write functions inside the classes.
  3. Use modules to translate and define your class or a substance of it.

Services - Execution, Triggers & UI Slots

Deploy a service from a package:

  1. Run the init method on deployment, to upload heavy models only once.
  2. Execute your functions manually.
  3. Set triggers on your services to have them executed automatically when the time/criteria is met.
  4. Assign functions to UI slots, to make them available from the platform’s UI. For example, a function that works on an image can become accessible to the annotator in the annotation-studio

Prerequisites

  • Working with Dataloop SDK requires you to have Python installed with the Dtlpy package. Read these instructions for more information.

  • In order to use FAAS you have to be part of an Organisation or to create your own. For more information how to create one, please follow instruction on this page.

Read more about FaaS related SDK (dtlpy) Repositories and Entities here.


What's Next