FaaS Introduction
-
Print
-
DarkLight
-
PDF
FaaS Introduction
-
Print
-
DarkLight
-
PDF
The Dataloop Function As A Service enables users to deploy packages and run services from them, with access to computing resources and data from the Dataloop system. Altogether, it serves as an amazingly flexible platform that provides outstanding flexibility, increasing the platform’s capabilities, and essentially allows achieving any needs and requirements, and automating many processes.
The FaaS guideline uses Dataloop SDK, Click here for the FaaS section in the SDK cheat sheet.
How Does it Work
Compute
FaaS uses the Kubernetes cluster, located near our backend for very low latency requests with minor operational costs.
CPU Environment
OS:
ubuntu:18.04
Python version:
Python 3.6.3
Python packages:
Package Version ------------------- ----------- absl-py 0.9.0 aiohttp 3.6.2 asn1crypto 0.24.0 astor 0.8.1 async-timeout 3.0.1 attr 0.3.1 attrs 19.3.0 certifi 2020.6.20 chardet 3.0.4 cryptography 2.1.4 cycler 0.10.0 decorator 4.4.2 dictdiffer 0.8.1 docopt 0.6.2 dtlpy 1.19.18 dtlpy-agent 1.19.18.0 ffmpeg-python 0.2.0 future 0.18.2 fuzzyfinder 2.1.0 gast 0.3.2 grpcio 1.26.0 h5py 2.10.0 hdf5storage 0.1.15 idna 2.10 idna-ssl 1.1.0 imageio 2.6.1 imgaug 0.2.9 Jinja2 2.11.2 Keras 2.1.6 keyring 10.6.0 keyrings.alt 3.0 kiwisolver 1.1.0 Markdown 3.1.1 MarkupSafe 1.1.1 matplotlib 3.1.2 multidict 4.7.6 networkx 2.4 nms 0.1.6 numpy 1.19.2 opencv-python 3.4.2.17 pandas 1.1.2 pathspec 0.8.0 pika 1.0.1 Pillow 7.2.0 pip 19.3.1
GPU Environment
OS:
ubuntu:16.04
Python version:
Python 3.6.3
Python packages:
Package Version ---------------------------------- -------------------- absl-py 0.9.0 aiohttp 3.6.2 alabaster 0.7.10 anaconda-client 1.6.5 anaconda-navigator 1.6.9 anaconda-project 0.8.0 asn1crypto 0.22.0 astor 0.8.1 astroid 1.5.3 astropy 2.0.2 async-timeout 3.0.1 attr 0.3.1 attrs 19.3.0 Babel 2.5.0 backports.shutil-get-terminal-size 1.0.0 beautifulsoup4 4.6.0 bitarray 0.8.1 bkcharts 0.2 blaze 0.11.3 bleach 2.0.0 bokeh 0.12.10 boto 2.48.0 Bottleneck 1.2.1 certifi 2017.7.27.1 cffi 1.10.0 chardet 3.0.4 click 6.7 cloudpickle 0.4.0 clyent 1.2.2 colorama 0.3.9 conda 4.8.2 conda-build 3.0.27 conda-package-handling 1.6.0 conda-verify 2.0.0 contextlib2 0.5.5 cryptography 2.8 cycler 0.10.0 Cython 0.26.1 cytoolz 0.8.2 dask 2.12.0 datashape 0.5.4 decorator 4.1.2 dictdiffer 0.8.1 distributed 1.19.1 docopt 0.6.2 docutils 0.14 dtlpy 1.19.12 dtlpy-agent 1.19.12.0 entrypoints 0.2.3 et-xmlfile 1.0.1 fastcache 1.0.2 ffmpeg-python 0.2.0 filelock 2.0.12 Flask 0.12.2 Flask-Cors 3.0.3 future 0.18.2 fuzzyfinder 2.1.0 gast 0.3.3 gevent 1.2.2 glob2 0.5 gmpy2 2.0.8 greenlet 0.4.12 grpcio 1.27.2 h5py 2.7.0 hdf5storage 0.1.15 heapdict 1.0.0 html5lib 0.999999999 idna 2.6 idna-ssl 1.1.0 imageio 2.2.0 imagesize 0.7.1 imgaug 0.2.9 ipykernel 4.6.1 ipython 6.1.0 ipython-genutils 0.2.0 ipywidgets 7.0.0 isort 4.2.15 itsdangerous 0.24 jdcal 1.3 jedi 0.10.2 Jinja2 2.11.2 joblib 0.14.1 jsonschema 2.6.0 jupyter-client 5.1.0 jupyter-console 5.2.0 jupyter-core 4.3.0 jupyterlab 0.27.0 jupyterlab-launcher 0.4.0 jwt 0.6.1 Keras 2.1.6 lazy-object-proxy 1.3.1 llvmlite 0.20.0 locket 0.2.0 lxml 4.1.0 Markdown 3.2.1 MarkupSafe 1.0 matplotlib 2.1.0 mccabe 0.6.1 mistune 0.7.4 mpmath 0.19 msgpack-python 0.4.8 multidict 4.7.6 multipledispatch 0.4.9 navigator-updater 0.1.0 nbconvert 5.3.1 nbformat 4.4.0 networkx 2.0 nltk 3.2.4 nms 0.1.6 nose 1.3.7 notebook 5.0.0 numba 0.35.0+10.g143f70e90 numexpr 2.6.2 numpy 1.18.0 numpydoc 0.7.0 odo 0.5.1 olefile 0.44 opencv-python 3.4.2.17 openpyxl 2.4.8 packaging 16.8 pandas 1.1.2 pandocfilters 1.4.2 partd 0.3.8 path.py 10.3.1 pathlib2 2.3.0 pathspec 0.8.0 patsy 0.4.1 pep8 1.7.0 pexpect 4.2.1 pickleshare 0.7.4 Pillow 7.2.0 pip 20.0.2 pkginfo 1.4.1 ply 3.10 prompt-toolkit 3.0.7 protobuf 3.11.3 psutil 5.6.7 ptyprocess 0.5.2 py 1.4.34 py3nvml 0.2.5 pycodestyle 2.3.1 pycosat 0.6.3 pycparser 2.18 pycrypto 2.6.1 pycurl 7.43.0 pyflakes 1.6.0 Pygments 2.2.0 PyJWT 1.7.1 pylint 1.7.4 pyodbc 4.0.17 pyOpenSSL 17.2.0 pyparsing 2.2.0 PySocks 1.6.7 pytest 3.2.1 python-dateutil 2.8.1 pytz 2017.2 PyWavelets 0.5.2 PyYAML 3.12 pyzmq 16.0.2 QtAwesome 0.4.4 qtconsole 4.3.1 QtPy 1.3.1 requests 2.24.0 requests-toolbelt 0.9.1 rope 0.10.5 ruamel-yaml 0.11.14 scikit-image 0.14.2 scikit-learn 0.21.3 scipy 1.1.0 seaborn 0.8 setuptools 36.5.0.post20170921 Shapely 1.7.0 simplegeneric 0.8.1 singledispatch 3.4.0.3 six 1.11.0 snowballstemmer 1.2.1 sortedcollections 0.5.3 sortedcontainers 1.5.7 Sphinx 1.6.3 sphinxcontrib-websupport 1.0.1 spyder 3.2.4 SQLAlchemy 1.1.13 statsmodels 0.8.0 sympy 1.1.1 tables 3.4.2 tabulate 0.8.7 tblib 1.3.2 tensorboard 1.9.0 tensorflow-gpu 1.9.0 termcolor 1.1.0 terminado 0.6 testpath 0.3.1 toolz 0.8.2 torch 1.1.0 tornado 6.0.2 tqdm 4.42.1 traitlets 4.3.2 typing 3.6.2 typing-extensions 3.7.4.3 unicodecsv 0.14.1 urllib3 1.22 validators 0.18.1 wcwidth 0.1.7 webencodings 0.5.1 webvtt-py 0.4.3 Werkzeug 0.12.2 wheel 0.29.0 widgetsnbextension 3.0.2 wrapt 1.10.11 xlrd 1.1.0 XlsxWriter 1.0.2 xlwt 1.3.0 xmltodict 0.12.0 yarl 1.5.1 zict 0.1.3
Packages
Packages are installed into the Dataloop FaaS system via SDK or directly from the platform UI:
- Define classes.
- Write functions inside the classes.
- Use modules to translate and define your class or a substance of it.
Services - Execution, Triggers & UI Slots
Deploy a service from a package:
- Run the init method on deployment, to upload heavy models only once.
- Execute your functions manually.
- Set triggers on your services to have them executed automatically when the time/criteria is met.
- Assign functions to UI slots, to make them available from the platform’s UI. For example, a function that works on an image can become accessible to the annotator in the annotation-studio
Prerequisites
-
Working with Dataloop SDK requires you to have Python installed with the Dtlpy package. Read these instructions for more information.
-
In order to use FAAS you have to be part of an Organisation or to create your own. For more information how to create one, please follow instruction on this page.
Read more about FaaS related SDK (dtlpy) Repositories and Entities here.
IP Address Whitelisting
If a FaaS is intended to be hosted on a remote server, or send events/info to one, and you require to whitelist Dataloop IPs, these is the full relevant IP addresses. Note that additional IPs may be added from time to time. If you whitelist these IPs, please infom our support/success team to be notified on future additons:
34.76.183.75
104.199.5.216
35.241.168.191
35.240.12.251