What

What is Datalayer.

Datalayer Offering

We ship Pilot to create a Kubernetes cluster on Amazon AWS from scratch.

We also ship a Data Explorer for Data Scientists who are willing to get the best from Apache Spark and Hadoop.

The Explorer helps the Data Science project actors to collaborate and share their knowledge in a Library which exposes searchable higher level of abstraction such as Datasets, Notes, Gists, Functions and Models.

This is the overall architecture that sustain those goals is depicted here after. We fully support the usage of the exploration artifacts stored in the library into a Alpha environment.

Architecture

Like building a house, we need strong foundations and well organized layers that are secured (IAM, RBAC) and managed (OPS).

Pilot is the Datalayer offering to manage the stack.

+----------------------------------------------------------------+
|             DATALAYER BIG DATA SCIENCE PLATFORM                |
|----------------------------------------------------------------|
|    K    | 7 |                   ACCESS                         |
|    U    |   |                  Ingress                         |
|    B    |------------------------------------------------------|
|    E    | 6 |                    UI                            |
|    R    |   |          UI | CLI | JSApps | Widgets             |
|   CLI   |------------------------------------------------------|
|   SDK   | 5 |                  SPITIFIRE                       |
|   LIB   |   |         Spark | TensorFlow | Markdown | ...      |
|         |------------------------------------------------------|
|   OPS   | 4 |                    UTILS                         |
|         |   |       Kubeless | Batch | Kafka | Kubeflow        |
|  MONIT  |------------------------------------------------------|
|         | 3 |                   STORE                          |
| BACK/RES|   |       HBase | Solr | JanusGraph | Etcd           |
|         |------------------------------------------------------|
|  DIAG   | 2 |                 FILE-SYSTEM                      |
|         |   |             HDFS | IFPS | Minio                  |
|  IAM    |------------------------------------------------------|
|         | 1 |                K8S-CLUSTER                       |
|  RBAC   |   | Kubeadm | Kubicorn/Kubespray | Reshifter/ARK     |
|         |   |            Gitkube | Kubed-sh                    |
+----------------------------------------------------------------+