Networking
This section describes how the components interact with each other.
-
From Web Application Server and API - Celery Worker:
-
To File Management: For file read/write operations
- To Metadata Database: For metadata read/write operations
-
To Redis: For task queuing and status-checks
-
From Spark - Celery worker:
-
To Redis: For task queuing and status-checks
-
To submit Spark Jobs: To submit spark jobs based on the task queue
-
From Jupyter Notebook:
-
To API: For metadata access
- To Web Application: For authentication (if internal authentication is used)
- To submit Spark Jobs: To submit spark jobs based on the task queue
Sequence Diagram
The interaction flow between the different components can be better understood with the below sequence diagram.
