flowmium 0.0.0

CLI tool to validate and format json files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

<img src="./logo.svg" width="128px"><br>

# Flowmium

Flowmium is a workflow orchestrator that uses kubernetes written in rust.

## Draft design notes

These designs are still in early stages, detailed documentation is yet to be done.

### Endpoints and job models

-   `POST /api/v1/job` and `PUT /api/v1/job`

_DAG container job definition_

```yaml
name: ""
schedule: ""
tasks:
  - name: ""
    image: ""
    depends: ["", ""]
    cmd: [],
    env:
      - name: ""
        value: ""
      - name: ""
        fromInput: ""
      - name: ""
        fromSecret: ""
    inputs:
      - from: ""
        path: ""
    outputs:
      - name: ""
        path: ""
config:
  active_deadline_seconds: 34
  affinity: 34
  tolerations: 34
  image_pull_secrets: 34
  priority: 3
  limits: 23
  requests: 23
```

_Python framework job definition_

```yaml
python:
    image: ""
    registry: ""
```

-   `GET /api/v1/job/` and `GET /api/v1/job/{name}`
-   `DELETE /api/v1/job/{name}`
-   `POST /api/v1/registry` and `PUT /api/v1/registry`
-   `GET /api/v1/registry/`
-   `DELETE /api/v1/registry`
-   `POST /api/v1/secret` and `PUT /api/v1/secret`
-   `GET /api/v1/secret/`
-   `DELETE /api/v1/secret`
-   Artifacts

### Python framework design and usage example

```python
from flowmium import flow

@flow.task()
def foo1(context):
    return 1

@flow.task({'arg_1': foo1})
def foo2(context, arg_1)
    print(context.secrets)
    return arg_1 + 1

@flow.task({'arg_1': foo1}, workers=8)
def foo3(context, arg_1)
    print(context.worker_id)    # used for manual sharding
    return arg_1 + 2

@flow.task(depends={'arg_1': foo2, 'arg_2': foo3}, secrets=[""], config={})
def foo3(context, arg_1, arg_2)
    return arg_1 * arg_2

flow.run(name="", schedule="", secrets=[""], config={})
# prints yaml DAG container job definition to run each function (task) as a pod
# which function to run in will be specified by CLI arguments
# the flow object will take care of parsing the arguments
```