84 Commits
480d4f4f3b
...
b139b6556d
Jenkins
Fabian Vowie
Roman Zipp
27 changed files with 1783 additions and 117 deletions
-
7.gitignore
-
297README.md
-
3build.sh
-
21config/example.json
-
12go.mod
-
38go.sum
-
11lithium.log
-
16lithium.md
-
193main.go
-
106main_test.go
-
28middlewares/authorization.go
-
50middlewares/authorization_test.go
-
43middlewares/ratelimiter.go
-
64middlewares/ratelimiter_test.go
-
99pipelines/executable_step.go
-
18pipelines/executable_step_test.go
-
67pipelines/pipeline.go
-
541pipelines/pipeline_test.go
-
39pipelines/step.go
-
3pipelines/step_test.go
-
16settings.json
-
91settings/settings.go
-
57settings/settings_test.go
-
57storage/storage.go
-
23storage/storage_test.go
-
BINtests/files/800x500.jpg
-
BINtests/files/900x900.jpg
@ -0,0 +1,297 @@ |
|||
# Lithium |
|||
|
|||
Micro-service for file storage and processing written in Go. |
|||
|
|||
## Features |
|||
|
|||
- Image processing pipelines for various transformations |
|||
- Web API for storing & retrieving files |
|||
|
|||
## Requirements |
|||
|
|||
- [Go 1.17+](https://go.dev/) |
|||
- [_Docker_](https://docs.docker.com/) (optional) |
|||
|
|||
## Setup |
|||
|
|||
#### 1. Clone repository |
|||
|
|||
```shell |
|||
git clone git@gogs.informatik.hs-fulda.de:FabianVowie/Lithium.git |
|||
``` |
|||
|
|||
#### 2. Pull dependencies |
|||
|
|||
```shell |
|||
go get |
|||
``` |
|||
|
|||
#### 3. Build & start application |
|||
|
|||
```shell |
|||
go run . |
|||
``` |
|||
|
|||
**Run using [Docker](https://docs.docker.com/) container** |
|||
|
|||
```shell |
|||
docker run --rm -p 8000:8000 -v "$PWD":/usr/src/lithium -w /usr/src/lithium golang:1.17 go run . |
|||
``` |
|||
|
|||
## Testing |
|||
|
|||
```shell |
|||
go test ./... |
|||
``` |
|||
|
|||
### Run tests in verbose logging mode |
|||
|
|||
```shell |
|||
go test ./... -v |
|||
``` |
|||
|
|||
### Run tests in [Docker](https://docs.docker.com/) container |
|||
|
|||
```shell |
|||
docker run --rm -v "$PWD":/usr/src/lithium -w /usr/src/lithium golang:1.17 go test ./... |
|||
``` |
|||
|
|||
## Configuration |
|||
|
|||
Config options can be adjusted via the [`settings.json`](settings.json) file in the root directory. |
|||
|
|||
```json |
|||
{ |
|||
"endpoint": "0.0.0.0:8000", |
|||
"token": "changeme", |
|||
"rate_limiter": { |
|||
"requests_per_minute": 20, |
|||
"allowed_burst": 5 |
|||
}, |
|||
"storage_provider": { |
|||
"type": 0, |
|||
"base_path": "assets" |
|||
} |
|||
} |
|||
``` |
|||
|
|||
## Rate Limiting |
|||
|
|||
By default, the rate limiting takes place on a per-route basis. When the limit for a specific route is hit, the response will return a status code `429: Too Many Requests`. |
|||
|
|||
| Headers | Explanation | |
|||
| --------------------- | -------------------------------- | |
|||
| X-Ratelimit-Limit | Allowed requests per minute | |
|||
| X-Ratelimit-Remaining | Remaining requests | |
|||
| X-Ratelimit-Reset | Seconds until requests replenish | |
|||
|
|||
## API |
|||
|
|||
### `GET` `/` |
|||
|
|||
Show application information. |
|||
|
|||
**Required headers**: |
|||
|
|||
```shell |
|||
Authorization: Bearer <Token> |
|||
``` |
|||
|
|||
**Example response**: |
|||
|
|||
```json |
|||
{ |
|||
"name": "Lithium", |
|||
"version": "0.1.0" |
|||
} |
|||
``` |
|||
|
|||
### `GET` `/pipelines/{pipeline}` |
|||
|
|||
Show pipeline information. |
|||
|
|||
**Required headers**: |
|||
|
|||
```shell |
|||
Authorization: Bearer <Token> |
|||
``` |
|||
|
|||
**Example response**: |
|||
|
|||
```json |
|||
{ |
|||
"name": "example pipeline", |
|||
"slug": "example", |
|||
"type": 0, |
|||
"remove_metadata": false, |
|||
"steps": [ |
|||
{ |
|||
"name": "resize image", |
|||
"type": 0, |
|||
"options": { |
|||
"width": 1280, |
|||
"height": 720, |
|||
"upscale": false |
|||
} |
|||
} |
|||
], |
|||
"output": { |
|||
"format": "jpg", |
|||
"quality": 90 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
## Pipelines |
|||
|
|||
The project uses a pipeline system defined by [JSON](https://en.wikipedia.org/wiki/JSON) files located in the [config](config) folder. |
|||
Take a look at the [example pipeline configuration file](config/example.json). |
|||
|
|||
Available pipeline `type`s: `0` (Image), `1` (Video) |
|||
|
|||
```json |
|||
{ |
|||
"name": "example pipeline", |
|||
"slug": "example", |
|||
"type": 0, |
|||
"removeMetadata": false, |
|||
"steps": [], |
|||
"output": {} |
|||
} |
|||
``` |
|||
|
|||
### Images pipeline |
|||
|
|||
The image pipeline offers the following additional output options. |
|||
|
|||
Available `format` types: `jpg` (or `jpeg`), `png`, `gif`, `tif` (or `tiff`) and `bmp` |
|||
The `quality` field can contain any integer between 1 and 100. |
|||
|
|||
```json |
|||
{ |
|||
"output": { |
|||
"format": "jpg", |
|||
"quality": 90 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
### Available pipeline steps |
|||
|
|||
Each pipeline step consists of an optional `name`, a predefined `type` and configurable `options`. |
|||
See the available options below for more information. |
|||
|
|||
```json |
|||
{ |
|||
"name": "step name", |
|||
"type": 0, |
|||
"options": {} |
|||
} |
|||
``` |
|||
|
|||
#### Resizing images |
|||
|
|||
Resize an image by a given `width` and `height`. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 0, |
|||
"options": { |
|||
"width": 1280, |
|||
"height": 720 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
#### Rotating images |
|||
|
|||
Rotate an image with a given `angle` in degrees. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 1, |
|||
"options": { |
|||
"angle": 90.0 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
#### Flipping images |
|||
|
|||
Flip an image with a given direction. |
|||
Allowed values for the `direction` option are `"h"` (horizontal), `"v"` (vertical) |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 2, |
|||
"options": { |
|||
"direction": "h" |
|||
} |
|||
} |
|||
``` |
|||
|
|||
#### Grayscale |
|||
|
|||
Convert the colorspace of an image into grayscale. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 3 |
|||
} |
|||
``` |
|||
|
|||
#### Fit |
|||
|
|||
Scales down the image to fit the specified maximum width and height. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 4, |
|||
"options": { |
|||
"height": 300, |
|||
"width": 200 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
#### Invert |
|||
|
|||
Invert image colors. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 5 |
|||
} |
|||
``` |
|||
|
|||
#### Blur |
|||
|
|||
Blur image using Gaussian functions. |
|||
|
|||
**Step definition**: |
|||
|
|||
```json |
|||
{ |
|||
"type": 6, |
|||
"options": { |
|||
"sigma": 50.0 |
|||
} |
|||
} |
|||
``` |
|||
|
|||
## Authors |
|||
|
|||
- [Fabian Vowie](https://gogs.informatik.hs-fulda.de/FabianVowie) |
|||
- [Roman Zipp](https://gogs.informatik.hs-fulda.de/roman.zipp) |
|||
@ -0,0 +1,3 @@ |
|||
#!/bin/bash |
|||
|
|||
GIT_COMMIT=$(git rev-parse --short HEAD); go build -ldflags "-X main.GitCommit=$GIT_COMMIT" |
|||
@ -0,0 +1,11 @@ |
|||
{"level":"info","msg":"Pipeline routes registered successfully","time":"2022-01-23T18:28:12+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-01-23T18:28:12+01:00"} |
|||
{"level":"info","msg":"Pipeline routes registered successfully","time":"2022-01-23T18:28:17+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-01-23T18:28:17+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-01-26T17:37:28+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-01-26T17:47:13+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-01-26T17:48:17+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-02-06T15:30:47+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-02-06T15:38:27+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-02-06T15:45:25+01:00"} |
|||
{"level":"info","msg":"Lithium started, listening for requests...","time":"2022-02-06T16:03:30+01:00"} |
|||
@ -1,16 +0,0 @@ |
|||
# Lithium |
|||
|
|||
Micro-service for file storage and processing. |
|||
|
|||
## Features |
|||
|
|||
- File storing with various providers |
|||
- S3 (or S3 compatible like MinIO) |
|||
- Locally (for development purposes) |
|||
- File processing "pipelines" for various formats |
|||
- Compression, Resizing for images |
|||
- Encoding for videos |
|||
- Remove metadata (e.g. EXIF) |
|||
- File-based configuration for pipelines |
|||
- JSON/YAML/TOML? |
|||
- Web api to store and retrieve files |
|||
@ -0,0 +1,28 @@ |
|||
package middlewares |
|||
|
|||
import ( |
|||
"net/http" |
|||
"strings" |
|||
) |
|||
|
|||
type AuthenticationMiddleware struct { |
|||
secret string |
|||
} |
|||
|
|||
func (middleware AuthenticationMiddleware) Middleware(next http.Handler) http.Handler { |
|||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
|||
authToken := r.Header.Get("Authorization") |
|||
|
|||
if authToken == "" || strings.HasPrefix(authToken, "Bearer ") == false || authToken[7:] != middleware.secret { |
|||
http.Error(w, "Forbidden", http.StatusForbidden) |
|||
} else { |
|||
next.ServeHTTP(w, r) |
|||
} |
|||
}) |
|||
} |
|||
|
|||
func CreateAuthenticationMiddleware(secret string) AuthenticationMiddleware { |
|||
return AuthenticationMiddleware{ |
|||
secret, |
|||
} |
|||
} |
|||
@ -0,0 +1,50 @@ |
|||
package middlewares |
|||
|
|||
import ( |
|||
"net/http" |
|||
"net/http/httptest" |
|||
"testing" |
|||
|
|||
"github.com/bxcodec/faker/v3" |
|||
"github.com/stretchr/testify/assert" |
|||
) |
|||
|
|||
func TestAuthenticationMiddleware(t *testing.T) { |
|||
token := faker.Word() |
|||
middleware := CreateAuthenticationMiddleware(token) |
|||
|
|||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
|||
w.WriteHeader(http.StatusOK) |
|||
}) |
|||
|
|||
middlewareHandler := middleware.Middleware(handler) |
|||
|
|||
t.Run("AuthenticationMiddleware returns 403 response when authorization header is incorrect", func(t *testing.T) { |
|||
request, _ := http.NewRequest("GET", "/", nil) |
|||
responseRecorder := httptest.NewRecorder() |
|||
|
|||
middlewareHandler.ServeHTTP(responseRecorder, request) |
|||
|
|||
assert.Equal(t, 403, responseRecorder.Code) |
|||
}) |
|||
|
|||
t.Run("AuthenticationMiddleware returns 403 response when authorization header is missing Bearer prefix", func(t *testing.T) { |
|||
request, _ := http.NewRequest("GET", "/", nil) |
|||
request.Header.Set("Authorization", token) |
|||
responseRecorder := httptest.NewRecorder() |
|||
|
|||
middlewareHandler.ServeHTTP(responseRecorder, request) |
|||
|
|||
assert.Equal(t, 403, responseRecorder.Code) |
|||
}) |
|||
|
|||
t.Run("AuthenticationMiddleware continues when authorization header is correct", func(t *testing.T) { |
|||
request, _ := http.NewRequest("GET", "/", nil) |
|||
request.Header.Set("Authorization", "Bearer "+token) |
|||
responseRecorder := httptest.NewRecorder() |
|||
|
|||
middlewareHandler.ServeHTTP(responseRecorder, request) |
|||
|
|||
assert.Equal(t, 200, responseRecorder.Code) |
|||
}) |
|||
} |
|||
@ -0,0 +1,43 @@ |
|||
package middlewares |
|||
|
|||
import ( |
|||
"net/http" |
|||
|
|||
"github.com/throttled/throttled/store/memstore" |
|||
"github.com/throttled/throttled/v2" |
|||
) |
|||
|
|||
type RateLimiterMiddleware struct { |
|||
rateLimiter throttled.HTTPRateLimiter |
|||
} |
|||
|
|||
func (middleware RateLimiterMiddleware) Middleware(next http.Handler) http.Handler { |
|||
return middleware.rateLimiter.RateLimit(next) |
|||
} |
|||
|
|||
func CreateRateLimiterMiddleware(requestsPerMinute int, allowedBurst int) (*RateLimiterMiddleware, error) { |
|||
store, err := memstore.New(65536) |
|||
|
|||
if err != nil { |
|||
return nil, err |
|||
} |
|||
|
|||
quota := throttled.RateQuota{ |
|||
MaxRate: throttled.PerMin(requestsPerMinute), |
|||
MaxBurst: allowedBurst, |
|||
} |
|||
|
|||
rateLimiter, err := throttled.NewGCRARateLimiter(store, quota) |
|||
if err != nil { |
|||
return nil, err |
|||
} |
|||
|
|||
httpRateLimiter := throttled.HTTPRateLimiter{ |
|||
RateLimiter: rateLimiter, |
|||
VaryBy: &throttled.VaryBy{Path: true}, |
|||
} |
|||
|
|||
return &RateLimiterMiddleware{ |
|||
rateLimiter: httpRateLimiter, |
|||
}, nil |
|||
} |
|||
@ -0,0 +1,64 @@ |
|||
package middlewares |
|||
|
|||
import ( |
|||
"net/http" |
|||
"net/http/httptest" |
|||
"testing" |
|||
|
|||
"github.com/stretchr/testify/assert" |
|||
) |
|||
|
|||
func ExecuteRequest(middlewareHandler http.Handler) int { |
|||
request, _ := http.NewRequest("GET", "/", nil) |
|||
responseRecorder := httptest.NewRecorder() |
|||
|
|||
middlewareHandler.ServeHTTP(responseRecorder, request) |
|||
|
|||
return responseRecorder.Code |
|||
} |
|||
|
|||
func TestRateLimiterMiddleware(t *testing.T) { |
|||
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { |
|||
w.WriteHeader(http.StatusOK) |
|||
}) |
|||
|
|||
t.Run("AuthorizationMiddleware returns 200 response when rate limit is not hit", func(t *testing.T) { |
|||
middleware, err := CreateRateLimiterMiddleware(1, 0) |
|||
assert.Nil(t, err) |
|||
|
|||
middlewareHandler := middleware.Middleware(handler) |
|||
|
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
}) |
|||
|
|||
t.Run("AuthorizationMiddleware returns 429 response when rate limit is hit", func(t *testing.T) { |
|||
middleware, err := CreateRateLimiterMiddleware(1, 0) |
|||
assert.Nil(t, err) |
|||
|
|||
middlewareHandler := middleware.Middleware(handler) |
|||
|
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
assert.Equal(t, 429, ExecuteRequest(middlewareHandler)) |
|||
}) |
|||
|
|||
t.Run("AuthorizationMiddleware returns 200 response when rate limit with burst is not hit", func(t *testing.T) { |
|||
middleware, err := CreateRateLimiterMiddleware(1, 1) |
|||
assert.Nil(t, err) |
|||
|
|||
middlewareHandler := middleware.Middleware(handler) |
|||
|
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
}) |
|||
|
|||
t.Run("AuthorizationMiddleware returns 429 response when rate limit with burst is hit", func(t *testing.T) { |
|||
middleware, err := CreateRateLimiterMiddleware(1, 1) |
|||
assert.Nil(t, err) |
|||
|
|||
middlewareHandler := middleware.Middleware(handler) |
|||
|
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
assert.Equal(t, 200, ExecuteRequest(middlewareHandler)) |
|||
assert.Equal(t, 429, ExecuteRequest(middlewareHandler)) |
|||
}) |
|||
} |
|||
@ -0,0 +1,16 @@ |
|||
{ |
|||
"endpoint": "127.0.0.1:8000", |
|||
"authentication": { |
|||
"enabled": true, |
|||
"token": "changeme" |
|||
}, |
|||
"rate_limiter": { |
|||
"enabled": true, |
|||
"requests_per_minute": 20, |
|||
"allowed_burst": 5 |
|||
}, |
|||
"storage_provider": { |
|||
"type": 0, |
|||
"base_path": "assets" |
|||
} |
|||
} |
|||
@ -0,0 +1,91 @@ |
|||
package settings |
|||
|
|||
import ( |
|||
"encoding/json" |
|||
"os" |
|||
"path/filepath" |
|||
|
|||
"github.com/spf13/afero" |
|||
) |
|||
|
|||
const ( |
|||
Local FileSystemType = iota |
|||
) |
|||
|
|||
type FileSystemType int |
|||
|
|||
type Settings struct { |
|||
Endpoint string `json:"endpoint"` |
|||
Authentication AuthenticationSettings `json:"authentication"` |
|||
RateLimiter RateLimiterSettings `json:"rate_limiter"` |
|||
StorageProvider StorageSettings `json:"storage_provider"` |
|||
} |
|||
|
|||
type StorageSettings struct { |
|||
Type FileSystemType `json:"type"` |
|||
BasePath string `json:"base_path"` |
|||
} |
|||
|
|||
type AuthenticationSettings struct { |
|||
Enabled bool `json:"enabled"` |
|||
Token string `json:"token"` |
|||
} |
|||
|
|||
type RateLimiterSettings struct { |
|||
Enabled bool `json:"enabled"` |
|||
RequestsPerMinute int `json:"requests_per_minute"` |
|||
AllowedBurst int `json:"allowed_burst"` |
|||
} |
|||
|
|||
func parseSettings(data []byte) Settings { |
|||
settings := Settings{} |
|||
|
|||
err := json.Unmarshal(data, &settings) |
|||
if err != nil { |
|||
return Settings{} |
|||
} |
|||
|
|||
return settings |
|||
} |
|||
|
|||
func LoadSettings(fileSystem afero.Fs) (Settings, error) { |
|||
workingDirectory, _ := os.Getwd() |
|||
path := filepath.Join(workingDirectory, "settings.json") |
|||
|
|||
// Load file and parse file
|
|||
data, err := afero.ReadFile(fileSystem, path) |
|||
|
|||
if err == nil { |
|||
return parseSettings(data), nil |
|||
} |
|||
|
|||
// If file does not exist, create default settings
|
|||
defaultSettings := Settings{ |
|||
Endpoint: "127.0.0.1:8000", |
|||
Authentication: AuthenticationSettings{ |
|||
Enabled: false, |
|||
Token: "changeme", |
|||
}, |
|||
RateLimiter: RateLimiterSettings{ |
|||
Enabled: true, |
|||
RequestsPerMinute: 20, |
|||
AllowedBurst: 5, |
|||
}, |
|||
StorageProvider: StorageSettings{ |
|||
Type: Local, |
|||
BasePath: "assets", |
|||
}, |
|||
} |
|||
|
|||
serializedSettings, err := json.MarshalIndent(defaultSettings, "", "\t") |
|||
if err != nil { |
|||
return Settings{}, err |
|||
} |
|||
|
|||
err = afero.WriteFile(fileSystem, path, serializedSettings, os.ModePerm) |
|||
if err != nil { |
|||
return Settings{}, err |
|||
} |
|||
|
|||
return defaultSettings, nil |
|||
} |
|||
@ -0,0 +1,57 @@ |
|||
package settings |
|||
|
|||
import ( |
|||
"os" |
|||
"path/filepath" |
|||
"testing" |
|||
|
|||
"github.com/spf13/afero" |
|||
"github.com/stretchr/testify/assert" |
|||
) |
|||
|
|||
func TestSettingsParsing(t *testing.T) { |
|||
const file string = `{ |
|||
"endpoint": "0.0.0.0:8000", |
|||
"authentication": { |
|||
"enabled": true, |
|||
"token": "foobar" |
|||
}, |
|||
"rate_limiter": { |
|||
"enabled": true, |
|||
"requests_per_minute": 20, |
|||
"allowed_burst": 5 |
|||
}, |
|||
"storage_provider": { |
|||
"type": 0, |
|||
"base_path": "assets" |
|||
} |
|||
}` |
|||
|
|||
t.Run("Settings parsing is successful", func(t *testing.T) { |
|||
settings := parseSettings([]byte(file)) |
|||
|
|||
assert.Equal(t, "0.0.0.0:8000", settings.Endpoint) |
|||
assert.Equal(t, "foobar", settings.Authentication.Token) |
|||
assert.Equal(t, "assets", settings.StorageProvider.BasePath) |
|||
}) |
|||
} |
|||
|
|||
func TestSettingsLoading(t *testing.T) { |
|||
t.Run("Settings loading creates default settings.json when none is present", func(t *testing.T) { |
|||
fileSystem := afero.NewMemMapFs() |
|||
workingDirectory, _ := os.Getwd() |
|||
|
|||
path := filepath.Join(workingDirectory, "settings.json") |
|||
|
|||
// Settings file does not exist in the beginning
|
|||
doesFileExist, _ := afero.Exists(fileSystem, path) |
|||
assert.False(t, doesFileExist) |
|||
|
|||
_, err := LoadSettings(fileSystem) |
|||
assert.Nil(t, err) |
|||
|
|||
// Settings file should be present after calling LoadSettings
|
|||
doesFileExist, _ = afero.Exists(fileSystem, path) |
|||
assert.True(t, doesFileExist) |
|||
}) |
|||
} |
|||
|
Before Width: 800 | Height: 500 | Size: 66 KiB After Width: 800 | Height: 500 | Size: 66 KiB |
|
Before Width: 900 | Height: 900 | Size: 128 KiB After Width: 900 | Height: 900 | Size: 128 KiB |
Write
Preview
Loading…
Cancel
Save
Reference in new issue