Micro-service for file storage and processing written in Go
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Jenkins d76986ae2a Merge commit 'b5c0c9700883928856820a678e432e78558862a6' into HEAD 2 years ago
.github/workflows Add CI 2 years ago
config Update output config format to allow string extensions 2 years ago
controllers Add check for directories in files endpoint 2 years ago
middlewares Add allowed burst for requests to rate limiting middleware 2 years ago
pipelines Add structured logging 2 years ago
settings Exclude index page from authentication 2 years ago
storage Add storage provider open method 2 years ago
tests/files Add pipeline test image dimension assertions 2 years ago
.gitignore Exclude files directory from vcs 2 years ago
README.md Add get files endpoint readme docs 2 years ago
build.sh Add commit hash to index metadata 2 years ago
go.mod Add structured logging 2 years ago
go.sum Add structured logging 2 years ago
main.go Add get files endpoint 2 years ago
main_test.go Add get files endpoint 2 years ago
settings.json Fix store methods not being public 2 years ago

README.md

Lithium

Micro-service for file storage and processing written in Go.

Features

  • Image processing pipelines for various transformations
  • Web API for storing & retrieving files

Requirements

Setup

1. Clone repository

git clone git@gogs.informatik.hs-fulda.de:FabianVowie/Lithium.git

2. Pull dependencies

go get

3. Build & start application

go run .

Run using Docker container

docker run --rm -p 8000:8000 -v "$PWD":/usr/src/lithium -w /usr/src/lithium golang:1.17 go run .

Testing

go test ./...

Run tests in verbose logging mode

go test ./... -v

Run tests in Docker container

docker run --rm -v "$PWD":/usr/src/lithium -w /usr/src/lithium golang:1.17 go test ./...

Configuration

Config options can be adjusted via the settings.json file in the root directory.

{
  "endpoint": "0.0.0.0:8000",
  "token": "changeme",
  "rate_limiter": {
    "requests_per_minute": 20,
    "allowed_burst": 5
  },
  "storage_provider": {
    "type": 0,
    "base_path": "assets"
  }
}

Rate Limiting

By default, the rate limiting takes place on a per-route basis. When the limit for a specific route is hit, the response will return a status code 429: Too Many Requests.

Headers Explanation
X-Ratelimit-Limit Allowed requests per minute
X-Ratelimit-Remaining Remaining requests
X-Ratelimit-Reset Seconds until requests replenish

API

GET /

Show application information.

Required headers:

Authorization: Bearer <Token>

Example response:

{
  "name": "Lithium",
  "version": "0.1.0"
}

GET /files

Show files.

Required headers:

Authorization: Bearer <Token>

Example response:

{
  "files": [
    {
      "bucket": "foo",
      "file_name": "output.jpg",
      "size": 14523,
      "modified_at": "2022-02-14T14:19:07.491308411+01:00"
    }
  ]
}

GET /pipelines/{pipeline}

Show pipeline information.

Required headers:

Authorization: Bearer <Token>

Example response:

{
  "name": "example pipeline",
  "slug": "example",
  "type": 0,
  "remove_metadata": false,
  "steps": [
    {
      "name": "resize image",
      "type": 0,
      "options": {
        "width": 1280,
        "height": 720,
        "upscale": false
      }
    }
  ],
  "output": {
    "format": "jpg",
    "quality": 90
  }
}

Pipelines

The project uses a pipeline system defined by JSON files located in the config folder. Take a look at the example pipeline configuration file.

Available pipeline types: 0 (Image), 1 (Video)

{
  "name": "example pipeline",
  "slug": "example",
  "type": 0,
  "removeMetadata": false,
  "steps": [],
  "output": {}
}

Images pipeline

The image pipeline offers the following additional output options.

Available format types: jpg (or jpeg), png, gif, tif (or tiff) and bmp
The quality field can contain any integer between 1 and 100.

{
  "output": {
    "format": "jpg",
    "quality": 90
  }
}

Available pipeline steps

Each pipeline step consists of an optional name, a predefined type and configurable options. See the available options below for more information.

{
  "name": "step name",
  "type": 0,
  "options": {}
}

Resizing images

Resize an image by a given width and height.

Step definition:

{
  "type": 0,
  "options": {
    "width": 1280,
    "height": 720
  }
}

Rotating images

Rotate an image with a given angle in degrees.

Step definition:

{
  "type": 1,
  "options": {
    "angle": 90.0
  }
}

Flipping images

Flip an image with a given direction. Allowed values for the direction option are "h" (horizontal), "v" (vertical)

Step definition:

{
  "type": 2,
  "options": {
    "direction": "h"
  }
}

Grayscale

Convert the colorspace of an image into grayscale.

Step definition:

{
  "type": 3
}

Fit

Scales down the image to fit the specified maximum width and height.

Step definition:

{
  "type": 4,
  "options": {
    "height": 300,
    "width": 200
  }
}

Invert

Invert image colors.

Step definition:

{
  "type": 5
}

Blur

Blur image using Gaussian functions.

Step definition:

{
  "type": 6,
  "options": {
    "sigma": 50.0
  }
}

Authors