Skip to content

Backends

Ralph supports various storage and database backends that can be accessed to read from or write learning events to. Implemented backends are listed below along with their configuration parameters. If your favourite backend is missing to the list, feel free to submit your implementation or get in touch!

Key concepts

We distinguish storage from database backends as the semantic and concepts are quite different in terms of code, but those two types of backends can be considered as “backends” from a Ralph user perspective as the CLI can use both in most of its commands.

Each backend has its own parameters that are required to use it. Those parameters can be set as command line options or environment variables; the later is the recommended solution for sensible data such as service credentials. If we consider the os_username (OpenStack user name) parameter of the OpenStack Swift backend, it can be set as a command line option using swift as the option prefix (and replacing underscores in its name by dashes):

$ ralph list --backend swift --swift-os-username johndoe # [...] more options

Alternatively, this parameter can be set as an environment variable (in upper case, prefixed by the program name, e.g. RALPH_):

$ export RALPH_SWIFT_OS_USERNAME="johndoe"
$ ralph list --backend swift # [...] more options

The general patterns for backend parameters are:

  • --{{ backend_name }}-{{ parameter | underscore_to_dash }} for command options, and,
  • RALPH_{{ backend_name | uppercase }}_{{ parameter | uppercase }} for environment variables.

Storage backends

OVH - Log Data Platform (LDP)

LDP is a nice service built by OVH on top of Graylog to follow, analyse and store your logs. Learning events (aka tracking logs) can be stored in GELF format using this backend.

For now the LDP backend is read-only as we consider that it is mostly used to collect primary logs and not as a Ralph target. Feel free to get in touch to prove us wrong, or better: submit your proposal for the write method implementation.

Backend parameters

To access OVH’s LDP API, you need to register Ralph as an authorized application and generate an application key, an application secret and a consumer key.

While filling the registration form available at: eu.api.ovh.com/createToken/, be sure to give an appropriate validity time span to your token and allow only GET requests on the /dbaas/logs/* path.

  • endpoint: the API endpoint (e.g. ovh-eu)
  • application_key: use generated application key
  • application_secret: use generated application secret
  • consumer_key: use generated consumer key

The following parameters are required to fetch archives from an LDP account stream:

  • service_name: the LDP account name (e.g. ldp-xx-xxxxx)
  • stream_id: the identifier of the stream you are querying (e.g. a UUID hex representation: 77ec6e4a-ac15-4bcf-8043-7429bf275e49)

For more information about OVH’s API client parameters, please refer to the project’s documentation: github.com/ovh/python-ovh.

OpenStack Swift

Swift is the OpenStack object storage service. This storage backend is fully supported (read and write operations) to stream and store log archives.

Backend parameters

Primarily required parameters correspond to a standard authentication using OpenStack Keystone service:

  • os_identity_api_version: keystone API version you will authenticate to (defaults to 3)
  • os_auth_url: the authentication URL (defaults to OVH’s Swift https://auth.cloud.ovh.net/)
  • os_project_domain_name: the project domain name (defaults to Default)
  • os_user_domain_name: the user domain name (defaults to Default)
  • os_username: the name of your openstack swift user
  • os_password: the password of your openstack swift user

Secondary parameters are required to work with the target container:

  • os_storage_url: the URL of the target container
  • os_region_name: the region where your container is
  • os_tenant_name: the name of the tenant of your container
  • os_tenant_id: the identifier of the tenant of your container

File system

The file system backend is a dummy template that can be used to develop your own backend. It’s a “dummy” backend as it’s not required in a UNIX Shell context, the ls and cat commands used along with UNIX streams will do a better job.

Backend parameters

The only required parameter is the path we want to list or stream content from.

WebSocket

The webSocket backend is read-only and can be used to get real-time events.

If you use OVH’s Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

Backend parameters

  • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)

Database backends

Elasticsearch

Elasticsearch backend is mostly used for indexation purpose (as a datalake) but it can also be used to fetch indexed data from it.

Backend parameters

Elasticsearch backend parameters required to connect to a cluster are:

  • hosts: a list of cluster hosts to connect to (e.g. ["http://elasticsearch-node:9200"])
  • index: the elasticsearch index where to get/put documents
  • client_options: all options from the official client are supported and can be specified as a python dict object (e.g. {'use_ssl': True, 'verify_certs': True})

For a complete list of supported client_options, please refer to the official client’s documentation.