Upgrade

All instructions to upgrade this project from one release to the next will be documented in this file. Upgrades must be run sequentially, meaning you should not skip minor/major releases while upgrading (fix releases can be skipped).

This project adheres to Semantic Versioning.

5.1.0 — partial success (issue #622)

partialSuccess (issue #622)

Optional bulk ingestion when some statements in a batch are invalid.

This feature is disabled by default and does nothing until you turn it on. POST /xAPI/statements stays atomic (xAPI-strict) unless one of the following applies:

Trigger	Set by
?partialSuccess=true or ?ignoreInvalid=true	the client, per request
RALPH_LRS_PARTIAL_SUCCESS_DEFAULT=true (default: false)	the server, globally

Clients that cannot add a query parameter — the Moodle logstore_xapi plugin is the typical case — are only covered by the server setting. Upgrading the image without setting it changes nothing for them.

Once the server default is on, a client can still force strict behaviour per request with ?partialSuccess=false.

See the partial success tutorial.

Elasticsearch per-statement isolation (5.0.2-beta1)

In partial-success mode, statements that pass Pydantic validation but fail Elasticsearch indexation are skipped individually (_write_statements_partial) instead of failing the whole batch with HTTP 500.

Elasticsearch-compatible dict keys (5.1.0)

When RUNSERVER_BACKEND=es and RALPH_LRS_ELASTICSEARCH_VALIDATE_KEYS=true (default), POST /xAPI/statements rejects statements whose JSON object keys are incompatible with Elasticsearch before indexation:

• empty string key ("") — e.g. quiz match extension with no response key
• non-IRI key containing . — e.g. nested.key inside an extension map

xAPI extension IRI keys (URLs with dots) remain valid. Only local / non-IRI keys with a dot are rejected.

Opt out: RALPH_LRS_ELASTICSEARCH_VALIDATE_KEYS=false.

Enable RALPH_LRS_PARTIAL_SUCCESS_DEFAULT first. In strict mode this check rejects the whole batch before any write, so nothing is indexed. Up to 5.0.2-beta1, such a batch returned HTTP 500 while the valid statements were still written to Elasticsearch. Deployments relying on that partial write will lose those statements unless partial-success mode is turned on beforehand.

4.x to 5.y

Upgrade learning events models

xAPI learning statements validator and converter are built with Pydantic. Ralph 5.x is compatible with Pydantic 2.x. Please refer to Pydantic migration guide if you are using Ralph models feature.

Most of fields in Pydantic models that are optional are set with None as default value in Ralph 5.y. If you serialize some Pydantic models from ralph and want to keep the same content in your serialization, please set exclude_none to True in the serialization method model_dump.

3.x to 4.y

Upgrade user credentials

To conform to xAPI specifications, we need to represent users as xAPI Agents. You must therefore delete and re-create the credentials file using the updated cli, OR you can modify it directly to add the agents field. The credentials file is located in { RALPH_APP_DIR }/{ RALPH_AUTH_FILE } (defaults to .ralph/auth.json). Each user profile must follow the following pattern (see this post for examples of valid agent objects) :

{
  "username": "USERNAME_UNCHANGED",
  "hash": "PASSWORD_HASH_UNCHANGED",
  "scopes": [ LIST_OF_SCOPES_UNCHANGED ],
  "agent": { A_VALID_AGENT_OBJECT }
}

Agent can take one of the following forms, as specified by the xAPI specification: - mbox:

"agent": {
      "mbox": "mailto:john.doe@example.com"
}

- mbox_sha1sum:

"agent": {
        "mbox_sha1sum": "ebd31e95054c018b10727ccffd2ef2ec3a016ee9",
}

- openid:

"agent": {
      "openid": "http://foo.openid.example.org/"
}

- account:

"agent": {
      "account": {
        "name": "simonsAccountName",
        "homePage": "http://www.exampleHomePage.com"
}

For example here is a valid auth.json file:

[
  {
    "username": "john.doe@example.com",
    "hash": "$2b$12$yBXrzIuRIk6yaft5KUgVFOIPv0PskCCh9PXmF2t7pno.qUZ5LK0D2",
    "scopes": ["example_scope"],
    "agent": {
      "mbox": "mailto:john.doe@example.com"
    }
  },
  {
    "username": "simon.says@example.com",
    "hash": "$2b$12$yBXrzIuRIk6yaft5KUgVFOIPv0PskCCh9PXmF2t7pno.qUZ5LK0D2",
    "scopes": ["second_scope", "third_scope"],
    "agent": {
      "account": {
        "name": "simonsAccountName",
        "homePage": "http://www.exampleHomePage.com"
      }
    }
  }
]

Upgrade Ralph CLI usage

If you are using Ralph’s CLI, the following changes may affect you:

• The ralph fetch command changed to ralph read
• The -b ws backend option changed to -b async_ws
  • The corresponding --ws-uri option changed to --async-ws-uri
• The -c --chunk-size option changed to -s --chunk-size

• The DEFAULT_BACKEND_CHUNK_SIZE environment variable configuration is removed in favor of allowing each backend to define their own defaults:

  Backend	Environment variable for default (read) chunk size
  async_es/es	RALPH_BACKENDS__DATA__ES__READ_CHUNK_SIZE=500
  async_lrs/lrs	RALPH_BACKENDS__DATA__LRS__READ_CHUNK_SIZE=500
  async_mongo/mongo	RALPH_BACKENDS__DATA__MONGO__READ_CHUNK_SIZE=500
  clickhouse	RALPH_BACKENDS__DATA__CLICKHOUSE__READ_CHUNK_SIZE=500
  fs	RALPH_BACKENDS__DATA__FS__READ_CHUNK_SIZE=4096
  ldp	RALPH_BACKENDS__DATA__LDP__READ_CHUNK_SIZE=4096
  s3	RALPH_BACKENDS__DATA__S3__READ_CHUNK_SIZE=4096
  swift	RALPH_BACKENDS__DATA__SWIFT__READ_CHUNK_SIZE=4096

• The ralph push command changed to ralph write

• The -c --chunk-size option changed to -s --chunk-size

• The DEFAULT_BACKEND_CHUNK_SIZE environment variable configuration is removed in favor of allowing each backend to define their own defaults:

  Backend	Environment variable for default (write) chunk size
  async_es/es	RALPH_BACKENDS__DATA__ES__WRITE_CHUNK_SIZE=500
  async_lrs/lrs	RALPH_BACKENDS__DATA__LRS__WRITE_CHUNK_SIZE=500
  async_mongo/mongo	RALPH_BACKENDS__DATA__MONGO__WRITE_CHUNK_SIZE=500
  clickhouse	RALPH_BACKENDS__DATA__CLICKHOUSE__WRITE_CHUNK_SIZE=500
  fs	RALPH_BACKENDS__DATA__FS__WRITE_CHUNK_SIZE=4096
  ldp	RALPH_BACKENDS__DATA__LDP__WRITE_CHUNK_SIZE=4096
  s3	RALPH_BACKENDS__DATA__S3__WRITE_CHUNK_SIZE=4096
  swift	RALPH_BACKENDS__DATA__SWIFT__WRITE_CHUNK_SIZE=4096

• Environment variables used to configure backend options for CLI usage (read/write/list commands) changed their prefix: RALPH_BACKENDS__{{DATABASE or HTTP or STORAGE or STREAM}}__{{BACKEND}}__{{OPTION}} changed to RALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}

• Environment variables used to configure backend options for LRS usage (runserver command) changed their prefix: RALPH_BACKENDS__{{DATABASE}}__{{BACKEND}}__{{OPTION}} changed to RALPH_BACKENDS__LRS__{{BACKEND}}__{{OPTION}}

Upgrade history syntax

CLI syntax has been changed from fetch & push to read & write affecting the command history. You must replace the command history after updating: - locate your history file path, which is in { RALPH_APP_DIR }/history.json (defaults to .ralph/history.json) - run the commands below to update history

sed -i 's/"fetch"/"read"/g' { my_history_file_path }
sed -i 's/"push"/"write"/g' { my_history_file_path }

Upgrade Ralph library usage (backends)

If you use Ralph’s backends in your application, the following changes might affect you:

Backends from ralph.backends.database, ralph.backends.http, ralph.backends.stream, and ralph.backends.storage packages have moved to a single ralph.backends.data package.

Ralph v3 (database/http/storage/stream) backends	Ralph v4 data backends
ralph.backends.database.clickhouse.ClickHouseDatabase	ralph.backends.data.clickhouse.ClickHouseDataBackend
ralph.backends.database.es.ESDatabase	ralph.backends.data.es.ESDataBackend
ralph.backends.database.mongo.MongoDatabase	ralph.backends.data.mongo.MongoDataBackend
ralph.backends.http.async_lrs.AsyncLRSHTTP	ralph.backends.data.async_lrs.AsyncLRSDataBackend
ralph.backends.http.lrs.LRSHTTP	ralph.backends.data.lrs.LRSDataBackend
ralph.backends.storage.fs.FSStorage	ralph.backends.data.fs.FSDataBackend
ralph.backends.storage.ldp.LDPStorage	ralph.backends.data.ldp.LDPDataBackend
ralph.backends.storage.s3.S3Storage	ralph.backends.data.s3.S3DataBackend
ralph.backends.storage.swift.SwiftStorage	ralph.backends.data.swift.SwiftDataBackend
ralph.backends.stream.ws.WSStream	ralph.backends.data.async_ws.AsyncWSDataBackend

LRS-specific query_statements and query_statements_by_ids database backend methods have moved to a dedicated ralph.backends.lrs.BaseLRSBackend interface that extends the data backend interface with these two methods.

The query_statements_by_ids method return type changed to Iterator[dict].

Ralph v3 database backends for lrs usage	Ralph v4 LRS data backends
ralph.backends.database.clickhouse.ClickHouseDatabase	ralph.backends.lrs.clickhouse.ClickHouseLRSBackend
ralph.backends.database.es.ESDatabase	ralph.backends.lrs.es.ESLRSBackend
ralph.backends.database.mongo.MongoDatabase	ralph.backends.lrs.mongo.MongoLRSBackend

Backend interface differences

• Data backends are read-only by default
• Data backends that support write operations inherit from the ralph.backends.data.base.Writable interface
• Data backends that support list operations inherit from the ralph.backends.data.base.Listable interface
• Data backends that support LRS operations (query_statements/query_statements_by_ids) inherit from the ralph.backends.lrs.BaseLRSBackend interface
• __init__(self, **kwargs) changed to __init__(self, settings: DataBackendSettings) where each DataBackend defines it’s own Settings object For example the FSDataBackend uses FSDataBackendSettings
• stream and get methods changed to read
• put methods changed to write

Backend usage migration example

Ralph v3 using ESDatabase:

from ralph.conf import ESClientOptions
from ralph.backends.database.es import ESDatabase, ESQuery

# Instantiate the backend.
backend = ESDatabase(
  hosts="localhost",
  index="statements"
  client_options=ESClientOptions(verify_certs=False)
)
# Read records from backend.
query = ESQuery(query={"query": {"term": {"modulo": 0}}})
es_statements = list(backend.get(query))

# Write records to backend.
backend.put([{"id": 1}])

Ralph v4 using ESDataBackend:

from ralph.backends.data.es import (
  ESClientOptions,
  ESDataBackend,
  ESDataBackendSettings,
  ESQuery,
)

# Instantiate the backend.
settings = ESDataBackendSettings(
  HOSTS="localhost",
  INDEX="statements",
  CLIENT_OPTIONS=ESClientOptions(verify_certs=False)
)
backend = ESDataBackend(settings)

# Read records from backend.
query = ESQuery(query={"term": {"modulo": 0}})
es_statements = list(backend.read(query))

# Write records to backend.
backend.write([{"id": 1}])

Upgrade ClickHouse schema

If you are using the ClickHouse backend, schema changes have been made to drop the existing JSON column in favor of the String version of the same data. See this issue for details.

Ralph does not manage the ClickHouse schema, so if you have existing data you will need to manually alter it as an admin user. Note: this will rewrite the statements table, which may take a long time if you have many rows. The command to run is:

-- If RALPH_BACKENDS__DATA__CLICKHOUSE__DATABASE is 'xapi'
-- and RALPH_BACKENDS__DATA__CLICKHOUSE__EVENT_TABLE_NAME is 'test'

ALTER TABLE xapi.test DROP COLUMN event, RENAME COLUMN event_str to event;