Initial xAPI/Caliper Comparison

Executive Summary

At the 1EdTech Quarterly meeting, Monday 15^th August 2016, an initial meeting to discuss the differences between xAPI and Caliper was held. There were over fifty participants from across the World. The meeting was focused on establishing a common understanding of the two specifications. A comparison of the core features of the two specifications was based upon:

a) Use-cases, Scenarios and Motivations;

b) Service Endpoints;

c) Data Models;

d) Security Mechanisms;

e) Transport Mechanisms;

f) Vocabularies, Metric Profiles, Profiles and Recipes;

g) Data Science.

The key conclusions are:

a) Caliper and xAPI have very different origins. The core xAPI is to enable any type of experience and evidence tracking, both electronic and physical performance and not limited to just web-based courses (as is the case for SCORM). Caliper is the manifestation of the 1EdTech Learning Analytics Framework and the Sensor API and Metric Profile(s) are the first two components of that framework. xAPI and Caliper are NOT equivalent. Adoption should not be ‘one-or-the-other’, instead it is a ‘horses-for-courses’ decision;

b) While both xAPI (Actor/Verb/Object) and Caliper (Actor/Action/Activity) use a data model based upon a triple statement structure there are considerable differences in the detailed structure and usage of the Object and Activity definitions. However, it should be possible for each specification to make use of the other’s Verb/Action;

c) Formal processes for the definition and/or modification of the Vocabularies, Metrics Profiles, Profiles and Recipes must be established with exemplars created to demonstrate the best practices when producing the corresponding documentation;

d) Any further work on either/both standards must include explicit participation of Data Scientists with knowledge of Learning Analytics. We need to ensure that the use of xAPI/Caliper can produce useful learning analytics information and not just data;
From a technology realisation perspective, for a next generation, it would simplify common adoption and convergence to agree common payload binding (currently JSON against JSON-LD), common security framework (currently OAuth 1 against APIkey), a common secure transport mechanism (currently HTTPS) and a common endpoint definition approach (including common agreement on the use of query parameters and URL construction).

1. Context for the Comparison

At the 1EdTech Quarterly meeting, Monday 15^th August 2016, an initial meeting to discuss the differences between xAPI and Caliper was held (the venue was Utah Valley University, Orem Utah, USA). There were over fifty participants from across the World. The meeting was focused on establishing a common understanding of the two specifications. A comparison of the core features of the two specifications was based upon:

a) Use-cases, Scenarios and Motivations – identification and clarification of the original scope and context for xAPI and Caliper;

b) Service Endpoints – identification of the types of data exchange that is supported and how this data is exchanged between the endpoints;

c) Data Models – a comparison of the core data features i.e. this analysis does not work down through the detailed data structures;

d) Security Mechanisms – the data authentication, authorization and encryption mechanisms that are supported and/or preferred;

e) Transport Mechanisms – the payload exchange technology i.e. the ways in which the data definitions are physically exchanged across the networking technology;

f) Vocabularies, Metric Profiles, Profiles and Recipes – the mechanisms used to define the data vocabularies and the tailoring of the specification for specific application domains and use-cases;

g) Data Science – identification of the actual learning analytics that can be created and the associated data science perspectives e.g. statistical significance.

Areas out-of-scope for the meeting were:

a) A detailed comparison of the approaches used by xAPI and Caliper for a specific use-case;

b) Establishing the scenarios for which xAPI or Caliper should be deployed;

c) The meeting was an information gathering exercise. It was not charged with making decisions.

2. Detailed Comparison

xAPI Features

Caliper Features

Use-cases/Scenarios/Motivations

The core use-cases are:

· To enable Learning within the SCORM-context and beyond-the-browser, outside of the LMS and outside of the SCORM package;

· Distributed Content: any type of learning content or experience can be delivered from a local computer, local network or on any remote servers;

· Distributed Data: learning data can be stored and shared across one or more systems;

· Usage & Performance Data: paradata about learning resources that include not just quantitative metrics, but also pedagogic context, skills, and performance;

· Team-Based Scenarios: data associated with users can now be aggregated and associated with a team or group of users;

· Instructor/Facilitator Scenarios: instructor or facilitators may observe and send or receive feedback or annotations to users during a learning or performance activity using real-time data collection displayed in an interface or dashboard.

· To provide System-to-System data transfer (including non-LRS based) that allows identification of data ownership, multi-agent statements, with an extensible data model and agnostic of security model.

The core usages are:

· To enable the creation of quantitative metrics for learning;

· To provide real-time data messaging to enable responsive learning engagement as opposed to just archive-based metrics;

· To provide details on student engagement in learning activities;

· To resolve the LTI/Black-box conundrum.

Caliper is 1EdTech’s Learning Analytics Framework of which the Sensor and Metric Profiles are just two components.

Service Endpoints

Supports both ‘reading data from’ and ‘writing data to’ an LRS. Explicit support for:

· Statement API – Create/Read

· State API – CRUD

· Agent Profile API – CRUD

· Activity Profile API – CRUD

· About Resource - read information about the endpoint

The Sensor API is used to Write/Post data to Repository endpoint

· send () – to transmits event data

· describe () – to transit entity data

At present, Caliper does NOT support reading data from a Data Repository i.e. the Sensor API is for writing data to a repository.

Data Models

The data model is based upon the Statement and this a { Actor, Verb, Object} triple:

· An Actor is an Agent or Group (two or more Agents);

· There are four types of Object i.e. Activity, Agent, Statement or Sub-statement. Statements can be composed of sub-statements;

· The Vocabulary for the Verb, Activity Types and Extensions are open.

xAPI can be considered an ‘Activity Scripting Language’.

The data model is based upon the {Actor, Action, Activity } triple:

· The vocabulary for the Action is controlled and constrained by the Metric Profiles;

The data is exchanged either as a set of Events or Entities with an Entity used to describe Actors and Activities. Entities provide context for the Events. Each Event is defined by a ‘Metric Profile’.

Events do not have explicit dependencies i.e. they must be associated through the use of a Session.

Caliper can be considered an ‘Event Scripting Language’.

Security Mechanisms

Support for:

· Basic HTTP Authentication;

· Use of 2-legged and 3-legged OAuth 1.0 (with HMAC-SHA1, RSA-SHA1 and PLAINTEXT) for statement authorization.

There is lot of information on Authorization.

Use of API Key.

Use of HTTPS/TLS 1.3 is recommended to secure the message exchange.

Very little discussion of security.

Transport Mechanisms

The transport is HTTP/HTTPS with JSON payloads. Supports both the requestor (source) and the respondent (LRS) allocating the unique identifier for a Statement that is to be stored.

Statements can be signed and the signature may also be stored in the LRS.

The transport is HTTP/HTTPS with JSON-LD payloads (note that the linked data aspects are not subject to conformance). The message is not signed. For conformance 1EdTech do not address the Linked Data aspects and treat the payload as formal JSON.

There is a Best Practices recommendation for using LTI to provide the sensor endpoint and the corresponding API Key.

Vocabularies, Metric Profiles, Profiles and Recipes

An xAPI Profile is similar to an Application Profile i.e. it is a definition of how to use the xAPI specification in a specific application domain. A Recipe is used to describe how xAPI supports a specific use-case in the application domain e.g. the sequence of statements, etc. xAPI Profiles have the following characteristics:

· Documentation on how to use the profile;

· IRIs used as part of vocabulary;

· Vocabulary Metadata (new vocab guidance suggests all should be marked up and published using RDF / JSON-LD) and returned if IRI requested. Could this be improved to support conformance testing or profile conformance?

· Recipes, for standardized tracking of common activities;

· Rules for authentication, security;

· Specific launch requirements;

· Specific packaging requirements;

· Storage requirements using Document API;

· Specific sequence of statements or rules (cmi5 uses initialized verb first);

· Granularity or aggregation of statements.

A Metric Profile describes either a learning activity or an activity that facilitates learning e.g. grading. Profiles are composed of one or more events. Each event specifies a controlled vocabulary of actions as well as a set of entities that provide a representation of the actor, object of the interaction as well as other elements that together comprise the learning context in which the action is situated.

For Conformance, a vendor must identify the Metric Profiles supported by their implementation. Support for Metric Profiles is optional. Also, an implementation is only required to support a predefined subset of the Actions for a Metric Profile.

There is no established process for the definition and/or modification of a Profile and/or Recipe.

There is no established process for the definition and/or modification of a Metric Profile. Once broader adoption of Caliper has been established 1EdTech will also enable domain profiling (as provided for all deployed 1EdTech specifications) of the Metric Profiles i.e. refinement of the Metric Profile for a market specific requirement.

Data Science

Provides support for identifying an Authority (an Agent or Group of Agents) who asserts statement(s) are true. The Authority is achieved using 2-legged or 3-legged OAuth. An LRS must ensure that all Statements stored have an authority.

Establishment of Metric Profiles is meant to reflect instructionally meaningful interactions. However, the current version represents only initial versions of the metric profiles.

Missing aspects for data science are:

· The required instrumentation of data sources to provide sufficiently rich/useful learning analytics;

· Data provenance and curation;

· Workflows between the various systems;

· Consistent usage/mapping of verbs/actions taking into account evolution of the standards to produce consistent data science;

· Supporting and processing of extensions to the standards and the process for disseminating/adopting such extensions.

Missing aspects for data science are:

· The required instrumentation of data sources to provide sufficiently rich/useful learning analytics;

· Data provenance and curation;

· Workflows between the various systems;

· Consistent usage/mapping of verbs/actions taking into account evolution of the standards to produce consistent data science;

· Supporting and processing of extensions to the standards and the process for disseminating/adopting such extensions.

3. Key Conclusions

The key conclusions are:

a) Caliper and xAPI have very different origins. The core xAPI use-case is to enable learning in a SCORM context (including the replacement of the IEEE CMI). Caliper is the manifestation of the 1EdTech Learning Analytics Framework and the Sensor API and Metric Profile(s) are the first two components of that framework. xAPI and Caliper are NOT equivalent. Therefore, adoption of the current versions should not be ‘one-or-the-other’, instead it is a ‘horses-for-courses’ decision;

Potential Recommendation: ADL/1EdTech produce a White Paper explaining xAPI/Caliper complementary nature.

Potential Recommendation: ADL/1EdTech undertake more detailed exploration of the detailed similarities and differences for xAPI/Caliper.

e) While both xAPI (Actor/Verb/Object) and Caliper (Actor/Action/Activity) use a data model based upon a triple definition there are considerable differences in the structure and usage of the Object and Activity definitions. However, it should be possible for each specification to make use of the other’s Verb/Action definitions;

Potential Recommendation: ADL/1EdTech produce a White Paper describing cross Verb/Action usage. This includes establishing best practices for the use of 1EdTech Metric Profiles with xAPI transport.

f) Formal processes for the definition and/or modification of the Vocabularies, Metrics Profiles, Profiles and Recipes must be established with exemplars created to demonstrate the best practices when producing the corresponding documentation;

Potential Recommendation: Key area for joint ADL/1EdTech collaboration.

g) Any further work on either/both standards must include explicit participation of Data Scientists with knowledge of Learning Analytics. We need to ensure that the use of xAPI/Caliper can produce useful learning analytics information and not just data;

Potential Recommendation: Key area for joint ADL/1EdTech collaboration.

h) From a technology realization perspective, for next generation development, it would simplify common adoption and convergence to agree common payload binding (currently JSON against JSON-LD), common security framework (currently OAuth 1 against APIkey), a common secure transport mechanism (currently HTTPS) and a common endpoint definition approach (including common agreement on the use of query parameters and URL construction).

Potential Recommendation: Key area for joint ADL/1EdTech collaboration.