1EdTech Logo

1EdTech Caliper Analytics™ Implementation Guide

 

Version 1.0 Final

Date Issued:            15 October 2015

Latest version:         http://www.imsglobal.org/caliper

IPR and Distribution Notices
Recipients of this document are requested to submit, with their comments, notification of any relevant patent claims or other intellectual property rights of which they may be aware that might be infringed by any implementation of the specification set forth in this document, and to provide supporting documentation.

1EdTech takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on 1EdTech’s procedures with respect to rights in 1EdTech specifications can be found at the 1EdTech Intellectual Property Rights web page: http://www.imsglobal.org/ipr/imsipr_policyFinal.pdf.

Copyright © 2015 1EdTech Consortium. All Rights Reserved.

Use of this specification to develop products or services is governed by the license with 1EdTech found on the 1EdTech website: http://www.imsglobal.org/license.html.

Permission is granted to all parties to use excerpts from this document as needed in producing requests for proposals.

The limited permissions granted above are perpetual and will not be revoked by 1EdTech or its successors or assigns.

THIS SPECIFICATION IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NONINFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY USE OF THIS SPECIFICATION SHALL BE MADE ENTIRELY AT THE IMPLEMENTER'S OWN RISK, AND NEITHER THE CONSORTIUM, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER, DIRECTLY OR INDIRECTLY, ARISING FROM THE USE OF THIS SPECIFICATION.

© 2015 1EdTech Consortium, Inc.
All Rights Reserved.
Trademark information: http://www.imsglobal.org/copyright.html
Document Name:  1EdTech Caliper Analytics Implementation Guide – Final v1.0
Revision: 15 October 2015

 

Executive Summary

The 1EdTech Caliper Analytics™ Learning Measurement Framework represents a valuable, standards oriented, high impact solution to a very significant problem amidst the online learning delivery ecosystem. Currently, this ecosystem has an insufficient, inconsistent, and fragmented underlying measurement and metrics capability across fast emerging of federated content/learning activity elements sourced from a wide array of providers. By providing the necessary alignment and structure to what is and should be measured, along with a framework to support the capture and marshaling of data, 1EdTech Caliper version 1.0  represents an achievable and valuable starting position. Moreover, the Caliper framework is extensible and aims to  facilitate the likely iterative build-out, refinement, and enhancement of its capabilities.

 

Table of Contents

Executive Summary

1       Introduction

   1.1        Scope

   1.2        High Level Illustration

   1.3        Nomenclature

   1.4        References

2       Metric Profiles

   2.1        Base Metric Profile

   2.2        Session Metric Profile

      Entities

     Actions

     Session Event

   2.3        Reading Metric Profile

      Entities

      Actions

      Reading Event

   2.4        Annotation Metric Profile

      Actions

      Annotation Event

   2.5        Assignable Metric Profile

     Actions

     Assignable Event

   2.6        Assessment Metric Profile

      Entities

      Actions

      Assessment Event

   2.7        Assessment Item Metric Profile

      Entities

      Actions

      Assessment Item Event

   2.8        Outcome Metric Profile

      Entities

      Actions

      Outcome Event

   2.9        Media Metric Profile

      Entities

      Actions

      Media Event

   2.10     Illustration of Metric Profile Interactions

   2.11     Engagement Scenario

3       Sensor Implementation Details

   3.1        Java Sensor

   3.2        Javascript Sensor

   3.3        PHP Sensor

   3.4        Python Sensor

   3.5        Ruby Sensor

   3.6        .NET Sensor

4       Caliper Transport Support

   4.1        Transport-Only Scope for Sensor API and CaliperEvent

   4.2        Caliper-Transport Event Sample Fixture

   4.3        Caliper Transport and Conformance/Compliance Validation

5       Use Cases for Caliper Analytics

   5.1        Caliper Enabled Engagement Sequence

   5.2        Example Reading Sequence

   5.3        Example Assessment Sequence

   5.4        Example Media Sequence

6       Conformance Testing Details

   6.1        Caliper Conformance Testing Framework Diagram

   6.2        Caliper Conformance Testing Details

7       Appendix

   7.1        Caliper Common Fixtures JSON

About This Document

List of Contributors

Revision History

 

1  Introduction

The purpose of the 1EdTech Caliper project is to define a standard for enabling the collection of rich contextual data about learning interactions  and a Sensor API™ for capturing and reporting this data. This work will enable learning environments to capture data from learning interactions and share it with other learning environments and consumers of learning analytics.  This work will build on existing 1EdTech standards and culminate in a new 1EdTech standard. 

The 1EdTech Caliper Framework is designed to meet the following requirements:

  • Establish a means for consistently capturing and presenting measures of learning activity, which will enable more efficient development of learning analytics features in learning environments
  • Define a common language for labeling learning data, which will set the stage for an ecosystem of higher-order applications of learning analytics
  • Provide a standard way of measuring learning activities and effectiveness, which will enable designers and providers of curriculum to measure, compare and improve quality
  • Leverage data science methods, standards and technologies
  • Build upon existing 1EdTech standards

 

1.1  Scope

The project deliverable includes creating multiple artifacts to support the implementation and use of Caliper Analytics. These deliverables include:

  • Creation of  1EdTech Metric Profiles to establish an  extensible, common format for grouping learning activity data gathered across multiple learning environments.
  • Creation of the 1EdTech Sensor API to define basic learning events and to standardize and simplify the gathering of learning metrics across learning environments. 
  • Creation of a reference 1EdTech Caliper Event Store primarily as a sample event persistence endpoint integration and to demonstrate the end-to-end event transmission and endpoint receipt / storage.  Caliper does not formally include an open, standards based event store/LRS in its initial scope.
  • Development  and continuing support of 1EdTech Caliper Conformance Tools.
  • Provide reference Caliper sensor implementations instrumented with Caliper sensors.
  • Leverage and extend the 1EdTech Learning Tools Interoperability ® (LTI®) standard by enhancing and integrating granular, standardized learning measurement with tools interoperability and the underlying learning information models, inclusive of course, learner, outcomes and other critical associated context.

1.2  High Level Illustration

 

1.3  Nomenclature

Throughout this document, we use the following terminology when describing Caliper Analytics.

  • Caliper:  The name of the 1EdTech Caliper Analytics Learning Measurement Framework which provides a standard for representing, capturing, and marshalling metrics generated by learning activity and targeted for consumption by any conforming analytics store or service.
  • Learning Activity: Any activity considered a component of a learning sequence presented in a digital learning environment (e.g. LTI tool) typically denoted as a lesson.  Activities can be assignable and/or gradable as required.
  • Caliper Sensor API:  Defines and specifies the interaction between Learning Application and Caliper.
  • Caliper Sensor:  Code/library that assists in the measurement of Caliper Events using the Caliper Sensor API.  Deployed into the Learning/Digital Application.  Sensors are implemented in Java, Javascript, PHP, Python, Ruby and .NET.
  • Metric Profile:  The Information Model for Caliper – organized by Learning Activity Type.
  • EventStore: System that stores event data emitted by Caliper Sensors.

 

1.4  References

1EdTech LTI®

     1EdTech Learning Tools Interoperability® http://www.imsglobal.org/lti/

1EdTech LIS

     1EdTech Learning Information Services http://www.imsglobal.org/lis/

1EdTech QTI®

     1EdTech Question and Test Interoperability® http://www.imsglobal.org/apip/alliance.html

FOAF

     FOAF http://xmlns.com/foaf/spec/

SCHEMA.ORG

     Schema.org http://schema.org/docs/schemas.html

OPEN ANNOTATION

 http://www.openannotation.org/

EPUB3

     http://www.idpf.org/epub/vocab/structure/

JSON-LD

     http://www.w3.org/TR/json-ld/
     http://json-ld.org/

Activity  Streams

     http://activitystrea.ms/JSON-LD

       xAPI

     http://www.adlnet.gov/tla/experience-api/

 

2  Metric Profiles

The events and entities collected into metric profiles serve as the information model for Caliper.  Metric Profiles are organized by learning activity type, for e.g., Assessment, Reading, Media, etc.. 

All of the events and entities defined in Caliper Metric Profiles are specified as Javascript Object Notation (JSON) and conform to the JSON for Linked Data (JSON-LD) format.  As a result,  events will specify both @context and @type element while entities will include an @id element in addition to specifying a @context and a @type.  Note that these properties are not included in the tables below.

The sections below outline each Metric Profile specified for Caliper v1.  As can be seen, for each Metric Profile, we define three types of information:

  • Entities - These are the Entities that participate in Learning Interactions (e.g. Person, Assessment, Video, etc.)
  • Actions - These are the actions performed as part of a Learning Interaction
  • Event - captures the Entities and Actions performed as a “learning event”

2.1  Base Metric Profile

The base activity context is universally applicable to all activity types and contains common Entities and attributes used in the other profiles.

 

Entities

Property

Data Type

Description

Source

Entity (baseline for all other Entities)

name

String

Name of Entity

schema.org/Thing

description

String

Description of Entity

schema.org/Thing

extensions

mapOf:extensions (String, String)

Map of Extensions

 

dateModified

long

time object was modified

schema.org/CreativeWork

dateCreated

long

time object was created

schema.org/CreativeWork

 

 

Property

Data Type

Description

Source

DigitalResource

objectType

listOf: Assignable, Gradeable, Sharable …

Type of Object

Note: This property is not currently used and may be removed in a future version

 

alignedLearningObjective

listOf: associated learning objectives

Associated Learning Objectives (Los) as context

 

keywords

listOf: keywords

Associated Topics as context.  List of Strings

schema.org/CreativeWork

 

isPartOf

DigitalResource

Parent reference

Parent/Referrer Learning Activity containing or linking to this Learning Activity

schema.org/CreativeWork

datePublished

long

time object was published

schema.org/CreativeWork

version

String

version or Entity

schema.org/CreativeWork

 

 

Event (baseline for all other Events)

 

Event Attribute

Required

Details (Type)

Comments

actor

Yes

User or System entity (FOAF:Agent)

 

action

Yes

Defined per Metric Profile

Action performed by Agent

object

Yes

Entity (Variable based on profile))

 

generated

No

Entity (Variable based on Profile)

 

target

No

Entity (Variable based on Profile)

Optional, but recommended

federatedSession

No

session id (String)

Optional, but recommended

edApp

No

edApp Entity

Optional, but recommended

group

No

Organization Entity

Optional, but recommended

membership

No

Membership Entitty

 

eventTIme

Yes

Event time (ISO-8601)

 

 

 

2.2Session Metric Profile

The Session Metric Profile is the vocabulary for tracking user session related activity/events.

Entities

Property

Data Type

Description

Source

Session

actor

FOAF:Agent (user)

Required.  Agent for whom the Session exists 

http://xmlns.com/foaf/spec/

startedAtTime

ISO-8601 timestamp

Date that the session was started by Agent

 

endedAtTime

ISO-8601 timestamp

Date that the session was ended

 

duration

xsd:duration

Date that the assignable object is Active

An xsd:duration (http://books.xmlschemata.org/relaxng/ch19-77073.html)

 

Actions

Action

URI

Description

Logged In

http://purl.imsglobal.org/vocab/caliper/v1/action#LoggedIn

Action indicating log in to start a session

Logged Out

http://purl.imsglobal.org/vocab/caliper/v1/action#LoggedOut

Actor initiates end of session

Timed Out

http://purl.imsglobal.org/vocab/caliper/v1/action#TimedOut

Session time out.

 

Session Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

edApp (in most cases)

 

generated

Yes

Session entity (from table above)

 

target

No

Frame/webpage, etc.

Optional, but recommended

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

 

eventTime

Yes

Event time

 

 

2.3  Reading Metric Profile

Vocabulary for activity/events related to the reading of textual content.  The reading object is the main collection.  Frame is the location within the content.  Using the EPUB3 structural semantics, Volume is a component of a collection. Views are generated as part of or the entire object of the interaction with a Reading.

Entities

Property

Data Type

Description

Source

ePubVolume

ePubPart

ePartSubChapter

frame

index

integer

The numeric index of the location relative to sibling locations in the content

 

 

Actions

Action

URI

Description

Searched

http://purl.imsglobal.org/vocab/caliper/v1/action#Searched

Actor searches for a Digital Resource

Viewed

http://purl.imsglobal.org/vocab/caliper/v1/action#Viewed

Actor views a Digital Resource

Navigated To

http://purl.imsglobal.org/vocab/caliper/v1/action#NavigatedTo

Actor navigates to a Digital Resource

 

Reading Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Digital Resource in most cases

 

generated

Yes

Search Result (for searched Action)

 

target

Yes

Digital Resource f

Optional, but recommended

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTime

Yes

Event started at time

 

 

2.4  Annotation Metric Profile

The vocabulary for activity/events related to the annotation of a target resource.  The annotation metric profile is a complimentary profile and can be combined with any learning activity such as a Reading Activity, an Assessment Activity, etc.. 

Entities

Property

Data Type

Description

Source

Annotation

annotated

DigitalResource

Required.  The object being annotated

 

BookmarkAnnotation

bookmarkNotes

String

The notes that the user entered with the bookmark

 

HighlightAnnotation

textPositionSelector

Object

Selector which describes a range of text based on its start and end positions

 

selectionText

String

The text that was selected

 

SharedAnnotation

withAgents

list of LISGroup or LISPersons

The group or people that the annotation was shared with

 

TagAnnotation

tags

List of Strings

List of Strings that the Actor tagged the content with

 

 

Actions

Action

URI

Description

Attached

http://purl.imsglobal.org/vocab/caliper/v1/action#Attached

Attach/detach a doc, media element, etc.  with a target resource

Bookmarked

http://purl.imsglobal.org/vocab/caliper/v1/action#Bookmarked

Record a position/place/location in a target resource

Classified

http://purl.imsglobal.org/vocab/caliper/v1/action#NavigatedTo

Actor navigates to a piece of content

Commented

http://purl.imsglobal.org/vocab/caliper/v1/action#Commented

A comment action associated/disassociated with a target resource

Described

http://purl.imsglobal.org/vocab/caliper/v1/action#Described

A description or abstract is added or removed from an annotated resource

Disliked

http://purl.imsglobal.org/vocab/caliper/v1/action#Disliked

Negative preference regarding the target resource

Highlighted

http://purl.imsglobal.org/vocab/caliper/v1/action#Highlighted

Mark a selected segment of the target resource for particular emphasis

Identified

http://purl.imsglobal.org/vocab/caliper/v1/action#Identified

Assign an identity to a target resource

Liked

http://purl.imsglobal.org/vocab/caliper/v1/action#Liked

Indicate a positive preference regarding the target resource

LInked

http://purl.imsglobal.org/vocab/caliper/v1/action#Linked

Link a resource related to the target resource

Questioned

http://purl.imsglobal.org/vocab/caliper/v1/action#Questioned

Represents asking a question about the target resource(s). For example to ask for assistance with a particular section of text, or question its veracity

Ranked

http://purl.imsglobal.org/vocab/caliper/v1/action#Ranked

Assignment of value or quality to the target resource(s). For example annotating an Annotation to moderate it up in a trust network or threaded discussion

Recommended

http://purl.imsglobal.org/vocab/caliper/v1/action#Recommended

Recommend a modification to the target resource

Shared

http://purl.imsglobal.org/vocab/caliper/v1/action#Shared

The annotated resource is shared to a target destination network and recipient

Subscribed

http://purl.imsglobal.org/vocab/caliper/v1/action#Subscribed

The annotated resource is subscribed to (or unsubscribed from) a subscription service by a subscriber

Tagged

http://purl.imsglobal.org/vocab/caliper/v1/action#Tagged

A tagging action is taken on a target reading element context

 

Annotation Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Digital Resource

 

generated

Yes

Annotation

 

target

N/A

 

 

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTime

Yes

Event time

 

 

 

2.5  Assignable Metric Profile

The assignable metric profile provides coverage for all activity types that are assignable to the learner for completion according to specific terms.  Assignable is an interface that is implemented by the AssignableDigitalResource.  Any Entity can implement AssignableDigitalResource and become “assignable”.

Entities

Property

Data Type

Description

Source

Assignable

dateCreated

ISO-8601 timestamp

Date that the assignable object was created

schema.org/CreativeWork

dateModified

ISO-8601 timestamp

Date that the assignable object was modified

schema.org/CreativeWork

datePublished

ISO-8601 timestamp

Date that the assignable object was published

schema.org/CreativeWork

dateToStartOn

ISO-8601 timestamp

Date that the assignable object is allowed to be started by Agent

 

dateToActivate

ISO-8601 timestamp

Date that the assignable object is Active

 

dateToShow

ISO-8601 timestamp

Date that the assignable object shows to students

 

dateToSubmit

ISO-8601 timestamp

Date that the assignable object should be submitted

 

maxAttempts

int

Maximum number of attempts

 

maxSubmits

int

Maximum number of submits

 

maxScore

double

Maximum score allowed

 

Attempt

actor

FOAF:Agent (user, system)

Required.  Agent that performed action.

http://xmlns.com/foaf/spec/

assignable

Assignable Object

The object that the agent is  attempting

 

count

int

Number of attempts

 

startedAt

ISO-8601 timestamp

Required time in milliseconds

http://www.w3.org/TR/prov-o/

 

endedAt

ISO-8601 timestamp

Time in milliseconds

http://www.w3.org/TR/prov-o/

 

duration

xsd:duration

 

http://books.xmlschemata.org/relaxng/ch19-77073.html

 

Actions

Action

URI

Description

Abandoned

http://purl.imsglobal.org/vocab/caliper/v1/action#Abandoned

Actor abandoned the assignment

Activated

http://purl.imsglobal.org/vocab/caliper/v1/action#Activated

Assignable Digital Resource is available to Actors who are assigned it

Competed

http://purl.imsglobal.org/vocab/caliper/v1/action#Completed

Actor completed the Assignable Digital Resource

Deactivated

http://purl.imsglobal.org/vocab/caliper/v1/action#Deactivated

Assignable Digital Resource is no longer available to Actors who are assigned it

Hid

http://purl.imsglobal.org/vocab/caliper/v1/action#Hid

Assignable Digital Resource is hid from the view of the Actors

Reviewed

http://purl.imsglobal.org/vocab/caliper/v1/action#Reviewed

 Assignable Digital Resource is reviewed

Showed

http://purl.imsglobal.org/vocab/caliper/v1/action#Showed

The Assignable Digital Resource was shown to an Actor

Started

http://purl.imsglobal.org/vocab/caliper/v1/action#Started

The Actor assigned started the assignment

 

Assignable Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Assignable Digital Resource,N/A

 

generated

Yes

Attempt,N/A

 

target

N/A

 

 

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTIme

Yes

Event time

 

 

2.6  Assessment Metric Profile

Provides coverage for Assessment, Quiz/Homework and similar assignable / gradable activity

Entities

Property

Data Type

Description

Source

Assessment

assessmentItems

List of Assessment Items

Required. 

 

 

Actions

Action

URI

Description

Started

http://purl.imsglobal.org/vocab/caliper/v1/action#Started

The Actor started the Assessment

Paused

http://purl.imsglobal.org/vocab/caliper/v1/action#Paused

The Actor paused the Assessment

Restarted

http://purl.imsglobal.org/vocab/caliper/v1/action#Restarted

The Actor restarted the Assessment

Submitted

http://purl.imsglobal.org/vocab/caliper/v1/action#Deactivated

The Actor submits the Assessment

 

Assessment Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Assignable Digital Resource,Attempt

 

generated

No

Attempt, N/A

Required in some cases

target

No

N/A, Assignable

Required in some cases

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTime

Yes

Event time

 

 

2.7  Assessment Item Metric Profile

Covers the Actor’s interaction with assessment items.  

Entities

Property

Data Type

Description

Source

Assessment

isTimeDependant

boolean

If the time to answer the question is important and should be recorded

 

Response

assignable

Object

DigitalResource associated with the Response

 

actor

Object

Agent associated with the response

 

attempt

Object

Attempt Object associated with the Response

 

startedAtTime

ISO-8601 timestamp

Required time in milliseconds

http://www.w3.org/TR/prov-o/

 

endedAtTime

ISO-8601 timestamp

 Time in milliseconds

http://www.w3.org/TR/prov-o/

 

duration

xsd:duration

 

http://books.xmlschemata.org/relaxng/ch19-77073.html

FilInTheBlankResponse

values

ListOf

List of values representing the response that was entered.

ListOf

MultipleChoiceResponse

Value

String

value choosen

 

MultipleResponseResponse

Values

ListOf

values choosen

 

SelectTextResponse

Values

ListOf

List of values representing the response

 

TrueFalseResponse

value

String

value choosen

 

 

Actions

Action

URI

Description

Started

http://purl.imsglobal.org/vocab/caliper/v1/action#Started

The Actor started the Assessment Item

Viewed

http://purl.imsglobal.org/vocab/caliper/v1/action#Viewed

The Actor viewed the Assessment Item

Skipped

http://purl.imsglobal.org/vocab/caliper/v1/action#Skipped

The Actor skipped the Assessment Item

Reviewed

http://purl.imsglobal.org/vocab/caliper/v1/action#Reviewed

The Actor reviewed the Assessment Item

Completed

http://purl.imsglobal.org/vocab/caliper/v1/action#Completed

The Actor completed the Assessment Item

 

Assessment Item Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Assignable Digital Resource, Attemp

 

generated

No

Assignable Digital Resource

Required in some cases

target

No

N/A, Assignable

Required in some cases

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

 

 

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTime

Yes

Event time

 

 

2.8  Outcome Metric Profile

Applies to all activity types that are assignable and also yield some measurable result and/or outcome

Entities

Property

Data Type

Description

Source

Result

assignable

Object

Assignable object associated with the Outcome

 

actor

Object

Agent associated with the response

 

normalScore

double

 

 

penaltyScore

double

 

 

totalScore

double

 

 

extraCreditScore

double

 

 

totalScore

double

 

 

curvedTotalScore

double

 

 

curveFactor

String

 

 

comment

Object

Assignable object associated with the Outcome

 

scoredBy

Agent that performed action.

http://xmlns.com/foaf/spec/

 

Required.  Agent that performed action.

 

Actions

Action

URI

Description

Graded

http://purl.imsglobal.org/vocab/caliper/v1/action#Started

Actor or Software Application Grades the Assignment

 

Outcome Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Attempt

 

generated

Yes

Result

 

target

Yes

Assignable

 

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

 

eventTime

Yes

Event time

 

 

2.9  Media Metric Profile

This metric profile captures the interaction with audio/video systems.

Entities

Property

Data Type

Description

Source

MediaObject

Media Location

currentTime

String

The time value (from beginning of media) that indicates the current location

Media fragment URI

AudioObject

volumeMin

String

Minimum volume

 

volumeMax

String

Maximum volume

 

volumeLevel

String

Level of volume

 

muted

boolean

Whether the Muted the audio

 

VideoObject

ImageObject

 

Actions

Action

URI

Description

Enabled Closed Captioning

http://purl.imsglobal.org/vocab/caliper/v1/action#EnabledClosedCaptioning

The Actor enabled Closed Captioning

Disabled Closed Captioning

http://purl.imsglobal.org/vocab/caliper/v1/action#DisabledClosedCaptioning

The Actor disabled Closed Captioning

Changed Volume

http://purl.imsglobal.org/vocab/caliper/v1/action#ChangedVolume

The Actor changed the volume

Muted

http://purl.imsglobal.org/vocab/caliper/v1/action#Muted

The Actor muted the volume

Unmuted

http://purl.imsglobal.org/vocab/caliper/v1/action#Unmuted

The Actor unmuted the volume

Changed Speed

http://purl.imsglobal.org/vocab/caliper/v1/action#ChangedSpeed

 The Actor changed speed

Changed Volume

http://purl.imsglobal.org/vocab/caliper/v1/action#ChangedVolume

 The Actor changed the volume

Ended

http://purl.imsglobal.org/vocab/caliper/v1/action#Ended

The Actor ended the media interaction

Jumped To

http://purl.imsglobal.org/vocab/caliper/v1/action#JumpedTo

The Actor jumped to a new media location

Forwarded To

http://purl.imsglobal.org/vocab/caliper/v1/action#ForwardedTo

The Actor forwarded to a new media location

Paused

http://purl.imsglobal.org/vocab/caliper/v1/action#Paused

The Actor paused the media

Resumed

http://purl.imsglobal.org/vocab/caliper/v1/action#Resumed

The Actor resumed the media

Rewound

http://purl.imsglobal.org/vocab/caliper/v1/action#Rewound

The Actor rewound the media

Started

http://purl.imsglobal.org/vocab/caliper/v1/action#Started

The Actor started the media

Changed Resolution

http://purl.imsglobal.org/vocab/caliper/v1/action#ChangedResolution

The Actor changed the resolution on the display

Changed Size

http://purl.imsglobal.org/vocab/caliper/v1/action#ChangedSize

The Actor changed the viewer size

Exited Fullscreen

http://purl.imsglobal.org/vocab/caliper/v1/action#ExitedFullscreen

 The Actor exited full screen view

Entered Fullscreen

http://purl.imsglobal.org/vocab/caliper/v1/action#EnteredFullscreen

 The Actor entered full screen view

Opened Popout

http://purl.imsglobal.org/vocab/caliper/v1/action#OpenedPopout

The Actor opened popout

Closed Popout

http://purl.imsglobal.org/vocab/caliper/v1/action#ClosedPopout

The Actor closed popout

Disable Closed Captioning

http://purl.imsglobal.org/vocab/caliper/v1/action#DisabledClosedCaptioning

The Actor disabled closed captioning

Enabled Closed Captioning

http://purl.imsglobal.org/vocab/caliper/v1/action#EnabledClosedCaptioning

The Actor enabled closed captioning

 

Media Event

Event Attribute

Required

Details

Comments

actor

Yes

Person, EdApp

 

action

Yes

From table above

 

object

Yes

Media Object

 

generated

N/A

 

 

target

N/A

 

 

federatedSession

No

session id

Optional, but recommended

edApp

No

edApp

Optional, but recommended

group

No

Organization

Optional, but recommended

membership

No

Membership

Optional, but recommended

eventTime

Yes

Event time

 

 

2.10  Illustration of Metric Profile Interactions

 

2.11  Engagement Scenario

As an example of just one common learning activity measurement,  this scenario contains a list of Events and corresponding actions from the current set of Caliper Metric Profiles that indicate a baseline level of student engagement with Learning Activities.  The determination of the what metrics constitute “engagement” may certainly differ across learning contexts so the metrics leveraged can be adjusted accordingly to align with any given engagement definition.

  • In order to quantify engagement, three elements can be measured:  (1) the overall duration of a “session” comprising multiple events;  (2) the duration of engagement with each individual activity; and, (3) the total number of (each) type of event in a session.
  • Each event has an “opening” action that indicates engagement.  E.g. “started”.  Other actions in the sequence (e.g. paused, resumed, completed) can also be included, including a “closing” action to indicate the end of the sequence.  However, to simply get an indicator of (one or many) learner engagement with a specific Learning Activity, the opening action could be sufficient.
  • All attributes/objects (e.g. actor, action, object, startedAtTime, endedAtTime, duration, etc) of Events are implicitly part of the Engagement Scenario.

 

Event

Action

Metric Profile

Comments

SessionEvent

loggedIn

Session

 

Navigation Event

navigatedTo

Mulitiple

Common across all Learning Activity types

ViewedEvent

viewed

Reading

 

AnnotationEvent

All

Annotation

 

MediaEvent

started, [paused,resumed,etc]

Media

 

AssignableEvent

started [completed,submitted]

Assignable

 

AssessementEvent

started, [submitted]

Assessment

 

 

3  Sensor Implementation Details

To insure that all the of the Sensor’s are producing the same JSON-LD and for ease in implementing the Conformance Testing Framework, the Caliper Common Fixtures project was created.  Version 1.0 aligned Caliper Common Fixtures are included in the Appendix of this document.

3.1  Java Sensor

The Reference Implementation of the Java Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

3.2  Javascript Sensor

The Reference Implementation of the Javascript Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

3.3  PHP Sensor

The Reference Implementation of the PHP Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

3.4  Python Sensor

The Reference Implementation of the Python Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

3.5  Ruby Sensor

The Reference Implementation of the Ruby Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

3.6  .NET Sensor

The Reference Implementation of the .NET  Sensor is located in the chart below:

Directions to install and build the project are located in README of the project.

 

3.1

Caliper Java Code

3.2

Caliper Javascript Code

3.3

Caliper PHP Code

3.4

Caliper Python Code

3.5

Caliper Ruby Code

3.6

Caliper .NET Code

 

4  Caliper Transport Support

In order to support some additional requirements related to the work being done with Caliper in the 1EdTech HED Analytics group (formerly RAM) as it relates to requirements to leverage a more minimal event with a free-form payload, the current released Caliper v1.0 sensorAPI and CaliperEvent implementation and conformance validation support will be revised to a “transport” level of use.   This essentially means that only the most minimal and generally applicable event structure, nomenclature and supporting properties will be leveraged and the more comprehensive CaliperEvent via metric profile modeled event definition and context will not be required. 

4.1  Transport-Only Scope for Sensor API and CaliperEvent

Events must include:

  1. The CaliperEvent structured “triple” of Actor, Action and Object (must adhere to Caliper naming conventions/vocabulary)
  • Actor: object with the @id attribute required.
  • Object: object with the @id attribute required.
  • Action:  a string with the @type attribute required (recommended format URI)
  • eventTime: the event ISO timestamp
  1. Optional
  • generated object (optional, If specified, must have an @id attribute. e.g. used for attempt, result, response etc)
  • target object (optional, If specified, must have an @id attribute)

4.2  Caliper-Transport Event Sample Fixture

{
  "sensor": "https://example.edu/sensor/001",
  "sendTime": "2015-09-15T11:05:01.000Z",
  "data": [
    {
      "@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
      "@type": "http://purl.imsglobal.org/caliper/v1/Event",            
      "actor": {
        "@id": "https://example.edu/user/554433",
        "@type": "http://purl.imsglobal.org/caliper/v1/lis/Person" },
      "action": "http://purl.imsglobal.org/vocab/caliper/v1/action#Viewed",
      "eventTime": "2015-09-15T10:15:00.000Z",
      "object": {
        "@id": "https://example.com/viewer/book/34843#epubcfi(/4/3)",
        "@type": "http://www.idpf.org/epub/vocab/structure/#volume" }
    }
  ]
}

4.3  Caliper Transport and Conformance/Compliance Validation

With this more recent transport-only support, the following levels of conformance test compliance and/or certification will be enabled and supported:

  1. Caliper-Transport Compliance: compliance-only validation of the SensorAPI and minimal CaliperEvent structure defined by the transport-only requirements….therefore, no 1EdTech certification will be issued for this level of compliance with Caliper.
  2. Caliper-Event Conformance: compliance and certification for implementations of the SensorAPI and one or more declared supported metric profiles. (Note: this is the primary and default Caliper implementation leveraging the SensorAPI and the CaliperEvent modeled event data and contextual payloads)
  3. Caliper-LTI Conformance: compliance and certification for implementations of LTI Provider and/or Consumer which include support for the defined Caliper Service Profile service extension. (Note: planned for v1.1 release to with completion of the related v1.1 ref implementation)

     

5  Use Cases for Caliper Analytics

This section describes common scenarios for the Caliper Specification. 

5.1  Caliper Enabled Engagement Sequence

This profile contains a list of Events and corresponding actions from the current set of Caliper Metric Profiles that indicate student engagement with Learning Activities.

Use Case Title:

Example Engagement Sequence

Brief description:

A student’s engagement with a common learning sequence

Level:

Summary

Actors:

Student

LMS

Electronic Reading Educational App

Quizzing Application

Basic Flow of Events:

  1. Student logs in to interact with a course.  A SessionEvent is sent by the sensor with an action of logged in.
  2. The student navigates to page in the reading.  A NavigationEvent is sent by the sensor with an action of navigated to.
  3. The student adds a tag to the page.  A TagAnnotation is generated.  An AnnotationEvent is sent by the sensor with an action of tagged..
  4. The student starts a quiz and generates an Attempt.  An AssessmentEvent is sent by the sensor with an action of started..
  5. The student starts question 1.  Generates an Attempt.  An AssessmentItemEvent is sent by the sensor with an action of started.
  6. The student completes question 1 and generates a response.  An  AssessmentItemEvent is sent by the sensor with an action of completed.
  7. Repeats 5-6 for all questions
  8. The student submits the quiz. An AssessmentEvent is sent by the sensor with an action of submitted.
  9. System grades the quiz and generates a Result An OutcomeEvent is sent by the sensor with an action of graded
  10. The student logs out of the course. A SessionEvent is sent by the sensor with an action of logged out.

 

5.2  Example Reading Sequence

Use Case Title:

Example Reading Sequence

Brief description:

A student in a course interacts with two readings from two reading eduApps. In the process of interacting with the apps, she performs various reading and annotation activities. These are defined in the Caliper Reading and Annotation profiles respectively.

Level:

Summary

Actors:

Student

LMS

Electronic Reading Educational App 1

Electronic Reading Educational App 2

Basic Flow of Events:

  1. The student navigates to a specific volume in the reading Educational App 1 from their LMS.    A NavigationEvent is sent by the sensor with an action of navigated to
  2. The student navigates to page one in that volume.    A NavigationEvent is sent by the sensor with an action of navigated to
  3. The student views page one.  A viewedEvent is sent by the sensor with the action of viewed.
  4. The student navigates to page two.   A NavigationEvent is sent by the sensor with an action of navigated to.  The student views page two.  A viewedEvent is sent by the sensor with the action of viewed.
  5. The student highlighted fragment in Page 2 from index 455 to 489 in the reading Educational App 1.  A highlightedEvent is sent by the sensor with the action of highlighted. Generating a highlight annotation.
  6. The student navigates to page three. A navigatedEvent is sent by the sensor with the action of navigated to.
  7. The student views page three.  A viewedEvent is sent by the sensor with the action of viewed.
  8. The student bookmarked Page 3.  A bookmarkedEvent is sent by the sensor with the actio of bookmarked.  Generating a bookmark annotation
  9. The student navigated to Reading provided by Electronic Reading Educational App 2.  A navigatedEvent is sent by the sensor with the action of navigated to.
  10. The student navigated to page in the Reading.  A navigatedEvent is sent by the sensor with the action of navigated to.
  11. The student viewed a page in the Reading. A viewedEvent is sent by the sensor with the action of viewd.
  12. The student tagged the page. A taggedEvent is sent by the sensor with the action of tagged.  Generating a tag annotation.
  13. The student shared the page with other students.  A sharedEvent is sent by the sensor with the action of tagged.  Generating a shared Annotation.

 

5.3  Example Assessment Sequence

Use Case Title:

Example Assessment Sequence

Brief description:

Student in a course interacts with assignable entities within the course. The assignable here is an Assessment. In the process of interacting, she performs various assignable and assessment related interactions. These are defined in the Caliper Assignable, Assessment and Outcomes profiles respectively

Level:

Summary

Actors:

Student

LMS

Assessment EduApp

Grading Engine

 

Basic Flow of Events:

  1. The student navigates to an assessment that was assigned in the LMS using the LMS Assessment Tool.  Sensor generates an navigatedEvent with the action of navigated to..
  2. The student starts the assignment in the LMS Sensor sends AssignableEvent with the action of started and generates an attempt Object
  3. The student starts the assessment in the Assessment EduApp.  Sensor sends AssessmentEvent with the action of started.
  4. The student starts question 1 in the assessment in the Assessment EduApp.  Sensor sends AssessmentItemEvent with the assessmentItem action of started.
  5. The student completes question 1.  Sensor sends AssessmentItemEvent with the assessmentItem action of completed.
  6. Steps 4-5 repeat for each question in the assignment.
  7. Student submits the assessment in the Assessment EduApp.  Sensor send an AssessmentEvent with the action of completed.
  8. Assessment autograded by grading engine.  Sensor generates an OutcomeEvent with the action of graded.  Including the attempt – result pairing.

 

5.4  Example Media Sequence

Use Case Title:

Example Media Sequence

Brief description:

A student in a course interacts with multi-media entities within the course. The events and entities referenced here are defined in the Caliper Media Metric Profile

Level:

Summary

Actors:

Student

LMS

Media Player

Basic Flow of Events:

  1. The student navigated to a video in the LMS.  Sensor generates a navigatedEvent with action of navigated to.
  2. The student started playing Video in the media player.  Sensor generates a mediaEvent with the action of started with the target of the media location.
  3. The student pauses the video in the media player.  Sensor generates a mediaEvent with the action of paused.
  4. The student resumes the video in the media player.  Sensor generates a mediaEvent with the action of resumed.
  5. The student completes playing the video in the media player.  Sensor generates an  ended event. Including the media object and the media location.

 

6  Conformance Testing Details

6.1  Caliper Conformance Testing Framework Diagram

 

6.2  Caliper Conformance Testing Details

Caliper Conformance Tests is a set of tooling to help verify if events are aligned with the Caliper format and to show if the metric profiles are complete.

For more details and instructions for Conformance Testing, please refer to the 1EdTech Caliper Conformance and Certification Guide available on the  Caliper Developer Site

 

7  Appendix

7.1  Caliper Common Fixtures JSON

The Common Fixtures resource can be found and referenced as part of  the Caliper codebase (https://github.com/IMSGlobal/caliper-common-fixtures-public).

 

About This Document

 

Title:                                       1EdTech

Editor:                                    Mary Millar, Elsevier

Co-chairs:                              Chris Vento, Intellify Learning, Chris Millet, Penn State University

Version:                                  Final v1.0

Version Date:                          10 October 2015

Revision Information:            Version 1.0 Final

Document Location:               http://www.imsglobal.org/caliper/

 

List of Contributors

The following individuals contributed to the development of this document:

Viktor Haag

D2L

Mary Millar

Elsevier

Prashant Nayak

Intellify Learning

Chris Vento

Intellify Learning

Anthony Whyte

University of Michigan

 

Revision History

Version No.

Release Date

Comments

Final v1.0

15 October 2015

Final formal release of this specifcation.

 

 

 

 

 

1EdTech Consortium, Inc. (“1EdTech”) is publishing the information contained in this document (“Specification”) for purposes of scientific, experimental, and scholarly collaboration only.

1EdTech makes no warranty or representation regarding the accuracy or completeness of the Specification.

This material is provided on an “As Is” and “As Available” basis.

The Specification is at all times subject to change and revision without notice. 

It is your sole responsibility to evaluate the usefulness, accuracy, and completeness of the Specification as it relates to you.

1EdTech would appreciate receiving your comments and suggestions.

Please contact 1EdTech through our website at http://www.imsglobal.org.

Please refer to Document Name: 1EdTech Caliper Analytics Implementation Guide v1.0

Date: 15 October 2015