Open edX 101

A Source Code Review

Régis Behmo (@regisb)
Open edX Conference, June 14 2016 | Stanford, CA
fun-mooc.com

Orlando
fun-mooc.fr

https://www.fun-mooc.fr
350 courses, 700 000 users

Open edX from above

# LMS + CMS
/edx/app/edxapp/edx-platform

# Forum service (Ruby code)
/edx/app/forum/cs_comments_service

# Programs-based product lines such as edX's XSeries offering.
/edx/app/programs/programs

# Payment services (seldom installed)
/edx/app/ecommerce/ecommerce
/edx/app/ecomworker/ecomworker

# Theme customization (optional)
/edx/app/edxapp/themes
                    
# Python virtual environment for edx-platform dependencies
/edx/app/edxapp/venvs/edxapp
    ...

Open edX from above

edx-platform

How many lines of code in edx-platform? (1 Dj = 1 Django)

  • CPython 3.5.1 (Python, C, C++) 967 725 # 4.23 Dj
  • Moodle (php) 672 331 # 2.94 Dj
  • ElasticSearch (Java) 590 318 # 2.58 Dj
  • Wordpress (php) 291 709 # 1.28 Dj
  • Django 228 381 # 1 Dj     
  • Sentry 175 073 # 0.77 Dj
  • Mercurial 122 671 # 0.54 Dj
  • Celery 44 022 # 0.19 Dj
  • Flask 9 072 # 0.04 Dj

Open edX from above

edx-platform

How many lines of code in edx-platform? (1 Dj = 1 Django)

  • CPython 3.5.1 (Python, C, C++) 967 725 # 4.23 Dj
  • Moodle (php) 672 331 # 2.94 Dj
  • ElasticSearch (Java) 590 318 # 2.58 Dj
  • edx-platform 427 321 # 1.87 Dj
  • Wordpress (php) 291 709 # 1.28 Dj
  • Django 228 381 # 1 Dj     
  • Sentry 175 073 # 0.77 Dj
  • Mercurial 122 671 # 0.54 Dj
  • Celery 44 022 # 0.19 Dj
  • Flask 9 072 # 0.04 Dj

Inside edx-platform

cloc --exclude-dir=<vendor-folders> edx-platform
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                        1791          64982          98146         244140 # 57.1%
Javascript                     718          13745          10806          79920 # 18.7%
SASS                           300          10054           3769          43727 # 10.2%
HTML                           493           4716          30435          27928 # 6.5%
CoffeeScript                   118           2660           1459          14558 # 3.4%
CSS                             12            510            461           6675 # 1.5%
SQL                              2              8              9           4137
XML                            286            172             33           3451
YAML                            40            270            367           1827
Bourne Shell                    13            219            222            700
make                             2             31              6            143
ActionScript                     1             21             23             74
XSD                              1              8              0             41
-------------------------------------------------------------------------------
SUM:                          3777          97396         145736         427321
-------------------------------------------------------------------------------

Inside edx-platform

# LMS + CMS
/edx/app/edxapp/edx-platform       # 427321

# Forum service (Ruby code)
/edx/app/forum/cs_comments_service # 5399

# Programs-based product lines such as edX's XSeries offering.
/edx/app/programs/programs         # 4906

# Payment services (seldom installed)
/edx/app/ecommerce/ecommerce       # 25670
/edx/app/ecomworker/ecomworker     # 643

# Theme customization (optional)
/edx/app/edxapp/themes
                    
# Python virtual environment
/edx/app/edxapp/venvs/edxapp

Inside edx-platform

dependencies

/edx/app/edxapp/venvs/edxapp
    ora2                        # 31245
    edx-search                  # 3321
    opaque-keys                 # 3089
    recommender-xblock          # 3001
    xblock-poll                 # 2194
    edx-submissions             # 2193
    edx-milestones              # 1953
    event-tracking              # 1777
    edx-sga                     # 1567
    edx-reverification-block    # 1418
    xblock-google-drive         # 1261
    edx-user-state-client       # 1083
    ccx-keys                    # 748
    rate-xblock                 # 598
    acid-xblock                 # 750
    edx-jsme                    # 607
    done-xblock                 # 534

Open edX from the inside

Viewing a course

Open edX from the inside

Viewing a course

$ curl -L https://raw.github.com/edx/configuration/.../Vagrantfile > Vagrantfile
$ OPENEDX_RELEASE="named-release/dogwood.3" vagrant up && vagrant ssh
$ sudo su edxapp
$ paver devstack lms
...
Starting development server at http://0.0.0.0:8000/

Open edX from the inside

Viewing a course

courseware url 
lms/urls.py:
url(
    r'^courses/{}/courseware/(?P<chapter>[^/]*)/(?P<section>[^/]*)/$'.format(
        settings.COURSE_ID_PATTERN,
    ),
    'courseware.views.index',
    name='courseware_section',
)

Open edX from the inside

Viewing a course

lms/djangoapps/courseware/views.py:
def index(request, course_id, chapter, section):
    ...
    course = get_course_with_access(..., course_key, ...)
    section_module = get_module_for_descriptor(
        user,
        request,
        section_descriptor,
        field_data_cache,
        course_key,
        position,
        course=course
    )
pprint(course.__class__.__mro__) # class and all base classes of 'course'
(<class 'xblock.internal.CourseDescriptorWithMixins'>,
 <class 'xmodule.course_module.CourseDescriptor'>,
 <class 'xmodule.course_module.CourseFields'>,
 <class 'xmodule.seq_module.SequenceDescriptor'>,
 <class 'xmodule.seq_module.SequenceFields'>,
 <class 'xmodule.seq_module.ProctoringFields'>,
 <class 'xmodule.mako_module.MakoModuleDescriptor'>,
 <class 'xmodule.mako_module.MakoTemplateBlockBase'>,
 <class 'xmodule.xml_module.XmlDescriptor'>,
 <class 'xmodule.xml_module.XmlParserMixin'>,
 <class 'xmodule.x_module.XModuleDescriptor'>,
 <class 'xmodule.x_module.HTMLSnippet'>,
 <class 'xmodule.x_module.ResourceTemplates'>,
 <class 'lms.djangoapps.lms_xblock.mixin.LmsBlockMixin'>,
 <class 'xmodule.modulestore.inheritance.InheritanceMixin'>,
 <class 'xmodule.x_module.XModuleMixin'>,
 <class 'xmodule.x_module.XModuleFields'>,
 <class 'xblock.core.XBlock'>,
 <class 'xblock.mixins.XmlSerializationMixin'>,
 <class 'xblock.mixins.HierarchyMixin'>,
 <class 'xmodule.mixin.LicenseMixin'>,
 <class 'xmodule.modulestore.edit_info.EditInfoMixin'>,
 <class 'xblock.XBlockMixin'>,
 <class 'xblock.core.XBlockMixin'>,
 <class 'xblock.mixins.ScopedStorageMixin'>,
 <class 'xblock.mixins.RuntimeServicesMixin'>,
 <class 'xblock.mixins.HandlersMixin'>,
 <class 'xblock.mixins.IndexInfoMixin'>,
 <class 'xblock.mixins.ViewsMixin'>,
 <class 'xblock.core.SharedBlockBase'>,
 <class 'xblock.plugin.Plugin'>,
 <type 'object'>)

Open edX from the inside

Viewing a course

http://.../courses/...edX+DemoX+Demo_Course/courseware/interactive_demonstrations/19a30.../
courseware.views.index
course = get_course_with_access(...)
xblock.internal.CourseDescriptorWithMixins
?

Btw, what is a mixin?

Wikipedia: "In object-oriented programming languages, a mixin is a class that contains methods for use by other classes"
class Shape(object):
    def __init__(self):
        self.edges = []

    def perimeter(self):
        return sum([e.size for e in self.edges])

class Square(Shape):
    def __init__(self):
        self.edges = make_square()

class Triangle(Shape):
    def __init__(self):
        self.edges = make_triangle()

class ColorMixin(object):
    def colorize(self, color):
        for edge in self.edges:
            edge.color = color

class ColoredSquare(Square, ColorMixin):
    pass
pprint(course.__class__.__mro__) # class and all base classes of 'course'
(<class 'xblock.internal.CourseDescriptorWithMixins'>,
 <class 'xmodule.course_module.CourseDescriptor'>,
 <class 'xmodule.course_module.CourseFields'>,
 <class 'xmodule.seq_module.SequenceDescriptor'>,
 <class 'xmodule.seq_module.SequenceFields'>,
 <class 'xmodule.seq_module.ProctoringFields'>,
 <class 'xmodule.mako_module.MakoModuleDescriptor'>,
 <class 'xmodule.mako_module.MakoTemplateBlockBase'>,
 <class 'xmodule.xml_module.XmlDescriptor'>,
 <class 'xmodule.xml_module.XmlParserMixin'>,
 <class 'xmodule.x_module.XModuleDescriptor'>,
 <class 'xmodule.x_module.HTMLSnippet'>,
 <class 'xmodule.x_module.ResourceTemplates'>,
 <class 'lms.djangoapps.lms_xblock.mixin.LmsBlockMixin'>,
 <class 'xmodule.modulestore.inheritance.InheritanceMixin'>,
 <class 'xmodule.x_module.XModuleMixin'>,
 <class 'xmodule.x_module.XModuleFields'>,
 <class 'xblock.core.XBlock'>,
 <class 'xblock.mixins.XmlSerializationMixin'>,
 <class 'xblock.mixins.HierarchyMixin'>,
 <class 'xmodule.mixin.LicenseMixin'>,
 <class 'xmodule.modulestore.edit_info.EditInfoMixin'>,
 <class 'xblock.XBlockMixin'>,
 <class 'xblock.core.XBlockMixin'>,
 <class 'xblock.mixins.ScopedStorageMixin'>,
 <class 'xblock.mixins.RuntimeServicesMixin'>,
 <class 'xblock.mixins.HandlersMixin'>,
 <class 'xblock.mixins.IndexInfoMixin'>,
 <class 'xblock.mixins.ViewsMixin'>,
 <class 'xblock.core.SharedBlockBase'>,
 <class 'xblock.plugin.Plugin'>,
 <type 'object'>)

Open edX from the inside

Viewing a course

xblock/runtime.py:
class Mixologist(object):
    def __init__(self, mixins):
        self._mixins = tuple(mixins)

    def mix(self, cls):
        ...
        return type(
            base_class.__name__ + 'WithMixins',   # class name
            (base_class, ) + self._mixins,        # class bases
            {'unmixed_class': base_class}         # class attributes
        )

Open edX from the inside

Viewing a course

xblock/runtime.py:
class Mixologist(object):
    def __init__(self, mixins):
        self._mixins = tuple(mixins)

    def mix(self, cls):
        ...
        return type(
            base_class.__name__ + 'WithMixins',   # class name
            (base_class, ) + self._mixins,        # class bases
            {'unmixed_class': base_class}         # class attributes
        )

common/lib/xmodule/xmodule/modulestore/__init__.py:
mixologist = Mixologist(settings.XBLOCK_MIXINS)

Open edX from the inside

Viewing a course

lms/envs/common.py
cms/envs/common.py
# These are the Mixins that should be added to every XBlock.
# This should be moved into an XBlock Runtime/Application object
# once the responsibility of XBlock creation is moved out of modulestore
XBLOCK_MIXINS = (
    LmsBlockMixin,
    InheritanceMixin,
    XModuleMixin,
    EditInfoMixin,
    AuthoringMixin, # (In the CMS only)
)

Open edX from the inside

Viewing a course

# These are the Mixins that should be added to every XBlock.
# This should be moved into an XBlock Runtime/Application object
# once the responsibility of XBlock creation is moved out of modulestore
  1. What is an "XBlock"?
  2. What is an "XBlock Runtime/Application"?
  3. What is a "modulestore"?

XBlocks from the inside

General explanation of Open edX and XBlocks (2013) (2'24): https://www.youtube.com/watch?v=dTS-nsf7d3Q

"XBlocks all the way down" -- Ned Batchelder, Appsembler webinar (15'): http://www.appsembler.com/blog/open-edx-xblocks-webinar/

Examples: course, poll, peer assessed exams,
jsme (molecule editor)...

The xblock directory: http://xblocks.com http://xblocks.org/

XBlocks from the inside

"XBlocks all the way down" -- Ned Batchelder

XBlocks from the inside

"XBlocks all the way down" -- Ned Batchelder

pprint(course.__class__.__mro__) # class and all base classes of 'course'
(<class 'xblock.internal.CourseDescriptorWithMixins'>,
 <class 'xmodule.course_module.CourseDescriptor'>,
 <class 'xmodule.course_module.CourseFields'>,
 <class 'xmodule.seq_module.SequenceDescriptor'>,
 <class 'xmodule.seq_module.SequenceFields'>,
 <class 'xmodule.seq_module.ProctoringFields'>,
 <class 'xmodule.mako_module.MakoModuleDescriptor'>,
 <class 'xmodule.mako_module.MakoTemplateBlockBase'>,
 <class 'xmodule.xml_module.XmlDescriptor'>,
 <class 'xmodule.xml_module.XmlParserMixin'>,
 <class 'xmodule.x_module.XModuleDescriptor'>,
 <class 'xmodule.x_module.HTMLSnippet'>,
 <class 'xmodule.x_module.ResourceTemplates'>,
 <class 'lms.djangoapps.lms_xblock.mixin.LmsBlockMixin'>,
 <class 'xmodule.modulestore.inheritance.InheritanceMixin'>,
 <class 'xmodule.x_module.XModuleMixin'>,
 <class 'xmodule.x_module.XModuleFields'>,
 <class 'xblock.core.XBlock'>,
 <class 'xblock.mixins.XmlSerializationMixin'>,
 <class 'xblock.mixins.HierarchyMixin'>,
 <class 'xmodule.mixin.LicenseMixin'>,
 <class 'xmodule.modulestore.edit_info.EditInfoMixin'>,
 <class 'xblock.XBlockMixin'>,
 <class 'xblock.core.XBlockMixin'>,
 <class 'xblock.mixins.ScopedStorageMixin'>,
 <class 'xblock.mixins.RuntimeServicesMixin'>,
 <class 'xblock.mixins.HandlersMixin'>,
 <class 'xblock.mixins.IndexInfoMixin'>,
 <class 'xblock.mixins.ViewsMixin'>,
 <class 'xblock.core.SharedBlockBase'>,
 <class 'xblock.plugin.Plugin'>,
 <type 'object'>)

XBlocks from the inside

XBlock-poll: "A user-friendly way to query students."
https://github.com/open-craft/xblock-poll

poll/poll.py:
@XBlock.wants('settings')class PollBlock(XBlock):
    answers = List(scope=Scope.settings, help="The answer options on this poll.")
    choice = String(scope=Scope.user_state, help="The student's answer")
    ...
    def studio_view(self):
        ...
        return xblock.fragment.Fragment(some_html_code)
    def img_alt_mandatory(self):
        """
        Determine whether alt attributes for images are configured to be mandatory.
        """
        settings_service = self.runtime.service(self, "settings")
        if not settings_service:
            return True
        xblock_settings = settings_service.get_settings_bucket(self)
        return xblock_settings.get('IMG_ALT_MANDATORY', True)

XBlock runtimes

Runtime responsibilities:

1. Instantiate xblocks

class XBlock(...):
    def __init__(self, runtime, ...):
        ...
class Runtime(object):
    def construct_xblock_from_class(self, cls, scope_ids,
                                    field_data=None, *args, **kwargs):
        return self.mixologist.mix(cls)(runtime=self, scope_ids=scope_ids,
                                        field_data=field_data, *args, **kwargs)

2. Provide service to xblocks

settings_service = self.runtime.service(self, "settings")

XBlock runtimes

XBlock serialization/deserialization ('field-data' service)

xblock/core.py:
class XBlock(..., ScopedStorageMixin, ...):
    ...
xblock/mixins.py:
@RuntimeServicesMixin.needs('field-data')
class ScopedStorageMixin(...):     
    @property
    def _field_data(self):
        return self.runtime.service(self, 'field-data')

    def force_save_fields(self, field_names):
        ...
        self._field_data.set_many(self, fields_to_save_json)
xblock/fields.py:
class Field(...):
    def __get__(self, xblock, ...):
        field_data = xblock._field_data

        if field_data.has(xblock, self.name):
            return value = self.from_json(field_data.get(xblock, self.name))
        else: ...

XBlock runtimes

XBlock serialization/deserialization ('field-data' service)

xblock/field_data.py:
class FieldData(object):
    @abstractmethod
    def get(self, block, name):
        raise NotImplementedError

    @abstractmethod
    def set(self, block, name, value):
        raise NotImplementedError

    @abstractmethod
    def delete(self, block, name):
        raise NotImplementedError

    @abstractmethod
    def has(self, block, name):
        try:
            self.get(block, name)
            return True
        except KeyError:
            return False

    def set_many(self, block, update_dict):
        for key, value in update_dict.items():
            self.set(block, key, value)

Course components

are actually

StuffWithMixins

are also

XBlocks

instantiated by

a runtime

with a

'field-data' service

that stores data in

where?

XBlock storage

"The scope of an xblock field defines the set of xblock instances over which the field takes the same value."

lms/djangoapps/lms_xblock/field_data.py:
class LmsFieldData(SplitFieldData):
        def __init__(self, authored_data, student_data):
            # See also CmsFieldData in cms/lib/xblock/field_data.py
            ...
            super(LmsFieldData, self).__init__({
                # one block, all users
                Scope.content: authored_data,            # all courses
                Scope.settings: authored_data,           # one course
                Scope.user_state_summary: student_data,  # aggregated user data

                # one user
                Scope.user_state: student_data,          # one block, one course
                Scope.user_info: student_data,           # all blocks
                Scope.preferences: student_data,         # all blocks from same type

                # XBlock-specific properties
                Scope.parent: authored_data,
                Scope.children: authored_data,
            })
class PollBlock(XBlock):
    answers = List(scope=Scope.settings, help="The answer options on this poll.")
    choice = String(scope=Scope.user_state, help="The student's answer")
    ...

XBlock storage

Student data

lms/djangoapps/courseware/module_render.py:
def get_module_for_descriptor(user, request, descriptor, ..., course_key, ...):
    return get_module_system_for_user(
        ...
        student_data=KvsFieldData(DjangoKeyValueStore(...))
        ...
    )

XBlock storage

Authored data

lms/envs/common.py:
MODULESTORE = {
    'default': {
        'ENGINE': 'xmodule.modulestore.mixed.MixedModuleStore',
        'OPTIONS': {
            'stores': [
                {
                    'NAME': 'split',
                    'ENGINE': 'xmodule.modulestore.split_mongo.split_draft.DraftVersioningModuleStore',
                    ...
                },
                {
                    'NAME': 'draft',
                    'ENGINE': 'xmodule.modulestore.mongo.DraftMongoModuleStore',
                    ...
                },
                {
                    'NAME': 'xml',
                    'ENGINE': 'xmodule.modulestore.xml.XMLModuleStore',
                    ...
                }
            ]
        }
    }
}
# These are the Mixins that should be added to every XBlock.
# This should be moved into an XBlock Runtime/Application object
# once the responsibility of XBlock creation is moved out of modulestore
  1. What is an "XBlock"?
  2. What is an "XBlock Runtime/Application"?
  3. What is a "modulestore"?

XModules vs XBlocks

Introduction of XBlocks:

commit 789ac3fc875aa26380fc7f0865dc5c89a7359473
    Author: Calen Pennington <calen.pennington@gmail.com>
    Date:   Fri Jan 4 16:19:58 2013 -0500

        Use the XBlock library as the base for XModule, so that we can
        incrementally rely on more and more of the XBlock api

XModules vs XBlocks

Introduction of XBlocks:

# bisect from latest release to first commit
$ git bisect start named-release/dogwood.3 cc1de22e2
$ git bisect run ./bisect.sh
$ cat ./bisect.sh
#! /bin/bash
if [ "$(git grep -i xblock | wc -l)" -le "10" ]; then
    exit 0
else
    exit 1
fi

XModules (deprecated)

XModuleDescriptor instantiation:

ModuleSystem  -------------  |
                             |   XModuleDescriptor
Modulestore   -------------  |
(author data)

Bind XModuleDescriptor to user:

ModuleSystem  -------------  |
                             |
Modulestore   -------------  |
(authored data)              |
                             |   XModule
Modulestore   -------------  |
(student data)               |
                             |
Student       -------------  |

XBlock SDK

XBlock instantiation:

Runtime     ---------------  |
                             |
Field data  ---------------  |
(authored data)              |
                             |   XBlock
Field data  ---------------  |
(student data)               |
                             |
Student     ---------------  |

Open edX XBlocks (backward-compatible)

XBlock instantiation:

Runtime     ---------------  |
                             |   XBlock (partially working)
Field data  ---------------  |
(authored data)

Bind XBlock to user:

Runtime     ---------------  |
                             |
Field data  ---------------  |
(authored data)              |
                             |   XBlock
Field data  ---------------  |
(student data)               |
                             |
Student     ---------------  |

The LMS/CMS runtimes

lms/djangoapps/lms_xblock/runtime.py:
class LmsModuleSystem(ModuleSystem):
    ...
cms/djangoapps/contentstore/views/preview.py:
class PreviewModuleSystem(ModuleSystem):
    ...
common/lib/xmodule/xmodule/x_module.py:
class ModuleSystem(..., xblock.runtime.Runtime):
    ...

The LMS/CMS runtimes

Adding new runtime services

lms/djangoapps/courseware/module_render.py:
system = LmsModuleSystem(
    ...,
    services={
        'i18n':  ModuleI18nService(),
        'fs': FSService(),
        'field-data': ...,
    },
    ...
)

The LMS/CMS runtimes

Adding new runtime services

lms/djangoapps/lms_xblock/runtime.py:
class LmsModuleSystem(ModuleSystem):
    # settings.LMS_RUNTIME_SERVICES = {
    #    "myservice": callable,
    #    ...
    # }
    def __init__(self, ..., services, ...):
        for service_name, callable in settings.LMS_RUNTIME_SERVICES.items():
            if service_name not in services:
                services[service_name] = callable()
        ...

Get in touch

Régis

Régis Behmo
regis@fun-mooc.fr

Richard

Richard Moch
richard@fun-mooc.fr

Sylvain

Sylvain Toé
sylvain@fun-mooc.fr

Slides available at
https://github.com/regisb/openedx-conference-2016