Notes from the exploration and selection process for Chico's next Learning Management System

Teaching and Learning Evaluation Results Summary

To perform the actual evaluation of teaching and learning, each faculty and instructional support member of our evaluation team was given access to both a standard Moodle 1.9 and Blackboard Learn 9.1 environment, including blank and populated courses, and access to each other’s courses as faculty and students. The Learn environment included the Content module, although many of its collaborative and content-sharing features were not formally evaluated. No formal training was given in each LMS, but online tutorial references were supplied to evaluators including step-by-step tutorials and quick videos.

Team members used our LMS Evaluation Rubric (PDF) to evaluate each LMS feature or capability on functionality as well as easy of use. All evaluators were asked to select those features they use, or would like to use, because the list was too exhaustive to expect active faculty to try everything, including LMS features with which they were not familiar. Comments on each feature and overall for each LMS were also solicited. The six-week timeline for this evaluation process (detailed in a previous post) was extended several weeks due to our busy semester, and in the end not all team members elected to submit completed rubrics. However, we feel the end results do represent the overall tenor of the evaluation team.

These results were presented to the evaluators in a meeting in December, 2010, with the understanding that this data would help to shape an LMS recommendation but it did not necessarily dictate what that recommendation would be. The final recommendation would take into account this data as well as factors such as cost, staff impact, technical requirements, license structure and governance.

The Evaluation Verdict

In short, Blackboard Learn 9.1 had a slight edge in both the Ease of Use and the Functionality measures over Moodle, in almost all feature areas.

Several faculty evaluators expressed a preference for either Moodle or Blackboard, but nearly all members of the group felt that both systems could capably serve the teaching and learning functions of Chico State.

Now, let’s have some charts!

First, here’s a graph showing our evaluators’ ratings of the top 10 tools used by faculty on our campus (according to the faculty survey):

chart showing ratings for faculty's top 10 features

Next, a graph showing our evaluation team ratings of the top ten tools used by students (from the student survey):

evaluator ratings of students' top 10 features

And here are the results across all teaching and learning functions evaluated. The order of features is taken from the evaluation rubric’s groupings.

Chart of results for all features by evaluators

I think even through the graph is difficult to read, it’s possible to get an overall sense from the bar height of the general preference towards Blackboard Learn 9.1.

Here are the functionality and ease-of-use results separated, and displayed in a way which shows the preference via a center point on each two-color bar.

evaluator ratings of functionality

evaluator ratings for ease of use

3 Responses to “Teaching and Learning Evaluation Results Summary”

  1. Blackboard v Moodle « julian Says:

    [...] while I’m in blogging mode, I thought this comparison of Blackboard and Moodle, effectively the two leading VLEs in the sector, was interesting. Of course it’s only one [...]

  2. Fred Barnes Says:

    So I guess you’ll have to ask yourselves – are the small differences your group found enough to warrent the extra money? If you think of the difference in cost between Moodle and Blackboard in terms of say, adjuncts – how many adjuncts will you have to fire in order to afford Blackboard?

    Or Secretaries? Groundskeepers? IT Staff?

  3. Peter DiFalco Says:

    Hi Fred,

    I think that each institution approaches IT budgets in a different way. The “free software is cheaper” argument really depends on your enterprise software organization’s capacity, strengths, and goals. Your implication that a Blackboard license fee would come at the cost of existing staff salary doesn’t reflect our reality.