SLCM INNOVATIONS TO SUPPORT TEACHING AND LEARNING – University of Montevideo HERUG 2014

April 2014
HERUG - University of Montevideo
1
SLCM INNOVATIONS TO SUPPORT
TEACHING AND LEARNING
HERUG 2014 – University of Montevideo
Kathy Gates
Chief Information Officer
University of Mississippi
April 2014
HERUG - University of Montevideo
University of Mississippi Profile
• Public university
located in
Oxford, Mississippi
(about 60 miles south of
Memphis, TN)
• Liberal arts focus
• Enrollment
• Total for Fall 2013: ~
19K
• Undergraduate
Statistics
• 25% minority enrollment
• 38% out of state
2
April 2014
IT Landscape
• SAP on Sun hardware
and Oracle database
• Initial “go lives” in 1999
• SLcM in 2003
• Medical Center in 2009
• Blackboard
• Strong but lean staff
HERUG - University of Montevideo
3
April 2014
HERUG - University of Montevideo
A Tech-Enabled University
4
April 2014
HERUG - University of Montevideo
A Sampling of SLcM Projects
• Attendance Scanners
• Advising Features
• Degree Audit
• Reaching Students with Text Messaging
• Online Grade Changes
• Academic Discipline Workflow
• Faculty Activity Reports
• Student Rating of Instruction
5
April 2014
HERUG - University of Montevideo
Automated Attendance Tracking
• Provost’s Office had requested
this for several years
• Developed in partnership with
faculty
• System is designed to:
• Target lower level courses and
large classrooms
• Help develop good habits in
underprepared students
• Support faculty preferences and
reduce administrative burden
• Work with established retention
efforts
6
April 2014
HERUG - University of Montevideo
7
Motorola Symbol MK500 MicroKiosk
• Chosen by Helpdesk
Staff and Classroom
Technology
• 2D Barcode Imager
• Touch-Screen
Interface
• Speaker
• Wired/Wireless
Networking
• Power Adapter or
Power-Over-Ethernet
April 2014
HERUG - University of Montevideo
Scanners in Action
• Attendance records are updated
in SAP when student scans ID
• Instructor can view the results in
myOleMiss
8
April 2014
HERUG - University of Montevideo
9
Attendance Tracking Operation
• Start of the day, all students are marked as absent for a
configured class.
• Scans that occur outside of a configured scan time
respond with a negative beep and the message “Not a
valid scan time.”
• Within the scan window, student scans present a positive
beep and the message “Scan successful.”
• Scans are processed and students will be marked as
present in the system. (15 minute interval)
• Freshman Attendance Based Initiative (FABI) results are
updated hourly.
April 2014
HERUG - University of Montevideo
10
More Attendance Tracking Details
• About 50 classrooms enabled with scanners
• During the first two weeks of class, any students marked
absent are sent email notification
• Parents and students can see absences
• Custom solution developed for Mathematics Lab
• Solution for performing arts center in progress
• Inquiries from other universities including one looking for
a method to determine last day of attendance for financial
aid
April 2014
HERUG - University of Montevideo
11
April 2014
12
HERUG - University of Montevideo
3
Average # of Absences
2,5
2
1,5
1
0,5
0
A
A-
B+
B
B-
C+
C
Midterm Grades - 2013
C-
D
F
April 2014
HERUG - University of Montevideo
13
April 2014
HERUG - University of Montevideo
14
April 2014
HERUG - University of Montevideo
15
Attendance-Related Tweets
• 2/10/2014 7:16:45 PM
• Ole Miss needs to chill out with these attendance scanners
• 1/30/2014 12:56:39
• The scanners at ole miss have now made it possible for auditorium
classes to have an attendance policy. #notcool
• 10/14/2013 10:09:57
• These attendance scanners make being discretely late for class
practically impossible. #OleMissProblems
• 10/2/2013 9:59:56
• Im convinced that attendance scanners are definitely instruments of
the devil #OleMissProblems
• 8/28/2013 7:50:50 AM
• I'm really sick of these scanners already & it's only day 3.
@OleMissProblems
April 2014
HERUG - University of Montevideo
Advising Features
• Integrated Planning and Advising Systems (IPAS)
• The Ole Miss IPAS
• Multiple advisor types
• Instructor alerts
• New “Academic Risk” status
• Tactical
• Trigged by:
• Instructor alerts
• Absences
• Two or more D or F midterm grades
• Advisors are alerted via daily email
• Dashboard gives advisors easy way
to see advisees with an active “Academic Risk” status
16
April 2014
HERUG - University of Montevideo
17
April 2014
HERUG - University of Montevideo
18
April 2014
Degree Audit
HERUG - University of Montevideo

• Launched in 2008
• Challenges
• Complicated user interface
• Substitutions
• Maintenance of catalog
rules
• Trust in the system
• Progress
• Provost support
• Simplified degree progress
report
• Campus-wide adoption
19
• Current work
• Percent complete
• Mass reports
• Planned work
• Analytics to …
• Find students who are not
making satisfactory progress
toward a degree
• Plan future course offerings
based on demand
• Centralized degree
conferment with increased
automation
April 2014
HERUG - University of Montevideo
20
April 2014
HERUG - University of Montevideo
21
SMS Text Messaging
• 2SMS cloud service functions as a tightly integrated
extension to on-premises SAP system
• SPROXY is used to build ABAP proxy from vendor WSDL
• Use cases
• Emergencies
• Instructor and advisor messages
• Automated alerts from SAP
• WebID is ready
• Registration window is about to open / has opened
• Time to complete teacher evaluations
• See your advisor
April 2014
HERUG - University of Montevideo
22
April 2014
HERUG - University of Montevideo
23
April 2014
HERUG - University of Montevideo
Online Grade Changes
• New process to replace paper grade
change cards
• Go live in Summer 2012
• 5737 grade changes
• Process
• Grade change initiated by instructor or
academic administrator
• Parties informed throughout the process
• Opportunities for transcription errors eliminated
• Grade is changed in SAP automatically on dean
or dean delegate approval
• Support for changes within normal grade
submission window
24
April 2014
HERUG - University of Montevideo
25
April 2014
HERUG - University of Montevideo
Academic Discipline Workflow
• Initiated by instructors from myOleMiss portal
• Instructor selects / provides:
• Sanction(s)
• Case summary and details
• Optional supporting materials
• Upload files (pdf only) or
• Send to Provost’s Office
• Documentation Examples
• Safe Assignment report (direct link and pdf copy)
• Syllabus containing academic dishonesty policy
• Communication between faculty member and student
• Discuss that meeting or attempt(s) to meet with student
• Did the student accept the proposed sanction(s)?
26
April 2014
HERUG - University of Montevideo
27
April 2014
HERUG - University of Montevideo
28
Academic Discipline Process
• Once case is initiated …
• Automatic email to:
• student
• faculty member
• chair of Academic Discipline
Committee
• department offering the
course
• college/school offering the
course
• university officials
• Case number w/ link to
view details
• Warning not to drop course
• Appeal window (14
calendar days) begins
when case initiated
• Reminder email about
appeal window sent 2
days before window
closes
• Registrar’s Office sends
letter with case details to
student’s home address
April 2014
HERUG - University of Montevideo
Sanction Type
FAILURE ON ASSIGNMENT
336
REDUCED GRADE ON
ASSIGNMENT
REPEAT ASSIGNMENT
101
7
OTHER
104
PROBATION
49
REDUCED GRADE ON COURSE
107
FAILURE IN COURSE
SUSPENSION
EXPULSION
266
13
6
29
Fall 2005
Spring 2006
Summer 2006
Fall 2006
Spring 2007
Summer 2007
Fall 2007
Spring 2008
Summer 2008
Fall 2008
Spring 20009
Summer 2009
Fall 2009
Spring 2010
Summer 2010
Fall 2010
Spring 2011
Summer 2011
Fall 2011
Spring 2012
Summer 2012
Fall 2012
Srring 2013
Summer 2013
Fall 2013
April 2014
HERUG - University of Montevideo
30
Number of Cases
90
80
70
60
50
40
30
20
10
0
April 2014
HERUG - University of Montevideo
31
Faculty Activity Reports
• Go live in Spring
2005
• Revolt in 2009
• Redesign with good
results
1000
900
800
700
600
Complete
500
In Progress
400
300
200
• How long does it take
for academic projects
to achieve full
adoption?
100
0
2006 2007 2008 2009 2010 2011 2012 2013 2014
April 2014
HERUG - University of Montevideo
32
April 2014
HERUG - University of Montevideo
33
Student Ratings of Instruction
• Much-studied topic
• Most extensive area of
study in higher education?
• Most studies focus on very
small data sets.
• Fall 1999
• UM moved to an online
process for reporting
results.
• Fall 2003
• UM moved to an online
process for collecting
results.
• Sentiment analysis of
narrative feedback is a
big focus area right now.
• Movie reviews
• Product ratings
• More …
• UM’s database of
comments is a sentiment
analysis gold mine.
April 2014
HERUG - University of Montevideo
34
Target Questions
• Q10: How would you rate the
difficulty level of this course,
compared to other courses you
have taken so far at Ole Miss?
•
•
•
•
•
Easy (1)
Average (2)
Difficult (3)
Very Difficult (4)
Extremely Difficult (5)
• Q11: How would you rate the
instructor’s overall performance in
this course?
•
•
•
•
•
Poor (1)
Marginal (2)
Good (3)
Excellent (4)
Superior (5)
• NAR1: What are some positive
characteristics or strengths of the
course and/or instructor?
NAR2: What are some negative
characteristics or weaknesses of
the course and/or instructor?
April 2014
HERUG - University of Montevideo
35
Variables of Interest
• Student
•
•
•
•
•
•
•
Age
Gender
Ethnicity
Classification
Overall GPA
Grade in course
Academic area
• Instructor
•
•
•
•
Age
Gender
Ethnicity
Rank
• Course
• Term when course was taught
• Campus where course was
•
•
•
•
•
•
•
taught
Time of day when course was
taught
Period within term when
ratings were collected
Section size
Level
Category (lecture, laboratory,
studio, etc.)
Delivery mode (online,
interactive video, traditional)
Academic area
April 2014
HERUG - University of Montevideo
36
Technical Tools
• Raw evaluation
responses in an Oracle
database.
• Extract that gathers
related student, course,
and instructor metadata
and stores with target
responses in a mySQL
database.
• Java programs and SQL
queries to process and
synthesize the results.
• Plan to acquire a more
sophisticated reporting
toolset to:
• Reduce the technical
skillset required to interact
with the data
• Speed up development of
reports
• Provide better visualization
capabilities
• Handle very large datasets
April 2014
HERUG - University of Montevideo
37
Database Overview
• 999,992 total evaluations
• 497,938 with NAR1 or
NAR2 having sentiment
• 330,729 NAR1
• 167,209 NAR2
• Earliest evaluation was
submitted on 11/11/2005
• Most recent evaluation
was submitted on
01/07/2014
• 6,258 rules to correct
spelling errors
• 23,163 unique tagged
words
• 10,511,936 occurrences of
tagged words
• Average positive sentiment
for evals containing
detectable sentiment: 2.88
• Average negative
sentiment for evals
containing detectable
sentiment: .75
April 2014
HERUG - University of Montevideo
38
Average Length in Characters of Non-Null Responses by Q11
Rating
(Overall Instructor Performance)
250,00
212,66
200,00
150,92
150,00
100,00
97,67
95,01
91,22
82,54
67,55
75,11
64,63
47,16
50,00
0,00
1 (Poor)
2 (Marginal)
3 (Good)
4 (Excellent)
5 (Superior)
Nar1: Positive characteristics or
strengths of the course and/or
instructor
Nar2: Negative characteristics or
weaknesses of the course and/or
instructor
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 6.938
Negative: 0.000
Normalized -- Positive: 100.000
Negative: 0.000
Category Scores
Engaging: 2.438 (positive) 0.000 (negative)
Learning: 1.875 (positive) 0.000 (negative)
General: 2.625 (positive) 0.000 (negative)
very smart and enthusiastic
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 4.688
Negative: 0.000
Normalized -- Positive: 100.000
Negative: 0.000
Category Scores
Knowledgeable: 2.250 (positive) 0.000 (negative)
Engaging: 2.438 (positive) 0.000 (negative)
Positive Sentiment
He is very enthusiastic about the material. It is essential to buy the
books b c the tests come straight from the books, not the lecture! It was
neat because at the beginning of every class he would give us an overview
of what was happening in the world at that point!! This kept me very
informed of international news.
39
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 18.494
Negative: 0.000
Normalized -- Positive: 100.000
Negative: 0.000
Category Scores
Helpful: 1.375 (positive) 0.000 (negative)
Knowledgeable: 6.094 (positive) 0.000 (negative)
Engaging: 3.000 (positive) 0.000 (negative)
Learning: 3.750 (positive) 0.000 (negative)
General: 4.275 (positive) 0.000 (negative)
Y is a CLEAR speaker and his tests are very straight forward. He's
a fair guy
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 5.125
Negative: 0.000
Normalized -- Positive: 100.000
Negative: 0.000
Category Scores
Fair: 1.625 (positive) 0.000 (negative)
Clear: 3.500 (positive) 0.000 (negative)
Positive Sentiment
Y is an excellent Honors teacher and I am "honored" to be in the last
cohort of students that he taught in Honors 102. He is very intelligent and
very knowledgeable in a variety of subjects. Grades papers fairly, assigns
work that is challenging and intriguing. Helps to make his students think,
which is what the class is meant to do. He gave the entire class a book
that he thought we might enjoy.
40
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
I really did not like this course. The positive was there were quiz grades that
could bring up crappy grades on tests.
Category Scores
General: 1.625 (positive) 2.875 (negative)
You think you're just memorizing useless crapp, but you're actually learning to
speak spanish
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 4.188
Negative: 1.625
Normalized -- Positive: 72.043
Negative: 27.957
Category Scores
Clear: 1.125 (positive) 0.000 (negative)
Learning: 3.063 (positive) 1.625 (negative)
His accent sometimes makes it hard to take notes during lectures but he does a
very good job of updating blackboard and keeping the notes available.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 1.375
Negative: 0.800
Normalized -- Positive: 63.218
Negative: 36.782
Category Scores
Helpful: 1.375 (positive) 0.000 (negative)
Clear: 0.000 (positive) 0.800 (negative)
Mixed Sentiment
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 1.625
Negative: 2.875
Normalized -- Positive: 36.111
Negative: 63.889
41
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
42
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 0.000
Negative: 1.500
Normalized -- Positive: 0.000
Negative: 100.000
Category Scores
Clear: 0.000 (positive) 1.500 (negative)
The lecture could get pretty monotoned so sometimes it was hard to pay attention
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 0.000
Negative: 2.625
Normalized -- Positive: 0.000
Negative: 100.000
Category Scores
Engaging: 0.000 (positive) 2.625 (negative)
not enough writting assignments
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 0.000
Negative: 1.000
Normalized -- Positive: 0.000
Negative: 100.000
Category Scores
Learning: 0.000 (positive) 1.000 (negative)
Negative Sentiment
too fast
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 1.250
Negative: 2.438
Normalized -- Positive: 33.898
Negative: 66.102
Category Scores
Learning: 1.250 (positive) 0.000 (negative)
General: 0.000 (positive) 2.438 (negative)
I will not remember any of what I learned.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 1.625
Negative: 1.125
Normalized -- Positive: 59.091
Negative: 40.909
Category Scores
Learning: 1.625 (positive) 1.125 (negative)
Challenging
His tests were hard as hell because he went to far in depth with the
subject matter
43
February 2014
Copyright © 2014 The University of Mississippi. All Rights Reserved.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 4.125
Negative: 1.500
Normalized -- Positive: 73.333
Negative: 26.667
Category Scores
Knowledgeable: 1.500 (positive) 0.000 (negative)
Engaging: 1.625 (positive) 0.000 (negative)
Learning: 1.000 (positive) 0.000 (negative)
Accountability: 0.000 (positive) 1.500 (negative)
where do i begin
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 0.000
Negative: 0.000
Normalized -- Positive: 0.000
Negative: 0.000
Y is the man.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 0.000
Negative: 0.000
Normalized -- Positive: 0.000
Negative: 0.000
Challenging
he knows his [expletive deleted]. he loves the subject and it shows.
44
April 2014
45
HERUG - University of Montevideo
Sentiment by Grade in Class
4,000
3,500
3,360
3,23
3,180
3,000
2,760
2,600
2,500
2,660
2,350
2,180
2,060
2,000
1,920
1,500
1,000
0,720
0,650
0,500
0,840
0,820
0,750
0,950
0,970
0,890
0,840
1,120
0,000
A
A-
B+
B
B-
C+
C
Average Positive Summary Sentiment Score
Average Negative Summary Sentiment Score
C-
D
F
Average Positive Sentiment
Average Negative Sentiment
0,00
PARKS, RECREATION, LEISURE, AND
FITNESS STUDIES
HOMELAND SECURITY, LAW
ENFORCEMENT, FIREFIGHTING AND…
BIOLOGICAL AND BIOMEDICAL SCIENCES
HERUG - University of Montevideo
COMPUTER AND INFORMATION
SCIENCES AND SUPPORT SERVICES
MATHEMATICS AND STATISTICS
PHYSICAL SCIENCES
ENGINEERING
BUSINESS, MANAGEMENT, MARKETING,
AND RELATED SUPPORT SERVICES
FAMILY AND CONSUMER
SCIENCES/HUMAN SCIENCES
SOCIAL SCIENCES
LIBERAL ARTS AND SCIENCES, GENERAL
STUDIES AND HUMANITIES
COMMUNICATION, JOURNALISM, AND
RELATED PROGRAMS
VISUAL AND PERFORMING ARTS
PHILOSOPHY AND RELIGIOUS STUDIES
HISTORY
AREA, ETHNIC, CULTURAL, GENDER, AND
GROUP STUDIES
1,00
ENGLISH LANGUAGE AND
LITERATURE/LETTERS
3,67
PSYCHOLOGY
1,50
PUBLIC ADMINISTRATION AND SOCIAL
SERVICE PROFESSIONS
3,00
MULTI/INTERDISCIPLINARY STUDIES
0,97 0,66
FOREIGN LANGUAGES, LITERATURES,
AND LINGUISTICS
2,00
EDUCATION
3,50
HEALTH PROFESSIONS AND RELATED
PROGRAMS
4,00
LEGAL PROFESSIONS AND STUDIES
April 2014
46
Sentiment by Academic Area
5,00
4,50
0,93
0,75 0,78 0,60 0,63
0,71 0,71 0,76 0,67
0,71 0,68 0,48 0,78 0,64
0,83 0,82 0,80 0,81 0,84 0,83 0,57 0,46
2,50
3,27 3,41 3,13 3,03 3,19 3,10 3,01
2,99 2,91 2,96 2,87 2,86 3,04 2,73 2,80
2,60 2,59 2,61 2,54 2,47 2,48 2,65 2,70
0,50
April 2014
HERUG - University of Montevideo
47
(51591612D79865D0E1000000824A0708)
I truly believe Dr. Watson is one of the smartest men I've ever been around. He's
absolutely brilliant, and he knows all things Faulkner. He's also extremely easy to talk to
which is not always the case with really smart people. He's very professional at all times,
but also laid back and non-intimidating. I would take his class every semester if I could.
He's one of my favorite professors. Even if I hated all my other English professors (which
I don't), his class would make me glad I chose Ole Miss. Even though he's a published
Faulkner scholar, he makes his students feel like our comments in class discussion on the
novels are so interesting and insightful. Dr. Watson is an encourager. He's also really
serious about us turning in good work and showing up for class and working hard. The way
that he leads discussion to get the class thinking and thinking about the right stuff is so
perfect. He allows students to present original ideas, and he gives us his own insight. He
works really hard outside of class to help us understand the texts by making us charts and
graphs and handouts. I've been so amazed every day of class by his efforts. His class is
the only class I've had perfect attendance in so far. I love it.
Overall Sentiment Scores (Excluding Difficulty)
Total -- Positive: 43.129
Negative: 0.000
Normalized -- Positive: 100.000
Negative: 0.000
Category Scores
Helpful: 1.375 (positive) 0.000 (negative)
Knowledgeable: 12.539 (positive) 0.000 (negative)
Engaging: 3.344 (positive) 0.000 (negative)
Learning: 9.750 (positive) 0.000 (negative)
Difficulty: 0.000 (positive) 2.031 (negative)
General: 16.121 (positive) 0.000 (negative)
WARNING: 2 phrase(s) not processed due to complexity.
Most
Positive
Sentiment
Fall 2013
April 2014
HERUG - University of Montevideo
Next Steps
• Portal Redesign
• Responsive design as with campus website
• Accessibility
• Role-based dashboards
• The idea of a dashboard to get a users attention.
• Incorporation of IPAS features
• More Automation
• Suite on HANA
• Enterprise Reporting Tool
48
April 2014
HERUG - University of Montevideo
49
Questions & Discussion
Kathy Gates
[email protected]
Thanks to Jimmy Ball, Frank Mathew, Chris Reichley, and
Margaret Walden for their contributions to this presentation.