Event Name - 022--Objective Eval-A Search for Parity in an Online... (SHM197166)


Janet Groen: Hi Scott - how are you?
Scott Walker: Hi.
Scott Walker: Just getting used to this web environment.
Scott Walker: While we're waiting. Where are you?
Scott Walker: Hello Deborah.
Deborah Conklin: Hi Scott, do we go through the PPT ourselves or do you do that?
Scott Walker: I'll go through it.
Deborah Conklin: Good enough!
Iskandar Hack: right
Scott Walker: I always like to know who I'm talking to, so please let us know where you are and what you do.
Janet Groen: Hi Scott:  I figured out how to access the text messaging box - sorry for the delay.  I'm up here in Calgary Alberta
Scott Walker: "Up there" is right. The great white north, eh?
Scott Walker: 'Tis 80 degree F and sunny here in Texas.
Deborah Conklin: I'm from Indiana Purdue Fort Wayne. I'm Ex. Dir for Continuing Studies
Janet Groen: I'm an assistant professor within Work Place and Adult Learning and a majority of my instruction is through distance.  Actually it isn't looking too white right now - we've had chinook winds and the temp was in the low 50s earlier this week.
Scott Walker: I think I know a bit about Iskandar, been getting a lot of messages from him lately!
Janet Groen: Hi again:  will this session be don via direct messaging or is there talking capability
Scott Walker: No talking.
Scott Walker: All text and ppt
Janet Groen: Oh okay - Iskandar - is this the process for all of the sessions - just interested because I present on Friday
Scott Walker: Want to start by asking who here has done distance ed program evaluation?
Scott Walker: Ah, silence.
Scott Walker: I'll assume that means no one?
Deborah Conklin: Our office is responsible for Student services, we have done online surveys/eval's
Janet Groen: Hi again Scott:  We have not systematically evaluated our distance education program and we will need to evaluate our Doctorate program soon
Scott Walker: IC
Iskandar Hack: It is up to you, some ppl are just using chat without the ppt - I prefer to see ppl present ppt then go to a chat session. (sorry for interupting)
Scott Walker: Great. Then let's talk about evaluation.
Janet Groen: okay - I'm ready
Scott Walker: Please interrupt at anytime with questions. I would like for this to be interactive.
Scott Walker: Also, I’m seeking positive criticism of this paper, hard questions, etc., as I would like to publish it…consider this a field test where you can chime in to challenge my thoughts OR agree with them.
Scott Walker: Let me first introduce myself.
Scott Walker: My name is Scott Walker. I have a, let’s say, non-linear background that brings me here to my 4th TOHE conference. I’m just coming away from 7 years at a small liberal arts college where I was on the education faculty. I created that university’s first online degree program, a M.Ed. with a concentration as Master Technology Teacher (which is a teaching certificate in Texas). http://education.ollusa.edu/mtt/
Scott Walker: Altogether I have now 10 years experience developing, implementing, teaching, and learning in online environments.
Scott Walker: Nowadays, I find myself on the Dept. of Geography faculty at Texas State University in San Marcos, Texas. I mostly "do" geographic education, but I was hired, in part, for my distance ed background (I have a masters in geography too).
Scott Walker: My distance ed background landed me a project within the first few weeks in my new university job. Our (interim) dept chair called me in one day and said, "Scott, I understand we have some issues with our online GIS certificate program. Don’t spend a lot of time on it, but look into it for me."
Scott Walker: Knowing one can’t just "look into" something meaningfully and not spend a lot of time on it I opted to leverage my new task by getting more than a shoddy report out of it. I opted to do it right and present it here, then turn it into a publication. I refer to this as a "rapid" evaluation because I conducted it much faster than I would have liked, but based my work on a solid literature foundation (and those 10 yrs. of defending distance ed to deans and provosts, and university presidents).
Scott Walker: This is my story...
Scott Walker: I'm assuming the ppt is changing on your screen too.
Janet Groen: yes
Scott Walker: Want to be clear on some terms.
Scott Walker: Evaluation is different from assessment in the world according to Scott.
Scott Walker: Assessment is how one determines what students are achieving.
Scott Walker: Evaluation, while including assessment in some cases, is looking at a program overall.
Scott Walker: Two parts to evaluation
Scott Walker: 1) measuring
Scott Walker: 2) judging
Scott Walker: If we don't have quality measurement in place we often can't make quality judgements.
Scott Walker: Also, if our measurement is done well, we reduce the need for reliance on beliefs and opinions.
Scott Walker: That said, the politics of it all (part of this conf. theme) results from beliefs and others trying to influence with theirs.
Scott Walker: A 2002 paper I have reads that we're in the 5th generation of distance ed.
Scott Walker: But, I think we're still in the 1st generation of evaluating DE programs.
Scott Walker: (DE) distance ed
Scott Walker: Literature says our evaluation is "less than rigorous"
Scott Walker: I agree with that.
Scott Walker: You can see on the slide the categories evaluation of programs usually falls into.
Scott Walker: Do any of you evaluate with only these measures? (rhetorical question)
Janet Groen: sigh!
Scott Walker: How many have seen a made up survey as the primary means of measurement?
Scott Walker: Rhetorical again.
Scott Walker: Janet don't fret. I offer solutions here.
Janet Groen: Okay!
Scott Walker: Just laying out the background.
Scott Walker: !!!
Scott Walker: Program evaluation has an overall goal or purpose as noted in the PPT
Scott Walker: It also has techniques (strategies), being able to do it well, and playing the game (politics).
Scott Walker: On the ppt are three things evaluation often seeks to do
Scott Walker: Do we need it, is it going as planned, and does it work.
Scott Walker: In the world according to Scott, there are "sub" purposes.
Scott Walker: Anyone ever had to do evaluation for #3?
Scott Walker: Digging through publications on program evaluation (not necessarily DE eval) I found these 6 approaches.
Scott Walker: Briefly on each one.
Scott Walker: Does the program do what you set out to do?
Scott Walker: Well, if you don't have solid objectives how do you know?
Scott Walker: What about hidden agendas? Those that are not articulated, but are expected.
Scott Walker: ex. making money, generating more student hours.
Scott Walker: and such
Scott Walker: Mis-measurement can be assuming student grades = meeting the objectives.
Scott Walker: In other words, if the instructor's students are making the grade, does that mean the program is high-quality?
Scott Walker: Ever heard of grade inflation?
Scott Walker: Kim says yes.
Scott Walker: I have evidence of such coming up.
Solomon Isiorho: :)
Scott Walker: Consumer-oriented is often "satisfaction" evaluation at the end.
Scott Walker: Problem is, can't fix the program at the end.
Scott Walker: Also, assumes somewhat of a one-size fits all scenario.
Scott Walker: Seen this one before?
Solomon Isiorho: nods
Scott Walker: If I'm evaluating my program and funding is contingent upon a good outcome, of course it will turn out that the program is the best in the world.
Kim Shipman: Can you give us an example?
Scott Walker: Often we eval our own programs. At least in my experience.
Scott Walker: Sure Kim.
Scott Walker: I started an online degree program at OLLU.
Scott Walker: I had a budget of $xx,000
Scott Walker: I had to report to the provost on the health of the program after its first year.
Scott Walker: What do you think I told him?
Solomon Isiorho: the best program in the world
Scott Walker: yes
Scott Walker: Actually, I downplayed the negatives, up played the positives.
Kim Shipman: I understand your point, but isn't there a rubric to measure it against that someone wants to see?
Scott Walker: Didn't lie though!!!
Scott Walker: Sure there was.
Scott Walker: But the outcome would be different perhaps if someone else had done the eval.
Scott Walker: In Texas we're accredited by SACS.
Janet Groen: what's SACS
Kim Shipman: Were you the only one who evluated it?
Scott Walker: In a recent review I had to turn in all of the syllabi for my online classes.
Scott Walker: Kim, yes.
Scott Walker: Turning in syllabi doesn't equate to a high-quality program.
Scott Walker: Given that SACS does have other DE standards.
Scott Walker: Get the naysayers involved so you can win them over.
Scott Walker: This is this approach.
Scott Walker: If you can address their points, you can get their support.
Scott Walker: But, if not, watch out.
Scott Walker: This approach involves asking what folks think about the program.
Scott Walker: Stakeholders with more influence or the loudest voice may be able to influence the outcomes despite the reality of the program.
Scott Walker: So, there. 6 approaches glossed over very quickly.
Scott Walker: Just hitting the highlights of those.
Kim Shipman: ok
Scott Walker: In reality we seldom use those approaches exclusively. We mix them and match them.
Scott Walker: Chances are we won't all agree on which approaches are best.
Scott Walker: Thus, little consensus.
Scott Walker: There are standards that abound for DE and that is why they turn up.
Scott Walker: I found 7 different standards in looking into this from 7 organizations.
Scott Walker: Go figure.
Kim Shipman: Were there consistencies in the standards?
Scott Walker: Also, we often tend to mix course evaluation (one unit of analysis) with the eval of the entire program (yet another Unit of Analysis).
Scott Walker: Kim. Yes.
Scott Walker: Got the refs in my paper.
Scott Walker: P. 4.
Scott Walker: is the discussion
Scott Walker: For the sake of time I'll skip over the consistencies in standards.
Cree Stout: How does class size measure program effectiveness?
Scott Walker: Survey says...we tend to eval programs most based on student grades.
Scott Walker: Cree.
Scott Walker: Exactly.
Scott Walker: It doesn't.
Scott Walker: Unless one of the objectives is to have a certain class size
Scott Walker: Objective-oriented approach
Cree Stout: I know we as teachers like to use that method to reduce the class size, but..
Scott Walker: This is a list of what I found in the literature about program evaluation.
Scott Walker: I don't agree with it all, this is just what I turned up.
Scott Walker: Sort of to give us the state of program eval.
Scott Walker: So, given all that as background from published literature, here is what I did recently.
Scott Walker: Not perfect, perhaps, but more informed.
Scott Walker: Questions on this program before I go on?
Scott Walker: The four courses are taught by two instructors
Scott Walker: They are PhD students who have strong content knowledge, but no DE knowledge.
Scott Walker: That evaluator would be me.
Scott Walker: We mostly elimated the problems with the expertise-oriented approach because I know the content and the DE methods, stategies, etc.
Scott Walker: When I was given this task the mandate was vague. "Look into it" is what I was told.
Scott Walker: So, to conform to what I learned over time and the literature, I ID 3 purposes.
Scott Walker: I think that well defined purposes should drive the strategies.
Scott Walker: And that time should be spent on defining why you're doing the eval.
Scott Walker: I opted for 6 strategies.
Scott Walker: I'll go through each rapidly. Stop me if you have a question.
Scott Walker: I used the Distance Ed Learning Environments Instrument to determine the status of the environment.
Scott Walker: Here are the results.
Scott Walker: 5 is that this always happens
Scott Walker: 1 is seldom occurs
Scott Walker: There are 5 scales for the learning environment and one (enjoyment) that is an affective-scale.
Scott Walker: I surveyed all of the students in all four courses this term. And surveyed the instructors.
Scott Walker: Where the students' perceptions are greater than the instructors we need work.
Scott Walker: That's mostly on interaction here.
Scott Walker: This qued me in to take a close look at the interaction going on between students.
Scott Walker: and the instructor
Scott Walker: The nitty gritty of the DELES is elswhere. I can point you to it.
Scott Walker: Cree?
Cree Stout: Am I misreading the graph?  It appears in the interaction part, the instructor perceptions are greater than the student.  this is not what you said, I don't think
Scott Walker: CORRECT
Scott Walker: Cree is correct.
Scott Walker: It's the other way around.
Scott Walker: Where the instructor has a "higher" perception and the students a lower one, there is a point to focus on.
Scott Walker: Thanks for the correction.
Cree Stout: with you
Scott Walker: Just rattling off at the mouth here.
Thomas Smyth: At some point, please define "parity" with face-to-face classes.
Thomas Smyth: and why.
Scott Walker: Since part of the purpose is to consider the program with the face-to-face (f2f) versions of the same classes I looked at the content of each class.
Scott Walker: No. Of one class.
Scott Walker: ONe class in detail. reviewed the other 3
Scott Walker: Parity = equal
Thomas Smyth: Equal in what way?
Thomas Smyth: (I'm not being pedantic here; I am interested in why they must be equal.)
Scott Walker: Why? Part of the problem after interviewing staff was that there were different text books.
Thomas Smyth: yes, and..
Scott Walker: This prompted the question of what content is being taught in the two versions.
Thomas Smyth: ahh
Scott Walker: They should be similar just as any two f2f sections should be similar to a degree.
Scott Walker: This is according to our accreditation standards.
Thomas Smyth: so you are focusing on the content primarily?
Scott Walker: In this particular strategy, yes.
Scott Walker: In the previous it was the psychosocial environment.
Scott Walker: Got 4 more.
Thomas Smyth: OK.  There are other issues wiht trying to make ftf and online course "equivalent."
Scott Walker: There are many issues. Yes.
Scott Walker: We're looking at content. To see if what is being addressed in class x is similar to the content of class y.
Scott Walker: Here's what I found
Scott Walker: in one class.
Scott Walker: This one is in-depth.
Scott Walker: In the content part there were crucial components missing. 6 are here on the left.
Scott Walker: THis is equivellent to the "lecture" portion of the f2f class (noting there is no real "lecture" in the online version).
Cree Stout: Were theses missing or were they included?
Scott Walker: ON the right are the skills components, otherwise called the lab.
Scott Walker: Missing
Scott Walker: I've thrown them all together for the presentatation. They seperated in the paper in a table.
Scott Walker: Point is, there are 11 content areas missing from one class or another.
Scott Walker: This does not say the online class is poor, or the f2f class is better. I'm making a factual statement that they are different and they should not be to this extent.
Scott Walker: I.e. something needs fixin'
Solomon Isiorho: that is odd
Scott Walker: This if part of the management-oriented approach here.
Scott Walker: How much is one costing over the other?"
Scott Walker: If we put a full-time facutly on the online courses, the cost will increas.
Janet Groen: wow this is interesting - I think from our perspective, we are finding it more $$$ to run the DE program courses. Actually, I think we would probably reverse the numbers -
Scott Walker: Note this is a simple comparison.
Solomon Isiorho: I am suprised there is so much content diference between FTF and online section
Janet Groen: As you have indicated - the key variance is the faculty cost - almost all of our DE courses are run by full-time faculty
Scott Walker: No classroom costs are figured, no infrastructure, etc.
Janet Groen: Isnt' there extra costs imbedded in terms of faculty P.D., technical support and those kind of things?
Scott Walker: This prompts the question for this program, do we use FT faculty or not.
Scott Walker: The students are about to graduate!
Scott Walker: STudent instructors that is
Scott Walker: Benefits.
Scott Walker: Benefits and salary make them more expensive.
Scott Walker: Also, there is no lab instructor in the online versions
Scott Walker: These are 4 semester hour classes.
Cree Stout: Does this mean there is no lab?
Scott Walker: In the online version they turn in projects, but don't have a lab to attend.
Scott Walker: The instructors in the online class do it all.
Solomon Isiorho: If there is no lab then the two are not the same in my opinion
Scott Walker: Solomon, you have a valid point.
Cree Stout: Do the f2f students have the same type of projects to do in lab/
Scott Walker: Cree, the right-hand column here is lab or skill content differences.
Scott Walker: So, the answer, Cree, is yes, to a degree.
Scott Walker: I've noted that grades are the most common eval measures.
Scott Walker: Here I combined all of the grades from all of the online classes over 3 years.
Scott Walker: and all of the grades from the f2f classes over 3 years.
Cree Stout: Are the tests reasonably similar for f2f and DE?
Scott Walker: The unit of analysis on both = program grades, not indiv. class grades.
Cree Stout: o.k.
Scott Walker: Cree, they
Scott Walker: Again, Cree, they're both project oriented.
Scott Walker: So, yes. They are similar
Scott Walker: This was a bit scary to me.
Scott Walker: ONline classes in magenta, dotted line = f2f
Scott Walker: Looking a bit inverse, eh?
Scott Walker: Tells me there is a problem.
Scott Walker: They should be somewhat similar.
Scott Walker: Or at least follow similar trends
Scott Walker: This prompted a look at online grades only...no comparison.
Scott Walker: Not the lack of C' and Ds.
Scott Walker: Lots of As and bump in Fs.
Scott Walker: Several Withdrawels (W).
Thomas Smyth: Scott, surely you're controlling fro different instructors, etc.
Scott Walker: Grade inflation in this program.
Scott Walker: This is all instructors combined
Scott Walker: Simple mean
Solomon Isiorho: something to think about
Scott Walker: On this one I even removed on term where one class of 5 had all As.
Scott Walker: Again, two parts to eval.
Scott Walker: 1) measurement, 2) judgement
Scott Walker: I'm just measuring.
Scott Walker: Do students stick with it?
Scott Walker: Pretty much, yes.
Scott Walker: These are the program vs. the university stats
Solomon Isiorho: What is the total number of students involved?
Scott Walker: Solomon. Depends. Thus the percentages.
Solomon Isiorho: Just a rough estimate for the online
Scott Walker: Finally, we needed to know how the program looked in contrast to others.
Scott Walker: Solomon n=151 for the online in this program under eval.
Scott Walker: Don
Scott Walker: I don't know the university one off hand.
Solomon Isiorho: Thanks
Scott Walker: This is more oriented to the management aspect.
Scott Walker: How do we stack up?
Scott Walker: Or program is right in line with other online GIS cert. programs.
Scott Walker: Finally, (note the time), I presented this to the powers that be in terms of SWOT
Scott Walker: Strengths
Scott Walker: Weaknesses
Scott Walker: Opportunities
Scott Walker: Threats
Scott Walker: SWOT is oft used in business for quick eval, but seldom in academia. It's just a way of stating where we stand
Scott Walker: The full description of the outcomes in terms of SWOT are in the paper coming up.
Scott Walker: When the dept chair and the undergrad committee look at this program now, they know what threats it contains.
Scott Walker: Such as not being in compliance with standards perhaps.
Scott Walker: So, to wrap up.
Gundars Kaupins: Isn't SWOT used a lot for accreditation?
Scott Walker: This has been 6 strategies that resulted in providing data for the dept.
Scott Walker: To reduce the politics (theme of this conf) we can now make data-based decisions on what to do with this program.
Scott Walker: Got a few minutes for questions.
Gail Rathbun: I think that Scott's multimodal approach is very helpful. I hope that he will remember to save his chat log and send it to tohe@ipfw.edu
Scott Walker: The paper is the WALKER_S_L...there on the bottom left.
Scott Walker: Gail, thanks for the reminder.
Solomon Isiorho: Thanks Scott
Cree Stout: Scott, you did very well.  Thank you.
Janet Groen: Thanks very much Scott - most helpful!
Gail Rathbun: I want to remind everyone to please complete the evaluation form for the session.
Solomon Isiorho: where might that be?
Gail Rathbun: When you close the window to leave it should pop up.
Scott Walker: Solomon it'llcome up when you close
Solomon Isiorho: :)
Thomas Smyth: thanks and adios
Gail Rathbun: Thanks for your Qs, Thomas
Scott Walker: Okay then, thanks gang, I'm off to a f2f class then.