Author - Hans Westman, Chair, Media Arts & Animation, The Art Institute of Pittsburgh

Methods for assessment have become increasingly important in education, especially in regards to complying with accreditation guidelines. The question is, how can this be done effectively at a school whose core curriculum focuses on teaching art? For a department that teaches courses through both campus and online modalities, a web-driven process has become an effective tool, allowing campus and online students to be reviewed side by side by both campus and online faculty alike.

Motivation

So where does one begin? First, it has to be established that there is a need as well as a reason to initiate an assessment process. For the Media Arts & Animation (MAA) Department at The Art Institute of Pittsburgh, it has been a natural evolution, triggered by internal prompts.

Not only do students spend 12 quarters attending classes to earn a Bachelor if Science degree in Media Arts & Animation, they are also preparing to take on roles as professionals in a related field. Once the Career Services department starts assisting the graduate to find placement, it is quickly ascertained if students have acquired the necessary skills to compete. For that reason alone, the need to evaluate student skills throughout their education became clear, serving the dual purpose of holding the student as well as the institution accountable to meet and, hopefully, exceed program expectations.  At The Art Institute of Pittsburgh, the challenge to implement a manageable process to evaluate skills of 20 to 80 students during any given quarter has been in development and has changed in form over the last 10 years.

Back Story

To begin with, around 1997 the MAA department initiated a faculty panel review to evaluate student exit portfolios that were comprised of a demo reel and a flat-work portfolio. A standard form was created for each faculty member to fill out while evaluating the student’s work.

Around 2000, an additional review was introduced to evaluate students’ 3D modeling, texturing, lighting, rigging and animation skills. This took place in the 6th quarter, which is midway through the program. The “3D Review” was assigned a faculty member to meet 1-on-1 with the student. A similar form to fill out was also created for this review.

A “2D Review” was added in 2002. Taking place in the 4th quarter,  a faculty was assigned to meet with each student to evaluate their traditional drawing skills. Here, too, a similar form was developed for the written evaluation.

2006 the Exit Portfolio review was reassessed and reassigned as a faculty panel that evaluated the student’s full set of traditional and digital skills they acquired during their studies. This “Pre-Portfolio” review has become significantly different from the other two as the primary objective is to assess the program and how effectively skills are taught. The secondary objective is to identify student's strengths, weaknesses and what their portfolio presentation's focus should be.


First Pre-Portfolio program/student assessment form

The reviews have evolved into a process of collecting quantitative data that shows trends in both student and program performance. With increased volume, the time required to hold the reviews as well as collect data has presented new challenges. How could this be simplified to ensure consistency in reviewing work and collecting evaluations for ongoing assessment purposes?

Web Based Solution

Leaving a literal paper trail did not seem optimal to the process. It was clear that an electronic collection method might offer a clutter free solution.  Fall 2007, an Internet based survey format was implemented to collect faculty evaluations during Pre-Portfolio review sessions and has been used every quarter since then to collect and store evaluations "online". This has simplified the mechanics involved in the assessment process considerably.


First web based Pre-Portfolio Review form (Survey Monkey)

Virtual Reviews

The Pre-Portfolio review is department driven, meaning the faculty deem the review important enough to meet once a quarter outside of their regular teaching schedule to look at student work, speak with the students individually and document their evaluations. Adapting a database driven method for collecting and analyzing reviews has made the whole process much more manageable, but the faculty’s ability to meet at the times scheduled as well as donate the time required, quarter after quarter, has not been consistent . Ideally, the department preference would be a 100% faculty attendance, but the expectation has been that there is at least 50% of the faculty in attendance at every Pre-Portfolio review. This made the need for additional refinements to the process increasingly apparent.

In the beginning phase, students were required to bring a DVD for the skills review, the “interview” and evaluation. Faculty reviews were already being accessed and collected digitally through an online survey form. Moving from a DVD presentation to a web based Flash video file (.flv) presentation was a natural progression, which the MAA department implemented in 2008. This has allowed pre-screening, or “real-time” screening and reviewing of student work from virtually anywhere, giving faculty the flexibility to participate either on site or from a remote location. To support this, students were asked to submit their files prior to the review in order to create the access web page, which was configured manually. By design, the online survey was also incorporated, giving quick access to the class list of students, their portfolios and the review form, all from the same page. Once faculty became comfortable with the online review process, both the 2D and 3D reviews were formatted in the same way. Up to this point, the concern was to ensure continuity in the process of reviewing and assessing student work, which had become an integral part of the program.


First web based “3 in 1” Pre-Portfolio Review. Campus and Online enrolled students for the quarter are listed in the left frame. Student names are links to Flash video file (.flv) presentations that appear in the main frame. The web based assessment form is accessed in the bottom frame.
Example of student 2D animation work displayed during a review session, student - Gregory Whetstone

“3 in 1” Dynamic Website and Volume Management

To handle the volume of student work more efficiently, the html based web page has been updated as a “self formatting” php database driven solution, consolidated the three web based review pages into 1 and allowing easy access to any 2D, 3D or Pre-Portfolio review from any given quarter from the same site.  Students now upload their work themselves, which means “cutting out the middle man” through an automated process that updates web page content in real-time.



 Using php scripting, a database driven version of the 3 in 1 solution was developed, enabling the reviewer to retrieve any archived student work by choosing the class, quarter and year in the left frame, all from the same site.




Example of 3D student work displayed during a Pre-Portfolio Review session.
Student – Owen Carson

Benefits

Nine years ago, MAA had boxes brimming with student portfolios turned in on VHS cassettes. There are filing cabinets stuffed with paper evaluations. I have a bookcase in my office with shelves packed with DVDs of past MAA graduate portfolios. Now students submit digital versions of their portfolios directly to a server, categorizing their work to a search-able database in real time. Anywhere between 30 to 100 students can graduate from the MAA campus division every year. As the online division student population grows, the number of MAA students graduating will increase. They, too, are required to submit their work digitally. Add to that the work campus and online students submit for the 2D, 3D and pre-portfolio reviews. Multiply that with 10 years of students submitting their work and you get an idea of the volume we are talking about. With each submission properly categorized, digital examples of student work can be retrieved though a single web based source at any time. Whether for student or program assessment, this enables a constructive academic dialog through easy access of student work and suggests benefits that will be easier to define in the years that follow.

Conclusion

Moving to a digitally based method to collect, categorize and evaluate student art work greatly simplifies the actual assessment process. However, the "mechanism" has no purposed if it is not used. The students must still submit their work and faculty must still perform the evaluations and we continue to identify what works well and what requires improvement. Already there has been evidence that students viewing each others' work has resulted in improvements in the examples submitted by students. The reviews have also triggered faculty discussions about what can be done to improve program content and renew program objectives and direction overall.

Acknowledgments

The framework on which I developed the concept for this web based student review process is founded on the collaboration of the Campus Division Media Arts & Animation faculty at The Art Institute of Pittsburgh, which started prior my arrival in 2001. MAA faculty: Rick Catizone, Sunil Ketty, Angela Love, Francis Pionati, Mike Schwab and Jeff Zehner.

Once the look and functionality of the 3 in 1 webpage solution was established, the next phase was to create a dynamic solution. This was accomplished with the assistance of Web Design student Jason Katke, who after a series of meetings was able to successfully implement a database driven solution while preserving the original conceptual design.

About the Author:

Hans Westman

has been active as an artist and art educator in both Sweden and the U.S from 1976 on. He discovered animation as an art form in the late ‘70s and started experimenting with animation through the support of Filmverkstan in Stockholm, Sweden. In 1987 he was exposed to "computer manipulated imagery" and has been working with the medium ever since. He is currently the Academic Department Chair for the Media Arts & Animation Department at The Art Institute of Pittsburgh.


Contact information:
email: hans_westman@siggraph.org
Tel: +1 (412) 291-6409