Thanks, Rob, for the kind words and detailed response. I hope you're still lurking, as Lloyd said you might be. I was out of town Thursday through Sunday and didn't have a chance to respond until now.
A warning to those not interested, this is a LONG post, but this is also a complex topic....
I've also received personal posts and would like to respond to both publicly....
If you mailed me directly, please read on.
SNIP
> >I find the emerging thread of iterative design (successive approximation) an intriguing subject....
We seem to have some of the terminology problems in communication (due to perhaps different backgrounds) that Rob mentions as justification for full, detailed documentation during an IMM project. Note that I parenthetically defined iterative design as successive approximation. I meant this to apply to both design and development (or production), yet Rob clearly distinguishes between the two phases. I don't think the differences were clear to either of us, nor perhaps the forum at large... hence this long post.
> I am not sure how well our model can be applied to the *real* world. > Perhaps it can, given your comments below....
Rob, please do interpret ANY of my following comments as critical. They are NOT. You have done a remarkable job of addressing many of the critical issues of IMM design and development. (I look forward to your clearing the legal hurdles so that your book be published. There's an empty spot in my bookcase waiting for it....) To the other less-experienced IMM designers, I also recommend that you carefully heed Rob's advice. It is obviously based upon lots of experience (i.e. lessons learned), and can help you avoid many of the worst pitfalls of trying to do this difficult task for a living (or even an enjoyable hobby).
I am merely trying to "push the envelope" in this professional forum and gather others' inputs or insight. I sincerely believe, given the state of today's design and development tools (4th generation tools, data-driven designs, etc.), that we are poised to make giant strides in our ability to develop very effective IMM very cost-effectively. I think that we have the tools today to iteratively develop IMM also, in addition to iteratively design it.... But, how do we do that and predict budgets and schedules?
> >As I alluded to in my previous posting, our SALES process institutes a > >traditional linear (i.e. waterfall) process of a design phase at a fixed > >cost which produces a prototype and a set of functional requirements, > >client review/signoff, then a firm fixed-price bid for developing and > >implementing the custom software package defined by the prototype and > >functional requirements.
> The particular incremental prototyping model that we use for IMM > development starts with a definition of the problem, coupled to a > feasibility study (discussed earlier), and then production proceeds through > a cycle consisting of Design, Develop, Evaluate, until the project is > finished and Implemented. > Experience has shown that this incremental prototyping model is appropriate > for the production of *quality* educational interactive multimedia. Under > this model all aspects of the project should be formatively evaluated and > revised until the project team is satisfied with their effectiveness. The > word 'quality' here may differentiate between the commercial and academic > developers (in a nice way :-).
But, is the client satisfied with the quality? The student? I admit, Rob, I'm somewhat confused here. It sounds like you iteratively prototype (Design, Develop Evaluate), then "lock in" the design prior to production. So do we, in our traditional linear or waterfall method, though I failed to describe that we typically present the prototype to the client about three times before we finish it, and "lock in" the design. Is this what you meant? I assume so, and the remainder of my comments come from that perspective. If I'm wrong again, I apologize in advance.
>Our projects cover such a wide range of > fields, and we feel that the range of topics dictates a different user > interface for each topic. You can't shoehorn drug dosage calculations and > scientific inquiry skills into a standard template. At least we don't. We > try to find the most appropriate user interface for the topic at hand, > given our particular subjective feelings about educational theory. The > commercial folks probably don't have this luxury.
We also produce a different interface, "look and feel" for each client, but I think that you may be undervaluing templates, if they are designed and built correctly, and the textual content. While I was with Allen Communication (previous job) we produced a series of 21 different courses, averaging over 10 hours each (estimated 240 hours of total contact time) for the Air Force using a total of only 24 screen templates (23 pre-defined, and one free-form). I'll admit it was for the same client (one look and feel) but for about five different end-user groups as I recall. Many of those 23 pre-defined templates dealt with multiple choice questions with four distractors, five distractors, etc. and variations of those same multiple choice distractors but with a picture (it was interactive videodisc) or graphic supporting the text. I came to the conclusion that most of the variations in screen design needed were made to allow the student an opportunity to interact -- not to present the content. This, combined with readings such as like M. David Merrill and his ID2 theory which at the time had defined some 17 or 19 (I don't honestly recall) different instructional transactions.... This train of thought, combined with the intense levels of effort required to thorougly document IMM design, led to my design of the first prototype of Designer's Edge.... A series of tested interaction templates that could be easily customized to change the "look and feel" simply by substituting backgrounds, graphics or buttons.... Then to "feed" these templates with a database of content that also contains instructions to the "code" on which template to use next and which content to present.... (This is truly not a "plug" for ANY Allen Communication product -- we have IconAuthor, Authorware, Director, and Toolbook all at my current job and use all of them selectively, but the team I've assembled uses Quest daily since the least experienced of us has over five years experience with it.) Though the current functionality of Designer's Edge bears only limited resemblance to that first crude Visual Basic prototype, I honestly think these types of tools are where the future of IMM design and development lie. Also, I'm very encouraged by the recent announcement of a version of Designer's Edge supporting Toolbook, but that's a different issue. Let's leave tools as an aside and continue on with the process and effectiveness discussion.
> However, a key point of our development methodology is to separate the > Design Process from the Production Process. Both of these processes follow > the design, develop, evaluate model described above, but the design process > has a higher weighting of the design component, while the production > process is mainly concerned with development. Evaluation is important in > both cycles. If we get it right the production process is a spiral with > very few turns.
> >> However, it clearly doesn't produce the most effective solution, in my > >>experience, because the product almost always ships with a list of > >>"deferred enhancements" that were defined as a result of user test and > >>acceptance process. Typically, only the critical "enhancements" or the > >>ones that can be implemented within the original estimated time/budget will > >>be incorporated. This negotiation process with the client at the critical, > >>late stages in the process is often quite "stormy" at times, as we much > >>reach agreement on the differences between bugs, design errors (of omission > >>or comission) or true enhancements. Unless it is a bug, it isn't free. > >>Even design errors are "joint responsibility" since the client approved the > >>design....
It seems in Rob's discussion that he also is just as concerned with "locking in" an approved design (prototype plus specifications) and content (storyboards).
> That's why we design the project as comprehensively as possible on paper, > in order to produce a quality project within budget. After a comprehensive > requirements specification of the overall structure of the program has been > produced, we create a complete storyboard of all of the content of the > program. Production should only commence after the storyboard has been > agreed on by the development team. This milestone in the process is > formalised by the signing of a contractual agreement that this stage has > been reached. The signing off procedure is an important incentive to > ensure that the design is complete, and that members of the team will not > attempt to change some components of the design at a later stage.
I think that your design/development teams and the client are also likely left with a long list of enhancements that they would like to also implement, if only there was time or money left on the project.... I always did.
> Evaluation of the design usually consists of constructive criticism and > reflection within the development team, and with other experts and > end-users.
I've also found that the "best laid plans of mice and men" and the best thought out designs still must be modified once the final products are being formatively evaluated.... Or, in military terms "no battle plan survives contact with the enemy" -- not that learners are our enemies -- they are our true customers!
> After the design has been completed, the Production Process commences. > Some aspects of production may have already been completed as part of the > prototyping, such as technically difficult programming issues. In the > production process, the resources, such as graphics and sounds are > prepared, and the major part of the programming work is done.
> >The other issue is effectiveness of the design, and the quality of the end > >products delivered by the iterative cycle. We've now developed a few > >CBT/IMM courses that are entirely data-driven, with all the content and > >selective logic being stored in a database. This allows several very > >powerful changes to the traditional, linear design and development > >approach. We can print a complete set of current, correct storyboards at > >any point and obtain client sign-off and approval. ... As long as we do > >not need to produce additional graphics or sound files, we are essentially limited in > >making revisions only by our typing speed. As an example, we can make > >significant changes during individual tryouts overnight, and have them > >available for further testing the next day. However, we can get them even > >more involved in the design process....
> Yes, the data-driven approach is really the only way to go, if you have a > choice. In current development, we develop storyboards on a wordprocessor. > Text is then automatically 'sucked in' to the authoring package. The use > of a wordprocessor simplifies revision of the storyboard, and minimises > spelling errors. Does anyone know a programmer who can spell? :-)
Instead of using a word processor, we're using an ODBC database (created in Access) with a Quest compiled executable that reads that same database and presents it to the learner. I'll admit that we've not spent the time to try mapping the entire Designer's Edge database to use it as a storyboarding tool, though we've used it to prepare the first drafts of Design Documents (i.e., specifications in Rob's terms) and have found the large library of Quest templates valuable. Supposedly, Allen will be "opening up" the database specification for Designer's Edge in the next release, so that the content would "automatically" flow from the storyboards into the Quest lessons, with the templates in the library (.QLB file) serving as the "pipe" between the two. We're doing something very similar, using Access and ODBC, with a much "cruder" Access interface to those storyboards (though it does support the all-important spell-checking!). Some of our smaller IMM courses contain only six or seven different interaction templates....
> >We've successfully used the technique of one student reviewing the course > >on one machine and the ID making the desired changes on an adjacent machine > >in "real-time" and presenting them back to the reviewer to critique. We > >try to do this with at least three "novices" who've never been exposed to > >the content and three "subject matter experts." The content, especially > >for learner prompts and feedback, then has been re-designed and refined > >"on-the-fly" based upon live input from the end-users. (Note that I said > >the ID, not a programmer!) This has dramatically improved the "richness" > >of the dialogue between the learner and the computer-based learning > >experience. We now can capture and address subtle distinctions never > >before possible, add elaborations where proven needed, etc. We can correct > >the causes of misunderstandings with the feedback to user mistakes, > >distinguish and award "partial credit" based upon typical users' > >explanations for their actions, etc.
> However, I would get these people involved at the prototype stage, rather > than during the final debugging.
In an interative development (as opposed to design) approach, this isn't final debugging, merely refinement of what was planned. Using our database tools, we can present the content (at its current state of development) in the actual interface of the final course at any time. We can perform what the top-down structure programming approach calls "structured walk-throughs" at any time. We can present the storyboards on-screen, including a textual definition of the graphics or media to be produced. The text content is as it will appear in the final product unless someone's desire to change it. If so, they edit the database and "Voila," the new text appears (both on screen and on any printed storyboard from the database). We can do this incrementally, for one lesson, one module or as much as we have defined at that point. All the interactions are already there, with the exception of "hot spot" graphics, there the user interacts with the graphic itself. (We try to limit this, if possible, to only simulation strategies.) The more complex media elements (especially animations and hot-spot graphics) may need to be "manually storyboarded" by sketching them on paper, and getting client review and signoff of these sketches prior to production.
Once the content has been thoroughly reviewed as story-boards, we go off and produce the media elements and incorporate them. Then we have the client review the course again. Now all the PLANNED content is included and is reviewed. If changes are needed (and they always are), we can make them on the spot as long as the graphics or audio elements do not need to be changed. We can change any text almost immediately. We can fine-tune the interactions by modifying the Quest templates (examples: Adding a "time-out" to every quiz question, adding "a second try" with a "Try again" feedback message before we disclose the correct answer, etc.). This typically requires editing no more than two or three screen templates in Quest, yet the results are evident throughout the entire course. We often have learners in during these "Alpha" tests, and significantly refine the course (minor design "tweaks" and sometimes major content changes) based upon their input. This can happen even though the same students reviewed and approved the "on-screen storyboards" earlier. This is likely due to at least two reasons. Until the entire course is developed, you simply cannot get a good sense of continuity, depth of content coverage, the level of prompting/feedback needed to make the human/computer-based tutor dialogue more realistic, pacing, etc. There's also the issue that some people cannot "visualize" well, i.e., they cannot "picture in their minds" what the final product will look like. They will withhold comment until they really see it.... Then, on to Beta test (i.e. small-group tryouts) for even more refinement....
ASIDE #1: We've found that most "hot spot" graphics can be incorporated in a database simply by defining the XY coordinates of each of the hotspots in the database and otherwise treating them like a multiple-choice question. Only irregularly-shaped hotspots pose a problem. When we encounter them, we try for "maximum re-use" of that graphic, since it must be defined as a unique Quest template, though the content in the database can allow multiple uses of it for exploratory learning, quizzing, branching like a menu, etc.
> Summary > ------- > The key points made in this post to efficiently produce educationally > effective IMM materials are shown below: > * Use an incremental prototyping model, but > * Separate design from production > * Design the project as comprehensively as possible on paper through a: > - thorough Requirements Specification > - comprehensive storyboard > * Use prototypes where appropriate > * Document all aspects of the development > > However, Kent, I am not sure that any of this will help resolve your problem.
Though there were a few valuable insights, Rob, especially regarding teams and relationships, the process discussion didn't help much. Been there, done that, got that T-shirt! I'm trying to go further. Any further comments? Anyone else? Who else is trying to use databases directly? IconAuthor has supported them for quite a while. Toolbook support came somewhat later, and Authorware does in it's last release.
ASIDE #2 The linking of Quest and Designer's Edge is where my original prototype and the current, commercial product differ most. The original prototype contained this automated link, the current version does not. The original prototype was designed from the "authoring/production" perspective, designing instructional interactions first, and then working backwards into the design steps of storyboarding and flowcharting, with linking only back to the point of instructional objectives, where direct correlations broke down. The current version begins at step one of the design process (defining the mission or goals), proceeds to audience analysis, then objectives, etc., with loose correlations throughout, yet lacks the automated link of storyboards to the final, authored lesson. Why did I work backwards? In my experience, storyboarding was taking 25-30% of the total effort, with authoring another 25% or so, and debugging another 10-15%. Linking the storyboard automatically to automatically become the authored product theoretically yields at least a 25% time savings, with significantly decreased bugs to be found and corrected in the test/debug steps.... It also allowed the content expert to do much of the final development, just as Rob uses them to produce storyboards. Programmers were needed only to develop the databases, interaction templates, and the links between the two....
ASIDE #3 We're now developing a Visual Basic "quiz engine" that's entirely database driven. Why VB? It responds to the user quickly, is more "database friendly" and has great add-in reporting capabilities such as Crystal Reports. There's no reason why Borland's Delphi (for those who know Pascal) or the IBM Visual Age family of systems couldn't be used similarly (choice of Smalltalk, C, or Basic languages). PowerSoft's PowerBuilder also has the same potential, perhaps with less robust built-in scripting (I'm not sure) and perhaps several more tools that I'm not familiar with. Hints:
(1) Focus on designing typical instructional interactions -- don't focus on content first.
(2) Treat graphics as either interface elements or just another way of representing content.
(3) If it's content, define it or a cross-reference to the final picture, for example, in a database.
(4) Define the instructional sequencing also in the database....