ITFORUM PAPER
MARCH 1997
Ian Hart
University of Hong Kong

 

CONTENTS 

 

 

NUD*IST 

NUD*IST (Non-numeric, Unstructured Data - Indexing, Searching & Theorizing) , is a qualitative research tool which runs under both Mac and Windows environments, published by Qualitative Solutions & Research Pty. Ltd., 2 Research Avenue, La Trobe University, Vic. Australia 3083. Version 4.0 has just been released.

 

Preliminary NUD*IST tree dervived from the paper

ROOT

(1)Base Data

(1 1)Contributors

(1 1 1)IanPaper

(1 1 2)Foshay debate

(1 2)Case studies

(1 2 1)CS#1

(1 2 2)CS#2

(1 2 3)CS#3

(2)Paradigms

(2 1)Positivist

(2 1 1)Behaviorist

(2 1 2)Scientific

(2 2)Interpretivist

(2 2 1)Ecological

(2 2 2)Postmodern

(2 2 3)Artistic

(2 2 4)Realworld

(2 3)Purpose of the research

(2 3 1) Pure research

(2 3 2)Applied research

(2 3 3)Action research (change)

(3)Methods

(3 1)Analytic (quantitative)

(3 1 1)Experimental

(3 1 1 1)Comparison studies

(3 1 1 2)Hypothesis testing

(3 1 2)Statistical (surveys, etc)

(3 2)Mixture of methods

(3 2 1)Evaluative studies

(3 2 2)Prototyping

(3 3)Systemic (qualitative)

(3 3 1)Ethnographic

(3 3 2)Data-driven

(3 3 2 1)Recording data

(3 3 2 2)Categorizing & indexing

(3 3 2 3)Storing & sorting

(3 3 2 4)Exploring

(3 3 3)Action research

(3 3 4) - empty node -

(3 3 5)Computer tools

(3 3 5 1)NUD*IST

(4)Outcomes

(4 1)Quantitative

(4 1 1)Proof of hypothesis

(4 1 2)Statistical figures

(4 1 3)Graphs

(4 2)Qualitative

(4 2 1)Categories

(4 2 2)Theories

(4 2 3)Graphics, diagrams, mind maps, etc.

(4 3)Recommendations

(5)Issues

(5 1)Validity

(5 1 1)Internal

(5 1 2)External

(5 1 3)Triangulation

(5 2)Laboratory or life?

(5 2 1)Laboratory

(5 2 2)Life

(5 3)Complexity

(6)References

 

Free Nodes

(F1) Metaphors

 

MENU


INDEXING THE PAPER

The paper has been indexed according to the NUD*IST categories in the tree above.

To simplify the presentation for the purposes of this esercise, the basic text unit is the SECTION, though in my project data base, the text is indexed by LINE.
 

INDEX 
TEXT SECTIONS 
(1 1 1) 

(F1) 

Between the idea and the reality...: the case for qualitative research 


Between the idea 
And the reality 
Between the motion 
And the act 
Falls the shadow 

- T.S.Eliot "The Hollow Men"

 
(1 1 1)  RESEARCH QUESTION #1: Where lies the shadow? 

Do you think it is drawing too long a semiotic bow to suggest that "the shadow" might represent the outer darkness into which academic "hollow men" are cast by the gatekeepers of academic life if they do not produce "the right kind of research"? You do? Just close your eyes, lie back and relax and maybe it will all make sense in the end...

 

(1 1 1) 

(2 1 1)

(2 2 1)

(2 2 2)

(2 2 3)

(F1) 

DATA: The importance of titles 

This wasn't the original title of this piece. I first though of calling it "Exploring the slimy swamp" after a memorable ecological metaphor from Schon (1987) writing about what he called the "real-life problems of the classroom eco-system." Then I thought of "Opening the black box" which is an equally significant metaphor because of its historical association with the operant conditioning model of learning which disavowed the thought process "because it cannot be observed." An article by Robert Kozma (1994) also suggested a sexy post-modern title: "Photographing the whirlwind" when he characterized the positivist approach to the study of learning as similar to examining the effects of a tornado by taking photographs before and after the event: "the photographs enable us to assess the extent of the damage but not the process by which the damage was wrought" (p.10).

But in the end I plumped for The Bishop's famous lines because the poet's insight is possibly the highest achievement of any research. (And of course it adds a pretentious literary "frisson" to a potentially dry topic.) 

 

(1 1 1) 

(2 2)

(3 1)

(3 2) 

MEMO: The substance 

This essay is not a polemic in praise of humanistic, ethnographic, holistic, phenomenographic, artistic, warm and fuzzy, trendy, "soft" or postmodern research methods. My purpose here is to make a case for considering non-quantitative methods as an indispensible monkey-wrench in the research toolbox; and to demonstrate that the outcomes of qualitative investigation can be equally productive, though often differently valid, to the selective, positivist, analytico-deductive, scientific, statistical research methods sanctioned by the gatekeepers of our discipline.

 

(1 1 1) 

(3 1 1 1) 

DATA: Letter from grandma 

My grandmother just sent me a newspaper clipping in which it is reported that a gentleman from California taught half his class using lectures and tutorials and the other half using the Internet: on the final test the students on the Internet course did better than the classroom group. I think there was a number somewhere which proved this. "You must be so pleased, dear," wrote granny, "to know that you have finally been proved right." OK, I confess, I've been delivering course material via the Internet since 1994 but until now it has been a guilty secret between me and the students, and they said they liked it so I kept going in spite of the lack of research evidence. (By the way, granny asked me to thank the nice California man for his Christmas present of a laser-driven virtual egg-sucker, "The piano doesn't wobble now," she says.)

 

(1 1 1) 

(2)

(3 1)

(3 2)

(3 3) 

DATA: Definitions 

Let us agree, first of all, that the terms "quantitative" and "qualitative" are ambiguous: they are commonly used for both the contrasting paradigms and the methods associated with them. However the contrasting paradigms could equally well employ either or both quantitative and qualitative methods. Although adherents of the quantitative paradigm are more likely to use experimental and quasi-experimental tools, while qualitative researchers are more likely to employ more descriptive techniques. (Fetterman,1988)

Salomon (1993) contrasts "analytic" and "systemic" approaches to research design. The goal of analytic research is to manipulate and control situations so as to increase internal validity and isolate specific causal mechanisms and processes; whereas systemic research is based on the assumption that "each event, component, or action in the classroom has the potential of affecting the classroom as a whole." He proposed that ethnographic or naturalistic methods, such as long-term observation, interviews and artifact analysis provide a richness of detail about the social processes within which cognition is embedded.

The two approaches are by no means exclusive: they can co-exist perfectly well and the methodologies can complement one-another.

 

(1 1 1) 

(1 2 1)

(2 1 1)

(3 1 1)

(3 1 1 2)

(3 2)

(3 3 1)

(4 1 1)

(4 1 2)

(4 3)

(5 1)

(5 1 3)

(5 3) 

CASE STUDY 1: Race, memory and culture 

An article in the Australian Journal of Psychology by Kearins (1986) reports an investigation of Lockard's (1971) hypothesis that human populations are shaped by natural selection to fit a particular ecological niche. Why is it, asks Lockard, that Aboriginal people do so poorly on IQ tests, yet possess such remarkable survival skills? Perhaps there are different, genetically determined patterns of intelligence.

Kearins first conducted a series of carefully controlled experiments to establish whether the IQ hypothesis was valid. She used a standardized "visual memory test" in which groups of suburban European and Western Desert Aboriginal children were given 30 seconds to memorize a set of objects presented on a tray, then to recall them: the Aboriginal children scored significantly worse at this test than the Europeans. She then administered a set of "spatial relocation tests" in which objects which could not always be differentiated by name (rocks, leaves, twigs) were arranged on a grid and presented for 30 seconds; they were then disarrayed and the children were asked to replace them in the original positions. The Aboriginal children scored significantly better than the Europeans at these tasks. Her studies confirmed Lockhard's hypothesis that the Aboriginal children had differently developed visual memory skills, but the statistically significant results provided no clues to the underlying causes of the difference.

Kearins notes Rowe's (1985) observation that, "Intelligence does not operate in a vacuum... if our assessment of intelligence is to increase in validity... we shall have to observe the individual's functioning in real life, rather than in a laboratory or in a standardized testing situation" (p.10).

Kearins' recognized serious problems in trying to conduct experimental studies in a cross-cultural situation. Her follow-up research used a more ethnographic methodology, consisting of long-term observation, interviews with teachers, artifact collection, etc. The outcomes of these studies were even more fascinating: e.g. she observed that Aboriginal babies' heads are not supported when they are carried, forcing them to develop their neck muscles, and consequently their visual acuity, much earlier than European children; additionally, Aboriginal value systems have little or no sense of "ownership" and the memorization of objects by name ("I want that") is of far less importance to Aboriginal than to European children, whereas the need to observe and memorize spatial relationships in a desert landscape is an essential navigation skill.

Most fascinating of all, "White children who performed well on the visual spatial memory test were... derogated by teachers, who considered them lazy, inattentive underachievers - views not supported by school records. It is possible that their cognitive strategies did not fit teacher expectations..." (p.212)

Kearins' work is an example of the complexity which arises once you begin to delve into Schon's swamp or to peep beneath the lid of the Pandora's box opened by qualitative research. And complexity does not sit comfortably with those who search for simple, definitive answers within what Biggs (1995) terms "the whistle-clean, four-square symmetry of the psycho-lab" (p.50). 

(1 1 1) 

(2)

(2 1)

(2 1 2)

(2 2)

(5 1 1)

(5 2)

(5 2 1) 

MEMO: Lab vs. life 

The "laboratory vs. life" debate is controversial in the social sciences, particularly psychology, a discipline which has long craved scientific respectability. Salomon (1993) quotes a debate in the field of memory research which stemmed from a chapter by Neisser (1978) in "Practical Aspects of Memory" about the validity and scientific merits of laboratory vs. real-life research. A precursor to the educational technology debate provoked by Clark (1983), Neisser had attacked the laboratory approach that emphasizes internal validity over external validity, charging that nothing interesting or important had resulted from roughly 100 years of effort in the laboratory. Ten years later, Banaji & Crowder (1989) were still arguing that "the more complex a phenomenon, the greater need to study it under controlled conditions, and the less it ought to be studied in its natural complexity" (p.1192)

The fundamental difference between quantitative and qualitative positions is based on philosophical and epistemological, not methodological, grounds. The paradigms are derived from the philosophical positions of positivism/ objectivism on the one hand and phenomenology on the other. Typically, positivists search for social facts apart from the subjective perceptions of individuals; and by contrast, phenomenologically oriented researchers seek to understand human behaviour from the "insider's" perspective. A qualitative researcher argues that what people believe to be true is more important than any objective reality.

 

(1 1 1) 

(1 2 2)

(3 1 2)

(3 3 1)

(3 3 2)

(4 2 2) 

CASE STUDY 2: Dried fish 

My colleague Dr. Daniel Lam Tai Pong <tplam@hku.hk>, of the University of Hong Kong's General Practice Unit, is researching the the health and lifestyle beliefs of the Hong Kong fishing community, who share a strong ethnic identity. They call themselves "Seui Song" meaning "people who live on the water" and are thought to be the original inhabitants of these islands.

Epidemiological studies have clearly demonstrated a link between the consumption of dried fish and nasopharyngeal cancer and Hong Kong has abnormally high levels of this form of cancer due to the popularity of the food in the southern, coastal diet. This link could only have been established by using large statistical samples and sophisticated quantitative methods. Since the link became widely known, dried fish consumption and nasopharyngeal cancer has begun to fall - everywhere but in the tightly knit fishing community.

In his harbourside clinic Daniel is conducting open-ended interviews with fishing families to see how lifestyle factors contribute to health problems. He has found that the Seui Song are quite aware of the link between their traditional food and cancer, but they attribute it to the commercial drying process "which uses chemicals"; in contrast to their own "natural" product, which they continue to consume with confidence. (Not true, unfortunately - the carcinogen is related to the drying process.) They justify their "traditional" lifestyle choice by quoting the slogans of the health food industry.

People act on what they believe.

 

(1 1 1) 

(2 2)

(2 2 3)

(3 1 1 1) 

MEMO: Personal reminiscences 

My approach to research could also be described as a "lifestyle choice" based on my own background and beliefs. I have worked for over 20 years as a documentary film maker: the highly subjective and most unscientific art of recording, structuring and interpreting other people's lives on film and videotape. My heroes were Dziga-Vertov, Flaherty, Rouche, Wiseman, the Maysles bros., Levy-Strauss...

When I gave up full-time film-making in the late 1970s to work in a university, which required me to do research into educational technology, I stood aghast before the body of media comparison studies which uniformly regard the central participant in the learning process, the learner, as no more than a "black box." I wrote a rather bitter polemic against it (Hart, 1981) in the Journal of Educational Television (still quoted back to me at conferences as a source of embarrassment.) But at the time, I recall seeing Richard Clark's notorious "trucks and nutrition" metaphor (1983) as a personal vindication of my position.

 

(1 1 1) 

(3 1 2)

 

DATA: The million monkeys theory 

There's a statistical theory that if you gave a million monkeys typewriters and set them to work, they'd eventually come up with the complete works of Shakespeare. Thanks to the Internet, we now know this isn't true.

 

(1 1 1) 

(2 2)

(3 3 2)

(3 3 2 2)

(3 3 2 3)

(3 3 2 4)

(4 2 1)

(4 2 2)

(6) 

DATA: What qualitative researchers actually do 

Qualitative research involves the gradual development of ideas about data and the exploration of these ideas. Sometimes the project begins with descriptive categories, derived from research or intuition; more often categories are derived from the data during the project and linked in ways that describe the data. New theories are constructed and tested by exploring their links with data.

Researchers normally use some or all of the following methods:

  • creating categories for thinking about the data, from prior theory and/or ideas emerging from the data; 
  • indexing or coding segments of documents at these index categories so all material about a category can be retrieved; 
  • storing factual information about documents, cases, people and sites; 
  • recording ideas about indexing categories and developing these as understanding grows; 
  • using index references for bringing together linked passages of text for interpretation, detailed description, theory testing or analysis; 
  • sorting and re-sorting data to locate patterns; 
  • expressing, exploring, testing and validating theories about the data. 
(Fielding & Lee, 1991; Miles & Huberman, 1994) 

Qualitative researchers typically collect text material, divide it up by content and file it under descriptive headings. The same piece of information may need to be cross-indexed under a number of headings and before computer programs enabled this to be accomplished electronically it was done with the aid of photocopier, scissors, adhesive tape, manila folders and a cataloguing system based on library cards and memo paper.

The test of the ultimate conclusion is to see how elegantly and methodically the evidence was shaped into the conclusion, how the conclusion was coaxed (never forced) to "emerge" from the data, "how evidence and grand account form a well-connected, seamless web of belief that illuminates and enriches our perceptions and understanding of phenomena we see every day. To be credible, the report must show these processes in action, and demonstrate how the conclusions were reached." (Richards & Richards, 1992)

There are many texts that describe and justify versions of the methodology in detail, e.g. Burgess (1984), Strauss (1987; 1990). The point to note here is that data-driven research is widely practised in many fields, including educational research, is essential to many research problems, and is not hypothesis-driven

 

(1 1 1) 

(1 2 3)

(2 2 4)

(2 3 2)

(2 3 3)

3 3 2 1)

(3 3 2 2)

(3 3 2 4)

(3 3 3)

(3 3 5 1)

(4 2 2)

(4 3) 

CASE STUDY 3: Learners as designers 

In 1993 I was invited to collaborate on a research project to establish whether a constructivist, problem-based approach to the teaching of Architecture was as valid as the conventional lecture-assignment method. Validation of teaching methods is important for professional accreditation and funds were available for a study.

It was proposed that we divide the class into two: one half would take the lectures, the other half would be thrown into the deep end of the computer lab to fend for themselves... (sounds familiar?). I managed to persuade the Architecture Department that this neo-scientific paradigm had been discredited for over 10 years and there were other, equally sound, methods of validating the new curriculum. Why not use Action Research, I suggested, which is a perfectly acceptable model? (I liked Rob Foshay's distinction between "medical research" and "clinical practice" in last month's paper.) So, for my pains, I was given the direction of the project.

The course we chose was Building Systems which covers issues of construction materials, maintenance and management. Students were required to work collaboratively on real-life problems which required them to build three-dimensional computer models of buildings. They had to master very complex modelling software (on SGI workstations) and to apply it to existing structures in Hong Kong. Their final presentation was to be a multimedia report explaining these issues to others. (URL below)

My job was to track their learning and produce a document at the end of the term which could be used in the accreditation process.

I instinctively began with the methodology of the documentary film maker: record everything without discrimination, then try to make sense of it through reflection at the "editing" stage. We rigged up a video splitting device which enabled us to record both the student and the computer screen as they worked; we conducted regular interviews with the students, individually and in groups; we gave them standardized tests; we asked them to draw concept maps and we generated Pathfinder Nets; and we systematically collected their work. We also provided feedback - both to the teaching staff and to the students themselves - according to the action research "spiral" (act - observe - reflect). For example, when our concept maps clearly demonstrated that the reason the students were losing data was that they had inaccurate mental models of the network file structure, we recommended both adding a self-paced module on network computing to the resource base for the course and simplifying the file system.

The data was indexed and analyzed using the NUD*IST software package* - a sophisticated and flexible qualitative research tool developed at La Trobe University in Melbourne. This too was a cyclic process: the indexing classifications were first developed using a grounded theory approach and were then fed back into our research methodology and further refined by an iterative process. This is not as difficult as it sounds with NUD.IST's flexibility and memo facility.

The project ran for 18 months. You will not be surprised that we did not feel able to conclude there was "no significant difference" between the old lecture method and the project method - although this was what the accreditation body wanted to hear. We concluded that the course was a substantial improvement on the lecture method. We were able to demonstrate that

  • the problem-based, computer-intensive curriculum promoted a "deep" approach to learning, 
  • students valued the situated nature of the task, 
  • for the more successful students, the computer system was a cognitive tool, which the they manipulated consciously, and 
  • the outcomes (the projects) demonstrated the achievement of complex cognitive constructions 
We were also able to point to aspects of learning which needed to be more strongly reinforced, such as:
  • metacognition, 
  • distributed cognition, and 
  • contextual (in contrast to declarative and procedural) knowledge. 
 
(1 1 1) 

(4 2 3) 

DATA: Building Systems URL 

Check out one of the projects at the following URL

http://arch.hku.hk/projects/bsys1/buildingServices/servicemenu/mainmenu.html 

 

(1 1 1) 

(2 1 2)

(5 3)

(F 1) 

DATA: Isaac Asimov 

Isaac Asimov, in his great "Foundation" series of novels, devised the ultimate quantitative science of "psychohistory" by which Harry Seldon was able to use statistical methods to predict the future of the galaxy for a thousand years and build a "Foundation" on a distant planet which would eventually be ready to step in and save humanity from itself. Throughout the first two novels it appears to work remarkably well, but in the final episode it is revealed that a secret "Second Foundation" of powerful thinkers and philosophers has been in the background all the time to nudge things back on track when the mathematics of psychohistory went astray.

 

(1 1 1) 

(2)

(2 2 3)

(F 1) 

MEMO: The end of all our exploring 

This is a complex argument and I've tried to tackle it by providing you with some unstructured data, and making the optimistic assumption that insights may emerge from the next week's discussion... I've also carefully avoided committing myself on contentious issues such as internal vs. external validity, the merits and restrictions of comparative methodologies, empiricism vs. connoisseurship, etc. No doubt somebody will be brave enough to make an unequivocal statement.

So let us conclude with this: if it is the purpose of Science to explain what it happening, it is up to the poets and philosophers (and maybe the qualitative researchers) to explain why.

 

And the end of all our exploring 
Will be to arrive where we started 
And know the place for the first time 
- T.S.Eliot "Four Quartets" Little Gidding
MENU 
EXTRA DATA: From the Foshay debate 
(1 1 2) 

(2 1 2)

(3 3 3) 

On 29-2-97 the the qualitative-quantitative argument reared its head: 

>>Concerning "scientifically proven principles": Since it is true that there are few scientifically proven facts to rely on when it comes to instructional design (David Merrill may disagree), these become very important. Consider this pyramidal paradigm of instruction: Instruction ought to be based on scientifically proven facts. As we get further from the base, we move more towards the point, which represents pure speculation. In-between are theories and hypotheses based on varying degrees of scientific evidence. Most instruction should be designed near the base of the pyramid. Experimentation takes place nearer the tip of the pyramid.

>>Data collection, however, works in just the opposite direction, much like a funnel. The more speculative the instructional process, the more data must be collected, and the more scientifically-based the instruction, the less data must be collected, at least in terms of scientific evidence for the process. The funnel paradigm is used here because all data is funneled into developing more scientific evidence.

 

(1 1 2) 

(2 2 4)

(3 2 2)

(5 3) 

Rob Foshay replied: 

>>My only disagreement would be that since real-world instructional problems are far more complex than our theories are capable of explaining, a complete instructional solution for a real-world education/training need will always involve a combination of formal design knowledge and gut feel intuition. We don't have the option of designing only what we really know how to. Example: when we build PLATO courseware, we have all kinds of design standards. To the extent possible, they are based on my judgement of the weight of research evidence and theory in ID and other fields (e.g., interface design). But there's lots that in there just because we think it should be, and it made sense at the time. Because of this, I have a basic principle in the application of our design standards: you can violate any design principle, if you prototype it and try it on some actual t-pop members using accepted methodology, and your design works.

MENU 
(6) 

NOT YET INDEXED 

REFERENCES 

Asimov, Isaac: The "Foundation" series of novels includes "Foundation" (1951), "Foundation and Empire" (1952), "Second Foundation" (1953). The Foundation and Robot novels were eventually brought together in "Foundation's Edge" (1982) and "Foundation and Earth" (1983)

Banaji, M. R., & Crowder, R. G. (1989). The bankruptcy of everyday memory. American Psychologist, 44, 1185-1193.

Biggs, J. B. (1995). Quality in education: A perspective from learning research and theory. In P.-k. Siu & T.-k. P. Tam (Eds.), Quality in education: Insights from different perspectives (pp. 50-69). Hong Kong: Hong Kong Educational Research Association.

Burgess, R. G. (1984). In the field: an introduction to field research. London: Allen & Unwin.

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459. 

Fetterman, D. M. (Ed.) (1988). Qualitative approaches to evaluation in education . New York: Praeger.

Fielding, N. G., & Lee, R. (Eds.). (1991). Using computers in qualitative analysis. Berkeley: Sage.

Hart, I. (1981) Educational television - the gulf between researchers and producers, Journal of Educational Television, 8, 91-98

Kearins, J. (1986) Visual spatial memory in Aboriginal and white Australian children, Australian Journal of Psychology, 38(3) 203-214

Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research & Development, 42(2), 7-19.

Lockard, R. B. (1971) Reflections on the fall of comparative psychology - is there a message for us all? American Psychologist, 26, 1680179

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: an expanded sourcebook. (2 ed.). Thousand Oaks, Ca: Sage.

Neisser, U. (1978). Memory: what are the important questions? In M. M. Gruneberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory (pp. 3-24). San Diego, CA: Academic Press.

Richards, T., & Richards, L. (1992, ). Qualitative computing: making data work. Paper presented at the International Conference of the Australian Evaluation Society, Melbourne.

Rowe, H. (1985) So intelligence tests don't work. First Australian Conference on Testing and Assessment of Ethnic Minority Groups, Darwin, 17-18 October, 1985

Salomon, G. (1993). No distribution without individuals' cognition: a dynamic interactional view. In G. Salomon (Ed.), Distributed cognitions: Psychological and educational considerations (pp. 111-138). New York: Cambridge University Press.

Schon, D. A. (1987). Educating the reflective practitioner: toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.

Strauss, A. L. (1987). Qualitative data analysis for social scientists. New York: Cambridge University Press. 

Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage Publications Inc.

 

MENU


Mind Map of the paper

Another qualitative method of presenting information is as a mind map. Think of this as a mind map of the ideas in the paper overlaid with the data items in the form of a multidimensional scale (MDS).

Indexed as an "off-line" document at: (4 2 3) Outcomes/Qualitative/Graphics, etc.


SOME QUESTIONS FOR DISCUSSION

In developing this debate, you may like to consider some of the following questions:

  1. Clark tells us that the media (trucks) are only the vehicles which deliver instruction (nutrition), but many correspondents to this forum believe that there is a substantial difference in the quality of the instruction (e.g. McDonalds vs vegibranburgers) delivered by different media (e.g. refrigerated trucks). How might we investigate this now we know that media comparison studies are a dead-end street?

  2. Do Tom Reeves' arguments about pseudoscience apply equally to qualitative research? What kinds of qualitative outcomes could be branded as pseudo- ? (and what should be the root of this compound noun?)

  3. How might we investigate the quality of learning experiences available from web-delivered instruction?

  4. What kind of validity is appropriate to studies of media and learning? (And what does external validity mean anyway?)