SUMMARY AND CONCLUSIONS

It would be highly presumptuous for us, on the basis of six case studies, to offer general solutions to the problems of adult literacy, or even to technology applications to adult literacy. However, we believe that it is both necessary and appropriate for us to offer for OTA's consideration some summary observations arising from our cases, which probably find application in a broad range of cases, if not all. We will start with the good news and the bad news.

In our cases, we found many encouraging things that lead us to believe that technology can be an important component of efforts toward adult literacy. Some of the more notable items of good news include the following:

It is reasonably clear that there is a wide variety of effective information technologies available to adult literacy programs. Such tools range from simple to complex computer applications, to viewer-oriented or interactive video, to telecommunications-based shared systems. In our cases, we saw at least a sampling of each of these in action. They are not always used to fullest advantage, either because of design problems or implementation complications, but they are out there. Access to such tools is open to even relatively small-scale programs with limited resources, although of course large and expensive installations are also possible. When such tools are used, there is a definite sense, although not a lot of hard evidence, that learning is improved and progress toward literacy is speeded. Technology appears to be both an efficient method of reaching a larger number of clients than traditional methods can accomplish, and probably cost-effective in terms of resources expended.[1]

The overwhelming majority of the people interviewed in our study were highly enthusiastic about the benefits of information technologies in adult literacy efforts. Almost everyone agreed that technology helps attract adult learners to the programs and helps keep them there. In addition to its instrumental value in the teaching of reading, use of information technology, particularly computers, is often seen as a pathway to a vocational skill. Mastering technology enhances self esteem and increases motivation to learn; students will work with well‑ designed learning programs for hours if given the opportunity. Computer literacy programs designed to familiarize users with specific applications such as word processing and spreadsheets are often more popular than classroom-oriented basic skills courses, and in at least some cases can be used as effectively as more specifically literacy-oriented software to teach these same basic skills.

There is no substitute for the dedicated and effective teacher. However good the tools are, their application is no better than the underlying quality of instruction offered by human beings. Learning, particularly for adults, is at bottom a process of interaction between the minds of the teachers and those of the students. The tools can sharpen and focus this interaction, and can spell the teacher during the more repetitive parts of the learning process; they can also facilitate the administration of teaching. But ultimately it is the teacher who must guide the use of technology and shape its contribution to the overall learning context; only teachers are capable of responding to the whole person, not just the reading skills problems that may be presented in public. The best parts of any of the programs that we studied are those that reach out to clients across the full context of personal and social needs.

Creative use of information technology can support the open‑entry, open‑exit programs that many believe essential for adult literacy instruction. For example, it can support individualized, self‑paced instruction, by adjusting to different skill levels, by providing a "private" environment so that adult learner can avoid potential embarrassment, and by providing immediate, individualized feedback to learners. Technology can also be used to manage the complex administrative arrangements that flexible programming often entails, and to coordinate the multiple sources of funding that adult programs typically must reach for.[2]

Along with the positive elements of the picture, we feel compelled to point out some problems and concerns experienced by our respondents that lead us to moderate our expectations for technology successes. Some of the more notable issues are as follows:

It is absolutely clear that the populations most in need of adult literacy programs are also usually in need of a more or less significant array of other social support services. Among many of the clients of the programs that we studied at least, the inability to read is only one element of a syndrome that may include unemployment or underemployment, poor housing, nutritional problems, lack of access to child care, and other social maladjustments. No matter how good our tools for enhancing access to the printed word, we will have only limited success in resolving these other problems through literacy as such. Much as we would like to think so, reading alone lifts relatively few bootstraps. Moreover, the learning process itself is often impeded by these other factors of peoples' lives. Unless programs can develop ways of addressing the full range of life issues that their clients face, their contributions are likely to be swamped in a sea of troubles.

Not everyone is equally enchanted by information technology. Any program that plans to open itself to a full range of clients needs to make available learning opportunities that are based on technologies other than those involving computers or even video. Some people are simply resistant; others have had negative experiences that lead to the tools' becoming more of a barrier than a facilitator to learning. Those for whom technology does not work are often (but not always) older and less generally equipped to cope with modern toys. But they deserve as much consideration as we can give to those more comfortable with state-of-the-art tools. Thus, efficiency is almost always going to be compromised by the need to make programs accessible to technophobes as well as technophiles.

For learners who lack basic computer skills, use of information technology in adult learning can require significant amounts of time to be spent orienting students to the technology. This is not just an issue of "user-friendliness"; even the most "friendly" interface available takes some time to learn to command. For most clients who come into adult literacy programs, computer or even video technology is something that is outside the realm of the familiar, and time must be invested in learning how to use the tools themselves before they can be used to make meaningful contributions to learning. While some technologies are easier to access than others -- e.g., touch‑screen and voice‑oriented systems, even software programs that teach the use of the mouse -- the fact remains that at this point there are no "transparent" information tools in the literacy arena. The problem is compounded when system developers, themselves highly computer-literate, have a hard time putting themselves in the place of a person who has never seen a computer, much less manipulated it.

Traditional methods of training teachers, either professional or volunteer, for participation in adult literacy are not for the most part oriented toward technology use. Many adult educators lack computing experience or aptitude. Even when they can operate the equipment, they may not be able to manage the technology infrastructure: keeping it running, diagnosing common problems, setting up different applications, or performing simple maintenance. They may also have limited capabilities in the areas of hardware and software evaluation and selection. Teachers must generally become, if not computing experts, at least well-informed amateurs. For some, this is not just an imposition -- it can even go against the basic motivations for teaching in the first place. But it is essential if information tools are to find full utility in the teaching process. In our cases, we interviewed both those who have made this adaptation and those who have had a hard time with it. Both groups include talented and capable and dedicated teachers. It is no more easy to predict what kind of teacher will make this jump than it is to predict who will become any other kind of computer "guru".[3]

There is a truism in computing that the more powerful the technology, the more it can look like magic. That is to say, the better the tools, the more we tend to rely on them without completely understanding what they do and how they do it. In the context of adult literacy, teachers can easily cultivate a tendency to augment what the computer does, rather than find ways to have the technology augment what they do. There can also be a parallel tendency for teachers to become "lab managers," technology facilitators rather than teachers.[4] The very "privacy" afforded by the technology can create problems: instructors may fail to recognize when students need assistance. There is a certain irony in finding that the better the tools we use, the more problems we can engender with this use. This is not a plea for limiting technology, merely for recognizing that one must always remember that it is an "idiot servant" rather than the "savant" its elegant interface might imply.

There has been a proliferation in recent years of systems that purport to be all-inclusive, even "turnkey" arrangements for handling the complete teaching task.[5] While these systems can reduce the need for specialized support personnel, a quality particularly valued by the smaller programs, ILS's as a class (there are individual exceptions) share some serious deficiencies:

1. They cannot be customized (easily or at all) for specific learner needs -- e.g., in some settings, instructors wish to enable or disable testing and timing.

2. They have frequently been designed for children, not adults, and can be perceived as condescending; students lose interest and attempt to finish as soon as possible, rather than taking full advantage of the opportunities presented.

3. They may offer little opportunity for student control -- e.g., students cannot advance to new lessons (or skip over certain lessons) as quickly as they would like; when this happens motivation suffers.

4. While ILS's usually provide session‑specific student tracking, some of them have difficulty in tracking student progress/attendance over time, and thus can frustrate effective management of the program.

5. Some ILS's have limited "save and resume" capabilities, so that students cannot easily restart lessons. This is especially problematic for students with only a short time to work on lessons (especially in workplace programs where students may go in on breaks).

Again, this should not be read as a blanket indictment of ILS's, simply as a plea to recognize that there is no single "technology fix" that overcomes all the limits of program design and management. We shall return later to this overriding issue of the need for effective interrelationship of the social and technological elements in program operations.

Information technology for adult learning seems to have made little headway in overcoming problems associated with delivering the service to the clients whenever and wherever they need it. Typically, there is little use of "distance learning" in the programs we studied. Most of our group (except Cleveland) generally require clients to come to program facilities within specified hours in order to receive instruction. There are some very good reasons for this ‑ for example, the need for social reinforcement and the need to develop job relevant skills such as punctuality, personal appearance management, and social interaction. However, in almost all of our sites, clients experienced major difficulties in going to program facilities (lack of adequate child care, lack of transportation, need to work during program hours, incarceration). At the same time, program administrators and technology support personnel complained about difficulty in providing access to the technology whenever and wherever it was needed. Equipment often sat idle at some times, while being over‑demanded at others. People complained that the technology was insufficiently "portable" to be taken to remote learning satellites. One potentially important opportunity for distance learning might be to provide opportunities for those who have exited a program for whatever reason (some programs have six month maximums, tied to funding restrictions) to continue to participate.

Software developers have not as a class been responsive to the needs of adult learners. Many of the program staff we interviewed expressed the opinion that vendors are discouraged by the apparently small and fragmented market for high quality adult programming. It is our assessment that they have underestimated the size of the market for such programs. While there are certainly unique local needs, we believe that there is a large market for high quality programming that can be easily customized by instructors to meet the special needs of local client populations. The needs are particularly acute in the following areas: reading instruction, ESL, foreign language instruction (e.g., Spanish as a second language), instruction that is integrated across subjects (e.g., reading and math), software to identify and assist in overcoming learning disabilities, and adult education that is not specifically "test" focused (e.g., GED oriented). More adult programming of the "external high school diploma" type is required.

The current software offerings are also limited in another key respect. To our knowledge, almost all technology‑based instructional approaches (with the exception of non‑interactive videos) are designed to be used by individual learners working on their own. While this supports individualized, self‑paced, open‑entry, open‑exit, anytime, anyplace instruction, many adult educators emphasize the importance of learning in groups.[6] All of the programs we studied made extensive use of group interaction in some part of their curriculum. In view of the increasing importance of "groupware" in the commercial software environment, it is hard to understand the virtually complete absence of support for group‑oriented learning technologies.[7]

In recent years there has been a virtual explosion of easily accessible video technologies. Yet in only a few settings does there seem to be extensive applications of this approach; the LA County Jail system is certainly the most effective illustration of what can be done with video under the right conditions.[8] Granted its limitations, it does remain cheap and more accessible to many people than the much more widely applied computer-based tools. Perhaps it is sometimes seen as too "low-tech", not worthy of attention precisely because it is accessible. It is certainly true, as the jail case also illustrates, that a substantial front-end investment in production tools is required, and an ongoing commitment to the medium requires significant and specific talents and expertise not found in the typical community-based literacy program. In short, doing it right is not cheap, and might even be seen as a diversion of resources from more pressing needs. However, not every site has to have the full production facilities that the jail system has developed; a few centralized facilities could probably meet lots of specialized needs. While the jail system has shared its materials with a number of other programs, there is room for much more diffusion of this technology and its applications than is being currently explored.

Possibly the saddest piece of news that characterizes the parts of the adult literacy/technology scene that we have investigated is the striking absence of networking, between sites and often even within sites. We use the term generically. At the most basic level, there is far less use of networking among machines even within sites than the state of the art would suggest possible and even desirable. This is paralleled by a general isolation of personnel in one facility at a site from those at other facilities. This is not universal -- some sites have been able to create cross-fertilization among their personnel -- but isolation is far too characteristic to allow the literacy community to be comfortable. And there is so little contact between sites as to suggest that programs in this area are often operating in a virtual intellectual and functional vacuum.

Coordination of services with other agencies is frequently desired, but singularly underachieved. Despite the capability to use their own technology to exchange information, there are few attempts made to help learners identify or access other social services available to them (e.g., child care, food, housing, transportation), other than where mandated by law as a part of a welfare reform initiative or similar incentive package. In short, the potential for information exchange is singularly untapped in the agencies we studied, despite their wishes for it. Like everything else, networking costs resources, and it is often hard to justify taking funds from service delivery to support it.[9]

Detection of good and bad news in the overall picture of technology applications in adult literacy is only one part of the picture. Of even more overriding importance, in our opinion, is a set of issues and concerns revolving around the social and organizational contexts within which technology is applied in such programs. On balance, it should come as no surprise that context critically conditions and mediates the impacts of technology on organizational participants, both providers and clients -- it is one of the principal commonplaces of any systematic analysis of technology. Nevertheless, it is appropriate here to offer some observations on how socio-technical interactions pose both opportunities and limitations for technology use in literacy improvement.

We would like to believe that adult literacy is an educational issue, one "above politics" in which the interests of individuals can selflessly and seamlessly be served. Unfortunately, as soon as we set up social organizations to help people, we are inextricably enmeshed in a web of politics. Sometimes this is politics writ small -- the direct interactions of different interest groups trying to maximize their own leverage over particular social institutions. Sometimes it is Politics in the larger sense -- that is, the interactions of large social groups vying to shape society in ways that enhance their quality of life. But always conflicts of interests can be observed, and must be coped with in diverse ways.

Sometimes literacy programs become a direct part of the political agenda, as we observed in Baltimore. While this does have advantages in terms of increasing visibility for such programs and thereby improving access to social resources, it does pose the danger that he who lives by the vote can die by the vote. If the next Mayor decides that Baltimore should be "the City that Sleeps Safely" instead of "the City that Reads", something critical will have been lost that may prove very difficult to regain. The public agenda is notoriously fickle, and while there is a certain satisfaction in being the Cause of the Month, it always the case that there will be another Cause next Month. Adult literacy is a long-term issue, and will not go away with the next shift of public opinion or media hype. It helps to be closely tied to a wide range of institutions, as in McAllen, where the program appears to be firmly fixed in the local environment. But politics is politics, and nothing is ever permanent.

The political dimension is no less real in other settings, though it may be more subtle. In the UAW/Ford program, for example, the politics are organizational rather than societal. The activities of the projects in the auto plants are played out in the context of labor/management relations generally, and the highly sensitive environment of the domestic automobile industry in particular. Passions run high in this environment, and threats to interests are strongly felt and quickly reacted to. The LA County jail program is likewise entangled in the entire set of issues dealing with crime and punishment generally, and the particularly difficult environment of contemporary Los Angeles, with its deep social, racial, and economic divisions. What may appear on the surface to be simple technical decisions turn out on further inspection to have major ramifications for long-term organizational survival and development.

Increasingly, instability is a hallmark of the current American political and social scene. Issues come and go in prominence, and there is little consensus on long-term directions and priorities. Moreover, there is increasingly bitter competition among priorities for what appears to be a steadily shrinking pot of resources, both public and private. Under these conditions, cobbling together support for programs becomes an art form of high order. All of the programs that we visited drew on multiple sources of support. This does allow programs a degree of flexibility in that they are not tied inextricably to any one set of funds (with the possible exception of the UAW/Ford program); on the other hand, it also requires agencies to respond to goals and priorities of many different funding sources, some of which may be incompatible if not downright contradictory.

This instability in turn makes the effective implementation of technology-based strategies for literacy programs problematical at best. Such programs cannot be put together overnight, nor can they be easily retooled to respond to a new set of priorities. Rather, they require time to set up facilities, procure hardware and software, learn how to use it effectively, recruit an ongoing client base, and generally establish a credible reputation. Programs have to become expert at reinterpreting themselves in different terms to different groups and at different times. Constant coping with the environment, while it does keep programs on their toes, also exacts costs in terms of time and energy at least. Working out the place of adult literacy in the nation's and the local agenda looks to be an ongoing and difficult process.

The essence of politics is constituency -- that is, the group whose interests are being served by the process. If, as we have suggested, adult literacy programs almost always operate in a political environment, this implies that such programs must develop and maintain a constituency to remain viable -- and perhaps to attract better attention from software developers and vendors. The difficulty is that those who stand to benefit most from these programs are perhaps the group in society least equipped to mobilize on its behalf -- that is to say, those who have limited reading skills. Their very need for the program serves to cut them off from the normal channels by which political influence is exercised. Moreover, if the program works and people actually do learn to read, they generally sever their connection with the program -- just at the point where they might begin to be able to do it some good. There is a cruel paradox at work here.

This perhaps overstates the point. In fact, there are some constituencies for literacy programs derived from the general social consciousness of the country, and, as the UAW/Ford case illustrates, an increasingly mobilizable constituency based on workplace interests. Part of the problem remains that it is small business more than large business that bears the brunt of the impact of worker illiteracy; large firms can afford to be more choosy about whom they hire and retain. Yet small firms are notoriously unable to form themselves into consortia capable of exercising systematic influence and putting together joint programs. Another possibility -- if "family values" becomes more than this year's catch phrase, perhaps the family can itself become a literacy focus; certainly family-based programs are among the more successful ones in operation.

In any case, the mobilization of support for literacy remains diffuse and complex, and at a considerable disadvantage relative to more organized social interests and problems. Limited literacy remains to a significant degree a "hidden" social problem, much less visible than, say, homelessness or unemployment. Nevertheless, as our cases illustrate, programs do get created and maintained, usually through the political and organizational competence of extraordinary individuals. The nature of the system does little to make their tasks easier

Another key theme illustrated by our cases is that the only thing permanent about them is change. Yet another paradox -- only through continuous changes in structure, emphasis, and procedures can any semblance of continuity in programs be maintained. There are a number of different sources of pressures for change, some of them social, some technical. On the technical side, there is the pace of technological developments themselves -- bigger, faster computers at a more manageable price, better software, interactive video displays, and the full range of new toys that are continually coming to market. On the social side, there is the constantly shifting wind of the political agenda noted earlier. Moreover, there is more or less steady personnel turnover, both in terms of providers and clients, and thus a regularly shifting body of program participants on both sides of the curtain.

Again and again, in our cases as in virtually any other technology application that we discussed in our opening frame of the problem, we see that the key to effective use of technology is the capacity to "reinvent" -- that is, to modify existing hardware, software, and systems to new uses and new purposes. This may range from the glosses applied by a capable teacher to make a software program designed for six-year-olds palatable to their parents and grandparents, to full-scale customized installations of networked stations. At all stages, it is the imagination and talent of particular individuals that makes systems manageable. If systems fail to be able to see the necessity of change, they die.

The infrastructure of programs may be described as the set of all aspects of the program, other than those that actually provide services to clients, that are required to maintain it. Technology does not operate without an infrastructure, and the more a system depends on technology, the more vulnerable it is to disruptions in that infrastructure. As programs come to depend more on "open architectures" with widely differing combinations of machines and software, the more they need specialized personnel to evaluate hardware and software, perform systems integration, troubleshoot and solve systems problems, switch over equipment to different applications, and handle the myriad of other strange and wonderful things that arise daily.

The situation is compounded by problems arising from equipment and software diversity. Many programs, even those that rely heavily on ILS's, tend to have a very diverse portfolio of equipment. Often they have acquired much of their equipment in the form of miscellaneous gifts and "hand me downs." This increases the difficulties of teachers in using the equipment, and of equipment support and maintenance personnel as well.

Even highly computer-sophisticated organizations experience system integration problems; yet again and again we find literacy programs that can generously be described as "computer-shy" coping with and effectively resolving highly complex integration issues. Much as this is a testimony to the dedication and capabilities of the individuals involved, it can best be described as a disaster waiting to happen. Each and every one of the programs we visited is in critical respects staggering under a potentially deadly load of infrastructure problems in the form of equipment failure, personnel turnover, and knowledge obsolescence.

Perhaps the most critical element of the infrastructure, and the one least well coped with, is the level of "local expertise" available to program sites. In all the sites we visited, as in virtually every other information technology-using site that the authors are familiar with, there are one or more "gurus" who, by virtue of interest and capability (seldom formal training) manage to provide the well of expertise and knowledge to nourish staff usage by those who cannot equally grasp the underlying mechanics of the system. These individuals are absolutely critical to the functioning of the programs. Yet there is almost never a formal mechanism set up to stimulate and reward guru-hood -- even to "clone" or back up people who have assumed the role in practice. Often the role must be carried out over the passive or even active disincentives established by the organization. We believe that the contributions of such individuals need to be cultivated and encouraged, since it is they who manage to keep the infrastructures patched together with baling wire and cellophane tape -- not to mention recycled disk drives and self-taught midnight programming.

In short, while it is relatively easy to evaluate the pedagogical advantages and disadvantages of particular software programs, it is far more difficult to assess what is going to be necessary to make the program usable in terms of administrative and technical backup. But if the infrastructure fails, it matters not how good the program was -- nobody is going to benefit from it. Unfortunately, there are few lessons in how to manage infrastructures to be drawn from the literature or the experiences of others -- just an almost endless array of illustrations of different ways to fail in varying degrees. The one thing that we do know is that the organizations that experience the fewest infrastructure problems are those that recognize that it is there and needs systematic care and feeding. "We deal with it by talking about it" is the sine qua non of coping.

We conclude this analysis with a few simple recommendations derived largely from suggestions offered by our respondents during our interviews. Some lend themselves to policy suggestions; some are offered more in the spirit of cheer leading than specific changes.

While it is no panacea for resolving issues of adult literacy in this country, it is nevertheless likely to contribute to effectiveness of programs in which it is used in conjunction with trained teachers and appropriate infrastructure support resources. Adult learners like it, use it, and benefit from it. There should be more of it in use, and in more sites, and of a higher level of sophistication.

How do we get more technology utilization, particularly if, as we have suggested, much of it depends on local initiative and reinvention rather than on central direction? One way would be to try to focus resources onto technology, either by mandating certain investments out of Federal grant funds or by setting up some new grant program emphasizing technology in particular. In general, this is probably less effective than it might seem, and would be likely to introduce technology biases into the system. There are, after all, certain "heavy hitters" in the literacy technology game who would be likely to end up with a disproportionate share of the resources. Reliance on the market is better.

What could be done, and relatively easily, is to expand the resources available for sharing information and ideas about creative applications of technology in literacy. We discuss this point in more detail further below in our comments on networking. At this point, we believe the barriers to technology applications are more conceptual than financial. Money never hurts, but the answer is not a great infusion of new funds; rather, the problem is to use what is there in a more focused fashion.

Since there will always be local situations and problems requiring customized solutions with limited applicability elsewhere, there will always be a need for tools (both ILS's and products designed for an open architecture) that permit easy modifications by instructors and flexible usage patterns by adult learners. The incredible diversity of situations even across six cases illustrates that "one size does not fit all". Both program design and technological content need to be responsive to both the immediate and the ever-changing local context.

This concern can probably be best addressed by a firm resolution to keep hands off. There will be a temptation on the part of government, particularly if there are additional resources to be allocated, to establish "standards", define "best practices", and in general get involved with the nuts-and-bolts of program design and implementation, all, of course, in the interests of effectiveness. This temptation needs to be resisted. A hundred -- or a thousand -- flowers need to bloom, and the good that might be done by preventing one or a few programs from making fatal mistakes is likely to be more than compensated for by the harm that would be done in reducing innovation and creativity. We take this opportunity to issue a strong plea for the encouragement of diversity.

We believe that communication is the key to effective implementation of any improvements in the entire technology/literacy picture. While flexibility and reinvention are Good Things, we have seen many local innovations with widespread potential applicability, both pedagogical and administrative whose light remains hidden under a local bushel. Along with our respondents, we believe that information dissemination services for adult literacy administrators, educators, and technology support persons are both essential and largely ignored at present.

This is something that government could profitably become involved with. There are ample precedents that might be applied, such as the National Innovation Network operated by the Department of Education to facilitate the diffusion of new ideas in teaching generally. This might usefully be supplemented with a modest program of support for technical assistance rendered by program people in local sites to each other. Creative people can and would be happy to help each other try new ideas, share software, tell war stories, and in general support each other's efforts if resources were available to fund travel and related costs. Again, what is needed is not a large Federal presence in technical assistance, but rather encouragement for people on the firing line to get together and be resources to each other.

At the very least, there is room for electronic bulletin boards devoted to evaluations of and experiences with particular commercial or locally developed educational software or supplemental materials. Such boards are inexpensive to operate, but they do take initial investments and continuing attention. For a set of small grants to appropriate regional and/or national organizations, a great deal of effective interchange could be encouraged. With modems available for less than $50, there is virtually no excuse for any computer system not to become part of the "cyberspace" of information exchange.

What conferences and professional organizations there are in this area do an excellent job, but they barely scratch the surface of its potential. Some additional funding for regional and occasional national conferences would probably be well received and of significant impact.

Again, there have been useful efforts made to compare different approaches to use of information technology in adult education and to disseminate the best approaches. But they have generally been under-funded and single-shot studies, without an ongoing underlying research paradigm. There is need for a fairly extensive set of studies in this field to address the full range of socio-technical issues that we have discussed, as well as many that have undoubtedly escaped our attention in the whirlwind of this project.

What we would suggest needs emphasis is not research on literacy as such, or even on appropriate technology for teaching literacy, but rather research on the processes by which successful community and workplace-based programs can be created and maintained. This is not to say that technology or content-focused research is not useful; it is only to reinforce that we know so much less about what to do with what we do know that it might be appropriate to let our ability to apply knowledge catch up with the knowledge itself for a while. We suggest that a combination of quantitative and qualitative research aimed at understanding how implementation of technology in the field of literacy is both like and unlike implementation in areas that have been more studied, such as schools, offices, and industrial settings. This would, in our opinion, enrich our knowledge about both literacy and technology in general, at the same time.

Clearly, research on new techniques should also be encouraged. The developments under way at ETS toward creation of new group-based interactive software show that there is always room for new kinds of tools to enter the arena. In addition, technical research should continue to explore innovative ways of networking both individual sites and groups of sites. We believe that this research is probably more effectively funded by vendors and other private sources rather than government, which does not have a particularly good track record "picking winners" in technology choices. The exception might be greater encouragement to agencies with literacy interests such as the Departments of Education and Labor to use their Small Business Innovation Research programs to encourage the development of innovative technologies for literacy improvement.[10]

Another area that might profitably be explored is that of evaluation at the local project level. Very few of the sites we visited had anything approximating a formal evaluation capacity. This is not to say that they were not interested in what they were doing, or with what effect -- simply that they had other things to invest their resources in that appeared to be of more pressing importance. Evaluation has never been particularly popular at the project level. Yet where outside resources can be provided to support it, it can have some very valuable consequences. The major lesson to be derived from a good many years of Federal research on program evaluation is that it cannot be imposed from without -- rather, the capacity has to be grown within the project as part of its basic structure. Moreover, it must concentrate on the formative aspects of evaluation rather than the summative or judgmental. In any case, a modest program of funding to support the development of systematic evaluation programs within at least a few programs might have significant payoff.

This is not an area that calls for a single unified research agenda, particularly not one set by a Federal agency. Rather, it will be best served by a diversity of approaches and strategies fueled by the different research interests of a range of sponsors, public and private. One step that the Federal government might most profitably undertake is to sponsor an annual or at least regular research conference at which various aspects of the technology/literacy interaction could be explored and cross-fertilized. Such conferences, generally costing less than one individual study of modest scope, can be of enormous value in encouraging research progress.

We echo the heartfelt plea of many of our respondents that whatever resource commitments may be made to adult literacy and specifically to technology in teaching be continuing and predictable. A modest level of resources that could be counted on steadily would be far more effective than a large one-time injection of resources that could not be digested easily. Even worse would be "seed-money" type commitments that are directed toward particular types of technologies; these establish perverse incentives that are particularly destructive of systematic organizational operations. While everyone would like more money, there is general agreement that it should be the right kind of money.

A steady pattern of support is a key component of the "infrastructure" that we described earlier. Specifically, steady support allows systematic planning for technology acquisition and upgrade, staff training and development in both teaching per se and technology applications, and the careful monitoring of outcomes across time that alone can tell what is actually happening as a result of different approaches. Environments of resource fluctuation are particularly damaging to evaluation as well as planning.

Finally, we offer the observation, again suggested by many of our respondents, that the real problem is literacy, not technology utilization. Tools need to be selected to meet needs, not to enhance the image of tool-builders. There is little room in this field for the kind of "technology-push" that has bedeviled other areas of technology application, particularly in manufacturing -- a drive to secure advanced technology simply because one believes that one will be outdated without it, rather than an intelligent procurement based on understanding of both technology advantages and program goals and needs. Applications that are tool-driven are inherently less effective than those that are use-driven; this is the consistent lesson of previous research.

This is not to say that the evolution of technology should not continue, or that developers should not do all they can to create applications and sell them to as many as they can. As in any other area of technological evolution, the market is critical. However, program planning -- from the level of the community agency to the level of the Federal government -- needs to start and finish with creative pedagogy, not technological determinism.

The major need is now for a degree of "consciousness-raising" on the part of system developers and users alike. To this end, efforts such as OTA's present study are necessary and worthy. But a sustaining and continuing involvement with the issues will be required; as we noted, the political agenda is mutable and full of potholes. The interaction of technology and literacy is too important to be left to any one focal point to nourish and protect. There is plenty of creativity out there, as we have described; the challenge now is to harness and empower it for the greater good.

[1] It should be noted that there are very few systematic data available on cost-effectiveness as such. See our recommendation below regarding the need for systematic evaluation research.

[2] Again, it should be noted that there are fewer applications of information technology to program management than might be supposed. The reasons for this remain somewhat obscure.

[3] See our point later in this discussion on the absolute necessity to develop appropriate local expertise in the management and operation of technology-based learning systems.

[4] In our cases, we encountered this problem more as a fear than as something actually experienced. A number of teachers did report that this was a tendency that they had to remain continually alert to avoid.

[5] The WICAT environment used in several of our sites is a classic example. IBM's PALS system is related, but not fully integrated.

[6] A few programs could not really be called "open-entry," since classes started at fixed times and only when a minimum number of students had been enrolled.

[7] The system under development by ETS that we saw being tested at the Ripken Center in Baltimore is an interesting exception, and its further development and implementation should be well worth watching.

[8] Of course, the peculiar conditions of this case, with the large and constantly shifting client population, have made video virtually mandatory as a medium.

[9] In our Recommendations, we offer some suggestions as to how networking might be improved.

[10] At present, there is no way of effectively determining how much SBIR activity there may be in literacy. Assembling these data would be a useful exercise in itself.