Resources

Research

White Papers

Towards an Ecological Theory of Sustainable Knowledge Networks (part 2)

Overview. The Collaboration Assistant activity (abbreviated CA) will conceptualize, design, and implement a software prototype assistant suitable for use by pilot test groups. It will actively provide information, analyses, and advice to moderators and collaborators within a distributed, asynchronous virtual collaboration. The design of this assistant will be based upon, and represent a pragmatic embodiment of, our working hypotheses of sustainable knowledge networks. Thus, use and evaluation of this CA will constitute a strong form of hypothesis testing.

We conjecture that most of the sustainability factors exposed in the empirical studies (both Observations and Experiments) will be concerned with collaboration-specific phenomena, such as the distribution of knowledge in the group, the various roles the members take on, the conventions and practices of sharing knowledge, the rewards for being explicit, etc. These phenomena generally remain concealed by the current document centric desktop applications. Thus, we expect the CA will be predominantly concerned with collaboration-specific parameters such as amount of access to shared objects, awareness of other’s presence and activity in the shared space, and the other factors suggested in the description of Activity 1, Observations (p. *).

CA Functionality. The functionality of the Collaboration Assistant will be partitioned into two categories. The first category is "Shared Object Coordination," which is concerned with gathering, manipulating, and disseminating artifact information. As much as possible we will avoid creating application-specific features; rather, we will treat a complex shared object within a virtual collaborative space as a "black box" with links going into and out of it. For example, the MS Windows environment now makes it relatively easy to create web and OLE links between documents (artifacts), and the CA might track the creation and evolution of those links. The web of links could also be analyzed by the CA for patterns of traversal and use by different members of the team, perhaps leading to "usefulness" metrics on shared objects (Hypotheses #8 and #9).

The second category of functionality is "Process Coordination," which is concerned with gathering, manipulating, and disseminating information about activities, participants, interactions, and formal/informal communications. For example, the CA might have features for orienting virtual team members in the knowledge network, such as a graphic view of all of the objects and interactions in the space color-coded for how recently each object was created or changed. Previous studies on the utility of graphical views of communication spaces (Begeman 1990) have shown that such a view would allow team members to tell at a glance where the "hot spots" were in the shared space (Hypothesis #10). The scheduling and managing of work tasks (e.g. workflow capabilities) is another one of its important functions concerning process (#2). We understand that much of the team knowledge is passed along by informal interactions. Thus, it is useful for the CA to be cognizant of which participants frequently chat with whom. We thus intend that the CA will have built-in dimensions of awareness, including some degree of task awareness, organizational awareness, and social awareness.

When groups are not co-located, the issues of team cohesion and shared context tend to be particularly aggravating. We believe that the Collaboration Assistant can include features that may significantly help to address these issues. For example, within a large team, it is useful simply to know who is currently working on the artifacts, and who has been contributing or non-contributing. This is one specific example of our working hypothesis that "cues about the activities of others will make the system more sustainable" (Hypothesis #1). This information can be presented to participants specifically (by name) or statistically by the CA. The CA can maintain certain barometers of team viability. It is thus possible that the CA can help team cohesion by noting team deficiencies, monitoring group member satisfaction, or suggesting helpful actions to appropriate team members or to those playing a mediator role.

Design Process. The CA will be conceptualized, designed, and implemented using techniques of iterative team design (Norman and Draper, 1986) and participatory design (Schuler & Namioka, 1992; Kyng, 1991). It will encode and utilize aspects of our theory of sustainable knowledge networks. The design team will perform conceptualization work, as well as producing a specific implementation; this work will start in the first year.

We envision that the CA will connect to and augment existing virtual collaboration systems (e.g. Lotus Notes™, QuestMapÔ , and Consensus @nyWARE®). Within the conceptualization work, we propose to first explore taxonomies of collaboration primitives that are common among such collaborative systems. We will investigate possibilities of a collaboration language or protocol (e.g. Lai et al, 1988) that enables our prototype, through a standard interface (see Figure 1), to communicate with different groupware systems.

Within the implementation work, there will be an initial (potentially throwaway) prototype designed and programmed by a team of implementers and users at the University of Colorado, followed by one or more enhanced prototypes developed at GDSS. The initial prototype will be constructed within a three-month coding period, and will be useful to show functionality and interface possibilities. Comments from experts and potential user communities will be elicited, and then one or more redesign and re-implementation efforts will take place during the second year of the research work. This will produce a prototype that is suitable for use by external pilot test groups. Testing, evaluation, and some continued development will proceed during the third year of the research.

The proposed research will yield two technologies to be evaluated in Activity 5, a Handbook representing an ecological theory of sustainable knowledge networks including guidance for distributed teams, and a prototype Collaboration Assistant that advises participants in distributed teams.

The evaluation strategies for these technologies will differ because their purposes differ. The Handbook is intended to serve as a resource for researchers, developers, and end users, and it will evolve as research and development unfold. Its value will be measured by how it is used.

The theory in the Handbook will be tested in Activity 3. The Handbook itself will be accessible on the web where we will monitor its use. We will track the frequency of access to different parts of the Handbook, thereby measuring the degree of interest in those facets of the theory and guidance represented in those parts. We will also solicit input from visitors to the Handbook. They may comment on parts of the theory, provide anecdotes or data supporting or refuting the theory, and evaluate the Handbook’s utility.

The Collaboration Assistant will be evaluated in field experiments. Because of its prototype status, we will not deploy it to teams engaged in mission-critical work. Instead, we will recruit distributed teams that can accommodate the uncertainties and difficulties experienced when using prototypes. Examples of potential evaluation teams include a conference organizing committee or a team concerned with computing support. These are both examples of distributed teams composed of people with computing expertise. If the Collaboration Assistant proves successful with such teams, we will consider teams that are less receptive to new technologies.

Results generated in the course of this research will be disseminated through the traditional channels of reports, white papers, conference papers, and journal articles. In addition, we plan to use the Handbook (Activity 3) as a major interactive forum for the on-going dissemination of research ideas, theories, and results. Finally, the Collaboration Assistant prototype will be available via Internet download to the public.

The University of Colorado Department of Computer Science is an active research department with 25 full time faculty members. The department is closely aligned with the Institute for Cognitive Science, and provides all of its faculty with up-to-date computing equipment in their offices and homes. Since 1985, the department has been constantly supported by NSF Computing Infrastructure grants (NSF CER grants followed by NSF RI grants), along with being the recipient of numerous industrial grants and gifts. Department equipment includes modern computing equipment on every employee’s desk, labs full of PCs, MACs, Sun workstations, mainframes, etc. The most recent RI program grant to the department focussed upon providing networking and multimedia distributed computing facilities using high speed network technology, fiber optics, and ATM switches. The University of Colorado is committed to excellence in research. Thus, computing equipment, infrastructure, and other resources of the University will be made available to perform this research.

Group Decision Support Systems, Inc (GDSS) is an consulting and facilitation firm specializing in the use of state-of-the-art synchronous and asynchronous communication tools for group planning and decision making. GDSS’s facilities in Georgetown in Washington, DC, offer a modern high-tech environment for research into collaboration. These facilities will be completely available to Dr. Jeff Conklin in support of this research project. In addition, the face-to-face conferences held by the research team will be hosted by GDSS in its meeting room facilities.

George Washington University currently provides Lynn R. Offermann with a private office complete with a Dell Pentium PC with ISN line to the GW mainframe, non-graphics access to the Web through Lynx, and a DeskJet printer. Ethernet connection to the Web is available down the hall, with current installation plans for full desktop internet access for her building scheduled for Fall 1998. She also shares a research room with PC and printer on the fourth floor, currently housing one of her doctoral students. Research space on the first floor is available to all faculty on an as-needed, scheduled basis; this semester she has had use of 5 rooms for a pilot study on distributed team decision making. Conference rooms are also available on a reservations basis.

Boeing’s Applied Research and Technology (AR&T) organization is a center for advanced applied research in computing application development and has facilities to support that mission. This includes a variety of UNIX servers, including HP, IBM, SGI, and Sun that support over 600 users and over 550 workstations. The workstation mix includes over 250 UNIX machines (Sun, SGI, HP, Dec, Alpha, IBM), roughly 220 PCs and an assortment of miscellaneous machines. An ATM backbone is also part of the environment which includes Internet access and access to Boeing’s mainframe computers. AR&T is located on a 90 acre site in Bellevue , Washington that houses the Boeing Shared Services Group Headquarters and Boeing’s largest data center. The entire complex consists of over 1,000,000 square feet of space devoted to computing support and applied research activities.

In support of its commitment to this proposal as discussed in the Letter of Intent, Bell Atlantic will provide sufficient work space and computing equipment to support Bell Atlantic personnel (specifically Albert M. Selvin) working on this project, as well as conference rooms in Bell Atlantic work locations as needed, high-speed Internet access, local- and wide-area-networking, remote access, speakerphones and telephone service to support teleconferencing, and other technologies as necessary. Bell Atlantic is currently both researching and developing distributed shared space technology and methods, so its participation and commitment to space and equipment for this proposal will result in greater leverage for Bell Atlantic's investment in this area.

The primary deliverable of this research project is a theory of sustainable knowledge networks. The deliverable comes in two forms. The first is an written "theory handbook," a textual and web-based compilation of data, hypotheses, principles, and annotations; it documents the research findings and emerging theory at many levels, from hand-on practical tips for virtual team leaders, to published papers on social and architectural principles of sustainability, to interview transcripts and other essentially raw data forms. The second form is a prototype implementation of the theory as a "Collaboration Assistant," a system that provides warnings, data views, analysis, and advice about counter-sustainability conditions emerging in a virtual team’s knowledge network.

The program to create this theory has five activities that span three years (see the Project Description for details of the activities):

Performance goals for the three years of the program are as follows.

Year 01. Our initial focus will be on the Observations and Handbook activities. The goal of Observations in Year 01 is to identify at least 4 virtual teams (both "live" and "deceased") within our partner corporations (World Bank and Boeing), gather data about their collaborations, analyze the data in terms of sustainability principles and factors, and document the analysis. The goal of Handbook in Year 01 is to construct a web-based "research collaboratory" using off-the-shelf tools (e.g. Consensus @nyWARE or Lotus Notes) and to assemble within it (as much as feasible) the linked network of data and observations from Observations. Handbook will include a continuous theory-building effort that includes all research staff in this program. Finally, an initial design effort will happen in Year 01 on the Experiments, and a first "throwaway" prototype of the Collaboration Assistant (CA) produced.

Year 02. In Year 02 our emphasis will be on Experiments and CA, but reduced levels of work will also continue in Observations and Handbook. In particular, we anticipate that the elements of an ecological theory – hypotheses and principles – will come into clear focus this second year. The goal of Experiments in Year 02 is to test specific hypotheses created in Observations and Handbook. The goal of CA in Year 02 is to take the rapid prototype from Year 01 and "harden it" into a working prototype, suitable for use by pilot test groups. The goal of Handbook in Year 02 is to develop a short course (and associated training materials) in moderating virtual teams. We anticipate presenting several papers at various conferences during this year (e.g. CSCW).

Year 03. The final year of the project focuses on Evaluation, both of the Handbook materials as well as of the prototype CA. The goal of Evaluation is, using pilot groups from various organizations (not restricted to Boeing and World Bank), to gain clear feedback about advantages and breakdowns encountered in real work situations using both Handbook and the CA prototype. This evaluative material and its analysis, as well as the required NSF reports, will be the major work products from Year 3.

A distributed research team such as one proposed here will require that special attention be given to creating and maximizing good coordination of studies and experiments, a high level of shared understanding about goals and issues, and exceptional sharing and collaboration about interim research results. Dr. Conklin will augment Dr. Ellis’s PI role by serving as the overall Project Manager. Drs. Ellis and Conklin worked together closely in the Software Technology Program at MCC and have a very high level of trust and respect for each other and an excellent working relationship.

In addition, each of the five research activities will have its own Lead Researcher (who is also either PI, co-PI, or a Senior Researcher), as shown in Table 2.

Table 2: Leaders of the five research Activities

Each Lead Researcher will oversee the coordination of research efforts and the sharing of information, electronic and otherwise. Bell Atlantic is donating 25% of Al Selvin’s time to this research effort, and he will be participating in both Observations (analysis of Bell Atlantic software project data) and CA (participating in design and testing).

In order to maximize cross-fertilization of ideas, hypotheses, and results among the Centers, we will use web-based collaboration tools, monthly conference calls, and a mid-year face-to-face conference (on the East Coast) to facilitate the research collaboration. For four months during Year 01 the Observations teams will have an additional weekly one-hour conference call for active sharing of ideas, issues, and hypotheses. Drs. Ellis and Conklin will arrange for additional face-to-face meetings twice a year for administration and management oversight coordination.




The web-based Handbook will serve not only as a medium of distribution of information and knowledge to a wider audience, but will also serve as a virtual collaboratory, a collaboration and coordination mechanism among all project researchers across the four centers, plus consultants and other members of our extended network. We plan to use the web-based Consensus @nyWARE® system for this interactive dimension of the research (Consensus @nyWARE® is being supplied by GDSS as part of its cost sharing).

The distribution of funds among the four Centers (the University of Colorado at Boulder and the three subcontractors) is described in the budgets. The chart at right describes the approximate distribution of resources among the five activities. As the chart shows, all activities are funded in each year (except Testing, which is only funded in Year 03), but different years emphasize different activities. The first major activity of the research project will be a series of planning meetings involving the entire research staff, via video teleconference, to plan the detailed milestones and accountabilities of the first year.

Hard copies of letters of commitment from the World Bank, Boeing, and Bell Atlantic are being sent along with the Cover Page.

Facilities, Equipment, and other Resources

The Psychology department at George Washington University has several observation rooms on the first floor that are currently available for research on a reservations basis. If the proposal is funded, on of these would need to be dedicated to the grant, with requested equipment installed, for the life of the grant. In addition, networked PC stations would need to be available. This could be accomplished through sharing use of the Psychology laboratory space (around class times) or using multiple research rooms (as is being done now during pilot testing).

The University of Colorado Computer Science Department equipment includes modern computing equipment on every employee’s desk, labs full of PCs, MACs, Sun workstations, mainframes, etc. The most recent RI program grant to the department focussed upon providing networking and multimedia distributed computing facilities using high speed network technology, fiber optics, and ATM switches.

Group Decision Support Systems, Boeing, and Bell Atlantic all have similar modern computing facilities. In the Psychology department at George Washington University Dr. Offermann has a Dell Pentium computer with an ISN line.

All the research staff listed on this proposal have their own offices and full access to modern office facilities, e.g. telephones with speaker phone capability, voice mail and email, copy and fax machines, administrative assistance, and meeting rooms. In addition to her own office, Dr. Offermann shares a research room on the 4th floor with another professor, used largely to house two research assistants, store data and materials, and occasionally to run subjects completing paper-and-pencil measures. This research space would be needed to house Research Assistants under this request. If full funding is granted, some additional space would be required to house a total of 4 students in year 01, and 5 students plus one research coordinator in years 02-03.

The World Bank has a newly installed observation laboratory that will allow researchers to observe real working groups engaged in collaborative work. The laboratory includes a two-way mirror to an observation room, whiteboards, a VCR and TV monitor, a computer, and tables and chairs that can be rearranged depending on the group’s work needs. In the near future, it will also have audio and video taping equipment. One of the co-PI’s (Offermann) is already engaged in activities which use this state-of-the-art research facility, and fully expects this facility to be available in support of the proposed research activity.

The Psychology department at George Washington University has one video-observation room set up and available for use (currently being used in pilot testing) on the first floor. This room has full capacity for videotaping, with intercom capability. The control room also supports video editing. A similar facility to be available on a dedicated basis would be set up with funds requested under this proposal.

At GDSS, Dr. Conklin has a strong working relationship with a staff of 25 first-rate organizational consultants and group facilitators, as well as professional computing and graphics services. A sister software company, The Soft Bicycle Company, is in the same building and has skilled programmers and successful products that overlap heavily with the proposed research. The programming effort for the Collaboration Assistant would be performed by subcontracted programming staff housed in and supported by The Soft Bicycle Company, thus assuring timely and professional code production.

At Boeing Dr. Poltrock is currently the director of the Workgroup Integration Technology project within Boeing’s Applied Research and Technology department, where he is currently supervising three research projects that overlap strongly with the proposed research. Dr. Grudin is currently a member of this research staff. Thus, Dr. Poltrock’s staff, laboratory, and research momentum would be directly and immediately applicable to the sustainability research proposed here.

At the University of Colorado at Boulder, Dr. Ellis is a Professor in the computer science department, faculty member of the Institute for Cognitive Science, and Co-Director of the Collaboration Technology Research Group. Dr. Ellis thus has an extraordinary resource of talented and experienced colleagues, already engaged in related research (e.g. Dr. Gerhard Fischer’s NSF-supported work on collaborative technology and organizational memory), as well as a pool of high quality graduate students to draw from in performing this research.

Dr. Offermann has been conducting on-going research on distributed team processes, using observation laboratories available at GW. The project she has been running has involved three research assistants. Like Dr. Ellis, Dr. Offermann is access to high quality doctoral students, some of whom would doubtless find rich opportunities in the proposed research.

Drs. Ellis and Conklin worked together for seven years in the Software Technology Program at Microelectronics and Computer Technology Corporation (MCC). They have a strong working relationship and a high level of trust and respect for each other, and this relationship is a strong asset for the management of this research project. Similarly, Drs. Grudin and Poltrock worked together for many years in the Human Interface Program at MCC, and have continued to work closely together since; for example, they are co-chairing CSCW ’98, the ACM-sponsored major conference in the field. Also, there was a strong connection between the Software Technology and Human Interface Programs at MCC, thus Drs. Ellis, Conklin, Grudin, and Poltrock have associated professionally for over 10 years. Dr. Conklin also has worked with Mr. Selvin (at Bell Atlantic) for nearly a decade. Thus, in addition to their individual credentials, many of the lead scientists on this team have a strong background in collaborative research together in the CSCW field. During the proposal writing process Dr. Offermann became a strong member of the team, adding her expertise in experimental psychology to the mix.