User needs and application area
User needs to be addressed and description of application
sites
Introduction: Problems with Scientific and Technical information
exchange on the Internet
The Internet has opened up important new opportunities for knowledge exchange
between scientific, technical, professional and other users. Some important such
knowledge techniques in the Internet are:
Search-oriented techniques like The World Wide Web, where
a vast amount of factual information is available to anyone searching for specific
kinds of information using hyperlinks, subject trees or subject structures and web
search engines.
News-oriented techniques like simultaneous computer conferencing
services such as the Usenet News, electronic journals and various e-mail mailing
lists. The EU Telematics project TAP Web4Groups is developing systems in this area
which provide better facilities than current techniques.
The difference between these two techniques is that search-oriented
techniques are mainly oriented towards a user who wants to search for specific
information already stored in computers or browse through knowledge depositories.
News-oriented techniques are mainly oriented towards distribution of news
within specific topic areas, and for a continuous, ongoing discussion and exchange
of ideas between people with special interests in various fields.
Sometimes, the user need is to find particular information on
particular topics, in other cases the need of the user is to update knowledge, keep
up-to-date with recent developments, and increase contacts with other people with
the same interests and speciality.
There is no sharp distinction that the WWW is used only for search-oriented
techniques and e-mail and conference systems only for news-oriented techniques, the
reverse also happens. It is however important to be aware of the difference, since
much of the experience in the field known as information retrieval is primarily
aimed at search-oriented, not at news-oriented usage. This means that some of the
knowledge from information retrieval area may not always be directly applicable for
news-oriented usage.
In news-oriented usage, a user is looking for something new but
of interest, so difference from previously known knowledge may be a valuable quality.
In search-oriented techniques, the user usually knows fairly well what he is looking
for and looks for information which match the question. Thus, similarity to known
documents is often a user need in search-oriented techniques, while too much similarity
(i.e. too little new) may be a non-requirement in news-oriented techniques.
All information available is not of high quality. Since anyone
is allowed to put up any information they like on the Internet, there is no quality
control (like that done by the editors and reviewers of magazines and journals).
Another related problems is that there is often too much information, making it difficult
for a user to find what is of most interest and relevance to that user. Users need
tools to handle their time, so that they can read the most valuable items within
the limited time they can spend on reading information.
Methods to help users find the best information
We believe that the free nature of the Internet is very important. Thus, it is
not our intent to implement techniques for censoring and forbidding information.
Instead, our intention is to develop and implement techniques to aid users of the
Internet in finding the information which is of highest quality and relevance for
the particular interests and needs of that user.
Many people and research groups have tried to tackle these issues
on the Internet in many different ways. For an overview. SELECT will tackle them
using two related techniques:
Rating is methods for users of the Internet to input their
evaluation of the quality of various messages, articles and web pages. These evaluations
are stored and used to aid other users to select documents to read. The interest
and knowledge profiles of the users, as shown by their evaluation, can be matched
with other users, so that the evaluations done for one user is used for aiding other
users with the same interests, values or knowledge.
Example 1: A user with a particular religion or political affiliation
may prefer to find information which has been highly rated by other people with the
same personal values.
Example 2: A specialist in an area may want to find high quality
information. Information which is of high quality for the specialist may be too complex
for a beginner or amateur. Information which the specialist finds trivial and unimportant
may be very valuable for a non-specialists who wants to learn the basics about a
particular topic.
In the Internet, rating may be applied to many kinds of objects,
like web pages, messages, electronic journal papers, public domain software.
The purpose of rating may be to increase the quality of the documents
you read, or to avoid certain documents deemed unsuitable in certain communities
for certain groups of readers (example: violence, pornography).
In the world before the Internet, rating was commonly provided
by services such as:
- Newspapers, magazines, books, which are rated by their editors or publishers,
selecting information which they think their readers will want.
- Consumer organisations and trade magazines which evaluate and rate products.
- Published reviews of books, music, theatre, films, etc.
- Peer review method of selecting submissions to scientific journals.
Information filtering: Methods to scan through new information arriving
in the Internet through group communication tools like computer conferencing, Usenet
News, e-mail mailing lists, electronic journals and new web pages. This scanning
has some similarities to information retrieval, but is also different in many aspects.
Filtering may be based on different criteria like:
- Keywords in the document.
- Semantic analysis of the document.
- Analysis of the stylistic and genre qualities of the document.
- Analysis of the similarities between the document and other documents which the
same user has rated highly.
- Documents directly related to other documents of high interest to the user, for
example by being replies or having hyperlinks to the document of interest.
- Rating of the document done by other people or by the author him/herself.
Note that filtering is not only a task of dividing all documents into two categories,
"good" and "bad", for a particular user. Often, what the user
needs is instead a list of documents sorted by a matching index. Also, the user often
wants the filter to sort information of interest into different areas or folders
representing different interests of that user.
The relations between rating and filtering is shown by the fact
that rating is also known under the names "social filtering" and "collective
filtering".
User needs to be addressed
Several of the partners in the SELECT consortium have already carried out studies
on user requirements:
DSV has performed a three-year study on information filtering.
In this study, investigation of user requirements was an important part. The most
important user needs reports in this project were [Lantz 1993, Lantz 1995, Fåhræus
1997]. The DSV study was performed on users of Usenet News, because this is an Internet
application where filtering needs are especially large. The methods used in the DSV
studies was (i) to collect groups of Usenet News users to discuss user requirements
and making notes from these discussions (ii) to ask Usenet News users to manually
filter articles and explaining how they decided which articles to read and not to
read (iii) by letting Usenet News users use a prototype of a newsreader with filtering
capabilities, and interviewing them on their experience from this usage. A full overview
of reports from this research project can be found at URL http://dsv.su.se/jpalme/fk/if_Doc/IntFilter.html.
The TAP Web4Groups has studied user requirements for rating. These
results are reported in [Scmutzer et al 1997, Irmay 1997]. (The Web4Groups project
will implement a limited system for collaborative rating in small user groups.)
One conclusion from these studies is that different users have
different filtering requirements. It is thus not possible to specify a single filtering
system to satisfy all users. The system must be able to adapt to the needs of different
users.
Trust
Important in the design of filtering systems is trust. A user is willing to let
a filtering system filter messages only if the user can trust the filter. Users,
who are afraid that the filter system will delete important messages without asking
them, are not willing to use the filtering system at all.
In the case of e-mail, for most users personally addressed messages
sent to them are of potentially high importance, and most users do not want a filter
to remove such messages automatically. E-mail also contains messages coming from
mailing lists. Some of these mailing lists are small closed groups of people doing
important work together, most users do not want such messages to be automatically
removed by a filter. Other mailing lists are large lists with many members, who exchange
information within the topic of the list. In some such lists, the activity is high,
and some users want the filter to select only the most important messages from such
lists.
In cases where the user is not willing to trust the filter to
automatically remove messages of less interest, the users prefer that the filter
orders the messages, so that the most interesting are shown first. The user can then
manually scan the list produced by the filter and make a final decision of which
messages to read and which to skip or defer reading of to a later time.
Users want to be in control. They want to be able to specify that
filters should only be applied to messages from some mailing lists, newsgroups and
forums, not to messages belonging to other, more important groups. When searching
using Internet search engines, many users want to control whether filtering is to
be used and what kind of filtering to use.
On the other hand, many users do not want to be troubled by the
need to specify how filtering is to be done. They just want to perform their searches,
and they will prefer to use the search engine which most often give them the information
they are searching for. Whether that search engine is using rating and filtering
is something these users are not interested in.
Different requirements from different users
Every user is different. Every user has different needs. One user is interested
in medieval religious beliefs, another is interested in particle physics. One user
wants an overview of the knowledge on a certain topic, another wants to find the
latest news. One user wants to get the maximum amount of information of value in
a limited time, another user wants to browse and entertain at leisure. One user is
an expert, another user is a novice, in the subject area they are retrieving information
on.
How then, can tools be designed to cater for all these differing
users with differing user needs? Because even though users are different, they are
common in that each user wants to find information of value to him or her. This is
the basic user need which this project will address. The goal of the project is not
to find "the" good information, according to some particular criterion
of goodness.
Our aim is to develop tools which will make it easier for each
Internet user to find the important and interesting information for that particular
user.
Here is a list of potentially conflicting user requirements:
- A user is getting too much information from the groups (mailing lists / conferences
/ newsgroups) subscribed to. And much of the information is not of interest. The
user wants to see only a selection of the items of highest interest.
- Another user may prefer to see all messages from a group, but sorted with the
most interesting items first in the list.
- Another user may want to see all messages in the ordinary chronological order
of threads, but wants each message shown with the important terms for this user highlighted
so that the user can rapidly manually decide what to read and what to skip.
- One user is not willing to do anything to aid the software in filtering. The
software should automatically, be looking at the user behaviour, deduce what is of
interest to this user.
- Another user is willing to classify items as interesting or uninteresting, or
in some other way, to aid the software in knowing what is interesting to this user.
- Another user is willing to explain in simple terms to the software why a certain
item was interesting or not.
- Another user prefers to see and set filtering conditions in a special language
for this.
Asynchronous group communication is an increasingly important tool for the exchange
of information and ideas between people in different places and countries. However,
there are also problems in this area. Many people feel that too much is written in
the discussion areas of interest to them, so that they do not have time to read all
the new items [Denning 1982, Palme 1984, Hiltz and Turoff 1985, Malone 1987]. They
also feel that the quality of the information provided is sometimes not high enough.
Too many uninteresting items are shown to them. These two problems are to some extent
two sides of the same coin, and one solution is information filtering.
Selection criteria
Filters should be capable of selecting items based on all attributes of items,
including attributes specially defined for special applications. Filters should also
be capable of selecting on attributes which are derived from the text of the message,
such as style and genre, degree of new information, etc. Example of basic attributes
for filtering are time flow (date and time in reference to other messages), author,
recipient, group to which the item was sent, topic, conversation/thread, subject
heading, keywords, relations to other items and text. A user may wish to give higher
priority to items which are direct or indirect responses to what the user him/herself
has written. Other filtering attributes can be special categories of items, such
as notifications, questions, replies etc. The degree to which semantic analysis of
the text is meaningful for selection will be investigated.
Filtering action
Based on selection criteria, the filter should be able to select items into categories
such as:
- Items to be read immediately
- Items to be saved for later reading
- Items belonging to different subject areas
- Items to be forwarded to someone else
- Items to be automatically processed by special software
- Items to be listed to the user, but then discarded unless the user specifically
overrides the recommendation of the filter
- Items to be discarded without showing them to the user
The choice of categories should be adjustable to the wishes of each user.
User interface and the creation of filtering conditions
Most existing filtering software like Elm [Taylor 1987], Procmail [Berg 1993],
MailFilter [Wyle 1992] are not easy to set up and require the user to prepare special
control files in a language that is not easy to understand for non-computer specialists
[McGough 1994]. It is easy to make a mistake in specifying the filtering conditions.
And such a mistake can have disastrous effects, e.g. automatically throwing away
important messages. The increasing usage of messaging by people whose speciality
is not in the computer area, will create a demand for filtering software that is
easier to use. This can be achieved by better user interfaces.
The risk of disastrous mistakes in specifying filtering conditions
might become even more of a problem if the filtering software aids the user in producing
filters, giving the user less direct control of the filtering conditions. It is important,
if people are to use filters, that they are able to trust the filters. One way of
achieving this, would be that new filtering conditions would in the beginning not
automatically trash messages, but rather put the deselected messages into a list
which the user can approve manually. Only when the user after some time of experience
is fully satisfied, the filter might trash messages without user final approval.
Studies on user behaviour when using filters show that filter
conditions set up by users are generally very simple [Mackay 1989]. This means that
advanced ways of specifying filters using complex logical language may be more of
a disadvantage than an advantage for most users, unless the complexity can be hidden
from the user by the user interface [Karlgren 1994C]. Viewing filter rules by frames
is usually easier for users than seeing them as logical rules in some programming-like
language.
Another way to make it easier for a user to specify filtering
conditions, would be to ask the user to input an evaluation on those messages which
the user believed were filtered wrong (e.g. a low evaluation on messages accepted
by the filter but not wanted by the user). To make this easy, the user interface
could be designed so that a single digit, input after reading a message, would be
interpreted as an evaluation on a scale from 0 to 9. The filtering program could
use this digit plus the message it applies to, to deduce a filtering rule, possibly
in co-operation with the user. This could be seen as an expert system, where the
filtering program in co-operation with the user builds an expert system data base
with filtering conditions for that particular user.
Some users want the filter to act automatically, deriving filtering
rules and improving its performance, or whether it should regularly communicate with
the user, asking for advice, showing prospective new filtering rules etc. The reason
users want filter is to make their life easier. Other users prefer regular contact
between filter software and user to increase user control of and user confidence
in the filter. Some users find it important to be able to inspect and understand
the filtering conditions, while others do not bother with this.
One user study (Fåhræus 1997) indicated that users
might prefer a filter which did not sort or reject documents, but which marked up
the documents or provided information to make it easy for the user to manually filter
the documents. This design choice will also be studied in this project.
User modelling
Important in filtering is to model different categories of users. The system will
be able to observe user behaviour and from this infer their needs, so that the system
adjusts itself to different user categories. This is especially important for new
and inexperienced users, since systems which do not explicitly model such users are
often too difficult to use for them.
Rating user requirements
There are many different kinds of rating with different user requirements. Some
examples:
- In a small, closed groups, users want to rate options. This kind of rating could
also be named voting or straw voting, and is handled by the TAP Web4Groups project.
This project will not develop support specifically for this kind of rating.
- People are employed, often also paid, for making ratings. This is very common
outside the electronic area, most newspapers and journals have some rating system
to decide what to publish and what to omit, even if they do not use the word rating
for what they are doing. A special case is the peer review system used for filtering
contributions to scholarly scientific and technical journals and conferences. In
the electronic publishing area, this kind of rating is applied by some search services,
the most well-known of which is Yahoo (see page 25). In Usenet News and e-mail mailing
lists, moderated groups publish only contributions which have been approved by one
or more moderators. A big disadvantage with such human moderators is the delay they
cause in publishing. In newsgroups and mailing lists, the time interval between one
message and a reply to it is often only a few hours, in moderated lists, this time
is lengthened to usually about a week. It is obvious that the rapid interaction in
discussions is severely hindered by this. On the other hand, the aid of moderators
is needed by many users who do not have time to read messages like "please remove
me from this mailing list".
- A variant of this is in education, where the teacher rates the submissions from
students.
- Instead of having special people who perform the rating, some systems allow anyone
or almost anyone to rate any document. They sometimes just use an average of these
ratings, but some systems (for example Firefly, see page 25) rate objects based on
other people who have similar tastes (views, values, competence) as the person the
rating is done for. A variant of this is to put people into different categories,
so that a user might specify that he prefers documents rated highly by other people
in his own category (political or religious group, scientist, etc.)
- The author of a document can provide his own rating. The advantage with this
is that more documents get rated, and that the ratings are easily transmitted with
the document. The disadvantage is that people may sometimes rate their own documents
too high. This disadvantage can be reduced by choosing a rating scale which does
not make such misratings easy to do, for example a scale with the values
9 = ph.d. thesis or equivalent
8 = accepted for publication in peer-reviewed scientific or technical
journal
7 = accepted for presentation at peer-reviewed scientific or technical
conference
6 = scientific research report
5 = other scholarly scientific och technical text
4 = popular science written by a scientist
3 = other newspaper or journal article
2 = discussion item in newsgroup, mailing list or other on-line
forum
1 = document of interest only to very few people.
We intend to develop support primarily of type 4 (but maybe also type 5) above.
Rating of kind 1-3 above is provided by other EU projects or proposals.
For rating of type 4, here are some user requirements:
- If anyone is allowed to submit ratings, there is a risk of misuse by people putting
in high ratings on their own documents, or collusion between two people putting high
ratings on their own documents. A check for the domain of the rater and the document
can stop ratings by people in the same domain, this is not a full protection. People
known to misuse the rating system in this way can be identified and put on a stop
list. Social codes that such misuse is not permitted may also help.
- A problem with such rating systems is how to get people to provide ratings. A
good solution to this problem is that used by for example Firefly, where you have
to provide your own ratings to get aid from the ratings data base. A variant of this
is that a filtering system may use the ratings by a user as a tool in developing
filtering conditions. Either the filtering system can automatically deduce the filtering
conditions by looking at the messages which a user rates high or low, or the filtering
system can ask the user (when needed, because the user rating did not agree with
already known filtering conditions) to specify some property, for example keywords,
to identify future message which the user wants to be filtered in similar ways.
- Important is that users, who so want, can control when rating is used or not.
- Some user studies have shown that users prefer not to rate documents into a scale
from interesting to uninteresting, but want to use several scales (for example a
scale of agree or disagree) or want to sort messages into folders/mailboxes for different
sets of messages. However, in a social filtering system where anyone can provide
rating on any document should not use more than one scale, because a major problem
is to get people to provide ratings, and this will be even more of an obstacle if
they have to provide ratings on several scales.
- Some users may want to read new documents as soon as they arrive, even if no
one has yet rated them. Other users may want to delay reading new documents in less
important categories for them, in order to wait for ratings by other people to arrive.
- We will also investigate whether ratings can be deduced from people's behaviour.
If this works, it can help in collecting ratings data.
Market situation and prospects
The amount of documents on the Internet is growing exponentially. The quality
and value of the documents is varying. The market for tools to aid users in finding
what is of highest value to them is very large. If message handling systems (e-mail
systems, news readers, non-simultaneous conference systems) and information retrieval
systems (like web search services) are equipped with features to help a user find
information of high interest, and to skip information with less interest, users can
be expected to choose to use such tools.
Internet users are already spending more than a thousand million
hours a year reading Internet documents. If this can be made only 10 % more efficient,
the benefit will be thousands of millions of ECU/year.
This benefit also will open a market for commercial companies
who provides this service to users. This market consists of the following products:
- Search services on the Internet: This market is today dominated by American
companies with services such as Alta Vista, Infoseek, Yahoo, etc. The competiton
is fierce in this market. A few European companies have entered this market, and
two of these companies, Euroseek (Euroseek search service) and OTM (Arianna search
service) are partners in this proposals. SELECT will give these companies a larger
market, because they will be able to give users, who so prefer, a better selection
of documents in the search results.
- Software tools for information filtering: Good such tools are obviously
useful, and there will thus be a market for good such software tools, probably as
add-ons to existing messaging or web retrieval software.
- Value-added information services for specific specialist groups: Two of
the partners in the SELECT proposal is MediBRIDGE/MGC and ISCN. MediBRIDGE provides
information to physicians and other people in the health care area. In Belgium today,
there are 34000 general practitioners, 8000 of these have a PC, many of these physicians
are already customers to MediBRIDGE. The market for this kind of information service
to physicians is presently increasing with 35 % per year. This is only for service
to one particular group of specialists, physicians, in one EU country, Belgium. Similar
services can be offered to other groups of specialists in other EU countries, which
means that the total future market for these services will be large. A user of the
medical discussion groups, which is one of the services of MediBRIDGE, receives on
average 35 messages/day. Only 4-5 of these are of primary interest. Even if a filtering
system is not able to automatically find only these 4-5 messages, i.e. providing
100 % recall and precision, a filtering system will still increase the value to the
users, and may also be a factor making the service worthwhile to subscribe to. MGC
is a Belgian organisation of physicians which will represent the users. ISCN is an
organisation of specialists on software quality in different European countries,
and they are an examples of the many networks of specialists which are expected to
benefit from this project.
More information about market situation can be found in section 10.1.4 Is there
a need for a new project on page 25
Work content
Phase of the project
Project methods
This project will pursue its work in the following ways:
- Collect a data base of documents and ratings on them made by a number of users.
This data base can be used to develop and test different filtering and rating methods,
without having to go back to the users every time a new method is to be tested.
- Define an architecture for filtering and rating with well-defined interfaces
between modules. This is important, because a good such architecture will allow partners
in different countries to develop different modules so that they can co-work well.
A first draft of such an architecture is provided in the chapter 1.4.2 Architecture
of the rating and filtering system on page 25.
- Develop a basic system, within the defined architecture or a subset of it, which
will not provide all advanced functionalities, but which can be used in real usage
to gain user experience. The basic system should be ready within 10 months of the
project start, and user tests with it should start not later than 12 months after
the project start.
- Based on the defined architecture, different partners can find existing software,
or develop new software, or modify existing software for the different modules. In
some cases, different partners in the project can find or develop alternate versions
of the same architectural module. For example, one partner might develop a module
for filtering based on semantic analysis, another might develop a module for filtering
based on similarity to other documents using the so-called cosine method. A competition
will thus exist within this project between different filtering methods, and the
users experience will decide which methods is best.
- Experts on Human-Computer Interaction can make studies, prototypes and user tests
to find good user interfaces.
- Combined systems of modules developed as described above can then be put up and
tested on users, and the user evaluations of different methods can be collected to
improve the methods.
Many of the partners in this project are organisations who have already done research
in the area of information filtering. An important part of this project is to combine
and test different filtering methods and evaluate how well they satisfy users. We
have chosen a two year project period because the market needs these services now,
and if we are too late, American services may have achieved dominance on the market.
Because of this short project period, the project is not split into phases which
succeed each other in time. Instead, the phases overlap, so that some development
can start in parallel with collection of user requirements and collection of the
test data base. Also the important phase for filter experiments will go on in parallel
throughout the whole project.
Filtering methods
Here are some filtering methods which will be developed and evaluated and tested
on users within the unified architectural framework we develop:
- Filters on keywords supplied by the user. New user interfaces will be tested,
where users can easily supply keywords in a dialogue with the software. Using this
method, the software will ask the user when needed questions like "Why did you
like/dislike this document" (when the user evaluation did not agree with what
the filter expected).
- So-called intelligent filters, by what is meant that the user supplies a quality
evaluation of documents, and the filter system deduces how to recognise high-quality
documents for this user in the future.
- Filtering on information in collaborative filtering data bases.
- Filtering on computer evaluations of the style and genre of documents.
Phases of the project
The major phases of the project can however be said to be:
- Collection of user requirements and specification of a base system based on existing
user requirements (work package U1 , S1, S2 and S3).
- Development of a simpler base system, which can be used by users but also as
a tool in testing rating filtering methods (workpackage D1, D2 and D4).
- Development of an advanced system, where the major difference from the base system
is that filtering is based on rates submitted by users with similar tastes, interests
and competence as the users for whom information is filtered (work package S4, D5,
D6, U4, D7).
- Parallel and continuous development and user testing of new filtering methods
throughout the project (tasks D2, U3, U4).
- External coordination and development of an exploitation plan (work package X,
M and C).
|
