How the Usenet News Protocols Work

Usenet News is a distributed computer conferencing system. It is interesting because of this age, early versions of Usenet News were operational already around 1980. To my knowledge no distributed conferending system existed before Usenet News, even if there were some non-distributed systems operating before Usenet News. such as Forum-Planet, KOM and Emisary. For many years, Usenet News was the largest world-wide conference service, even if it today is more and more getting replaced by more modern social internet systems, such as Facebook. But UsenetNews is not dead, it is still in operation when this is written (in the year 2016).     Conferences in Usenet News are called newsgroups, and messages are called articles. Usenet News is also used for dstribution of binary objects like object code, which is then encoded into a text-like format. Usenet News is one of the oldest distributed conferencing system in the internet, it got supported already in the year 1980. At that time, much of the communication was not done in the internet, but using various other protocols for communication between servers.    Topics of newsgroups are about computer usage, but also about cultural and life issues.    The basic principle of Usenet News is that local servers handle most of the functionality. Usenet News standardizes two variants of the NNTP protocols: One for communication between adjacent servers, one for communication between a client and a server. Each server can download as much as it wants of what is available on any of the adjacent servers. Loop control is handled both by a trace list and a list of the Message-IDs of received messages stored by each server, so that the server can reject the same message coming back again. This procedure for distribution of news can be compared to pouring water onto a flat horizontal surface; the water flows out in all directions (see the figure below).

"Pouring water" principle of Usenet News distribution

The figure below shows how new articles are forwarded from server to server in Usenet News. A server tells its adjacent servers which items it offers, the server requests those it has not already got via another route. Sometimes, there is only one way for news to be distributed between groups of servers, for example between Europe and America when Usenet News was young, and communication across the Atlantic was very expensive compared to how it is today.

This figure shows how new articles were originally forwarded from server to server in Usenet News.

Information about a user, such as how much this user has seen, is stored in the client. The server need not even know which users are using it. There are many different user-interface softwares for Usenet News, Some of them, of course, do not provide all the available functions.    A disadvantage with the procedure described in this figure is that it causes wait times between a IHAVE and a SENDME command. To avoid this, an alterative procedure much used today is to instead use two commands, CHECK and TAKETHIS. CHECK asks a server whether it wants certain articles, and TAKETHIS sends articles. The server which provides articles can use TAKETHIS to send articles, which it believes the receiving server wants, even if the receiving server has not asked for this. This allows for a stream of TAKETHIS commands sending probably new articles. Transmission is faster because the sending server can send articles which the receiving server already has. This might seem to be inefficient, but the advantage with streaming new articles is more valuable if more than 95 % of the articles are actually new.    In addition, Usenet News provides an interesting functionality which restricts communication to only those members of a newsgroup who work in the same organization or live in the same area or country. This functionality, however, is not used very much, and its existence is controversial, since it means that different users will get different views of the same newsgroup.    Usenet News has a cancel command, which can delete messages already sent out. Only the author of the cancelled message and the local newsserver administrator is allowed to cancel a message. Since, however, it is very easy to fake your identity, this command poses an obvious security risk, and the command is known to have been used to cancel messages for political reasons. The command is also used (not quite appropriate) by cancelbots, robots (= automatic programs) which cancel obvious spams by identifying messages with the same content sent to many disparate newsgroups. Usenet News also often has a Supersedes header field, which refers from a new message to an old message. This header usually cancels the old message.    Obsoletes in X.400 has some similarities to Supersedes in Usenet News, and can also be used to get an effect similar to the cancel command, by obsoleting a message with an empty message. However, cancel in Usenet News really deletes messages, while obsoletes is information to the recipient UA, which need not cause deletion. Many UA-s store both the new and the old version, so that the recipient can choose to see the obsoleted version if he so wishes.    The most important restriction of Usenet News is that closed groups are not well supported. For closed groups, moderated groups are used. For moderated groups, all contributions are sent via e-mail to the moderator, who accepts them and sends them for distribution. The moderator adds an approved header, and Usenet News softare will not accept any articles in moderated newsgroups, which do not have the approved header.    There are lots of ways for faking information in Usenet News, for example adding an approved header not provided by the moderator. But there are also robots continously spanning usenet hews to detect faked information and remove it. Because of this, faking is not a large problems in Usenet News.    The Usenet News protocol is called network news transfer protocol (NNTP) and is specified in RFC3977. The standard for the format of Usenet News articles is specified in RFC5536. The archicecture of Usenet News is specified in RFC5537. The table below lists the most common NNTP commands:

article [<Message-ID> | <Number>] Return text of designated article. If no parameter is given, the next article is returned. The current article pointer is put at the fetched article. body [<Message-ID>| <Number>] As article, but only returns body group <newsgroup> Go to the designated newsgroup head [<Message-ID> | <Number>] As article, but only returns head help Lists available commands ihave <messageID> Informs the server of an available article. The server can then ask for the article or refuse it. last Sets current article pointer to last message available, return the number and Message-ID. list [active | newsgroups | distributions | schema] Returns a list of valid newsgroups in the format: group last first newgroups <yymmdd hhmmss> ["GMT"] [<distributions>] List newgroups created since a certain datetime. "distributions" can be e.g. alt to only get newsgroups in the alt category. newnews <newsgroups> <yymmdd hhmmss> ["GMT"] [<distributions>] List Message-ID of articles posted to one or more newsgroups after a specific time. newsgroups can be. e.g. net.*.unix to match more than one newsgroups. distributions checks for articles which also has this other newsgroup as recipient. next Current article pointer is advanced. Returns number and Message-ID of current article. post Submit a new article from a client. check Asks a server whether it wants certain articles. takethis Sends new articles, many can be sent in succession and some of the articles may be articles which the receiving server already has.

There is a problem with the newsgroups header in a message sent via e-mail. Different systems use this header in two different ways: To indicate that this message has also been sent via Usenet News to the indicated newsgroups. To indicate that this is a personal reply, sent only via e-mail, to a message posted on the indicated newsgroups. Because of this problem, it is better to use the Posted-To header in e-mail to indicate that a message has also been sent to certain newsgroups, and e-mail recipients should ignore any newsgroups heading in an e-mail message. Traditionally, Usenet News has not been using the MIME standard for binary attachments like images. Instead of BASE64, UUENCODING or yEnc is often used in Usenet News to include binary attachments, even if MIME nowadays is also used very much. This did not exist in the beginning of Usenet News, of course, since Usenet News is much older than MIME. Handling binary attachments the MIME way is common in mostly textual discussion groups, which sometimes contain some binary code, like a picture, while yENC is mostly used in special newsgroups intended mostly for distribution of binary code (usually object code of programs). Because of message size restrictions, large attachments are often split into several messages in Usenet News. This also occurs in e-mail, but is more frequent in Usenet News, since some Usenet News servers try to save space by not accepting articles above a certain size limit. Both MIME and Usenet News have methods of indicating how a client can automatically combine parts into a complete message or attachments.