07 Jan 2005 (permalink)

Several things are worth mentioning since the beginning of this year:

First, I have finally joined Ukrainian translation team. Fortunately, it did not require any paperwork as our usual disclamer by default covers any other language teams one may want to join. Would it be so with our GNU disclamer as well! I hate waiting for a snail-mail to arrive from Boston to me and for my reply to reach Boston again only in order for my 25-line patch to Emacs being finally applied to CVS (by the way it is a real case, Ted have sent it almost a month ago and I'm still waiting for it to arrive, and the patch to rmail is still waiting in the queue, although everybody knows how many assignments I have already made to GNU project... I wonder why would not RMS create a new assignment form for this and any other project whatsoever I may wish to join or create within GNU or (whatever else) in the future).

I've started to rant, it seems... Too bad.

Secondly, Mailutils finally has vacation extension action in Sieve. To be precise, we already had it as an extension to Scheme implementation of Sieve. I lacked time (or was too lazy?) to do the same for our Sieve library until Felix Egli sent us his version which didn't quite match our standards and was incompatible with the existed one, so I decided to finally write it:) Massive thanks, Felix!

Oh, yes, and the mess with diagnostic messages is fixed, too.

Another news in Mailutils is ODBC SQL driver, which actually means that from now on Mailutils is able to interact with virtually any existing DBMS (since any serious DBMS has an ODBC interface).

As usual most of the time is taking tar. After 1.15.1 release I have started a major rewrite of buffer handling and incremental archive parts.

Several fixes were applied to Anubis as well. Recent letters from Paolo made me finally gather my wits and set forth a draft describing the proposed method of handling multi-part messages.

This, in its turn, forced me to finally attack and solve the problem of using gendocs.sh script to generate web documentation in our projects. The resulting Makefile.am is very elegant. However, this patch must be applied to the script in order to use it. I'll submit the patch upstream tomorrow.

Take a look at the resulting manual page.

The only thing left with this approach is the absence of per-chapter html output, which I am going to fix shortly (it will require sending a new assignment form to Boston. At the end, I think, I'd better send quite a bunch of them beforehand).

21 Jan 2005 (permalink)

Ukrainian translation for www.gnu.org

Evgeniy Sudyr translated the main GNU site to Ukrainian. Of course it is now the default for www.gnu.org.ua.

Anubis MDA

I have implemented new MDA mode for Anubis. In this mode, it acts as a mail delivery agent, receiving incoming mail from the standard input, processing it and piping modified message to the real local mailer.

Two command line options must be specified in this mode: --mode=mda, telling Anubis that it should operate in MDA mode, and --local-mta, specifying the local mailer to use when actually delivering the message.

In my sendmail.cf I have:

Mlocal, P=/usr/local/sbin/anubis,
        S=EnvFromL/HdrFromL, R=EnvToL/HdrToL,
        A=mail --mode=mda \
            -l '/usr/local/libexec/mail.local -f %sender %recipient' $u

This allows me to first process incoming mail through Anubis and then through Sieve (invoked by mail.local)!

14 Feb 2005 (permalink)

EiFUB, which means "Everything is Fucked Up Badly".

Sorry about the language...

16 Feb 2005 (permalink)

Simultaneous access to mailboxes

How to handle a shrink [of a mailbox]? meaning, the &^$^@%#@^& user start two browsers and deleted emails in one session. My views is that we should scream bloody murder and hunt them with a machette. But for now just play dumb, but maybe the best approach is to pack our things and leave i.e exit()/abort().
-- Alain Magloire

Imagine the following synopsis: your imap4d daemon is working with a mailbox. Part of its data is cached into memory, any modifications to the mailbox are not flushed to disk until user explicitely requires it.

Now, the same user launches another mail client that opens the same mailbox, deletes some messages, and closes it.

The user makes some modifications to the mailbox in imap4d, requests EXPUNGE and closes the mailbox. Imap4d performs flush using the data cached in memory and, voilà - the mailbox is trashed.

You'd say the obvious solution is to lock the mailbox while it is being operating upon. But no, that won't work for imap4d, because appending to the mailbox is quite normal, and it should be allowed. Locking the mailbox will cause mail delivery to be blocked during the client's session (and as a sysadmin, I may say that the number of users that keep their imap session open for 24 hours a day is surprisingly big!)

Of course, the mailbox must be locked while it is being written to, and we do this, but that's another story.

In GNU mailutils we use a function called mailbox_is_updated to determine whether the mailbox was modified recently. The function returns 0 if it was modified and 1 otherwise. For plain UNIX mailboxes a simple approach is used: the function checks if the size of the mailbox file has changed.

It is assumed that if it the new size is greater than previously recorded one, than some new message were appended to the mailbox. If the size is less than the recorded one, then, well... see the quote at the start of this article :)

The method is week since:

Both cases will result in mailbox corruption.

And finally, the case of mailbox size shrinkage was still not handled in imap4d. Still, because the issue has quite long history: it dates back to April of 2001, when I first joined the Mailutils team.

Here is a short historical essay on the subject:

21 Apr 2001

I noticed that pop3d broke on simultaneous accesses to the mailbox. The story with pop3d is quite different, since it does use mailbox locking (well, that's to say now. Back then it still did not...), anyways:

... what seems to be a real problem is simultaneous access to
the maildrop. All locking/unlocking works just fine, but I have
come across the following situation: two sessions access the same
mailbox simultaneously, both do LIST, then one of them does
DELE on some message, then QUIT. The remaining session does a RETR
and dies miserably, since its internal indices are already out of sync
with the real maildrop contents. The comment on mbox_is_updated()
function addresses this but the function itself is never called.
So I added a call to mailbox_is_updated() to the pop3_mainloop()
function just before the processing of a keyword. Also I have added
ERR_MBOX_SYNC define to help handle the situation when
mbox_is_updated() bails out.
After these changes, the second session just gets the message

      -ERR Mailbox updated by other party or corrupt

and the connection is closed. This behaviour seems to be compatible
with RFC, but maybe it would be better to deny simultaneous
accesses altogether?
23 Apr 2001

Alain agreed to the proposed solution.

30 May 2001

Nice comment on a correctness of the name mailbox_is_updated. It raised a small discussion (see my next posting and Dawe's answer to it), but did not lead to renaming the function :)

As a side note, the story of linguistic inventions in bug-mailutils list is quite amusing... I certainly should catalogue them :^)

10 Nov 2001

In course of an unrelated discussion, Alain reminds that

We are trying to be smart by guessing that if the newsize is larger,
it is because a new mail was ___APPENDED___ to the mailbox so the offsets
maintained by the mailbox_t will still be sane and mailbox_expunge ()
will do the right thing (i.e. seeing that they was new mail when doing
the ftruncate())

13 Mar 2002

An unrelated issue, but still worth mentioning: on some machines disabling mmap speeds up pop3d. Rationale: mailbox_is_updated indirectly calls msync which makes the daemon spend quite some time in uninterruptible sleep.

16 Feb 2005

After a four-year delay I have added to imap4d support similar to that of pop3d.

The approach used by imap4d is a bit different from what we use in pop3d. Essentially, upon selecting a mailbox imap4d does the following:

The drawback of this approach is that all modifications to the mailbox the user might have made during the imap4 session are discarded. In my opinion it is a reasonable price for preserving the mailbox contents. However, this issue can be addressed too. More on this later.

While solving mailbox shrink problem, the method does not solve the described above weaknesses of mailbox_is_updated.

Correct Diagnosis is 50% of Treatment

To actually solve the issue mailbox_is_updated must be able to tell that the mailbox was modified, and how exactly it was modified, independently of its size. Unfortunately this is easier said than done. Currently mailutils is able to handle various mailbox formats and the way to do the proper diagnosis will vary with the format.

Let's first see what can be done for plain UNIX mailboxes. Obviously, mailbox_is_updated must first check mtime of the mailbox file. If it did not change, the function can return 1 immediately. Otherwise, it should assume some changes were done to the mailbox. Now, the question is whether these changes affect the cached data. The code in mailbox/mbox/mbox.c keeps offsets of UNIX 'From ' lines in the mailbox file. It will suffice to compare these with real offsets to decide whether the mailbox was modified (in the terms of libmailutils corrupted) or a new message was added to it. If any of the registered offsets differ, we scream bloody murder and raise MU_EVT_MAILBOX_CORRUPT event1) . In any case, we return 0 indicating that the mailbox has been changed.

A similar approach could be used for other mailbox formats.

As a side note, we could also prevent undesirable modifications of the plain UNIX mailbox if we apply kernel exclusive lock on the file from offset 0 up to its current size...

1) It may also be reasonable to invalidate all cached data in this case

17 Mar 2005 (permalink)
[image] Nunc est bibendum!
22 Mar 2005 (permalink)
Aequinoctium vernum! Gaudete!


There is also another good news, apart from the equinox. I have resurrected cflow project. It was initially written in 1997-1999 and fell into oblivion shortly afterwards. Somehow, I got distracted by other things... Now, a recent thread in gnu-devel mailing list has shown that there is a demand for this kind of program and this made me remember my past attemtps. So I have (quite happily) resurrected the package. During this week I have managed to turn it into GNU-compatible project. I have also sent a registration request to RMS.

The question is: where was my brain during these seven years? Another question is: how many useful programs still lie buried in my archives?

The program is compatible with POSIX specs and offers quite a number of improvements (of course it will offer more still:). Here is a sample output tree describing the project itself. The tree was obtained using chart.cflow rule in src/Makefile.am.

03 Apr 2005 (permalink)

"Potykaniem dobrym potykałem się, wiaręm zachował, zawodum dokonał. Na koniec przyznaczon mi jest wieniec sprawiedliwości."

Papież Jan Paweł II, wielki syn Narodu Polskiego, odszedł od nas. Cześć Jego pamięci.

15 Apr 2005 (permalink)


RMS has finally dubbed cflow a GNU package. I am now installing a repository on Savannah and working on other procedure issues (docs, web page, FSF directory entry etc.)

MIX language

There are quite some news in this respect. Firstly, I have prepared a patch to GNU MDK that fixes several incompatibilities between the language described by Dr.Knuth and the language accepted by MDK itself. These are:

José has agreed to include the patch into the future version of MDK

I have also returned to work on Douglas Laing MIX package. Douglas has kindly given me his permission to distribute the sources under GPL and has informed Dr.Knuth about the new package.

Why work on another MIX assembler/simulator when MDK is available? Well, there are quite a few reasons of doing so. While being a nice package, MDK does not quite satisfy my needs. What I want is a MIX assembler/simulator as closely conforming to TAOCP as is possible. Douglas's MIX meets this requirement, especially after I added back TTY device (by the way, I'm looking for a name for the new package, Conformant MIX seems to be quite a good choice).

Secondly, any MIX package is meant first of all for those who are studying TAOCP and are experimenting with the programs from its examples and excercises. This audience will certainly welcome a debugging feature. I have implemented MIX terminal mode, which, apart from remembering me IBM 360 console of the good ol' times, allows to debug MIX programs.

Another thing, given its special audience, a MIX package should be extremely educational itself. Again, MIX fits this description quite well. Its simulator is initially written closely following the guidelines set forth in TAOCP itself. Besides, it is written in WEB. And this brings me to the third objective of the package:

I want to rewrite the entire package in WEB. The primary motivation is explained above: it is to make sources well-documented and educational to the very extreme. There is a secondary motivation as well: I want to explore the limits of usability of CWEB tools, by pushing them to the very edge. At the risk of being accused of heresy, I must admit I doubt whether they could effectively be used for a real-world programming. Writing in literate programming style requires one to switch attention between TEX subtleties and writing good code, which is a difficult task (the new CWEB major mode for Emacs might help in this regard, though) and does not improve productivity. Besides I do not like much the way cweave formats large C data (e.g. arrays of structures). It is simply awful. So the purpose is to explore all pros et contras of CWEB tools.

The majority of MIX sources are rewritten in WEB. I have even managed to produce a valid YACC source file from a .w source (see mixal.w)!

The only two sources left are lex.l, which is downright impossible to write in WEB, and automatically-generated optab.c, which, being processed by cweave, looks like a pile of ..., well, garbage.

Trying to make my best to handle these, too...

Oh, well, and last but not the least: the package will provide a MIX simulator written entirely in Elisp. There exists a preliminary implementation already.


I have published my modifications to Warren Toomey's apout package. The modifications permit to run all programs from the earliest UNIX binaries found so far: so called 1972_stuff, binaries dated 1973, discovered by Dennis Ritchie.

Here is my message to Warren describing the details. It is interesting to notice the date, by the way.

Other things

Have you ever tried to download, compile and install a dozen and a half aspell dictionaries on one machine? If you ever need it, use this script.

18 Apr 2005 (permalink)


It was possible to write entire MIX in CWEB! Even including optab.c and lex.l!

However, I had to write a special post-processing program to handle lex and yacc sources, created by ctangle. Here it is.

Another good news: Dr.Knuth's home page already includes a link to MIX.

11 Aug 2005 (permalink)

Still here

I have had neither time nor desire to write any articles within the recent three months. Sure enough, there were plenty of news during this time. I am still a bit short of time for a detailed description (maybe next time), but a can provide a very brief summary:

13 Aug 2005 (permalink)
Cumpleaños felices, Fidel
30 Aug 2005 (permalink)

Straight to the business: first, the very first batch of fotos from Lwow is available (it probably will take me at least a couple of weeks to publish them all, at the present rate ...) Anyway, enjoy! If the gallery is missing a translation to your native language, just let me know.

Secondly, thanks to Wojciech's kind assistance, we have managed to bring mailutils namespace to a unified form. It is a great step for our team. Just think that currently mailutils codebase counts 193759 lines of C text, this will show you what it feels like to ferret out all this code, changing global symbols, without breaking anything. Should you ever plan to go into such a risky adventure, consider using this script. The docs are also available. If these are not enough, write me, I'll be glad to answer your questions.

To tell the truth, I have never thought I would wish to make my scripts repository available, but now I think I should do it.

Returning to mailutils, now that it has taken its new form, I have finally undertaken final API cleanup. First goes mu_folder_list and friends. Untill now it has been (1) non-recursive, and (2) used a kludgy struct folder_list, which seems to have been a leftover from the dark times when mu_list_t has not yet been there... So, this has changed. Now, one can write:

     mu_folder_t folder;
     mu_list_t list;

     mu_folder_create (&folder, dir);
     mu_folder_open (folder, MU_STREAM_READ);
     mu_folder_list (folder, dir, "*", 5, &list);

and obtain a list of all folders under dir up to fifth nesting level! If one is careful enough (or, if one knows pretty well what he/she is doing, which is the same thing, anyway), one may write:

     mu_folder_list (folder, dir, "*", 0, &list);

0 meaning infinite nesting level, i.e. recurse untill you have where to recurse.

Oh, well, I mustn't forget to tell Jocke about it! It was him who made me think about this folder business.

All in all, given all the recent changes to mailutils, I am inclined to think that the next release version will not be 0.x, but 1.0:) I am pretty ready to delay the release for a pair of months and to release a couple of alpha versions more, to achieve the result.

Oh, by the way, now mailutils interfaces with GNU radius. Strange, I haven't thought about it earlier...

26 Sep 2005 (permalink)

Cflow: heading for the release

The documentation for the GNU cflow project is finally available. I ask you to kindly take a look and report me your opinions, suggestions and the like.

Also, new alpha release is available for download. Give it a try!

Finally, the first stable release (version 1.0) is planned for this saturday.

27 Sep 2005 (permalink)

GNU.ORG.UA finally has new DNS!

Домени GNU.ORG.UA та TUHS.ORG.UA нарешті мають третій DNS. Огромна подяка хлопцям з Івано-Франківська та Адріанові Барану зі Львова!

Domeny GNU.ORG.UA i TUHS.ORG.UA nareszcie mają trzeci DNS. Wielkie dzięki chłopcom z Iwano-Frankiwska (Stanisławowa) i Adrianowi Baranowi ze Lwowa!

06 Oct 2005 (permalink)

I dag er Thor Heyerdahls gebursdagen. Jeg lengter etter ham nå som han er bort...

Det er en utmerket biografi om Thor på Kon-Tiki siden. Legg dessuten merke til valget av den vesentligste Nordmann, arrangert av NRK. Thor er en av kandidatene. Gi stemmen din til ham.