new cddb server software with UTF-8 support finally approaching

[Joerg Hevers]

Hello,

finally, after more than two years, a new version of the cddb server
software is almost ready. This new server software version introduces
a new protocol level 6. Clients using protocol level 6 will receive
all data in utf-8, instead of ISO-8859-1. This is a huge improvement,
since it finally allows CDs from China, Japan, Poland, Russia etc. to
be entered properly in freedb.
Additionally this new server software version finally allows rejection
mails to be sent for discid collisions and submissions with a revision
level that is lower than the revision of the existing entry.
Previously people kept wondering why their submissions wouldn't get
through in case of a discid collision, now they get to know what's
going on.
Both of these changes are features that were often requested!
The implementation of protocol level 6 was started by Edmund Grimley
Evans about two years ago and now finished by Jon F. Zahornacky and
myself. The sending of rejection e-mails in the cases mentioned above
has been implemented by Jon F. Zahornacky with a few bugfixes by
myself.

Full backwards compatibility has been maintained. To ensure that
entries with characters, that can only be represented in UTF-8 (which
are mostly shown as question marks to older clients), cannot be
overwritten by old clients, only submissions which were sent with a
UTF-8 charset headers are allowed to overwrite entries that contain
such characters. So new clients should always specify UTF-8 as charset
in the headers of the http-submission or the submission e-mail, even
if the submission contains only characters from the US-ASCII character
set.

Within the past two weeks the improved server software has been
tested and all bugs that we found have been fixed. It needs some more
testing, but I believe it's stable enough to warrant an announcement
in this list, so you can start playing with it if you like. If
everything goes as expected and nobody finds any bugs, I plan to get
this released before Christmas!

Currently no application supports the new protocol level, but I
suppose that will change quickly after the introduction of this new
version on the official servers. I can send a test application to
people who are really interested in testing this. It has been written
by Joshy, the author of CD'n'Go, about two years ago, when Edmund
implemented the basic utf-8 support, and I have permission to send it
to interested people.

The new version of the server software has been tested to compile on:

Linux 2.4.x (x86)
Linux 2.2.20 (Alpha)
Linux 2.4.19 (AMD64)
Cygwin 1.5.5
FreeBSD 4.9-PRERELEASE
NetBSD 1.6.1
Solaris 9 / SunOS 5.9 (Sparc)
MacOS X 10.x
HP-UX 10.20
Mips-SGI-Irix 6.5

For some of these system you need to change the makefile, since they
need some specific settings (FreeBSD, NetBSD, HP-UX, Solaris). You
just need to remove the "#" before the respective line - the required
settings are already in the makefile, just commented out.
If you have access to different systems, which are not listed here,
please try to compile the server software there and let me know if
there are any warnings/errors, that need to be fixed.

You can download a snapshot of the latest code from
http://www.freedb.org/internal/cddbd-1.5.tar.gz
and the new submit.cgi from
http://www.freedb.org/internal/submit.cgi-0.7.tar.gz

If you use the submit.cgi to submit utf-8 entries, make sure to change
the submit-addresses in it, to point to your local server.
The documentation in the cddbd-1.5.tar.gz package should be
up-to-date. If you find any mistakes or anything missing there, please
let me know!

If there's enough interest I might also set up my test server
properly, so you can use it to test the new features, even if you
don't have the possibility to set the server software up on your own
system. http-access won't be possible on that machine, though - only
cddbp access and e-mail-submissions and/or e-mail-commands. If you're
interested, just let me know!

If you have any comments or questions, have found bugs or tested the
software on a new platform, please let us know on this list!

- Joerg

[Joerg Hevers]

Hello,

well, doing a follow-up to my own e-mail, which I wrote a week ago. I
must say that I am a bit surprised, that we have a list explicitly for
people interested in the server software development, to which 120
people are subscribed, and not even one of you seems to be interested
enough to test a new server software beta version bringing interesting
new features, or at least comment/react in any other way to the
announcement.
I really don't mean to complain, but if an approaching server software
release is not interesting for people supposedly interested in server
software development (why else would you be subscribed to this list),
I wonder what the purpose of this list is or should be. Maybe we
should close it, which would save us admins the job of deleting the
spam via the web-interface every few days to prevent you from getting
it via this list.

Anyway - I have replaced the server software beta, which you can still
get from
http://www.freedb.org/internal/cddbd-1.5.tar.gz
with a slightly newer version (same filename). The biggest change is
an attempt to stop looping clients in cddbp-mode (did you ever wonder,
why the weekly and monthly statistics looked a bit strange sometimes? -
looping clients!). If a query for a discid is repeated or a read for a
discid from a category is repeated, the cddbp connection is closed by
the server with an error-message. No idea, if this really helps, but I
think it's worth a try.

Unless you guys (or someone else) find any bugs, my next goal will be
to update the freedb.howto to reflect the changes made in this new
server software version.

Joerg

[Joerg Hevers]

Hello,

and another follow-up to myself.

The documentation is ready and a new submit.cgi. For completeness I
have also uploaded the latest cddbd server code, but the changes are
only cosmetic.
For the submit.cgi I have dropped the quoted printable encoding, since
it made rejection notices too hard to read and confused users. I hope
that nowadays all mail-systems are 8bit-secure, so this shouldn't make
a difference.

You can get the new stuff here.

http://www.freedb.org/internal/cddbd-1.5.tar.gz
http://www.freedb.org/internal/submit.cgi-0.7.tar.gz
http://www.freedb.org/internal/freedb.howto.txt.gz

If anything is unclear in the freedb.howto or you think something
should be changed, now is the time to tell us!

Best regards, Joerg