Monday, February 8, 2010

The java packaging nightmare...

... or why I mutilated your java package in Debian

This post is especially addressed to what we Debian Developers refer to as upstream maintainers, that is the people who write/maintain the software that we package for Debian. It is meant to explain why, in some cases, Debian Developers prefer to mutilate your work rather than upload it to Debian Archives in a state that gives it proper credit. And to apologize.

These opinions are my own, but I have a faint feeling that I'm not the only Debian Developer working like that, and that this few words could help soothe the relation betwenn Debian Developers and Upstream Maintainers, which can be quite tense some times...

It happens unfortunately quite often (too often!) that I disable (or don't enable) features in programs I package. Exact reasons vary, but they essentially fall into two categories:

  • A feature requires code that does not comply to Debian Free Software Guidelines, which Debian cannot distribute. This is a no-go.
  • A relatively minor feature depends on other software that are not packaged yet, and in which I don't have specific interest, which means I would not be able to maintain them properly (or it would require too much packaging work and so on...).
When a program has features which falls in one of these categories, I have two choices: either I give up packaging the software or I give up some of the features... For packages for which I care, obviously the second option looks better from my point of view as a Debian Developer.

Java packages are especially affected by this problem, for a very simple reason. Java is one of the few (I don't say only because I'll probably be flamed to death) programming languages which allows very easy distribution of binary platform-independent libraries, in the form of JAR files. Many projects simply reuse and embed JAR files published for other projects in their own JAR files, often without precise references to where the source code can be found. Packaging a moderately-sized Java application often rely on dozens of embedded JARs, which we need to remove from the packages we build (as Debian does not distribute binaries without sources for many reasons), and for which we need to hunt down the original source code and often packaging the original project, which might pull in yet other dependencies and so on... A true nightmare ! That's why it is so tempting some times to simply drop some of the code, possibly write patches to disable references to removed libraries, and be done with it.

Therefore, I would wish to publicly apologize to any upstream developer I've offended by stripping down apparently carelessly their beautiful software before uploading to the Debian Archives; I hope they now understand why I've done that.

Note to users: it is not because one of the features of your favorite software has been disabled by its (Debian) maintainer that you need to turn away from Debian ! Rather, file a bug report against the package. You'll get the reasons why the feature was disabled, and you have a chance to convince the maintainer to switch it back on.

12 comments:

directhex said...

The situation you describe is also common with Mono, which is obviously a very similar platform in this regard. The worst offenders seem to be web apps, which is why there are no ASP.NET apps in the archive yet.

Vincent Fourmond said...

On behalf of Warren Overholt:
As a sysadmin dealing with a java centric environment (mostly on Solaris), I share your pain. We get frequent requests to install this (.jar|.war|.ear) with no sense of where it came from. When the code breaks (not if), everyone is left scratching around trying to figure out versions. Much of the java development runs in the "use the current" approach so how do you revert to where you were 3 weeks or 6 months ago?

I view it as a fundamental flaw that at a minimum version numbers are not required in (.jar|.war|.ear)'s. Open Source also requires a package source, in the commercial world that would simply be nice.

I think your post would be stonger if it addressed the lack of versions as well.

Armagon said...

That was a very sound explanation, thank you. Not the typical "doesn't work" - but it actually has valid reasons.

Thomas Koch said...

We've had a very similar discussion at fosdem two days ago: http://www.fosdem.org/2010/schedule/events/java_debian_packaging. One idea is, to create and promote a documentation for java upstream on how to be a good upstream

Torsten Werner said...

"One idea is, to create and promote a documentation for java upstream on how to be a good upstream"

very good idea

directhex said...

@Vincent Fourmond

Fortunately, that's one issue that's NOT shared with Mono - it's very very simple to version your libraries in a way that's enforced by linking - and to carry compatibility data around allowing mismatched versions to function when (and only when) they're considered completely compatible. As an example, The current gtk-sharp package in the archive is version 2.12.9, which identifies itself as 2.12.0.0 as far as any apps linking against it are concerned, and carries "Policy" files with it which mark it as capable of running any app linked against 2.10.0.0, 2.8.0.0, 2.6.0.0, 2.4.0.0

slashdotaccount said...

I don't suppose you'd be willing to mutilate this Java-based game?

http://bugs.debian.org/551873

Not as pretty as the thing it clones, but fun nonetheless.

Torsten Werner said...

@directhex

Some new kind of versioning is in the works for Java 7.

http://blogs.sun.com/mr/entry/jigsaw

directhex said...

@Torsten Werner

Yeah, I'd heard they had lots of plans to make the tracking of dependencies better behaved - but like the plan to move the classlib into human-sized chunks, it seems a bit up in the air for JDK7

Vincent Fourmond said...

@slashdotaccount:

Nice try ;-)... Why don't you package it ? I'd be happy to review/upload...

Henri said...

Very interesting article. As one of the founder of JPackage project, I know how hard it is to repackage a consistent distribution.

BTW, you miss some important point :

- Repository in Java allready exist, Maven like (local or remote) and OBR, so there is no real need to provide a duplicate on native OS.

- Often developpers need to use, at the same time, differents versions of the same library, and it could became a nightmare to handle all the cases in a packaging project. On native side, a package update should do a replace of previously installed, in Java it's not the case

I think Java Package for OS (Unix flavors and Windows) should primary ensure perfect integration on OS of Java runtime apps (ie: Tomcat, JBoss, ActiveMQ), so DevOps could just see them as native components.

It's allready a huge works since only on Linux, packages should cover DEB/RPM (and variants) and differences between distributions.

I see no reason why there couldn't be a cross packaging project covering these needs :)

Emmanuel L├ęcharny said...

Don't mutilate Java packages, or otherwise you'll fell again in a trap Debian peeps have already been caught in : http://www.links.org/?p=327

Otherwise, just do what we do at Apache : create a debian repository where the validated Java packages are stored, and use it to build new packages. With most of the current Java project using Maven those days, that should not be a big deal...