Rubyforge is dead, but ctioga2 goes on...

In the light of the frequent recent downtimes of rubyforge, and thanks to the information from @copiousfreetime, I finally decided to move the hosting of ctioga2 away from rubyforge, to sourceforge. Transition went smooth, git is now the only VCS. Code is hosted at sourceforge and mirrored at github.

Work goes on on ctioga2. I've recently implemented a decent support for histograms. It is already much more powerful than the one in the old ctioga, but it is far from being feature-full. Here's a preview.

I'm slowly preparing a new release for ctioga2, that would incorporate quite some fixes since the last time, and quite a few nice features in addition to the histograms. Stay tuned !

8 months with a macbook pro retina 15''

Back in February, I wanted a new laptop for my work, so I bought a fancy Dell Latitude E6540 laptop. Bad luck, it was giving my sore eyes and horrible headaches, to the point I couldn't work more than a dozen minutes with it. As in my work, we were not allowed to buy PC laptops with any other vendor, my boss convinced me to get a 15'' macbook pro retina (yes that's about as expensive a laptop as you can get). Although quite reluctant at first, I did need a new laptop, so I gave in. Of course, I had to install debian on it. I booted once or twice on macos to leave enough space for Debian and install rEFIt so that I could boot into it. I had to use version 3.9 of linux else it wouldn't boot.

After about 8 month, I must say I'm quite pleased with the beast. The retina display is very neat (a 2880x1800 resolution!), and it is pretty fast. There are two annoyances though:

• I never managed to get either the non-free NVIDIA drivers or the integrated intel graphics to work. The NVIDIA driver give blank screen on X startup, while it doesn't even start with the intel drivers (and I've tried all the tricks I could find around). The fine point is that I don't have any hardware 3D acceleration !
• The wireless network is pretty unstable.

Another point is the update of GRUB: Debian's post-install scripts don't play nice with the EFI setup, so here is what I have to do to update to a newer version of grub, (and it's also what I had to do initially to install GRUB):

# mount /dev/sda1 /mnt
# grub-install --efi-directory=/mnt

The /dev/sda1 partition is the so-called EFI partition. I think it is created by rEFIt and it is essentially empty.

Release 0.7 of ctioga2

Quite a bit of work has gone into ctioga2 since my last post. Among the most interesting features, ctioga2 now features:

• full windows support, including proper file types associations
• fill syntax as powerful as gnuplot's
• an impulses plot style
• unclipped plot elements along with full control of the curves depth: in front of axes, behind background lines...
• control of symbol size through a Z axis

The latter feature proved actually instrumental in a research project under way in my lab (which is why I wrote it in the first place). It is demonstrated in the image above, made using this code.

I've also started the gnuplot versus ctioga2 gallery I was speaking about in my last post, where I try to reproduce most of gnuplot's examples using ctioga2; it can be found here. While I don't like so much the look of the gnuplot graphs, and I try to keep the look of ctioga2's as close as possible to that look, it proved invaluable in detecting small bugs, fixing small glitches and providing inspiration for new features.

In the meantime, my todo-list for ctioga2 doesn't seem to get any smaller. Let's hope I'll implement them fast enough ! Keep posted...

Removing the background of an image

While in the process of polishing up a paper, I had to draw figures using a well-known proprietary program for drawing chemical formula whose name I will not write down, out of decency (but you know which one it is anyway). I'm using the mac version (yes, no version for Linux) which has a neat glitch in that the "Save As" button does not work at all: no way to export your chemical formula as a SVG/PDF/PNG picture or whatever ! So I resorted to the neat Print as PDF feature of macintosh. I had to edit the resulting PDF files by hand as some of the colors were wrong (great !), and when I tried to include the PDF, I realized they had a white background... No way !

So I dug in ImageMagick's convert documentation, and came up with the following code that converts the PDF file into a PNG with a transparent background:

convert -density 600 figure.pdf -channel alpha \
  -fx '((r == 1 && g == 1 && b == 1) ? 0 : 1)' figure.png

There are still some few points that look white but are probably not that white (due to antialiased rendering of the PDF file ?). There are many more things that can be done using the -fx operator, this page was helpful to me !

Edit: while the alpha channel seems to be on by default for PDF files, it is not necessarily the case for all images. If the above doesn't work, try adding -alpha Set before the -fx bit.

First release of Tioga that works on win32

While I usually don't announce releases of tioga, as it is now mostly in maintenance mode, with not a lot of new features, today is a different story. I finally got around to try and build it on windows, and after a bit of a fight, I got it to compile and work there (I got hit by the classical fopen(file, "w") for binary files, that don't make any difference on Linux and Mac, but that do on Windows...). The release is already available on the rubyforge download page. Gems will be available later on today.

To actually make it work on windows, here is what you need to install:

• Ruby, using the one-click installer. The 1.9.3 version works just fine. Don't forget to click the box to add the executables to the PATH
• the Ruby DevKit, at the same location. Make sure you take the one matching the Ruby version you installed (there's an explicative text on the side). You need to unpack it in the directory where Ruby got installed
• you need to get a LaTeX distribution, try out proTeXt
• at this point, you may need to reboot
• then, just open a windows terminal (Execute cmd.exe) and run:
  

  gem install tioga
  

And that's it ! That also means that ctioga2 works on windows, if you run:

gem install ctioga2

Announcing version 0.5 of ctioga2

I've just released version 0.5 of ctioga2, my plotting program. While there is still quite a lot of improvements I'd like to get in some day, the past few days have seen a lot of activity. Probably the most important change is the new space-separated command file format, much simpler than the old one (which is still supported, although marked clearly as deprecated now). Another important change for the users is the large improvement of error reporting: in most cases, ctioga2 pinpoints the exact location where things go wrong; it also features reporting on LaTeX errors. The last important feature is also the introduction of real contour plots, such as the one below, coming from the documentation:

Now, I feel like ctioga2 is coming close to gnuplot in terms of features and ease of use. Part of my plans for the website is to create a gnuplot mockup gallery, in which I would take as many examples from the gnuplot website as possible (ie all the ones ctioga2 can reproduce) and compare the code and look of both.

Among features I'd like to implement for ctioga2, here are the ones that I think are the most important ones:

• plot styles (histograms, bars, impulses), in the spirit of the plot with instruction of gnuplot
• complex axes: broken axes and linked axes
• full control of the tick labels and positions (alternative formats, placing ticks at multiples of π, and labelling them as such...)
• full styling using CSS-like stylesheets
• a windows port would be nice, too

But maybe you have other ideas you'd like to see before that ?

Importing lots of files into git-annex

If, like me, you find git-annex very appealing, but have been moving files without it for a while, here is a little script that may help you through the transition. I've just converted my MP3 library to git-annex, but, now I must convert all my copies of the library on various computers to track one (or more) of them. The trick is to teach git-annex about the already existing files, as re-downloading a 20GB library through a 80 KB/s upload DSL line doesn't sound that funny. Here is how I did it.

First, convert one repository to using git-annex. Then, switch to another computer, move away the directory containing the files to a backup directory, and git clone the first repository. Then, from the directory created, run:

~ ./import path/to/old/backup/copy

Where import is the following script:

#! /bin/bash

src="$1"

IFS=$'\n' 
for f in $(find -type l); do 
    if stat -L "$f" >/dev/null 2>/dev/null; then
        echo "File $f ok"
    else
        tg=$(readlink "$f" | sed 's/.*git/.git/' )
        dir=$(dirname "$tg")
        if [ -r "$src/$f" ]; then
            mkdir -p "$dir"
            cp -avl "$src/$f" "$tg"
        fi
    fi
done

Once this is done, you'll need to run

~ git annex fsck

Of course, there is no warranty !! It saved me lots of download time, but it could irretrievably damage your data, put your dog in danger or set fire to your house, so use with care !

Introducing dvdcopy, an error-resistant command-line DVD copier

My wife likes watching DVDs a lot, which is why she borrows many from the public library. The problem is that many of those were handled by people who have no clue whatsoever about the proper handling of DVDs, which means that they are in a state ranging from moderately scratched in the best cases to properly scarred in the worst ones... xine or VLC fail miserably to read such DVDs. What's worse is that failures happen right in the middle of DVDs, which is quite frustrating !

That's why I've started quite some time ago to write a small C wrapper around libdvdread to first copy the DVD files onto the disk and then read them from there. It was also an easy solution to copy files to a driveless netbook to watch the DVD there. With time, this project evolved into a neat C++ command-line utility that's able to keep track of badly read sectors and come back to them later.

Today is the 0.0 release of dvdcopy. You can get the tarball there. It follows the standard autoconf dance:

~ ./configure
~ make
~ make install

Using it is as simple as running:

~ dvdcopy /dev/dvd MyMovieDirectory

If everything went fine, you're set, and you can open the directory created with vlc or xine:

~ xine dvd://`pwd`/MyMovieDirectory

(yes, xine needs the full path, I haven't checked with VLC, but from what I remember, you can open DVD directories directly from a menu).

If you're not that lucky and dvdcopy found some bad sectors, try reading again using

~ dvdcopy --second-pass /dev/dvd MyMovieDirectory

I found that in many cases, one or two second passes get all the sectors back. While that wasn't really the intended purpose, the fact that most copy protection schemes (in addition to CSS) are implemented using bad sectors make dvdcopy quite resistant to them.

I have no plan of making a Debian package for it at the time being, but if there is demand for that, I may reconsider. Enjoy !

Lots of hard drive accesses with MacOS

Sorry, a post about MacOS again, but this is a problem my boss has been faced with, and judging by what google searches gave us as clues to go on, I thought it would be useful for others to know about this.

So here it is: for a couple of weeks, my boss's Mac had been slowing down very painfully, to the point that he was just sitting in front of it waiting for web pages to show up or applications to start... We tried a whole bunch of things, such as disabling spotlight, dropbox, and even the time machine, but nothing worked. Finally, he had a look at Disk Utility that said that the disk was gone beyond repair. Fortunately, a new colleague suggested to restore from the Time Machine, holding Command-R during startup. It took ages (over a slow network), but now the machine is up again, without any slowing down.