Wednesday, December 9, 2015

Announcing QSoas, a powerful y=f(x) data analysis software

Why a new data analysis software ?

I'm a researcher at the interface between physics, chemistry and biology, and in our team, we pride ourselves on making the most of the data we acquire, especially through quantitative analysis and modelling. In fact, we spend a lot of time doing fitting simple formulas or complex differential equations to our data. As we were not really satisfied with the data fitting capacities of the software available, we've had our custom data processing/fitting tool, SOAS, for ages. However, that tool was hard to maintain (Fortran + Fortran libraries interfacing with X11 with ABI changing every once in a while without notice), impossible to port to non-X11 platforms, not very user-friendly, and not easy to extend at all. So, when I got my permanent position, a rewrote a completely new version from scratch, called QSoas using C++, Qt, Ruby and the GNU Scientific Library. The result is incomparably more powerful, more easy to maintain, more user-friendly, and more portable (I build it for Linux, Mac and Windows).

What does it do ?

The main features of QSoas are:
  • all standard data processing capacities (filtering, baseline subtraction, arithmetic transformations);
  • fully scriptable (can process hundreds of data files in one go);
  • easy to use data fitting interface, based on the recognition that initial parameters are very important for difficult fits, and that being able to change them easily is an absolute must (screenshot);
  • easy fit parameter save and reuse;
  • very powerful "global fits", with parameters that can be common to several datasets and others that can be dataset-specific (at the option of the user);
  • integration of differential equations and kinetic systems;
  • quick data browsing;
  • peak detection;
  • arbitrary fit functions, and many built-in fit functions with smart initial parameters detection;
  • infinite undo/redo and much more...

What has it done already ?

We've relied heavily on QSoas's functionalities for the past 3-4 years, and a great part of the team's publications just wouldn't be there without QSoas. More precisely on selected examples:
I hope it will also help you get more than you previously could from your data (and faster, too !).

I want it, where can I get it ?

You can download QSoas version 1.0 on its website. The source code is fully available under the GNU General Public License. For those not too compilation-savvy, we sell pre-built binaries for windows and mac, in collaboration with Satt Sud-Est and eValorix. Compilation under Linux is very simple, but I'm willing to come up with a Debian package, should some of you want that. You should definitely have a look at the tutorial and the command reference.

Monday, November 16, 2015

Purely shell way to extract a numbered line from a file

I feel almost shameful to write it down, but as it took me a long time to realize this, I'll write it down anyway. Here's the simplest portable shell one-liner I've found to extract only the, say, 5th line from file:

~ cat file | tail -n +5 | head -n1

Hope it helps...

Update: following the comments to this post, here are a couple of other solutions. Thanks to all who contributed !

~ cat file | sed -ne 5p
~ cat file | sed -ne '5{p;q}' 

The second solution has the advantage of closing the input after line 5, so if you have an expensive command, you'll kill it with a SIGPIPE soon after it produces line 5. Other ones:

~ cat file | awk 'NR==5 {print; exit}'
~ cat file | head -n5 | tail -n1 

The last one, while simpler, is slightly more expensive because all the lines before the one you're interested in are copied twice (first from cat to head and then from head to tail). This happens less with the first solution because, even if tail passes on all the lines after the one you're interested in, it is killed by a SIGPIPE when head closes.

Thursday, September 3, 2015

Releases 1.19.1 of Tioga and 0.13.1 of ctioga2

I've just released the versions 1.19.1 of Tioga and 0.13.1 of ctioga2. They both fix installation problems with recent versions of Ruby. Update as usual, though that isn't strictly necessary if you've managed to install them properly.
~ gem update tioga ctioga2

Saturday, June 13, 2015

Rescan-scsi-bus and slow DVD drives

For reasons that fail my, my internal SATA DVD drive is very seldom seen by the kernel at startup. My guess is that it takes very long to start, and the kernel doesn't wait that long before deciding that it had all SCSI devices, so it misses it. It's actually very annoying, since you can't use the drive at all. After digging around, I finally stumbled on the rescan-scsi-bus tool from the scsitools package. Running (as root, of course)
~ rescan-scsi-bus -w -l
(sometimes two or three times) is enough to get the device back up, with the /dev/dvd udev symlink.

Hope this'll help !

Thursday, June 11, 2015

Release 0.13 of ctioga2

Today is ctioga2's release. Unlike most other release, this one does not bring many visible features, but quite a few changes nevertheless, including:
  • finally customizable output PDF resolution, which was asked some time ago
  • ways to average successive Y values (for the same X value), setting the error bars to the standard deviation
  • handling of histograms with missing X values (SF issue #1)
  • improvements in the emacs mode (including contextual help)

As usual, the new version is available as a gem

~ gem update ctioga
Enjoy !

Sunday, June 7, 2015

Release 1.19 of Tioga

I've just released Tioga version 1.19. It fixes a couple of bugs, and finally brings the possibility to change the resolution of the target PDF file. A version of ctioga2 that needs this new feature should be published soon. Enjoy !

Monday, March 23, 2015

Release 0.12 of ctioga2

Out is the new version of ctioga2, which brings:
  • a much better handling of heterogeneous x,y coordinates in heat maps: ctioga2 now automatically splits the data into homogeneous segments;
  • control on the properties of the fill and the stroke of symbols (image)
  • decent improvement of error messages
  • and some bug fixes and other minor improvements
As usual, the new release is available as a gem:
~ gem update ctioga2
The website has also been decently improved, with now a search box for finding images in the gallery

Monday, January 26, 2015

Linux kernels for a macbook pro retina

I was unhappy about the recent Linux (3.14-3.16, and I think 3.17 too) kernels on my Macbook Pro Retina (15'), for a few reasons:

  • the nouveau graphics driver was not handling the graphics card very well (hangs when using the DRM after putting the computer to sleep once, garbage screen on various apps, slow 3D rendering), and I could never get the proprietary nvidia drivers to work (would give blank screen at boot time)
  • very unstable wireless (at least with my box home, but not with all the ones I've tried)
  • and the most painful was the need to recompile the kernel by hand, with the following modifications from stock debian kernel:
    +# CONFIG_X86_SYSFB is not set
    +# CONFIG_FB_SIMPLE is not set
    Without these modifications, the screen would be garbled some 5-6 seconds after boot (but SSH would still work, as far as I remember).
The latest 3.18-trunk kernel fixes essentially all the above problems, which is just great. Kudos to everyone involved ! Hope it helps...

Friday, January 2, 2015

Release 0.11 of ctioga2

The new year is starting with a new release of ctioga2, with a lot of new features, such as:

  • patterned fills (shown on the right)
  • loops in command files
  • a CSS-like styling system (with full XPATH support)
  • a backend for mathematical functions of two variables
  • the possibility to hide elements that have been drawn before, useful for making animations using output-now for instance
  • the possibility to have grid elements spanning two or more columns and/or rows
  • many other improvements and bug-fixes

The new release is of course available using rubygems:

~ gem update ctioga2

It can also be downloaded from sourceforge.

The possibilities of the new styling system are particularly interesting, and I'm working on ways to make it more powerful, and providing series of default style files that anyone could use as they want. Among other future changes, I want to improve the position of ticks, especially when using non-linear axes, and add functions to draw vector fields (though this still needs some thinking). Enjoy, and a happy new year to everyone !

New home for Tioga

Now, it's been a long time since Rubyforge has become unusable, and the Tioga project (on which ctioga2 is based) was still hosted there. I first wanted to thanks Rubyforge for hosting our projects for so long, and then say that Tioga is now maintained in sourceforge in a git repository. The web page is hosted by sourceforge as well, and that includes the documentation. A clone of the git repository is available on github. Enjoy !