Announcing QSoas, a powerful y=f(x) data analysis software

Why a new data analysis software ?

I'm a researcher at the interface between physics, chemistry and biology, and in our team, we pride ourselves on making the most of the data we acquire, especially through quantitative analysis and modelling. In fact, we spend a lot of time doing fitting simple formulas or complex differential equations to our data. As we were not really satisfied with the data fitting capacities of the software available, we've had our custom data processing/fitting tool, SOAS, for ages. However, that tool was hard to maintain (Fortran + Fortran libraries interfacing with X11 with ABI changing every once in a while without notice), impossible to port to non-X11 platforms, not very user-friendly, and not easy to extend at all. So, when I got my permanent position, a rewrote a completely new version from scratch, called QSoas using C++, Qt, Ruby and the GNU Scientific Library. The result is incomparably more powerful, more easy to maintain, more user-friendly, and more portable (I build it for Linux, Mac and Windows).

What does it do ?

The main features of QSoas are:

• all standard data processing capacities (filtering, baseline subtraction, arithmetic transformations);
• fully scriptable (can process hundreds of data files in one go);

• easy to use data fitting interface, based on the recognition that initial parameters are very important for difficult fits, and that being able to change them easily is an absolute must (screenshot);
• easy fit parameter save and reuse;
• very powerful "global fits", with parameters that can be common to several datasets and others that can be dataset-specific (at the option of the user);
• integration of differential equations and kinetic systems;
• quick data browsing;
• peak detection;
• arbitrary fit functions, and many built-in fit functions with smart initial parameters detection;
• infinite undo/redo and much more...

What has it done already ?

We've relied heavily on QSoas's functionalities for the past 3-4 years, and a great part of the team's publications just wouldn't be there without QSoas. More precisely on selected examples:

• we used advanced fitting (fitting derivatives along with data) in Fourmond et al, JACS 2013;
• we used heavy batch processing and global fitting for Hajj et al, EES 2014;
• we used the integration of kinetic systems in Orain et al, JACS 2015;
• we made heavy use of the global fitting capacities in Ceccaldi et al, BBA 2015...

I hope it will also help you get more than you previously could from your data (and faster, too !).

I want it, where can I get it ?

You can download QSoas version 1.0 on its website. The source code is fully available under the GNU General Public License. For those not too compilation-savvy, we sell pre-built binaries for windows and mac, in collaboration with Satt Sud-Est and eValorix. Compilation under Linux is very simple, but I'm willing to come up with a Debian package, should some of you want that. You should definitely have a look at the tutorial and the command reference.

Purely shell way to extract a numbered line from a file

I feel almost shameful to write it down, but as it took me a long time to realize this, I'll write it down anyway. Here's the simplest portable shell one-liner I've found to extract only the, say, 5th line from file:

~ cat file | tail -n +5 | head -n1

Hope it helps...

Update: following the comments to this post, here are a couple of other solutions. Thanks to all who contributed !

~ cat file | sed -ne 5p
~ cat file | sed -ne '5{p;q}' 

The second solution has the advantage of closing the input after line 5, so if you have an expensive command, you'll kill it with a SIGPIPE soon after it produces line 5. Other ones:

~ cat file | awk 'NR==5 {print; exit}'
~ cat file | head -n5 | tail -n1 

The last one, while simpler, is slightly more expensive because all the lines before the one you're interested in are copied twice (first from cat to head and then from head to tail). This happens less with the first solution because, even if tail passes on all the lines after the one you're interested in, it is killed by a SIGPIPE when head closes.

Releases 1.19.1 of Tioga and 0.13.1 of ctioga2

I've just released the versions 1.19.1 of Tioga and 0.13.1 of ctioga2. They both fix installation problems with recent versions of Ruby. Update as usual, though that isn't strictly necessary if you've managed to install them properly.

~ gem update tioga ctioga2

Rescan-scsi-bus and slow DVD drives

For reasons that fail my, my internal SATA DVD drive is very seldom seen by the kernel at startup. My guess is that it takes very long to start, and the kernel doesn't wait that long before deciding that it had all SCSI devices, so it misses it. It's actually very annoying, since you can't use the drive at all. After digging around, I finally stumbled on the rescan-scsi-bus tool from the scsitools package. Running (as root, of course)

~ rescan-scsi-bus -w -l

(sometimes two or three times) is enough to get the device back up, with the /dev/dvd udev symlink.

Hope this'll help !

Release 1.19 of Tioga

I've just released Tioga version 1.19. It fixes a couple of bugs, and finally brings the possibility to change the resolution of the target PDF file. A version of ctioga2 that needs this new feature should be published soon. Enjoy !

Linux kernels for a macbook pro retina

I was unhappy about the recent Linux (3.14-3.16, and I think 3.17 too) kernels on my Macbook Pro Retina (15'), for a few reasons:

• the nouveau graphics driver was not handling the graphics card very well (hangs when using the DRM after putting the computer to sleep once, garbage screen on various apps, slow 3D rendering), and I could never get the proprietary nvidia drivers to work (would give blank screen at boot time)
• very unstable wireless (at least with my box home, but not with all the ones I've tried)
• and the most painful was the need to recompile the kernel by hand, with the following modifications from stock debian kernel:
  

  -CONFIG_X86_SYSFB=y
  +# CONFIG_X86_SYSFB is not set
  -CONFIG_FB_SIMPLE=y
  +# CONFIG_FB_SIMPLE is not set
  

  Without these modifications, the screen would be garbled some 5-6 seconds after boot (but SSH would still work, as far as I remember).
  

The latest 3.18-trunk kernel fixes essentially all the above problems, which is just great. Kudos to everyone involved ! Hope it helps...

New home for Tioga

Now, it's been a long time since Rubyforge has become unusable, and the Tioga project (on which ctioga2 is based) was still hosted there. I first wanted to thanks Rubyforge for hosting our projects for so long, and then say that Tioga is now maintained in sourceforge in a git repository. The web page is hosted by sourceforge as well, and that includes the documentation. A clone of the git repository is available on github. Enjoy !