Thursday, February 4, 2016

Making oprofile work again with recent kernels

I've been using oprofile for profiling programs for a while now (and especially QSoas, because it doesn't require specific compilation options, and doesn't make your program run much more slowly (like valgrind does, which can also be used to some extent for profiling). It's a pity the Debian package was dropped long ago, but the ubuntu packages work out of the box on Debian. But, today, while trying to see what takes so long in some fits I'm running, here's what I get:
~ operf QSoas
Unexpected error running operf: Permission denied

Looking further using strace, I could see that what was not working was the first call to perf_event_open.
It took me quite a long time to understand why it stopped working and how to get it working again, so here's for those of you who googled the error and couldn't find any answer (including me, who will probably have forgotten the anwser in a couple of months). The reason behing the change is that, for security reason, non-privileged users do not have the necessary privileges since Debian kernel 4.1.3-1; here's the relevant bit from the changelog:

  * security: Apply and enable GRKERNSEC_PERF_HARDEN feature from Grsecurity,
    disabling use of perf_event_open() by unprivileged users by default
    (sysctl: kernel.perf_event_paranoid)

The solution is simple, just run as root:
~ sysctl kernel.perf_event_paranoid=1

(the default value seems to be 3, for now). Hope it helps !

Monday, January 25, 2016

QSoas tips and tricks: using the fit interface to explore parameters to a function

As I said before, one of the very strong points of QSoas, my data analysis software, is its fitting capacities, and its fitting interface. Today, as a first post in a series of tips and tricks about QSoas, I will show you that the fit interface can also be used for a quick exploration of the effect of parameters on a given function. For that, I have chosen a simple system, the Van der Pol oscillator. The differential equation can be written for QSoas in the following form (in a file named vanderpol.ode):

x = x_0
y = dx_0

d_x = y
d_y = mu * (1 - x**2) * y - x

Note the blank line between the initial conditions and the expression for the derivatives. This is the representation of the form using y as the derivative of x. The integration variable in QSoas is the time t. You can learn more about how to specify differential equations in QSoas from the manual. The game is now simply to use fit-ode to launch the fit interface, but before that, one needs data to serve as a template, generated using generate-buffer:

QSoas> generate-buffer 0 100 1
QSoas> fit-ode vanderpol.ode

Now, the fit interface looks like this:

You can play around with the parameters, and in particular the value of mu... For every modification, hit the Update button, or just Ctrl+U, to see the effect. For data you'd wish to take a closer look at, use the Push current to stack action from the Data... menu, which creates new buffers on the stack. A nice thing is that QSoas keeps the fit parameters as meta-data, which you can see later on using show:

See the values of mu in the terminal above ?

This feature of QSoas makes it easy to quickly explore the effect of the parameters of a model on the shape of the resulting curve. I have used that very heavily in my research, I hope others will find it useful too. Enjoy !

Monday, January 11, 2016

Ghost in the machine: faint remanence of screen contents across reboots in a Macbook pro retina

As was noted a few times before, I happen to own a Macbook Pro Retina laptop I mostly use under Linux. I had noticed from time to time weird mixes between two screens, i.e. I would be looking at a website, but, in some areas with uniform colors, I would see faint traces of other windows currently opened on another screen. These faint traces would not show up in a screenshot. It never really bothered me, and I attributed that to a weird specificity of the mac hardware (they often do that) that was not well handled by the nouveau driver, so I had simply dismissed that. Until, one day, I switch off the computer, switch back on, boot to MacOS and see this as a boot screen:
Here is a close-up view of the top-left corner of the screen:
If you look carefully, you can still see the contents of the page I was browsing just before switching off the computer ! So this problem is not Linux-specific, it also affects MacOS... To be honest, I don't have a clue about what's happening here, but it has to be a serious hardware malfunction. How can two video memory regions be composed upon display without the computer asking explicitly for it ? Why does that problem survives a reboot ? I mean, someone switches on my computer and can see the last thing I did on it ? I could actually read the address line without difficulty, although you'll have to take my word for it, since the picture does not show it that well. That's scary...

Wednesday, December 9, 2015

Announcing QSoas, a powerful y=f(x) data analysis software

Why a new data analysis software ?

I'm a researcher at the interface between physics, chemistry and biology, and in our team, we pride ourselves on making the most of the data we acquire, especially through quantitative analysis and modelling. In fact, we spend a lot of time doing fitting simple formulas or complex differential equations to our data. As we were not really satisfied with the data fitting capacities of the software available, we've had our custom data processing/fitting tool, SOAS, for ages. However, that tool was hard to maintain (Fortran + Fortran libraries interfacing with X11 with ABI changing every once in a while without notice), impossible to port to non-X11 platforms, not very user-friendly, and not easy to extend at all. So, when I got my permanent position, a rewrote a completely new version from scratch, called QSoas using C++, Qt, Ruby and the GNU Scientific Library. The result is incomparably more powerful, more easy to maintain, more user-friendly, and more portable (I build it for Linux, Mac and Windows).

What does it do ?

The main features of QSoas are:
  • all standard data processing capacities (filtering, baseline subtraction, arithmetic transformations);
  • fully scriptable (can process hundreds of data files in one go);
  • easy to use data fitting interface, based on the recognition that initial parameters are very important for difficult fits, and that being able to change them easily is an absolute must (screenshot);
  • easy fit parameter save and reuse;
  • very powerful "global fits", with parameters that can be common to several datasets and others that can be dataset-specific (at the option of the user);
  • integration of differential equations and kinetic systems;
  • quick data browsing;
  • peak detection;
  • arbitrary fit functions, and many built-in fit functions with smart initial parameters detection;
  • infinite undo/redo and much more...

What has it done already ?

We've relied heavily on QSoas's functionalities for the past 3-4 years, and a great part of the team's publications just wouldn't be there without QSoas. More precisely on selected examples:
I hope it will also help you get more than you previously could from your data (and faster, too !).

I want it, where can I get it ?

You can download QSoas version 1.0 on its website. The source code is fully available under the GNU General Public License. For those not too compilation-savvy, we sell pre-built binaries for windows and mac, in collaboration with Satt Sud-Est and eValorix. Compilation under Linux is very simple, but I'm willing to come up with a Debian package, should some of you want that. You should definitely have a look at the tutorial and the command reference.

Monday, November 16, 2015

Purely shell way to extract a numbered line from a file

I feel almost shameful to write it down, but as it took me a long time to realize this, I'll write it down anyway. Here's the simplest portable shell one-liner I've found to extract only the, say, 5th line from file:

~ cat file | tail -n +5 | head -n1

Hope it helps...

Update: following the comments to this post, here are a couple of other solutions. Thanks to all who contributed !

~ cat file | sed -ne 5p
~ cat file | sed -ne '5{p;q}' 

The second solution has the advantage of closing the input after line 5, so if you have an expensive command, you'll kill it with a SIGPIPE soon after it produces line 5. Other ones:

~ cat file | awk 'NR==5 {print; exit}'
~ cat file | head -n5 | tail -n1 

The last one, while simpler, is slightly more expensive because all the lines before the one you're interested in are copied twice (first from cat to head and then from head to tail). This happens less with the first solution because, even if tail passes on all the lines after the one you're interested in, it is killed by a SIGPIPE when head closes.

Thursday, September 3, 2015

Releases 1.19.1 of Tioga and 0.13.1 of ctioga2

I've just released the versions 1.19.1 of Tioga and 0.13.1 of ctioga2. They both fix installation problems with recent versions of Ruby. Update as usual, though that isn't strictly necessary if you've managed to install them properly.
~ gem update tioga ctioga2

Saturday, June 13, 2015

Rescan-scsi-bus and slow DVD drives

For reasons that fail my, my internal SATA DVD drive is very seldom seen by the kernel at startup. My guess is that it takes very long to start, and the kernel doesn't wait that long before deciding that it had all SCSI devices, so it misses it. It's actually very annoying, since you can't use the drive at all. After digging around, I finally stumbled on the rescan-scsi-bus tool from the scsitools package. Running (as root, of course)
~ rescan-scsi-bus -w -l
(sometimes two or three times) is enough to get the device back up, with the /dev/dvd udev symlink.

Hope this'll help !