SciYAG has been released !

Finally, I've released SciYAG ! I actually released it quite a few days ago. And it has seen quite some modifications after that (among which a critical bug fix...). Among the new features, let's name a few little ones:

• icons for experiments, directories and groups, adapted from gnome and KDE iconset, all in SVG as I discovered that Qt4 natively deals with SVG icons (great !)
• You can now edit Experiment Groups in the Type view as well
• Some usability fixes, such as shortcuts for commonly repeated things
• It starts to actually look good !

I'm starting to use it heavily for my own work, so it will have to improve fast !

SciYAG has a search engine now !

I wanted to release SciYAG not so long ago, but it came to me that it was still missing something to be of real interest: a search engine. It lets you find experiments according to data. Code used is real Ruby code, so you get a pretty good deal of power, and it stays reasonably intuitive.

It starts to feel like SciYAG is really getting ready and interesting ! Now, I'll try it rather heavily, and we'll see if it can stand a heavy use with reasonable performance. I hope so, as I did take care about this (although some improvements can still be expected somehow). Oh, by the way, the gnuplot backend is also available for SciYAG now !

gnuplot-compatibility backend

I used to work a lot with gnuplot. For what matter, I still do. I find it very convenient to write simple models (the one you can solve analytically), especially with the help of gnuplot-mode, a mode for editing gnuplot files and interacting directly with gnuplot, rather more smoothly than the standard gnuplot interface. But, you might have guessed, I don't like how gnuplot's output looks. After all, that's why I did write ctioga!

So what ? I won't use ctioga for playing with my models, as it has not been designed with interactivity in mind, and will most probably never be (that would somehow defeat it's purpose). But I don't want to do the same work twice, especially if the second time is way more tedious than the first...

So I wrote a new backend, gnuplot, whose job is to feed a file to gnuplot and tweak it so that ctioga gets the data of all the plots. Imagine you have a file biniou.gnu in the spirit of

a(x) = x**2 + 2*x + 1
plot a(x), x**2 - x

Then, you can plot both a(x) and x**2 - x with the following command-line:

ctioga --gnuplot biniou.gnu@1##2

Note that it only takes data ! Style is completely dismissed.

SciYAG is nearly ready !

I haven't been writing for a long time about SciYAG, but it does not mean that I didn't work on it. For what matters, I've been progressing quite a fair bit, to the point that I consider it releasable. Here comes the current features:

• fast display of experiments, including quick selection of several experiments and zooming
• editing of experiment types
• saving the database in the form of a YAML file
• editing type-related experiment "meta-data"
• using experiment groups for meta-data shared across a whole group of experiments (say, same buffer, so same pH values)
• different views of the experiments (type-based, file-based and group-based)

Although this really doesn't make all the features that are planned (see here for a reminder), it makes something I'd qualify as usable.

Actually, the only thing that refrains me from releasing is that I really should come up with a rudimentary documentation before I do so, and I simply don't have the energy/time/will for that now... Some time later (soon, hopefully) ? I really prefer adding some more features !

Yet again some improvements in SciYAG

Well, I didn't stop so easily on my enthusiasm of the morning. So I went on. I'll probably seem to repeat myself, but I won't ever say it enough: QGraphicsScene and others are simply fantastic ! I implemented today ticks and a grid for the plot using the neat trick of subclassing QGraphicsView and providing custom handlers for drawBackground and drawForeground - so easy ! As a result, the code for the new plotting system is way more elegant than the old one (and more compact, although file size doesn't show it as it is also significantly more documented). Once again, many thanks to developers at Trolltech for their wonderful work !

SciYAG's new plot system

Today (or rather, yesterday), I started using SciYAG at work. It did prove useful, even in the stage of non-completion in which it was. Great ! However, I found that something was missing sorely: the ability to display several curves at the same time (yes, SciYAG is only pre-alpha software...). A quick look tonight showed me that it was time to change the way plots were implemented. I therefore changed completely the structure to profit from the QGraphicsScene scheme. Attached is a result obtained with the first commit... It looks great ! QGraphicsScene is very easy to use, thanks to the Qt developers.

However, I found out that when I was switching antialiasing on, the rendering time was excessively great for big plots (around a second for 7000 points on my machine). So I decided to use OpenGL rendering. It turns out that it is as simple as this:

format = Qt::GLFormat.new()
format.set_sample_buffers(true)
@plot_view.set_viewport(Qt::GLWidget.new(format))
@plot_view.setRenderHints(Qt::Painter::Antialiasing)

Fantastic, isn't it ??

SciYAG goes on...

Well, work is going on on SciYAG. I added an view to select on the experiment type, and I tried for the first time a little scale up: I brutally imported my ca. 2000 data files into the program. It takes a few seconds to import, but then, it is really fast. Some remarks:

• I'm making very heavy use of the QStandardItemModel Qt class, and it scales up really easily, even with QtRuby. If you plan to make a model/view application in QtRuby, do not subclass QAbstractItemModel! The underlying model/view architecture make many calls to functions of the model, and calls from C++ to Ruby in QtRuby are prohibitive (compared to C++/C++ calls). Rather, fill in a QStandardItemModel, it is likely to work much faster.
• I'm glad it scales up nicely: the navigation between my 2000 data files is very smooth and neat.
• Currently, the function for reading text datafiles is written in pure Ruby, and it is apparently much too slow. I had planned a long time ago to rewrite that in C, and it looks like time has come to do so

wegben and inclusion of manual pages

Funnily enough, I haven't spoken about webgen yet. webgen is a very powerful static website generation tool written in Ruby. It was written with a very good plugin architecture in mind which makes it trivial to extend it.

Most of the sites I maintain (namely, SciYAG's, Tioga's and my former web page, the latter not being maintained anymore) are written using webgen. I recommend you have a look at those sites and at the website of webgen itself, you'd be amazed of the possibilities it offers.

SciYAG's website is making a heavy use of webgen's extension possibilities. I did write plugins for things as various as

• including a thumbnail of an image and a link to it with a code as simple as:

  {linkImage: Electrode_cleaning.png}

• converting automatically ctioga's shell script files into PDF, and converting any PDF into a PNG image with a thumbnail
• having a random image at every page (this really looks great in my humble opinion)
• use SVN information to display the name of the last author
• and many more things

My last addition to this wealth of plugins is a way to include a manual page within a webgen page, so that you profit from the CSS and from the navigation bar. You can get an idea of how it looks there. The source code is available as usual in the SVN repository.

SciYAG is going on...

Well, even if I didn't post here, I've kept myself decently busy. The commit that just went into SciYAG's SVN repository sees a new step in the completion of a public release: it is now possible to edit experiment types. Here's a picture of the current dialog box for editing experiment types. It is admittedly not very comfortable yet, but it works. I'll take time to improve it later when that becomes a priority.

Of course, I'll keep you posted about my progresses !

A first draft of SciYAG...

After some time, I finally have the first working draft of the upcoming SciYAG ! Admittedly, it currently doesn't do much more than just displaying the entries, but that already is a start, because you can browse pretty fast...

Here are the main things I'll have to focus on over the next days or weels:

• Provide a decent interface to change experiment types and then, most importantly, experience values (that would be under the current window)
• Provide a way to group experiences so they share the same data when that makes sense (when they really share some experimental conditions - when the exact same buffer or temperature or... were used)
• For the first version of SciYAG a year ago, I used QPainter::drawPath for painting my curves, which was fine at that time, but resulted in much hassle and too much Ruby code (much slower than C++). So I need to switch to the new 2D framework, QGraphicsScene/QGraphicsView - which wasn't available when I started back then
• Put back all the navigation features (zoom and the like) that were part of the old SciYAG at one point in the past

As you can see, I have things to keep myself busy... I'll keep you posted here !

Hoorray !

The commit that just got into SciYAG's SVN repository is the first one that actually gets a user interface running ! There is not much, admittedly (just dummy entries for experiments you can't change yet), but, still, it shows that I've made decent progresses in the architecture. I'll keep you posted about future developments !

Change of plans for SciYAG

Initially, the SciYAG program in the SciYAG project was intended as a direct competitor for the interactive part of gnuplot, while ctioga is definitely a good competitor of the "output" part (get convinced there). I had some part working, with a command-line working exactly as ctioga and some reasonable features.

However, with time, I found that what I'm really missing in my everyday scientific work is not an interactive gnuplot clone. As far as I can tell, gnuplot is jolly good itself for that, and we have a neat home-brewed program called Soas that does the job pretty well, and is tailored to our needs.

So, then, what ? What I really miss is a good program to organise my data. Here is the features I'm missing:

• First, I want to be able to attach meta-data to my datafiles, such as the buffer used for the experiment, the pH values, and so on. All these should be fully customizable on a per-experiment or per-experiment-type basis
• I want to be able to browse quickly to my experiments, sorting them according to their type, their values of their meta-data
• Of course, I'm interested to actually display the data, possibly massively, possibly as a function of some of the meta-data
• I also want to be able to associate series of fits of different kinds to each of the datafiles, and be able to quickly check them (visually, by displaying the fit and the data), and plot the parameters found there as a function of meta-data.
• And more !

This may sound ambitious, but actually, with a great programming language like Ruby, it is fun. A great deal of the architecture is already in place. What I'm currently working on is a way to store all the data. The trick is to make it so it can evolve easily, as I'll be using this program extensively for my own needs, and I don't want to reenter any meta-data already in. A good architecture should also please the lazy in me (which amounts to, say, 95%...) and have me type in only what is absolutely necessary - so most values for meta-data should be shared or guessed.

There is currently no release of SciYAG, though you can chance a look at it's SVN repository. Be sure I'll keep you posted about my progresses on the field !

Ruby

Well, I couldn't really start a blog without writing a small article about this fantastic programming language, Ruby. It all started when a friend of mine posted a message about the Tioga plotting library.

He said that this library was producing great plots (he saw directly plots made by its creator, Bill Paxton). When I saw this video, I was sincerely convinced !

So then, as the library was only for the Ruby programming language, I started immediately to learn Ruby, using the Pragmatic Programmer's Programming Ruby. I found it an extremely enjoyable experience. Within an hour, I had made my first Ruby script using Tioga. Before the end of the day, I was sending my first bug report to Bill. The following day, I had already started to work on a (very poor at that time) first version of ctioga...

I've gone on working with Ruby ever since... I wrote no less than 9000 lines of code only in the SciYAG project on Rubyforge, not mentioning other projects using Ruby and the countless scripts I have written for my everyday science... Ruby is extremely comfortable to program with. A pure delight !

ctioga's new legends

Since release 1.6 of ctioga, the legends had moved away from the graph, and were not anymore aligned with the plot's internal boundaries. It might not seem much, but the look was horrible, and, even worse, a lot of white space was inserted between the legend and the graph. I've just commited a fix to the SVN repository, and it now looks nearly like before 1.6. Great !