Open the current .sl4 file format ?

Motivation

( [I]See the bottom of the message for the 'Suggestion' part .[/I] )

1.
Not all the experiments arise from the scratch every time )
– sometimes (and very often) one may have a serious need to do a set of routine changes in his or her previous experiments;

for example, one may look at shafi’s post in
http://community.cedrus.com/showthread.php?t=45 , [0]
at bgon’s post at
http://community.cedrus.com/showthread.php?t=57 , [1]
and at WSHorton’s post at
http://community.cedrus.com/showthread.php?t=199 . [2]

Last but not least ),
I can also report that we have tens of old experiments,
having tens of ‘events’ stimuli in each, tens of “trials” and tens of blocks also .
( Making a brief note, I should say that using stimulus lists is not an option due to some technical reasons like lack of control )

I was told that making a set of simple uniform changes in all of them via the straightforward “click-and-edit” way took like one month of human time ,
though this amount can be slightly overestimated .

2.
As it can be easily seen from [0] – [2] and
http://community.cedrus.com/archive/index.php/t-130.html [3] ,
Superlab users seem to have expected to be able to do such a “mass change” operation on experiment files .

3.
Cedrus seem to admit that this feature is one of the most demanded
( [1], Hisham’s answer ) , promising to implement sort of a solution in the upcoming intermediate SuperLab 4.5 release, but there is obviously not enough time to implement that yet:
see
http://community.cedrus.com/showthread.php?t=106 [4],
Hisham’s answer in [2], and the FAQ link
http://www.cedrus.com/support/superlab/tn1501_faq.htm [5] ,
stating that the 4.5 version release date is estimated to be the Fall of 2007 .

At the moment I am writing this,
there is no evidence of version 4.5 in the list of the new updates
( http://community.cedrus.com/forumdisplay.php?f=13 ),
the 4.0.5 version release info also does not have any notifications about the new features from the “top demand” list
( http://community.cedrus.com/showthread.php?t=316 )

4.

So the only solution to deal with “old” SuperLab 4.* files that is available at the moment is the Greg Shenaut’s ‘sl4util’  [4] (manages only stimulus lists): officially unsupported by Cedrus ( see the Hisham's reply ) but sort of recommended to use (see [3] ) .       
Before writing sl4util, Greg has also proposed to open the .sl4 data format (as an option, see [0] ), but at that moment this proposal was not accepted.     
From the other hand, it looks like Greg's attempts to hack the SuperLab data format were encouraged by the Cedrus team ( [4] ) .       

The Suggestion

Is obviously to open the .sl4 experiment file format: from the one hand, I personally can not imagine how the one can be used to run an experiment without the SuperLab software; from the other hand, we see (see the above links) that Cedrus obviously lacks the time to develop a solution for the "mass editing" problem, while the users (who can concentrate on their own needs) can sometimes do that surprisingly quickly (see Greg's report where he says it took like one weekend for him to make some significant progress even having no experiment file format description).
I do not call Cedrus to make any "stated in the stone" statements -- let it be something "unsupported" and "subject to change" -- basically, I personally just don't want to waste time hacking a thing that can be simply (and harmlessly) documented .     

Actually, if there is a significant lack of time ( so it is hard to devote it to the .sl4 file format description), I might even suggest to open the code of the .sl4 file saving procedure (well, in the case if there are no solutions one will want to keep undisclosed).
Anyway, thanks for reading this,

George

PS. I’d like to say that the program itself is very nice designed, and, what impressed me even more – the support on the forum looks to have an immediate reaction .
This looks to be very positively accepted by the users, so I wish to Cedrus to go in a more intensive collaboration with them .

Thank you for the post and the time that you took to “make the case”.

Regarding your suggestion to open up the .sl4 file saving procedure: SuperLab 1 and 2 had a single function to write and another to read. SuperLab 4.0 is an object oriented C++ program and alas, there is no single function for reading or writing. Each object throughout the program is responsible for saving its own “state”, saving and restoring links to other objects, and maintaining backward compatibility. It would be impossible to open up the file format without essentially making all of SuperLab open source.

You are right regarding the SuperLab 4.5 promise. We’re late with it, and the end result may be different than what we promised. That’s one of the main reasons why we no longer divulge information about our product plans (except perhaps in the vaguest of terms, see my blog entry, or when it makes sense).

Having said that, it is clear that we have seriously underestimated how important it is for our users to be able to manipulate the experiment file directly. Mea culpa. Rather than “open up” the code that reads files, we are exploring ways where we can open up the file format itself and use human readable/text files in future products. Suggestions are welcome.

And last but not least, thank you for the kind words regarding SuperLab’s design and our support. It’s not perfect but we keep trying to improve.

Well, the first thing that comes to mind is to use xml files.

Cons:
[LIST=1]

  • imho, they are approximately half away from binaries -- if we think about the ease of direct human editing;
  • the xml files are generally much slower in access than the binary ones . [/LIST]

    Pros:
    [LIST=1]

  • we significantly ease the process of data editing for the users ;
  • the xml format need not to be a replacement, being just an alternative . [/LIST]

    How one can do that (add an xml file format support for the .sl4 files) in a relatively easy way?

    Suppose that one has a hierarchy of objects, each having friend ifstream/ofstream operators that are responsible for saving and restoring their state from the file.

    Irrespectfully of the size of such a hierarchy, I suppose that every high-level <</>> operation can be expressed in the terms of the <</>> operations of the contained objects, having only few “elementary” operations on the lowest level: these are probably “reading/writing a number” and “saving/restoring a string”, where the numbers can be integer or floating-point – or (raw) “byte”, if we want to reserve the possibility to store some “raw” data. On the next level, we might need to have the possibility to read/write the arrays of the described elementary items.

    Having that implemented, there is no principal difference between storing (or reading) binary or xml data:
    when the first will be implemented in the terms of operations like
    ourfile << 5.4
    or
    ourfile << “abc”
    ,
    the second will look the same,
    except for the implementation of ourfile object,
    that could be a descendant of ofstream with some basic operations redefined:
    say, the string output operation will basically look like


    class xmlstream : public ofstream
    {
    // …
    xmlstream& operator << (const char* pszStr)
    {
    return ( this->ofstream::operator<< ( “<StringData>” ) << pszStr << “</StringData>” ) ;
    }
    // …
    };

    Obviously, it will be nice if the high order data save/restore operations will add (and check) some information that is meaningful at their level:
    say,
    <StimList> … </StimList>
    , etc .

    PS. Sorry if the above looks somewhat unclear or contains typos, but I hope the trick is still clear: if one can express the store/retrieve procedures in the terms of a few elementary ones (I presume this is how it is implemented at the moment), then it is relatively simple to change the meaning of these elementary operations to make them handle, say, xml data instead of “raw binary” one.

    This provides some low-level type information “for free” and allows to add high-level information “incrementally” with every editing of any store/retrieve data function pair (one can just add a <FunctionName> … </FunctionName> wrapper in the writing part – and check for it’s presence during the reading) .

    Anyway, it looks like direct hacking of the .sl4 file format will be much faster for our purposes.
    It will be nice then if it will be possible to ask for some yes/no confirmations during the process – in the case if that won’t violate Cedrus policy .

    Best,
    George

  • You’re right about XML being half away from binary. Notice in my reply that I said “human readable” files! XML is poetry for programmers but barely readable by all others.

    What about making half the way?

    However, XML would be better than the solution which we have at the moment which requires manual editing of each individual item that you want to change. As a non programmer myself I don’t find XML formats that hard to get my head round once I’ve worked out what I’m looking at!

    Certainly colleagues of mine that used Superlab v2 miss the abilty to bulk edit items that was apparently in that version so I think addressing this issue would be a good move for Cedrus and the future of Superlab…

    open the .sl4 file

    Are there any news on manipulating the superlab code directly?
    Thx,

    Unfortunately not.