Page maintained by
The Isdat tutorial.
Author: Tobias Eriksson
This paper assumes that you already have built the isdat clients to use. They are built the same way as the server but with only clients and isdatbase checked out from the CVS. You also need a working Isdat server Server to contact, either locally on your machine or remote on any machine on your network. You don't need access to any data since we use the test database that is available in all Isdat servers.
To set these things up please refer to the documentation in the source code release. Specially the INSTALL file in the isdat root directory and various papers in the doc directory.
What is Isdat?
Isdat (current version 2.7.1) is a client-server application to access high resolution time series data from different space missions. The server part (data base handler) keeps track of all data and unpacks new data files as they are added into the database. The client part fetches data from the server and presents them to the user. The main advantage of this design is that all data is only kept in one place in the institution (or on the Internet) reducing maintenance, improving data integrity and allowing fast data distribution to all users.<! Simply put Isdat is a computer tool to help scientists to keep track of enormous amounts of time series data. In order to make things as easy as possible for the people using and administrating Isdat the program is split into two main parts. There is one part that keep track of all data and unpacks new data files as they are added into the database. This part is called the server and it provides a uniform interface to data from as vast sources as the Viking satellite orbiting the earth in the late 80s and the Cassin spacecraft soon to deliver data from the moons of Saturn. As the other part there are the various clients that fetch data from the server in order to present it in a nice way to the user. Thus Isdat is a so called client-server application. The main advantage of this design is that all data is only kept in one place in the institution or even just one place on the whole Internet. This radically reduces maintenance and makes it easier to update data. All this gives god data integrity and fast data distribution to all users. In this tutorial we will give an introduction to some of the different clients available and how to make them perform their duty, fetch data. At the moment of writing the current Isdat version is 2.7.1. As the set of available clients is constantly increasing and existing clients is being updated this paper is only to be considered valid for the above version of Isdat. Hopefully this document will be kept updated and the figure above will be in sync with future Isdat versions. >
At the moment there are 16 clients in the Wec Isdat distribution. All of different status and functionality, some (like igr) are graphical applications intended to display and perform various calculations on the data. This type of clients are started from ctm (Client and Time Manager) who's main function is to manage the clients and to help you select a start time and a stop time and provide this to the clients so they know which data to fetch.
Another class of clients is the so called API:s. These have only one sole use, to provide a bridge between the Isdat server and third party software such as Matlab and IDL. With the help of these handy clients you are free to do almost any calculations and visualizations on the data in the Isdat database. <! Later we will show you an excellent example on this. >
Getting started using iscmd.
iscmd is a simple and flexible way of getting data out of the Isdat Database. It is a simple shell program that takes a number of command line arguments and displays tabulated data on standard output so it can piped to any other command line tool or simply being displayed on the screen. iscmds status is someware between an stand alone client and a API for the shell programing language. You can use it both ways and thous it makes a good introduction to the Isdat system.
In order for iscmd (and any other Isdat client) to work there are some things to set up. First your shell must find the iscmd program. When Isdat compiles it does not copy or link its executables to any of the folders were the shell normally looks for programs. This has to be set up manually by setting a number of shell environment variables. Our first variable, ISDAT_HOME, makes sure that the Isdat clients can find themselves. It must point to the base of the Isdat installation. If you downloaded Isdat by CVS and installed it under /usr/local then typing:
would do the job. Now we can use the ISDAT_HOME variable to extend the PATH variable in a correct way.
It's important that the $PATH: string is included on the right side of the equality sign. Otherwise the shell won't be able to find any of the normal UNIX commands any longer. The final step is to tell iscmd were to look for its missing half, it must find the Isdat server. This is done by setting ISDAT_DATABASE. In order to do this you must know on witch machine the server is running and witch Isdat database number it has assigned to it. When you have this information proceed with:
Now you should be ready to run Isdat clients. Naturally all this is not what you would like to type into your computer every morning you come to work. And of course there is a simple way to get around it. In the Isdat distribution there is a skeleton script for setting all environment variables needed. Copy it to your home directory:
or if you are running c-shell:
Now edit this file according to the above recipe and include the following row in one of your startup files.
For Bash users a typical startup file is ~/.profile and for c-shell users ~/.cshrc. But beware there are numerous exceptions to this.
Will give you a listing of all your "dot-files" in Linux, on other os ls -a will have to do. If you don't have any of the above files then check your local documentation for your correct startup file.
With all this done you are finally ready to start looking at data. iscmd has four
modes of operation. Simply typing iscmd at the prompt will give you an short help
message together with an listing of the command line options for the different
modes. The first option to use is q, it stands for query. It's purpose is to let
you explore the structure of the isdat data tree. In many ways this tree
reminds of the file system of your local hard drive, it has a root from were
everything originates as the directory's in a file system. Let us call Isdats
"directory's" for logical instruments. The big difference in this comparison is
that when you can put datafiles in any directory. Isdat data only shows up in
the leaf of the data tree. A Isdat logical instrument can only contain data or other
logical instruments not both.
Will give you an listing of the databases on your Isdat server. There will be one entry called Test and if you are working in Wec most probably another one called Cluster. To explore further you can use the slash '/' separator between entry's. Thousiscmd q /Test
will give you the next lever in the Test logical instrument tree. Give yourself some time to familiarize yourself with your local Isdat data tree. When you encounter an logical instrument that is empty it probably isn't. Try
to view the contents of the data set in this logical instrument. Although, there is a hatch here. Even though iscmd c reports that there should be data in a specific time interval it's not guaranteed to be there. This is be course the Isdat server only looks for the existence of datafiles and does not check their contents when reporting data available. This is an issue for future Isdat versions.
<! Now let us look at a specific event. On the last day of year 2000, the 31 of December 2000, approximately at 10.17 Cluster made an magnetopause crossing that is clearly visible in the data set. To simply view this data in raw asci format on the screen type: > Now let us look at a specific event. In the Test database there is 100 sec of an built in square wave signal. It is accessed by:
After a small delay you will see a lot of data pass through the screen. If you want to view it, pipe it to less.
Now you will be able to see the top of the dataset. It tells you a great deal about the data, its place in the Isdat database, its type, Isdat server version and so on. Just before the data there is an row describing the type of data. In this case Time and Volt. The time is relative from the requested start time. Time is the main key that all data in Isdat is ordered after so this row will always be there. The unit of next column will of course vary depending on the requested data, for some measurements there will be several columns since these are multi dimensional.
Let's examine the above command row more carefully. The d option tells iscmd to ask the Isdat server for data and if there is any iscmd delivers it to standard out. Second we have the full qualified logical instrument path to the data. Third the start time for the wanted data set and last the duration of the data set in seconds. iscmd is a bit strict about the format of the start time it should be:
Where MM is the two digit number for the month and mm two digit number for the minute. The only optional part of the above time is the nanoseconds.
An alternative way of calling iscmd is with the isdat url syntax. In that case the database is added before the isdat logical instrument path, thous eliminating the need of the ISDAT_DATABASE shell variable. In that case the above command would look like:
if the Isdat server is running on a machine called localhost on DB port 10. For now iscmd is the only isdat client that supports this syntax.
Well asci data might not be very informative, piping it to xgraph might give us a better show.
xgraph is avaliable for download from http://www.cs.toronto.edu/~radford/xgraph.html
Installation is straight forward, at least on Suse Linux, un tar source package, go there, type:
I really recommend this small handy program.
Lesson two, use Igr.
One of the most common ways of accessing the isdat database is through the Igr client. To start Igr type ctm &. This will start the client and time manager. Your first quest will be to select the same data set as used with iscmd. On the left side on the ctm panel there is three drop down boxes. Project, Member and Instrument. These corresponds to the first three levels of the Isdat logical instrument tree. The default settings might be Cluster, 1, efw. If so, change to Test, * and wave. Next to select a time interval you use the start, stop and interval boxes on the right. We will continue to work with the above data (1993-08-01 15:10:00.0 interval 100s). You will now discover that to type text into isdat text field you must always have the cursor positioned over the text field, for people used to the Windows environment this can be a bit confusing in the beginning. Next start igr from the drop down menu, Clients -> General -> Igr.
Igr is a client program for plotting and looking at Isdat data, once you got
a grip of the main idea the program it gets quite easy to use. The two
abstract main components to work with is quantities and plots. A
quantity is a storage box (named q0 - q132) into whom you can put a
data set from any of the logical instruments. You can then make a
number of operations and calculations on these quantities. Finally you
bind the result to a named plot for displaying your result. There are two
control panels for doing this tasks. First the quantity dialog (Select
-> quantity, or
Ctrl-Q), second the Calculator. At any moment just one quantity and
one plot can be active, activating an quantity means that it's
possible to fil it with data. The active quantity and panel are
displayed right under the drop down menus.
It is possible to add more plot panels to igr. Graphics -> panel -> create will show up a dialog box with buttons for each one of 30 possible plot panels. The first panel corresponds to the top one on the left side. Pressing one of the highlight buttons will add one panel in that place. Click so you have four panels avaliable for plotting. In the right upper corner of each plot there is four buttons, think of these as shortcuts to a plot and quantity. When you click one of these you will change both the current plot and quantity. For example, if you press the last button in the left upper plot you will activate plot 3 and quantity 3. Let's read in some data in q3. Now you will see a new row being added to the calculator. Right click in the panel to see the plot. This is the fast way if you just want to look at raw data.
More complicated plots!
Let us now try to make a fft spectrogram on our event. This is a more complicated operation and will require several steps. First exit igr and goto Options -> clean client start in ctm, relaunch igr. This will assure that all quantities and plots are cleared (the default behavior of igr is to start in the same state as it was stopped in). Now select quantity 0 (Select -> quantity) and fill it with square test data (Test -> * -> wave -> square -> wf). In the drop down menu Operator you will see igrs built in operators. To make a fft spectrogram plot we will use the fft operator. <! we also need to introduce a variable in the calculator. Let's call it v0, v0 = despin(q0) in the formula box of igr. This calls igrs builtin despin operator on the dataset q0 and gives us a new a new dataset v0 that not is bound to any plot panel and thus is 'invisible'. Next step is to make an fft on the data set v0. Add p0y = fft(v0) in the calculator. Both the despin and fft operator can take optional additional arguments to customize their behavior. > The fft operator takes some optional additional arguments that customize its behavior. This is done by marking the used operator with the mouse and then Clicking Edit -> Operator. In this case we want to tell the fft operator to sweep over the signal and transform it in many small overlapping pieces. In the options dialog box change mode to Manual, fft size to 64, real samples to 64 and shift to 32 also change spectra to All. Don't press the apply button in the calculator at this stage this would erase those fft options just chosen and restore the default ones. Instead press left mouse button in panel p0 to view the results. As you see there are now many transformed signals in the same plot. To view all these as a spectrogram instead just Click Graphics -> Plot -> Spectra. If you feel happy with your plot and would like to save it to a file or print it. Choose File -> print, here you have the option to save the panel to a postscript file or a gif file or simply to print it.
<! Breaking Free - The APIs. Finally a example on how to use the Matlab API to make a plot. > For comments, suggestions etc. firstname.lastname@example.org