Department of Physics and Astronomy

IDL Tutorial: Reading and Writing ASCII data

The simplest sort of data you might receive from an experiment would be simple (x, y) pairs. For example, you might have access to a weather monitoring station that records the date and time, temperature, and windspeed. Ultimately you may wish to plot the temperature vs. date, or temperature vs. windspeed, or who knows what. Often these data appear as ASCII (text) tables, the sort that you can readily view with a text editor.

Here's a snippet of a data file containing the spectrum of a fluorescent lamp, coincidentally, one in my office.

lambda_nm signal_cts
380 0.00143708199983
381 0.00153702295425
382 0.00154965916688
383 0.00130114698519
384 0.00144397447945
385 0.00139113213573
386 0.00167180922229
387 0.00178974720682
388 0.00167602129317
389 0.0018824127661
390 0.00194980590012

There's a single header line describing the columns, in this case, wavelength in nm and signal level in detector counts per second (or something like that). How do we get these data into IDL?

The Easy Way

The IDLastro package maintained at NASA's Goddard Space Flight Center has a handy tool to read in columnar data; it's a procedure called readcol.

Try it out. Copy the text file spectrum.txt into your workspace.

readcol, 'spectrum.txt', lambda, signal

Readcol seeks out columnar data and loads the data into array variables. Try,

print, lambda

plot, lambda, signal, xtitle='Wavelength (nm)', ytitle='Counts per second'

There appears to be an overall background level in the count rate; that is, ideally "no signal" means 0 counts per second, but instead there seems to be some constant offset in the count rate. Suppose that instrumental background level were uniformly 0.002 counts per second across the spectrum. We can easily remove that background,

signal = signal - 0.002

Now it might be useful to save the background-subtracted data. There's a nice tool called writecol that behaves similarly to readcol, which I believe is included in the XIDL distribution produced by Lick Observatory (try googling IDL writecol). The link will take you to my copy of it. It will not be installed by default on the system, you'll have to copy it to wherever you're working, or wherever you keep your personal IDL codes.

writecol, 'spectrum_bgsub.txt', lambda, signal

And now you should have saved the data for posterity.

Formatted Input with Readcol

Readcol, by default, looks for numbers, but sometimes you have a column filled with text, names of celestial objects, for example. You can tell readcol to look for text in a given column by including the format statement.

readcol, 'data_with_text.txt', x, y, names, format='(f,f,a)'

This command tells readcol to look for floating-point numbers in the first two columns, but text (ascii) in the third column.

Formatted Output with Writecol

Writecol behaves similarly: the expectation is that you are writing columns of numbers. To force an output column to be ascii,

writecol, 'data_with_text.txt', x, y, names, fmt='(f,f,x,a)'

Notice that the keyword changes from format to fmt for no good reason. The 'x' in the format statement asks writecol to put in a blank space between y and names.

Try the following experiment:

n = n_elements(lambda)

goofy = replicate('goofy', n)

writecol, 'goofy_file.txt', lambda, signal, goofy, fmt='(f,f,x,a)'

Have a look at 'goofy_file.txt' with a text editor. Then try,

readcol, 'goofy_file.txt', x, y, names, format='(f,f,a)'

and see if you effectively copy the contents of goofy into names. (Which you could have done by the command names = goofy, but the point here is to practice reading and writing text files!)

The Hard Way

You will run into limitations with readcol and writecol, in which case you'll have to do things the hard way. The hard way involves writing a for loop to read and write ASCII data one element at a time. Files are handled by assigning them a "logical unit number," or LUN. Nothing to be scared of - the point here is that rather than having the input/output routines deal with filenames like "really_long_filename.txt" all the time, you just assign a number (LUN) to that file, and access it by that number. I'll just give you some simple examples, using the lambda and signal data we already have loaded. Here's an procedure that would read the spectrum from an external file.

pro readspectrum, filename, LAMBDA=lambda, SIGNAL=signal
;
; Reads in a spectrum from "filename," stores the data into "lambda" and "signal"
;
get_lun, in ; get_lun just automatically chooses a free "LUN" for your file. Here, I'm storing the LUN in the variable "in."
openr, in, filename ; open the file for reading: assign LUN "in" to filename
lambda = 0.0
signal = 0.0
;
; loop
;
junktext = ' '; establish that junktext is a string variable
readf, in, junktxt, format='(a)' ; read in and throw away the header line
while not eof(in) do begin ; do the following steps so long as we're not at the end-of-file (EOF)
readf, in, x, y; read in the next line
lambda = [lambda, x]; add the new data point to the array
signal = [signal, y]
endwhile
;
; lose the initial junk element
lambda = lambda[1:*]
signal = signal[1:*]
close, in ; we no longer need the file
free_lun, in ; we no longer need the LUN
end

Try out this procedure:

readspectrum, 'spectrum.txt', x, y
plot, x, y

Exercise

Try writing a similar procedure called "writespectrum.pro" to spit a spectrum out to some arbitrary filename chosen by the user. Check the documentation on openw and printf.

Final Note

printf and readf can each take format= keyword arguments to clean up the reading and writing.

Back to Index

Next Tutorial

Last modified by Jack Gallimore.