^Ms in Unix files.. the secret!

Solving the ^M Mystery
(and Other Puzzles FTPing Text Files Between Unix and DOS)

Have you ever tried to edit a file in your Unix account (sweetgum, cypress, sunset, cedar, etc.), only to find that someone has riddled it with ^M characters, or removed the end-of-file marker? That mysterious someone was no ghost--it was your file transfer program.

Well, maybe not tecnically. But since DOS and Unix handle low level details of text files differently, there is sometimes a need to
convert these files to the format of the target operating system; and sometimes, this taken care of by the file transfer program; other times, you have to do it yourself.

On Windows/DOS systems, lines are terminated by a carriage return character followed by a line feed character. On Unix systems, lines are terminated by just a line feed character. On Mac systems, lines are terminated by just a carriage return character.

With FTP, you could always specify "ASCII" or "binary" mode when transferring files. "ASCII" mode was generally used for text files, which resulted in carriage returns being removed (when transferring from PC/DOS/Windows to Unix) or added (when transferring from Unix to PC/DOS/Windows.) Failure to use ASCII mode when transferring text files from PS/DOS/Windows systems to Unix systems results in the carriage returns being left in the file; when the file is viewed under Unix, these carriage returns show up as ^M characters at the ends of lines.

With Secure Shell File Transfer Protocol (SFTP) version 2.4, ASCII is only an option if you use the command-line version of the tool (sftp2). It is not possible to specify a transfer mode in the GUI version of the 2.4 release of the tool--all files are transferred in binary mode. Thus, all text files so transferred from PC/DOS/Windows to Unix systems will arrive with these unwanted ^M characters. This problem has been corrected with version 3.0 of SSH/SFTP, now available for download to UM campus computers from the UM IT Helpdesk. (Operation->File Transfer Mode->Ascii)

Some Windows programs are smart enough to add carriage returns when they detect their absence in data files. Similarly, most Unix programs can recognize and ignore carriage returns. However, sometimes this is not the case, and so you may want to add or remove them, as appropriate.

Option One: Windows-Side Conversion

Windows-side utilities are widely available for download from the web to convert back and forth between the two formats. One such set of utilities, dos2unix and unix2dos, is available for download from: http://www.bastet.com/software/software.html.

Option Two: Unix-Side Conversion, using sed

The Unix sed tool can be used on a Unix system to convert a file between PC/DOS/Windows mode and Unix mode. We found the following two examples in a web article entitled "Common Threads: Sed by Example, Part 3" from November 2000 by Daniel Robbins posted on the IBM DeveloperWorks website: They illustrate the process.

Step One: Prepare for Windows Departure
Create a text file on your DOS/Windows PC, using some generic text editor such as Notepad.
Upload it to your Unix account using SFTP. (See the accompanying article on SFTP file permissions.)
View the file on Unix, using the cat or more commands, or an editor such as pico or vi.
Notice the ^M characters.
Step Two: Prepare for Unix Arrival
After the file has been uploaded, use the Unix utility, sed, to convert the file to an acceptable format. While still in your unix account:
sed -e 's/.$//' mydos.txt > myunix.txt
Replace the "mydos.txt" and "myunix.txt" with the filenames to be converted. For example, to convert
the DOS file foo.txt to a Unix format execute
sed -e 's/.$//' foo.txt > foounix.txt
Step Three: Prepare for Return Trip to DOS
The command to convert a file from Unix format to DOS format looks like this:
sed -e 's/$/\r/' myunix.txt > mydos.txt
Replace the "mydos.txt" and "myunix.txt" with the filenames to be converted. For example, to convert
the Unix file foo.txt to a DOS format execute
sed -e 's/$/\r/' foo.txt > foodos.txt

When we tried the DOS to UNIX conversion, it did succeed in removing the ^M characters (the carriage returns) from the ends of lines. The only goofy thing was, the last line didn't have a ^M, so when we ran the script, it removed the line feed character of the last line instead. So, when we edited it the last time, it gave an "incomplete last line message". But when we saved the file, it seemed to add the line feed character, because all was well the second time we edited it.