Showing posts with label Linux tips. Show all posts
Showing posts with label Linux tips. Show all posts

Wednesday, 5 September 2012

A Linux one-liner to find all the acronyms in your Latex files

At the beginning of my PhD thesis, I include a List of Acronyms. Of course, I would like to be sure that my list is comprehensive. I don't want any strange acronyms to appear in the text of my thesis, without first appearing in my list of acronyms. But how can I easily identify all of the acronyms in the Latex source, without having to read all 244 pages manually?

grep to the rescue, again

Like most other areas of my life, this problem can be easily solved with a Linux one-liner centered around grep:

cat *.tex | grep -wo "[A-Z]\+\{2,10\}" | sort | uniq -c | sort -gr

Let's take a look at the pipeline:
  • The cat *.tex outputs all my Latex to standard output.
  • The grep -wo  "[A-Z]\+\{2,10\}" matches whole words (the -w flag) that contain between 2 and 10 upper case letters. The -o flag returns only the match, not the entire line.
  • The first sort sorts the acronyms, which is useful for the next step.
  • The uniq gets rid of duplicates, but retains a counter because of the -c flag.
  • Finally, the second sort sorts the entries numerically (-g) and reverses the results (-r).

  • Here's the output on my thesis:

    
        292 IR
        241 LDA
        166 LSI
        125 VSM
         87 TCP
         80 EM
         35 APFD
         34 SUT
         29 HSD
         22 TOPIC
         22 II
         18 MALLET
         16 LOC
         14 PS
         14 OR
         14 IDE
         12 CALLG
         11 ICA
         10 RNDM
         10 MAP
         10 KL
         10 CS
         ...

Note that this command works with any text file; it is not unique to Latex. Just change the cat command.

Thursday, 26 July 2012

A Linux one-liner to count the unique words in a set of files

Say you have a bunch of text files, and you want to find out all the unique words in them, and how often each words appears. Using the brilliance of Linux commands, this can be achieved in a few strokes of the finger:

$ cat *.txt | tr " " "\n" | sort | uniq

Use tr to put all the words (separated by spaces, but you can make this more complex if you need) into their own line. sort will then sort all these words. Finally, uniq will get rid of duplicate lines, and the -c flag will add counts. I'll run this on a small set of bug reports from the Eclipse Bugzilla repository:

$ cat * | tr " " "\n" | sort | uniq -c
      1 ^^^^
      6 able
     12 about
      1 aboutdialog
      1 abovebackground
      1 absolute
     18 abstract
      2 abstractannotationprocessormanager
      4 abstractcompletiontest
      1 abstractdebugeventhandler
    [...]

The words will be displayed in alphabetical order. (Yes, the word "^^^^" actually appeared somewhere in the bug reports.) To instead display them by usage in decreasing order, run it all through sort one more time:

$ cat * | tr " " "\n" | sort | uniq -c | sort -gr
   1118 java
   1006 eclipse
   1005 at
    956 org
    825 the
    546 internal
    361 to
    356 jdt
    320 ui
    318 in
   [...]

The -g option sorts by numeric values, instead of alphanumeric, and the -r option reverses the sort. VoilĂ !

Monday, 23 July 2012

Removing spaces in filenames


Ever have a bunch of files with spaces in their names, and need to get rid of the spaces?

$ ls -1
My data.csv
Surfing USA - Beach boys.mp3
Who let the dogs out.mp3
Dear American Express.doc
thesis version2-oct2008 final revisions3 steves comments.tex

Some scripts and commands don't handle spaces nicely, without painful escaping of each and every space and each and every file name. You could use mv to rename each file individually, but I'd rather shove ghost peppers in my eyeballs, especially when there's an easy one-liner to do it for me.

The Easy Way

On the eight day, God created the rename command:
$ rename 's/ //g' ./*
The rename command takes a regular expression and a list of files, and renames each file according to the regular expression. See the man page for complete details, but the regular expression syntax is similar to Perl's. In our simple case, we search for any spaces ("/ /") and replacing them with nothing ("//").

The Hard Way

Another approach is to use bash to loop through each file, use sed to find spaces and replace them with nothing, and then use the mv command to finish it all up.
$ for file in *; do mv "$file" `echo $file | sed -e 's/  */_/g' -e 's/_-_/-/g' -e 's/[()]//g'`; done
Although much more involved and a pain to read, it achieves the same effect as the rename command. If the rename command is not available to you but bash is, then here's your ticket.