Working With History in Bash

Yesterday we talked about favorite bash features (on the ##textmate IRC channel). I figured it was worth posting mine to this blog, they mostly revolve around history, hence the title.

Setup

My shell history collects a lot of complex command invocations which take time to figure out. To ensure that I have access to them at a later time, I have the following 3 lines in my bash init:

export HISTCONTROL=erasedups
export HISTSIZE=10000
shopt -s histappend

The first one will remove duplicates from the history (when a new item is added). For example if you switch between running make and ./a.out in a shell, you may later find that the last 100 or so history items is a mix of these two commands. Not very useful.

The second one increase the history size. With duplicates erased, the history already holds a lot more actual information, but I still like to increase the default size of only 1,000 items.

The third line ensures that when you exit a shell, the history from that session is appended to ~/.bash_history. Without this, you might very well lose the history of entire sessions (rather weird that this is not enabled by default).

History Searching

Now that I have my history preserved nicely in ~/.bash_history there are a few ways to search it.

Using Grep

The most crude is grep. You can do:

history|grep iptables

For me (on this particular Linux server) I get:

4599  iptables -N http-block
4600  iptables -A http-block -s 58.60.43.196 -j DROP
4601  iptables -A INPUT -p tcp --dport 80 -j http-block
4602  iptables -L http-block
4603  iptables-save -c
4604  history|grep iptables

I do this often enough to have an alias for history (which is just h).

From the output I can either copy/paste the stuff I want, or repeat a given history event. You’ll notice that each history event has a number, you can repeat e.g. event number 4603 simply by running:

!4603

I will write a bit more about referencing history events in History Expansion.

Prefix Searching

Similar to how you can press arrow up for the previous history event, there is a function you can invoke for the previous history event with the same prefix as what is to the left of the insertion point.

This function is called history-search-backward and by default does not have a key equivalent. So to actually reach this function, I have the following in ~/.inputrc (or /etc/inputrc when I control the full system):

"\ep": history-search-backward

This places the function on P (escape P). So if I want to repeat the iptables-save -c history event we found in previous section, all I do is type ipt and hit P. If it finds a later event with the same prefix, hit P again to go further back.

This functionality is offered by the readline library, so if you setup this key, you have access to prefix searching in all commands which use this library.

Incremental Search

It is possible to press ⌃R to do an incremental (interactive) search of the history.

Personally I am not a big fan of this feature, so I will leave it at that :)

Update: The reason I dislike ⌃R is both because the interactive stuff just seems to get in the way (when P is what I need 99% of the time) and because it fails in cases where I “switch shell”, for example I may do: ssh mm press return, then instantly type: fP and again hit return (to execute free -m on the server called mm). I enter this before the connection to the server has been fully established, and here ⌃R would have been taken by the local shell, but it is the shell history at the server I want to search.

History Expansion

History Expansion was what we did above when we ran !4603. It is a DSL for referencing history events and optionally run transformations on these.

Anyone interested in this should run man bash and search for History Expansion, but just to give you a feel for what it is, I will reference a subset of the manual and provide a few examples.

Event Designators

First, an event designator starts with ! and then the event we want to reference. This can be:

«n»      Reference event number «n».
-«n»     Go «n» events back.
!        Last line (this is the default).
#        Current line.
«text»   Last event starting with «text».
?«text»  Last event containing «text».

So if we want to re-run our iptables-save -c we can do: !ipt.

What’s more useful though is to use history references as part of larger commands.

For example take this example:

% which ruby
/usr/bin/ruby
% ls -l $(!!)
lrwxr-xr-x  1 root  wheel  76 30 Oct  2007 /usr/bin/ruby -> ../../System/Library/Frameworks/Ruby.framework/Versions/Current/usr/bin/ruby

Or something like:

% make some_target
(no errors)
% path/to/target some arguments
(no errors)
% !-2 && !-1

Word Designators

In the previous section we referenced entire history events. It is possible to reference just a subset of a history event by appending a : to the event designator and then the word of interest, the two most useful are:

«n»      Reference the «n»’th word.
$        Reference the last word.

So for example we can do:

% mkdir -p /path/to/our/www-files
(no errors)
% chown www:www !$
(no errors)

Here we reference last word of last line. We can also reference stuff on the same line, e.g.:

% cp /path/to/important/file !#:1_backup

To reference the last word of last line one can also press _ which will immediately insert that word.

Modifiers

To make history substitution even more useful (and harder to remember), one can also add a modifier to the event designator.

The most useful modifiers are in my experience :h and :t, these are head and tail respectively or better know as dirname and basename.

An example could be:

% ls -l /path/to/some/file
(listing of file)
% cd !$:h
(change directory to that of file)

Brace Expansion

Somewhat related to the backup example where we reference the first argument as !#:1 and append _backup to this, another approach is bracket expansion.

Anywhere on a command line, one can write {a,b,c} which will expand to the 3 words a, b, and c. If we include a prefix/suffix, that will be part of each of the expanded words. We can also leave the word in the braces empty, and have it expand to just the prefix/suffix, so for example we can do:

% cp /path/to/important/file{,_backup}

This is functionally equivalent to:

% cp /path/to/important/file !#:1_backup

But lack of hardcoded word number is IMO an improvement.

46 Comments

  1. 28 Jun 2008 | # Anonymous wrote…

    This is interesting, but… any updates on 2.0? A year ago I thought for sure it would be out by now…

  2. 28 Jun 2008 | # oliver taylor wrote…

    Yo, Frakking lay off with the 2.0 shit. It's not like he's spent a year looking at YouTube.

  3. 28 Jun 2008 | # kch wrote…

    yea, and where's textmate for the iphone anyway? :P

  4. 28 Jun 2008 | # Steve L wrote…

    Wow … and all this time I've been thinking that I was pretty handy with the shell.

    Thanks for the schooling.

  5. 28 Jun 2008 | # Daniel Stockman wrote…

    Thanks Allan, this is great!

  6. 29 Jun 2008 | # Thomas Aylott wrote…

    Simply add bind Space:magic-space to your .bash_profile to be able to auto-expandy those crazy ! commands inline before executing them.

  7. 29 Jun 2008 | # Thomas Aylott wrote…

    Personally, instead of using the escape-p business, I just replaced the standard up/down character history search with the history search with context. If you want it to work without the context just move your cursor to the left.

    bind '"\e[A"':history-search-backward bind '"\e[B"':history-search-forward

  8. 29 Jun 2008 | # Thomas Aylott wrote…

    bind '"\e[A"':history-search-backward

    bind '"\e[B"':history-search-forward

    no, I'm not a Markdown pro :'(

  9. 29 Jun 2008 | # Naveed wrote…

    You should consider putting the following 2 lines in your .inputrc "\e[A": history-search-backward "\e[B": history-search-forward It will allow you to search backward just by pressing the up key.

  10. 29 Jun 2008 | # Peteris Krumins wrote…

    I have written an article which teaches how to work efficiently on the command line.

    It's called "[The Definite Guide to Bash History])http://www.catonmat.net/blog/the-definitive-guide-to-bash-command-line-history/)"

    Sincerely, Peter

  11. 29 Jun 2008 | # Peteris Krumins wrote…

    Fixing my previous comment:

    Article: The Definite Guide to Bash History

  12. 29 Jun 2008 | # Alok wrote…

    The reason I dislike ⌃R is both because the interactive stuff just seems to get in the way

    Here's how to use it properly.

    ^rgit will show you the last line in your shell history with git.

    From here you can:

    1. Press enter to re-execute your command immediately.
    2. Press ESC or CTRL-[ to edit the command line found, for what you need.
    3. And much better, type ^r to search more for previous instance, from the current position.

    In any case it is quicker than ESC-P and more powerful.

  13. 29 Jun 2008 | # Pádraig Brady wrote…

    Rather than using ^r to search the history, I usually type the start of the command, and then use the up/down arrows to complete it. Details here: http://www.pixelbeat.org/settings/.inputrc

  14. 29 Jun 2008 | # lee doolan wrote…

    Just a little note on history expansion. If you do this

     <event-designator>M-^
    

    the event will be moved into the command line buffer but it will not be executed. You can use editing commands to modify the line before typing to execute. For instance:

    !iptableM-^
    

    will move the last iptables command into the buffer and

    !4601
    

    would move this command (from the example in the main article) into the buffer:

    iptables -A INPUT -p tcp --dport 80 -j http-block
    

    This is a very handy facility. Try it, you'll like it.

  15. 29 Jun 2008 | # lee doolan wrote…

    Just one further note:

    M-
    

    is the meta key. On my keyboard it is bound to ALT, but you can change this binding. Consult the man page for details.

  16. 30 Jun 2008 | # A Fresh Cup » Blog Archive » Double Shot #238 wrote…

    [...] Working with History in Bash – Some tips from Allan Odgaard. [...]

  17. 01 Jul 2008 | # David Maas wrote…

    Didn't know the brace trick, very very very useful

  18. 01 Jul 2008 | # Geoff Hutchison wrote…

    A few other bash tricks:

    Add "cdspell" to the shopt -s line. This allows bash to correct for minor misspellings when doing cd

    export HISTIGNORE="&:ls:[bf]g:exit:cd ..:make:svn up:svn commit"
    # a bunch of command-lines separated by colons
    

    There's also a pile of great bash completions at Caliban.org. I use them so often that I find myself wondering why tab-completion "doesn't work" on someone else's computer.

  19. 01 Jul 2008 | # You Sexy Thing › Bookmarks for June 30th wrote…

    [...] TextMate Blog » Working With History in Bash – Great tips for Bash. Lots of great comments too. This was written by Kris Markel. Posted on Monday, June 30, 2008, at 9:13 pm. Filed under del.icio.us. Tagged bash, commandline, history, howto, Linux, reference, shell, terminal, tips, Unix. Bookmark the permalink. Follow comments here with the RSS feed. Post a comment or leave a trackback. [...]

  20. 01 Jul 2008 | # nogg3r5 wrote…

    Thank you. A lot.

  21. 01 Jul 2008 | # Håkan Eriksson wrote…

    Hey, I've recently been looking for a way to this kind of stuff in bash.

    One question; at a place where I used to work, someone had set up our environment so that Ctrl-ArrowUp (or Ctrl-P) did what is described as Esc-P above, i.e. jump to the previous line in history that starts like the one you're currently typing.

    I'd like to have Ctrl-P (for instance) instead of Esc-P for "history-search-backward" – how do I do that?

  22. 01 Jul 2008 | # Elia wrote…

    erasedups for HISTCONTROL isn't available for bash 2.05 (like that of OSX Tiger) insead you can use

    export HISTCONTROL=ignoreboth

    more info on: http://aplawrence.com/Linux/bash_history.html

  23. 01 Jul 2008 | # Stop Reading Here! » Bash History Explained wrote…

  24. 01 Jul 2008 | # C Grayson wrote…

    If you're man (or woman) enough to have learned vi, the same search commands (/, n, N), line navigation (k, l) and line editing (cw, dw, C, r, etc.) are all available to you on the command line if you just issue this command: "set -o vi".

  25. 01 Jul 2008 | # Decklin Foster wrote…

    There's an interactive equivalent to the "Word Designators" section that I really like:

    M-. (that's meta-dot) yanks the last word of the previous command. Give it an argument (like M-2 M-.) to yank the /n/th word, counting from zero.

    Very useful for running several commands on the same file without having to think about editing.

  26. 01 Jul 2008 | # Decklin Foster wrote…

    Another one: fixing mistakes. To correct something in a command (let's say the last command):

    !!:s/old/new/

    There is a shortcut for this (only works on !!):

    ^old^new

    But it doesn't let you replace every occurence on the line, like so:

    !!:gs/old/new/

  27. 01 Jul 2008 | # Decklin Foster wrote…

    One more and I'm done. Let's say you wanted to run the sequence of iptables commands from the first example, but with a different name for the chain. Doing an s/// substitution on each command would get annoying. The sequence is from numbers 4599 to 4603 in the history. So you run:

    fc 4599 4603

    and bash launches your text editor ($EDITOR, maybe vi if you don't have that set) with all the commands in it. Do a global search-and-replace to change "http-block" to whatever else (or make any other edits you want). Then save and quit, and bash will run each edited command, printing them before they are run.

    With no arguments, fc just edits the last command.

  28. 01 Jul 2008 | # Reading List, 1July08 » Karl Katzke | PHP, Puppies, and other Geekery wrote…

    [...] Working with History in Bash [...]

  29. 02 Jul 2008 | # Blue Sky On Mars » links for 2008-07-02 wrote…

    [...] TextMate Blog » Working With History in Bash Great collection of tips for managing your history in Bash. This has already helped me out. (tags: bash terminal tips development) [...]

  30. 03 Jul 2008 | # Sam wrote…

    Man the load of useful stuff here is amazing.

    Not much to add here but you can add the following to your .bash_profile you can tab through other completions rather than trying to remember what letter to write next. Useful if you don't use the bash very often and have a hard time remembering commands.

    bind '"\t":menu-complete'
    
  31. 05 Jul 2008 | # Blogaholic » Bash-Tricks (History) wrote…

    [...] TextMate Blog: Working with History in Bash [...]

  32. 10 Jul 2008 | # Thomas wrote…

    Where can we find product updates? I'm also interested in how close we are to the 2.0 release. Generally, I think it's a good idea to let customers know something about the software they're purchasing, and an expected arrival date/month/quarter would be nice. More accurately, knowing that a new release will make it out in 2008, for example, would certainly dissuade me from buying Komodo IDE. Or using MacVim as my primary editor, since I use GVim on Windows at the office.

  33. 23 Jul 2008 | # eric wrote…

    wow this is a great compilation of useful commands, thanks a lot

  34. 31 Jul 2008 | # nairda wrote…

    @Thomas: TextMate is dead, get used to it.

  35. 31 Jul 2008 | # Kevin Ballard wrote…

    TextMate isn't dead, it's just stagnating because Allan's spending his time working on 2.0 instead of incremental updates to 1.x. And the bundles are being updated all the time by the community.

  36. 02 Aug 2008 | # This month in bookmarks: July 2008 wrote…

    [...] Working With History in Bash [TextMate Blog] – Allan Odgaard wrote a hugely useful tutorial to help us master bash history. I found this very useful and I'm sure you will, too. [...]

  37. 06 Aug 2008 | # sohbet wrote…

    Thanks a lot

  38. 15 Aug 2008 | # QuickLinks vom 14. August — instant-thinking.de wrote…

    [...] Working With History in Bash – Some very good tips for the history command. [...]

  39. 22 Sep 2008 | # Recent URLs tagged Commandline - Urlrecorder wrote…

    [...] Recent public urls tagged "commandline" → TextMate Blog " Working With History in Bash [...]

  40. 19 Oct 2008 | # Customizing The Mac Terminal Bash Prompt | Mac Tricks And Tips wrote…

    [...] the bash prompt I would like to show you is the history. This comes in part from the TextMate blog. What you can do is increase or decrease the history size, remove duplicates and make sure it saves [...]

  41. 23 Oct 2008 | # Customizing The Mac Terminal Bash Prompt | Macbook Pro, iMac, iPod, iPhone, iTunes, Mac, Apple News and Shopping wrote…

    [...] the bash prompt I would like to show you is the history. This comes in part from the TextMate blog. What you can do is increase or decrease the history size, remove duplicates and make sure it saves [...]

  42. 28 Oct 2008 | # Working With History in Bash | MirthLab wrote…

    [...] If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!Working WIth History in Bash [...]

  43. 25 Nov 2008 | # Building’s the Foundation - Beige Sunshine wrote…

    [...] have them in any of it's shells. I mean seriously, how does anyone live without a searchable history in a command [...]

  44. 16 Feb 2009 | # NexNova » Blog Archive » Links del giorno: February 16, 2009 wrote…

    [...] TextMate Blog » Working With History in Bash [...]

  45. 21 Mar 2009 | # Bash history | codetoself wrote…

    [...] Allan Odgaard's excellent Working With History in Bash: export HISTCONTROL=erasedups export HISTSIZE=10000 shopt -s histappendPop out1 2 3 export [...]

  46. 05 May 2009 | # bash history size « arcierisinasce.wordpress.com wrote…

    [...] by marinz on May 5, 2009 Ho trovato questo articolo, in pratica basta inserire nel [...]

Comments closed, you can use the mailing list for discussion.