BASH Frequently Asked Questions

1. How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?

Don't try to use "for". Use a while loop and the read command:

2. How can I store the return value and/or output of a command in a variable?

Well, that depends on whether you want to store the command's output (either stdout, or stdout + stderr) or its exit status (0 to 255, with 0 typically meaning "success").

3. How can I find the latest (newest, earliest, oldest) file in a directory?

The tempting solution is to use ls to output sorted filenames and take the first result. As usual, the ls approach cannot be made robust and should never be used in scripts due in part to the possibility of arbitrary characters (including newlines) present in filenames. Therefore, we need some other way to compare file metadata.

4. How can I check whether a directory is empty or not? How do I check for any *.mpg files, or count how many there are?

In Bash, you can do this safely and easily with the nullglob and dotglob options (which change the behaviour of globbing), and an array:

5. How can I use array variables?

This answer assumes you have a basic understanding of what arrays are. If you're new to this kind of programming, you may wish to start with the guide's explanation. This page is more thorough. See links at the bottom for more resources.

6. How can I use variable variables (indirect variables, pointers, references) or associative arrays?

This is a complex page, because it's a complex topic. It's been divided into roughly three parts: associative arrays, evaluating indirect variables, and assigning indirect variables. There are discussions of programming issues and concepts scattered throughout.

7. Is there a function to return the length of a string?

The fastest way, not requiring external programs (but not usable in Bourne shells):

8. How can I recursively search all files for a string?

If you are on a typical GNU or BSD system, all you need is one of these:

9. What is buffering? Or, why does my command line produce no output: tail -f logfile | grep 'foo bar' | awk ...

Most standard Unix commands buffer their output when used non-interactively. This means that they don't write each character (or even each line) immediately, but instead collect a larger number of characters (often 4 kilobytes) before printing anything at all. In the case above, the grep command buffers its output, and therefore awk only gets its input in large chunks.

10. How can I recreate a directory hierarchy structure, without the files?

With the cpio program:

11. How can I print the n'th line of a file?

One dirty (but not quick) way is:

12. How do I invoke a shell command from a non-shell application?

13. How can I concatenate two variables? How do I append a string to a variable?

There is no (explicit) concatenation operator for strings (either literal or variable dereferences) in the shell; you just write them adjacent to each other:

14. How can I redirect the output of multiple commands at once?

Redirecting the standard output of a single command is as easy as:

15. How can I run a command on all files with the extension .gz?

Often a command already accepts several files as arguments, e.g.

16. How can I use a logical AND/OR/NOT in a shell pattern (glob)?

"Globs" are simple patterns that can be used to match filenames or strings. They're generally not very powerful. If you need more power, there are a few options available.

17. How can I group expressions in an if statement, e.g. if (A AND B) OR C?

The portable (POSIX or Bourne) way is to use multiple test (or [) commands:

18. How can I use numbers with leading zeros in a loop, e.g. 01, 02?

As always, there are different ways to solve the problem, each with its own advantages and disadvantages.

19. How can I split a file into line ranges, e.g. lines 1-10, 11-20, 21-30?

Some Unix systems provide the split utility for this purpose:

20. How can I find and safely handle file names containing newlines, spaces or both?

First and foremost, to understand why you're having trouble, read Arguments to get a grasp on how the shell understands the statements you give it. It is vital that you grasp this matter well if you're going to be doing anything with the shell.

21. How can I replace a string with another string in a variable, a stream, a file, or in all the files in a directory?

There are a number of techniques for this. Which one to use depends on many factors, the biggest of which is what we're editing.

22. How can I calculate with floating point numbers instead of just integers?

BASH's builtin arithmetic uses integers only:

$ echo "$((10 / 3))"
3

23. I want to launch an interactive shell that has special aliases and functions, not the ones in the user's ~/.bashrc.

Just specify a different start-up file:

24. I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?

In most shells, each command of a pipeline is executed in a separate SubShell. Non-working example:

25. How can I access positional parameters after $9?

Use ${10} instead of $10. This works for BASH and KornShell, but not for older BourneShell implementations. Another way to access arbitrary positional parameters after $9 is to use for, e.g. to get the last parameter:

26. How can I randomize (shuffle) the order of lines in a file? Or select a random line from a file, or select a random file from a directory?

To randomize the lines of a file, here is one approach. This one involves generating a random number, which is prefixed to each line; then sorting the resulting lines, and removing the numbers.

27. How can two unrelated processes communicate?

Two unrelated processes cannot use the arguments, the environment or stdin/stdout to communicate; some form of inter-process communication (IPC) is required.

28. How do I determine the location of my script? I want to read some config files from the same place.

There are two prime reasons why this issue comes up: either you want to externalize data or configuration of your script and need a way to find these external resources, or your script is intended to act upon a bundle of some sort (eg. a build script), and needs to find the resources to act upon.

29. How can I display the target of a symbolic link?

The nonstandard external command readlink(1) can be used to display the target of a symbolic link:

30. How can I rename all my *.foo files to *.bar, or convert spaces to underscores, or convert upper-case file names to lower case?

There are a bunch of different ways to do this, depending on which nonstandard tools you have available. Even with just standard POSIX tools, you can still perform most of the simple cases. We'll show the portable tool examples first.

31. What is the difference between test, [ and [[ ?

[ ("test" command) and [[ ("new test" command) are used to evaluate expressions. [[ works only in Bash, Zsh and the Korn shell, and is more powerful; [ and test are available in POSIX shells. Here are some examples:

32. How can I redirect the output of 'time' to a variable or file?

Bash's time keyword uses special trickery, so that you can do things like

33. How can I find a process ID for a process given its name?

Usually a process is referred to using its process ID (PID), and the ps(1) command can display the information for any process given its process ID, e.g.

34. Can I do a spinner in Bash?

Sure!

35. How can I handle command-line arguments (options) to my script easily?

Well, that depends a great deal on what you want to do with them. There are several approaches, each with its strengths and weaknesses.

36. How can I get all lines that are: in both of two files (set intersection) or in only one of two files (set subtraction).

Use the comm(1) command:

37. How can I print text in various colors?

Do not hard-code ANSI color escape sequences in your program! The tput command lets you interact with the terminal database in a sane way:

38. How do Unix file permissions work?

See Permissions.

39. What are all the dot-files that bash reads?

See DotFiles.

40. How do I use dialog to get input from the user?

Here is an example:

41. How do I determine whether a variable contains a substring?

In BASH:

42. How can I find out if a process is still running?

The kill command is used to send signals to a running process. As a convenience function, the signal "0", which does not exist, can be used to find out if a process is still running:

43. Why does my crontab job fail? 0 0 * * * some command > /var/log/mylog.`date +%Y%m%d`

In many versions of crontab, the percent sign (%) is treated specially, and therefore must be escaped with backslashes:

44. How do I create a progress bar? How do I see a progress indicator when copying/moving files?

The easiest way to add a progress bar to your own script is to use dialog --gauge. Here is an example, which relies heavily on BASH features:

45. How can I ensure that only one instance of a script is running at a time (mutual exclusion)?

We need some means of mutual exclusion. One way is to use a "lock": any number of processes can try to acquire the lock simultaneously, but only one of them will succeed.

46. I want to check to see whether a word is in a list (or an element is a member of a set).

If your real question was How do I check whether one of my parameters was -v? then please see FAQ #35 instead. Otherwise, read on....

47. How can I redirect stderr to a pipe?

A pipe can only carry standard output (stdout) of a program. To pipe standard error (stderr) through it, you need to redirect stderr to the same destination as stdout. Optionally you can close stdout or redirect it to /dev/null to only get stderr. Some sample code:

48. Eval command and security issues

The eval command is extremely powerful and extremely easy to abuse.

49. How can I view periodic updates/appends to a file? (ex: growing log file)

tail -f will show you the growing log file. On some systems (e.g. OpenBSD), this will automatically track a rotated log file to the new file with the same name (which is usually what you want). To get the equivalent functionality on GNU systems, use tail -F instead.

50. I'm trying to put a command in a variable, but the complex cases always fail!

Variables hold data. Functions hold code. Don't put code inside variables! There are many situations in which people try to shove commands, or command arguments, into variables and then run them. Each case needs to be handled separately.

51. I want history-search just like in tcsh. How can I bind it to the up and down keys?

Just add the following to /etc/inputrc or your ~/.inputrc:

52. How do I convert a file from DOS format to UNIX format (remove CRs from CR-LF line terminators)?

Carriage return characters (CRs) are used in line ending markers on some systems. There are three different kinds of line endings in common use:

53. I have a fancy prompt with colors, and now bash doesn't seem to know how wide my terminal is. Lines wrap around incorrectly.

53.1. Escape the colors with \[ \]

You must put \[ and \] around any non-printing escape sequences in your prompt. Thus:

54. How can I tell whether a variable contains a valid number?

First, you have to define what you mean by "number". The most common case when people ask this seems to be "a non-negative integer, with no leading + sign". Or in other words, a string of all digits. Other times, people want to validate a floating-point input, with optional sign and optional decimal point.

55. Tell me all about 2>&1 -- what's the difference between 2>&1 >foo and >foo 2>&1, and when do I use which?

Bash processes all redirections from left to right, in order. And the order is significant. Moving them around within a command may change the results of that command.

56. How can I untar (or unzip) multiple tarballs at once?

As the tar command was originally designed to read from and write to tape devices (tar - Tape ARchiver), you can specify only filenames to put inside an archive (write to tape) or to extract out of an archive (read from tape).

57. How can I group entries (in a file by common prefixes)?

As in, one wants to convert:

58. Can bash handle binary data?

The answer is, basically, no....

59. I saw this command somewhere: :(){ :|:& } (fork bomb). How does it work?

This is a potentially dangerous command. Don't run it! The "trigger" is omitted from the question above, leaving only the part that sets up the function.

60. I'm trying to write a script that will change directory (or set a variable), but after the script finishes, I'm back where I started (or my variable isn't set)!

Consider this:

   #!/bin/sh
   cd /tmp

61. Is there a list of which features were added to specific releases (versions) of Bash?

Here are some links to official Bash documentation:

62. How do I create a temporary file in a secure manner?

There does not appear to be any single command that simply works everywhere. tempfile is not portable. mktemp exists more widely (but still not ubiquitously), but it may require a -c switch to create the file in advance; or it may create the file by default and barf if -c is supplied. Some systems don't have either command (Solaris, POSIX).

63. My ssh client hangs when I try to logout after running a remote background job!

The following will not do what you expect:

   ssh me@remotehost 'sleep 120 &'
   # Client hangs for 120 seconds

64. Why is it so hard to get an answer to the question that I asked in #bash?

Maybe nobody knows the answer (or the people who know the answer are busy). Maybe you haven't given enough detail about the problem, or you haven't presented the problem clearly. Maybe the question you asked is answered in this FAQ, or in BashPitfalls, or in the BashGuide.

65. Is there a "PAUSE" command in bash like there is in MSDOS batch scripts? To prompt the user to press any key to continue?

Use the following to wait until the user presses enter:

66. I want to check if [[ $var == foo || $var == bar || $var == more ]] without repeating $var n times.

The portable solution uses case:

67. How can I trim leading/trailing white space from one of my variables?

There are a few ways to do this. Some involve special tricks that only work with whitespace. Others are more general, and can be used to strip leading zeroes, etc.

68. How do I run a command, and have it abort (timeout) after N seconds?

FIRST check whether the command you're running can be told to timeout directly. The methods described here are "hacky" workarounds to force a command to terminate after a certain time has elapsed. Configuring your command properly is always preferable to the alternatives below.

69. I want to automate an ssh (or scp, or sftp) connection, but I don't know how to send the password....

STOP!

70. How do I convert Unix (epoch) times to human-readable values?

The only sane way to handle time values within a program is to convert them into a linear scale. You can't store "January 17, 2005 at 5:37 PM" in a variable and expect to do anything with it....

71. How do I convert an ASCII character to its decimal (or hexadecimal) value and back?

If you have a known octal or hexadecimal value (at script-writing time), you can just use printf:

72. How can I ensure my environment is configured for cron, batch, and at jobs?

If a shell or other script calling shell commands runs fine interactively but fails due to environment configurations (say: a complex $PATH) when run noninteractively, you'll need to force your environment to be properly configured.

73. How can I use parameter expansion? How can I get substrings? How can I get a file without its extension, or get just a file's extension? What are some good ways to do basename and dirname?

Parameter expansion is an important subject. This page contains a concise overview of parameter expansion.

74. How do I get the effects of those nifty Bash Parameter Expansions in older shells?

Most of the extended forms of parameter expansion do not work with the older BourneShell. If your code needs to be portable to that shell as well, sed and expr can often be used.

75. How do I use 'find'? I can't understand the man page at all!

See UsingFind.

76. How do I get the sum of all the numbers in a column?

This and all similar questions are best answered with an AWK one-liner.

77. How do I log history or "secure" bash against history removal?

If you're a shell user who wants to record your own activities, see FAQ #88 instead. If you're a system administrator who wants to know how to find out what a user had executed when they unset or /dev/nulled their shell history, there are several problems with this....

78. I want to set a user's password using the Unix passwd command, but how do I script that? It doesn't read standard input!

OK, first of all, I know there are going to be some people reading this, right now, who don't even understand the question. Here, this does not work:

{ echo oldpass; echo newpass; echo newpass; } | passwd
# This DOES NOT WORK!

79. How can I grep for lines containing foo AND bar, foo OR bar? Or for files containing foo AND bar, possibly on separate lines? Or files containing foo but NOT bar?

This is really four different questions, so we'll break this answer into parts.

80. How can I make an alias that takes an argument?

You can't. Aliases in bash are extremely rudimentary, and not really suitable to any serious purpose. The bash man page even says so explicitly:

81. How can I determine whether a command exists anywhere in my PATH?

POSIX specifies a shell builtin called command which can be used for this purpose:

82. Why is $(...) preferred over `...` (backticks)?

`...` is the legacy syntax required by only the very oldest of non-POSIX-compatible bourne-shells. There are several reasons to always prefer the $(...) syntax:

83. How do I determine whether a variable is already defined? Or a function?

There are several ways to test for a defined or non-empty variable or function depending upon the requirements.

84. How do I return a string (or large number, or negative number) from a function? "return" only lets me give a number from 0 to 255.

Functions in Bash (as well as all the other Bourne-family shells) work like commands: that is, they only "return" an exit status, which is an integer from 0 to 255 inclusive. This is intended to be used only for signaling errors, not for returning the results of computations, or other data.

85. How to write several times to a fifo without having to reopen it?

In the general case, you'll open a new FileDescriptor (FD) pointing to the fifo, and write through that. For simple cases, it may be possible to skip that step.

86. How to ignore aliases or functions when running a command?

Sometimes it's useful to ignore aliases (and functions, including shell built-in functions). For example, on your system you might have this set:

87. How can I get a file's permissions (or other metadata) without parsing ls -l output?

There are several potential ways, most of which are system-specific. They also depend on precisely why you want the information; in most cases, there will be some other way to accomplish your real goal. You don't want to parse ls's output if there's any possible way to avoid doing so.

88. How can I avoid losing any history lines?

This method is designed to allow you to store a complete log of all commands executed by a friendly user; it is not meant for secure auditing of commands - see securing bash against history removal.

89. I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!

When reading a file line by line, if a command inside the loop also reads stdin, it can exhaust the input file. For example:

90. How do I prepend a text to a file (the opposite of >>)?

You cannot do it with bash redirections alone; the opposite of >> does not exist....

91. I'm trying to get the number of columns or lines of my terminal but the variables COLUMNS / LINES are always empty.

COLUMNS and LINES are set by BASH in interactive mode; they are not available by default in a script. On most systems, you can try to query the terminal yourself:

92. How do I write a CGI script that accepts parameters?

There are always circumstances beyond our control that drive us to do things that we would never choose to do on our own. This FAQ entry describes one of those situations.

93. How can I set the contents of my terminal's title bar?

If you have a terminal that understands xterm-compatible escape sequences, and you just want to set the title one time, you can use a function like this:

94. I want to get an alert when my disk is full (parsing df output).

Sadly, parsing the output of df really is the most reliable way to determine how full a disk is, on most operating systems. However, please note that this is a "least bad" answer, not a "best" answer. Parsing any command-line reporting tool's output in a program is never pretty. The purpose of this FAQ is to try to describe all the problems this approach is known to encounter, and work around them.

95. I'm getting "Argument list too long". How can I process a large list in chunks?

First, let's review some background material. When a process wants to run another process, it fork()s a child, and the child calls one of the exec* family of system calls (e.g. execve()), giving the name or path of the new process's program file; the name of the new process; the list of arguments for the new process; and, in some cases, a set of environment variables. Thus:

96. ssh eats my word boundaries! I can't do ssh remotehost make CFLAGS="-g -O"!

ssh emulates the behavior of the Unix remote shell command (rsh or remsh), including this bug. There are a few ways to work around it, depending on exactly what you need.

97. How do I determine whether a symlink is dangling (broken)?

The documentation on this is fuzzy, but it turns out you can do this with shell builtins:

98. How to add localization support to your bash scripts

Looking for examples of how to add simple localization to your bash scripts, and how to do testing? This is probably what you need....

99. How can I get the newest (or oldest) file from a directory?

This page should be merged with BashFAQ/003

100. How do I do string manipulations in bash?

Bash can do string operations. LOTS of string operations. This is an introduction to bash string manipulations and related techniques. It overlaps with the Parameter Expansion question, but the information here is presented in a more beginner-friendly manner (we hope).

101. Common utility functions (warn, die)

(If you were looking for option processing, see BashFAQ/035.) The following functions are frequently asked for in #bash, so we hope you find them useful.

102. How to get the difference between two dates

It's best if you work with timestamps throughout your code, and then only convert timestamps to human-readable formats for output. If you must handle human-readable dates as input, then you will need something that can parse them.

103. How do I check whether my file was modified in a certain month or date range?

Doing date-related math in Bash is hard because Bash has no builtins in place for doing math with dates or getting metadata such as modification time from files.

104. Why doesn't foo=bar echo "$foo" print bar?

This is subtle, and has to do with the exact order in which the BashParser performs each step.

105. Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected?

set -e was an attempt to add "automatic error detection" to the shell. Its goal was to cause the shell to abort any time an error occurred, so you don't have to put || exit 1 after each important command.

106. I want to tee my stdout to a log file from inside the script. And maybe stderr too.

This requires some tricky file descriptor manipulation, and either a named pipe or Bash's ProcessSubstitution. We're going to focus on the Bash syntax.

107. How do I add a timestamp to every line of a stream?

There are numerous ways to do this, but all of them are either limited by the available tools, or slow. We'll show a few examples.

108. How do I wait for several spawned processes?

There are numerous ways to do this, but all of them are either limited by the available tools. I have come up with the following solutions.

109. How can I tell whether my script was sourced (dotted in) or executed?

Usually when people ask this, it is because they are trying to detect user errors and provide a friendly message. There is one school of thought that says it's a bad idea to coddle Unix users in this way, and that if a Unix user really wants to execute your script instead of sourcing it, you shouldn't second-guess him or her. Setting that aside for now, we can rephrase the question to what's really being asked:

110. How do I copy a file to a remote system, and specify a remote name which may contain spaces?

All of the common tools for copying files to a remote system (ssh, scp, rsync) send the filename as part of a shell command, which the remote system interprets. This makes the issue extremely complex, because the remote shell will often mangle the filename. There are at least three ways to deal with the problem: NFS, careful encoding of the filename, or submission of the filename as part of the data stream.

111. What is the Shellshock vulnerability in Bash?

"Shellshock" refers to two remotely-exploitable vulnerabilities in Bash, discovered in September 2014. The first vulnerability exploits the mechanism that Bash used to export and import functions, and allowed arbitrary command execution. The second vulnerability exploits a parser bug and allowed local files to be created.

112. What are the advantages and disadvantages of using set -u (or set -o nounset)?

Bash (like all other Bourne shell derivatives) has a feature activated by the command set -u (or set -o nounset). When this feature is in effect, any command which attempts to expand an unset variable will cause a fatal error (the shell immediately exits, unless it is interactive).