Overcome command line length limitations

Reviewer: name contact BSD flavour

Concept

The command line length is limited, and often a command should be applied to more arguments than fit on a command line. Understand how to run the command multiple times with different arguments for each call using xargs(1)(((xargs))) or a shell "while(((while)))" read loop.

Introduction

The shell has a limit for the command line length and the system has a limit on how many bytes can be used for the arguments (and environment variables) when starting a new process. (Some shells also have a limit on the command line length saved in its history.)

NOTES:

TODO: Just briefly mention the following, but focus on generic skills of using xargs and sh while loop instead of adjusting the more advanced tunables.

ksh 1024 on command line

tcsh 8190 on command line

tcsh history is 4096

On DragonFly: TODO: sh has "abort" or "segmentation fault" and exits noticed with around 1920 characters -- report this bug or commit fix

NCARGS on FreeBSD and DragonFly is 65536. On OpenBSD and NetBSD it is 262144

TODO: mention "getconf ARG_MAX"??

TODO: do all BSD's have kern.argmax?

Common error message is "Argument list too long."

TODO: some xargs have -J option for replacements, but I probably won't cover that here and will use "while" instead

TODO: mention find -print0 and xargs -0

TODO: example using Bourne style shell:

"" find . -type f | while read line ; do "" # do something with ${line} "" done

TODO: do real example above

Examples

The following is an example of having too many arguments:

"" $ ls -l /usr/ports/*/*/Makefile* "" /bin/ls: Argument list too long.

A work-around for this is to use shell built-in "echo" and pipe the output to "xargs", for example:

""$ echo /usr/ports/*/*/Makefile* | xargs ls -l

(Note that that output is not shown here, because it was over 15,000 lines.)

Practice Exercises

run a command multiple times with different arguments for each call: ls | xargs md5 (spits out an md5 hash for each file in a dir)

More information

xargs(1), find(1)