An Engineering Introduction to The Command Line

The command line is an essential interface that allows users to interact with computers using text-based commands. It serves as a powerful alternative to graphical user interfaces (GUIs), offering greater flexibility and control. Imagine a scene from The Matrix, where streams of code cascade down the screen, representing the inner workings of a computer. While not exactly as dramatic, the command line gives you a glimpse into the “matrix” within your own system. A fun way to visualize this is by running the `cmatrix` command, which simulates falling green code on your terminal.

Computers provide two main types of interfaces: graphical (GUI) and textual (CLI). GUIs are user-friendly, while CLIs are efficient and indispensable in scenarios where no monitor is available, such as accessing remote servers or managing embedded systems. Consider working on a remote computer in a data center or a robot operating in the field—these scenarios necessitate the use of the command line.

The first step in mastering the command line is to access it. On Windows, you can search for "Command Prompt" (cmd) in the Start menu¹. For more powerful features, you can use PowerShell, a command-line shell and scripting language.

An even better alternative is Git Bash, which brings Unix-like functionality to Windows. You can download and install Git Bash from https://git-scm.com/downloads. Read more at https://www.atlassian.com/git/tutorials/git-bash.

Advanced users may prefer the Windows Subsystem for Linux (WSL), which enables Linux commands on a Windows system. Installing WSL involves opening PowerShell and typing `wsl –install`. Once set up, you can start WSL from the Start menu and enjoy a full Linux command-line experience.

For MacOS and Linux users, the command line is readily available through the Terminal app. On MacOS, consider using iTerm2, an enhanced terminal emulator with additional features like split panes and advanced search capabilities.

The command line lets you interact with your computer in ways that GUIs cannot. To get started, try some basic commands:

1. `date`: Displays the current date and time.
2. `cal`: Shows a calendar for the current month.
3. `echo`: Writes text to the screen, e.g., `echo "Hello, world!"`.

Here are some example input and output pairs:

cal
echo 'Hello, world!'

     April 2024
Su Mo Tu We Th Fr Sa
    1  2  3  4  5  6
 7  8  9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30

Hello, world!

These commands provide simple but powerful ways to interact with your computer.

A key aspect of the command line is navigating and managing the file system. The following commands are fundamental:

• `pwd`: Prints the current directory.
• `ls`: Lists the contents of a directory.

• `cd`: Changes the current directory. For example:

  • `cd ..` moves up one directory level.
  • `cd ~` navigates to your home directory.

Try the following in your commandline:

pwd
cd ..
pwd
cd ~
pwd

File and folder management is another essential skill. The commands `mkdir` (make directory), `touch` (create an empty file), `rm` (delete files), `cp` (copy files or directories), and `mv` (move or rename files) allow you to organize your system efficiently. For example, you can create a new folder, create a file inside it, and then copy or move that file to another folder:

ls
mkdir testfolder
ls
cd testfolder
ls
touch testfile.txt
ls
mkdir testsubfolder
cp testfile.txt ./testsubfolder/testCopy.txt
rm testfile.txt

Special symbols like `.` and `..` represent the current and parent directories, respectively, and are crucial for relative navigation.

ls
cd testsubfolder
ls
cd ..
ls
cd ./testsubfolder
cp testCopy.txt ../testfileRestored.txt
mv testCopy.txt ../
ls
cd ..
ls

The `cat` command is used to print the contents of a file directly to the screen. It is particularly useful for quickly viewing the entire contents of a text file without opening it in an editor. For example, the following command creates a file named `testfilefoo.txt` containing the text "Hello, world." and then displays its contents:

echo 'Hello, world.' >testfilefoo.txt # creates a file testfilefoo.txt with content Hello, world. We will talk about the details later.
cat testfilefoo.txt

Hello, world.

The `head` command displays the first few lines of a file, making it useful for previewing file contents without scrolling through the entire file. For instance, the following script populates a file with multiple lines of text and demonstrates how to use both `cat` and `head`:

rm testfilefoo.txt
echo 'Hello, world 1.' >>testfilefoo.txt
echo 'Hello, world 2.' >>testfilefoo.txt
echo 'Hello, world 3.' >>testfilefoo.txt
echo 'Hello, world 4.' >>testfilefoo.txt
echo 'Hello, world 5.' >>testfilefoo.txt
echo 'Hello, world 6.' >>testfilefoo.txt
echo 'Hello, world 7.' >>testfilefoo.txt
echo 'Hello, world 8.' >>testfilefoo.txt
echo 'Hello, world 9.' >>testfilefoo.txt
echo 'Hello, world 10.' >>testfilefoo.txt
echo 'Hello, world 11.' >>testfilefoo.txt
echo 'displaying file contents with cat:'
cat testfilefoo.txt
echo 'displaying file contents with head:'
head testfilefoo.txt

displaying file contents with cat:
Hello, world 1.
Hello, world 2.
Hello, world 3.
Hello, world 4.
Hello, world 5.
Hello, world 6.
Hello, world 7.
Hello, world 8.
Hello, world 9.
Hello, world 10.
Hello, world 11.
displaying file contents with head:
Hello, world 1.
Hello, world 2.
Hello, world 3.
Hello, world 4.
Hello, world 5.
Hello, world 6.
Hello, world 7.
Hello, world 8.
Hello, world 9.
Hello, world 10.

To learn more about any command, use the `man` command, which displays the manual or help page for the specified command. For example, to learn about the `mkdir` or `ls` commands, run the following:

man mkdir
man ls

After reading the manual, press `q` to quit and return to the command line. This is an invaluable tool for exploring command options and usage details.

A powerful use of the command line is automating repetitive tasks, such as creating multiple folders and files. Here’s a script to create 10 subfolders in a folder called `testfolder`, and then populate each subfolder with 50 test files:

#! /bin/sh
mkdir testfolder
cd testfolder
nfolder=10
nfile=50
for ((i=1; i <= $nfolder; ++i))
do
    mkdir "testsubfolder$i"
    cd "testsubfolder$i"
    for ((j=1; j <= $nfile; ++j))
    do
         touch "testfile$j"
    done
    cd ..
done

Brace expansion provides a convenient shorthand for creating sequences. For example, the script below also creates 10 subfolders and 50 test files in each, but it uses brace expansion for conciseness:

#!/bin/sh
mkdir testfolder
cd testfolder
for i in {1..10}
do
    mkdir testsubfolder$i
    cd testsubfolder$i
    for j in {1..50}
    do
         touch testfile$j
    done
    cd ..
done

However, brace expansion does not handle variables well. If you need to use variables for sequences, the `seq` function is a more robust option:

#!/bin/sh
mkdir testfolder
cd testfolder
nfolder=10
nfile=50
for i in $(seq 1 $nfolder)
do
    mkdir testsubfolder$i
    cd testsubfolder$i
    for j in $(seq 1 $nfile)
    do
         touch testfile$j
    done
    cd ..
done

For an even more compact approach, you can use a combination of brace expansion and `touch` to create multiple files in one command:

#!/bin/sh
mkdir testfolder
cd testfolder
for i in {1..10}
do
    mkdir testsubfolder$i
    touch testsubfolder$i/testfile{1..50}
done

These examples demonstrate the versatility of the command line for file management and automation. Mastering these techniques can save significant time and effort.

Variables are an essential feature of shell scripting, allowing you to store values and reuse them in your scripts. In shell scripts, variables are defined without spaces around the `=` sign. For example:

#!/bin/sh
mkdir testfolder
cd testfolder
nfolder=10 # variable definition; no space before and after = allowed
nfile=50
for ((i=1; i <= $nfolder; ++i)) # reference nfolder via $nfolder
do
    mkdir "testsubfolder$i"
    cd "testsubfolder$i"
    for ((j=1; j <= $nfile; ++j))
    do
         touch "testfile$j"
    done
    cd ..
done

In the above example, the variables `nfolder` and `nfile` define the number of subfolders and files to create, respectively. You can print the value of a variable using the `echo` command:

echo $nfolder

This script demonstrates how variables make your code dynamic and reusable.

Wildcards are symbols that represent one or more characters, allowing you to match patterns in filenames. They are incredibly useful for searching, listing, or performing batch operations on files.

The asterisk (`*`) is the most frequently used wildcard:

• `ls *.png`: Lists all `.png` files in the current directory.
• `ls test*`: Lists all files or directories starting with "test".
• `ls python*`: Lists all files or directories starting with "python".

The question mark (`?`) matches a single character:

touch {python,pythonx,python3,xp}.txt | ls python?.txt

Square brackets (`[]`) define a set of characters to match:

touch {python,pythonx,python3,xp}.txt | ls python[x3].txt

These examples show how wildcards can streamline file operations by matching patterns instead of specifying exact filenames.

The `find` command is a powerful tool for locating files and directories. Its syntax is: ```shell find /path/to/search -options criteria

For instance, let us first set up some test files:

mkdir testfolder
touch testfolder/testfile{1..9}.txt
find testfolder

testfolder
testfolder/testfile8.txt
testfolder/testfile9.txt
testfolder/testfile4.txt
testfolder/testfile5.txt
testfolder/testfile7.txt
testfolder/testfile6.txt
testfolder/testfile2.txt
testfolder/testfile3.txt
testfolder/testfile1.txt

To find files matching a pattern, such as all .txt files with numbers in their names:

find . -name “testfile[1-9].txt”

./testfolder/testfile8.txt
./testfolder/testfile9.txt
./testfolder/testfile4.txt
./testfolder/testfile5.txt
./testfolder/testfile7.txt
./testfolder/testfile6.txt
./testfolder/testfile2.txt
./testfolder/testfile3.txt
./testfolder/testfile1.txt

You can also search for specific file types or contents: • find . -name "python.org": Finds .org files containing “python” in their names. • find . -type d: Lists all directories in the current directory. • find . -empty: Identifies empty files or directories.

The find command can not only locate files but also act on them. For example, you can delete specific files interactively:

touch testfile99.txt
find . -name “testfile99.txt” -exec rm -i {} ;

Alternatively, you can delete files without confirmation (use with caution!):

touch testfile98.txt
find . -name “testfile98.txt” -delete

The following table summarizes common find options:

With these commands, you can efficiently search for files and perform actions, saving time on repetitive tasks.

Input/Output (I/O) operators allow you to redirect the output of commands to files or use the output of one command as input to another. These tools are essential for efficient scripting and command-line operations.

The `>` operator redirects output to a destination other than the screen, such as a file. For example, you can collect all files whose names start with "python" and store them in a file:

ls *python* > allFilesWithPythonInFilename.txt

The `>>` operator works similarly but appends the output to an existing file instead of overwriting it:

echo 'the above are all files whose titles contain python' >> allFilesWithPythonInFilename.txt

The `|` operator, known as a pipe, sends the output of one command to the input of another. For instance, you can concatenate two files and add additional text:

echo 'Hello, world in file 1.' > testfilefoo.txt
echo 'Hello, world in file 2.' > testfilefoo2.txt
cat testfilefoo.txt | cat >> testfilefoo2.txt
echo 'some additional texts' >> testfilefoo2.txt
cat testfilefoo2.txt

The `<` operator redirects input from a source other than the keyboard, enabling automation of input processes in scripts. These operators are indispensable for managing data flows in command-line environments.

The `grep` command, which stands for "global regular expression print," is used to search for text or strings defined by the user within files. Its syntax is as follows: ```shell grep [options] pattern [FILE]

To illustrate, let’s set up a test file containing filenames:

ls python > allFilesWithPythonInFilename.txt
cat allFilesWithPythonInFilename.txt

To find all .org files listed in allFilesWithPythonInFilename.txt, you can use:

grep -i ‘.org’ allFilesWithPythonInFilename.txt

how-to-intall-python.org
learn-python.org
org-python.org
python-and-controls.org
python-basics.org
python-flow-control.org
python-functions.org
python-installed-packages.org
python-intro.org
python-tutorials.org
python-version.org

However, this approach might also match unwanted patterns, such as files named “abc.organization.” To ensure accuracy, you can search for strings ending with .org by appending a $:

grep -i ‘.org$’ allFilesWithPythonInFilename.txt

You can refine your search further to match files starting with “python” and ending with .org by adding a ^ at the start:

grep -i ‘^python.*.org$’ allFilesWithPythonInFilename.txt

Finally, you can redirect the results to a new file:

grep -i ‘^python.*.org$’ allFilesWithPythonInFilename.txt > allOrgFilesStartingWithPython.txt

These examples demonstrate how grep enables precise and efficient searches within files.

The clear command clears the command-line screen or window, providing a fresh workspace for new commands. This is particularly useful when your terminal becomes cluttered with output.

For further exploration of the command line, refer to the following resources: • Get started with the Linux command line and the Shell - Training | Microsoft • Learn Shell - Free Interactive Shell Tutorial • GitHub - BurntSushi/ripgrep: A powerful recursive search tool for files

These resources will deepen your understanding of command-line concepts and tools, empowering you to work more effectively in shell environments.

Write a script that performs the following:

• Create a test folder named "testFolderApril2024".
• Inside testFolderApril2024, create 100 subfolders named "subfolder" followed by number 1 to 100.

• Inside each subfolder,

  • create 50 empty test .txt files named "testfile" follow by number 1 to 50.
  • create 50 empty test .md files named "test" followed by number 1 to 50.
• Find all .txt files in testFolderApril2024 and all its subfolders.
• Find all .md files in testFolderApril2024 and all its subfolders.
• Create a file containing names of all .txt files in testFolderApril2024 and all its subfolders.