Some Examples of Bash for Use in Test

Time has come to finally write something of technical nature, not just another advice on how others should work to make me happy. So here is a cheatsheet for a tester who spends at least some of her time with nix console. For obvious reasons it wasn't my intention to cover everything, so please take it just as a memory refresher (or a source of ideas it you find it useful).

NB: It is assumed that as long as you understand what man is, you are fine (use it if my explanation is not enough).


Contents

1. On what's this and where we are
2. Looking or this fantastic file
3. Looking inside this fantastic file
4. Writing to a fantastic file
5. Doing bulk operations with a fantastic file
6. Running and killing something
7. How do I get this fantastic file from a remote machine
8. Making one fantastic file out of many fantastic files
9. Network: a couple of examples with ports
10. Sources

#1 On what's this and where we are

The ultimate source of information is a man command (man is a shorthand for manual). Mans are almost always there for you unless you are using some pseudo-cli under windows with a limited functionality or a mobile device

Easiest way to read a man:

man <command name>

Looking for a particular section of a man:

man [ -s ] <command section> <command name>

Some other interesting commands:

whatis
apropos
info
help

Sometimes the information you are looking for is contained within special system files like this (finding your nix distribution name and version):

/etc/lsb_release file
lsb_release command
/proc/version file

More basic info about what's going on on your system (processes, memory, state of the file system):

top
free
du -sch
df -h



#2 Looking for this fantastic file

Your current location:

pwd

Getting home:

cd or cd ~

Let's look around (remember to use tab key for name expansion + use man to find out more about the options):

ls -lhat

Looking for a file by its name within the directory you are in (returns a subset of whatever ls returns limited by a given substring):

ls *substring*

Getting all files starting with r from /etc directory:

ls /etc/* | grep ^r

Finding a file of type 'file' (alternative is d for directories) and with a specific name (alternative is -iname or case-insensitive search by name):

find ~/env/myfolder/ -type f -name simulator.properties

Looking for a file by its contents from your current location:

grep -r 'textinside' .

Doing something with a found file (like opening it in cat and applying grep to it, reads 'find here by type and execute something and process output as I say'):

find . -type f -exec cat {} \+ | grep -i 123456789

find . -exec cat {} \+ | grep -i mykeyword
 
find . -type f -exec {} \+ -exec cat {} \+ | grep 'blah.blah.string'
 
find . -type f -exec grep mykeyword {} \+

Doing some regexp (complex regexp is only supported by egrep, reads specific set of values from a set of xml files):

egrep -r textinside\>[0-9]\{0,15\} *.xml

Searching for contents by date (cut parses by delimiters, xargs redirects output):

ls -laht | grep Mar | cut -d ' ' -f11 | xargs cat

ls -laht | grep 'Mar  2 14:59' | cut -d ' ' -f11 | xargs cat



#3 Looking inside this fantastic file

Open and read:

less +F

less +G

tail -10000

tail +f

cat

Search for something inside:

\ or ? (searching for a pattern inside an editor)

grep

sed

Copying a part of a log to a separate file (tee command forks the output which helps to write it to a file and read with less at the same time; date may be automatic or manual)

inclusive:

sed -n '/11:59:17,206/,/12:02:14,606/p' ~/env/logs/mybeautifulapp.log | tee ~/tmp/log_excerpts/mybeautifulapp-$(date +%F_%H-%M-%S) | less

sed -n '/17:27:52,425/,/17:37:44,166/p' ../env/logs/server.log | tee excerpt-$(date +%F) | less

sed -n '/17:23:35,997/,/17:23:36,009/p' ../env/logs/server.log | tee excerpt-2014-06-18 | less

not inclusive:

sed '1,/17:23:35,996/d;/17:23:36,009/,$d' ../env/logs/server.log | tee excerpt-2014-06-18 | less

Reading more than one log at a time:

watch -n1 cat <path to log1> <path to log2> <path to log3>

Reading data from csv or other delimited source

CSVs:

cut -d \; -f 7,11,12 *

Output of ls:

ls -lhat | tr -s ' ' | cut -d' ' -f9



#4 Writing to a fantastic file

Create an empty one:

touch

vim

Redirect output:

>

>>

tee

xargs

Playing with the file descriptors (all errors to regular output and then all to a file by appending):

2>&1>>myfile.txt
Write something to csv:

ls -lha <FILE TO PROCESS> | sed 's/[ \t]/,/g' > result.csv
Writing file structure to a file:

find ~/etc/myfolder/ -type f | sed 's/[ \t]/,/g' > result.csv
Writing a list of files to csv:

ls -lha ~/env/folder | sed 's/[ \t]/,/g' > result.csv



#5 Doing bulk operations with a fantastic file

Copying and moving:

cp -avr <source> <target>

mv <source> <target>

Copy several found files somewhere else (this solution may overwrite files with identical names):

find . -type f -name \*partofaname\*.xml -exec \cp -avr {} . \;

find . -type f -exec sh -c 'cp -avr {} . ' ;

Copy file structure only:

cd destination/dir

find /source/dir -type d -printf "%P\n" | xargs mkdir -p

Bulk content update with sed (this string searches for all items of type "file" and extention ".xml", edit files in place (makes backup if extension supplied) according to the provided pattern):

find . -type f -name \*.xml -exec sed -i.xml-backup 's|<PATTERN TO CHANGE>|<REPLACEMENT>|g' {} +
Bulk rename files (selects all files starting with 'myfile', passes them one by one to sed, sed keeps 'myfile' part of a filename and substitutes the rest, then mv & is applied to the match, and renames myfile.xml to myfile-1.xml, them passes it to shell for processing):

ls myfile*.xml | sed 's/myfile\(.*\)/mv & myfile\1/' | sh
Using xargs (removing all except):

ls -lhat | tr -s ' ' | cut -d' ' -f9 | grep -v 20141205 | xargs rm



#6 Running and killing something

Starting executables:

./

source

Make sure your permissions are ok:

chown

chmod

Make file executable:

chmod a+x

chmod 755

Find process:

ps aux | grep -i keyword | grep -i --color=auto keyword2

ps aux | grep java | grep -v keyword

And kill it:

kill -9 <PID>



#7 How do I get this fantastic file from a remote machine

Remote to local

Pattern:

scp -r <USER>@<HOST>:<REMOTE SOURCE FOLDER/FILE> <LOCAL TARGET FOLDER/FILE>

Example:

scp -r user@remotehost:~/remotefolder/sourcefile.tar ~/localfolder/targetfile.tar

Local to remote

Pattern:

scp -r <SOURCE FOLDER/FILE> <USER>@<HOST>:<TARGET FOLDER/FILE>

Example:

scp -r ~/localfolder/sourcefile.tar user@remotehost:~/remotefolder/targetfile.tar



#8 Making one fantastic file out of many fantastic files

Sometimes you need to quickly get stuff from a remote machine. It's quicker to archive it first.

tar

comress

tar -cvf my_archive.tar folder/

decompress

tar -xvf archive.tar

tar -xvf archive.tar <target=name of root directory archived>

tar.gz

compress

tar -zcvf my_archive.tar.gz <target folder>

decompress

tar -zcvf my_archive.tar.gz <target folder>

view tar without extracting it

tar -tvf my_arch.tar



#9 Network: a couple of examples with ports

Checking if a port is being listened on:

lsof -i :<PORT>

netstat -lnt | grep <PORT>

nc -z localhost <PORT>



#10 Sources

http://www.google.com
http://en.wikipedia.org
http://stackoverflow.com/

The ABS guide:
http://tldp.org/LDP/abs/html/

Regexps & backreferences:
http://www.robelle.com/smugbook/regexpr.html
http://code.snipcademy.com/tutorials/shell-scripting/sed/backreferences-ampersands

Editors (vim):
http://vim.wikia.com/
http://www.radford.edu/%7Emhtay/CPSC120/VIM_Editor_Commands.htm

Editors (nano):
http://www.nano-editor.org/

Working with CSVs:
http://bconnelly.net/working-with-csvs-on-the-command-line/
http://www.linuxquestions.org/questions/linux-newbie-8/howto-ls-output-to-csv-format-808545/
http://www.unix.com/unix-for-dummies-questions-and-answers/221439-create-csv-output-filenames-file-size.html
http://stackoverflow.com/questions/14573262/convert-ls-output-into-csv

Using find:
http://www.computerhope.com/unix/ufind
http://www.bergercity.de/linux/find-exec-mit-pipe/comment-page-1/

Get structure only (folders): http://www.unix.com/shell-programming-and-scripting/119654-copy-only-folder-structure.html

Getting your distro name: http://www.cyberciti.biz/faq/find-linux-distribution-name-version-number/

Copying and moving:
http://www.bergercity.de/linux/find-exec-mit-pipe/comment-page-1/
http://www.computerhope.com/unix/ufind
http://www.unix.com/shell-programming-and-scripting/119654-copy-only-folder-structure.html
http://linuxconfig.org/bash-printf-syntax-basics-with-examples (man find at hand, for -printf)

Permissions:
http://stackoverflow.com/questions/18596778/difference-between-using-chmod-ax-and-chmod-755

Bulk update:
http://stackoverflow.com/questions/13364514/batch-replace-text-inside-text-file-linux-osx-commandline

Checking the ports:
http://stackoverflow.com/questions/4922943/how-to-test-if-remote-tcp-port-is-opened-from-shell-script
http://stackoverflow.com/questions/9609130/quick-way-to-find-if-a-port-is-open-on-linux
http://stackoverflow.com/questions/21897119/shell-script-how-do-i-determine-if-a-port-is-in-use-e-g-via-netstat

SCP in console:
http://stackoverflow.com/questions/11304895/how-to-scp-a-folder-from-remote-to-local
http://unix.stackexchange.com/questions/115560/use-scp-to-transfer-a-file-from-local-directory-x-to-remote-directory-y