scp [[user@]from-host:]source-file [[user@]to-host:][destination-file]

Description of options

from-host

Is the name or IP of the host where the source file is, this can be omitted if the from-host is the host where you are actually issuing the command

user

Is the user which have the right to access the file and directory that is supposed to be copied in the cas of the from-host and the user who has the rights to write in the to-host

source-file

Is the file or files that are going to be copied to the destination host, it can be a directory but in that case you need to specify the -r option to copy the contents of the directory

destination-file

Is the name that the copied file is going to take in the to-host, if none is given all copied files are going to maintain its names

Options

-p

Preserves the modification and access times, as well as the permissions of the source-file in the destination-file

-q

Do not display the progress bar

-r

Recursive, so it copies the contents of the source-file (directory in this case) recursively

-v

Displays debugging messages

Examples

scp *.txt user@remote.server.com:/home/user/

This will copy all files with .txt extension to the directory /home/user in the remote.server.com host

scp -r miguel@10.1.2.2:/home/miguel/ miguel@10.1.2.3:/home/miguel/

This is going to recursively copy all files from miguel’s Home directory on 10.1.2.2 host to his Home directory in 10.1.2.3 host.

——————————

Using SSH and compressed tar gzipped (compressed for faster transfer).

tar czf - </path/to/file_or_folder> --exclude LEAVE | ssh <user@host> tar xzf - -C </path/to/copy_to>

Note:
If you are familiar with the tar command then you know that if you start at say “/” and tar the path “/var/www/html” then when you un-tar you will have the full path in that tar file (/var/www/html). If you wish to avoid this, the cd to “/var/www/html” and “tar ./*” so that you do NOT have full path recursion.

—–

Examples

Copy the file “foobar.txt” from a remote host to the local host

$ scp your_username@remotehost.edu:foobar.txt /some/local/directory

Copy the file “foobar.txt” from the local host to a remote host

$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory

Copy the directory “foo” from the local host to a remote host’s directory “bar”

$ scp -r foo your_username@remotehost.edu:/some/remote/directory/bar

Copy the file “foobar.txt” from remote host “rh1.edu” to remote host “rh2.edu”

$ scp your_username@rh1.edu:/some/remote/directory/foobar.txt \ your_username@rh2.edu:/some/remote/directory/

Copying the files “foo.txt” and “bar.txt” from the local host to your home directory on the remote host

$ scp foo.txt bar.txt your_username@remotehost.edu:~

Copy multiple files from the remote host to your current directory on the local host

$ scp your_username@remotehost.edu:/some/remote/directory/\{a,b,c\} .

$ scp your_username@remotehost.edu:~/\{foo.txt,bar.txt\} .

scp Performance

By default scp uses the Triple-DES cipher to encrypt the data being sent. Using the Blowfish cipher has been shown to increase speed. This can be done by using option -c blowfish in the command line.

$ scp -c blowfish some_file your_username@remotehost.edu:~

It is often suggested that the -C option for compression should also be used to increase speed. The effect of compression, however, will only significantly increase speed if your connection is very slow. Otherwise it may just be adding extra burden to the CPU. An example of using blowfish and compression:

$ scp -c blowfish -C local_file your_username@remotehost.edu:~

—

The most simple case

In the most simple case, you can connect to a server that supports ssh with a syntax as short as this:

[rechosen@localhost ~]$ ssh yourserver

Note: If you do not have any ssh server nearby that you can access, you can also try this command with your own computer as a server. To do this, replace “yourserver” with “localhost”.

Of course, yourserver should be replaced by a hostname or an ip address of the server you want to connect to. As you can see in the terminal snippet, I am logged in as rechosen. If you do not specify a username (I’ll explain how to do that later in this tutorial), SSH will assume that you want to login with the username you’re currently logged in with. So, in this case, SSH will try the username rechosen.

Of course, you need to be sure that the server supports ssh connections. The ssh client tries to connect to port 22 defaultly. This means that, if you want to connect to a remote host with the default settings, you should make sure that, if applicable, port 22 is forwarded to the server you’re trying to connect to. You will find more regarding the SSH port further in this tutorial.

Now, back to the command we ran. If the server supports SSH connections and you can reach it by port 22, you should be prompted for a password (if this is the first time you try to connect to the server, ssh will first ask the question if you want to continue connecting, which can generally just be answered with a ‘yes’). If you type a password here, you won’t see asterisks appearing. Don’t panic, this is ssh’s normal behaviour. It makes connecting using ssh even more safe, because any accidental spectators won’t be able to see the length of the password. After entering the password, if the username and the password were correct, you should be running a shell on the server. If not, make sure you are connecting to a server of which you know that you should be able to login with your username and the specified password. You could try connecting to your own computer (see the note beneath the terminal quote) or read on to learn how to specify an other username.

Once you’re done trying the ssh shell, you can exit it by pressing Ctrl + D.

Specifying a username

It’s actually quite simple to specify a different username. You might even already be familiar with it. See the following example:

[rechosen@localhost ~]$ ssh yourusername@yourserver

The above will make ssh try to connect with the username “yourusername” instead of (in my case) rechosen. This syntax is also used by a lot of other protocols, so it’ll always come in handy to know it. By the way, you will still be asked for a password. For security reasons, it is not even possible to directly specify the password in the syntax. You will always be asked interactively, unless you start configuring the server in an advanced way (which is exactly why that topic is out of this tutorials scope: this tutorial documents how to use the clients, not how to configure the server).

Specifying a port

There are many reasons to move the ssh service to an other port. One of them is avoiding brute-force login attempts. Certain hackers try to get access to ssh servers by trying a lot of common usernames with common passwords (think of a user “john” with password “doe”). Although it is very unlikely that these hackers will ever get access to the system, there is an other aspect of the brute-force attacks that you’ll generally want to avoid: the system and connection load. The brute-force attacks usually are done with dozens or even thousands of tries a second, and this unnecessarily slows down the server and takes some bandwidth which could’ve been used a lot better. By changing the port to a non-default one, the scripts of the hackers will just be refused and most of the bandwidth will be saved.

As the ssh command can’t just guess the port, we will have to specify it if it’s not the default 22 one. You can do that this way:

[rechosen@localhost ~]$ ssh -p yourport yourusername@yourserver

Of course, you will have to replace “yourport” with the port number. These is an important difference between ssh and scp on this point. I’ll explain it further on.

Running a command on the remote server

Sometimes, especially in scripts, you’ll want to connect to the remote server, run a single command and then exit again. The ssh command has a nice feature for this. You can just specify the command after the options, username and hostname. Have a look at this:

[rechosen@localhost ~]$ ssh yourusername@yourserver updatedb

This will make the server update its searching database. Of course, this is a very simple command without arguments. What if you’d want to tell someone about the latest news you read on the web? You might think that the following will give him/her that message:

[rechosen@localhost ~]$ ssh yourusername@yourserver wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”

However, bash will give an error if you run this command:

bash: !”: event not found

What happened? Bash (the program behind your shell) tried to interpret the command you wanted to give ssh. This fails because there are exclamation marks in the command, which bash will interpret as special characters that should initiate a bash function. But we don’t want this, we just want bash to give the command to ssh! Well, there’s a very simple way to tell bash not to worry about the contents of the command but just pass it on to ssh already: wrapping it in single quotes. Have a look at this:

[rechosen@localhost ~]$ ssh yourusername@yourserver ‘wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”‘

The single quotes prevent bash from trying to interpret the command, so ssh receives it unmodified and can send it to the server as it should. Don’t forget that the single quotes should be around the whole command, not anywhere else.

SCP

The scp command allows you to copy files over ssh connections. This is pretty useful if you want to transport files between computers, for example to backup something. The scp command uses the ssh command and they are very much alike. However, there are some important differences.

The scp command can be used in three* ways: to copy from a (remote) server to your computer, to copy from your computer to a (remote) server, and to copy from a (remote) server to another (remote) server. In the third case, the data is transferred directly between the servers; your own computer will only tell the servers what to do. These options are very useful for a lot of things that require files to be transferred, so let’s have a look at the syntax of this command:

[rechosen@localhost ~]$ scp examplefile yourusername@yourserver:/home/yourusername/

Looks quite familiar, right? But there are differences. The command above will transfer the file “examplefile” to the directory “/home/yourusername/” at the server “yourserver”, trying to get ssh acces with the username “yourusername”. That’s quite a lot information, but scp really needs it all. Well, almost all of it. You could leave out the “yourusername@” in front of “yourserver”, but only if you want to login on the server with your current username on your own computer. Let’s have a closer look at the end of the command. There’s a colon over there, with a directory after it. Just like Linux’s normal cp command, scp will need to know both the source file(s) and the target directory (or file). For remote hosts, the file(s)/directory are given to the scp command is this way.

You can also copy a file (or multiple files) from the (remote) server to your own computer. Let’s have a look at an example of that:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile .

Note: The dot at the end means the current local directory. This is a handy trick that can be used about everywhere in Linux. Besides a single dot, you can also type a double dot ( .. ), which is the parent directory of the current directory.

This will copy the file “/home/yourusername/examplefile” to the current directory on your own computer, provided that the username and password are correct and that the file actually exists.

You probably already guessed that the following command copies a file from a (remote) server to another (remote) server:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile yourusername2@yourserver2:/home/yourusername2/

Please note that, to make the above command work, the servers must be able to reach each other, as the data will be transferred directly between them. If the servers somehow can’t reach each other (for example, if port 22 is not open on one of the sides) you won’t be able to copy anything. In that case, copy the files to your own computer first, then to the other host. Or make the servers able to reach each other (for example by opening the port).

Well, those are the main uses of scp. We’ll now go a bit more in-depth about the differences between ssh and scp.

*: Actually you can also use it just like the normal cp command, withhout any ssh connections in it, but that’s quite useless. It requires you to type an extra ‘s’ =).

Specifying a port with scp

The scp command acts a little different when it comes to ports. You’d expect that specifying a port should be done this way:

[rechosen@localhost ~]$ scp -p yourport yourusername@yourserver:/home/yourusername/examplefile .

However, that will not work. You will get an error message like this one:

cp: cannot stat `yourport’: No such file or directory

This is caused by the different architecture of scp. It aims to resemble cp, and cp also features the -p option. However, in cp terms it means ‘preserve’, and it causes the cp command to preserve things like ownership, permissions and creation dates. The scp command can also preserve things like that, and the -p option enables this feature. The port specification should be done with the -P option. Therefore, the following command will work:

[rechosen@localhost ~]$ scp -P yourport yourusername@yourserver:/home/yourusername/examplefile .

Also note that the -P option must be in front of the (remote) server. The ssh command will still work if you put -p yourport behind the host syntax, but scp won’t. Why? Because scp also supports copying between two servers and therefore needs to know which server the -P option applies to.

Another difference between scp and ssh

Unlike ssh, scp cannot be used to run a command on a (remote) server, as it already uses that feature of ssh to start the scp server on the host. The scp command does have an option that accepts a program (the -S option), but this program will then be used instead of ssh to establish the encrypted connection, and it will not be executed on the remote host.

Tips & Tricks with ssh and scp

Quite a handy thing about scp is that it supports asterisks. You can copy all files in a remote directory in a way like this:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/* .

And you can also just copy a whole directory by specifying the -r (recursive) option:

[rechosen@localhost ~]$ scp -r yourusername@yourserver:/home/yourusername/ .

Both of these also work when copying to a (remote) server or copying between a (remote) server and another (remote) server.

The ssh command can come in handy if you don’t know the exact location of the file you want to copy with scp. First, ssh to the (remote) server:

[rechosen@localhost ~]$ ssh yourusername@yourserver

Then browse to the right directory with cd. This is essential Linux terminal knowledge, so I won’t explain it here. When you’re in the right directory, you can get the full path with this command:

[rechosen@localhost ~]$ pwd

Note: pwd is an abbreviation of Print Working Directory, which is a useful way to remember the command.

You can then copy this output, leave the ssh shell by pressing Ctrl + D, and then paste the full directory path in your scp command. This saves a lot of remembering and typing!

You can also limit the bandwidth scp may use when copying. This is very useful if you’re wanting to copy a huge amount of data without suffering from slow internet for a long time. Limiting bandwidth is done this way:

scp -l bandwidthlimit yourusername@yourserver:/home/yourusername/* .

The bandwidth is specified in Kbit/sec. What does this mean? Eight bits is one byte. If you want to copy no faster than 10 Kbyte/sec, set the limit to 80. If you want to copy no faster than 80 Kbyte/sec, set the limit to 640. Get it? You should set the limit to eight times the maximum Kbyte/sec you want it to be. I’d recommend to set the -l option with all scp’ing you do on a connection that other people need to use, too. A big amount of copying can virtually block a whole 10 Mbit network if you’re using hubs.

Final Words

Well, that was it! I hope you learned a lot. Of course, you can always have a quick look at this tutorial again if you forgot something. Please tell other people who might be interested about this tutorial, you’ll help this blog to grow if you do =). Thank you for reading and have a lot of fun with your new knowledge!

Makes copy of subset of dir/ and below with files that match the criteria. It keeps the nested directory structure. Uses

-print0 | xargs -0

to handle spaces in Windows names

find /cygdrive/f/dir1/ -name '*.doc' -print0 | xargs -0 cp -a --target-directory=/cygdrive/c/Temp --parents

This little script needs a lot of help But it will recurse down through the current directory and create a listing of files, their folders, sizes, and modification dates and times. it was written to run on Windows under Cygwin.

#!/bin/sh
# v1 jcz 30-dec-2009
# This script will search for files of a certain type and create a text file of the results
# TODO:
#

############################
# enable for debugging #####
############################
# set -vx

############################
#  Global script variables block
############################
# Date and other variables pretty self explanatory, S is seconds
# date format is currently YYYYMMDD_HHMMSS
dater=$(date +%Y-%m-%d %H:%M:%S)
dayer=$(date +%a)
namer=$(whoami)
hoster=$(hostname)
directory=$(pwd)
filenamer=$(date +%Y%m%d_%H%M%S).txt
# sets day of the week for incremental backups
set $(date)

############################
#  Clear the screen and introduce the user to the script
############################

clear
echo “”
echo “WELCOME TO THE FIND TO LIST SCRIPT”
echo “”

############################
#  Wait for the user to enter a new file extension and capture the value as a variable
############################
echo -n “Enter file extension to search for, without the leading dot (e.g. txt): ”
read fileext

############################
#  Wait for the user to enter a new file destination
############################
echo -n “Enter a new log file destination without ending slash (e.g., /cygdrive/c ): ”
read filedest

############################
#  Create the log file for the script named after the file extension
############################
echo “—-” >> $filedest/$filenamer
# echo “—-” > $filedest/$fileext_files_from_$directory_on_$dater.txt
echo “File created on: “$dater  >> $filedest/$filenamer
echo “Setup script was run in: “$directory >> $filedest/$filenamer
echo “By user” $namer  >> $filedest/$filenamer
echo “Searching for files ending in: ” $fileext >> $filedest/$filenamer
echo “This file was written to: ” $filedest/$filenamer >> $filedest/$filenamer
echo “***************************”  >>$filedest/$filenamer
echo “” >> $filedest/$filenamer

find . -name ‘*.’$fileext -type f -print0 | xargs -0  stat -c ‘file: %N | bytes: %s | modtime: %y’ >> $filedest/$filenamer

echo -n “Hit enter to continue ”
read none

echo “”
echo “* Now I will show you the file and be done”
echo “”
echo -n “Hit enter to list or Ctrl-c to quit ”
read none
less $filedest/$filenamer

This is another one of those notes to myself because I look this stuff up every six months.

Basics of find. The following is a complete rip off of the content at: http://content.hccfl.edu/pollock/unix/findcmd.htm

But I wanted to keep a copy safe here. Thanks Wayne Pollock on 12/30/2009 10:27:30.

FIND

find . -name foo\*bar

find / -type f -mtime -7 | xargs tar -rf weekly_incremental.tar
gzip weekly_incremental.tar

find / -name core | xargs /bin/rm -f
find / -name core -exec /bin/rm -f '{}' \; # same thing
find / -name core -delete                  # same if using Gnu find

find / -mmin -10

find . -perm -o=w

find / -name /dev -prune | xargs tar ...

find . -mtime 0   # find files modified between now and 1 day ago
                  # (i.e., within the past 24 hours)
find . -mtime -1  # find files modified less than 1 day ago
                  # (i.e., within the past 24 hours, as before)
find . -mtime 1   # find files modified between 24 and 48 hours ago
find . -mtime +1  # find files modified more than 48 hours ago

find . -mmin +5 -mmin -10 # find files modified between
                          # 6 and 9 minutes ago

find . -maxdepth 1 -name '[!.]*' -printf 'Name: %16f Size: %6s\n'

find . -path './*' -prune ...

find . ! -name . -prune ...

 find . \( -name . -o -prune \) ...

find / -noleaf -wholename '/proc' -prune \
     -o -wholename '/sys' -prune \
     -o -wholename '/dev' -prune \
     -o -wholename '/windows-C-Drive' -prune \
     -o -perm -2 ! -type l  ! -type s \
     ! \( -type d -perm -1000 \) -print

  find mount-point -xdev -inum inode-number

  find whatever... | xargs command

find . -name \*.txt | xargs cp /tmp  # This won't work!

find / -name core -exec /bin/rm -f '{}' +

sh -c 'command-line' [ command-name [ args... ] ]

find . -name '*.txt' -type f \
  -exec sh -c 'exec cp -f "$@" /tmp' find-copy {} +

find \( expression \) -print

$ find -name tmp -prune -o -name \*.txt
./bin/data/secret.txt
./tmp
./missingEOL.txt
./public_html/graphics/README.txt
./datafile2.txt
./datafile.txt

$ find -name tmp -prune -o -name \*.txt -print
./bin/data/secret.txt
./missingEOL.txt
./public_html/graphics/README.txt
./datafile2.txt
./datafile.txt

find . \( -name tmp -prune -o -name \*.txt \) -print

See Also Stat

$ stat –help
Usage: stat [OPTION] FILE…
Display file or file system status.

-L, –dereference     follow links
-f, –file-system     display file system status instead of file status
-c  –format=FORMAT   use the specified FORMAT instead of the default;
output a newline after each use of FORMAT
–printf=FORMAT   like –format, but interpret backslash escapes,
and do not output a mandatory trailing newline.
If you want a newline, include \n in FORMAT.
-t, –terse           print the information in terse form
–append-exe      append .exe if cygwin magic was needed
–help     display this help and exit
–version  output version information and exit

The valid format sequences for files (without –file-system):

%a   Access rights in octal
%A   Access rights in human readable form
%b   Number of blocks allocated (see %B)
%B   The size in bytes of each block reported by %b
%C   SELinux security context string
%d   Device number in decimal
%D   Device number in hex
%f   Raw mode in hex
%F   File type
%g   Group ID of owner
%G   Group name of owner
%h   Number of hard links
%i   Inode number
%n   File name
%N   Quoted file name with dereference if symbolic link
%o   I/O block size
%s   Total size, in bytes
%t   Major device type in hex
%T   Minor device type in hex
%u   User ID of owner
%U   User name of owner
%x   Time of last access
%X   Time of last access as seconds since Epoch
%y   Time of last modification
%Y   Time of last modification as seconds since Epoch
%z   Time of last change
%Z   Time of last change as seconds since Epoch

Valid format sequences for file systems:

%a   Free blocks available to non-superuser
%b   Total data blocks in file system
%c   Total file nodes in file system
%d   Free file nodes in file system
%f   Free blocks in file system
%C   SELinux security context string
%i   File System ID in hex
%l   Maximum length of filenames
%n   File name
%s   Block size (for faster transfers)
%S   Fundamental block size (for block counts)
%t   Type in hex
%T   Type in human readable form

This might be handy with xargs from find. Here’s an example from cygwin that outputs a formatted display of filename, size in bytes, and date time when the file was last modified.

$ stat -c “file: %N | bytes: %s | modtime: %y” *.ini
file: `XrxWm.ini’ | bytes: 1064 | modtime: 2009-09-14 12:15:19.531250000 -0400
file: `ntuser.ini’ | bytes: 178 | modtime: 2009-12-29 09:19:04.843750000 -0500

This little command works nicely o Windows (to esacpe the spaces in file names).

$ find . -name ‘*.enl’ -type f -print0 | xargs -0  stat -c “file: %N | bytes: %s | modtime: %y” >> find-output.txt

To produce

file: `./Sample library.enl’ | bytes: 13918 | modtime: 2009-12-18 15:02:41.671875000 -0500

Edit

/etc/network/interfaces

and adjust it to your needs (in this example setup I will use the IP address 192.168.0.100):
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# This is a list of hotpluggable network interfaces.
# They will be activated automatically by the hotplug subsystem.
mapping hotplug
script grep
map eth0


# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.100
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
gateway 192.168.0.1

Then do

sudo /etc/init.d/networking restart

to restart the network.

My latest project is to turn my favorite Linux desktop configurations into stand alone distributions that I can run as LiveCDs or install anywhere.

Here are some links so that I don’t forget.

http://www.ubuntugeek.com/creating-custom-ubuntu-live-cd-with-remastersys.html