Sending files over SFTP in an automated fashion

The exerpts below describe how to use SFTP to transfer files from one
machine to another in automated fashion, such as using shell scripts. I
needed to do this as part of a back up script.

Use the ssh authorized_keys functionality instead of trying to pass in
a username and password. Create a key on the client side using
ssh-keygen (it will have to be nonpassworded if you want this to be
fully automated) and copy the public key to the target user’s
authorized_keys file on the ssh/sftp server. For extra security, an
option (“from”) is available to limit the use of that key to
connections coming from your client server. This will give ssh or sftp
access to anyone who comes from the client server with the right
private key (no worse than having a name and password hardcoded in a
script). See the SSH docs for more detail.

Set that up and you’ll be past the authentication issues. Then check
out the -b option of sftp to give it a batch file with sftp commands to
automatically execute:

sftp -b batchfile targetuser@targethost

Now, you have to configure the server ahead of time to consider the source trusted. Use the following steps to make that happen:

create the keys for the origin account, i.e. the account that performs the copy:

$ cd ~/.ssh

$ ssh-keygen -t dsa

You are asked for a passphrase, do not enter a passphrase, type for empty passphrase.

verify the creation of the 2 files:

~/.ssh/id_dsa

~/.ssh/id_dsa.pub

copy ~/.ssh/id_dsa.pub to the destination node

login into the destination node and verify if file ~/.ssh/authorized_keys is already present, if not do:

$ cd ~/.ssh

$ mv id_dsa.pub authorized_keys

Verify ~/.ssh/authorized_keys and add/replace id_dsa.pub as needed.

Then, run your sftp with a -b extension. This will put it in batch mode
and allow it to draw its commands from a text, or batchfile. You need
to specify the batch filename after -b.

—————————————————————————————-

Example 2

Log in to your account on the front-end node (the client machine). Here
we describe the machine you want to login TO as the client (compute
node). In the example below we are using root, but you should likely
use a standard user name. Go to the .ssh directory of your home
directory. If it isn’t there, you may create one with the command

mkdir ~/.ssh

In the examples below, you should conceptually replace /root/ with /home/youruser/.ssh.

Create an rsa key pair by executing ssh-keygen with the

“rsa” option:

ssh-keygen -t rsa

The ssh-keygen program will respond with:

Generating a public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Type the Enter key to accept the default value:

[Enter]

The ssh-keygen program will respond with:

Created directory ‘/root/.ssh’.

Enter passphrase (empty for no passphrase):

Type the Enter key to accept the default value:

[Enter]

The ssh-keygen program will respond with:

Enter same passphrase again:

Type the Enter key again:

[Enter]

The ssh-keygen program will respond with:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

[fingerprint] root@[hostname]

Create the ssh directory for each compute node’s root

account (on the first use of ssh the system automatically

creates the ssh directory for you).

Login to a compute node. Type:

ssh root@[compute node address]

Enter the root password.

Check if a .ssh directory exists. Type:

ls -la

If there is no .ssh directory listed, type:

mkdir .ssh

ls -la

chmod go-rwx .ssh

ls -la

The .ssh listing should look like:

drwx—— 2 root root 4096 [date & time] .ssh

The .ssh directory is now only accessible by the user root.

Log out of the compute node. Type:

logout

Use sftp (secure ftp) to copy the generated rsa public key

to each compute node’s root account secure shell directory

as the file authorized_keys. Type:

sftp root@[compute node address]

If prompted to continue connecting, type:

yes

Log in with the root password for that compute node.

You will get the sftp prompt:

sftp>

Change to the secure shell directory. Type:

cd .ssh

Copy the rsa public key to the compute node. Type:

put /root/.ssh/id_rsa.pub authorized_keys

Exit sftp. Type:

exit

Repeat this procedure for each compute node.

To test that the secure automatic login is working

properly from the front-end node, type:

ssh root@[compute node address]

The system should log you in without prompting for a

password.

Log out of the compute node. Type:

logout

Also copy the rsa public key to the authorized_keys file on

the front-end node. Type:

cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys

root on the front-end node can now securely access all nodes

in the cluster without having to enter a password.

Automate your life

What is crontab?

If you’ve worked with Perl
scripts I’m sure you’ve heard or seen the words “cron”,

“crontab”, or “cron job” before. If not, you’ll learn
something new today!

The

crontab command allows you to tell your server to execute a file at a specific

time, as often as you want – like once a day or once a minute. Most commonly,
crontab

is used to execute a cgi script to perform a certain task repeatedly rather than

doing it manually. Saves a lot of time!

NOTE: many hosting
companies disable crontab on their hosting accounts because

it is often abused. Using it to repeatedly execute a file takes up precious cpu

time and if misused can slow down a server immensely. However, if you talk to
your

hosting company and let them know what file you are going to run they more than

likely will be happy to enable crontab on your system. If they won’t, then you
might want to think about getting a dedicated server…

Where do I start?

To use crontab, you have
to be able to telnet into your server. This is accomplished

in Windows by going to Start->Run and typing in “telnet yourdomain.com”
and hitting

“OK”. A new window will pop up and you will have to put in your
username and password.

If you normally use an FTP client, usually it’s the same username and password.

If successful. you will
then get a command prompt: $

First, you can see the
crontab usage info by typing in “crontab” and hitting return.

Here’s what it looks like on my server:

usage: crontab [-u user]
file

crontab [-u user] { -e | -l | -r }

-e (edit user’s crontab)

-l (list user’s crontab)

-r (delete user’s crontab)

So, if you type:

crontab -l

you will get a list of the crontab jobs already running on your system. Try it
out. You probably don’t have any running so you will get an empty list…

How do I set up a
crontab job?

While you can edit the crontab file directly through telnet, I’ve found that the
easiest way for a beginner to start a crontab job is to create a text file
containing your crontab instructions, upload it to your main directory, telnet
into your system, and then just type:

crontab myfile.txt

and the crontab job will
be created.

So what do I put in the text file? What is the syntax?

This text file will
minimally only have to have one line containing the information

for your cron job. Here is a run down of the syntax:

0 1 * * * /path/to/cgi-bin/yourscript.cgi

| | | | |

| | | | |________________ day of week

| | | |__________________ month of year

| | |____________________ day of month

| |______________________ hour of day

|________________________ minute of hour

The * in the above example
basically means “every”. So, in this example the script would execute
*every* month, *every* day of the week, *every* day of the month, and then ONLY
at hour 1 of the day. So – this script would execute once a day at 1 am in the
morning.

Say you wanted to execute
it 4 times a day, you would type in:

0 1,7,13,19 * * * /path/to/cgi-bin/autocron.cgi

This executes autocron.cgi
four times a day about every 6 hours. Notice that

you can use commas to add in more times – but always keep the spaces to
delineate

the time frames.

Another example, to
execute autocron.cgi twice a month:

0 1 1,15 * * /path/to/cgi-bin/autocron.cgi

This line would execute
the file on the 1st and 15th of the month at 1AM.

Can I run multiple
crontab jobs?

In a nutshell – YES! Just
add multiple lines to the text file using the syntax above for every

script that you would like to run. Make sure that the paths to the scripts are

correct and you’ve double checked your time settings.

How can I tell that it’s working?

Simple, after you have
executed the text file with your one line crontab, just

type:

crontab
-l

At your command prompt and you will see something like this:

# (cronjob.txt installed on Tue Apr 11 21:19:12 2000)

# (Cron version — $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)

0 1 * * * /path/to/cgi-bin/yourscript.cgi

All of the running jobs
will be listed.

Can I edit my
crontab file through telnet?

Yes, however, I prefer to
just delete the current crontab file and create a new

one with a new text file. Just type:

crontab
-e

and your crontab file editor will pop up and you can edit the file. Some telnet

clients make it hard to edit this file though – which is why I just delete

and recreate my crontab file.

How do I delete a crontab file?

This is easy! At the
telnet prompt, type:

crontab
-r

and the crontab file will be completely deleted. You can check this by using the

“crontab -l” command. And of course, you can recreate your file by
typing

“crontab myfile.txt”. (or whatever you named the text file containing
your

crontab lines)

Linux.com: Bugs in your shell script?

By: Larry Reckner
Topics: Shell
Subsection: Intermediate
If your writing a shell script and want to watch exactly what is going on (very usefull for debugging purposes), add the line

set -vx

in the beginning of the script.

The shell script will then output what it’s doing so you can watch.

This can also be done via command line by doing

sh -x filename

The power of xargs

Excerpted (stolen) from http://www.linux.com/print.pl?sid=04/04/13/211209

xargs is your friend. Using xargs, you can pull off feats of greatness
and not have to write a script to do it. xargs can take care of things
right on the command line. Though I focus mainly on files in this
article (it’s what I use it for almost exclusively), it’s important to
remember that xargs acts on standard input, which could mean lines
redirected from /var/log/messages or urls or whatever else you can
manage to point in its direction.

$ > rpm -qa | grep mozilla | xargs -n1 -p rpm -e –nodeps

What this says in English, is “Using RPM, query all (-qa) packages,
look for mozilla in the package name, and send the results one at a
time (-n1), to RPM’s uninstall command, and I don’t care about
dependencies, thank you very much (“rpm -e –nodeps”). Also, in case
there’s something that contains the word “mozilla” that I DON’T want
erased, prompt me (-p) before uninstalling.” The above command saves
you from having to manually list the packages containing the string
“mozilla,” then manually running separate “rpm -e” commands against
them one at a time.

$ > find / -name *.mp3 -type f -print | xargs tar -cvzf mp3z.tar.gz

This finds all the mp3z on my entire drive and puts ‘em all in a tar
file, and then I can untar them wherever I want :) I actually could’ve
piped that xargs “tar” line into a “tar xvzf” line to automatically
untar them. I also could’ve left out the “-type f” if I had grip set up
to use a custom directory structure that I wanted to preserve. You get
the idea :) PS – this works for other types of files, too, like finding
all the files that belong to you, tarring them and sending the tar to a
backup somewhere, so it does have legitimate use.

$> ls *.mp3 | xargs -n1 -i cp {} backup/.

This command takes all of the MP3 files in the current directory, and
feeds them one at a time (-n1) to the cp command, where the file
argument coming in from ls will replace the curly braces. Notice I
didn’t specify a string with “-i.” I don’t think I’ve ever had to. The
default string that xargs will look to replace when using the -i flag
is the curly braces. As your command lines get a little more complex,
or you start using xargs in scripts, there are a couple of useful
troubleshooting flags you may find helpful if you run into issues. One,
the -p flag, will prompt you for a yes or no before executing a command
on anything. The other, which is a real life saver, is “-t,” and it
does NOT prompt you for a yes or no (unless you use it with -p), but it
will output the command it’s trying to execute, so if something isn’t
quite right, you’ll be able to spot it right away. Comments:

$ > rpm -qa | grep mozilla | xargs -n1 -p rpm -e –nodeps

How about:

rpm -e –nodeps `rpm -qa|grep mozilla`

or if you want a prompt:

for pkg in `rpm -qa`
do
echo “Remove package $pkg? (y/n)”
read ans
if [ "$ans" == "y" ]; then
rpm -e –nodeps $pkg
fi
done

Far clearer. The use for xargs is cases where you want to use tools
(such as GNU Grep) which have limits on the amount of input they can
take. For example:

grep foo `find / -type f -print`

might be too much for grep to cope with;

find / -type f -print | xargs grep foo
> ls *.mp3 | xargs -n1 -i cp {} backup/.

Search remotely and send the results to local machine

If you want to grep (search) through a log file while you’re ssh’d in
to a server and then get that output to yourself on your workstation. I
usually do “grep … > ~/file.txt” and then scp it over.

But you can also do:

ssh remotehost -l remoteuser “grep regexpr logfile” > localfilename

Or if you are already on the remote but want the file to end up locally:

grep … | ssh localhost cat \>file.txt

~ from the gang at Milwaukee LUG

Mounting your old linux HD to recover files

A quickie lesson in mounting – If I have an old linux drive with stuff
on it that I want to use in another linux system (and I do), these are
the steps that I would follow. I would use the SLAVE, MASTER, CABLE
SELECT jumper on the harddrive to set the set the drive to be a slave,
and boot up my new linux install on another physical drive set to be
master. I closely watch the boot messages, or try going to a place like
/var/log/dmesg to see what hdX (e.g., hda, hdb, hdc) my old drive was
assigned at boot. Partitions on that physical drive get numbers. With
four partitions on my old drive, the root partition on my second drive
was at hdd4. So, I needed to create a mount point to mount my drive to.
For simplicity, I chose to mkdir /mnt/hdd4.

As my old drive was of the ext3 filesystem type, so all mounting
operations need to be told what type (-t for mount) the drive is. To
mount this partitiion, as root I typed mount -t ext3 /dev/hdd4
/mnt/hdd4. I could then cd to /mnt/hdd4 and copy off my files. If I
wanted to reboot with the partition mounted, simply I would just add
something like /dev/hdd4 /mnt/hdd4 etx3 to my /etc/fstab file.

How to set a static IP in Ubuntu from the shell

Edit
<span class="punct">/</span><span class="regex">etc</span><span class="punct">/</span><span class="ident">network</span><span class="punct">/</span><span class="ident">interfaces</span> </pre> and adjust it to your needs (in this example setup I will use the IP address  192.168.0.100):<br /><br /><pre><span class="comment"># This file describes the network interfaces available on your system</span><br /><span class="comment"># and how to activate them. For more information, see interfaces(5).</span><br /><br /><span class="comment"># The loopback network interface</span><br /><span class="ident">auto</span> <span class="ident">lo</span><br /><span class="ident">iface</span> <span class="ident">lo</span> <span class="ident">inet</span> <span class="ident">loopback</span><br /><br /><span class="comment"># This is a list of hotpluggable network interfaces.</span><br /><span class="comment"># They will be activated automatically by the hotplug subsystem.</span><br /><span class="ident">mapping</span> <span class="ident">hotplug</span><br />        <span class="ident">script</span> <span class="ident">grep</span><br />        <span class="ident">map</span> <span class="ident">eth0</span><br /><br /><span class="comment"># The primary network interface</span><br /><span class="ident">auto</span> <span class="ident">eth0</span><br /><span class="ident">iface</span> <span class="ident">eth0</span> <span class="ident">inet</span> <span class="ident">static</span><br />        <span class="ident">address</span> <span class="number">192.168</span><span class="punct">.</span><span class="number">0.100</span><br />        <span class="ident">netmask</span> <span class="number">255.255</span><span class="punct">.</span><span class="number">255.0</span><br />        <span class="ident">network</span> <span class="number">192.168</span><span class="punct">.</span><span class="number">0.0</span><br />        <span class="ident">broadcast</span> <span class="number">192.168</span><span class="punct">.</span><span class="number">0.255</span><br />        <span class="ident">gateway</span> <span class="number">192.168</span><span class="punct">.</span><span class="number">0.1</span><br /><br />Then do<br /><br /><span class="ident">sudo</span> <span class="punct">/</span><span class="ident">etc</span><span class="punct">/</span><span class="ident">init</span><span class="punct">.</span><span class="ident">d</span><span class="punct">/</span><span class="ident">networking</span> <span class="ident">restart</span><br /><br />to restart the network.<br /><br />