How to setup Cron Jobs

Cron Jobs

What are Cron Jobs? A cron job is a command or a series of commands set to run at specific times on your Linux or Unix based system. Cron jobs are very flexible when used in conjuction with programs and shell scripts and can automate various parts of system administration (.such as logfile rotation, program execution, gather system information and sending emails etc.)

Summary of Commands

crontab filename - Install filename as your crontab file.

crontab -e - Edit your crontab file.

crontab -l - Show your crontab file.

crontab -r - Remove your crontab file.

Creating a Cron job

To Create a cron job, run the command

 crontab -e

This will open the crontab for the user and allow you to create a cron job following the syntax below:

Syntax
Your cron job looks like as follows:
1 2 3 4 5 /path/to/command arg1 arg2

Where,

1: Minute (0-59)
2: Hours (0-23)
3: Day (0-31)
4: Month (0-12 [12 == December])
5: Day of the week(0-7 [7 or 0 == sunday])
/path/to/command - Script or command name to schedule
Same above five fields structure can be easily remembered with following diagram: 

* * * * * command to be executed
- - - - -
| | | | |
| | | | —– Day of week (0 - 7) (Sunday=0 or 7)
| | | ——- Month (1 - 12)
| | ——— Day of month (1 - 31)
| ———– Hour (0 - 23)
————- Minute (0 - 59)Insert non-formatted text here

This Example cronjob is set to run every minute and execute the omgiambreakingtheserver.sh script. Note the actual example in relation to the syntax above.

 [root@training-linux yum]# crontab -l
 */1 * * * * /script/omgiambreakingtheserver.sh

Another set of examples as shown in the man page:

 EXAMPLE CRON FILE
      # use /bin/sh to run commands, no matter what /etc/passwd says
      SHELL=/bin/sh
      # mail any output to âpaulâ, no matter whose crontab this is
      MAILTO=paul
      #
      # run five minutes after midnight, every day
      5 0 * * *       $HOME/bin/daily.job >> $HOME/tmp/out 2>&1
      # run at 2:15pm on the first of every month -- output mailed to paul
      15 14 1 * *     $HOME/bin/monthly
      # run at 10 pm on weekdays, annoy Joe
      0 22 * * 1-5   mail -s "Itâs 10pm" joe%Joe,%%Where are your kids?%
      23 0-23/2 * * * echo "run 23 minutes after midn, 2am, 4am ..., everyday"
      5 4 * * sun     echo "run at 5 after 4 every sunday"

External References

Cron man Page

Cron on wikipedia

N00b intro to cron

How to use the dpkg package managment tool

dpkg package managment tool What is dpkg?

dpkg is the software at the base of the Debian package management system. dpkg is used to install, remove, and provide information about .deb packages. dpkg itself is a low level tool; higher level tools, such as [APT], are used to fetch packages from remote locations or deal with complex package relations. End users are encouraged to use tools like aptitude or synaptic, having a more sophisticated way of dealing with package relationships and a friendlier interface. The Debian package “dpkg” provides the dpkg program, as well as several others necessary for run-time functioning of the packaging system, including dpkg-statoverride, dpkg-divert and update-alternatives. You will encounter this tool on Debian and Ubuntu.

Basic Use

dkpg -i Installs a .deb package

dpkg -l Lists installed packages

dpkg -r Removes the package

External References

dpkg man page

Wikipedia dpkg article

How to use the RPM Command

The RPM Command

What is RPM? RPM is the RPM Package Manager. It is an open packaging system available for anyone to use. It allows users to take source code for new software and package it into source and binary form such that binaries can be easily installed and tracked and source can be rebuilt easily. It also maintains a database of all packages and their files that can be used for verifying packages and querying for information about files and/or packages. Permission is granted to use and distribute RPM royalty free under the GPL. You will find RPM being used on all RPM based distrobutions including CentOS, RedHat, Fedora, OpenSUSE, and VMWare ESX.

General Use

rpm -v the v switch stands for verbose and provides more detailed info to the installation of packages

rpm -i the i switch is for installation of a package

rpm -h the h switch that enables hash marks that show the progress of a package installation

rpm -U the U switch is for upgrade and will install a package and upgrade any previous versions of the package

rpm -e the e switch is for removal of packages

rpm -q the q switch is for querying the install rpm packages

Examples of use

installing packages

 [root@training-linux ~]# rpm -ivh webmin-1.420-1.noarch.rpm
 warning: webmin-1.420-1.noarch.rpm: V3 DSA signature: NOKEY, key ID 11f63c51
 Preparing…                ########################################### [100%]
 Operating system is CentOS Linux
 1:webmin                 ########################################### [100%]
 Webmin install complete. You can now login to https://training-linux:10000/

installing packages from the internet

 [root@training-linux ~]# rpm -ivh http://internap.dl.sourceforge.net/sourceforge/webadmin/webmin-1.420-1.noarch.rpm
 Retrieving http://internap.dl.sourceforge.net/sourceforge/webadmin/webmin-1.420-1.noarch.rpm
 warning: /var/tmp/rpm-xfer.nMhGFY: V3 DSA signature: NOKEY, key ID 11f63c51
 Preparing…                ########################################### [100%]
 Operating system is CentOS Linux
 1:webmin                 ########################################### [100%]
 Webmin install complete. You can now login to https://training-linux:10000/

removing packages

 [root@training-linux ~]# rpm -e webmin-1.420-1.noarch
 Running uninstall scripts ..
 Deleted init script /etc/rc.d/init.d/webmin

searching for a package

 [root@training-linux ~]# rpm -q webmin
 webmin-1.420-1

finding out what package a file belongs to

 [root@training-linux ~]# rpm -qf /usr/sbin/ipmievd
 OpenIPMI-tools-2.0.11-3.fc8

External Resources

RPM man page

Redhat Article on RPM

RPM main project page

RPM HowTo

How to use the APT Package management tool

APT Package management tool

What is APT?

The Advanced packaging tool, or APT, is a user interface that works with core libraries to handle the sometimes difficult process of installing software on Linux. APT simplifies the process of managing software on Unix-like computer systems by automating the retrieval, configuration and installation of software packages, either from binary files or by compiling source code. You will most commonly see APT on Debian and Ubuntu servers here at SoftLayer, though some people do use it for RedHat & CentOS package management.

General Use

Using APT is very easy however does have a bit of a learning curve if you are used to YUM or PORTS. Below I will go over some of the basic usage functions and what they do.

apt-get update Pulls information from the apt respositories as listed in the configuration to see what packages are available on the package mirrors. I will go over the configuration later.

apt-get upgrade Updates all packages installed the in the system.

apt-get install Installs packages that are requested

apt-cache search Searches for specified packages

apt-get remove Uninstalls packages and dependencies

Configuration

/etc/apt/sources.list - Mirrors that the packages are fetched from

/etc/apt/apt.conf - The Main APT configuration file

/etc/apt/apt.conf.d/ - The APT configuration file fragments .

/etc/apt/preferences - version preferences file. This is where you would specify “pinning”, i.e. a preference to get certain packages from a separate source or from a different version of a distribution.

/var/cache/apt/archives/ - storage area for retrieved package files.

/var/cache/apt/archives/partial/ - storage area for package files in transit.

/var/lib/apt/lists/ - storage area for state information for each package resource specified in sources.list

/var/lib/apt/lists/partial/ - storage area for state information in transit.

External Resources on APT

APT man pages

RPMForge Apt Repositories for CentOS,RedHat & Fedora

Debian APT howto

Using the YUM Package managment tool

YUM Package managment tool

What is YUM?

The Yellow dog Updater, Modified (YUM) is a free software/open source command line package management utility for RPM-compatible Linux operating systems. It was developed by Seth Vidal and a group of volunteer programmers, and is currently maintained as part of Duke University’s Linux@DUKE project. Though yum is a command line utility, several other tools provide graphical user interfaces to yum, among them pup, pirut, and yumex. Seth Vidal now works for Red Hat and a number of other Red Hat programmers are involved in the development of yum. You will encounter this package management tool at SoftLayer on CentOS, and RedHat Enterprise 5+ and Fedora.

General Use

yum install Command used for installing packages - Example: yum install httpd

yum update Command used for updating packages - Example: yum update httpd

yum upgrade Command used for upgrading packages - Example: yum upgrade httpd

yum check-update Command used for checking available updates - Example: yum check-update httpd

yum search Command used to search for packages - Example: yum search httpd

yum remove Command used to remove packages - Example: yum remove httpd

yum clean Command used to clear yum caches and other functions - Example: yum clean packages

yum groupinstall Command used to install a group of packages - Example: yum groupinstall X Window System

Configuration

/etc/yum.conf The master configuration file for yum, you can add exclusions to packages to be updated and other options.

/etc/yum.repos.d/ A directory for storing yum repositories

/etc/yum/ main yum directory - contains further configurations

/etc/yum/yum-updatesd.conf Configuration file for the yumupdatesd Daemon

/etc/yum/pluginconf.d/ Configuration directory for yum plugins

Managing Yum Repositories

Yum repos are by default located in /etc/yum.repos.d/ . The repositories can be modified to suit your needs at any time. Note that in the in the repo configs, you will find various entries for the various types of repositories with comments on what they are. To enable a repo from the configuration, just uncomment out the base url and this will allow it to be added to your yum package sources. There is also 3rd party repositories that customers may be using (such as rpmforge

3rd Party Yum Repositories list

RPM Forge

DAG Repos

ATrpms

KBS Extras

External resources on yum

CentOS.org - Managing Software with YUM

Official YUM project page

YUM Mailing List

Yum Wiki

APF (Advanced Policy Firewall)

APF (Advanced Policy Firewall)

What is APF? Advanced Policy Firewall (APF) is an iptables(netfilter) based firewall system designed around the essential needs of today’s Internet deployed servers and the unique needs of custom deployed Linux installations. The configuration of APF is designed to be very informative and present the user with an easy to follow process, from top to bottom of the configuration file. The management of APF on a day-to-day basis is conducted from the command line with the ‘apf’ command, which includes detailed usage information and all the features one would expect from a current and forward thinking firewall solution.

APF Commands

apf -f Stops the firewall and all scripts that restart it. Also flushes the rules from iptables.

apf -s Starts the firewall and loads all rulesets

apf –status checks the status of APF

apf -l lists the running rules of APF

apf -r restarts the APF firewall

apf -d blocks a specific ip address

apf -e refreshes & resolves dns names in trusted rules

Installing APF

Download the software

 wget http://www.rfxnetworks.com/downloads/apf-current.tar.gz

Extract the contents

 tar xvzf apf-current.tar.gz

Enter the Directory

 cd apf-0.9.6-3/

Execute the install file

 sh install.sh

Configuration of APF

APF has multiple configuration files - Please see the official APF site for more advanced descriptions

/etc/apf/ - Main configuration directory

/etc/apf/conf.apf - Main apf firewall configuration file.

List of configuration files in /etc/apf:

 [root@somelinuxbox apf]# ls
 allow_hosts.rules  ds_hosts.rules        internals          sysctl.rules
 apf                ecnshame_hosts.rules  log.rules          VERSION
 bt.rules           extras                main.rules         vnet
 conf.apf           firewall              postroute.rules
 deny_hosts.rules   glob_allow.rules      preroute.rules
 doc                glob_deny.rules       sdrop_hosts.rules

Below is the example of a typical configuration of rules in /etc/apf/conf.apf - no control panel.

 # Common ingress (inbound) TCP ports
 IG_TCP_CPORTS=" 21,22,25,53,80,110,443,3306"

 # Common ingress (inbound) UDP ports
 IG_UDP_CPORTS="21,53"

 # Common ICMP (inbound) types
 # 'internals/icmp.types' for type definition; 'all' is wildcard for any
 IG_ICMP_TYPES="3,5,11,0,30,8"

 # Common egress (outbound) TCP ports
 EG_TCP_CPORTS="21,22,25,53,80,110,443,3306"

 # Common egress (outbound) UDP ports
 EG_UDP_CPORTS="20,21,53"

 # Common ICMP (outbound) types
 # 'internals/icmp.types' for type definition; 'all' is wildcard for any
 EG_ICMP_TYPES="all"

[Tips and Tricks

General

Remember, APF is nothing but a wrapper for iptables - So many of the same things apply.

Troubleshooting APF

DEVEL_MODE= is a useful configuration line in /etc/apf/conf.apf - if enabled to "1" it will start a cron job that every 5 minutes will stop the firewall. Useful when diagnosing issues with configuration.

 # [Main]
 ##
 # !!! Do not leave set to (1) !!!
 # When set to enabled; 5 minute cronjob is set to stop the firewall. Set
 # this off (0) when firewall is determined to be operating as desired.
 DEVEL_MODE="0"

This line should be set for our customers as there is no need to run apf on the private network and this will completely bypass APF on the listed interfaces after a firewall restart

 # Trusted Network interface(s); all traffic on defined interface(s) will by-pass
 # ALL firewall rules, format is white space or comma separated list.
 IFACE_TRUSTED="eth0"

Basic Differences between lighttpd and apache web servers.

Below is some of the basic differences between lighttpd and apache.

Basic configuration

Apache:

DocumentRoot /var/www/html
CustomLog /var/www/logs/access
ErrorLog /var/www/logs/error
User www
Group www

lighttpd:

server.document-root = “/var/www/html”
accesslog.filename = “/var/www/logs/access”
server.errorlog = “/var/www/logs/error”
server.username = “www”
server.groupname = “www”
server.modules = ( “mod_accesslog” )

Authentication and Authorization .htaccess

Apache:
AuthName “My Special Directory”
AuthType Basic
AuthUserFile /var/www/passwords/users
Order deny,allow
require valid-user

lighttpd:

auth.backend = “htpasswd”
auth.backend.htpasswd.userfile = “/var/www/passwords/users”
auth.require = ( “/special/” =>
(
“method” => “basic”,
“realm” => “My Special Directory”,
“require” => “valid-user”
)
)

Virtual Host configuration

Apache:

NameVirtualHost *

ServerName “scratch.example.com”
DocumentRoot “/var/www/hosts/scratch/docs”

ServerName “sniff.example.com”
DocumentRoot “/var/www/hosts/sniff/docs”

lighttpd:

$HTTP["host"] == “scratch.example.com” {
server.document-root = “/var/www/hosts/scratch/docs/” }
$HTTP["host"] == “sniff.example.com” {
server.document-root = “/var/www/hosts/sniff/docs/” }

Virtual Host Modules:

Apache:

LoadModule vhost_alias_module modules/mod_vhost_alias.so
VirtualDocumentRoot /var/www/hosts/%1/docs

lighttpd:

server.modules = ( …, “mod_evhost”, … )
evhost.path-pattern = “/var/www/hosts/%3/docs”

PHP

Apache:

LoadModule php5_module modules/libphp5.so
AddType application/x-httpd-php .php

lighttpd:

server.modules = ( …, “mod_fastcgi”, … )
fastcgi.server =
( “.php” =>
( “localhost” =>
(
“socket” => “/tmp/php-fastcgi.socket”,
“bin-path” => “/usr/local/bin/php”
)
)
)

Installing lighttpd, php5 fast cgi in Debian etch 4.0

Installing lighttpd in Debian is easy. All you have to do is perform the following steps:

Step1. Execute the following command from shell apt-get install lighttpd php5-cgi

Step2. vi /etc/php5/cgi/php.ini and add the following entry to the end of the file

cgi.fix_pathinfo = 1

Step3. vi /etc/lighttpd/lighttpd.conf and add the following mod to the mod listing:

“mod_fastcgi”,

Step4. add the following entry at the end of the php.ini file:

fastcgi.server = ( “.php” => ((
“bin-path” => “/usr/bin/php5-cgi”,
“socket” => “/tmp/php.socket”
)))

Step5. Save the file and execute /etc/init.d/lighttpd restart and you should be good to go!

How to test your Linux web server to see what it can do - Benchmarking

Below is a list of utilities that can help you test and identify potential bottlenecks on your webserver.
Database Performance testing

DOTS = Open Source Database Performance test
http://sourceforge.net/projects/ltp/

Dbgrinder
= Perlscript to stress MySQL http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ltp/utils/database/dbgrinder

OSDL Database Testsuite - Database testing
http://www.osdl.org/lab_activities/kernel_testing/osdl_database_test_suite/

Filesystem Performance testing

Bonnie++ = Performs hard drive and filesystem tests
http://www.coker.com.au/bonnie++/

Postmark = Simulates enterprise load on a filesystem
http://www.netapp.com/tech_library/postmark.html

Stress = You can specify load to place on the system
http://weather.ou.edu/%7Eapw/projects/stress/

Tiobench = Tests I/O on the system
http://sourceforge.net/projects/tiobench/

Network Performance testing

Connectathon = NFS connection testing
http://www.connectathon.org/nfstests.html

Netperf = Can test various protocals
http://www.netperf.org/

Iperf = Tests UDP and TCP
http://dast.nlanr.net/Projects/Iperf/

Http Performance testing

httperf
= tool for measuring web server performance
http://www.hpl.hp.com/research/linux/httperf/

Seige = a http regression testing and benchmarking utility
http://www.joedog.org/JoeDog/Siege

ApacheBench = a command line computer program for measuring the performance of HTTP web servers - This is included with the apache web server.
http://www.apache.org/

Basic iptables firewall script with ranges

A basic iptables rule script with ranges.

iptables -A INPUT -p icmp -i eth0 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 20:22 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 25 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 53 -j ACCEPT
iptables -A INPUT -p udp -m udp -i eth0 –dport 53 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 110 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 143 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp -i eth0 –dport 9999 -j ACCEPT
iptables -A INPUT -m state –state RELATED,ESTABLISHED -i eth0 -j ACCEPT
iptables -A INPUT -s 10.0.0.0/24 -i eth0 -j ACCEPT
iptables -A INPUT -s 127.0.0.1 -i eth0 -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables-save > /etc/sysconfig/iptables