legal contact
 

Installation

Build a server to host the service...

As I'am a friend of Ubuntu, I've choosen the Ubuntu 12 tree as it's marked to have a stable install source for the CIF.

So check: http://cdimage.ubuntu.com/netboot/12.04/ and install the system. - I've used (to try first) an VirtualImage VM with 3GB, but 8GB are recomended.

Once the Ubuntu is up and running, check the following to be done and finished correctly:

Bootstrap the CPAN:

$ sudo su - root
$ mkdir -p /root/.cpan/CPAN
$ vi /root/.cpan/CPAN/MyConfig.pm

And copy below into MyConfig.pm

$CPAN::Config = {
'applypatch' => q[],
'auto_commit' => q[0],
'build_cache' => q[100],
'build_dir' => q[/root/.cpan/build],
'build_dir_reuse' => q[0],
'build_requires_install_policy' => q[yes],
'bzip2' => q[/bin/bzip2],
'cache_metadata' => q[1],
'check_sigs' => q[0],
'colorize_output' => q[0],
'commandnumber_in_prompt' => q[1],
'connect_to_internet_ok' => q[1],
'cpan_home' => q[/root/.cpan],
'ftp_passive' => q[1],
'ftp_proxy' => q[],
'getcwd' => q[cwd],
'gpg' => q[/usr/bin/gpg],
'gzip' => q[/bin/gzip],
'halt_on_failure' => q[0],
'histfile' => q[/root/.cpan/histfile],
'histsize' => q[100],
'http_proxy' => q[],
'inactivity_timeout' => q[0],
'index_expire' => q[1],
'inhibit_startup_message' => q[0],
'keep_source_where' => q[/root/.cpan/sources],
'load_module_verbosity' => q[none],
'make' => q[/usr/bin/make],
'make_arg' => q[],
'make_install_arg' => q[],
'make_install_make_command' => q[/usr/bin/make],
'makepl_arg' => q[INSTALLDIRS=site],
'mbuild_arg' => q[],
'mbuild_install_arg' => q[],
'mbuild_install_build_command' => q[sudo ./Build],
'mbuildpl_arg' => q[--installdirs site],
'no_proxy' => q[],
'pager' => q[/usr/bin/less],
'patch' => q[/usr/bin/patch],
'perl5lib_verbosity' => q[none],
'prefer_external_tar' => q[1],
'prefer_installer' => q[MB],
'prefs_dir' => q[/root/.cpan/prefs],
'prerequisites_policy' => q[follow],
'scan_cache' => q[atstart],
'shell' => q[/bin/bash],
'show_unparsable_versions' => q[0],
'show_upload_date' => q[0],
'show_zero_versions' => q[0],
'tar' => q[/bin/tar],
'tar_verbosity' => q[none],
'term_is_latin' => q[1],
'term_ornaments' => q[1],
'test_report' => q[0],
'trust_test_report_history' => q[0],
'unzip' => q[],
'use_sqlite' => q[0],
'version_timeout' => q[15],
'wget' => q[/usr/bin/wget],
'yaml_load_code' => q[0],
'yaml_module' => q[YAML],
};
1;
__END__

Run "perl -MCPAN install ..." to auto-configure a list of local CPAN mirrors

 
  1. su root
  2. $ sudo su - root
    
  3. Install the base dependencies from the Ubuntu repositories (as root), you may need to do a "aptitude update && aptitude safe-upgrade" if you run into a lot of conflicts first
  4. $ aptitude -y install perl rng-tools build-essential postgresql apache2 apache2-threaded-dev gcc g++ make libexpat1-dev libapache2-mod-perl2 libclass-dbi-perl libnet-cidr-perl libossp-uuid-perl libxml-libxml-perl libxml2-dev libmodule-install-perl libapache2-request-perl libdbd-pg-perl bind9 libregexp-common-perl libxml-rss-perl libapache2-mod-gnutls libapreq2-dev rsync libunicode-string-perl libconfig-simple-perl libmodule-pluggable-perl libmime-lite-perl libfile-type-perl libtext-csv-perl libio-socket-inet6-perl libapr1-dbg libdate-manip-perl libtry-tiny-perl libclass-accessor-perl pkg-config vim libjson-xs-perl perl-modules libdigest-sha-perl libsnappy-dev libdatetime-format-dateparse-perl liblwp-protocol-https-perl libtime-hires-perl libnet-patricia-perl libnet-ssleay-perl liblog-dispatch-perl libregexp-common-net-cidr-perl libtext-table-perl libdatetime-perl libencode-perl libmime-base64-perl libhtml-table-perl libzmq-dev libzmq1 libzeromq-perl libssl-dev cpanminus
    
  5. upgrade cpanm to work with github
  6. $ sudo cpanm --self-upgrade
    
  7. work-around for the Linux::Cpuinfo dep
  8. $ sudo cpanm git://github.com/gitpan/Linux-Cpuinfo.git@1.7 --force
    
  9. Which I had problem with. But what worked though was:

  10. download the zip file directly from GitHub, unzip and:

  11. perl Makefile.PL
    
  12. make
    
  13. make install
    
  14. Install the remaining perl dependencies from CPAN (as root)
  15. $ PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Test::SharedFork,Test::TCP,Net::Abuse::Utils,Google::ProtocolBuffers,Iodef::Pb::Simple,Compress::Snappy,Net::Abuse::Utils::Spamhaus,Net::DNS,Net::DNS::Match,Snort::Rule,Parse::Range,Log::Dispatch,Net::SSLeay,ZeroMQ,Sys::MemInfo,LWP::Protocol::https,LWPx::ParanoidAgent'
    
  16. Resolver Config

    Configure the static interface to use 127.0.0.1 as the nameserver. Bind will be configured next.

     
    1. edit /etc/network/interfaces
    $ sudo vi /etc/network/interfaces
    
    1. replace (or add) dns-nameservers with 127.0.0.1

    # The primary network interface
    iface eth0 inet
    dns-nameservers 127.0.0.1
    1. Restart networking
    $ sudo ifdown eth0 && sudo ifup eth0
    
    1. Verify resolveconf
    $ cat /etc/resolv.conf
    

    Should look similar to:

    # Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
    # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
    nameserver 127.0.0.1

    Default CIF user

    Create your "cif" user/group (the configure script will default to this user "cif")

    $ sudo adduser --disabled-password --gecos '' cif
    

    CIF Router Configuration (Apache)

    Some of the "CIF" values will be created later in the doc, for now just follow the config as is, don't worry about creating things like "/home/cif" etc.

    1. enable the default-ssl site (debian):
    $ sudo a2ensite default-ssl
    $ sudo a2enmod apreq
    $ sudo a2enmod ssl
    1. unless you know what you're doing with virtual hosts, comment out the port-80 stuff in /etc/apache2/ports.conf

    $ sudo vi /etc/apache2/ports.conf

    # If you just change the port or add more ports here, you will likely also
    # have to change the VirtualHost statement in
    # /etc/apache2/sites-enabled/000-default
    # This is also true if you have upgraded from before 2.2.9-3 (i.e. from
    # Debian etch). See /usr/share/doc/apache2.2-common/NEWS.Debian.gz and
    # README.Debian.gz

    + #NameVirtualHost *:80
    + #Listen 80

    <IfModule mod_ssl.c>
    # If you add NameVirtualHost *:443 here, you will also have to change
    # the VirtualHost statement in /etc/apache2/sites-available/default-ssl
    ...
    1. configure apache2, add this line to your default-ssl site (or default if you're not using TLS)
    $ sudo vi /etc/apache2/sites-available/default-ssl
    
    <IfModule mod_ssl.c>
    <VirtualHost _default_:443>
    + PerlRequire /opt/cif/bin/http_api.pl
    + Include /etc/apache2/cif.conf
    ....
    1. create your config at /etc/apache2/cif.conf, which should look like:
    $ sudo vi /etc/apache2/cif.conf
    
    <Location /api>
    SetHandler perl-script
    PerlResponseHandler CIF::Router::HTTP
    PerlSetVar CIFRouterConfig "/home/cif/.cif"
    </Location>
    1. add your "www-data" user (whoever apache is set to run under) to the group "cif" (/etc/group):
    $ sudo adduser www-data cif
    

    Random Number Generator

    The "rng-tools' service helps with random number generation (mainly used for generating security certificates in bind and apache, speeds up the entropy process).

    1. modify /etc/default/rng-tools to use /dev/urandom as the seed
    $ echo 'HRNGDEVICE=/dev/urandom' | sudo tee -a /etc/default/rng-tools
    
    1. restart rng-tools
    $ sudo service rng-tools restart
    

Bind

Configure Bind

  1. modify the config file
    • Debian Based (Debian, Ubuntu, etc)
    • $ sudo vi /etc/bind/named.conf.options
      
    • RHEL Based (RHEL, CentOS, etc)
    • $ sudo vi /etc/named.conf
      
  2. Configure the file to look something like this:
  3. options {
    // If there is a firewall between you and nameservers you want
    // to talk to, you may need to fix the firewall to allow multiple
    // ports to talk. See http://www.kb.cert.org/vuls/id/800113

    // If your ISP provided one or more IP addresses for stable
    // nameservers, you probably want to use them as forwarders.
    // Uncomment the following block, and insert the addresses replacing
    // the all-0's placeholder.
    forward only;
    forwarders {
    8.8.8.8;
    8.8.4.4;
    };

    auth-nxdomain no; # conform to RFC1035
    // listen-on-v6 { any; };
    listen-on { 127.0.0.1; };
    };

    // bypass the Google public servers
    zone "cymru.com" {
    forward only;
    type forward;
    forwarders { };
    };

    zone "zen.spamhaus.org" {
    forward only;
    type forward;
    forwarders { };
    };

    zone "dbl.spamhaus.org" {
    forward only;
    type forward;
    forwarders { };
    };
  4. reload bind
    • Debian 6.0.x
    • $ sudo /etc/init.d/bind9 restart
      
    • Ubuntu 12.04
    • $ sudo service bind9 restart
      
    • RHEL
    • $ sudo /etc/init.d/named restart
      
  5. verify bind is working
  6. $ dig ns1.google.com
    
    ...
    ns1.google.com. 21588 IN A 216.239.32.10
    ...
    ;; SERVER: 127.0.0.1#53(127.0.0.1)

Postgresql

Postgres Authentication Configuration
  1. Modify your postgres config accordingly (note the 'trust' setting, make sure your iptables are up to date!):
  2. $ sudo vi /etc/postgresql/X.X/main/pg_hba.conf
    
     # (autovacuum, daily cronjob, replication, and similar tasks).
    #
    # Database administrative login by UNIX sockets
    -local all postgres ident sameuser
    +local all postgres trust

    # TYPE DATABASE USER CIDR-ADDRESS METHOD

    # "local" is for Unix domain socket connections only
    -local all all ident sameuser
    +local all all trust
    # IPv4 local connections:
    -host all all 127.0.0.1/32 md5
    +host all all 127.0.0.1/32 trust
    # IPv6 local connections:
    -host all all ::1/128 md5
    +host all all ::1/128 trust

Performance Configuration

NOTE: These recommend numbers have been tested on a machine with 4 cores and 8 GB of ram. During testing we found that these values may be too high for a machine with 4 GB of ram. If you are testing this on a machine with less than 8 GB of ram, you may want to skip this section all together or reduce the numbers these shell script spit out.

  1. Create backups of system files:
  2. sudo cp /etc/sysctl.conf /etc/sysctl.conf.orig
    sudo cp /etc/postgresql/X.X/main/postgresql.conf /etc/postgresql/X.X/main/postgresql.conf.orig
  3. create shmsetup.sh to configure:
    • shared memory (to about 1/2 - 2/3 the amount of system ram)
    • control virtual memory overcommit and swappiness
    • $ vi shmsetup.sh
      
      #!/bin/bash
      page_size=`getconf PAGE_SIZE`
      phys_pages=`getconf _PHYS_PAGES`
      shmall=`expr $phys_pages / 2`
      shmmax=`expr $shmall \* $page_size`
      echo kernel.shmmax = $shmmax
      echo kernel.shmall = $shmall
      echo vm.overcommit_memory = 2
      echo vm.swappiness = 0
      # If you install CIF on a machine with limited ram and
      # have out of memory issues, uncomment the next line
      #echo vm.overcommit_ratio = 100
  4. run the script
  5. $ /bin/bash shmsetup.sh | sudo tee -a /etc/sysctl.conf
    
  6. reload the kernel settings
    • Debian / Ubuntu / RHEL 6.x
    • $ sudo sysctl -p
      
    • RHEL 5.x
    • $ sudo /sbin/sysctl -p
      
  7. Comment out existing shared_buffers and max_connections settings so it can be set below
  8. sudo sed -i 's/shared_buffers/#shared_buffers/' /etc/postgresql/X.X/main/postgresql.conf
    sudo sed -i 's/max_connections/#max_connections/' /etc/postgresql/X.X/main/postgresql.conf
  9. create postgressetup.sh to configure better defaults for your CIF installation
  10. $ vi postgressetup.sh
    
    #!/bin/bash
    page_size=`getconf PAGE_SIZE`
    phys_pages=`getconf _PHYS_PAGES`
    total_ram_b=`expr $page_size \* $phys_pages`
    total_ram_kb=`expr $total_ram_b / 1024`
    total_ram_mb=`expr $total_ram_kb / 1024`
    ten_percent_total_ram=`expr $total_ram_mb / 10`

    work_mem=`expr $total_ram_mb / 8`
    shared_buffers=$ten_percent_total_ram
    effective_cache_size=`expr $ten_percent_total_ram \* 6`

    echo ""
    echo ""
    echo "#------------------------------------------------------------------------------"
    echo "# CIF Setup "
    echo "#------------------------------------------------------------------------------"
    echo "# Rough estimates on how to configured postgres to work with large data sets"
    echo "# See the following URL for proper postgres performance tuning"
    echo "# http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server"
    echo "wal_buffers = 12MB" " # recommended range for this value is between 2-16MB"
    echo "work_mem = $work_mem""MB" " # minimum 512MB needed for cif_feed"
    echo "shared_buffers = $shared_buffers""MB" "# recommended range for this value is 10% on shared db server"
    echo "checkpoint_segments = 10" " # at least 10, 32 is a more common value on dedicated server class hardware"
    echo "effective_cache_size = $effective_cache_size""MB" " # recommended range for this value is between 60%-80% of your total available RAM"
    echo "max_connections = 8" " # limiting to 8 due to high work_mem value"
  11. run the script
  12. $ /bin/bash postgressetup.sh | sudo tee -a /etc/postgresql/X.X/main/postgresql.conf
    

Testing

  1. restart postgres
    • Debian
    • sudo /etc/init.d/postgresql restart
      
    • Ubuntu/RHEL
    • $ sudo service postgresql restart
      
  2. make sure your user can log in:
  3. $> psql -U postgres
    
    postgres=#
    postgres=#\l
    List of databases
    Name | Owner | Encoding | Collation | Ctype | Access privileges
    -----------+----------+----------+------------+------------+-----------------------
    postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
    template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres
    : postgres=CTc/postgres
    template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres
    : postgres=CTc/postgres
    (3 rows)
    postgres=#\q