-
Notifications
You must be signed in to change notification settings - Fork 94
Setting up GeoLite Data
The GeoLite databases are free IP geolocation databases. They are updated on the first Tuesday of each month. These databases are offered in the binary and csv formats. We will be using csv for this setup. Download the files from the following link: http://dev.maxmind.com/geoip/legacy/geolite/
Unzip the zip file and you will find two csv files: GeoLiteCity-Blocks.csv and GeoLiteCity-Location.csv. Place both files in the same directory.
Now setup your instance of MySQL database if you haven't already done so. Setting up MySQL is beyond the scope of this article. Once MySQL is up and running enter MySQL shell and type the following code:
CREATE DATABASE IF NOT EXISTS GEO;
USE GEO;
DROP TABLE IF EXISTS `blocks`;
CREATE TABLE `blocks` ( `startIPNum` int(10) unsigned NOT NULL,`endIPNum` int(10) unsigned NOT NULL,`locID`
int(10) unsigned NOT NULL, PRIMARY KEY (`startIPNum`,`endIPNum`) )
ENGINE=MyISAM DEFAULT CHARSET=latin1 PACK_KEYS=1 DELAY_KEY_WRITE=1;
DROP TABLE IF EXISTS `location`;
CREATE TABLE `location` (`locID` int(10) unsigned NOT NULL,`country` char(2) default NULL,`region` char(2)
default NULL,`city` varchar(45) default NULL,`postalCode` char(7) default NULL,`latitude` double default
NULL,`longitude` double default NULL,`dmaCode` char(3) default NULL,`areaCode` char(3) default NULL,PRIMARY KEY
(`locID`),KEY `Index_Country` (`country`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1 ROW_FORMAT=FIXED;
This will create the empty tables. Now we need to load the extracted csv files into these tables. To do so type the following command into the MySQL shell:
load data infile 'GeoLiteCity-Blocks.csv' into table `blocks` fields terminated by ',' optionally enclosed by
'"' lines terminated by '\n' ignore 2 lines;
load data infile 'GeoLiteCity-Location.csv' into table `location` fields terminated by ',' optionally enclosed
by '"' lines terminated by '\n' ignore 2 lines;
Finally, with the data loaded we need to define a stored function to make it easier on us to query the database. To do so enter the following commands in the MySQL shell:
DELIMITER $$
DROP FUNCTION IF EXISTS `IPTOLOCID` $$
CREATE FUNCTION `IPTOLOCID`( ip VARCHAR(15)) RETURNS int(10) unsigned
BEGIN
DECLARE ipn INTEGER UNSIGNED;
DECLARE locID_var INTEGER;
IF ip LIKE '192.168.%' OR ip LIKE '10.%' THEN RETURN 0;
END IF;
SET ipn = INET_ATON(ip);
SELECT locID INTO locID_var FROM `blocks` INNER JOIN (SELECT MAX(startIPNum) AS start FROM `blocks` WHERE startIPNum <= ipn) AS s ON (startIPNum = s.start) WHERE endIPNum >= ipn;
RETURN locID_var;
END
$$
DELIMITER ;
CIF Data can be loaded into hbase using the utility class com.opensoc.dataloads.cif.HBaseTableLoad. This class is in the OpenSoc-Dataloads folder.
The class takes a directory name and table name as inputs.
java -cp OpenSOC-Topologies-0.3BETA-SNAPSHOT.jar com.opensoc.dataloads.cif.HBaseTableLoad directoryname hbaseTableName
The hbase configuration is loaded from hbase-site.xml file. There is a hbase-site.xml file within OpenSoc-Dataloads folder. You can override the hbase-site.xml by passing in a different hbase-site.xml file in the classpath. i.e. java -cp /etc/hbase/conf/hbase-site.xml:OpenSOC-Topologies-0.3BETA-SNAPSHOT.jar com.opensoc.dataloads.cif.HBaseTableLoad directoryname hbaseTableName
The class assumes the source files are in gz compressed and data is in json. As of now, domain, email and infrastructure data is being loaded. URL and malware data is not being loaded.
The following datasets are being loaded into the hbase table.
domain_botnet/ domain_fastflux/ domain_malware/ domain_phishing/ domain_spam/ domain_spamvertising/ domain_suspicious/ domain_whitelist/ email_phishing/ email_registrant/ email_spam/ email_spamvertising/ email_suspicious/ email_whitelist/ infrastructure_botnet/ infrastructure_fastflux/ infrastructure_malware/ infrastructure_phishing/ infrastructure_scan/ infrastructure_spam/ infrastructure_spamvertising/ infrastructure_suspicious/ infrastructure_warez/ infrastructure_whitelist/
The first part of the directory is the column family name, the second part is the column qualifier. For e.g. within domain_botnet, domain is the family name and botnet is the qualifier name.
The loader uses the value for the json field "address" as the hbase row key. The value stored is a simple boolean flag "Y".