Pastebin gatherings

   ->  

Search the Pastebin for interesting stuff by using some Yara rules to map for keywords and expressions. After that, display your getherings on a nice splunk view for making your findings more convenient.

This is still work in progres, but could be done so far is pretty straight forward...

Prerequisits

Get yourself a pro account from pastebin to be able to scrath pastebin. And eventually an account at GitHub to retrieve gists as well.

Find your Pastebin API here:

And whitelist your office IP at Pastbin. -> here

Install PasteHunter (and give respect to TheHermit)

Do a "git clone https://github.com/kevthehermit/PasteHunter" to receive the actual programm doing the work.
Install the missing stuff with "pip3 install -r requirements.txt"

Adjusting the config

at settings.json adjust the appropriate fields to your own data:

{
"inputs": {
"pastebin":{
"enabled": true,
"module": "inputs.pastebin",
"api_scrape": "https://pastebin.com/api_scraping.php",
"api_raw": https://pastebin.com/api_scrape_item.php?i=[your pastbin API key],
"paste_limit": 200,
"store_all": false
},
"dumpz": {
"enabled": false,
"module": "inputs.dumpz",
"api_scrape": "https://dumpz.org/api/recent",
"api_raw": "https://dumpz.org/api/dump",
"paste_limit": 200,
"store_all": false
},
"gists": {
"enabled": true,
"module": "inputs.gists",
"api_token": "[your Git oAuth key]",
"api_limit": 100,
"store_all": false,
"user_blacklist": [],
"file_blacklist": ["grahamcofborg-eval-package-list"]
}
},

Ensure you've enable the JSON output in the config:

"json_output": {
"enabled": true,
"module": "outputs.json_output",
"classname": "JsonOutput",
"output_path": "logs/json/",
"store_raw": true,
"encode_raw": true
},

P.S. As per the latest issue, check the comma at the [general] section is set after the "run_frequency" line.

Every paste gathered is checked against the yara rule logic that is stored in the corresponding Yara directory. - Check it out and write your own rules.

Depending on the installation directory, give it a try

/usr/bin/python3 /opt/pastehunter/pastehunter.py

The output shoud look something like:

INFO:pastehunter.py:Reading Configs
INFO:pastehunter.py:Configure Inputs
INFO:pastehunter.py:Enabled Input: gists
INFO:pastehunter.py:Enabled Input: pastebin
INFO:pastehunter.py:Configure Outputs
INFO:pastehunter.py:Enabled Output: elastic_output
INFO:pastehunter.py:Enabled Output: syslog_output
INFO:pastehunter.py:Enabled Output: csv_output
INFO:pastehunter.py:Enabled Output: json_output
INFO:pastehunter.py:Compile Yara Rules
INFO:pastehunter.py:Enable Blacklist Rules
INFO:pastehunter.py:Populating Queue
INFO:pastehunter.py:Fetching paste list from inputs.gists
INFO:gists.py:Remaining Limit: 4992. Resets at 2018-02-03T11:38:29
INFO:pastehunter.py:Fetching paste list from inputs.pastebin
INFO:pastehunter.py:Sleeping for 300 Seconds

Check if the corresponding json file has some entrys at (depending on your install directory) at: /opt/pastehunter/logs/json/

Configure Splunk

Install a SPlunkForwarder on your machine and get the data into your splunk.

The inputs.conf that works at my end is the below one.

[monitor:///opt/pastehunter/logs/json/*]
disabled = false
renderXml = true
index = main
sourcetype = Pastebin

Create the right sourcetype at you splunk server.

/opt/splunk/etc/system/local/props.conf

[Pastebin]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = true
NO_BINARY_CHECK = true
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true
SHOULD_LINEMERGE = true
LINE_BREAKER=([\r\n]+)\{

Job done ...

Now start everything and watch the sourcetype=Pastebin in Splunk.

I've created me a view with some searches that could act as an excample for you

sourcetype=pastebin | stats distinct_count(raw_paste) as pastes

sourcetype=pastebin "YaraRule{}"=* | rename "YaraRule{}" as Topics | stats count by Topics

sourcetype=pastebin "YaraRule{}"=* raw_paste=* | fillnull value="n.a." | rename "YaraRule{}" AS topic | eval result=raw_paste."--> ".scrape_url." <--"| stats count by topic title type syntax result

Already found an interesting paste of the quite personal data of a leak.
Check out yourself for cool stuff and if you find stuff being written about you, your family and your company. I'm doing this simply by adding a regex part within the search. (| regex _raw="mpauli.de|xxx|xxx|xxx|xxx|xxx" |)

supervisor config

To enable the whole thing within your supervisord, I created below file:

/etc/supervisor/conf.d/pastehunt.conf

[program:pastehunter]
command=/usr/bin/python3 /opt/pastehunter/pastehunter.py
directory=/opt/pastehunter/
process_name = pasthunter%(process_num)d
startsecs = 20
autostart = true
autoretstart = true
user = root
stderr_logfile=/var/log/pastehunter_error.log
stdout_logfile=/var/log/pastehunter.log