To forward the "intersting" data only and parse it into my leak dashboard, I've created a monitoring rule within splunk and some of the sourcetypes.
As long as the telegram-cli and the telegram-messages-dump support the JSON format, that was not to complicate to setup.
The import of the downloaded file attachments is done as easy as possible as well. I just take the .txt and .csv files that are provided. The remaining files are checked with an own script for interesting and usable stuff. More on that script to be available soon.
inputs.conf
[monitor:///var/log/TGMON/]
crcSalt = <SOURCE>
disabled = 0
host = telegram-monitor
index = tgmon
sourcetype = tgmon
whitelist = .*\.log$
[monitor:///[NFS Share] /.../]
crcSalt = <SOURCE>
disabled = 1
followTail = 1
host_segment = 8
index = leak
sourcetype = leak:TG_colon
whitelist = .*TG_monitor_.*/document/.*\.csv$|.*TG_monitor_.*/document/.*\.txt$
blacklist = .*\.ini
host =
[monitor:///[NFS Share]/Telegram/telegram-messages-dump/*.en]
disabled = 0
crcSalt =
recursive = false
followTail = 1
host_segment = 8
index = tg-msg-mon
sourcetype = TG_message_mon
[tcp://667]
connection_host = ip
index = telegram
source = telegram
sourcetype = TG_message_mon
//var/log/TGMON/ has the logging of the scripts themself. Helpfull for debugging and monitoring the whole stuff.
[monitor:///[NFS Share] /.../ has all the subdirectorys of the file and message dumps on the NFS share.
[monitor:///[NFS Share]/Telegram/telegram-messages-dump/*.
This is the only the directory of the messages being downloaded and translated. They are stored for historical analysis and if something goes wrong to still be able to retrieve them from my local disk.
[tcp://667]
The live feed of the groups that I've joined is retrieved (in almost realtime), tanslated and send to splunk via net-cat. As there is no historical data available, if that connection breaks, no data at all.
translate and transmit the messages to splunk
The transmission of the translated live-messages received via telegram is just very easy. Although I was able to "see" this simple solution at first.
telegram-cli --json | trans -no-autocorrect -b :en | nc [dest IP] 667
probs.conf
[tgmon]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
pulldown_type = true
disabled = false
BREAK_ONLY_BEFORE_DATE =
MAX_TIMESTAMP_LOOKAHEAD = 10
[TG_message_mon]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = json
SHOULD_LINEMERG = true
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = date
category = Structured
description = A variant of the JSON source type, with support for nonexistent timestamps
disabled = false
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 512
SEDCMD-strip_prefix = s/^[^{]+//g
SHOULD_LINEMERGE = false