Log Parser

Question: Log files are used ubiquitously in programming because they give insight on what the program is doing during its execution. Instead of searching through many log files on the filesystem with custom commands, we can load the data into kdb+ and query the logs in a standardized (and faster) manner. Log files can differ in format from program to program, so it is not unusal to write a custom log parser based on the log format. Given a log format of '%(asctime)s %(name)s %(levelname)s:%(message)s', define a function 'lp' that takes in a log file, represented as a file symbol, and returns a table with columns date, time, level, message.

Example

                                
                                $ cat sample.log
2019-03-27 21:16:41,200 INFO:starting function
2019-03-27 21:16:41,200 INFO:this file has 10 rows
2019-03-27 21:16:41,201 DEBUG:function completed for myfile.txt
2019-03-27 21:16:41,201 INFO:starting function
2019-03-27 21:16:41,202 ERROR:[Errno 2] No such file or directory: 'nonexistentfile.txt'
2019-03-27 21:16:41,202 DEBUG:function completed for nonexistentfile.txt

q)lp `:sample.log
date       time         level msg
------------------------------------------------------------------------------------------
2019.03.27 21:16:41.200 INFO  "starting function"
2019.03.27 21:16:41.200 INFO  "this file has 10 rows"
2019.03.27 21:16:41.201 DEBUG "function completed for myfile.txt"
2019.03.27 21:16:41.201 INFO  "starting function"
2019.03.27 21:16:41.202 ERROR "[Errno 2] No such file or directory: 'nonexistentfile.txt'"
2019.03.27 21:16:41.202 DEBUG "function completed for nonexistentfile.txt"
q)meta lp `:sample.log
c    | t f a
-----| -----
date | d
time | t
level| s
msg  | C

Solution

                                
                                ######## solution.q ########

lp:{
    lines:{sl:" " vs x; (@[;1;ssr[;",";"."]] @[;2;{first ":" vs x}]3#sl), enlist " " sv @[;0;{last ":" vs x}] 2_sl} each read0 x;
    flip `date`time`level`msg!"DTS*"$flip lines
 }

Explanation: To read the string contents of a file into a list of strings, use the keyword read0. Next, we extract the date, time, and level of each line. Given this log format, we can extract the date, time, and level by looking at the first three strings. Replace the comma with a period in the time string. Chop off the excess characters in the string that contains the level. We can extract the message by looking at the third (chop off level) string onward. Next join these fields together for each line and flip them to get the values in column wise format. Create a dictionary of column names to values and flip to get the resulting table.

Code:

https://bitbucket.org/alvikabir/dailyq/src/master/Intermediate/log-parser

Tags:

dictionaries iterators strings tables

Searchable Tags

algorithms api architecture asynchronous c csv data structures dictionaries disk feedhandler finance functions ingestion ipc iterators machine learning math multithreading optimizations realtime shared library sql statistics streaming strings tables temporal utility websockets

daily q

Log Parser

Example

Solution

Tags:

Searchable Tags