Breaking News
Loading...

Log analysis using Hive




 Example: Log analysis using Hive

In this example, a log is processed and info like number of TRACE,DEBUG,FATAL messages are retrieved using   HiveQL

Input Data:
2012-02-03 20:26:41 SampleClass3 [TRACE] verbose detail for id 1527353937
2012-02-03 20:26:41 SampleClass2 [TRACE] verbose detail for id 191364434
2012-02-03 20:26:41 SampleClass1 [DEBUG] detail for id 903114158
2012-02-03 20:26:41 SampleClass8 [TRACE] verbose detail for id 1331132178
2012-02-03 20:26:41 SampleClass8 [INFO] everything normal for id 1490351510
2012-02-03 20:32:47 SampleClass8 [TRACE] verbose detail for id 1700820764
2012-02-03 20:32:47 SampleClass2 [DEBUG] detail for id 364472047
2012-02-03 21:05:21 SampleClass6 [FATAL] system problem at id 1620503499

Step 1 : Create table and load data into it


Step2: Query for no of debug,fatal,INFO and Trace messages


Step3: usage of  UNION ALL

- See more at: http://labstrikes.blogspot.in/2012/08/adsense-middle-blog-post.html#sthash.gQgSkqx8.dpuf
 
Toggle Footer