Aggregations
來源:200-Areas/210-工程師修煉/ELK/ElasticSearch_Query
Aggregations
在ES的聚合中主要一共分為四大類：
- Bucket Aggregation：分桶類型，一些列滿足特定條件的文檔的集
- Metric Aggregation：指標分析類型，一些數學運算，可以對文檔欄位進行統計分析，比如計算最大值、最小值、平均值等
- Pipeline Aggregation：管道分析類型，對其他
聚合結果進行二次聚合 - Matrix Aggregation：矩陣分析類型，支援對多個字段的操作並提供一個結果矩陣
Bucket Aggregation
根據 Bucket 的分桶策略，常見的 Bucket 聚合分析如下：
- Terms：直接按照
term來分桶，如果是text類型，則按照分詞后的結果分桶 - Range：指定數值的範圍來設定分桶規則
- Date Range：指定日期的範圍來設定分桶規則
- Histogram：直方圖，以固定間隔的策略來分割數據
- Date Histogram：針對日期的直方圖或者柱狀圖，是時序數據分析中常用的聚合分析類型
針對IIS 進行 Port 進行 Group by
POST /ob-iis-obweb01-x-*/_search
{
"size": 0,
"aggs": {
"port": {
"terms": {
"field": "port.keyword",
"size": 3
}
}
}
Result
{
"took" : 652,
"timed_out" : false,
"_shards" : {
"total" : 46,
"successful" : 46,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"port" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "80",
"doc_count" : 35773267
},
{
"key" : "443",
"doc_count" : 941
}
]
}
}
}
change an aggregation's Scope
POST /ob-iis-obweb01-x-*/_search
{
"query": {
"range": {
"log_timestamp": {
"gte": "now-100d/d",
"lte": "now/d"
}
}
},
"size": 0, // return only aggregarion results
"aggs": {
"port": {
"terms": {
"field": "port.keyword",
"size": 3
}
}
}
}
Run multiple aggregations
POST /ob-iis-obweb01-x-*/_search
{
"query": {
"range": {
"log_timestamp": {
"gte": "now-100d/d",
"lte": "now/d"
}
}
},
"size": 0,
"aggs": {
"terms-by-port": {
"terms": {
"field": "port.keyword",
"size": 3
}
},
"terms-by-status": {
"terms": {
"field": "sc_status.keyword",
"size": 3
}
}
}
}
Results
{
"took" : 1680,
"timed_out" : false,
"_shards" : {
"total" : 46,
"successful" : 46,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"terms-by-port" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "80",
"doc_count" : 35773267
},
{
"key" : "443",
"doc_count" : 941
}
]
},
"terms-by-status" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 43762,
"buckets" : [
{
"key" : "200",
"doc_count" : 35537971
},
{
"key" : "304",
"doc_count" : 138464
},
{
"key" : "404",
"doc_count" : 54011
}
]
}
}
}
Metric Aggragation
- 單值分析： avg、max、min、sum、Median(中間值)、Cardinality(like Distinct)
- 多值分析：
- Stats、Extended Stats
- Percentiles、Percentile Ranks
- Top Hits
AVG、Max、Min 範例
POST /ob-iis-obweb01-x-*/_search
{
"query": {
"range": {
"log_timestamp": {
"gte": "now-100d/d",
"lte": "now/d"
}
}
},
"size": 0,
"aggs": {
"terms-by-port": {
"terms": {
"field": "port.keyword",
"size": 3
},
"aggs":{
"avg_byptes":{
"avg":{
"field": "sc_bytes"
}
},
"max_bytes":{
"max":{
"field": "sc_bytes"
}
},
"min_bytes":{
"min":{
"field": "sc_bytes"
}
}
}
}
}
}
Result
{
"took" : 3053,
"timed_out" : false,
"_shards" : {
"total" : 46,
"successful" : 46,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"terms-by-port" : { // 自定義名稱
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "80",
"doc_count" : 35773267,
"avg_byptes" : { // 自定義名稱
"value" : 15963.497536162968
},
"min_bytes" : { // 自定義名稱
"value" : 0.0
},
"max_bytes" : { // 自定義名稱
"value" : 3.127071E7
}
},
{
"key" : "443",
"doc_count" : 941,
"avg_byptes" : {
"value" : 5205.103081827842
},
"min_bytes" : {
"value" : 0.0
},
"max_bytes" : {
"value" : 463862.0
}
}
]
}
}
}
Aggragation Terms 又嵌套 Terms
POST /ob-iis-obweb01-x-*/_search
{
"query": {
"range": {
"log_timestamp": {
"gte": "now-100d/d",
"lte": "now/d"
}
}
},
"size": 0,
"aggs": {
"terms-by-status": {
"terms": {
"field": "sc_status.keyword",
"size": 5
},
"aggs":{
"my_port":{
"terms": {
"field": "port.keyword",
"size": 3
}
}
}
}
}
}
Results
{
// ....... 略 .......
"aggregations" : {
"terms-by-status" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 7589,
"buckets" : [
{
"key" : "200",
"doc_count" : 35537971,
"my_port" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "80",
"doc_count" : 35537066
},
{
"key" : "443",
"doc_count" : 905
}
]
}
},
// ....... 略 .......
]
}
}
}