记录一下工作中遇到的一例kibana无法启动的问题
先说一下我这边的环境
kubernetes: 1.17.9
helm: 2.16.x
efk: 7.8.0 官方oss镜像
安装elasticsearch使用的helm官方模板:
https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
安装kibana使用的deployment yaml文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana
namespace: log-system
labels:
kibana :
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
annotations:
## add-dep-hash-annotations
"e00d2a0ea4791bd589564b4792ad8461355748ed31ede88256abba4255c23e6e" :
labels:
app: kibana
spec:
# affinity for best effort to assign pod across nodes
# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
key: app
operator: In
values:
kibana
topologyKey: "kubernetes.io/hostname"
volumes:
name: kibana-config
secret:
secretName: kibana-config
#
containers:
name: kibana
# port
ports:
containerPort: 5601
name: kibana #关闭健康检查
readinessProbe:
exec:
command:
"echo"
"ok"
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
exec:
command:
"echo"
"ok"
initialDelaySeconds: 15
periodSeconds: 20
"kibana-oss:v7.8.0" :
imagePullPolicy: Always
env:
name: "ELASTICSEARCH_HOSTS"
value: "https://dev-elasticsearch.abc.com:443"
volumeMounts:
mountPath: /usr/share/kibana/config/kibana.yml
name: kibana-config
subPath: kibana.yml
# add-resources
resources:
# default-resources
# overwritten limits
limits:
cpu: 1
memory: 2Gi
# overwritten requests
requests:
cpu: 1
memory: 2Gi
YAML文件是没啥问题
应用kibana后pod不断重启,查看一下pod日志
{"type":"log","@timestamp":"2020-11-21T01:14:05Z","tags":["warning","plugins-discovery"],"pid":7,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}
{"type":"log","@timestamp":"2020-11-21T01:14:06Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,statusPage,share,newsfeed,mapsLegacy,kibanaUtils,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,console,apm_oss,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeMetric,visTypeTable,visTypeMarkdown,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement]"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["fatal","root"],"pid":7,"message":"TypeError: Cannot read property 'properties' of undefined\n at assertIsSupportedIndex (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:315:32)\n at fetchInfo (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:70:10)\n at process._tickCallback (internal/process/next_tick.js:68:7)"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","savedObjectsManagement"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","dashboard"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualize"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","inputControlVis"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMarkdown"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTable"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMetric"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTagcloud"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimelion"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimeseries"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVega"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVislib"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualizations"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","advancedSettings"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","management"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","console"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","home"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","data"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","expressions"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","discover"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","share"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","kibanaLegacy"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetry"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetryCollectionManager"],"pid":7,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Stopping plugin"}
FATAL TypeError: Cannot read property 'properties' of undefined
报错关键信息
FATAL TypeError: Cannot read property 'properties' of undefined
是不是一脸懵,看不出啥问题,就先google了,没任何结果,debug了半天,依然没有任何头绪,第二天一早继续,精神状态贼好
既然网上查不出啥,就慢慢梳理咯,首先构造一个post请求debug一下,看返回什么
curl -X POST "https://dev-elasticsearch.abc.com/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }' |jq .
我这里是返回一个网关的404页面的信息
果然是网关这边的原因,先看下网关的配置,这里只截取了location关键部分
location / {
# root html;
# index 50x.html index.htm;
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Nginx-Proxy true;
proxy_set_header Host $http_host;
proxy_intercept_errors on;
眼尖的可能已经发现了,是的,就是这个参数`proxy_intercept_errors on;`
先看下nginx官网对这个参数的解释: