记录一下工作中遇到的一例kibana无法启动的问题
先说一下我这边的环境
kubernetes: 1.17.9helm: 2.16.xefk: 7.8.0 官方oss镜像
安装elasticsearch使用的helm官方模板:
https://github.com/elastic/helm-charts/blob/master/elasticsearch/values.yaml
安装kibana使用的deployment yaml文件
apiVersion: apps/v1kind: Deploymentmetadata:name: kibananamespace: log-systemlabels:: kibanaspec:replicas: 1selector:matchLabels:app: kibanatemplate:metadata:annotations:## add-dep-hash-annotations: "e00d2a0ea4791bd589564b4792ad8461355748ed31ede88256abba4255c23e6e"labels:app: kibanaspec:# affinity for best effort to assign pod across nodes# ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinityaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:weight: 100podAffinityTerm:labelSelector:matchExpressions:key: appoperator: Invalues:kibanatopologyKey: "kubernetes.io/hostname"volumes:name: kibana-configsecret:secretName: kibana-config#containers:name: kibana# portports:containerPort: 5601name: kibana #关闭健康检查readinessProbe:exec:command:"echo""ok"initialDelaySeconds: 5periodSeconds: 10livenessProbe:exec:command:"echo""ok"initialDelaySeconds: 15periodSeconds: 20: "kibana-oss:v7.8.0"imagePullPolicy: Alwaysenv:name: "ELASTICSEARCH_HOSTS"value: "https://dev-elasticsearch.abc.com:443"volumeMounts:mountPath: /usr/share/kibana/config/kibana.ymlname: kibana-configsubPath: kibana.yml# add-resourcesresources:# default-resources# overwritten limitslimits:cpu: 1memory: 2Gi# overwritten requestsrequests:cpu: 1memory: 2Gi
YAML文件是没啥问题
应用kibana后pod不断重启,查看一下pod日志
{"type":"log","@timestamp":"2020-11-21T01:14:05Z","tags":["warning","plugins-discovery"],"pid":7,"message":"Expect plugin \"id\" in camelCase, but found: apm_oss"}{"type":"log","@timestamp":"2020-11-21T01:14:06Z","tags":["info","plugins-service"],"pid":7,"message":"Plugin \"visTypeXy\" is disabled."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Setting up [42] plugins: [usageCollection,telemetryCollectionManager,telemetry,kibanaLegacy,devTools,uiActions,statusPage,share,newsfeed,mapsLegacy,kibanaUtils,kibanaReact,inspector,embeddable,indexPatternManagement,esUiShared,discover,charts,bfetch,expressions,data,home,console,apm_oss,management,advancedSettings,telemetryManagementSection,visualizations,visTypeVislib,visTypeVega,visTypeTimeseries,visTypeTimelion,visTypeTagcloud,visTypeMetric,visTypeTable,visTypeMarkdown,inputControlVis,savedObjects,navigation,visualize,dashboard,savedObjectsManagement]"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","savedobjects-service"],"pid":7,"message":"Starting saved objects migrations"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["fatal","root"],"pid":7,"message":"TypeError: Cannot read property 'properties' of undefined\n at assertIsSupportedIndex (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:315:32)\n at fetchInfo (/usr/share/kibana/src/core/server/saved_objects/migrations/core/elastic_index.js:70:10)\n at process._tickCallback (internal/process/next_tick.js:68:7)"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins-system"],"pid":7,"message":"Stopping all plugins."}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","savedObjectsManagement"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","dashboard"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualize"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","inputControlVis"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMarkdown"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTable"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeMetric"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTagcloud"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimelion"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeTimeseries"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVega"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visTypeVislib"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","visualizations"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","advancedSettings"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","management"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","apm_oss"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","console"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","home"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","data"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","expressions"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","bfetch"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","discover"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","share"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","kibanaLegacy"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetry"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","telemetryCollectionManager"],"pid":7,"message":"Stopping plugin"}{"type":"log","@timestamp":"2020-11-21T01:14:07Z","tags":["info","plugins","usageCollection"],"pid":7,"message":"Stopping plugin"}FATAL TypeError: Cannot read property 'properties' of undefined
报错关键信息
FATAL TypeError: Cannot read property 'properties' of undefined
是不是一脸懵,看不出啥问题,就先google了,没任何结果,debug了半天,依然没有任何头绪,第二天一早继续,精神状态贼好
既然网上查不出啥,就慢慢梳理咯,首先构造一个post请求debug一下,看返回什么
curl -X POST "https://dev-elasticsearch.abc.com/_aliases" -H 'Content-Type: application/json' -d' { "actions" : [ { "add" : { "index" : ".kibana_1", "alias" : ".kibana" } } ] }' |jq .
我这里是返回一个网关的404页面的信息
果然是网关这边的原因,先看下网关的配置,这里只截取了location关键部分
location / {# root html;# index 50x.html index.htm;proxy_redirect off;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Nginx-Proxy true;proxy_set_header Host $http_host;proxy_intercept_errors on;
眼尖的可能已经发现了,是的,就是这个参数`proxy_intercept_errors on;`
先看下nginx官网对这个参数的解释:
