You are viewing documentation for KubeSphere version:v3.0.0

KubeSphere v3.0.0 documentation is no longer actively maintained. The version you are currently viewing is a static snapshot. For up-to-date documentation, see the latest version.

Last updated:

Edit

Feedback

KubeSphere Logging System

What is KubeSphere Logging System

KubeSphere provides a powerful, holistic and easy-to-use logging system for log collection, query and management. It covers logs at varied levels, including tenants, infrastructure resources, and applications. Users can search logs from different dimensions, such as project, workload, Pod and keyword. Compared with Kibana, the tenant-based logging system of KubeSphere features better isolation and security among tenants as tenants can only view their own logs. Apart from KubeSphere’s own logging system, the container platform also allows users to add third-party log collectors, such as Elasticsearch, Kafka and Fluentd.

For more information, see Log Query.

Enable Logging before Installation

Installing on Linux

When you install KubeSphere on Linux, you need to create a configuration file, which lists all KubeSphere components.

In the tutorial of Installing KubeSphere on Linux, you create a default file config-sample.yaml. Modify the file by executing the following command:
```
vi config-sample.yaml
```
Note
- If you adopt All-in-One Installation, you do not need to create a config-sample.yaml file as you can create a cluster directly. Generally, the all-in-one mode is for users who are new to KubeSphere and look to get familiar with the system. If you want to enable Logging in this mode (for example, for testing purposes), refer to the following section to see how Logging can be installed after installation.
- If you adopt Multi-node Installation and are using symbolic links for docker root directory, make sure all nodes follow the exactly same symbolic links. Logging agents are deployed in DaemonSets onto nodes. Any discrepancy in container log path may cause collection failures on that node.

In this file, navigate to logging and change false to true for enabled. Save the file after you finish.

logging:
    enabled: true # Change "false" to "true"

Note

By default, KubeKey will install Elasticsearch internally if Logging is enabled. For a production environment, it is highly recommended that you set the following values in config-sample.yaml if you want to enable Logging, especially externalElasticsearchUrl and externalElasticsearchPort. Once you provide the following information before installation, KubeKey will integrate your external Elasticsearch directly instead of installing an internal one.

es:  # Storage backend for logging, tracing, events and auditing.
  elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
  elasticsearchDataReplicas: 1     # total number of data nodes
  elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
  elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
  logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
  elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
  externalElasticsearchUrl: # The URL of external Elasticsearch
  externalElasticsearchPort: # The port of external Elasticsearch

Create a cluster using the configuration file:
```
./kk create cluster -f config-sample.yaml
```

Installing on Kubernetes

The process of installing KubeSphere on Kubernetes is same as stated in the tutorial of Installing KubeSphere on Kubernetes except the optional component Logging needs to be enabled first in the cluster-configuration.yaml file.

Download the file cluster-configuration.yaml and open it for editing.
```
vi cluster-configuration.yaml
```

In this local cluster-configuration.yaml file, navigate to logging and enable Logging by changing false to true for enabled. Save the file after you finish.

logging:
    enabled: true # Change "false" to "true"

Note

By default, ks-installer will install Elasticsearch internally if Logging is enabled. For a production environment, it is highly recommended that you set the following values in cluster-configuration.yaml if you want to enable Logging, especially externalElasticsearchUrl and externalElasticsearchPort. Once you provide the following information before installation, ks-installer will integrate your external Elasticsearch directly instead of installing an internal one.

es:  # Storage backend for logging, tracing, events and auditing.
  elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
  elasticsearchDataReplicas: 1     # total number of data nodes
  elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
  elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
  logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
  elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
  externalElasticsearchUrl: # The URL of external Elasticsearch
  externalElasticsearchPort: # The port of external Elasticsearch

Execute the following commands to start installation:

kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.0.0/kubesphere-installer.yaml

kubectl apply -f cluster-configuration.yaml

Enable Logging after Installation

Log in to the console as admin. Click Platform in the top-left corner and select Clusters Management.
Click CRDs and enter clusterconfiguration in the search bar. Click the result to view its detail page.

Info

A Custom Resource Definition (CRD) allows users to create a new type of resources without adding another API server. They can use these resources like any other native Kubernetes objects.
In Resource List, click the three dots on the right of ks-installer and select Edit YAML.

In this yaml file, navigate to logging and change false to true for enabled. After you finish, click Update in the bottom-right corner to save the configuration.

logging:
    enabled: true # Change "false" to "true"

Note

By default, Elasticsearch will be installed internally if Logging is enabled. For a production environment, it is highly recommended that you set the following values in this yaml file if you want to enable Logging, especially externalElasticsearchUrl and externalElasticsearchPort. Once you provide the following information, KubeSphere will integrate your external Elasticsearch directly instead of installing an internal one.

es:  # Storage backend for logging, tracing, events and auditing.
  elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
  elasticsearchDataReplicas: 1     # total number of data nodes
  elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes
  elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes
  logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
  elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log
  externalElasticsearchUrl: # The URL of external Elasticsearch
  externalElasticsearchPort: # The port of external Elasticsearch

You can use the web kubectl to check the installation process by executing the following command:
```
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
```
Tip

You can find the web kubectl tool by clicking the hammer icon in the bottom-right corner of the console.

Verify the Installation of the Component

Go to Components and check the status of Logging. You may see an image as follows:

logging

Execute the following command to check the status of Pods:

kubectl get pod -n kubesphere-logging-system

The output may look as follows if the component runs successfully:

NAME                                          READY   STATUS    RESTARTS   AGE
elasticsearch-logging-data-0                  1/1     Running   0          9m33s
elasticsearch-logging-data-1                  1/1     Running   0          5m12s
elasticsearch-logging-discovery-0             1/1     Running   0          9m33s
fluent-bit-qpvrf                              1/1     Running   0          4m56s
fluentbit-operator-5bf7687b88-z7bgg           1/1     Running   0          9m26s
logsidecar-injector-deploy-667c6c9579-662pm   2/2     Running   0          8m56s
logsidecar-injector-deploy-667c6c9579-tjckn   2/2     Running   0          8m56s

Previous : KubeSphere DevOps System Next : KubeSphere Events Next

What’s on this Page