Leo Liu
发布于:2019-08-25

基于 Kubernetes 部署 node.js APP

什么是 Kubernetes

Kubernetes 是一个开源容器编排引擎,可以帮助开发者或运维人员部署和管理容器化的应用,能够轻松完成日常开发运维过程中诸如 滚动更新,横向自动扩容,服务发现,负载均衡等需求。了解更多

安装 Kubernetes

可以通过快速安装 kubernetes 集群:

Kubernetes 术语介绍

Pod

Pod 是 Kubernetes 最小调度单位,是一个或一组容器的集合。

Deployment

提供对 Pod 的声明式副本控制。指定 Pod 模版,Pod 副本数量, 更新策略等。

Service

Service 定义了 Pod 的逻辑分组和一种可以访问它们的策略。借助Service,应用可以方便的实现服务发现与负载均衡。

Label & Selector

Kubernetes 中使用 Label 去关联各个资源。

  1. 通过资源对象(Deployment, etc.)上定义的 Label Selector 来筛选 Pod 数量。
  2. 通过 Service 的 Label Selector 来选择对应的 Pod, 自动建立起每个 Service 到对应 Pod 的请求转发路由表。
  3. 通过对某些 Node 定义特定的 Label,并且在 Pod 中添加 NodeSelector 属性,可以实现 Pod 的定向调度(运行在哪些节点上)。

Nodejs 模板项目

node-express-realworld-example-app 是一款 node.js 编写的后端示例应用。使用 Express 作为网络框架,使用 mongodb 持久化数据,使用 jwt 作用户认证等。

项目分析

  1. 分析 package.json, 将使用 npm start 作为容器启动命令,并依赖于一个 realworld-mongo 的容器服务。
{
  "scripts": {
    "mongo:start": "docker run --name realworld-mongo -p 27017:27017 mongo & sleep 5",
    "start": "node ./app.js",
    "dev": "nodemon ./app.js",
    "test": "newman run ./tests/api-tests.postman.json -e ./tests/env-api-tests.postman.json",
    "stop": "lsof -ti :3000 | xargs kill",
    "mongo:stop": "docker stop realworld-mongo && docker rm realworld-mongo"
  },
}
  1. 全局搜索 process.env.并分析代码, 生产环境需要 的环境变量有:
NODE_ENV                # 应设置为 production
MONGODB_URI             # mondodb 服务路径
PORT                    # nodejs 服务端口, 默认3000
SECRET                  # jwt token 的私钥

在项目根目录添加 Dockerfile:

FROM mhart/alpine-node:8
WORKDIR /root/demo
COPY ./package.json .
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

打包镜像

docker build -t example-app .

推送镜像到 Docker Hub

docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASSWORD
docker tag example-app $DOCKERHUB_USER/example-app:v0.1.0
docker push $DOCKERHUB_USER/example-app:v0.1.0

部署 MongoDB

此应用将分为两个服务部署,一个是包含 Nodejs 主程序的 example-app 服务,一个提供存储的 mognodb 服务。

mongodb 使用 https://hub.docker.com/_/mongo 作为镜像,因为有持久化数据,在单节点模式下可以使用 HostPath 挂载存储(多节点模式或挂载 PVC 存储形式推荐使用 KubeSphere 部署集群, 省去繁琐的配置),通过 Service 将服务暴露到 k8s 内部。

mongodb.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  labels: &ref_0
    app: mongodb
  name: mongodb
spec:
  replicas: 1
  selector:
    matchLabels: *ref_0
  template:
    metadata:
      labels: *ref_0
    spec:
      containers:
        - name: container-ur9rxw
          image: mongo
          imagePullPolicy: IfNotPresent
          ports:
            - name: http-mongo
              protocol: TCP
              containerPort: 27017
          volumeMounts:
            - readOnly: false
              mountPath: /data/db
              name: mongodb
      serviceAccount: default
      volumes:
      - hostPath:
          path: /data/db
          type: DirectoryOrCreate
        name: mongodb-pvc
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

配置文件中主要包含:

  1. 定义了 Pod 运行时的镜像 mongo
  2. 暴露 mognodb 服务到 27017 端口
  3. 声明了一个 hostPath,将主机上的 /data/db 路径挂载到容器的 /data/db 路径上(mongodb 默认数据存放路径)。

在 kubernetes 集群中执行 kubectl 部署

kubectl apply -f mongodb.yaml

查看 pod 运行情况:

root@i-auleonob:/home/ubuntu# kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
mongodb-54f949cb5c-r48bj   1/1     Running   0          42s

pod 已成功运行起来

查看 mongodb 服务日志:

root@i-auleonob:/home/ubuntu# kubectl logs mongodb-54f949cb5c-r48bj
2019-09-09T14:05:05.127+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-09-09T14:05:05.132+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=mongodb-54f949cb5c-r48bj
2019-09-09T14:05:05.132+0000 I  CONTROL  [initandlisten] db version v4.2.0
2019-09-09T14:05:05.132+0000 I  CONTROL  [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2019-09-09T14:05:05.132+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1  11 Sep 2018
...
2019-09-09T14:05:06.180+0000 I  NETWORK  [initandlisten] Listening on 0.0.0.0
2019-09-09T14:05:06.180+0000 I  NETWORK  [initandlisten] waiting for connections on port 27017
...

从日志可以看到 mongodb 已成功启动,并监听 27017 端口, 但此时在集群中还无法访问该端口,需要创建一个服务将端口映射到集群。

创建 MongoDB 服务

mongodb-service.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: default
  labels: &ref_0
    app: mongodb
  name: mongodb-service
spec:
  type: ClusterIP
  sessionAffinity: None
  selector: *ref_0
  ports:
    - name: http-mongo
      protocol: TCP
      port: 27017
      targetPort: 27017

在 kubernetes 集群中执行 kubectl 创建服务

kubectl apply -f mongodb-service.yaml

查看服务详情

root@i-auleonob:/home/ubuntu# kubectl describe svc/mongodb-service
Name:              mongodb-service
Namespace:         default
Labels:            app=mongodb
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"mongodb"},"name":"mongodb-service","namespace":"default"...
Selector:          app=mongodb
Type:              ClusterIP
IP:                10.102.156.99
Port:              http-mongo  27017/TCP
TargetPort:        27017/TCP
Endpoints:         192.168.219.10:27017
Session Affinity:  None
Events:            <none>

透过 Kubernetes DNS,在集群内部可以使用 mongodb-service.default:27017 访问到 mongodb 服务。

部署 Example App

example-app 无需挂载存储卷,但容器的启动时需要必要的环境变量。在 env 中填入必要的环境变量,通过 mongodb-service.default 来访问 mongodb 服务。该端口无需暴露到主机上,service 类型选择使用 ClusterIP

example-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  labels:
    version: v1
    app: example-app
  name: example-app-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      version: v1
      app: example-app
  template:
    metadata:
      labels:
        version: v1
        app: example-app
    spec:
      containers:
      - name: container-l6cx0m
        image: leoendlessx/example-app:v0.1.0
        imagePullPolicy: IfNotPresent
        ports:
        - name: http-app
          protocol: TCP
          containerPort: 3000
        env:
        - name: NODE_ENV
          value: production
        - name: MONGODB_URI
          value: mongodb-service.default    # 使 用kubernetes 内部 DNS 来访问 mongodb 服务
        - name: SECRET
          value: asecretkey
      serviceAccount: default
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%

在 kubernetes 集群中执行 kubectl 部署

kubectl apply -f example-app.yaml

查看 Pod 状态及日志

root@i-auleonob:/home/ubuntu# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
example-app-v1-5678d8db79-7nhkd   1/1     Running   0          26s
mongodb-54f949cb5c-r48bj          1/1     Running   0          71m
root@i-auleonob:/home/ubuntu# kubectl logs example-app-v1-5678d8db79-7nhkd

> conduit-node@1.0.0 start /root/demo
> node ./app.js

Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
Listening on port 3000

example-app 程序已成功启动,现在需将服务暴露主机节点上访问,以便主机网络将端口映射到外网。service 类型选择 NodePort

创建 Example App 服务到并暴露到主机

apiVersion: v1
kind: Service
metadata:
  namespace: default
  labels: &ref_0
    version: v1
    app: example-app
  name: example-app
spec:
  type: NodePort
  sessionAffinity: ClientIP
  selector: *ref_0
  ports:
    - name: http-app
      protocol: TCP
      port: 3000
      targetPort: 3000

在 kubernetes 集群中执行 kubectl 创建服务

kubectl apply -f example-app-service.yaml

查看服务详情

root@i-auleonob:/home/ubuntu# kubectl describe svc/example-app
Name:                     example-app
Namespace:                default
Labels:                   app=example-app
                          version=v1
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"example-app","version":"v1"},"name":"example-app","names...
Selector:                 app=example-app,version=v1
Type:                     NodePort
IP:                       10.106.109.178
Port:                     http-app  3000/TCP
TargetPort:               3000/TCP
NodePort:                 http-app  30375/TCP
Endpoints:                192.168.219.48:3000
Session Affinity:         ClientIP
External Traffic Policy:  Cluster
Events:                   <none>

在节点上访问:

root@i-auleonob:/home/ubuntu# curl 127.0.0.1:30375
{"errors":{"message":"Not Found","error":{}}}

可见 example-app 可以正常访问,不过仍需要确认是否能正常访问 mongodb。

example-app 容器需要等待 mongodb service 可用时, 才可以正常启动。在 mongodb service 可用后, 手动删除旧的 example-app Pod 即可。

测试 Example App 服务

注册用户

curl -X POST \
  'http://127.0.0.1:30375/api/users' \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'cache-control: no-cache' \
  -d '{"user":{"email":"john@jacob.com", "password":"johnnyjacob", "username":"johnjacob"}}'
{"user":{"username":"johnjacob","email":"john@jacob.com","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkNzY3MmMzMzdjNGI5MTAwMGVhZmIzYyIsInVzZXJuYW1lIjoiam9obmphY29iIiwiZXhwIjoxNTczMjI3NzE1LCJpYXQiOjE1NjgwNDM3MTV9.24OGisacl8-n4SooYr8kdcAOYTBvz27sC1mUyU3VkKM"}}

登录

curl -X POST \
  'http://127.0.0.1:30375/api/users/login' \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'cache-control: no-cache' \
  -d '{"user":{"email":"john@jacob.com", "password":"johnnyjacob"}}'
{"user":{"username":"johnjacob","email":"john@jacob.com","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkNzY3MmMzMzdjNGI5MTAwMGVhZmIzYyIsInVzZXJuYW1lIjoiam9obmphY29iIiwiZXhwIjoxNTczMjI3ODcwLCJpYXQiOjE1NjgwNDM4NzB9.a5p0mCLZLSNbTCECjK9j-5hrgg-tA7mI5iENKdik5Xc"}}

创建 Article

curl -X POST \
  'http://127.0.0.1:30375/api/articles' \
  -H 'Authorization: Token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkNzY3MmMzMzdjNGI5MTAwMGVhZmIzYyIsInVzZXJuYW1lIjoiam9obmphY29iIiwiZXhwIjoxNTczMjI3ODcwLCJpYXQiOjE1NjgwNDM4NzB9.a5p0mCLZLSNbTCECjK9j-5hrgg-tA7mI5iENKdik5Xc' \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'cache-control: no-cache' \
  -d '{"article":{"title":"How to train your dragon", "description":"Ever wonder how?", "body":"Very carefully.", "tagList":["dragons","training"]}}'
{"article":{"slug":"how-to-train-your-dragon-estl69","title":"How to train your dragon","description":"Ever wonder how?","body":"Very carefully.","createdAt":"2019-09-09T15:46:26.935Z","updatedAt":"2019-09-09T15:46:26.935Z","tagList":["dragons","training"],"favorited":false,"favoritesCount":0,"author":{"username":"johnjacob","image":"https://static.productionready.io/images/smiley-cyrus.jpg","following":false}}}

获取所有 Articles

curl -X GET \
  'http://127.0.0.1:30375/api/articles' \
  -H 'Authorization: Token eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjVkNzY3MmMzMzdjNGI5MTAwMGVhZmIzYyIsInVzZXJuYW1lIjoiam9obmphY29iIiwiZXhwIjoxNTczMjI3ODcwLCJpYXQiOjE1NjgwNDM4NzB9.a5p0mCLZLSNbTCECjK9j-5hrgg-tA7mI5iENKdik5Xc' \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-With: XMLHttpRequest' \
  -H 'cache-control: no-cache'
{"articles":[{"slug":"how-to-train-your-dragon-estl69","title":"How to train your dragon","description":"Ever wonder how?","body":"Very carefully.","createdAt":"2019-09-09T15:46:26.935Z","updatedAt":"2019-09-09T15:46:26.935Z","tagList":["dragons","training"],"favorited":false,"favoritesCount":0,"author":{"username":"johnjacob","image":"https://static.productionready.io/images/smiley-cyrus.jpg","following":false}}],"articlesCount":1}

通过测试可发现 example-app 与 mongodb 服务运行正常。