k8s(十三)、企业级docker仓库Harbor在kubernetes上搭建使用

发表于 | 阅读次数：
本文字数： 14k | 阅读时长 ≈ 13 分钟

前言

Harbor是Docker镜像仓库管理系统，用于企业级管理容器镜像，支持对接ldap进行权限管理,由VMVare中国团队开发。
项目地址:
https://github.com/goharbor/harbor/
安装方式:
1.Docker Compose方式部署:
https://github.com/goharbor/harbor/blob/master/docs/installation_guide.md
2.通过kubernetes直接部署:
新版本已废弃
3.通过helm/kubernetes方式部署:
https://github.com/goharbor/harbor-helm/blob/master/README.md
推荐使用,本文将采用此方式

Helm安装

helm简介

Helm是一个用于Kubernetes的包管理工具,每一个安装包构成一个chart基本单位.想象一下,一个完整的部署,可能包含前端\后端\中间件\DB的部署实例,以及k8s管理对应的secret/configmap/svc/ing/pv/pvc等众多资源类型,在部署管理或者卸载之前,如果逐一编辑管理这些资源的yaml文件,工作量比较庞大且容易疏漏.helm的出现很好的解决了这个问题,使用模板以及变量替换的方式,按需生成部署yaml文件包,并且向kubernetes api发送调度请求,虽然额外地引入了一些学习成本,但对于部署包的管理无疑是带来了很大的便利性的.

helm组件

Helm
Helm是一个cli客户端，可以完成如下内容：

1.创建/打包/调试Chart
2.创建本地Chart仓库,管理本地和远程Chart仓库
3.与Tiller交互并完成Chart的安装，升级，删除，回滚，查看等操作

Tiller
1.监听helm客户端的请求,根据chart生成相应的release
2.将release的资源信息发送给kubernetes api,实现资源增删改查.
3.追踪release的状态,实现资源的更新/回滚操作

安装:
安装较为简单,参考官方文档,不做复述:
https://docs.helm.sh/using_helm/#installing-helm

Harbor安装

这里采用次新版本的v1.5.3版本

1
2
3
root@h1:# wget https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz
root@h1:# tar -xf https://github.com/goharbor/harbor/archive/v1.5.3.tar.gz 
root@h1:# cd harbor-1.5.3/contrib/helm/harbor/

helm方式安装的harbor默认是https的,因此需要k8s集群ingress网关开启https协议,traefik开启https参考此前文章:
k8s（二）、对外服务

更新helm dependency:

harbor的helm部署依赖了postgresql的helm，在官方的安装文档没有明确说明，直接按照官方文档说明安装，就会缺失postgresql的部署，导致helm install failed

1
helm dependency update

修改value.yaml文件,添加pvc持久存储并将相应组件上下文中的volume部分注释打开:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    
adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

完整value.yaml文件如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
# Configure persisten Volumes per application
## Applications that require storage have a `volumes` definition which will be used
## when `persistence.enabled` is set to true.
## example
# mysql:
#   volumes:
#     data:
## Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
##   set, choosing the default provisioner.  (gp2 on AWS, standard on
##   GKE, AWS & OpenStack)
##
#     storageClass: "-"
#     accessMode: ReadWriteOnce
#     size: 1Gi

mysql:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
adminserver:
  volumes:
    config:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 1G

registry:
  volumes:
    data:
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 5G

## Configure resource requests and limits per application
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
# mysql:
#   resources:
#     requests:
#       memory: 256Mi
#       cpu: 100m

persistence:
  enabled: true

# The tag for Harbor docker images.
harborImageTag: &harbor_image_tag v1.4.0

# The FQDN for Harbor service.
externalDomain: harbor.my.domain
# If set to true, you don't need to set tlsCrt/tlsKey/caCrt, but must add
# Harbor FQDN as insecure-registries for your docker client.
insecureRegistry: false
#insecureRegistry: true
# The TLS certificate for Harbor. The common name of tlsCrt must match the externalDomain above.
tlsCrt: 
#tlsCrt: root/traefik/ssl/kokoerp.crt
tlsKey:
#tlsKey: root/traefik/ssl/kokoerp.key
caCrt:

# The secret key used for encryption. Must be a string of 16 chars.
secretKey: not-a-secure-key

# These annotations allow the registry to work behind the nginx
# ingress controller.
ingress:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/body-size: "0"
    ingress.kubernetes.io/proxy-body-size: "0"

adminserver:
  image:
    repository: vmware/harbor-adminserver
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  emailHost: "smtp.mydomain.com"
  emailPort: "25"
  emailUser: "sample_admin@mydomain.com"
  emailSsl: "false"
  emailFrom: "admin <sample_admin@mydomain.com>"
  emailIdentity: ""
  emailInsecure: "False"
  emailPwd: not-a-secure-password
  adminPassword: Harbor12345
  authenticationMode: "db_auth"
  selfRegistration: "on"
  ldap:
    url: "ldaps://ldapserver"
    searchDN: ""
    baseDN: ""
    filter: "(objectClass=person)"
    uid: "uid"
    scope: "2"
    timeout: "5"
    verifyCert: "True"
  ## Persist data to a persistent volume
  volumes:
    config:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## jobservice
#
jobservice:
  image:
    repository: vmware/harbor-jobservice
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
# resources:
#   requests:
#     memory: 256Mi
#     cpu: 100m

## UI
#
ui:
  image:
    repository: vmware/harbor-ui
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  secret: not-a-secure-secret
  privateKeyPem: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIJKAIBAAKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AQKCAgBEUigO8/4UJse6xKr3APHv7E94NjKtjMqPT8RhDCLhqAH/lRuClTVb8k0Y
    RILi6oHggsKGDvkS1vJEESCU5LfYBjDAX/r/M0I7gp6TU1AukAXKMdETvkfoMbg/
    9j7W/G152hF4KztvjwmcHyUd7aay+SDh0n1taPm/FzaXfgONwmQFmo40uQ2SfwhX
    I3tD6iMWjASLV4eRfe5w88WpJQ3r5IGYMNuKFF1RcV7MNL3xMHBAwl1kudmRWY4w
    p6+83Gc0m+2AQbY70TkQuRbeUFkIBsWn99yEqXC+7h2us+JLm57iGN1ByQvVnEwL
    Zs7Pl0Hge4leSxeZWhv+aE1R/jm/VdG4dglInuhED0ug8WAJg58IkDYfMKOOALHx
    +0CNHE02XqqUIFwboZJSYTjMYvFL1i14L30FWnqH/0kDs4whXHbnGWhVustsMSK9
    iyIGepuGhMnvtUF1wa/SrBd12qfDj68QHDXsKKbs6eTNYHfn3QL9uisrfMIa5HAt
    nX2YOsAVxg+yvxkWD6n1DU+a/+pAu6iAgiwyxSZiyn6vJUE2zO6pJNbk1kJW6jU3
    A69srtbO4jQn4EM859XYSqdqwXgJL+XJEYNbBcHalmiIOvRg9CCvDSKS7M5rJ0M1
    L7oCzl6EW+zUb4JHkSO7V5uxIZu2sEduw5gofQ3OT9L/qDhDIQKCAQEA8T/8okF2
    Q7SOj3su6KKX6H/ab31SvHECf/oeJtH8ZfLBYL55Yof0pZwq8iXQ26d8cH7FPKBo
    hz0RZ9i2S3bYkzEVCPv9ISFg1NACxL3dU0PMBnmbmg2vPhMzEuQI2JOUu6ILOXEN
    mImvfjZXps/b8OjQgzicH0skBBcbUlXT3a4fF52ktC8FiXgBG9JYg5CsXmfPRxci
    ITa4w4ZLEuECmtJieS0MdKXPLwUVv3e2BlNS6c1JzXyp6EyX/euJ8cCe3n/GHbTY
    2j1OO+xTQfQJVf6S9f2mSzjdHe9KZwWKgyxQ9dZ9Qtho2z/gUN9/UkL52fdljjlw
    ++b/z9Ppcl9K0QKCAQEA7y4Fv8dPFLLnr0R/S7eoAKa0S95xVe97EJHVUhWyOI09
    K9VdZHp6be8W0Yd9h/Ks8Zi4EPRiTTaF3yA3iADwdKFeZt49jGzeM+Gl7Q2Ll98W
    I5gOdJkHSVAP2uK7qSjZ8lPCu4iUYRsae+Psam7Yd6X17RP0M966PlUFj1nnrJjQ
    EN4zeh/m01q9vqebB9C1W/ZiJ6rpt6VVHAcOQQ69F/lKdTif4XCvbMIhIXTYNifk
    1oIv2qTDnfzzv+bgrlvpBJPpPYR0Oc7WoEpyd1Y9IzienLZi8RnujV//FXEmJ45E
    F9GE1HOmoERdEWA1bMYhOO5OfRY1HSMuFMA4+5ojSwKCAQEAmwubio/1uMemw3HQ
    kPRGGsdolDR/4tniWGtfy2UzCDY+r7Vaf8eOpIy8UQmatEBsykO+8RrKcvf9Yrc1
    WUSVJevqb+67HPq9p6fTz6uSPXwZ+KNZLGXVFVjzfxWM1dvrP7eB7TXKHhmG7t9v
    76Yw3SBTObI9LCN3jyVmisDcO+E23E+VVbPOpC260K2b81ocXUPsQ+0LIztu/UIm
    p4hyyxug6+3WznTttXNYKch+9IvCgr5Ly0NuUvw+xpMFAZjgwXBu3BKpN4Ek8YAN
    dhqnkVveCTguErQF78IlGBbIkUr+8TAbKsW4hggEWxV4V17yAnJsEz65bTtldqTj
    qHyzsQKCAQBGhv6g/2d9Rgf1cbBLpns+vel6Wbx3x6c1SptpmgY0kMlR7JeeclM5
    qX/EBzzn4pJGp27XaQi3lfVBxyE41HYTHiZVFQF3L/8Rs18XGKBqBxljI4pXrWwt
    nRMfyy3lAqvJvhM082A1hiV4FMx40fi4x1JON00SIoIusSlzjOI4zdLEtpDdWRza
    g+5hktCvLEbeODfXVJmYUoNXQWldm7f8osDm8eyLMIw5+MCGOgsrZPYgnsD3qxAX
    vSgvFSh5oZaDiA4F2tHe3fQBzhIUyHQ8t4xlz447ZBcozv7L1tKWZWgE0f5mGzgu
    GBqNbh4y1fWj8Plp/ytoTSBgdBIZdukjAoIBAELJPSVFnlf/gv6OWRCHyKxquGjv
    fEn/E8bw5WSqMcj/7wiSJozr0Y8oyWjtWXObliLRQXcEhC8w3lLMjNqnFzQOAI7s
    Oa6BQPigqyXZPXG5GK+V0TlUYvZQn9sfCq4YCxUBNtQ4GHbKKl3FGQL3rJiuFr6G
    fVcetuDFNCiIGYbUF+giJ2cEN3a/Q+7fR6V4xC7VDdL+BqM09wZ6R98G48XzCKKp
    ekNpEfmvJiuk9tFFQwDPWcQ6uyHqesK/Wiweo5nh5y2ZPipwcb0uBoYOQH60NqEL
    6MXRVNdtKujjl1XZkG053Nvcz/YfF6lFjDekwgfd9m49b/s0EGTrl7z9z8Y=
    -----END RSA PRIVATE KEY-----
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m

## MySQL Settings. Currently Harbor does not support an external
## MySQL server, only their own image. Until this is fixed, do not
## Change the settings below.
#
mysql:
  image:
    repository: vmware/harbor-db
    tag: *harbor_image_tag
    pullPolicy: IfNotPresent
  # If left blank will use the included mysql service name.
  host: ~
  port: 3306
  user: "root"
  pass: "registry"
  database: "registry"
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

registry:
  image:
    repository: vmware/registry-photon
    tag: v2.6.2-v1.4.0
    pullPolicy: IfNotPresent
  httpSecret: not-a-secure-secret
  logLevel:
# comment out one of the below to use your cloud's object storage.
#  objectStorage:
#    gcs:
#      keyfile: ""
#      bucket: ""
#      chunksize: "5242880"
#    s3:
#      region: ""
#      accesskey: ""
#      secretkey: ""
#      bucket: ""
#      encrypt: "true"
#    azure:
#      accountname: ""
#      accountkey: ""
#      container: ""
  rootCrt: |
    -----BEGIN CERTIFICATE-----
    MIIE0zCCArugAwIBAgIJAIgs3S+hsjhmMA0GCSqGSIb3DQEBCwUAMAAwHhcNMTcx
    MTA5MTcyNzQ5WhcNMjcxMTA3MTcyNzQ5WjAAMIICIjANBgkqhkiG9w0BAQEFAAOC
    Ag8AMIICCgKCAgEA4WYbxdrFGG6RnfyYKlHYML3lEqtA9cYWWOynE9BeaEr/cMnM
    bBr1dd91/Nm6RiYhQvTDU2Kc6NejqjdliW5B9xUoVKayri8OU81a8ViXeNgKwCPR
    AiTTla1zoX5DnvoxpO9G3lxyNvTKXc0cw8NjQDAXpaDbzJYLkshCeuyD9bco8R96
    /zrpBEX8tADN3+3yA3fMcZzVXsBm4BTpHJRk/qBpHYEPSHzxyH3iGMNKk3vMUBZz
    e0EYkK8NCA2CuEKMnC3acx9IdRwkx10abGvHQCLRCVY7rGoak+b0oZ99RJIRQ9Iq
    YXsn8fsMBQly6xxvSeY5XuSP7Xb6JKDt3y8Spi4gR1M/5aEzhuOyu201rMna7Rs/
    GPfaKjBlbX0jiLDa7v4zjsBPsPaf/c4uooz3ICLsdukaom+E538R0EiOkXt/wyw2
    2YmaWNCsYlEpke7cVC33e/0dPBq4IHsVflawSF9OWS23ikVAs/n+76KjuucEDmbT
    aKUYAJjvAmZL14j+EKc/CoplhCe6pKhavjmNIOfCSdlreIPBhOVbf1f817wKoSIZ
    qVyCA1AYNkI9RYS00axtJGBGMlKbdQqCNpLL58c6To2awmckIZCEcATKOp++NoGm
    Ib0bhdSasdGB5VCtwZVluN8bLl13zBKoxTGjNlEatUGDRnDAnLdZbXXffjsCAwEA
    AaNQME4wHQYDVR0OBBYEFCMYYMOL0E/Uyj5wseDfIl7o4ELsMB8GA1UdIwQYMBaA
    FCMYYMOL0E/Uyj5wseDfIl7o4ELsMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEL
    BQADggIBABG8fPvrrR+erpwQFuB/56j2i6sO+qoOJPpAMYwkzICrT0eerWAavwoy
    f0UAKN7cUeEJXjIR7s7CogGFijWdaWaQsXUD0zJq5aotLYZLimEc1O0uAmJEsfYC
    v7mG07eU6ge22sSo5hxhVplGt52hnXnT0DdgSRZpq2mvgd9lcopAidM+KHlaasXk
    IecHKM99KX9D8smr0AcQ6M/Ygbf2qjO9YRmpBIjyQWEake4y/4LWm+3+v08ecg4B
    g+iMC0Rw1QcPqgwaGaWu71RtYhyTg7SnAknb5nBcHIbLb0hdLgQTa3ZdtXgqchIi
    GuFlEBmHFZP6bLJORRUQ0ari5wpXIsYfrB4T8PybTzva3OCMlEsMjuysFr9ewhzM
    9UGLiSQNDyKA10J8WwlzbeD0AAW944hW4Dbg6SWv4gAo51T+6AukRdup5y6lfQ5a
    h4Lbo6pzaA369IsJBntvKvia6hUf/SghnbG7pCHX/AEilcgTb13HndF/G+7aZgKR
    mi9qvNRSDsE/BrgZawovp81+j6aL4y6UtXYspHr+SuWsKYsaH7pl5HspNCyJ5vV6
    dpJAwosFBqSEnI333wAunpMYmi/jKHH/j4WqjLnCInp0/wouzYu42l8Pmz591BSp
    Jag500bEBxqI2RLELgMt/bUdjp4N2M7mrxdrN+2579HTzb6Hviu9
    -----END CERTIFICATE-----
  ## Persist data to a persistent volume
  volumes:
    data:
      storageClass: "cephrbd"
      accessMode: ReadWriteOnce
      size: 5Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

clair:
  enabled: true
  image:
    repository: vmware/clair-photon
    tag: v2.0.1-v1.4.0
    pullPolicy: IfNotPresent
## The following needs to match the credentials
## in the `postgresql` configuration under the
## `postgresql` namespace below.
  postgresPassword: not-a-secure-password
  postgresUser: clair
  postgresDatabase: clair
# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
# pgResources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
#  volumes:
#    pgData:
#      storageClass: "cephrbd"
#      accessMode: ReadWriteOnce
#      size: 1Gi
  # resources:
  #  requests:
  #    memory: 256Mi
  #    cpu: 100m

## Notary support is not yet fully implemented in the Helm Charts
## Enabling it will just break things.
#
notary:
  enabled: false

## Settings for postgresql dependency.
## see https://github.com/kubernetes/charts/tree/master/stable/postgresql
## for further configurables.
postgresql:
  postgresUser: clair
  postgresPassword: not-a-secure-password
  postgresDatabase: clair
  persistence:
    # enabled: false
    enabled: true
    storageClass: "cephrbd"
    accessMode: ReadWriteOnce
    size: 8Gi

部署:

1
helm install . --debug --name hub --set externalDomain=hub.test.com

等待一段时间的镜像拉取后,查看pod:

1
2
3
4
5
6
7
hub-harbor-adminserver-0                 1/1       Running            1          3h
hub-harbor-clair-6c7d9dcdb7-q4lv4        1/1       Running            2          3h
hub-harbor-jobservice-75f7fbcc9c-ggwp4   1/1       Running            2          3h
hub-harbor-mysql-0                       1/1       Running            0          3h
hub-harbor-registry-0                    1/1       Running            1          3h
hub-harbor-ui-57b4674ff9-kcfq6           1/1       Running            0          3h
hub-postgresql-ccf8d56d5-jg4wq           1/1       Running            1          3h

添加dns,指向ingress gateway(即traefik的node ip):

1
echo "192.168.1.238  hub.test.com"  >> /etc/hosts

浏览器打开测试,默认登录口令为admin/Harbor12345
在这里插入图片描述
可以看到默认有一个公开项目library,尝试往这里推送镜像

Docker推送测试

找一台安装了docker的机器,修改docker服务脚本:

1
2
3
4
~# cat /lib/systemd/system/docker.service

#ExecStart=/usr/bin/dockerd -H fd://
ExecStart=/usr/bin/dockerd  --insecure-registry hub.kokoerp.com

重启docker并登录测试:

1
2
3
4
5
6
7
~# systemctl daemon-reload
~# systemctl restart docker

root@h2:~# docker login hub.test.com
Username (admin): admin
Password: 
Login Succeeded

推送镜像测试:

1
2
3
4
5
root@h2:~# docker tag busybox hub.test.com/library/busybox:test1
root@h2:~# docker push hub.test.com/library/busybox:test1
The push refers to a repository [hub.test.com/library/busybox]
8a788232037e: Pushed 
test1: digest: sha256:915f390a8912e16d4beb8689720a17348f3f6d1a7b659697df850ab625ea29d5 size: 527

可以看到推送成功

ldap配置:

在这里插入图片描述

harbor可以基于企业内部的ldap来获取人员信息,可以在ui上创建非公开项目,将ldap中获取的人员加入项目并赋予相应pull push权限,ui操作界面比较友好,这里就不展示了.

赏一瓶快乐回宅水吧～
-------------本文结束感谢您的阅读-------------