前言
多版本就是字面意思：在相同的bucket中保留对象的多个版本。功能目的是为了可以恢复因意外删除或覆盖操作而失去的对象。
好久之前玩过这个功能，但是没有记录下来，下面简单记录下。
1、实验环境
三个虚拟机组成ceph集群，每台虚拟机配置信息一样，如下：
[root@ceph05 ~]# hostnamectl
Static hostname: ceph05
Icon name: computer-vm
Chassis: vm
Machine ID: e8e9768445e2448fb3ff5e5b7138039c
Boot ID: 8be60bf35f1a4eb5be558934342009bb
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-957.el7.x86_64
Architecture: x86-64
搭建标准的三节点集群：
[root@ceph05 ~]# ceph -v
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
[root@ceph05 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.11691 root default
-3 0.03897 host ceph05
0 hdd 0.01949 osd.0 up 1.00000 1.00000
1 hdd 0.01949 osd.1 up 1.00000 1.00000
-5 0.03897 host ceph06
2 hdd 0.01949 osd.2 up 1.00000 1.00000
3 hdd 0.01949 osd.3 up 1.00000 1.00000
-7 0.03897 host ceph07
4 hdd 0.01949 osd.4 up 1.00000 1.00000
5 hdd 0.01949 osd.5 up 1.00000 1.00000
ceph05节点配置了一个rgw实例：
[root@ceph05 ~]# cat /etc/ceph/ceph.conf
...
# 配置rgw实例信息
[client.rgw.inst01]
rgw_frontends = civetweb port=9001
[root@ceph05 ~]# systemctl start ceph-radosgw@rgw.inst01
ceph06节点作为rgw的客户端，安装awscli工具：
[root@ceph06 ~]# yum install -y awscli
2、实际操作
注意：多版本是针对某个bucket去操作的。
2.1、上传对象
先说结论：一个没有开启多版本的bucket，上传相同对象名的对象会覆盖，但是开启了多版本的bucket，会保留对象之前的版本。
创建bucket01:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api create-bucket --bucket bucket01
上传一个名为test-key-1的对象:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket bucket01 --key test-key-1 --body /tmp/test-1
{
"ETag": "\"1181c1834012245d785120e3505ed169\""
}
查看bucket01下的对象信息:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket bucket01
{
"Versions": [
{
"LastModified": "2019-11-18T09:52:17.501Z",
"VersionId": "null", # 如果没有开启多版本，此字段为null
"ETag": "\"1181c1834012245d785120e3505ed169\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
再次上传一个名为test-key-1的对象:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket bucket01 --key test-key-1 --body /tmp/test-2
{
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\""
}
查看bucket01下的对象信息，看到之前上传的对象信息已经被覆盖了:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket bucket01
{
"Versions": [
{
"LastModified": "2019-11-18T09:52:32.016Z",
"VersionId": "null",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
开启bucket01的多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket bucket01 --versioning-configuration Status=Enabled
查看是否开启成功：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api get-bucket-versioning --bucket bucket01
{
"Status": "Enabled" # 表示开启成功
}
再次上传一个名为test-key-1的对象:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket bucket01 --key test-key-1 --body /tmp/test-4
{
"ETag": "\"45a62d3d5d3e946250904697486591bc\""
}
查看bucket01下的对象信息:
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket bucket01
{
"Versions": [
{
"LastModified": "2019-11-19T06:20:54.613Z",
"VersionId": "urD9LXRy9.ZgoDaYNagoZh911SqJlbG",
"ETag": "\"45a62d3d5d3e946250904697486591bc\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
},
{
"LastModified": "2019-11-18T09:52:32.016Z",
"VersionId": "null",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"Size": 4
}
]
}
从上面的信息中可以看出，现在有两个版本的名为test-key-1的对象了，VersionId分别是null和urD9LXRy9.ZgoDaYNagoZh911SqJlbG，其中VersionId是urD9LXRy9.ZgoDaYNagoZh911SqJlbG的对象的IsLatest字段为true，表示改版本的对象是当前最新的。
2.2、获取对象
先说结论：没有开启多版本时，相同对象名的对象只有一个，获取时也就只能获取这一个了。开启了多版本时，默认获取当前最新版本的对象，也可以显示指定–version-id参数来获取指定version的对象。
默认获取当前最新的版本的对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api get-object --bucket bucket01 --key test-key-1 test-key-1.out
{
"AcceptRanges": "bytes",
"ContentType": "binary/octet-stream",
"LastModified": "Tue, 19 Nov 2019 06:20:54 GMT",
"ContentLength": 4,
"VersionId": "urD9LXRy9.ZgoDaYNagoZh911SqJlbG",
"ETag": "\"45a62d3d5d3e946250904697486591bc\"",
"Metadata": {}
}
获取指定版本的对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api get-object --bucket bucket01 --key test-key-1 --version-id null test-key-1.out
{
"AcceptRanges": "bytes",
"ContentType": "binary/octet-stream",
"LastModified": "Mon, 18 Nov 2019 09:52:32 GMT",
"ContentLength": 4,
"VersionId": "null",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"Metadata": {}
}
2.3、删除对象
先说结论：没有开启多版本时，删除对象时将永久删除，且不可恢复。开启了多版本时，分两种情况，一种是不指定–version-id删除对象，此时rgw不会永久删除该对象，只是给该对象打上一个被删除标记，还有一种是显示指定–version-id参数来删除对象，此时rgw会永久删除该对象。
新建一个bucket test-bucket-2：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api create-bucket --bucket test-bucket-2
上传一个名为test-key-1的对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-2 --key test-key-1 --body /tmp/test-2
{
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\""
}
查看bucket里面的对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-2
{
"Versions": [
{
"LastModified": "2019-11-20T01:59:37.478Z",
"VersionId": "null",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
删除该对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api delete-object --bucket test-bucket-2 --key test-key-1
再次查看bucket里面对象的信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-2
可以看到对象被永久删除了。
下面看下开启多版本后删除对象情况，新建一个test-bucket-3 bucket，并开启多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api create-bucket --bucket test-bucket-3
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-3 --versioning-configuration Status=Enabled
上传一个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-3 --key test-key-1 --body /tmp/test-2
{
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\""
}
删除这个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api delete-object --bucket test-bucket-3 --key test-key-1
{
"VersionId": "ymH3s-OeK.D4kNAHaj8l8laPRn5hFtM",
"DeleteMarker": true
}
查看bucket里面对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-3
{
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"VersionId": "ymH3s-OeK.D4kNAHaj8l8laPRn5hFtM",
"Key": "test-key-1",
"LastModified": "2019-11-20T02:08:10.572Z"
}
],
"Versions": [
{
"LastModified": "2019-11-20T02:06:13.742Z",
"VersionId": "d2-tVH26q4tfS-a5gn7qFsPJ.tMgYq1",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"Size": 4
}
]
}
看到在DeleteMarkers下增加了一个标记对象信息，这个DeleteMarkers下面的信息只是用来标记哪些对象被删除过，暂时没发现有其他用处。然后把被删除对象的IsLatest字段变成了false（可以把这个对象恢复到当前最新版本），然后尝试获取该对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api get-object --bucket test-bucket-3 --key test-key-1 test-key-1.out
An error occurred (NoSuchKey) when calling the GetObject operation: Unknown
看到默认是获取不到对象了（可以指定version-id去获取），但是对象数据还是保留在rgw里面。开启多版本时永久删除对象方式就是指定version-id参数：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api delete-object --bucket test-bucket-3 --key test-key-1 --version-id d2-tVH26q4tfS-a5gn7qFsPJ.tMgYq1
{
"VersionId": "d2-tVH26q4tfS-a5gn7qFsPJ.tMgYq1"
}
2.4、恢复对象
先说结论：恢复一个对象，推荐的方法是copy你想恢复的对象版本到当前的bucket里面，那么该对象就是当前最新的了。
创建一个test-bucket-4：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api create-bucket --bucket test-bucket-4
开启多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-4 --versioning-configuration Status=Enabled
上传一个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-4 --key test-key-1 --body /tmp/test-2
{
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\""
}
查看对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-4
{
"Versions": [
{
"LastModified": "2019-11-22T06:20:05.328Z",
"VersionId": "CIkac6aeClGjppAcBpRvFKPQACWvdny",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
删除这个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api delete-object --bucket test-bucket-4 --key test-key-1
{
"VersionId": "pwUBcmPcwRh2BnXOVB4juJcQGN7vkrL",
"DeleteMarker": true
}
恢复对象到之前的版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api copy-object --bucket test-bucket-4 --copy-source test-bucket-4/test-key-1?versionId=CIkac6aeClGjppAcBpRvFKPQACWvdny --key test-key-1
{
"CopyObjectResult": {
"LastModified": "2019-11-22T06:22:35.892Z",
"ETag": "348bd3ce10ec00ecc29d31ec97cd5839"
}
}
查看对象信息，可以看到版本为9bThYMWjdOYtQh8mmYzyulzFcOQoEwn的对象已经是当前最新的对象了（IsLatest字段为true）：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-4
{
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"VersionId": "pwUBcmPcwRh2BnXOVB4juJcQGN7vkrL",
"Key": "test-key-1",
"LastModified": "2019-11-22T06:20:36.817Z"
}
],
"Versions": [
{
"LastModified": "2019-11-22T06:22:35.892Z",
"VersionId": "9bThYMWjdOYtQh8mmYzyulzFcOQoEwn",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
},
{
"LastModified": "2019-11-22T06:20:05.328Z",
"VersionId": "CIkac6aeClGjppAcBpRvFKPQACWvdny",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"Size": 4
}
]
}
2.5、关闭多版本
先说结论：关闭多版本后，上传对象会覆盖当前最新的对象信息，并且删除对象时会删除version-id是null的对象信息。
创建一个bucket：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api create-bucket --bucket test-bucket-6
开启多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-6 --versioning-configuration Status=Enabled
关闭多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-6 --versioning-configuration Status=Suspended
上传一个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-6 --key test-key-1 --body /tmp/test-2
{
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\""
}
查看对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-6
{
"Versions": [
{
"LastModified": "2019-11-22T07:42:23.645Z",
"VersionId": "null",
"ETag": "\"348bd3ce10ec00ecc29d31ec97cd5839\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
再次上传一个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-6 --key test-key-1 --body /tmp/test-3
{
"ETag": "\"40b134ab8a3dee5dd9760a7805fd495c\""
}
查看对象信息,可以看到关闭多版本之后，再次上传对象会覆盖之前的对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-6
{
"Versions": [
{
"LastModified": "2019-11-22T07:42:36.041Z",
"VersionId": "null",
"ETag": "\"40b134ab8a3dee5dd9760a7805fd495c\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
}
]
}
此时开启多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-6 --versioning-configuration Status=Enabled
上传一个对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-object --bucket test-bucket-6 --key test-key-1 --body /tmp/test-4
{
"ETag": "\"45a62d3d5d3e946250904697486591bc\""
}
查看对象信息：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-6
{
"Versions": [
{
"LastModified": "2019-11-22T07:43:06.724Z",
"VersionId": "JGTD3E5hznLQLprYC7pH3fGhYzId8Q6",
"ETag": "\"45a62d3d5d3e946250904697486591bc\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"Size": 4
},
{
"LastModified": "2019-11-22T07:42:36.041Z",
"VersionId": "null",
"ETag": "\"40b134ab8a3dee5dd9760a7805fd495c\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"Size": 4
}
]
}
关闭多版本：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api put-bucket-versioning --bucket test-bucket-6 --versioning-configuration Status=Suspended
删除对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api delete-object --bucket test-bucket-6 --key test-key-1
{
"VersionId": "i6BI5Yzm07lX2A.vwP.ttGuaudC7h3K",
"DeleteMarker": true
}
查看对象信息，这里可以看到，关闭多版本后，删除对象时，如果不指定version-id，会删除version-id为null的对象：
[root@ceph06 ~]# aws --endpoint=http://192.168.10.30:9001 s3api list-object-versions --bucket test-bucket-6
{
"DeleteMarkers": [
{
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": true,
"VersionId": "null",
"Key": "test-key-1",
"LastModified": "2019-11-22T07:47:34.591Z"
}
],
"Versions": [
{
"LastModified": "2019-11-22T07:43:06.724Z",
"VersionId": "JGTD3E5hznLQLprYC7pH3fGhYzId8Q6",
"ETag": "\"45a62d3d5d3e946250904697486591bc\"",
"StorageClass": "STANDARD",
"Key": "test-key-1",
"Owner": {
"DisplayName": "rgwuser02",
"ID": "rgwuser02"
},
"IsLatest": false,
"Size": 4
}
]
}
