hyperledger fabric 2.2 kubernetes 一、本地开发环境依赖 1 2 3 4 5 6 7 8 # Minikube 安装 v1.20.0 curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.20.0/minikube-darwin-amd64 chmod +x minikube sudo mv minikube /usr/local/bin/ # Minikube start minikube start --image-mirror-country cn \ --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.20.0.iso \ --registry-mirror=https://xxxxx.mirror.aliyuncs.com
1 2 3 4 5 6 # 执行以下指令下载 Hyperledger 命令列工具 curl -sSL http://raw.githubusercontent.com/hyperledger/fabric/master/scripts/bootstrap.sh | bash -s -- 2.2.0 1.5.0 # 或者 curl -sSL https://bit.ly/2ysbOFE | bash -s # 设定环境变数至下载下来的 bin 目录下 export PATH=<path to download location>/bin:$PATH
以上脚本会在本地下载k8s部署需要的容器和可执行文件，云上的话需要准备的镜像如下：
1 2 3 4 centos:latest hyperledger/fabric-orderer:2.2 hyperledger/fabric-peer:2.2.0 hyperledger/fabric-tools:2.2.0
用于测试的chaincode镜像，marbles是hyperledger官方给的一个sample，但是源码没有打镜像，这里其他人打的一个镜像，我们只是用来测试部署是否成功，之后会用我们开发的存证chaincode镜像替换
1 paragones /chaincode-marbles:1 .0
1 2 3 4 # 创建一个名为 hyperledger 的 namespace kubectl create namespace hyperledger # 切换至 hyperledger namespace kubectl config set-context --current --namespace=hyperledger
二、系统架构规划 2.1 节点规划 下图为本范例的部署架构，所有hyperledger 所需节点皆部署于 k8s 中的 hyperledger namespace 当中。架构中包含的节点如下
orderer0 : 排序节点0，用于排序区块
orderer1 : 排序节点1，用于排序区块
orderer2 : 排序节点2，用于排序区块
peer0-org1 : 组织1的Peer节点，用于区块的实际运算、背书以及记帐。
cli-org1-peer : 用于操纵组织1的Peer节点
peer0-org2 : 组织2的Peer节点，用于区块的实际运算、背书以及记帐。
cli-org2-peer : 用于操纵组织2的Peer节点
2.2 存储存放规划
节点
挂载路径
路径说明
PVC
PV hostPath
orderer0
/var/hyperledger/orderer/
存放凭证
orderer0-pvc
./fabric/orderer0
orderer0
/var/hyperledger/production
持久化资料
orderer0-persist-pvc
./fabric/orderer0persist
orderer1
/var/hyperledger/orderer/
存放凭证
orderer1-pvc
./fabric/orderer1
orderer1
/var/hyperledger/production
持久化资料
orderer1-persist-pvc
./fabric/orderer1persist
orderer2
/var/hyperledger/orderer/
存放凭证
orderer2-pvc
./fabric/orderer2
orderer2
/var/hyperledger/production
持久化资料
orderer2-persist-pvc
./fabric/orderer2persist
Org1-Peer
/etc/hyperledger/fabric/
存放凭证
peer0-org1-pvc
./fabric/peer0org1
Org1-Peer
/var/hyperledger/production
持久化资料
peer0-org1-persist-pvc
./fabric/peer0org1persist
Org1-Peer-CLI
/opt/gopath/src/github.com/ hyperledger/fabric/peer/crypto/
存放凭证
peer0-org1-pvc
./fabric/peer0org1
Org2-Peer
/etc/hyperledger/fabric/
存放凭证
peer0-org2-pvc
./fabric/peer0org2
Org2-Peer
/var/hyperledger/production
持久化资料
peer0-org2-persist-pvc
./fabric/peer0org2persist
Org2-Peer-CLI
/opt/gopath/src/github.com/ hyperledger/fabric/peer/crypto/
存放凭证
peer0-org1-pvc
./fabric/peer0org2
三、准备凭证 Hyperledger Fabric 于节点沟通时必须依赖凭证进行沟通，因此必须先签发凭证。在凭证的签发过程中可以使用两种方式签发凭证
cryptogen 命令
fabric-ca 服务
在此范例中将以 cryptogen 命令列生成各节点所需凭证，以下为签发凭证的指令。可以查看 git repo 中的 crypto-config.yaml 为签发凭证设定文件 ，crypto-config则为签发结果 。
1 cryptogen generate --config=crypto-config.yaml --output ./crypto-config
（notice：练习中，需要再次生成新的凭证设定文件时，需要将./crypto-config文件夹删除后，再行运行上述命令。否则凭证设定文件不会被更新覆盖！)
四、准备创世区块 1 configtxgen -profile TwoOrgsOrdererGenesis -channelID devchan -outputBlock ./channel-artifacts/genesis.block
五、产生Channel 所需档案 1 2 3 4 5 6 # 产生Channel 所需档案 configtxgen -profile TwoOrgsChannel -outputCreateChannelTx ./channel-artifacts/channel.tx -channelID "mychannel" # 产生 Org1 使用的 channel 设定文件 configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/org1Anchors.tx -channelID "mychannel" -asOrg org1MSP # 产生 Org2 使用的 channel 设定文件 configtxgen -profile TwoOrgsChannel -outputAnchorPeersUpdate ./channel-artifacts/org2Anchors.tx -channelID "mychannel" -asOrg org2MSP
六、同步档案至PV中 在前述三、四、五章节中我们已经准备了以下内容
由于 Hyperledger 之 container 必须先将以上档案放置至正确位置后 container 才能正常启动。但本次安装不使用NFS 预先将档案填入，而是先启临时的 container 同步档案。
container 名称
说明
orderer-bastion
填充 orderer0-pvc,orderer1-pvc,orderer2-pvc 所需要的文档
peer-bastion
填充 peer0-org1-pvc,peer0-org2-pvc 所需要的文档
以下为同步文档的步骤
6.1 启动临时 container
1 kubectl create -f /deploy-hyperledger-fabric-on-k8s/file-populate-bastion/
这个指令将会产生 pv/pvc/orderer-bastion/peer-bastion 这些资源
6.2 同步 Orderer 所需文档
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 # 登录进入 orderer-bastion kubectl exec -it ${orderer-bastion-pod-name} bash # 安装 git 并 clone deploy-hyperledger-fabric-on-k8s repo yum install git -y git clone https://github.com/willzhuang/deploy-hyperledger-fabric-on-k8s.git # 同步 orderer0 创世区块 cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer0-pvc/ # 同步 orderer0 凭证 cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer0/* /orderer0-pvc/ # 同步 orderer1 创世区块 cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer1-pvc/ # 同步 orderer1 凭证 cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer1/* /orderer1-pvc/ # 同步 orderer2 创世区块 cp deploy-hyperledger-fabric-on-k8s/channel-artifacts/genesis.block /orderer2-pvc/ # 同步 orderer2 凭证 cp -r deploy-hyperledger-fabric-on-k8s/crypto-config/ordererOrganizations/consortium/orderers/orderer2/* /orderer2-pvc/
6.3 同步 Peer 所需要的文档
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 # 登录进入 peer-bastion kubectl exec -it ${peer-bastion-pod-name} bash # 安装 git 並 clone deploy-hyperledger-fabric-on-k8s repo yum install git -y git clone https://github.com/willzhuang/deploy-hyperledger-fabric-on-k8s.git # 同步 org1 peer0 所需之凭证 cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/peerOrganizations/org1/peers/peer0-org1/* peer0-org1-pvc/ # 同步 org1 peer0 cli 所需之凭证 cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/* /peer0-org1-pvc/ # 同步 org1 peer0 所需之 channel设定文档 cp /deploy-hyperledger-fabric-on-k8s/channel-artifacts/* peer0-org1-pvc/ # 同步 org2 peer0 所需之凭证 cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/peerOrganizations/org2/peers/peer0-org2/* peer0-org2-pvc/ # 同步 org2 peer0 cli 所需之凭证 cp -r /deploy-hyperledger-fabric-on-k8s/crypto-config/* /peer0-org2-pvc/ # 同步 org2 peer0 所需之 channel设定文档 cp /deploy-hyperledger-fabric-on-k8s/channel-artifacts/* peer0-org2-pvc/
七、启动 Orderer 1 2 # 启动 orderer cluster kubectl create -f /deploy-hyperledger-fabric-on-k8s/orderer/
內含 orderer0,orderer1,orderer2 所需的 deployment 与 service 资源
八、启动 Peer 1 2 3 4 5 6 7 # 启动 org1 peer0 kubectl create -f /deploy-hyperledger-fabric-on-k8s/org1/ # 內含 org1 peer0/cli 所需的 deployment,configmap,service 资源 # 启动 org2 peer0 kubectl create -f /deploy-hyperledger-fabric-on-k8s/org2/ # 內含 org2 peer0/cli 所需的 deployment,configmap,service 资源
九、创建channel 9.1 将 org1 peer0 加入 channel
1 2 # 登录进入 cli pod kubectl exec -it ${org1-peer0-cli-pod-name} sh
1 2 # 产生 channel 区块 peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx --tls true --cafile $ORDERER_CA
运行成功后，log信息如下：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 /opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel create -o orderer0:7050 -c mychannel -f ./scripts/channel-artifacts/channel.tx - -tls true --cafile $ORDERER_CA 2021-08-10 05:41:03.212 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2021-08-10 05:41:03.242 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND} 2021-08-10 05:41:03.246 UTC [channelCmd] InitCmdFactory -> INFO 003 Endorser and orderer connections initialized 2021-08-10 05:41:03.447 UTC [cli.common] readBlock -> INFO 004 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2021-08-10 05:41:03.450 UTC [channelCmd] InitCmdFactory -> INFO 005 Endorser and orderer connections initialized 2021-08-10 05:41:03.651 UTC [cli.common] readBlock -> INFO 006 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2021-08-10 05:41:03.654 UTC [channelCmd] InitCmdFactory -> INFO 007 Endorser and orderer connections initialized 2021-08-10 05:41:03.857 UTC [cli.common] readBlock -> INFO 008 Expect block, but got status: &{SERVICE_UNAVAILABLE} 2021-08-10 05:41:03.866 UTC [channelCmd] InitCmdFactory -> INFO 009 Endorser and orderer connections initialized 2021-08-10 05:41:04.068 UTC [cli.common] readBlock -> INFO 00a Expect block, but got status: &{SERVICE_UNAVAILABLE} 2021-08-10 05:41:04.070 UTC [channelCmd] InitCmdFactory -> INFO 00b Endorser and orderer connections initialized 2021-08-10 05:41:04.273 UTC [cli.common] readBlock -> INFO 00c Expect block, but got status: &{SERVICE_UNAVAILABLE} 2021-08-10 05:41:04.277 UTC [channelCmd] InitCmdFactory -> INFO 00d Endorser and orderer connections initialized 2021-08-10 05:41:04.480 UTC [cli.common] readBlock -> INFO 00e Received block: 0
1 2 # 加入 channel peer channel join -b mychannel.block
运行成功后，log信息如下：
1 2 3 /opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel join -b mychannel.block 2021-08-10 05:48:59.716 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2021-08-10 05:48:59.736 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
1 2 # 查看是否在 channel 当中 peer channel list
9.2 將 org2 peer0 加入 channel
1 2 3 4 5 6 7 8 9 10 11 kubectl exec -it pod ${org2-peer0-cli-pod-name} sh peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA peer channel join -b mychannel.block peer channel list
运行成功后，log信息如下：
1 2 3 4 5 6 7 8 9 10 /opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel fetch 0 mychannel.block -c mychannel -o orderer0:7050 --tls --cafile $ORDERER_CA 2021-08-10 06:05:04.550 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2021-08-10 06:05:04.553 UTC [cli.common] readBlock -> INFO 002 Received block: 0 /opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel join -b mychannel.block 2021-08-10 06:05:16.493 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized 2021-08-10 06:05:16.513 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel /opt/gopath/src/github.com/hyperledger/fabric/peer # peer channel list 2021-08-10 06:05:24.222 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized Channels peers has joined: mychannel
十、Fabric节点上安装外的Chaincode 10.1 安装chaincode
打包org1
1 2 3 4 5 6 # 进入github的chaincode/packaging # 将connection.json 打包成为 code.tar.gz $ tar cfz code.tar.gz connection.json # 再將code.tar.gz metadata.json包成marbles-org1.tgz $ tar cfz marbles-org1.tgz code.tar.gz metadata.json
将org1 tar文档安装到peer cli pod中
1 2 3 4 5 6 7 8 9 10 11 kubectl cp marbles-org1.tgz hyperledger/${ cli-org1-name}:/opt/gopath/src/github .com/hyperledger/fabric/peer kubectl exec -it ${ cli-org1-name} -- /bin/bash $ peer lifecycle chaincode install marbles-org1.tgz$ peer lifecycle chaincode queryinstalled
打包org2，如打包or1一样的步骤，请修改connection.json中的address
1 "address" : "chaincode-marbles-org2.hyperledger:7052"
依照下列打包步骤
1 2 3 4 5 6 7 $ rm -f code.tar.gz$ tar cfz code.tar.gz connection.json$ tar cfz marbles-org2 .tgz code.tar.gz metadata.json$ peer lifecycle chaincode install marbles-org2 .tgz$ peer lifecycle chaincode queryinstalled
至此，已完成外部链码设置。
10.2 部署”智能合约”
制作golang的alpine镜像文件
1 2 3 4 5 $ docker build -t chaincode/marbles:1.0 . # 将文件推送至dockerhub $ docker login $ docker push chaincode/marbles:1.0
进入github中 chaincode\k8s ，找到org1-chaincode-deployment.yaml和org2-chaincode-deployment.yaml中的CHAINCODE_CCID
将yaml文件中的CHAINCODE_CCID更换成为对应的 chaincode 识别码
1 2 $ kubectl create -f chaincode/k8s
完成部署。
10.3审核链码的安装
请在peer-cli-org1/peer-cli-org2 这2个pod中允许链码的安装，记得修改CHAINCODE_CCID
1 2 -- -- -- . -- - -- - -- - -- -- -- - . , .
检查所有org允许状态
1 2 $peer lifecycle chaincode checkcommitreadiness --channelID mychannel --name marbles --version 1.0 --init-require d --sequence 1 -o -orderer0:7050 --tls --cafile $ORDERER_CA --signature-policy "AND ('org1MSP.peer','org2MSP.peer')"
至此，2个peer-cli都审核完毕。
之后提交到channel 中
1 2 peer lifecycle chaincode commit - o orderer0:7050 -- channelID mychannel -- name marbles -- version 1.0 -- sequence 1 -- init - required -- tls true -- cafile $ORDERER_CA -- peerAddresses peer0- org1:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org1/peers/ peer0- org1/tls/ ca.crt -- peerAddresses peer0- org2:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org2/peers/ peer0- org2/tls/ ca.crt -- signature- policy "AND ('org1MSP.peer','org2MSP.peer')"
10.4测试
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 # 查询安装的链码 peer lifecycle chaincode queryinstalled # 查询审核通过的链码 peer lifecycle chaincode queryapproved -- channelID mychannel -- name marbles # 查询提交完成的链码 peer lifecycle chaincode querycommitted -- channelID mychannel -- name marbles # 塞弹珠(第一次) peer chaincode invoke - o orderer0:7050 -- isInit -- tls true -- cafile $ORDERER_CA - C mychannel - n marbles \ -- peerAddresses peer0- org1:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org1/peers/ peer0- org1/tls/ ca.crt \-- peerAddresses peer0- org2:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org2/peers/ peer0- org2/tls/ ca.crt \- c '{"Args" :["initMarble" ,"marble1" ,"blue" ,"35" ,"tom" ]}' -- waitForEvent# 塞弹珠 peer chaincode invoke - o orderer0:7050 -- tls true -- cafile $ORDERER_CA - C mychannel - n marbles \ -- peerAddresses peer0- org1:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org1/peers/ peer0- org1/tls/ ca.crt \-- peerAddresses peer0- org2:7051 -- tlsRootCertFiles /opt/ gopath/src/ github.com/hyperledger/ fabric/peer/ crypto/peerOrganizations/ org2/peers/ peer0- org2/tls/ ca.crt \- c '{"Args" :["initMarble" ,"marble5" ,"red" ,"50" ,"tom" ]}' -- waitForEvent# 查询弹珠 peer chaincode query - C mychannel - n marbles - c '{"Args" :["readMarble" ,"marble5" ]}'
开发Client Application 目前hyperledger不对外，请使用kubectl port-forward的方式连进去。
1 kubectl port-forward ${pord-id} 7050 :7050
参考 fabric-samples 中typescript所撰写的范例 ※另有java， go 语言的范例!!
以下为typerscript 范例:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 async function main () { try { const ccpPath = path.resolve(__dirname, '..' , 'connection.json' ); const fileExists = fs.existsSync(ccpPath); if (!fileExists) { throw new Error('no such file or directory: ${ccpPath}' ); } const contents = fs.readFileSync(ccpPath, 'utf8' ); console.log(contents); const connectionProfile = JSON.parse(contents); const gateway = new Gateway(); const wallet = await buildWallet (walletPath) ; const identity: X509Identity = { credentials: { certificate: '-----BEGIN CERTIFICATE-----\nMIICBjCCAaygAwIBAgIRAKeXh5QGRlhFCf31amakNiowCgYIKoZIzj0EAwIwWzEL //crypto-config\peerOrganizations\org1\users\Admin@org1\msp\keystore privateKey: '-----BEGIN PRIVATE KEY-----\nMIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg9tkGBGl8aIWYbN/i\nqKjl5dK3D3qXx6ltn3YrlH4NFPehRANCAATMJIf0mY3FQDysZOevzbsBwqttfVuW\nwoIFV9Z0TwBEXXAQ0HPd9u77zV4my0onty3rJnWKF2kuhIn5PUzz61zk\n-----END PRIVATE KEY-----\n', }, mspId: org1UserId, type: 'X.509', }; await wallet.put (org1UserId, identity) ; const gatewayOpts: GatewayOptions = { wallet, identity: org1UserId, discovery: { enabled: false , asLocalhost: false }, // using asLocalhost as this gateway is using a fabric network deployed locally }; await gateway.connect (connectionProfile, gatewayOpts) ; const network = await gateway.getNetwork ('mychannel' ) ; const contract = network.getContract('marbles' ); let result = await contract.evaluateTransaction ('readMarble' , 'marble2' ) ; console.log(`*** Result: ${prettyJSONString(result.toString())}`); } catch (error ) { console.error (`******** FAILED to run the application: ${error }`); } } main();
上述为app.ts，此app.ts会读取connection.json档，此档用于定义peer url以及peer的tls凭证路径。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 { "name" : "Network" , "version" : "1.1" , "channels" : { "mychannel" : { "peers" : [ "peer0-org1" , "peer0-org2" ] } }, "organizations" : { "Org1" : { "mspid" : "org1MSP" , "peers" : [ "peer0-org1" ] }, "Org2" : { "mspid" : "org2MSP" , "peers" : [ "peer0-org2" ] } }, "peers" : { "peer0-org1" : { "url" : "grpcs://peer0-org1:7051" , "grpcOptions" : { "ssl-target-name-override" : "peer0-org1" }, "tlsCACerts" : { "path" : "crypto-config\p eerOrganizations\o rg1\t lsca\t lsca.org1-cert.pem" } }, "peer0-org2" : { "url" : "grpcs://peer0-org2:7051" , "grpcOptions" : { "ssl-target-name-override" : "peer0-org2" }, "tlsCACerts" : { "path" : "crypto-config\p eerOrganizations\o rg2\t lsca\t lsca.org2-cert.pem" } } } }
application连到peer的时候必须要有tls 凭证去验证peer service是否安全及正确。
当交易的时候需要提供该使用者(user or admin)的交易凭证(x509 cert +private key)
路径对应如下：tls CA Cert： crypto-config\peerOrganizations\org1\tlsca\tlsca.org1-cert.pemX509 certificate： crypto-config\peerOrganizations\org1\users\Admin@org1\msp\signcerts\Admin@org1-cert.pem X509 privateKey： crypto-config\peerOrganizations\org1\users\Admin@org1\msp\keystore\priv_sk
成功收到资料如图
附录、参考连结
Hyperledger 参考链接
Kubenetes 参考链接
NFS 架构参考链接
部署 fabric 和 CA 与 Kubenetes 之上
Deploy Hyperledger Fabric network on Kubernetes
