RL Course by David Silver

#Lecture 1: Introduction to Reinforcement Learning

两本书推荐

An Introduction to Reinforcement Learning, Sutton and Barto, 1998 概念多一些
Algorithms for Reinforcement Learning, Szepesvari 数学细节多一些

都有免费的online版本

什么是强化学习

多种交叉的学科

RL的特点

no supervisor, only a reward signal
Feddback is delayed, not instantaneous
time really matters（连续的处理）
Agent’s actions affect the subsequent data it receives(与环境产生交互)

强化学习

每个时刻都有一个 $R_t$ ,表示t时刻的reward。强化学习的目标就是最大化 $R_t$ 。

example of reward

Fly stunt manoeuvres in a helicopter
- +ve reward for following desired trajectory
- −ve reward for crashing
Defeat the world champion at Backgammon
- +/−ve reward for winning/losing a game
Manage an investment portfolio
- +ve reward for each $ in bank
Control a power station
- +ve reward for producing power
- −ve reward for exceeding safety thresholds
Make a humanoid robot walk
- +ve reward for forward motion
- −ve reward for falling over

目标：选择最大化 $R_t$ 的action

action可能影响时间很长
可能会延迟
可能需要放弃一些暂时的reward

Agent and Environment

在每个时间，对于agent而言

执行动作 $A_t$
收到环境信息 $O_t$
收到reward $R_t$

在每个时间，对于enviroment而言

收到动作 $A_t$
发出 $O_t$
发出 $R_t$

History and State

history： $H_t = O_1,R_1,A_1,…A_t-1,O_t,R_t$

将要发生的事情，取决于 $H_t$

State是信息的summary $S_t = f(H_t)$

State

environment state $S_t^e$

environment对于agent是不可见的，即使可见也包含不相干的内容

agent state $S_t^a$

可以是 $H_t$ 的任意函数 $S_t^a = f(H_t)$

information state(a.k.a Markov state) 含有所有的历史有信息,是个信息论的该你那当下面条件满足的时候 $S_t$ 是 Markov $P[S_{t+1}|S_t]=P[S_{t=1}|S_1,..,S_t]$

未来与过去是独立的 $H_{1:t} –> S_t –> H_{t+1: \infty}$
一旦state确定了，所有的history都可以被丢弃
environment state $S_t^e$ 是 Markov
history of everything $H_t$ 是Markov

例子

full Observable

$O_t = S_t^a=S_t^e$ 即可以观察到所有情况，这是个 Markov decision process(MDP)

Partal observability

agent state != environment state

通常这是个 partially observable Markov decision process(POMDP)

这样agent需要有自己的state，例如

完整的历史 $S_t^a = H_t$
对环境状态分布的估计 $S_t^a =(P[s_t^e=s1],…,p[S_t^e=s_n])$
RNN $S_t^a = \sigma(S_t-1^a W_S +O_tW_o)$

RL agent的组成

下面这些并不一定是都是必要的

Policy agent如何选择action
Value function 如何评价一个state或者是action
Model 表示environment

policy

policy是agent的行为他是state到action的映射 确定的policy $a=\pi(s)$ 随机的policy $\pi(a|s)=P[A_t=a|S_t=s]$

Value function

评估action

$V_{\pi}(s)=E_\pi[R_t+\gamma R_{t+1}+\gamma ^2 R_{t+2}

S_t=s]$

model

model预测环境 $P$ 预测next state $R$ 预测 next(immediate) review

例子

RL分类

Value Based
- No policy(implicit)
- Value Function
Policy Base
- Policy
- No value Function
Actor Critic 结合两者
- Policy
- Value Function

Model Free
- Policy and/or ValueFunction
- No model
Model base
- Policy and/or value function
- model

Learning and Planning

强化学习
- 环境不可知
- agent与环境交互
- 提升policy
Planning
- model可知
- agent的行为可由model得到
- agent提升policy
- a.k.a. deliberation, reasoning, introspection, pondering, thought, search 更像搜索

balance Exploration and Exploitation

Exploitation: 最大化以知最优 Exploration：最大化经验化环境

例子：

Restaurant Selection
- Exploitation Go to your favourite restaurant
- Exploration Try a new restaurant
Online Banner Advertisements
- Exploitation Show the most successful advert
- Exploration Show a different advert
Oil Drilling
- Exploitation Drill at the best known location
- Exploration Drill at a new location
Game Playing
- Exploitation Play the move you believe is best
- Exploration Play an experimental move

Prediction and Control

预测：Given a policy，衡量它控制：选择最好的策略

总结

$O_t,R_t,A_t$ 表示的是agent与环境的交互
$H_t$ , $S_t$ 表示的是历史的提炼
$S_t$ 是 Markov 可以理解为信息足够，可以进行推断
agent中model，policy，value funcion和分类
强化学习用于解决不同的问题，Learning和Planing，Prediction和Control，Exploration and Exploitation

Lecture 2: Markov Decision Process

Markov Processes

Introduction

Almost all RL problems can be formalised as MDPs

Markov Property

state是有足够的统计量用于预测未来

### Markov Chains

Markov Process(or Markov Chain)

S 是一组状态 P 是状态转移概论矩阵 $P_{ss’}=P[S_{t+1}=s’|S_t=s]$

### Example: Student Markov Chain

进行sample 有这些可能的序列

Transition Matrix

Markov Reward Process

带有value judgement的 Markov 过程

S 是一组状态 P 是状态转移概论矩阵 $P_{ss’}=P[S_{t+1}=s’|S_t=s]$ $R_s=E[R_{t+1}|S_t=s]$

### Student MRP

### return G： Goal $G_t=R_{t=1}+\gamma R_{t+2}=\sum_{k=0} \gamma^k R_{t+k+1}$

$\gamma$ 是 discount 属于 [0,1] $\gamma$ 接近0 会比较贪心,接近1则会导致比较长远的进化

为什么使用discount

数学上方便
防止Markov process中的环
没有环境的完美model，所与更远的reword,占比更小
人和动物都对马上发生的事情反映
如果在经济上，表示了interest
如果知道序列会终止，可以不使用discount

Value Function

$v(s)=E[G_t

S_t=s]$

如果你从state s开始，Goal是什么。

Example

$S_1=C_1$ 做sample，然后按照公式计算，gamma取0.5

gamma取0

gamma取0.9

### Bellman Equation

value function 由两部分组成

立刻的reward $R_{t+1}$
discounted 的接下来的状态 $\gamma v(S_{t+1})$

### Example

### Bellman Equation in Matrix Form

$v = R + \gamma P v$ R 马上的回报

### Solving the Bellman Equation

MDP Markov Decision Process

马尔科夫过程是一个元组 $S,A,P,R,\gamma$

$S$ 是一组有限的states集合 $A$ 是一组有限的action集合 $P$ 是一个状态转换概率矩阵 $P_{ss’}^a=P[S_{t+1}=s|S_t=s,A_t=a]$ $R$ 是reward function， $R_s^a=E[R_{t+1}|S_t=s,A_t=a]$ $\gamma$ 是 discount factor

例子

### policy

A policy π is a distribution over actions given states, $π(a|s) = P [A_t = a | S_t = s]$

policy 具有对于agent行为的完全的定义
在MDP中，action只基现在的状态

### Value Function state-value function: 从s状态，遵从policy pi 的期望

$v_π (s) = E_π [G_t

S_t = s]$

action-valute function:

在s 状态，采取a操作，然后遵从policy pi 的期望

$q π (s, a) = E π [G t

S t = s, A t = a]$

### value function example

### Bellman Expectation Equation

## Optimal Value Function

在所有的policies下有可能获得的最大值 $v_∗ (s) = \max v_π (s)$ π

在action下能获得的最大数值 $q_∗ (s, a) = \max q_π (s, a)$

optimal value fn 表明了MDP有可能的最好性能
如果optimal value fn能够被知道，那么MDP就被”解决了”

Example Optimal value fn

## Optimal Policy

$π ≥ π’ if v_π (s) ≥ v_{π’}(s), ∀s$

### Example Optimal Policy

## 如何计算

Bellman 优化等式是非线性的
没有近似的solution
采用迭代的方法
- Value iteration
- Policy Iteration
- Q-learning
- Sarsa

xionchen commented on Tue Jul 04 2017

哇,成功了
timchenxiaoyu commented on Fri Jan 05 2018

请问
mount --bind foo foo
有啥意义？
xionchen commented on Fri Jan 05 2018

@timchenxiaoyu
举个例子,可以通过这种方式来创造一个挂在点。
➜ ~ mkdir test_dir
➜ ~ sudo mount --bind test_dir test_dir
➜ ~ mount |grep test_dir
/dev/nvme0n1p6 on /home/xion/test_dir type ext4 (rw,relatime,errors=remount-ro,data=ordered)

这样就可以使用一些mount的属性，最简单的例子，例如：
sudo mount,ro --bind test_dir test_dir
可以让test_dir成为一个是read only的目录。无论改目录中的文件夹或者文件的权限是什么，这个文件夹都是只读的。
除此之外，mount有很多别的属性
yeweimian21 commented on Thu Mar 01 2018

您好，请问，我在用diskimage-builder制作镜像的时候，总是卡在一个地方执行不下去，不知道是哪里配置错了还是网络的原因。
下面是执行过程的输出：
2018-03-01 04:38:32.659 | Running hooks from /tmp/dib_build.cuv4VCZM/hooks/root.d
2018-03-01 04:38:32.671 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-bootloader-default-cmdline
2018-03-01 04:38:32.673 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-bootloader-default-cmdline
2018-03-01 04:38:32.674 | ++ export 'DIB_BOOTLOADER_DEFAULT_CMDLINE=nofb nomodeset vga=normal'
2018-03-01 04:38:32.674 | ++ DIB_BOOTLOADER_DEFAULT_CMDLINE='nofb nomodeset vga=normal'
2018-03-01 04:38:32.674 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-dib-init-system.bash
2018-03-01 04:38:32.676 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-dib-init-system.bash
2018-03-01 04:38:32.676 | ++++ dirname /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-dib-init-system.bash
2018-03-01 04:38:32.678 | +++ PATH=/root/dib-virtualenv/bin:/root/dib-virtualenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/..
2018-03-01 04:38:32.678 | +++ dib-init-system
2018-03-01 04:38:32.680 | ++ DIB_INIT_SYSTEM=upstart
2018-03-01 04:38:32.680 | ++ export DIB_INIT_SYSTEM
2018-03-01 04:38:32.680 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-ubuntu-distro-name.bash
2018-03-01 04:38:32.682 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/10-ubuntu-distro-name.bash
2018-03-01 04:38:32.682 | ++ export DISTRO_NAME=ubuntu
2018-03-01 04:38:32.682 | ++ DISTRO_NAME=ubuntu
2018-03-01 04:38:32.682 | ++ export DIB_RELEASE=xenial
2018-03-01 04:38:32.682 | ++ DIB_RELEASE=xenial
2018-03-01 04:38:32.682 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/14-manifests
2018-03-01 04:38:32.684 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/14-manifests
2018-03-01 04:38:32.684 | ++ export DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2018-03-01 04:38:32.684 | ++ DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2018-03-01 04:38:32.684 | ++ export DIB_MANIFEST_SAVE_DIR=image.d/
2018-03-01 04:38:32.684 | ++ DIB_MANIFEST_SAVE_DIR=image.d/
2018-03-01 04:38:32.685 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/50-dib-python-version
2018-03-01 04:38:32.687 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/50-dib-python-version
2018-03-01 04:38:32.687 | ++ '[' -z '' ']'
2018-03-01 04:38:32.687 | ++ '[' ubuntu == ubuntu ']'
2018-03-01 04:38:32.687 | ++ '[' xenial == trusty ']'
2018-03-01 04:38:32.687 | ++ '[' -z '' ']'
2018-03-01 04:38:32.687 | ++ DIB_PYTHON_VERSION=3
2018-03-01 04:38:32.687 | ++ export DIB_PYTHON_VERSION
2018-03-01 04:38:32.687 | ++ export DIB_PYTHON=python3
2018-03-01 04:38:32.687 | ++ DIB_PYTHON=python3
2018-03-01 04:38:32.687 | dib-run-parts Sourcing environment file /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/99-cloud-init-datasources.bash
2018-03-01 04:38:32.689 | + source /tmp/dib_build.cuv4VCZM/hooks/root.d/../environment.d/99-cloud-init-datasources.bash
2018-03-01 04:38:32.689 | ++ export DIB_CLOUD_INIT_DATASOURCES=Ec2
2018-03-01 04:38:32.689 | ++ DIB_CLOUD_INIT_DATASOURCES=Ec2
2018-03-01 04:38:32.689 | dib-run-parts Running /tmp/dib_build.cuv4VCZM/hooks/root.d/10-cache-ubuntu-tarball
2018-03-01 04:38:32.694 | Getting /root/.cache/dib/lockfiles/xenial-server-cloudimg-amd64-root.tar.gz.lock: Thu Mar 1 12:38:32 CST 2018
2018-03-01 04:38:32.695 | Fetching Base Image
2018-03-01 04:38:32.714 | * Hostname was NOT found in DNS cache
2018-03-01 04:38:32.714 | % Total % Received % Xferd Average Speed Time Time Time Current
2018-03-01 04:38:32.714 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 91.189.92.141...
2018-03-01 04:38:33.167 | * Trying 2001:67c:1360:8c01::8001...
2018-03-01 04:38:33.167 | * Immediate connect fail for 2001:67c:1360:8c01::8001: Network is unreachable
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to cloud-images.ubuntu.com (91.189.92.141) port 443 (#0)
2018-03-01 04:38:33.222 | * successfully set certificate verify locations:
2018-03-01 04:38:33.222 | * CAfile: none
2018-03-01 04:38:33.222 | CApath: /etc/ssl/certs
2018-03-01 04:38:33.222 | * SSLv3, TLS handshake, Client hello (1):
2018-03-01 04:38:33.222 | } [data not shown]
2018-03-01 04:38:33.485 | * SSLv3, TLS handshake, Server hello (2):
2018-03-01 04:38:33.485 | { [data not shown]
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* SSLv3, TLS handshake, CERT (11):
2018-03-01 04:38:34.507 | { [data not shown]
2018-03-01 04:38:34.508 | * SSLv3, TLS handshake, Server key exchange (12):
2018-03-01 04:38:34.508 | { [data not shown]
2018-03-01 04:38:34.508 | * SSLv3, TLS handshake, Server finished (14):
2018-03-01 04:38:34.508 | { [data not shown]
2018-03-01 04:38:34.509 | * SSLv3, TLS handshake, Client key exchange (16):
2018-03-01 04:38:34.509 | } [data not shown]
2018-03-01 04:38:34.509 | * SSLv3, TLS change cipher, Client hello (1):
2018-03-01 04:38:34.509 | } [data not shown]
2018-03-01 04:38:34.509 | * SSLv3, TLS handshake, Finished (20):
2018-03-01 04:38:34.509 | } [data not shown]
2018-03-01 04:38:34.767 | * SSLv3, TLS change cipher, Client hello (1):
2018-03-01 04:38:34.767 | { [data not shown]
2018-03-01 04:38:34.767 | * SSLv3, TLS handshake, Finished (20):
2018-03-01 04:38:34.767 | { [data not shown]
2018-03-01 04:38:34.767 | * SSL connection using ECDHE-RSA-AES128-GCM-SHA256
2018-03-01 04:38:34.767 | * Server certificate:
2018-03-01 04:38:34.767 | * subject: C=GB; L=London; O=Canonical Group Ltd; OU=IS; CN=cloud-images.ubuntu.com
2018-03-01 04:38:34.767 | * start date: 2017-07-31 00:00:00 GMT
2018-03-01 04:38:34.767 | * expire date: 2018-08-23 12:00:00 GMT
2018-03-01 04:38:34.767 | * subjectAltName: cloud-images.ubuntu.com matched
2018-03-01 04:38:34.767 | * issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
2018-03-01 04:38:34.767 | * SSL certificate verify ok.
2018-03-01 04:38:34.768 | > GET /xenial/current/SHA256SUMS HTTP/1.1
2018-03-01 04:38:34.768 | > User-Agent: curl/7.35.0
2018-03-01 04:38:34.768 | > Host: cloud-images.ubuntu.com
2018-03-01 04:38:34.768 | > Accept: /
2018-03-01 04:38:34.768 | > If-Modified-Since: Tue, 27 Feb 2018 07:45:32 GMT
2018-03-01 04:38:34.768 | >
2018-03-01 04:38:35.024 | < HTTP/1.1 200 OK
2018-03-01 04:38:35.024 | < Date: Thu, 01 Mar 2018 04:38:34 GMT
2018-03-01 04:38:35.024 | * Server Apache is not blacklisted
2018-03-01 04:38:35.024 | < Server: Apache
2018-03-01 04:38:35.024 | < Last-Modified: Wed, 28 Feb 2018 19:13:23 GMT
2018-03-01 04:38:35.024 | < ETag: "1492-5664a89703ac0"
2018-03-01 04:38:35.024 | < Accept-Ranges: bytes
2018-03-01 04:38:35.024 | < Content-Length: 5266
2018-03-01 04:38:35.024 | <
2018-03-01 04:38:35.279 | { [data not shown]
100 5266 100 5266 0 0 2052 0 0:00:02 0:00:02 --:--:-- 2052
2018-03-01 04:38:35.279 | * Connection #0 to host cloud-images.ubuntu.com left intact
2018-03-01 04:38:35.281 | Server copy has changed. Using server version of https://cloud-images.ubuntu.com/xenial/current/SHA256SUMS
2018-03-01 04:38:35.303 | * Hostname was NOT found in DNS cache
2018-03-01 04:38:35.303 | % Total % Received % Xferd Average Speed Time Time Time Current
2018-03-01 04:38:35.303 | Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 91.189.92.141...
2018-03-01 04:38:35.756 | * Trying 2001:67c:1360:8c01::8001...
2018-03-01 04:38:35.756 | * Immediate connect fail for 2001:67c:1360:8c01::8001: Network is unreachable
2018-03-01 04:38:35.808 | * Connected to cloud-images.ubuntu.com (91.189.92.141) port 80 (#0)
2018-03-01 04:38:35.808 | > GET /xenial/current/xenial-server-cloudimg-amd64-root.tar.gz HTTP/1.1
2018-03-01 04:38:35.808 | > User-Agent: curl/7.35.0
2018-03-01 04:38:35.808 | > Host: cloud-images.ubuntu.com
2018-03-01 04:38:35.808 | > Accept: /
2018-03-01 04:38:35.808 | > If-Modified-Since: Tue, 13 Feb 2018 03:24:38 GMT
2018-03-01 04:38:35.808 | >
2018-03-01 04:38:36.064 | < HTTP/1.1 200 OK
2018-03-01 04:38:36.064 | < Date: Thu, 01 Mar 2018 04:38:35 GMT
2018-03-01 04:38:36.064 | * Server Apache is not blacklisted
2018-03-01 04:38:36.064 | < Server: Apache
2018-03-01 04:38:36.064 | < Last-Modified: Wed, 28 Feb 2018 15:48:30 GMT
2018-03-01 04:38:36.064 | < ETag: "c1f6071-56647acb7ef80"
2018-03-01 04:38:36.064 | < Accept-Ranges: bytes
2018-03-01 04:38:36.064 | < Content-Length: 203382897
2018-03-01 04:38:36.064 | < Content-Type: application/x-gzip
2018-03-01 04:38:36.064 | <
2018-03-01 04:38:36.064 | { [data not shown]

每次都是卡在这里不继续下去了，我看过程中有好多data not shown，是不是网络被墙了，下载不了镜像，还是别的原因。我是第一次用这个工具，有好多不明白的地方，请多指教，谢谢。
xionchen commented on Thu Mar 01 2018

@yeweimian21
出现了很多Hostname was NOT found in DNS cache错误，
建议检查DNS，

可以把DIB_DEBUG_TRACE打开，看详细日志。
不行的话可以下载好，用离线模式运行，DIB_OFFLINE 或者指定本地缓存镜像
参考此处源码
https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/ubuntu/root.d/10-cache-ubuntu-tarball
yeweimian21 commented on Fri Mar 02 2018

您好，大神，我刚刚接触这个工具，平时可能有一些问题需要问您，方便加个QQ吗，我的QQ：1030060483，谢谢。同时祝您元宵节快乐🎉🎉
xionchen commented on Tue Mar 06 2018

@yeweimian21 现在基本不用qq了。。。有问题你可以直接留言或者发邮件。
能解答的基本都会解答
Sirius21c commented on Mon Jun 25 2018
你好，有两个问题想请教一下：
1. 我是在调研 openstack_octavia 的时候遇到 dib 的问题，跑 devstack 总是卡在了制作镜像处，为避免网络问题，已经换了国内的 ubuntu 源（阿里的），但还是出错：
  2018-06-25 02:51:45.639 | dib-run-parts Running /tmp/dib_build.shLTNvjU/hooks/root.d/50-build-with-http-cache
  2018-06-25 02:51:45.642 | dib-run-parts 50-build-with-http-cache completed
  2018-06-25 02:51:45.642 | dib-run-parts Running /tmp/dib_build.shLTNvjU/hooks/root.d/50-shared-apt-cache
  2018-06-25 02:51:45.653 | dib-run-parts 50-shared-apt-cache completed
  2018-06-25 02:51:45.653 | dib-run-parts Running /tmp/dib_build.shLTNvjU/hooks/root.d/50-trim-dpkg
  2018-06-25 02:51:45.669 | dib-run-parts 50-trim-dpkg completed
  2018-06-25 02:51:45.669 | dib-run-parts Running /tmp/dib_build.shLTNvjU/hooks/root.d/75-ubuntu-minimal-baseinstall
  2018-06-25 02:51:45.814 | Hit:1 http://mirrors.aliyun.com/ubuntu xenial InRelease
  2018-06-25 02:51:45.815 | Couldn't create tempfiles for splitting up /var/lib/apt/lists/mirrors.aliyun.com_ubuntu_dists_xenial_InReleaseErr:1 http://mirrors.aliyun.com/ubuntu xen
  ial InRelease
  2018-06-25 02:51:45.815 | Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:45.877 | Get:2 http://mirrors.aliyun.com/ubuntu xenial-updates InRelease [109 kB]
  2018-06-25 02:51:45.983 | Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/mirrors.aliyun.com_ubuntu_dists_xenial-updates_InReleaseIgn:2 http://mirrors.aliyu
  n.com/ubuntu xenial-updates InRelease
  2018-06-25 02:51:46.015 | Get:3 http://mirrors.aliyun.com/ubuntu xenial-backports InRelease [107 kB]
  2018-06-25 02:51:46.092 | Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/mirrors.aliyun.com_ubuntu_dists_xenial-backports_InReleaseIgn:3 http://mirrors.ali
  yun.com/ubuntu xenial-backports InRelease
  2018-06-25 02:51:46.124 | Get:4 http://mirrors.aliyun.com/ubuntu xenial-security InRelease [107 kB]
  2018-06-25 02:51:46.635 | Couldn't create tempfiles for splitting up /var/lib/apt/lists/partial/mirrors.aliyun.com_ubuntu_dists_xenial-security_InReleaseGet:5 http://mirrors.aliy
  un.com/ubuntu xenial-updates/main amd64 Packages [796 kB]
  2018-06-25 02:51:46.638 | Ign:4 http://mirrors.aliyun.com/ubuntu xenial-security InRelease
  2018-06-25 02:51:47.524 | Get:6 http://mirrors.aliyun.com/ubuntu xenial-updates/universe amd64 Packages [636 kB]
  2018-06-25 02:51:47.694 | Get:7 http://mirrors.aliyun.com/ubuntu xenial-backports/main amd64 Packages [4844 B]
  2018-06-25 02:51:47.694 | Get:8 http://mirrors.aliyun.com/ubuntu xenial-backports/universe amd64 Packages [7400 B]
  2018-06-25 02:51:47.706 | Get:9 http://mirrors.aliyun.com/ubuntu xenial-security/main amd64 Packages [511 kB]
  2018-06-25 02:51:48.359 | Get:10 http://mirrors.aliyun.com/ubuntu xenial-security/universe amd64 Packages [355 kB]
  2018-06-25 02:51:48.876 | Fetched 2632 kB in 3s (832 kB/s)
  2018-06-25 02:51:49.605 | Reading package lists...
  2018-06-25 02:51:49.632 | W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://mirro
  rs.aliyun.com/ubuntu xenial InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:49.632 | W: GPG error: http://mirrors.aliyun.com/ubuntu xenial-updates InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:49.632 | W: The repository 'http://mirrors.aliyun.com/ubuntu xenial-updates InRelease' is not signed.
  2018-06-25 02:51:49.632 | W: GPG error: http://mirrors.aliyun.com/ubuntu xenial-backports InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:49.632 | W: The repository 'http://mirrors.aliyun.com/ubuntu xenial-backports InRelease' is not signed.
  2018-06-25 02:51:49.632 | W: GPG error: http://mirrors.aliyun.com/ubuntu xenial-security InRelease: Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:49.632 | W: The repository 'http://mirrors.aliyun.com/ubuntu xenial-security InRelease' is not signed.
  2018-06-25 02:51:49.632 | W: Failed to fetch http://mirrors.aliyun.com/ubuntu/dists/xenial/InRelease Could not execute 'apt-key' to verify signature (is gnupg installed?)
  2018-06-25 02:51:49.632 | W: Some index files failed to download. They have been ignored, or old ones used instead.
  2018-06-25 02:51:50.345 | Reading package lists...
  2018-06-25 02:51:50.504 | Building dependency tree...
  2018-06-25 02:51:50.576 | Calculating upgrade...
  2018-06-25 02:51:50.632 | The following packages will be upgraded:
  2018-06-25 02:51:50.632 | apt base-files bash bsdutils coreutils dpkg gcc-5-base gnupg gpgv grep init
  2018-06-25 02:51:50.632 | init-system-helpers libapparmor1 libapt-pkg5.0 libaudit-common libaudit1
  2018-06-25 02:51:50.634 | libblkid1 libc-bin libc6 libcryptsetup4 libdb5.3 libfdisk1 libgcrypt20
  2018-06-25 02:51:50.634 | libkmod2 libmount1 libpam-modules libpam-modules-bin libpam-runtime libpam0g
  2018-06-25 02:51:50.634 | libprocps4 libseccomp2 libsmartcols1 libstdc++6 libsystemd0 libudev1
  2018-06-25 02:51:50.634 | libuuid1 locales login lsb-base makedev mount multiarch-support passwd
  2018-06-25 02:51:50.634 | perl-base procps sensible-utils systemd systemd-sysv tar tzdata util-linux
  2018-06-25 02:51:50.634 | zlib1g
  2018-06-25 02:51:50.659 | 52 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  2018-06-25 02:51:50.659 | Need to get 23.5 MB of archives.
  2018-06-25 02:51:50.659 | After this operation, 161 kB of additional disk space will be used.
  2018-06-25 02:51:50.659 | WARNING: The following packages cannot be authenticated!
  2018-06-25 02:51:50.659 | base-files bash bsdutils coreutils dpkg grep perl-base init-system-helpers
  2018-06-25 02:51:50.659 | init login libsystemd0 systemd libc6 libapparmor1 libaudit-common libaudit1
  2018-06-25 02:51:50.659 | libpam0g libpam-modules-bin libpam-modules passwd libuuid1 libblkid1
  2018-06-25 02:51:50.659 | libgcrypt20 libcryptsetup4 libkmod2 libmount1 libseccomp2 lsb-base
  2018-06-25 02:51:50.659 | util-linux mount tar locales libc-bin gcc-5-base libstdc++6 zlib1g
  2018-06-25 02:51:50.659 | libapt-pkg5.0 gpgv gnupg apt systemd-sysv libdb5.3 libfdisk1 libpam-runtime
  2018-06-25 02:51:50.659 | libsmartcols1 libudev1 multiarch-support sensible-utils libprocps4 makedev
  2018-06-25 02:51:50.659 | procps tzdata
  2018-06-25 02:51:50.662 | E: There were unauthenticated packages and -y was used without --allow-unauthenticated
  2018-06-25 02:51:51.261 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
  2018-06-25 02:51:51.305 | Unmount /tmp/dib_build.shLTNvjU/mnt/var/cache/apt/archives
  2018-06-25 02:51:51.321 | Unmount /tmp/dib_build.shLTNvjU/mnt/tmp/pip
  2018-06-25 02:51:51.522 | +/opt/stack/octavia/devstack/plugin.sh:build_octavia_worker_image:1 exit_trap
  2018-06-25 02:51:51.527 | +./stack.sh:exit_trap:510 local r=1
  2018-06-25 02:51:51.533 | ++./stack.sh:exit_trap:511 jobs -p
  2018-06-25 02:51:51.538 | +./stack.sh:exit_trap:511 jobs=
  2018-06-25 02:51:51.543 | +./stack.sh:exit_trap:514 [[ -n '' ]]
  2018-06-25 02:51:51.548 | +./stack.sh:exit_trap:520 '[' -f /tmp/tmp.WQZcukSekU ']'
  2018-06-25 02:51:51.553 | +./stack.sh:exit_trap:521 rm /tmp/tmp.WQZcukSekU
  2018-06-25 02:51:51.559 | +./stack.sh:exit_trap:525 kill_spinner
  2018-06-25 02:51:51.565 | +./stack.sh:kill_spinner:424 '[' '!' -z '' ']'
  2018-06-25 02:51:51.569 | +./stack.sh:exit_trap:527 [[ 1 -ne 0 ]]
  2018-06-25 02:51:51.574 | +./stack.sh:exit_trap:528 echo 'Error on exit'
  2018-06-25 02:51:51.574 | Error on exit
  2018-06-25 02:51:51.579 | +./stack.sh:exit_trap:530 type -p generate-subunit
  2018-06-25 02:51:51.584 | +./stack.sh:exit_trap:531 generate-subunit 1529894031 1080 fail
  2018-06-25 02:51:52.004 | +./stack.sh:exit_trap:533 [[ -z /opt/stack/logs ]]
  2018-06-25 02:51:52.009 | +./stack.sh:exit_trap:536 /opt/stack/devstack/tools/worlddump.py -d /opt/stack/logs
  2018-06-25 02:51:52.991 | +./stack.sh:exit_trap:545 exit 1
2. 把 dib 的代码单独 clone 下来，按照blog直接使用命令disk-image-create vm ubuntu-minimal也出现问题，感觉是 ubuntu-minimal 依赖的 dpkg ebootstrap 都没有成功加载，但日志显示已经copy脚本
  2018-06-25 07:32:52.168 | Building elements: base vm ubuntu-minimal block-device-mbr
  2018-06-25 07:32:52.242 | Expanded element dependencies to: dib-python install-types install-bin block-device-mbr ubuntu-minimal ubuntu-common vm sysprep base pkg-map debootstrap dib-init-system bootloader manifests package-installs dpkg
  ......
  2018-06-25 07:32:53.731 | dib-run-parts Running /tmp/dib_build.vPtWj9sd/hooks/root.d/08-debootstrap
  2018-06-25 07:32:53.734 | /tmp/dib_build.vPtWj9sd/hooks/root.d/08-debootstrap: line 53: dpkg: command not found
  2018-06-25 07:32:53.734 | /tmp/dib_build.vPtWj9sd/hooks/root.d/08-debootstrap: line 53: [: !=: unary operator expected
  2018-06-25 07:32:53.741 | sh: debootstrap: command not found
  2018-06-25 07:32:53.999 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
非常感谢！！！
xionchen commented on Thu Aug 02 2018

@Sirius21c 应该是缺少二进制依赖导致的，
可以clone dib的仓库
然后使用bindep安装二进制依赖。
适应diskimage-builder中的bindep.txt这个文件作为bindep的安装文件.
HF-CrisXu commented on Fri Nov 08 2019

2019-11-07 12:46:37.801 | + sudo udevadm settle
2019-11-07 12:46:37.807 | + sudo losetup -f
2019-11-07 12:46:37.811 | /dev/loop5
2019-11-07 12:46:37.812 | ++ sudo kpartx -av /tmp/tmp.JFLzt8mOgb/3cuH0l.raw
2019-11-07 12:46:37.812 | ++ awk '/loop[0-9]+p1/ {print $3}'
2019-11-07 12:46:37.818 | device-mapper: resume ioctl on loop5p1 failed: Invalid argument
2019-11-07 12:46:37.837 | create/reload failed on loop5p1
2019-11-07 12:46:37.838 | + ROOT_LOOPDEV=
2019-11-07 12:46:37.838 | ++ rm -r /tmp/tmp.JFLzt8mOgb
2019-11-07 12:46:38.101 | + /home/images/new-images/dib/diskimage-builder/diskimage_builder/lib/common-functions:run_d:1 : trap_cleanup
2019-11-07 12:46:38.103 | + /home/images/new-images/dib/diskimage-builder/diskimage_builder/lib/img-functions:trap_cleanup:36exitval=1

大神帮我看下是啥问题，可以吗

RL Course by David Silver

"RL基本的概念理解"

RL Course by David Silver

两本书推荐

什么是强化学习

RL的特点

强化学习

example of reward

Agent and Environment

History and State

State

例子

full Observable

Partal observability

RL agent的组成

policy

Value function

model

例子

RL分类

Learning and Planning

balance Exploration and Exploitation

Prediction and Control

总结

Lecture 2: Markov Decision Process

Markov Processes

Introduction

Markov Property

Markov Reward Process

Value Function

Example

MDP Markov Decision Process

例子

Example Optimal value fn

CATALOG

FEATURED TAGS

FRIENDS