Chybeta

pip-pop 源码阅读

pip-pop源码阅读

项目地址

https://github.com/heroku-python/pip-pop

按照commit记录来阅读。

lawyer up

commit记录: a84bc7439770063e457760a18119c10e5d802d3e

添加了LICENSE文件,采用MIT License

dummy dir

commit记录: 636935f9394165c1d55c0e0d878cea60428a434e

创建了 pip_pop文件夹,在其中创建空文件__init__.py。 此时项目结构如下:

1
2
3
4
5
6
.
├── LICENSE
└── pip_pop
└── __init__.py
1 directory, 2 files

READ IT

commit记录: ebdda7f8897403e9b77a2fa7023b2f4f8df1ecaa

项目结构如下:

1
2
3
4
5
6
├── LICENSE
├── README.rst
└── pip_pop
└── __init__.py
1 directory, 3 files

增加了README.rst文件。用于说明该项目的用处,计划中实现的功能,未来可能实现的功能。

1
2
3
4
5
6
7
8
pip-pop: tools for managing requirements files
==============================================
Planned Commands
----------------
Possible Future Commands
------------------------

docopt

commit记录: f0e51cc56f55c4615e29b7a12264b20dbe12db66

项目结构如下:

1
2
3
4
5
6
7
8
.
├── LICENSE
├── README.rst
├── pip_pop
│ └── __init__.py
└── requirements.txt
1 directory, 4 files

增加了requirements.txt文件。

note about blacklisting plans

commit记录: bf54913eaa70f9f505c414a7be328ff15040f37f

项目结构如下:

1
2
3
4
5
6
7
8
.
├── LICENSE
├── README.rst
├── pip_pop
│ └── __init__.py
└── requirements.txt
1 directory, 4 files

修改READEME.rst文件。

Update READEME.rst

commit记录: 2b444bc846071148dedf6773555e8b33f895765c

项目结构如下:

1
2
3
4
5
6
7
8
.
├── LICENSE
├── README.rst
├── pip_pop
│ └── __init__.py
└── requirements.txt
1 directory, 4 files

修改README.rst文件

exes

commit记录: fd65e4d148939f1c7405370e1f342f1fa1b3ea14

项目结构如下:

1
2
3
4
5
6
7
8
9
10
11
12
.
├── LICENSE
├── README.rst
├── bin
│ ├── pip-diff
│ └── pip-flatten
├── pip_pop
│ └── __init__.py
├── requirements.txt
└── setup.py
2 directories, 7 files

新增bin/pip-diffbin/pip-flattensetup.py

bin/pip-diffbin/pip-flatten均是空文件。

setup.py用于python库打包。代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
"""
pip-pop manages your requirements files.
"""
import sys
from setuptools import setup
setup(
name='pip-pop',
version='0.0.0',
url='https://github.com/kennethreitz/pip-pop',
license='MIT',
author='Kenneth Reitz',
author_email='me@kennethreitz.org',
description=__doc__.strip('\n'),
#packages=[],
scripts=['bin/pip-diff', 'bin/pip-flatten'],
#include_package_data=True,
zip_safe=False,
platforms='any',
install_requires=['docopt'],
classifiers=[
# As from https://pypi.python.org/pypi?%3Aaction=list_classifiers
#'Development Status :: 1 - Planning',
#'Development Status :: 2 - Pre-Alpha',
#'Development Status :: 3 - Alpha',
'Development Status :: 4 - Beta',
#'Development Status :: 5 - Production/Stable',
#'Development Status :: 6 - Mature',
#'Development Status :: 7 - Inactive',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
#'Programming Language :: Python :: 2.3',
#'Programming Language :: Python :: 2.4',
#'Programming Language :: Python :: 2.5',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
#'Programming Language :: Python :: 3',
#'Programming Language :: Python :: 3.0',
#'Programming Language :: Python :: 3.1',
#'Programming Language :: Python :: 3.2',
#'Programming Language :: Python :: 3.3',
'Intended Audience :: Developers',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: BSD License',
'Operating System :: OS Independent',
'Topic :: System :: Systems Administration',
]
)

setuptools导入setup函数,其中参数的含义如下:

参数 含义
name 包名字 pip-pop
version 包版本 0.0.0
url 程序官网地址 https://github.com/kennethreitz/pip-pop
license 授权信息 MIT
author 程序作者 Kenneth Reitz
author_email 作者邮箱 me@kennethreitz.org
description 程序简单描述 __doc__.strip(‘\n’)
scripts 指定可执行脚本,安装时脚本会被添加到系统PATH中 [‘bin/pip-diff’, ‘bin/pip-flatten’]
zip_safe 不压缩包,以目录形式安装 False
platforms 程序适合的平台 ‘any’
install_requires 安装时需要安装的依赖包 [‘docopt’]
classifiers 分类信息 详细见下

diffing works!

commit记录: d58196205cea3a4650d68443dd90132bbd4b2b4e

项目结构如下:

1
2
3
4
5
6
7
8
9
10
11
12
.
├── LICENSE
├── README.rst
├── bin
│ ├── pip-diff
│ └── pip-flatten
├── pip_pop
│ └── __init__.py
├── requirements.txt
└── setup.py
2 directories, 7 files

更改了bin/pip-diff文件。代码整体的格式如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Usage:
pip-diff (--fresh | --stale) <reqfile1> <reqfile2>
pip-diff (-h | --help)
Options:
-h --help Show this screen.
--fresh List newly added packages.
--stale List removed packages.
"""
import os
from docopt import docopt
from pkg_resources import parse_requirements
# TODO: ignore lines
IGNORABLE_LINES = '#', '-r'
VERSION_OPERATORS = ['==', '>=', '<=', '>', '<', ',']
def split(s):...
class Requirements(object):...
def diff(r1, r2, include_fresh=False, include_stale=False):...
def main():...
if __name__ == '__main__':
main()

第一行#!/usr/bin/env python,用于为脚本语言指定解释器,这样可以直接./*.py的方式执行,不要使用#!/usr/bin/python,因为python可能不是安装在默认的环境。

第二行# -*- coding: utf-8 -*-用于指定编码为 utf-8,这样可以在py文件中写中文,方便写注释和消息。

最下面的if __name__ == '__main__':的意思是,当该py文件被直接运行时,if __name__ == '__main__':之下的main()将被调用执行,当该py文件被以模块的形式导入时,if __name__ == '__main__':不被运行。

main()函数源代码如下:

1
2
3
4
5
6
7
8
9
10
11
def main():
args = docopt(__doc__, version='pip-diff')
kwargs = {
'r1': args['<reqfile1>'],
'r2': args['<reqfile2>'],
'include_fresh': args['--fresh'],
'include_stale': args['--stale']
}
diff(**kwargs)

通过args = docopt(__doc__, version='pip-diff') 来获取对应的命令行参数,参数要求见程序开头的那一段注释:

1
2
3
4
5
6
7
8
Usage:
pip-diff (--fresh | --stale) <reqfile1> <reqfile2>
pip-diff (-h | --help)
Options:
-h --help Show this screen.
--fresh List newly added packages.
--stale List removed packages.

args解析完命令行参数后,会返回一个Dict类型。然后通过kwargs解析出对应的变量。。--fresh--stale的作用是Generates a diff between two given requirements files. Lists either stale or fresh packages.。以命令行参数--fresh D:\temp\req1 D:\temp\req2为例

然后程序进入diff(**kwargs), diff函数:

1
2
3
4
5
6
7
8
9
10
11
12
def diff(r1, r2, include_fresh=False, include_stale=False):
# assert that r1 and r2 are files.
try:
r1 = Requirements(r1)
r2 = Requirements(r2)
except ValueError:
print 'There was a problem loading the given requirements files.'
exit(os.EX_NOINPUT)
results = r1.diff(r2)
print results

Requirements对象定义如下,其中的diff函数先暂时省略:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
class Requirements(object):
"""docstring for Requirements"""
def __init__(self, reqfile=None):
super(Requirements, self).__init__()
self.path = reqfile
self.requirements = []
if reqfile:
self.load(reqfile)
def __repr__(self):
return '<Requirements \'{}\'>'.format(self.path)
def load(self, reqfile):
if not os.path.exists(reqfile):
raise ValueError('The given requirements file does not exist.')
with open(reqfile) as f:
data = []
for line in f:
line = line.strip()
# Skip lines that start with any comment/control charecters.
if not any([line.startswith(p) for p in IGNORABLE_LINES]):
data.append(line)
for requirement in parse_requirements(data):
self.requirements.append(requirement)
# assert that the given file exists
# parse the file
# insert those entries into self.declarations
pass
def diff(self, requirements, ignore_versions=False):
。。。

Requirements(r1)为例,传入的参数为D:\\temp\\req1,在__init__中进入self.load(reqfile),首先判断了文件的存在。然后对于文件中的每一行(for line in f:),去除它末尾的换行符(line = line.strip()),然后判断其是否以注释或控制字符开头([line.startswith(p) for p in IGNORABLE_LINES]),若不是则将其加入到data中。之后调用parse_requirements(data)进行解析:

在pass之后,返回给r1

r2对象实例化后,进行results = r1.diff(r2),在class Requirements(object)中定义了diff方法代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
class Requirements(object):
...
def diff(self, requirements, ignore_versions=False):
r1 = self
r2 = requirements
results = {'fresh': [], 'stale': []}
# Generate fresh packages.
other_reqs = (
[r.project_name for r in r1.requirements]
if ignore_versions else r1.requirements
)
for req in r2.requirements:
r = req.project_name if ignore_versions else req
if r not in other_reqs:
results['fresh'].append(req)
# Generate stale packages.
other_reqs = (
[r.project_name for r in r2.requirements]
if ignore_versions else r2.requirements
)
for req in r1.requirements:
r = req.project_name if ignore_versions else req
if r not in other_reqs:
results['stale'].append(req)
return results

output for pip-diff works!

commit记录: d6ae563831228dd6d7e712d69763663032410391

项目结构如下:

1
2
3
4
5
6
7
8
9
10
11
12
.
├── LICENSE
├── README.rst
├── bin
│ ├── pip-diff
│ └── pip-flatten
├── pip_pop
│ └── __init__.py
├── requirements.txt
└── setup.py
2 directories, 7 files

根据参数的不同fresh或者stale,输出对应的结果。

req1内容如下:

1
2
req1
test1

req2内容如下:

1
2
req2
test2

则运行结果如下:

1
2
3
4
5
6
7
C:\Python27\python.exe D:/Learn/opensource/pip-pop/bin/pip-diff --stale D:\temp\req1 D:\temp\req2
req1
test1
C:\Python27\python.exe D:/Learn/opensource/pip-pop/bin/pip-diff --fresh D:\temp\req1 D:\temp\req2
req2
test2

cleanup

commit记录: 2c2ffe318e5c539fc3bdef4feda97c56c162062a

项目结构及代码部分未做改变。

删除了原 pip-diff 中的一些注释

tuples

commit记录: 58f9ae5f9668a7613f7c0f9f1c43a105b2604891

VERSION_OPERATORSlist改为tuple 。 其余无变化。

remove bunk files

commit记录: d1ff1029ca3d4bd765abe2d4e92b1c2700586702

项目结构变为:

1
2
3
4
5
6
7
│ LICENSE
│ README.rst
│ requirements.txt
│ setup.py
└─bin
pip-diff
pip-flatten

删除了pip-pop/__init__.py空文件

rely on pip

commit记录: d638b182d9302fa541efa48fbf99fa05f42a4565

项目结构未变

利用pip.req来解析req文件

getting simpler and simpler!

commit记录:69d9e22c10734d463bde67c04cc469f0b0bce072

项目结构未变

因为直接利用pip.req来解析req文件,删除无用变量

only check lines that have explicit requirements

commit记录: 0837d1133ee25c645d763f670f6683a20bf30240

只有当requirement.req为真时,才添加到self.requirements中。

附上最新版的pip中的 parse_requirements的代码:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# C:/Python27/Lib/site-packages/pip/req/req_file.py:64
def parse_requirements(filename, finder=None, comes_from=None, options=None,
session=None, constraint=False, wheel_cache=None):
"""Parse a requirements file and yield InstallRequirement instances.
:param filename: Path or url of requirements file.
:param finder: Instance of pip.index.PackageFinder.
:param comes_from: Origin description of requirements.
:param options: cli options.
:param session: Instance of pip.download.PipSession.
:param constraint: If true, parsing a constraint file rather than
requirements file.
:param wheel_cache: Instance of pip.wheel.WheelCache
"""
if session is None:
raise TypeError(
"parse_requirements() missing 1 required keyword argument: "
"'session'"
)
_, content = get_file_content(
filename, comes_from=comes_from, session=session
)
lines_enum = preprocess(content, options)
for line_number, line in lines_enum:
req_iter = process_line(line, filename, line_number, finder,
comes_from, options, session, wheel_cache,
constraint=constraint)
for req in req_iter:
yield req

最后会返回一个迭代器

initial version of pip-grep

commit记录: 3862c2f9a2f72bb962e7ed15416109ee0ec3e5ae

项目结构变为:

1
2
3
4
5
6
7
│ LICENSE
│ README.rst
│ requirements.txt
│ setup.py
└─bin
pip-diff
pip-grep

setup.py中:

pip-flatten变为pip-grep,代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Usage:
pip-grep <reqfile> <package>...
Options:
-h --help Show this screen.
"""
import os
from docopt import docopt
from pip.req import parse_requirements
class Requirements(object):
def __init__(self, reqfile=None):
super(Requirements, self).__init__()
self.path = reqfile
self.requirements = []
if reqfile:
self.load(reqfile)
def __repr__(self):
return '<Requirements \'{}\'>'.format(self.path)
def load(self, reqfile):
if not os.path.exists(reqfile):
raise ValueError('The given requirements file does not exist.')
for requirement in parse_requirements(reqfile):
self.requirements.append(requirement)
def grep(reqfile, packages):
try:
# 读取reqfile文件并解析
r = Requirements(reqfile)
except ValueError:
print 'There was a problem loading the given requirement file.'
exit(os.EX_NOINPUT)
# 对于reuqirement中的每一个
for requirement in r.requirements:
if requirement.req.project_name in packages:
# 如果找到了在 packages中
print 'Package {} found!'.format(requirement.req.project_name)
exit(0)
print 'Not found.'.format(requirement.req.project_name)
exit(1)
def main():
# 获取参数
args = docopt(__doc__, version='pip-grep')
kwargs = {'reqfile': args['<reqfile>'], 'packages': args['<package>']}
# 传入 reqfile package
grep(**kwargs)
if __name__ == '__main__':
main()

updated readme

commit记录:2116d8a7698bf8fece0ad5c32db9ec9f69c97e69

更新readme文档,添加pip-grep的使用说明

fix for pip-grep

commit记录:2116d8a7698bf8fece0ad5c32db9ec9f69c97e69

silent mode for pip-grep

commit记录: 78e3c31b3584bfb263c061317ccc798cfaddf061

增加silent参数选项。作用位置

silence “not found”

commit记录: 94c553879358aff40da2c3d2f536acb184703166

添加silent模式对not found情况的支持

python 3 compatibility

commit纪录:70af45d95fd38e0a93abdbdb400283dcc495a00f

修改了pip-greppip-diff,将其中的print 'xx' 改为print('xx')

Add a dummy finder so parse_requirement does not fail on —arguments

commit记录:2aa545fb3b80d78670d923be4333e85f0abb7309

1
2
3
4
5
6
7
8
9
10
from pip.index import PackageFinder
class Requirements(object):
。。。
finder = PackageFinder([], [])
for requirement in parse_requirements(reqfile, finder=finder):
self.requirements.append(requirement)
。。。

新增加一个finder=finder参数,避免parse_requirements失败。

v0.1.0

commit记录:2dc013300c4b0fb605fa9dd2a3fba5ecc81ac20c

修改setup.py,修改版本号为version='0.1.0'

Add option to print the requirement, if found

commit记录: a3f9a4ba40c02d6bc26318e589ae2db11304203f

修改pip-grep文件。

首先是Usage部分:

1
2
3
4
5
6
7
8
"""Usage:
pip-grep [-sp] <reqfile> <package>...
Options:
-h --help Show this screen.
-s --silent Suppress output.
-p --print-req If found, print the requirement.
"""

-p,在grep找到的情况下,打印出requirement

support for lastest pip

commit记录: 27f35700c7d8affb1fc3b399bd77fe38fb82bba1

修改pip-diff

由于parse_requirements中:

1
2
3
4
5
6
7
8
def parse_requirements(filename, finder=None, comes_from=None, options=None,
session=None, constraint=False, wheel_cache=None):
if session is None:
raise TypeError(
"parse_requirements() missing 1 required keyword argument: "
"'session'"
)

所以添加session参数:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from pip._vendor.requests import session
requests = session()
class Requirements(object):
。。。
def load(self, reqfile):
if not os.path.exists(reqfile):
raise ValueError('The given requirements file does not exist.')
finder = PackageFinder([], [], session=requests)
for requirement in parse_requirements(reqfile, finder=finder):
if requirement.req:
self.requirements.append(requirement.req)

Update pip-grep

commit记录:90eba89335af5aa1285d179aa9ea6aa9725bd712

修改内容同上,增加session参数。

Merge pull request #3 from thenovices/print-line Add option to print the requirement, if found.

commit记录:d572c00cc65a47f8d6e3d9446f8c21fb7aac685f

update from python buildpack

commit记录:097c4a94848897e693bf269150a49129d4019390

修改pip-diffpip-grep的一些细节,增删参数。

exclude in pip-diff

commit记录:047dd63d5dd0a754d3e515bef7aa33d1246a548b

修改pip-diff文件,增加excludes参数选项,用于指定排除,不进行比较的packages包

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Usage:
pip-diff (--fresh | --stale) <reqfile1> <reqfile2> [--exclude <package>...]
pip-diff (-h | --help)
Options:
-h --help Show this screen.
--fresh List newly added packages.
--stale List removed packages.
"""
import os
from docopt import docopt
from pip.req import parse_requirements
from pip.index import PackageFinder
from pip._vendor.requests import session
requests = session()
class Requirements(object):
def diff(self, requirements, ignore_versions=False, excludes=None):
。。。
for req in r2.requirements:
r = req.project_name if ignore_versions else req
if r not in other_reqs and r not in excludes:
results['fresh'].append(req)
。。。
for req in r1.requirements:
r = req.project_name if ignore_versions else req
if r not in other_reqs and r not in excludes:
results['stale'].append(req)
return results
def diff(r1, r2, include_fresh=False, include_stale=False, excludes=None):
。。。
excludes = excludes if len(excludes) else []
。。。
results = r1.diff(r2, ignore_versions=True, excludes=excludes)
。。。
def main():
kwargs = {
。。。
'excludes': args['<package>']
}
if __name__ == '__main__':
main()

README.rst Fix spelling error

commit记录:81587647408ff5adc13cc30a50ff84e36116505d

无他,修改README中的拼写错误

update

commit记录:4f5ebcd253ec299baf0f4cb10c99d06bc52cc91f

修改两个文件pip-diffpip-grep

pip-diff中将project_name改为name。原因是pip版本升级,经过parse_requirements后会是name属性。但在8.1.2版本之前并不存在,因此需要在load时进行检测,增加代码如下:

v0.0.1

commit记录:4dc238c79ca19974eeb434ec4be4285d7747bb38

修改setup.py中的版本号

update setup.py

commit记录:07562561ce6aa9c733a18135cf510fadd794433a

修改setup.py中的一些参数Programming LanguageDevelopment Status

Require pip>=1.5.0

commit记录:99d9f36ad765535946af1fa9fc181d33668ee146

修改setup.py中的install_requires,要求pip版本大于1.5.0

Remove unused wsgiref from requirements.txt

commit记录:47ad229596ade5024d9c4c4190e73972176bc58b

删除requirements.txt中的无用条目

Add a tox config and some very primitive pip-grep and pip-diff tests

commit记录:433e02ec7e294e171557514c55412cc3e06c1e53

项目结构:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
│ .gitignore
│ LICENSE
│ README.rst
│ requirements.txt
│ setup.py
│ tox.ini
├─bin
│ pip-diff
│ pip-grep
└─tests
test-requirements.txt
test-requirements2.txt

修改READEME.rstsetup.pyrequirments.txt,主要是增加了tox的依赖,相关环境的安装。

新增文件tests文件夹及其文件、.gitignoretox.ini

Add Travis config

commit记录:a40d8850701f08c99d66cab2eedf283a0b326731

新增.travis.yml 。修改README.rst文件

Update PyPI classifiers to reflect tested Python version

commit记录:e865cb31f4b43edd5f07aa8d40680d0b1eb08f28

阅读完毕。

微信扫码加入知识星球【漏洞百出】
chybeta WeChat Pay

点击图片放大,扫码知识星球【漏洞百出】

本文标题:pip-pop 源码阅读

文章作者:chybeta

发布时间:2018年10月12日 - 19:10

最后更新:2018年10月12日 - 19:10

原始链接:http://chybeta.github.io/2018/10/12/pip-pop-源码阅读/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。