When you develop a software system, you can choose any solution between two extreme approaches.
- Build and maintain all your dependencies.
- Rely on external repositories and build only your specific packages.
Having chosen the first approach, you can be sure that your users will use your great tuned packages of carefully chosen versions that are doomed to work properly. And when you see a problem in a dependency, you freely patch it and… congratulations, now you are a happy maintainer of a zoo of numerous packages containing software written in several languages!
The second way is clear: you build a dozen of your own packages and publish a relatively small repository. And when your third-party dependencies become unavailable, it will be a user’s problem.
Developing Altai, we started from the first solution: a user installs basic RHEL/CentOS and just adds our repository. Nowadays, we are moving to the second approach: continue to provide all required packages but stop rebuilding the stuff that’s available from EPEL.
So, the question is how much can we trust EPEL? Altai is a set of several specific packages, patched OpenStack, and their dependencies. EPEL contains OpenStack, so, it was quite logic to expect at least OpenStack’s dependencies in EPEL repositories, but…
The first surprise was all these cool package names like python-paste-deploy1.5 or python-webob1.0 where the version was a part and parcel of package name. Well, now my package will depend on python-paste-deploy1.5 instead of python-paste-deploy or python-paste-deploy = 1.5 – who cares?
However, there is nothing better than to see what’s inside these packages:
drwxr-xr-x 2 root root 0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg drwxr-xr-x 2 root root 0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO -rw-r--r-- 1 root root 718 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/PKG-INFO -rw-r--r-- 1 root root 3974 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/SOURCES.txt -rw-r--r-- 1 root root 1 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/dependency_links.txt -rw-r--r-- 1 root root 1 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/not-zip-safe -rw-r--r-- 1 root root 7 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/top_level.txt drwxr-xr-x 2 root root 0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes -rw-r--r-- 1 root root 5468 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.py -rw-r--r-- 2 root root 5733 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.pyc -rw-r--r-- 2 root root 5733 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.pyo ...
What? The python code resides inside its egg?! Of course, it cannot be imported out of the box:
# yum install python-routes1.12 ... # python -c 'import routes' Traceback (most recent call last): File "", line 1, in ImportError: No module named routes
You have to process the egg and add /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg directory to your sys.path, otherwise python won’t be able to find the module.
Here I have to confess that I believe that there should be only one package management system in a single installation. Do not use easy_install if you have in a solid system like emerge, rpm, or dpkg: find (or write) a ebuild, spec, or a debian package. If you just want to have a look, place the package to /usr/local, /opt, or to your home directory and delete it in two hours! Don’t kindle a package management hell of emerge, maven, ivy, easy_install, gem, and whatever else!
Ok, let’s see how EPEL deals with these incoherence packaging:
# yum install openstack-glance -y Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package openstack-glance.noarch 0:2012.1.1-1.el6 will be installed --> Processing Dependency: python-glance = 2012.1.1-1.el6 for package: openstack-glance-2012.1.1-1.el6.noarch --> Processing Dependency: openstack-utils for package: openstack-glance-2012.1.1-1.el6.noarch --> Running transaction check ---> Package openstack-utils.noarch 0:2012.1-2.el6 will be installed ---> Package python-glance.noarch 0:2012.1.1-1.el6 will be installed --> Processing Dependency: pyxattr for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-webob1.0 for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-sqlalchemy0.7 for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-paste-deploy1.5 for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-migrate for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-iso8601 for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-httplib2 for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-eventlet for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: python-crypto for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: pysendfile for package: python-glance-2012.1.1-1.el6.noarch --> Processing Dependency: MySQL-python for package: python-glance-2012.1.1-1.el6.noarch --> Running transaction check ---> Package MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6 will be installed --> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64 --> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64 ---> Package pysendfile.x86_64 0:2.0.0-3.el6 will be installed ---> Package python-crypto.x86_64 0:2.0.1-22.el6 will be installed ---> Package python-eventlet.noarch 0:0.9.17-1.el6 will be installed --> Processing Dependency: python-greenlet for package: python-eventlet-0.9.17-1.el6.noarch ---> Package python-httplib2.noarch 0:0.7.4-6.el6 will be installed ---> Package python-iso8601.noarch 0:0.1.4-2.el6 will be installed ---> Package python-migrate.noarch 0:0.6-6.el6 will be installed --> Processing Dependency: python-sqlalchemy for package: python-migrate-0.6-6.el6.noarch --> Processing Dependency: python-tempita for package: python-migrate-0.6-6.el6.noarch --> Processing Dependency: python-decorator for package: python-migrate-0.6-6.el6.noarch ---> Package python-paste-deploy1.5.noarch 0:1.5.0-5.el6 will be installed --> Processing Dependency: python-paste for package: python-paste-deploy1.5-1.5.0-5.el6.noarch ---> Package python-sqlalchemy0.7.x86_64 0:0.7.8-1.el6 will be installed ---> Package python-webob1.0.noarch 0:1.0.8-3.el6 will be installed ---> Package pyxattr.x86_64 0:0.5.0-1.el6 will be installed --> Running transaction check ---> Package mysql-libs.x86_64 0:5.1.61-4.el6 will be installed ---> Package python-decorator.noarch 0:3.0.1-3.1.el6 will be installed ---> Package python-greenlet.x86_64 0:0.3.1-6.el6 will be installed ---> Package python-paste.noarch 0:1.7.4-1.el6 will be installed --> Processing Dependency: pyOpenSSL for package: python-paste-1.7.4-1.el6.noarch ---> Package python-sqlalchemy.noarch 0:0.5.5-3.el6_2 will be installed ---> Package python-tempita.noarch 0:0.4-2.el6 will be installed --> Running transaction check ---> Package pyOpenSSL.x86_64 0:0.10-2.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: openstack-glance noarch 2012.1.1-1.el6 epel 47 k Installing for dependencies: MySQL-python x86_64 1.2.3-0.3.c1.1.el6 base 86 k mysql-libs x86_64 5.1.61-4.el6 base 1.2 M openstack-utils noarch 2012.1-2.el6 epel 16 k pyOpenSSL x86_64 0.10-2.el6 base 212 k pysendfile x86_64 2.0.0-3.el6 epel 9.3 k python-crypto x86_64 2.0.1-22.el6 base 159 k python-decorator noarch 3.0.1-3.1.el6 base 14 k python-eventlet noarch 0.9.17-1.el6 epel 262 k python-glance noarch 2012.1.1-1.el6 epel 494 k python-greenlet x86_64 0.3.1-6.el6 epel 21 k python-httplib2 noarch 0.7.4-6.el6 epel 69 k python-iso8601 noarch 0.1.4-2.el6 epel 12 k python-migrate noarch 0.6-6.el6 epel 208 k python-paste noarch 1.7.4-1.el6 base 757 k python-paste-deploy1.5 noarch 1.5.0-5.el6 epel 47 k python-sqlalchemy noarch 0.5.5-3.el6_2 base 1.3 M python-sqlalchemy0.7 x86_64 0.7.8-1.el6 epel 2.1 M python-tempita noarch 0.4-2.el6 base 38 k python-webob1.0 noarch 1.0.8-3.el6 epel 189 k pyxattr x86_64 0.5.0-1.el6 epel 24 k ...
Good job! We have python-sqlalchemy0.7 for python-glance-2012.1.1-1.el6.noarch and python-sqlalchemy (version 0.5.5; from base CentOS repo) for python-migrate-0.6-6.el6.noarch. You must guess that python-migrate is provided by EPEL but depends on the old sqlalchemy package!
Our sqlalchemies do not conflict ’cause one of them resides in its egg. Guess now which one is really imported and used by glance! (Answer: version 0.7 because there is no evil in EPEL.)
Fighting for peace in the whole world, people from EPEL patched OpenStack sources. This is a patch for glance:
$ cat openstack-glance-newdeps.patch
Delve into pkg_resources a little to get it to modify sys.path,
so that our parallel installed egg takes precedence over the
system default module versions.
diff -up glance-2011.3/glance/__init__.py.newdeps glance-2011.3/glance/__init__.py
--- glance-2011.3/glance/__init__.py.newdeps 2012-01-06 17:22:36.000000000 +0000
+++ glance-2011.3/glance/__init__.py 2012-01-06 17:25:01.019063547 +0000
@@ -18,3 +18,31 @@
import gettext
gettext.install('glance', unicode=1)
+
+import sys
+import pkg_resources
+
+# If there is a conflicting non egg module,
+# i.e. an older standard system module installed,
+# then replace it with this requirement
+def replace_dist(requirement):
+ try:
+ return pkg_resources.require(requirement)
+ except pkg_resources.VersionConflict:
+ e = sys.exc_info()[1]
+ dist=e.args[0]
+ req=e.args[1]
+ if dist.key == req.key and not dist.location.endswith('.egg'):
+ del pkg_resources.working_set.by_key[dist.key]
+ # We assume there is no need to adjust sys.path
+ # and the associated pkg_resources.working_set.entries
+ return pkg_resources.require(requirement)
+
+replace_dist("SQLALchemy >= 0.6.3")
+replace_dist("WebOb >= 1.0")
+replace_dist("Routes >= 1.12.3")
+
+replace_dist("PasteDeploy >= 1.5.0")
+# This hack is needed because replace_dist() results in
+# the standard paste module path being at the start of __path__.
+# TODO: See can we get pkg_resources to do the right thing directly
+import paste
+paste.__path__.insert(0, paste.__path__.pop(-1)
It’s insane to mimic EPEL making several “obvious” patches and installing two versions of sqlalchemy. So, that’s wise to maintain several own packages that duplicate the EPEL’s ones but do not need these egg tricks.
Reblogged this on OpenStack Blog.
Pingback: What’s New in Altai 1.0.2 from Maintainer’s Point of View | OpenStack Blog·