OpenStack EPEL: the Dependency Purgatory

When you develop a software system, you can choose any solution between two extreme approaches.

  1. Build and maintain all your dependencies.
  2. Rely on external repositories and build only your specific packages.

Having chosen the first approach, you can be sure that your users will use your great tuned packages of carefully chosen versions that are doomed to work properly. And when you see a problem in a dependency, you freely patch it and… congratulations, now you are a happy maintainer of a zoo of numerous packages containing software written in several languages!

The second way is clear: you build a dozen of your own packages and publish a relatively small repository. And when your third-party dependencies become unavailable, it will be a user’s problem.

Developing Altai, we started from the first solution: a user installs basic RHEL/CentOS and just adds our repository. Nowadays, we are moving to the second approach: continue to provide all required packages but stop rebuilding the stuff that’s available from EPEL.

So, the question is how much can we trust EPEL? Altai is a set of several specific packages, patched OpenStack, and their dependencies. EPEL contains OpenStack, so, it was quite logic to expect at least OpenStack’s dependencies in EPEL repositories, but…

One does not simply use EPEL in his project

The first surprise was all these cool package names like python-paste-deploy1.5 or python-webob1.0 where the version was a part and parcel of package name. Well, now my package will depend on python-paste-deploy1.5 instead of python-paste-deploy or python-paste-deploy = 1.5 – who cares?

However, there is nothing better than to see what’s inside these packages:

drwxr-xr-x    2 root    root                        0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg
drwxr-xr-x    2 root    root                        0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO
-rw-r--r--    1 root    root                      718 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/PKG-INFO
-rw-r--r--    1 root    root                     3974 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/SOURCES.txt
-rw-r--r--    1 root    root                        1 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/dependency_links.txt
-rw-r--r--    1 root    root                        1 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/not-zip-safe
-rw-r--r--    1 root    root                        7 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/EGG-INFO/top_level.txt
drwxr-xr-x    2 root    root                        0 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes
-rw-r--r--    1 root    root                     5468 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.py
-rw-r--r--    2 root    root                     5733 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.pyc
-rw-r--r--    2 root    root                     5733 Apr 26 18:22 /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/__init__.pyo
...

What? The python code resides inside its egg?! Of course, it cannot be imported out of the box:

# yum install python-routes1.12
...
# python -c 'import routes'
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named routes

You have to process the egg and add /usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg directory to your sys.path, otherwise python won’t be able to find the module.

Here I have to confess that I believe that there should be only one package management system in a single installation. Do not use easy_install if you have in a solid system like emerge, rpm, or dpkg: find (or write) a ebuild, spec, or a debian package. If you just want to have a look, place the package to /usr/local, /opt, or to your home directory and delete it in two hours! Don’t kindle a package management hell of emerge, maven, ivy, easy_install, gem, and whatever else!

Ok, let’s see how EPEL deals with these incoherence packaging:

# yum install openstack-glance -y
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package openstack-glance.noarch 0:2012.1.1-1.el6 will be installed
--> Processing Dependency: python-glance = 2012.1.1-1.el6 for package: openstack-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: openstack-utils for package: openstack-glance-2012.1.1-1.el6.noarch
--> Running transaction check
---> Package openstack-utils.noarch 0:2012.1-2.el6 will be installed
---> Package python-glance.noarch 0:2012.1.1-1.el6 will be installed
--> Processing Dependency: pyxattr for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-webob1.0 for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-sqlalchemy0.7 for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-paste-deploy1.5 for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-migrate for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-iso8601 for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-httplib2 for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-eventlet for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: python-crypto for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: pysendfile for package: python-glance-2012.1.1-1.el6.noarch
--> Processing Dependency: MySQL-python for package: python-glance-2012.1.1-1.el6.noarch
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.3-0.3.c1.1.el6 will be installed
--> Processing Dependency: libmysqlclient_r.so.16(libmysqlclient_16)(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64
--> Processing Dependency: libmysqlclient_r.so.16()(64bit) for package: MySQL-python-1.2.3-0.3.c1.1.el6.x86_64
---> Package pysendfile.x86_64 0:2.0.0-3.el6 will be installed
---> Package python-crypto.x86_64 0:2.0.1-22.el6 will be installed
---> Package python-eventlet.noarch 0:0.9.17-1.el6 will be installed
--> Processing Dependency: python-greenlet for package: python-eventlet-0.9.17-1.el6.noarch
---> Package python-httplib2.noarch 0:0.7.4-6.el6 will be installed
---> Package python-iso8601.noarch 0:0.1.4-2.el6 will be installed
---> Package python-migrate.noarch 0:0.6-6.el6 will be installed
--> Processing Dependency: python-sqlalchemy for package: python-migrate-0.6-6.el6.noarch
--> Processing Dependency: python-tempita for package: python-migrate-0.6-6.el6.noarch
--> Processing Dependency: python-decorator for package: python-migrate-0.6-6.el6.noarch
---> Package python-paste-deploy1.5.noarch 0:1.5.0-5.el6 will be installed
--> Processing Dependency: python-paste for package: python-paste-deploy1.5-1.5.0-5.el6.noarch
---> Package python-sqlalchemy0.7.x86_64 0:0.7.8-1.el6 will be installed
---> Package python-webob1.0.noarch 0:1.0.8-3.el6 will be installed
---> Package pyxattr.x86_64 0:0.5.0-1.el6 will be installed
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.61-4.el6 will be installed
---> Package python-decorator.noarch 0:3.0.1-3.1.el6 will be installed
---> Package python-greenlet.x86_64 0:0.3.1-6.el6 will be installed
---> Package python-paste.noarch 0:1.7.4-1.el6 will be installed
--> Processing Dependency: pyOpenSSL for package: python-paste-1.7.4-1.el6.noarch
---> Package python-sqlalchemy.noarch 0:0.5.5-3.el6_2 will be installed
---> Package python-tempita.noarch 0:0.4-2.el6 will be installed
--> Running transaction check
---> Package pyOpenSSL.x86_64 0:0.10-2.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                     Arch                                        Version                                                   Repository                                 Size
===================================================================================================================================================================================================================
Installing:
 openstack-glance                                            noarch                                      2012.1.1-1.el6                                            epel                                       47 k
Installing for dependencies:
 MySQL-python                                                x86_64                                      1.2.3-0.3.c1.1.el6                                        base                                       86 k
 mysql-libs                                                  x86_64                                      5.1.61-4.el6                                              base                                      1.2 M
 openstack-utils                                             noarch                                      2012.1-2.el6                                              epel                                       16 k
 pyOpenSSL                                                   x86_64                                      0.10-2.el6                                                base                                      212 k
 pysendfile                                                  x86_64                                      2.0.0-3.el6                                               epel                                      9.3 k
 python-crypto                                               x86_64                                      2.0.1-22.el6                                              base                                      159 k
 python-decorator                                            noarch                                      3.0.1-3.1.el6                                             base                                       14 k
 python-eventlet                                             noarch                                      0.9.17-1.el6                                              epel                                      262 k
 python-glance                                               noarch                                      2012.1.1-1.el6                                            epel                                      494 k
 python-greenlet                                             x86_64                                      0.3.1-6.el6                                               epel                                       21 k
 python-httplib2                                             noarch                                      0.7.4-6.el6                                               epel                                       69 k
 python-iso8601                                              noarch                                      0.1.4-2.el6                                               epel                                       12 k
 python-migrate                                              noarch                                      0.6-6.el6                                                 epel                                      208 k
 python-paste                                                noarch                                      1.7.4-1.el6                                               base                                      757 k
 python-paste-deploy1.5                                      noarch                                      1.5.0-5.el6                                               epel                                       47 k
 python-sqlalchemy                                           noarch                                      0.5.5-3.el6_2                                             base                                      1.3 M
 python-sqlalchemy0.7                                        x86_64                                      0.7.8-1.el6                                               epel                                      2.1 M
 python-tempita                                              noarch                                      0.4-2.el6                                                 base                                       38 k
 python-webob1.0                                             noarch                                      1.0.8-3.el6                                               epel                                      189 k
 pyxattr                                                     x86_64                                      0.5.0-1.el6                                               epel                                       24 k
...

Good job! We have python-sqlalchemy0.7 for python-glance-2012.1.1-1.el6.noarch and python-sqlalchemy (version 0.5.5; from base CentOS repo) for python-migrate-0.6-6.el6.noarch. You must guess that python-migrate is provided by EPEL but depends on the old sqlalchemy package!

Our sqlalchemies do not conflict ’cause one of them resides in its egg. Guess now which one is really imported and used by glance! (Answer: version 0.7 because there is no evil in EPEL.)

Fighting for peace in the whole world, people from EPEL patched OpenStack sources. This is a patch for glance:

$ cat openstack-glance-newdeps.patch
Delve into pkg_resources a little to get it to modify sys.path,
so that our parallel installed egg takes precedence over the
system default module versions.

diff -up glance-2011.3/glance/__init__.py.newdeps glance-2011.3/glance/__init__.py
--- glance-2011.3/glance/__init__.py.newdeps	2012-01-06 17:22:36.000000000 +0000
+++ glance-2011.3/glance/__init__.py	2012-01-06 17:25:01.019063547 +0000
@@ -18,3 +18,31 @@
 import gettext

 gettext.install('glance', unicode=1)
+
+import sys
+import pkg_resources
+
+# If there is a conflicting non egg module,
+# i.e. an older standard system module installed,
+# then replace it with this requirement
+def replace_dist(requirement):
+    try:
+        return pkg_resources.require(requirement)
+    except pkg_resources.VersionConflict:
+        e = sys.exc_info()[1]
+        dist=e.args[0]
+        req=e.args[1]
+        if dist.key == req.key and not dist.location.endswith('.egg'):
+            del pkg_resources.working_set.by_key[dist.key]
+            # We assume there is no need to adjust sys.path
+            # and the associated pkg_resources.working_set.entries
+            return pkg_resources.require(requirement)
+
+replace_dist("SQLALchemy >= 0.6.3")
+replace_dist("WebOb >= 1.0")
+replace_dist("Routes >= 1.12.3")
+
+replace_dist("PasteDeploy >= 1.5.0")
+# This hack is needed because replace_dist() results in
+# the standard paste module path being at the start of __path__.
+# TODO: See can we get pkg_resources to do the right thing directly
+import paste
+paste.__path__.insert(0, paste.__path__.pop(-1)

It’s insane to mimic EPEL making several “obvious” patches and installing two versions of sqlalchemy. So, that’s wise to maintain several own packages that duplicate the EPEL’s ones but do not need these egg tricks.

About these ads

2 responses to “OpenStack EPEL: the Dependency Purgatory

  1. Pingback: What’s New in Altai 1.0.2 from Maintainer’s Point of View | OpenStack Blog·

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s