I'm been working on creating my second Debian package for Mozilla's Release Engineering infrastructure and it's been a pain like the first one.
To be honest, it's been hard to figure out the correct flow and to understand what I was doing.
In order to help other people in the future, I decided to document the process and workflow.
This is not to replace the documentation but to help understand it.
If you're using a Mac or a Windows machine, notice that we have a VM available on EC2 that has the tools you need: ubuntu64packager1.srv.releng.use1.mozilla.com. The documentation can be found in "
How to build DEBs". You can use this blog post to help you get up to speed.
During our coming work week we will look at a complete different approach to make changes like this easier for developers to make without Release Engineering intervention. It is not necessarily a self-serve for Debian deployments.
Goal
We want to upgrade a library or a binary on our infrastructure.
For Linux, we use Puppet to deploy packages and we deploy it through a Debian repository.
Before we deploy the package through Puppet, we have to add the package to our internal Debian repository. This blog post will guide you to:
- Create the .deb files
- Add them to our internal Debian repository
- Test the deployment of the package with Puppet
Debian packaging
For a newbie, it can be a very complicated system that has many many parts.
In short, I've learned that there are involved three different files that allow you to recreate the .deb files. The files extensions are:
.dsc,
.orig.tar.gz and
.diff.gz. If you find
the source package page for your desired package, you will notice that these 3 files are available to download. We can use the .dsc file to generate all the .deb files.
For full info you can read the
Debian Packaging documentation and/or look at the
building tutorial to apply changes to an existing package.
Ubuntu version naming
If I understand correctly (IIUC),
"precise" is an identifier for a Ubuntu release. In our case it refers to
Ubuntu 12.04 LTS.
Versions of a package
IIUC, a package can have 3 different versions or channels:
- release. The version that came out with a specific release
- Ubuntu 12.04 came out with mesa 8.0.2-0ubuntu3
- security. The latest security release
- e.g. mesa 8.0.4-0ubuntu0.6
- updates. The latest update
- e.g. mesa 8.0.4-0ubuntu0.7
If you load the
"mesa" source package page, you will find a section called "Versions published" and you will see all three versions listed there.
Precise, not precise-updates
In our specific releng setup, we always use "precise" as the distribution and not "precise-updates".
I don't know why.
Repackage the current version or the latest one?
If you're patching a current package, do not try jump to the latest available version unless necessary. Choose the version closest to our current package to reduce the number of new dependencies.
In my case I was trying to go for
mesa 8.0.4-0ubuntu0.7 instead of
mesa 8.0.2-0ubuntu3.
Due to that, I had all sorts of difficulties and it had lots of new dependencies.
Even then, I realized later that I had to go for mesa 8.0.4-0ubuntu0.6 as a minimum.
Puppetagain-build-deb OR pbuilder?
From Mozilla's Release Engineering's prespective, we're only considering two ways of creating our .deb files: 1)
puppetagain-build-deb and 2) pbuilder.
FYI puppetagain-build-deb was written to make it very simple to create the required .deb files.
Unfortunately, in my case, puppetagain-build-deb could only handle the dependencies of 0.8.2 and not the ones of 0.8.4.
I describe how to use pbuilder in the section "Create the debian/ directory".
Below is the "puppetagain-build-deb" approach. Also documented in
here.
Puppetagain-build-deb
At this point we have the "package_name-debian" directory under modules/packages/manifests in Puppet. Besides that, we need to download ".orig.tar.gz" file.
To create the .deb files we need 1) the debian directory + 2) the original tar ball.
In most cases, we should be able to use ubuntu64packager1 and puppetagain-build-deb to build the deb files. If not,
NOTE: The .orig.tar.gz file does not need to be committed.
cd puppet
hg up -r d6aac1ea887f #It has the 8.0.2 version checked-in
cd modules/packages/manifests
wget https://launchpad.net/ubuntu/+archive/primary/+files/mesa_8.0.2.orig.tar.gz
# The .deb files will appear under /tmp/mesa-precise-i386
puppetagain-build-deb precise amd64 mesa-debian
# The .deb files will appear under /tmp/mesa-precise-amd64
puppetagain-build-deb precise i386 mesa-debian
Create the debian/ directory
In Puppet we have "debian" directories checked-in (e.g.
mesa-debian/) for any debian package we deploy to our systems through it. The debian directory is produced with the standard Debian packing instructions.
If you have access to a Linux machine you can follow
the steps that rail gave me to generate the deb files. You can also log-in to ubuntu64packager1 (you have to start it up first).
To make it work locally, I had to install pbuilder with "sudo apt-get install pbuilder".
I also needed to
create my own pbuilder images.
In short, to recreate .deb files without modifying them you can follow these steps:
- use dget to downloads all three required files (.dsc, .orig.tar.gz and .diff.gz)
- use pbuilder --build to generate the .deb files
Since we want to patch the libraries rather than use them as-is, we also have to run these steps in between step 1 & step 2:
- dpkg-source -x
- it extracts the source files
- download your patch under debian/patches
- append a line to debian/patches/series
- the line indicates the filename of your patch under debian/patches
- set DEBFULLNAME
- to bump the version when repackagin the source
- dpkg-source -b
- rebuild the source package
Keep track of the debian/ directory in Puppet
The previous section should have generated your desired "debian" directory.
We now need to check it inside of our puppet repository to keep track of it.
cp -r mesa-8.0.4/debian ~/puppet/modules/packages/manifests/mesa-debian
cd ~/puppet
hg addremove
hg diff
Having Debian packaging issues?
rail and
dustin have experience in this area, however, if we have further Debian packaging issues we can reach
sylvestre and
glandium.
Determine involved libraries
To create our Puppet patch, we have to determine which packages are involved.
For instance, the mesa bug required updating five different libraries.
rail explains on
comment 26 how to discover which libraries are involved.
You can list the package names you compiled with something like this:
ls *deb | awk -F_ '{print $1}' | xargs
# copy the list of names and run the following on the target machine:
dpkg -l 2>/dev/null | grep ^ii | awk '{print $2}'
Create a no-op puppet change (pinning the version)
If the package already exists on our infra but it is not managed by Puppet (e.g. the library came by default on the OS), then it is better to write first a puppet change to pin the versions.
To write the puppet change you will have to answer these questions:
- Do we want this change for the in-house and ec2 machines? Or a subset?
- Do we want the change for both 64-bit and 32-bit machines?
- What are the versions currently running on the machines that would be affected?
- Check on each pool you're planning to deploy it since we could have inconsistencies between them
Answering these questions will determine which files to modify in puppet.
Remember that you will have to test that your puppet change runs without issues.
Integrating your .deb files into the releng Debian repository and sync to the puppet masters
The documentation is
here. And here's what I did for it.
1 - Sync locally the Debian packages repository
We need to sync locally from the "
distinguished master" the "releng", "conf" and "deb" directories:
sudo su
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/ /data/repos/apt/releng/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/conf/ /data/repos/apt/conf/
rsync -av releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/ /data/repos/apt/db/
2 - Import your .deb files into the Debian repo
cd /data/repos/apt
cp ~armenzg/tmp/mesa_8.0.4.orig.tar.gz releng/pool/main/m/mesa
reprepro -V --basedir . include precise ~armenzg/tmp/out64/*.changes
reprepro -V --basedir . includedeb precise ~armenzg/tmp/out32/*.deb
If the package is new you will also have to place the .orig.tar.gz file under /data/repos/apt/releng. The reprepro will let you know as it will fail until you do.
3 - Rsync the repo and db back to the distinguished master
Push your file back to the official repository:
rsync -av /data/repos/apt/releng/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/releng/
rsync -av /data/repos/apt/db/ releng-puppet2.srv.releng.scl3.mozilla.com:/data/repos/apt/db/
Your files should show up in here:
http://puppetagain.pub.build.mozilla.org/data/repos/apt/releng/pool/main
NOTE: Pushing the .deb files to the repo does not update the machines.
4 - Fix the permissions at the distinguished master
ssh root@releng-puppet2.srv.releng.scl3.mozilla.com
puppetmaster-fixperms
Test that you can update
Before you can sync up a host with puppet you need to let the puppet servers sync up with the distinguished ma
ster.
For instance, my puppet runs were failing because the packages were missing at:
http://puppetagain-apt.pvt.build.mozilla.org/repos/apt/releng/pool/main/m/mesa
To test my changes, I created two EC2 instances. For other pools you will have to pull a machine from production.
1 - Prepare your user environment
ssh armenzg@releng-puppet2.srv.releng.scl3.mozilla.com
cd /etc/puppet/environments/armenzg/env
hg pull -u && hg st
2 - Run a no-op test sync from your loaned machines
puppet agent --test --environment=armenzg --server=releng-puppet2.srv.releng.scl3.mozilla.com
3 - Under your user environment on the puppet master, bump the versions and the repoflag
4 - Run puppet syncs again on the test instances and watch for the changes on the Puppet output
puppet agent --test --environment=armenzg --server=releng-puppet2.srv.releng.scl3.mozilla.com
5 - Review the package versions are the right ones
6 - Test a rollback scenario
You will have to remove the bumping of the versions from step #3 and bump the repoflag again.
Run steps 4 and 5 to see that we downgrade properly.
7 - Clean up ubuntu64packager1 and shut it off
8 - Deploy your change like any other Puppet change
Read all the steps at once
Commands and minimum comments:
https://bugzilla.mozilla.org/show_bug.cgi?id=975034#c37
This work by
Zambrano Gasparnian, Armen is licensed under a
Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.
