Getting Firefox nightly updates on Fedora through yum

This quarter I have had few students (Tarin, Brett van Gennip and Vitaly) from Seneca working with Release Engineering to try to provide nightly updates for Fedora users. This now works and has been tested with Fedora 14.

They have recently added this feature to their Seneca Fedora-Firefox repository and today I would like to share with you what getting updates on Fedora looks like.

Note that this is just a proof-of-concept and that they will (hopefully) continue dealing with all issues and get it integrated into our Mozilla systems and/or with Fedora's (suggestions/directions welcome!) in following quarters.

This post proves again that a well defined problem when given to Seneca's open-source students helps us move forward when we lack man-power. This is one of the first fruits out of the several MozillaReleng-Seneca projects.

The structure of the post will follow as:

• Getting the repository
• Install Minefield
• New update available
• Feedback
• Files
• What's next?
• Known issues or problems

Getting the repository
First we have to make Fedora to know where to pull the package from.
This means that we have to add our "Firefox Nightly" repository to our list of Software Sources.
To do so download this rpm and install it: firefox-repo-0.3-2.noarch.rpm
Once installed you should be able to see the following screen shot when you do this:

• System -> Administration -> Software Sources

Software Sources view with "Firefox Nightly" integrated into it

I believe we should be listed under "debug and development software sources". Let's see.

Install Minefield (a.k.a. Firefox nightly build)
Now that the software source has been added we can decide to install the latest Firefox nightly.
To do so follow one of these two methods:

• System -> Administration -> Add/Remove Software
• Search for "Minefield"
• Check the latest build and select "Apply"

NOTE: Having two entries looks like a bug to me. We will see.

Once installed, Fedora prompts you to run the application and it reminds you from where you can run the application

You can also install Minefield by typing this in the command line:

• yum install minefield 

NOTE: Both of these methods should add the following to your system:

• /usr/lib/minefield-4.0b8pre/
• /usr/bin/minefield -> /usr/lib/minefield-4.0b8pre/firefox
• Add a Menu item under "Applications->Internet"

That's it! You can now run Minefield and get updates for it through yum!

NOTE: I am not comfortable about where we unpack (/usr/lib/minefield-4.0b8pre) as I don't know what will happen when we bump the version.


New update available
How do I receive updates?

Software Update window showing a "Mozilla Firefox Web Browser Nightly" update

The same way as with any Fedora package!

Feedback
Let me know what your thoughts are or what we are doing wrong by:

• Adding a comment to this post.
• Adding a comment to Bug 600317.
• or you can email me directly armenzg [AT] mozilla.com 

Files
In case you wanted to have a look at the different files in the repository:
http://chile.proximity.on.ca/ffrepo

• no-arch 
• firefox-repo-0.3-2.noarch.rpm
• firefox-repo-0.3-2.src.rpm 
• i686
• minefield-4.0b8pre-20101208.fc13.i686.rpm
• minefield-4.0b8pre-20101208.fc13.src.rpm 
• x86_64
• minefield-4.0b8pre-20101208.fc13.src.rpm
• minefield-4.0b8pre-20101208.fc13.x86_64.rpm

What's next? 

• we need to make sure that we are doing everything according to the guidelines that Fedora has for packaging and third party sources
• get people to test it and take input from Fedora/Linux veterans
• integrate it into Release Engineering systems
• does our setup work for other distributions using yum? if not, why not?
• set things up for more branches than just "mozilla-central"
• set things up for all locales
• set things up for betas
• define what the ideal world would look like
  

Known problems or issues

• Current and previous Minefield nightly show up under "Add/Remove Software"

 

•  Setting Minefield as the default browser seems to mess up the "Preferred Web Browser" icon (I need to verify).

• Using the "Preferred Web Browser" launcher opens "file://home/armenzg/". Is it supposed to be like that?

• Should we be listed under "debug and development software sources" on the Software Sources manager?

• Minefield gets unpacked under /usr/lib/minefield-4.0b8pre. Should this be instead /usr/lib/minefield? (version agnostic)
• Minefield when launched from Applications->Internet->Minefield starts with the ProfileManager and allows running the nightly build with the other instances of Firefox ("minefield --no-remote --ProfileManager); should this not be a feature?
• Since /usr/lib/minefield-4.0b8 is owned by "root" Firefox's update system doesn't work. If we change ownership then updates work again and I wonder what bug would I hit if I can receive updates through yum and through AUS. Only one way to find out!
• I also believe that there is not a way to offer partial updates through yum as we do through our normal update system

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Installing Fedora14 through VMWare 2.0.6

I won't go too much into detail but I will describe the process I went through to get this installed.
I first tried the LiveDVD version which failed and then I tried the normal installation DVD.

LiveDVD Fedora-14-i686-Live-Desktop.iso

• I created a new VM from the ISO (I might have chosen Linux 2.6.x kernel {whatever version F14 is} - I can't recall)
• I double-clicked on the "Install to Hard Drive" link on my desktop
• It tells me:

You do not have enough RAM to use the graphical installer. Try the text mode installer by running:
      '/usr/bin/liveinst -T'
from a root terminal.

• The problem is that I probably created the VM with low memory or that I don't have installed the VMware Tools ("Virtual Machine > Install VMware Tools menu) installed.
• I run "/usr/bin/liveinst -T" from the command line
• I get this beautiful text mode installer which works very well with the offered defaults
• The problem is that somehow it froze on the last screen (I think) and I didn't manage to get it to functional state. I decide to start from 0 but from the normal installation DVD.

Installing now from Fedora-14-i386-DVD.iso

• Create new VM from ISO (this time I select more RAM - 728MB)
• After the VM is created and it boots up it seems that the DVD iso is not detected

"The Fedora disc was not found" yara yara

• I do a right click on the CD icon on the VMware window and I select "Connect CD/DVD"
• I right click again and select "Choose disk image..."
• I choose Fedora-14-i386-DVD.iso 
• The installation continues and I reach this menu:

• I choose the defaults except when I have to choose the environment I want to use
• I chose "Sofware Development" installation type instead of default option
• Additional repos can be added in here
• I added "Fedora 14 - i386"; it required configuring the network devices which worked :D
• 30-40 minutes I am asked to reboot
• I update to the latest updates and I happily have a working Fedora 14 :D

Next:
Try to use Seneca's student's RPM repo for Firefox.


PS = Excuse me if I am using VMware Fusion rather than an open source alternative (which I have used before). I can't recall when or why I continued with VMware. Probably because I had work license.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Test suites on Windows

On my previous post I raised that xpcshell on Windows was quite bad on the minis. I assumed that it was mainly because the hardware was slower but Ted commented on the discussion groups saying that it might xpcshell might be slow because of the OS and not just the hardware change.
Guess what? Graphs are our good friends!

I compared all test suites and confirmed that it doesn't have to do too much with the hardware running the test suites. It seems that Win7 in it self is bad for running reftests and xpcshell.
Look at this graph and you will see how the two lines separate quite a lot for xpcshell and reftests.



For comparison, we can see that Win7 is constantly bad for xpcshell and reftests for debug builds as well as optimized builds.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

The xpcshell case on Windows

If you care about end to end times you might want to read this post.

I have enabled debug unit tests on the Windows 7 testing minis and I started filing the permanent oranges for them.

After notifying one of them, philor pointed out something that caught my attention. Debug xpcshell runs on Win7 takes more than a 100 mins ( :S ) compared to 30-40 mins on the IX machines that running Win2003. That sounds like a lot!

I decided that if we are going to disable debug unit tests on the builders (Win2003) as we did before for other platforms we should look closely and see what is going on.

NOTE that at the time that we did the switch we did not have easy ways of tracking variations and the gain was tremendous (more CPU power and end-user OSes) compared to increasing the end to end time. We improved greatly the wait times (larger CPU available) but the end to end times were affected on some platforms since the minis have lower hardware specification (for instance on leopard we didn't loose that much).

Now that I am back to moving Windows debug unit tests to the minis (I have been away -kind of - for more than 2 months) there is something that there wasn't at that time; ssalbiz prepared two months ago a report out of the information from our schedulerdb that has averages for our test jobs. This report was in reaction to some good discussions I had with shaver about our tear down/tear up times.

I will break the rest of this post into data and conclusions.

NOTE:   I am using data from mozilla-central and for Dec. 2nd, 2010.

NOTE2: I am using averages. I know, it is what I have.
NOTE3: Some of the statements on this post do not apply to mozilla-1.9.1 and mozilla-1.9.2

DATA:   The spreadsheet containing the data and charts used. Please be gentle on drawing conclusions without knowing all context that I would love to help you understand.

Everything

How many test jobs (perf and unit tests jobs - opt and debug) do we run for mozilla-central? (I am ignoring JP, mozmill-all and mobile)

• 177

NOTE: We currently run concurrently debug unit tests on Win2003 and Win7. This will change.

In the next couple of months we will also add Windows XP.

• What is the average for each job? 

 Well, our reports can tell us now.

• What is the job that takes the longest for each platform? MAX() to the rescue!

If we take the worst average for any test job for each platform and we put them in a table. We can see the following.

Table 1 - This shows for each platform the worst average for any test job. In orange xpcshell. In purple Windows platforms.

We can see the worst for all test jobs is "debug xpcshell for Win7" with 106.18 minutes on average. Up until now it was "optimized xpcshell Win7x64" with 79.95 minutes (probably if we had debug on Win7x64 it would be even worst).

It is also noticeable on the list of worst three offenders for each platform that xpcshell only appears for Windows. That leads me to other questions.

Xpcshell on Windows

• What would the world look without xpcshell? (or a shorter run of it)

Table 2 - Let's not count xpcshell at all for Windows.

If we remove xpcshell for Windows in our calculations we can see that there is a new worst offender for each platform combination. For instance, for Windows 7 debug type jobs we have "mochitest-other" as the test job that would take the longest. Instead of taking 106.18 mins for having a complete Windows debug coverage we would then only have to wait less than an hour; this is a decrease of 45% which is not bad!!

Let's look now at what would the worst time for all test jobs for any platform look like with and without xpcshell being considered.

Table 3 - Test coverages completion for all platforms

Currently, we wait close to 80 minutes to have a complete coverage for Windows (well, kind of as we don't really pay too much attention to Win7x64 - yet).
If Win7 debug unit tests replace the Win2003 debug unit tests we would have to wait a 32.81% more to have complete Windows coverage. That is not good!
In the last two rows of the previous table you can see that if xpcshell was ignored the new worst offender would be Fedora mochitest-4 debug and developers would wait close to 10% less (not really as build times are in favor of Linux) regardless on where we run Windows unit tests (IX/VMs vs minis). This means that Windows would not be anymore on the way to have full platform coverage (not really as the worst build times are for Windows) but Linux.

Xpschell on all platforms

• How horribly does Windows compare to other platforms when running xpcshell?

Quite bad.
Let's look at the following chart:

You can easily see that every other platform besides Windows ( BLUE ) takes less than 30 minutes. Debug unit tests on the IX machines takes around 40 minutes while on the minis can take up to 100 minutes.
Something makes running xpcshells very very slow on Windows.
All other suites on Windows are not as dramatically as bad as we can see the gains when not considering it (45% gain on debug unit tests completion for Win7).
Ehsan suggested me to determine if xpcshell is going this slow because I/O by looking at the CPU usage that Windows provides.
If it is the reason or not someone needs to look at how to improve it by either fixing some underlying code or breaking xpcshell for Windows into two-three pieces to make it finish on the same range as other test suites.

As you might have noticed, this post is only considering at the times for complete coverage considering all platforms finished at the same time. This is not our reality as each platform takes a different time to finish a build (Windows debug is dramatically slow). This is worth another blog post and will be the basis for improving the left side of the equation (the build times) rather than the right side (the tests times).

This post is to be informative and to help us discover that we can improve the infrastructure even more. After we complete off-loading the builders from unit tests jobs into the minis we have to think of improving the suites, where instead could we run the test jobs, how we can make our builds even faster and others.

We face these new problems because we have stretched our infrastructure and to solve them we will have to reconsider many assumptions and keep on adding tools to allow us make better decisions.

Please don't expect me to do such detailed blog posts as they are quite time consuming. I should finish my goals first!

Questions welcome.

[1] Spreadsheet and charts

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Upcoming switching DEBUG unit tests from Win2003 build machines to Win7 test machines

We have enabled debug unit tests for Windows 7 Rev3 machines.
We are now running debug unit tests on both Win2003 builders and Win7 test machines (except 1.9.1 and 1.9.2).

You can see that we have double coverage for Windows debug unit tests

You should start seeing them on tbpl (except crashtest, reftests and xpcshell). If not check on the tinderbox page and make sure that the builder is "active" and "scrape" is checked as well.

We have enabled this for all branches (except 1.9.1 and 1.9.2) and we will soon adjust the try_chooser parsing to run them on the tryserver as well (it's a bug).

When will we do the switch?

• when we have no perma-orange
• when all test suites are shown on tbpl
• when all working branches are showing them on tbpl
• when we get approval to do so

If you have any questions, ask them on this blog post.
If you notice any fall-outs comment on bug bug 614956.

NOTE: This will impact slightly our wait times on the Win7 testing pool but reduce the wait times for build jobs.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to set up buildAPI locally

I wanted to give a hand to one of our Seneca students (Andrew Singh) that are contributing to release engineering. He is working on creating a report for buildapi.
To do so I had to set up buildapi locally and I was so lucky to be the guinea pig for Mac OS X ;)

Big thanks for ssalbiz and catlee for helping me debug this.

I followed the instructions that ssalbiz wrote https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#Getting_Started:

• Downloaded MySQL Community Server(Current Generally Available Release: 5.1.53)
• Install mysql-5.1.53-osx10.6-x86_64.dmg
• NOTE Make sure you are downloading the 64-bit version otherwise you will (I think) have match-o but wrong architecture problems!!!
• Add mysql to your path:

export PATH=/usr/local/mysql/bins:$PATH 

echo "export PATH=/usr/local/mysql/bins:$PATH" > ~/.bash_profile

• Create a new user to use with buildapi. I read this documentation.

mysql --user=root mysql

mysql> CREATE USER 'monty'@'localhost' IDENTIFIED BY 'some_pass';

mysql> GRANT ALL PRIVILEGES ON *.* TO 'monty'@'localhost' WITH GRANT OPTION;

• Create the databases and import them.

mysql -u -p

mysql> create database schedulerdb; create database statusdb;

mysql> exit

mysql -u -p schedulerdb < schedulerdb.sql

mysql -u -p statusdb < statusdb.sql

• Setup buildapi:

hg clone http://hg.mozilla.org/build/buildapi

cd buildapi; sudo python setup.py install; cd ..

mkdir dist

cd dist; wget http://google-visualization-python.googlecode.com/files/gviz_api_py-1.7.0.tar.gz

sudo easy_install gviz_api_py-1.7.0.tar.gz

cd ..

paster make-config buildapi config.ini

# edit your config.ini with the right information

# sqlalchemy.scheduler_db.url = mysql://monty:some_pass@localhost/schedulerdb

# sqlalchemy.status_db.url = mysql://monty:some_pass@localhost/statusdb

• Start buildapi

paster serve --reload --daemon config.ini

NOTE: I believe that if the following is respected you should not hit all the problems that I hit in the next section:

• Install the 64-bit version of MySQL
• Install buildapi with "python setup.py install" instead of "easy_install buildapi"

PROBLEMS

I tried to load http://localhost:5000 but it didn't work.

Now I tried it without --daemon and I got this:

armenzg-laptop $ paster serve --reload config.ini
Starting subprocess with file monitor
Traceback (most recent call last):
File "/usr/local/bin/paster", line 8, in
load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run
invoke(command, command_name, options, args[1:])
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke
exit_code = runner.run(args)
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run
result = self.command()
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command
relative_to=base, global_conf=vars)
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp
**kw)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 225, in loadobj
return context.create()
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 625, in create
return self.object_type.invoke(self)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 110, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call
val = callable(*args, **kw)
File "/Library/Python/2.6/site-packages/buildapi-0.1dev-py2.6.egg/buildapi/config/middleware.py", line 37, in make_app
config = load_environment(global_conf, app_conf)
File "/Library/Python/2.6/site-packages/buildapi-0.1dev-py2.6.egg/buildapi/config/environment.py", line 48, in load_environment
scheduler_engine = engine_from_config(config, 'sqlalchemy.scheduler_db.')
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/__init__.py", line 272, in engine_from_config
return create_engine(url, **opts)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/__init__.py", line 254, in create_engine
return strategy.create(*args, **kwargs)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/strategies.py", line 60, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/dialects/mysql/mysqldb.py", line 101, in dbapi
return __import__('MySQLdb')
ImportError: No module named MySQLdb

I will now list the SOLUTION and after that the other route I first took and lead me to nowhere.

For reference, this problem is hit in many different ways by many different people:

• http://stackoverflow.com/questions/3061277/python-mysql-wrong-architecture-error
• http://stackoverflow.com/questions/2414658/error-installing-mysql-python-on-mac-snow-leopard-os-10-6
• http://birdhouse.org/blog/2009/02/21/python-mysql-connections-on-mac-os/

SOLUTION

• Uninstall MySQL (because I installed the 32-bit by mistake) http://akrabat.com/computing/uninstalling-mysql-on-mac-os-x-leopard/
• Download and install mysql-5.1.53-osx10.6-x86_64.dmg (this is the 64-bit version)

rm -rf /Library/Python/2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg
easy_install mysql-python

• This should work now:

python -c 'import MySQLdb'

• I think it should work now

paster serve config.ini

FAILED ATTEMPT

NOTE: I am just typing this failed attempt for the record and to maybe bring some frustrated people with the same problem to "a solution".

Resuming from ImportError: No module named MySQLdb.

It seems that I am missing the MySQLdb for python:

sudo easy_install mysql-python

which errors on me:

EnvironmentError: mysql_config not found

The problem is that mysql_config in the package's site.cfg file (download the source package if you want to see that file) points to mysql_config = /usr/local/bin/mysql_config
You can fix this in several ways:

1. Add a symlink "ln -s /usr/local/mysql/bin/mysql_config /usr/local/bin/mysql_config"
2. Add /usr/local/mysql/bin to your PATH
3. Download the mysql-python package and modify site.cfg

I thought that installing mysql-python would fix things but I as you can see I can't even do this on python:

armenzg-laptop $ python -c 'import MySQLdb'
Traceback (most recent call last):
  File "", line 1, in
  File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in
  File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in
  File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/armenzg/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): no suitable image found.  Did find:
        /Users/armenzg/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so: mach-o, but wrong architecture

For the record I also tried this:

pip install -I mysql-python

OTHER

Something that I found to be interesting/nit:

armenzg-laptop $ file $(which python)
/usr/bin/python: Mach-O universal binary with 3 architectures
/usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python (for architecture i386): Mach-O executable i386
/usr/bin/python (for architecture ppc7400): Mach-O executable ppc
armenzg-laptop $ file $(which mysql)
/usr/local/mysql/bin/mysql: Mach-O executable i386

After I installed MySQL 64-bit:

armenzg-laptop $ file $(which mysql)
/usr/local/mysql/bin/mysql: Mach-O 64-bit executable x86_64

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.