Getting Firefox nightly updates on Fedora through yum

This quarter I have had few students (Tarin, Brett van Gennip and Vitaly) from Seneca working with Release Engineering to try to provide nightly updates for Fedora users. This now works and has been tested with Fedora 14.

They have recently added this feature to their Seneca Fedora-Firefox repository and today I would like to share with you what getting updates on Fedora looks like.

Note that this is just a proof-of-concept and that they will (hopefully) continue dealing with all issues and get it integrated into our Mozilla systems and/or with Fedora's (suggestions/directions welcome!) in following quarters.

This post proves again that a well defined problem when given to Seneca's open-source students helps us move forward when we lack man-power. This is one of the first fruits out of the several MozillaReleng-Seneca projects.

The structure of the post will follow as:

• Getting the repository
• Install Minefield
• New update available
• Feedback
• Files
• What's next?
• Known issues or problems

Getting the repository
First we have to make Fedora to know where to pull the package from.
This means that we have to add our "Firefox Nightly" repository to our list of Software Sources.
To do so download this rpm and install it: firefox-repo-0.3-2.noarch.rpm
Once installed you should be able to see the following screen shot when you do this:

• System -> Administration -> Software Sources

Software Sources view with "Firefox Nightly" integrated into it

I believe we should be listed under "debug and development software sources". Let's see.

Install Minefield (a.k.a. Firefox nightly build)
Now that the software source has been added we can decide to install the latest Firefox nightly.
To do so follow one of these two methods:

• System -> Administration -> Add/Remove Software
• Search for "Minefield"
• Check the latest build and select "Apply"

NOTE: Having two entries looks like a bug to me. We will see.

Once installed, Fedora prompts you to run the application and it reminds you from where you can run the application

You can also install Minefield by typing this in the command line:

• yum install minefield 

NOTE: Both of these methods should add the following to your system:

• /usr/lib/minefield-4.0b8pre/
• /usr/bin/minefield -> /usr/lib/minefield-4.0b8pre/firefox
• Add a Menu item under "Applications->Internet"

That's it! You can now run Minefield and get updates for it through yum!

NOTE: I am not comfortable about where we unpack (/usr/lib/minefield-4.0b8pre) as I don't know what will happen when we bump the version.


New update available
How do I receive updates?

Software Update window showing a "Mozilla Firefox Web Browser Nightly" update

The same way as with any Fedora package!

Feedback
Let me know what your thoughts are or what we are doing wrong by:

• Adding a comment to this post.
• Adding a comment to Bug 600317.
• or you can email me directly armenzg [AT] mozilla.com 

Files
In case you wanted to have a look at the different files in the repository:
http://chile.proximity.on.ca/ffrepo

• no-arch 
• firefox-repo-0.3-2.noarch.rpm
• firefox-repo-0.3-2.src.rpm 
• i686
• minefield-4.0b8pre-20101208.fc13.i686.rpm
• minefield-4.0b8pre-20101208.fc13.src.rpm 
• x86_64
• minefield-4.0b8pre-20101208.fc13.src.rpm
• minefield-4.0b8pre-20101208.fc13.x86_64.rpm

What's next? 

• we need to make sure that we are doing everything according to the guidelines that Fedora has for packaging and third party sources
• get people to test it and take input from Fedora/Linux veterans
• integrate it into Release Engineering systems
• does our setup work for other distributions using yum? if not, why not?
• set things up for more branches than just "mozilla-central"
• set things up for all locales
• set things up for betas
• define what the ideal world would look like
  

Known problems or issues

• Current and previous Minefield nightly show up under "Add/Remove Software"

 

•  Setting Minefield as the default browser seems to mess up the "Preferred Web Browser" icon (I need to verify).

• Using the "Preferred Web Browser" launcher opens "file://home/armenzg/". Is it supposed to be like that?

• Should we be listed under "debug and development software sources" on the Software Sources manager?

• Minefield gets unpacked under /usr/lib/minefield-4.0b8pre. Should this be instead /usr/lib/minefield? (version agnostic)
• Minefield when launched from Applications->Internet->Minefield starts with the ProfileManager and allows running the nightly build with the other instances of Firefox ("minefield --no-remote --ProfileManager); should this not be a feature?
• Since /usr/lib/minefield-4.0b8 is owned by "root" Firefox's update system doesn't work. If we change ownership then updates work again and I wonder what bug would I hit if I can receive updates through yum and through AUS. Only one way to find out!
• I also believe that there is not a way to offer partial updates through yum as we do through our normal update system

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Installing Fedora14 through VMWare 2.0.6

I won't go too much into detail but I will describe the process I went through to get this installed.
I first tried the LiveDVD version which failed and then I tried the normal installation DVD.

LiveDVD Fedora-14-i686-Live-Desktop.iso

• I created a new VM from the ISO (I might have chosen Linux 2.6.x kernel {whatever version F14 is} - I can't recall)
• I double-clicked on the "Install to Hard Drive" link on my desktop
• It tells me:

You do not have enough RAM to use the graphical installer. Try the text mode installer by running:
      '/usr/bin/liveinst -T'
from a root terminal.

• The problem is that I probably created the VM with low memory or that I don't have installed the VMware Tools ("Virtual Machine > Install VMware Tools menu) installed.
• I run "/usr/bin/liveinst -T" from the command line
• I get this beautiful text mode installer which works very well with the offered defaults
• The problem is that somehow it froze on the last screen (I think) and I didn't manage to get it to functional state. I decide to start from 0 but from the normal installation DVD.

Installing now from Fedora-14-i386-DVD.iso

• Create new VM from ISO (this time I select more RAM - 728MB)
• After the VM is created and it boots up it seems that the DVD iso is not detected

"The Fedora disc was not found" yara yara

• I do a right click on the CD icon on the VMware window and I select "Connect CD/DVD"
• I right click again and select "Choose disk image..."
• I choose Fedora-14-i386-DVD.iso 
• The installation continues and I reach this menu:

• I choose the defaults except when I have to choose the environment I want to use
• I chose "Sofware Development" installation type instead of default option
• Additional repos can be added in here
• I added "Fedora 14 - i386"; it required configuring the network devices which worked :D
• 30-40 minutes I am asked to reboot
• I update to the latest updates and I happily have a working Fedora 14 :D

Next:
Try to use Seneca's student's RPM repo for Firefox.


PS = Excuse me if I am using VMware Fusion rather than an open source alternative (which I have used before). I can't recall when or why I continued with VMware. Probably because I had work license.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Test suites on Windows

On my previous post I raised that xpcshell on Windows was quite bad on the minis. I assumed that it was mainly because the hardware was slower but Ted commented on the discussion groups saying that it might xpcshell might be slow because of the OS and not just the hardware change.
Guess what? Graphs are our good friends!

I compared all test suites and confirmed that it doesn't have to do too much with the hardware running the test suites. It seems that Win7 in it self is bad for running reftests and xpcshell.
Look at this graph and you will see how the two lines separate quite a lot for xpcshell and reftests.



For comparison, we can see that Win7 is constantly bad for xpcshell and reftests for debug builds as well as optimized builds.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

The xpcshell case on Windows

If you care about end to end times you might want to read this post.

I have enabled debug unit tests on the Windows 7 testing minis and I started filing the permanent oranges for them.

After notifying one of them, philor pointed out something that caught my attention. Debug xpcshell runs on Win7 takes more than a 100 mins ( :S ) compared to 30-40 mins on the IX machines that running Win2003. That sounds like a lot!

I decided that if we are going to disable debug unit tests on the builders (Win2003) as we did before for other platforms we should look closely and see what is going on.

NOTE that at the time that we did the switch we did not have easy ways of tracking variations and the gain was tremendous (more CPU power and end-user OSes) compared to increasing the end to end time. We improved greatly the wait times (larger CPU available) but the end to end times were affected on some platforms since the minis have lower hardware specification (for instance on leopard we didn't loose that much).

Now that I am back to moving Windows debug unit tests to the minis (I have been away -kind of - for more than 2 months) there is something that there wasn't at that time; ssalbiz prepared two months ago a report out of the information from our schedulerdb that has averages for our test jobs. This report was in reaction to some good discussions I had with shaver about our tear down/tear up times.

I will break the rest of this post into data and conclusions.

NOTE:   I am using data from mozilla-central and for Dec. 2nd, 2010.

NOTE2: I am using averages. I know, it is what I have.
NOTE3: Some of the statements on this post do not apply to mozilla-1.9.1 and mozilla-1.9.2

DATA:   The spreadsheet containing the data and charts used. Please be gentle on drawing conclusions without knowing all context that I would love to help you understand.

Everything

How many test jobs (perf and unit tests jobs - opt and debug) do we run for mozilla-central? (I am ignoring JP, mozmill-all and mobile)

• 177

NOTE: We currently run concurrently debug unit tests on Win2003 and Win7. This will change.

In the next couple of months we will also add Windows XP.

• What is the average for each job? 

 Well, our reports can tell us now.

• What is the job that takes the longest for each platform? MAX() to the rescue!

If we take the worst average for any test job for each platform and we put them in a table. We can see the following.

Table 1 - This shows for each platform the worst average for any test job. In orange xpcshell. In purple Windows platforms.

We can see the worst for all test jobs is "debug xpcshell for Win7" with 106.18 minutes on average. Up until now it was "optimized xpcshell Win7x64" with 79.95 minutes (probably if we had debug on Win7x64 it would be even worst).

It is also noticeable on the list of worst three offenders for each platform that xpcshell only appears for Windows. That leads me to other questions.

Xpcshell on Windows

• What would the world look without xpcshell? (or a shorter run of it)

Table 2 - Let's not count xpcshell at all for Windows.

If we remove xpcshell for Windows in our calculations we can see that there is a new worst offender for each platform combination. For instance, for Windows 7 debug type jobs we have "mochitest-other" as the test job that would take the longest. Instead of taking 106.18 mins for having a complete Windows debug coverage we would then only have to wait less than an hour; this is a decrease of 45% which is not bad!!

Let's look now at what would the worst time for all test jobs for any platform look like with and without xpcshell being considered.

Table 3 - Test coverages completion for all platforms

Currently, we wait close to 80 minutes to have a complete coverage for Windows (well, kind of as we don't really pay too much attention to Win7x64 - yet).
If Win7 debug unit tests replace the Win2003 debug unit tests we would have to wait a 32.81% more to have complete Windows coverage. That is not good!
In the last two rows of the previous table you can see that if xpcshell was ignored the new worst offender would be Fedora mochitest-4 debug and developers would wait close to 10% less (not really as build times are in favor of Linux) regardless on where we run Windows unit tests (IX/VMs vs minis). This means that Windows would not be anymore on the way to have full platform coverage (not really as the worst build times are for Windows) but Linux.

Xpschell on all platforms

• How horribly does Windows compare to other platforms when running xpcshell?

Quite bad.
Let's look at the following chart:

You can easily see that every other platform besides Windows ( BLUE ) takes less than 30 minutes. Debug unit tests on the IX machines takes around 40 minutes while on the minis can take up to 100 minutes.
Something makes running xpcshells very very slow on Windows.
All other suites on Windows are not as dramatically as bad as we can see the gains when not considering it (45% gain on debug unit tests completion for Win7).
Ehsan suggested me to determine if xpcshell is going this slow because I/O by looking at the CPU usage that Windows provides.
If it is the reason or not someone needs to look at how to improve it by either fixing some underlying code or breaking xpcshell for Windows into two-three pieces to make it finish on the same range as other test suites.

As you might have noticed, this post is only considering at the times for complete coverage considering all platforms finished at the same time. This is not our reality as each platform takes a different time to finish a build (Windows debug is dramatically slow). This is worth another blog post and will be the basis for improving the left side of the equation (the build times) rather than the right side (the tests times).

This post is to be informative and to help us discover that we can improve the infrastructure even more. After we complete off-loading the builders from unit tests jobs into the minis we have to think of improving the suites, where instead could we run the test jobs, how we can make our builds even faster and others.

We face these new problems because we have stretched our infrastructure and to solve them we will have to reconsider many assumptions and keep on adding tools to allow us make better decisions.

Please don't expect me to do such detailed blog posts as they are quite time consuming. I should finish my goals first!

Questions welcome.

[1] Spreadsheet and charts

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Upcoming switching DEBUG unit tests from Win2003 build machines to Win7 test machines

We have enabled debug unit tests for Windows 7 Rev3 machines.
We are now running debug unit tests on both Win2003 builders and Win7 test machines (except 1.9.1 and 1.9.2).

You can see that we have double coverage for Windows debug unit tests

You should start seeing them on tbpl (except crashtest, reftests and xpcshell). If not check on the tinderbox page and make sure that the builder is "active" and "scrape" is checked as well.

We have enabled this for all branches (except 1.9.1 and 1.9.2) and we will soon adjust the try_chooser parsing to run them on the tryserver as well (it's a bug).

When will we do the switch?

• when we have no perma-orange
• when all test suites are shown on tbpl
• when all working branches are showing them on tbpl
• when we get approval to do so

If you have any questions, ask them on this blog post.
If you notice any fall-outs comment on bug bug 614956.

NOTE: This will impact slightly our wait times on the Win7 testing pool but reduce the wait times for build jobs.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to set up buildAPI locally

I wanted to give a hand to one of our Seneca students (Andrew Singh) that are contributing to release engineering. He is working on creating a report for buildapi.
To do so I had to set up buildapi locally and I was so lucky to be the guinea pig for Mac OS X ;)

Big thanks for ssalbiz and catlee for helping me debug this.

I followed the instructions that ssalbiz wrote https://wiki.mozilla.org/ReleaseEngineering/BuildAPI#Getting_Started:

• Downloaded MySQL Community Server(Current Generally Available Release: 5.1.53)
• Install mysql-5.1.53-osx10.6-x86_64.dmg
• NOTE Make sure you are downloading the 64-bit version otherwise you will (I think) have match-o but wrong architecture problems!!!
• Add mysql to your path:

export PATH=/usr/local/mysql/bins:$PATH 

echo "export PATH=/usr/local/mysql/bins:$PATH" > ~/.bash_profile

• Create a new user to use with buildapi. I read this documentation.

mysql --user=root mysql

mysql> CREATE USER 'monty'@'localhost' IDENTIFIED BY 'some_pass';

mysql> GRANT ALL PRIVILEGES ON *.* TO 'monty'@'localhost' WITH GRANT OPTION;

• Create the databases and import them.

mysql -u -p

mysql> create database schedulerdb; create database statusdb;

mysql> exit

mysql -u -p schedulerdb < schedulerdb.sql

mysql -u -p statusdb < statusdb.sql

• Setup buildapi:

hg clone http://hg.mozilla.org/build/buildapi

cd buildapi; sudo python setup.py install; cd ..

mkdir dist

cd dist; wget http://google-visualization-python.googlecode.com/files/gviz_api_py-1.7.0.tar.gz

sudo easy_install gviz_api_py-1.7.0.tar.gz

cd ..

paster make-config buildapi config.ini

# edit your config.ini with the right information

# sqlalchemy.scheduler_db.url = mysql://monty:some_pass@localhost/schedulerdb

# sqlalchemy.status_db.url = mysql://monty:some_pass@localhost/statusdb

• Start buildapi

paster serve --reload --daemon config.ini

NOTE: I believe that if the following is respected you should not hit all the problems that I hit in the next section:

• Install the 64-bit version of MySQL
• Install buildapi with "python setup.py install" instead of "easy_install buildapi"

PROBLEMS

I tried to load http://localhost:5000 but it didn't work.

Now I tried it without --daemon and I got this:

armenzg-laptop $ paster serve --reload config.ini
Starting subprocess with file monitor
Traceback (most recent call last):
File "/usr/local/bin/paster", line 8, in
load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 84, in run
invoke(command, command_name, options, args[1:])
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 123, in invoke
exit_code = runner.run(args)
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/command.py", line 218, in run
result = self.command()
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 276, in command
relative_to=base, global_conf=vars)
File "/Library/Python/2.6/site-packages/PasteScript-1.7.3-py2.6.egg/paste/script/serve.py", line 313, in loadapp
**kw)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 204, in loadapp
return loadobj(APP, uri, name=name, **kw)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 225, in loadobj
return context.create()
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 625, in create
return self.object_type.invoke(self)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/loadwsgi.py", line 110, in invoke
return fix_call(context.object, context.global_conf, **context.local_conf)
File "/Library/Python/2.6/site-packages/PasteDeploy-1.3.3-py2.6.egg/paste/deploy/util/fixtypeerror.py", line 57, in fix_call
val = callable(*args, **kw)
File "/Library/Python/2.6/site-packages/buildapi-0.1dev-py2.6.egg/buildapi/config/middleware.py", line 37, in make_app
config = load_environment(global_conf, app_conf)
File "/Library/Python/2.6/site-packages/buildapi-0.1dev-py2.6.egg/buildapi/config/environment.py", line 48, in load_environment
scheduler_engine = engine_from_config(config, 'sqlalchemy.scheduler_db.')
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/__init__.py", line 272, in engine_from_config
return create_engine(url, **opts)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/__init__.py", line 254, in create_engine
return strategy.create(*args, **kwargs)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/engine/strategies.py", line 60, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/Library/Python/2.6/site-packages/SQLAlchemy-0.6.5-py2.6.egg/sqlalchemy/dialects/mysql/mysqldb.py", line 101, in dbapi
return __import__('MySQLdb')
ImportError: No module named MySQLdb

I will now list the SOLUTION and after that the other route I first took and lead me to nowhere.

For reference, this problem is hit in many different ways by many different people:

• http://stackoverflow.com/questions/3061277/python-mysql-wrong-architecture-error
• http://stackoverflow.com/questions/2414658/error-installing-mysql-python-on-mac-snow-leopard-os-10-6
• http://birdhouse.org/blog/2009/02/21/python-mysql-connections-on-mac-os/

SOLUTION

• Uninstall MySQL (because I installed the 32-bit by mistake) http://akrabat.com/computing/uninstalling-mysql-on-mac-os-x-leopard/
• Download and install mysql-5.1.53-osx10.6-x86_64.dmg (this is the 64-bit version)

rm -rf /Library/Python/2.6/site-packages/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg
easy_install mysql-python

• This should work now:

python -c 'import MySQLdb'

• I think it should work now

paster serve config.ini

FAILED ATTEMPT

NOTE: I am just typing this failed attempt for the record and to maybe bring some frustrated people with the same problem to "a solution".

Resuming from ImportError: No module named MySQLdb.

It seems that I am missing the MySQLdb for python:

sudo easy_install mysql-python

which errors on me:

EnvironmentError: mysql_config not found

The problem is that mysql_config in the package's site.cfg file (download the source package if you want to see that file) points to mysql_config = /usr/local/bin/mysql_config
You can fix this in several ways:

1. Add a symlink "ln -s /usr/local/mysql/bin/mysql_config /usr/local/bin/mysql_config"
2. Add /usr/local/mysql/bin to your PATH
3. Download the mysql-python package and modify site.cfg

I thought that installing mysql-python would fix things but I as you can see I can't even do this on python:

armenzg-laptop $ python -c 'import MySQLdb'
Traceback (most recent call last):
  File "", line 1, in
  File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in
  File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in
  File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/armenzg/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): no suitable image found.  Did find:
        /Users/armenzg/.python-eggs/MySQL_python-1.2.3-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so: mach-o, but wrong architecture

For the record I also tried this:

pip install -I mysql-python

OTHER

Something that I found to be interesting/nit:

armenzg-laptop $ file $(which python)
/usr/bin/python: Mach-O universal binary with 3 architectures
/usr/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64
/usr/bin/python (for architecture i386): Mach-O executable i386
/usr/bin/python (for architecture ppc7400): Mach-O executable ppc
armenzg-laptop $ file $(which mysql)
/usr/local/mysql/bin/mysql: Mach-O executable i386

After I installed MySQL 64-bit:

armenzg-laptop $ file $(which mysql)
/usr/local/mysql/bin/mysql: Mach-O 64-bit executable x86_64

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

OPSI nightmares on Windows 2008 R2 64-bit

Back in September I worked for five days trying to make OPSI work for our Windows 2008 64-bit reference machine and failed miserably.

I reached a point where I wanted to uninstall OPSI and install it again from the right mount share. The problem is that uninstall process got the machine wedged and impossible to login through RDP. We finally figured out that we had to remove two registry keys to allow the machine to reach the login page (We used IPMI and safe mode to do this).

I will just list some comments that might help somebody searching for answers related to this. I won't put too much effort on making it presentable or coherent.

NOTE: many many thanks for cshields for having helped me through those days!

The versions (I think) we are using are:

The version for staging-opsi is 3.4.0.0 and I don't see preloginloader installed on the win64-ix-ref.build.mozilla.org machine. Nevertheless when I click on the package I get 3.4 v27mozilla2.

From bug 596735:

Blue screen before reaching login dialog

What I could see through IPMI: "Opsi Login Blocker: service request failed"

I asked on the OPSI forums and I got this answer:

Re: Cannot uninstall on Windows 2008 x64

von wolfbardo » 17 Sep 2010, 08:19

Hello Armen,

armenzg hat geschrieben:The machine is MS2008 R2 x64 with OPSI versions 3.4.0.0 and preloginloader package 3.4 v27mozilla2.

In this version there some issues in the 64-Bit handling of Files and Registry

You should update your opsi-version to 3.4.0.14 and the opsi-packages
http://download.uib.de/opsi3.4/produkte ... 8.4-1.opsi
http://download.uib.de/opsi3.4/produkte ... .4-69.opsi

Also have a look at opsi 4.0 Release Candidate (viewtopic.php?f=10&t=1747)

armenzg hat geschrieben:
What could I try?

Delete the follwoing registry-Keys

Code: Alles auswählen

[Registry_vista_del_loginblocker] deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers\{d2028e19-82fe-44c6-ad64-51497c97a02a}] deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider Filters\{d2028e19-82fe-44c6-ad64-51497c97a02a    }]

regards
bardo wolf

The key was to login with IPMI on safe mode and remove those two registry keys.
After that the machine was good almost good (RDP was not working - jabba fixed this for me in bug 597573).

I also got another machine wedged in the exact same way but not from trying to uninstall OPSI but from forgetting to resume the opsiclientd service.

In bug 596735 comment 12 I mentioned:

This morning I found win64-ix-ref in the same condition and it is *not* from uninstalling OPSI but from having forgotten to resume last night opsiclientd service (which is mentioned here https://forum.uib.de/viewtopic.php?f=8&t=939#p4694).

After we fixed the prelogin issue bhearsum tried to mount from the right OPSI mount but he was not succesful either:

I tried a few things without success:
- preloginloader 3.4-27mozilla2 and 3.4-30 -- These versions successfully
talked to the config server but failed to mount the SMB share. Errors were like
this:
[5] [Oct 20 09:43:11] [action_processor_starter.exe] (2250,
'WNetCancelConnection2', 'The network connection could not be found.')
(Windows.pyo|237)
[4] [Oct 20 09:43:11] [action_processor_starter.exe] Mounting
'\\production-opsi\opt_pcbin' to 'P:' (Windows.pyo|239)
[2] [Oct 20 09:43:12] [action_processor_starter.exe] Cannot mount: (1312,
'WNetAddConnection2', 'A specified logon session does not exist. It may already
have been terminated.') (Windows.pyo|253)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] Traceback:
(Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] line 71 in
'' in file 'action_processor_starter.py' (Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] line 254 in 'mount'
in file 'OPSI\System\Windows.pyo' (Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] ==>>> Cannot mount:
(1312, 'WNetAddConnection2', 'A specified logon session does not exist. It may
already have been terminated.') (action_processor_starter.py|83)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Executing jsonrpc method
'setStatusMessage' (JSONRPC.pyo|225)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Options: {'params':
"Failed to process action requests: Cannot mount: (1312, 'WNetAddConnection2',
'A specified logon session does not exist. It may already have been
terminated.')"} (JSONRPC.pyo|229)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] Appending param: Failed
to process action requests: Cannot mount: (1312, 'WNetAddConnection2', 'A
specified logon session does not exist. It may already have been terminated.'),
type: (JSONRPC.pyo|238)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] jsonrpc string:
{"params":["Failed to process action requests: Cannot mount: (1312,
'WNetAddConnection2', 'A specified logon session does not exist. It may already
have been terminated.')"],"id":1,"method":"setStatusMessage"}
(JSONRPC.pyo|248)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] requesting:
'https://localhost:4441/opsiclientd', query '{"params":["Failed to process
action requests: Cannot mount: (1312, 'WNetAddConnection2', 'A specified logon
session does not exist. It may already have been
terminated.')"],"id":1,"method":"setStatusMessage"}' (JSONRPC.pyo|250)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Using method POST
(JSONRPC.pyo|283)

I also tried preloginloader 3.4-69, which failed to event connect the config server -- claimed that it timed out.

So yeah... you can understand if I ever say I don't like OPSI and suggest you never to get close to it.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Releng contributors from Seneca: Mozharness and buildapi

When I wrote my previous post trying to get students to contribute to Release Engineering I did not expect to have up to 8 students getting involved with us. 4 of them will be involved with the Fedora-Mozilla project.

This quarter we are going to have students contributing to the following projects:

• Add Firefox nightly/beta updates for Fedora - bug 600317
• Contribute to mozharness
• Contribute to buildapi
• Maybe some old and simple bugs

It is going to be very interesting to work with so many students as I have to be careful not be dragged down but at the same time I believe it is very valuable to both the students and our team.

You can ignore the rest of the post if you are not one of the students. Unless you are very curious to see what their options/projects are going to be.

I am going to extend a little bit on what the last 3 points involve. This should get adrianp, mustafaj, asingh, jing yan started. Please congratulate them when you see them on an IRC channel for taking up on the challenge.

To understand how our systems work checkout this new hire brownbag.

Mozharness
Mozharness is a new idea which is still being baked by aki. The goal is to pull much of the build logic that is currently intertwined on the buildbot factories (more than 40 different factories and we have passed 8 thousand lines in process/factory.py). The benefits of doing this is that it would become a standalone project which would be very easy to contribute to and run locally without having to setup buildbot.
Some problems would be that much of the API still on the works but at the same time it has a lot of value for students to face real life development and not school isolated development.

adrianp and mustafaj will be planning their development in here:
* http://zenit.senecac.on.ca/wiki/index.php/Mozharness
and here is the repo to work against:
* http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness

Here are some thoughts on where this project could get some love:

• logging options
• documentation
• add new scripts
• do a locale repackage (easy)
• setup the right environment
• download en-US nightly
• repackage with the right locale and strings
• run packaged unit tests
• download binaries and packaged tests and run them
• generating MAR files (updates)
• signing
• give love to mozharness/scripts/partner-repacks.py
• porting hgtools.py to mozharness as a supporting library
• some POCs (proof-of-concepts)

Tip on writing new scripts:

• Use the two mozharness existing scripts to understand what format to follow
• To find out what steps are used for a certain type of job you have two places to check:
• Look for logs on tinderbox. There is one for L10n as well. Each step starts with
  "======== BuildStep started ========"
• TinderboxPrint steps are metadata to show on the tinderbox boxes to be scrapped by other tools (which slave the job run on, changeset used, etc)
• Look at process/factory.py and ask a releng which script you are trying to write and to which factory it corresponds. For instance, L10n repacks corresponds to BaseRepackFactory and NightlyRepackFactory. NOTE that there is inheritance involved.

Thoughts on logging:

The most standalone item I could find in mozharness is fleshing out the logging options/todo items:

• http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/default/lib/log.py 
• http://hg.mozilla.org/users/asasaki_mozilla.com/mozharness/file/default/test/test_log.py

Namely:
 - network logging
 - ability to change log settings mid stream?  Meaning, can we have a logger that's set to INFO level; can we set it to DEBUG halfway through the script?  If not, that's ok, but I'd like to know.
 - per-module/class log settings: e.g. can we have anything that goes through a Mercurial object to be DEBUG with a timestamp, and everything else be INFO with no timestamp?
 - turn off global logging settings:  when I create two log objects, I think I end up getting duplicate log lines on screen and in logfiles, even if I'm only calling one of them.
 - generic log rotation that is configurable


Hopefully the patches would end up being generic and reusable, but optional.  For instance, if they created the SimpleNetworkLogger and MultiNetworkLogger objects that could be drop-in replacements for SimpleFileLogger or MultiFileLogger, that would be awesome.

Problems faced by aki:

I need to figure out how to keep the "generic harness" (anyone at any company using python scripts could find them useful, like log.py, config.py, errors.py, script.py) separate from the Mozilla-specific code (l10n.py or repack.py, aus2.py, talos.py, etc.) 

Porting hgtools.py

mozharness and hgtool.py will need to communicate somehow.
I was going to just call hgtool.py from commandline in AbstractMercurialScript, but Catlee wants to merge it in.
So I'd lean towards creating a generic mozharness/lib/source/mercurial.py that has generic methods.

(Generic meaning we would hopefully later be able to create a mozharness/lib/source/git.py, or a mozharness/lib/source/svn.py, and be able to use them interchangeably.  SOURCEMODULE.checkout() for example, where SOURCEMODULE is any of the above.)
We can take a lot of that from work already done in Buildbot's Mercurial/Git/etc. steps, but remove any Buildbot dependencies.  And of course Catlee has most of it written already in hgtool.py.

Thoughts on POC:

• Here are some thought of aki but I think POCs will make more sense a little later when we understand where we want to take mozharness to. aki's thoughts are good but involve setting buildbot up which I want students to avoid at first. We'll leave it for now.

I'd love to see that.  Maybe take a small project and try it in mozharness?
How about something like porting buildbot-configs/test-masters.sh + setup_master.py to mozharness?

Like |checkconfigMasters.py --list-masters| or |checkconfigMasters.py -8 --only-builder-masters| or something.

That should be a relatively small project, but potentially useful.  We should end up with something way more configurable, and verify that you know your way around the harness.
Open to other ideas for a first project too... whatever POC script should hopefully be doable in a day or two.

BuildAPI
ssalbiz (our current kick-ass intern) has taken some time to prepare instructions for our new students on how to setup BuildAPI locally.
I still scratch myself and try to understand why it is called BuildAPI because this part of the project is more of a web-based project. It is based on pylons (python web framework) and it also uses the Google Chart Tools API (I think).

asingh and jing yang would probably coordinate this in here:
http://zenit.senecac.on.ca/wiki/index.php/BuildAPI

What I need students to do is one of the two:

• generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts like this
• this is very useful and could move us forward towards having this information being published publicly for consumption
• I highly encourage this one as understanding the mental model behind it is easier
• write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)

They can get started by pulling the repo and these snapshots from August:

• http://hg.mozilla.org/build/buildapi
• https://wiki.mozilla.org/ReleaseEngineering/BuildAPI
• http://people.mozilla.org/~catlee/statusdb.sql.bz2
• http://people.mozilla.org/~catlee/schedulerdb.sql.bz2

Simple releng bugs
Out of the list of bugs that we have tagged as [simple] on the Whiteboard I have pulled the following bugs:

• bug 563941 - Fake signing for staging releases
• bug 437482 - create Mercurial bundles
• bug 510770 - make source-package
• bug 586664 - Normalize builder names
• bug 563939 - Staging release should download previous release to staging-stage
• bug 577696 - Automate sending an email to metrics team once a release is pushed live
• bug 590329 - fail early if the clobberer is broken (at least for releases) 

This work could be coordinate in here:
http://zenit.senecac.on.ca/wiki/index.php/Release_Simple_Bugs#Project_Description

NOTE: All of these bugs would require you to setup buildbot and use Dummy factories to skip certain parts of our automation. Three or four of these bugs could count as a one student project. Less than that there is no value.

[EDIT] I have fixed a couple of URLs

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How we use Narro to localize Firefox

Narro is a community-maintained web-based project (mainly alexxed AFAIK) which is hosted at https://l10n.mozilla.org/narro.

This web interface allows localizers to translate Firefox without having to deal with Mercurial, compare-locales or tools to generate language packages. It is easy to use, speeds up the translation process and improves collaboration between new contributors and veterans.

The way I have used it so far is:

• Our Armenian localizer lead visits the "Translate" tab and starts translating
• Since he is the main localizer he is also an approver and by default his changes are auto-approved
• He should also be visiting once in a while the "Review" tab to approve and discard suggestions from people on the community
• Once in a while I will visit the "Export" tab to generate a zip file (Image 1)
• I still have not figured out when to have these checkboxes checked before exporting:
• "Copy unhandled files" - I believe I don't need this one as it gives me .xml files
• "Clean export directory before exporting" - 
• I unzip the tar ball containing the Armenian repo exported from Narro on my Mercurial local checkout overwritting everything
• I run the following commands:
•  find . -type f -exec chmod -x {} \;
• modules=( browser dom extensions netwerk other-licenses security toolkit); for module in ${modules[@]}; do cd $module; for file in `find . -name "*dtd" -type f`; do cat /Users/armenzg/moz/armenian/patches/headers/header.dtd $file > $file.new; mv $file.new $file; done; for file in `find . -name "*properties" -type f`; do cat /Users/armenzg/moz/armenian/patches/headers/header.properties $file > $file.new; mv $file.new $file; done; cd ..; done
• I don't know yet on which point the permissions get changed to +x from the narro server to the zip file. According to alexxed, the Armenian project does not have +x on the files on disk.
• I also have to re-insert the license headers as it seems that Narro uses the very old headers which I have not yet learned how to overwrite
• After  analysing the output of "hg diff" I decide to commit
• My commit will add new translations that have been approved by the Armenian localization lead and new en-US strings that Narro has imported into the Armenian project
• My commit changes will be shown in tomorrow's nightly
• You can see that I name my commits with "Narro import for $date"
• You can see the status of the Armenian locale and the output of compare-locales after having pushed my changes
• The new compare-locale 0.9 can even catch parsing problems (Image 2):
• Fixing those with Narro can be real painful
• A trick is to uncheck "Show files" and "Show folders" when looking at the "Files" tab

I hope this feeds the curiosity of some people that have asked me how do I use Narro to drive the localization of Firefox.

I will explain more in other blog posts on how I use compare-locales, how I used bitbucket before we became an official localization team and how I sign-off my revision for a beta. So much to blog about!

Image 1 - This is the output of exporting a Narro project. It gives you a XPI, a diff and an exported project

Image 2 - Compare-locales 0.9 now also shows parsing errors and warnings besides missing strings

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Feedback wanted - Mozilla's Release Engineering presentation at FSOSS: draft 1

On October 29th I will be presenting at FSOSS in Toronto talking about Release Engineering at Mozilla through buildbot.

This is the first time that I am putting together a presentation of this scale and I have found preparing the right content for a presentation a difficult task. That is why I want to do it in the open to get feedback early and help me prepare something that is educative and worth the time of others.

The people attending the session will be of various sectors and not necessarily familiar to the functions of a release engineering team.

Link to draft.



Presentation's goals
My goals are the following:

• Explain in a progressive manner how our infrastructure help developers know if their changes are good
• Explain how our automation has grown and understand how much we have been able to do thanks to buildbot
• Explain the release process which is made of different teams joining forces to create a product that is delived to final users

Currently I have started to develop the first two goals as I would rather have two parts very well developed rather than three incomplete parts.

What do you think? Shall I give more emphasis to the third point? What do you think would be more interesting for people to learn from Release Engineering?

Structure of the presentation
You can look at the whole slide or look at this brief bullet point explaining the structure of the presentation.

• Brief introduction
• Cycle of a push
• Builds and results
• Scaling
• Release process
• Buildbot
• Q&A
• Wrap-up

NOTES:
I am tackling this project as an iterative process and allowing room for cutting scope without damaging the overall quality. I will provide full sources once done and provide all the images and diagrams for re-use.

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Seneca, FSOSS and Release Engineering

This quarter there are many things that are bringing me back to Seneca College for open source related reasons.

This time is about a couple of presentations and a joint project with Fedora.

1. Tomorrow I will be presenting at one of ctyler's courses ("Software Build and Release" - SBR600)with regards how our release engineering systems work.
2. Next month I will be presenting at FSOSS to a larger audience how we use Buildbot in our Release Engineering team.
3. Even more exciting is the joint project with Fedora and ctyler to setup Firefox nightly and beta builds for Fedora users.

I love presentations and I believe there are many good stories from our team that we can share.
The Fedora-Firefox repositories project is very interesting to me as it is an area that I have always wanted to investigate how it works. I also believe that we can have a benefitial impact for both Linux users and the open web.

Let's hack!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Release Engineering & Student Projects

This semester we are trying to see if we can get some students to contribute to Mozilla by working with the Release Engineering team on very diverse and interesting projects. You might not have heard of Release Engineering before. I had no knowledge of the area before I got involved as a student at Seneca. In short, a release engineer at Mozilla creates, maintains, and automates systems to deal with the release infrastructure.  Our work affects you every time you get an update with Firefox. We generate releases and updates for all different platforms; Linux, Mac, Windows and their 64-bit counterparts. Our systems also manage the continuous integration for developers working on Firefox and other Mozilla projects generating build, test, and performance results as developers commit changes and believe me they push our systems to the limit :P

We've put together a list of project we think might be interesting (and are manageable in size) to student contribution:

Teach the slave to be smarter! - The hgtools project

What if our pool of builders was better at managing all the different working directories (from Mercurial checkouts) we need at any give time?

If you look at this conversation from IRC you can see the benefits of this and a patch that has the initial work of it.

I am a releng and we are bad ass - ask me why :)

Give me a script and I will run it - The ScriptFactory project
Imagine that we did not have to touch that much our buildbot factories but instead we would maintain a bunch of script for all the different jobs we run? That would be easier and not even require our masters to be reconfigured to pick up the changes!
It would be good if we could create scripts that told a machine how to generate an optimized build, a debug build, unit tests, talos runs, locale repackages.
If you look in the tools/scripts repo you can see that we have a simple shell file to do this for the fuzzing automation. The buildbot factory that calls it is called ScriptFactory and it is very simple.





 
End-to-end project
How can we build faster and provide tests results faster to our developers?
That is what we are trying to figure out and we will be adding bugs to this tracking bug to optimize our infrastructure.

I don't like waiting - give me a CPU!
We have hundred jobs running per hour and we sometimes have jobs that have to wait before getting started. If we optimized the load we could make these jobs have to wait earlier. I will be adding bugs to this tracking bug to reduce our load on our pools and therefore reduce our waiting times.

Many many more projects...
We have a really large number of bugs unassigned and here are the queries for them in case any previous project did not call your attention:

• List of simple projects: http://tinyurl.com/25atlbw
• List of all unassigned projects: http://tinyurl.com/2cs77uo

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Firefox in Armenian: approved beta!

If you look at the L10n dashboard you can see that we are at 81% of translations for Firefox 4 and even better than that is that we our sign off for the following beta has been approved!!

We have been working really hard using Narro since we have been officially been recognised a locale for Firefox 4 a little more than a month ago.
You can see our progress in the last two weeks on Narro by looking at this screenshot that shows number of new translations VS approved translations (which :

If you would like to give us a hand translating into Armenian please visit this page and gives us your suggestions!

---

This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.