Showing posts with label win64. Show all posts
Showing posts with label win64. Show all posts

Friday, September 09, 2011

Win64 status update

Hi all,
Since my last status update a lot of things have happened:
  • The opt builds are going green
  • The builds are now showing on the developer's dashboard: tbpl
  • All branches have win64 builds (I have added the last few this morning)
    • We have try support for win64 (implied in previous point)
    • We don't have win64 support for aurora, beta, release and 1.9.2
What is it missing to be at par with other operating systems?
  • Testing infrastructure
    • We currently have 5 Win7 64-bit machines and they are only testing mozilla-central (as of today) to keep up
    • The other 50 that we had were repurposed 3 months ago for the other operating systems so we could increase their capacity by 8-10%. It was a tough call but it was necessary to keep up.
  • Debug builds
    • Symbols. It seems we are hitting a Microsoft bug. We will disable them for now
    • Packaging.
  • Symbols for the try server.
Perhaps some people won't agree that we should make the tests visible for mozilla-central since we don't have testing coverage for other branches. Nevertheless I believe that it makes sense that we could have a way of seeing tests failing rather than not at all. If it takes us 3-4 weeks to clone more machines we would be adding test failures without seeing them. I don't think it is asking for too much to try to file a bug for a test failure and carry on (even hide it) if we are not willing to back the change out or to fix the failure. At least we would have a merge to blame for or a range of pushes to help debugging the issues. If you disagree feel free to do what you think is best for everybody.

EDIT: Fixed typo 64-bi instead of 65-bit.
EDIT: For further info please follow the tracking bug.
EDIT: To try the build go to http://nightly.mozilla.org


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Monday, August 22, 2011

Win64 builds status update

Two things:
  • we now have all Win64 build machines cloned
  • we now have all branches with Win64 builds (Try coming soon)
The builds are currently not compiling but khuey and Makoto Kato are tackling each issue until we get back to compiling builds.

Meanwhile I will be setting up a subset of the pool to take care of doing Try builds.

I will give another status update when something new happens.
If you want to be up-to-date on all details please follow bug 558448 [1].
You can read my previous post for background information and

[1] Bug 558448 - (support-win64) [Tracking bug] officially support Windows 64-bit builds


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Tuesday, July 12, 2011

Firefox and Windows 64-bit builds (testing version; not a release version)

We now have a small set of Windows 2008 64-bit slaves ready to be put in our production systems that can generate the 64-bit version of Mozilla Firefox.

NOTE: This is not a released version but a testing version.




I will leave out any talking about when this could be released to our users and just focus on explaining where the project has been and where we are now from a Release Engineering point of view. Releasing depends on evaluating what the problems are on the product side before we would release it.
If you have done any comparison of the pros/cons of the 32-bit and 64-bit version of Firefox running on a 64-bit machine please let me know as I am interested to know.

Try it out
We have been producing Firefox 64-bit nightly builds since last week but we now have a small pool of machines and we are upstreaming the process to production levels.

You can give it a try by downloading the installer.

Help needed
Right now I have several bugs that I need help from developers to get them fixed.

  • bug 671000 - make buildsymbols takes 45 mins rather than 5 mins
  • bug 669384 - make buildsymbols fails for leak test builds
  • bug 670915 - make package fails for leak test builds
  • bug 670697 - sporadic make check failures 
There are many more bugs but the ones I mentioned are the ones that affect releng infrastructure.
We are using a tracking bug for product problems and another one for releng problems.

Background
I started working on this project last year on Q2 and by May 2010 I had some proof of concept going on. To my surprise the media picked up on this and brought a lot of attention to it.
By Q3 we started having problems with OPSI (a system to deploy changes to our machines) and all efforts started little by little shifting towards supporting developers to ship Firefox 4. On Q2 of this year all focus was to adapt to the new fast release cadence.

Nevertheless, by the end of March I had set up a machine to be cloned unto other machines.
Unfortunately the tools that we had in IT were not being able to clone the machine.
IT at that point started to look for a solution and in June we hired digipengi who has Windows experience (for real!).
We worked together and we had to recreate the Windows 64-bit machine from scratch with only one partition rather than three.
We are now at the point that we have 5 production slaves, another 4 to be added soon and we will be cloning the remaining ones in the near future.

If you want to know more CC yourself to the tracking bug.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Friday, March 18, 2011

How to disable Windows Error Reporting on Windows 2008

Sometimes programs crash on Windows and we all know that Windows might asks us to report back to them or just for us to be notified that something did not work properly.
This is quite good for users but not for automating jobs on machines.

I was setting up a Windows 64-bit machine to run our Firefox builds and I noticed that every time we reached the "make -k check" step the job would hang until it timed out.
I decided to run the step manually and discovered that we would get a prompt for the user to intervene.
jsapi-tests.exe crashed when running "make -k check" and Windows notifies the user

At first ted let me know that it might be related to disabling the JIT debugger (Visual Studio allows you to attach a debugger on programs outside of itself just-in-time!) but I figured out that it was disabled and this was the "post attaching the debugger" message.

I filed a bug to disable the jsapi-tests.exe crash until it gets fixed but soon after I found a post that gave me an idea.
I searched for "prevent stopped working" and I noticed this low-rated comment on stackoverflow that mentions how to disable the "Windows Customer Experience Improvement Program".
This was not what I wanted but it inspired me to look for something that would stop Windows from notifying the users of an error.
I filtered for the word "Error" (because that is what you do on Administrative tools on Windows instead of searching). After looking for a while I found "Prevent display of the user interface for critical errors" and voila! It did the trick.

Here are the steps I followed which I documented on the Win64 reference platform documentation:
  • Run "gpedit.msc"
  • Computer configuration -> Administrative Templates
  • Windows Components -> Windows Error Reporting
  • Set "Prevent display of the user interface for critical errors" to Enabled
From now on this machine won't let the user know that a program crashed and will just carry on.

Happy Windows automation!


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

How to install NSClient++ 0.3.8 on Windows 2008 x64 RC2

I am finally back to work on setting up the Windows 2008 64-bit machine to generate the 64-bit version of Firefox.

At Mozilla's Release Engineering we use Nagios to monitor our build and test machines (among other systems).
The NRPE addon (Nagios Remote Plugin Executor) is designed to allow you to execute Nagios plugins on remote Linux/Unix machines.
In the case of Windows we use NSClient++ which can be used for Nagios as its NRPE for Linux/Unix machines.

Therefore, for the Windows 64-bit machine I installed NSClient++ as we do for the Windows 32-bit machines (bhearsum did this long time ago and gave me a heads up from what he remembered - also thanks to ravi for checking things with me).

Enough background and let me show you what I did.

  • Download NSClient++ 0.3.8 for 64-bit machines
    • I am going to use the installer as it adds the firewall exceptions for me.
  • Start the installer and choose these settings:
    • "Enable common check plugins", "Enable nsclient server (check_nt)", "Enable NRPE Server (check_nrpe)", "Enable WMI checks"
  • Do not start the service and finish the installation
  • Rename C:\Program Files\NSClient++\NSC.ini as NSC.original.ini
  • Checkout mozilla/tools/nagios and copy NSC.ini to C:\Program Files\NSClient++
    • I am reusing the NSC.ini in our Win2k3 machines
    • In fact, the selected settings in the installation have no effect since we replace the NSC.ini but I thought you might be interested on having a rough idea on what to do yourself.
To check that everything went well do the following (from this documentation):
  • Run "C:\Program Files\NSClient++\nsclient++.exe" /test
  • We are going to run the following two checks (see at bottom of this post for output):
    • CheckDriveSize ShowAll MinWarnFree=10% MinCritFree=5% Drive=c:\
    • CheckCPU warn=80 crit=90 time=20m time=10s time=4
To check that everything works well from another machine do the following:
  • Add to allowed_hosts in C:\Program Files\NSClient++\NSC.ini the IP of a linux machine that has the nagios plugins installed
  • Reboot the Windows machine (we want to make sure that everything is on a clean state)
  • From the Linux machine do the following:
cd /usr/lib/nagios/plugins
./check_nrpe -H mw64-ix-slave01 -c check_load
CRITICAL: 1m: average load 0% > critical, 5m: average load 0% > critical, 15m: average load 0% > critical|'1m'=0%;0;0; '5m'=0%;0;0; '15m'=0%;0;0;
./check_nrpe -H mw64-ix-slave01 -c check_buildbot
OK: python.exe: 1
  • Now go back to the Windows machine and restore the allowed_hosts in NSC.ini to its original state
And that's it!
You can now use nagios with your Windows machine!

Here is the output of running |"C:\Program Files\NSClient++\nsclient++.exe" /test| and the two checks:
Launching test mode - client mode
d NSClient++.cpp(1178) Enabling debug mode...
d NSClient++.cpp(551) Attempting to start NSCLient++ - 0.3.8.76 2010-05-27
d NSClient++.cpp(969) Loading plugin: CheckDisk...
d NSClient++.cpp(969) Loading plugin: Event log Checker....
d NSClient++.cpp(969) Loading plugin: Helper function...
d NSClient++.cpp(969) Loading plugin: CheckSystem...
d NSClient++.cpp(969) Loading plugin: CheckWMI...
d \PDHCollector.cpp(73) Autodetected w2k or later, using w2k PDH counters.
d NSClient++.cpp(969) Loading plugin: File logger...
d \PDHCollector.cpp(110) Using index to retrive counternames
l \FileLogger.cpp(93) Log path is: C:\Program Files\NSClient++\\NSC.log
d NSClient++.cpp(969) Loading plugin: NRPE server (w/ SSL)...
d \NRPEListener.cpp(91) Loading all commands (from NRPE)
d \NRPEListener.cpp(121) Starting NRPE socket...
d \PDHCollector.cpp(130) Found countername: CPU:    \Processor(_total)\% Process
or Time
d \PDHCollector.cpp(131) Found countername: UPTIME: \System\System Up Time
d \PDHCollector.cpp(132) Found countername: MCL:    \Memory\Commit Limit
d \PDHCollector.cpp(133) Found countername: MCB:    \Memory\Committed Bytes
d NSClient++.cpp(969) Loading plugin: SystemTray...
d \Socket.h(669) Bound to: 0.0.0.0:5666
e \SysTray.cpp(51) SysTray is not installed (or it cannot interact with the desk
top) SysTray won't be loaded. Run NSClient++ SysTray install to change this.
d NSClient++.cpp(671) NSCLient++ - 0.3.8.76 2010-05-27 Started!
l NSClient++.cpp(455) Using settings from: INI-file
l NSClient++.cpp(456) Enter command to inject or exit to terminate...
CheckDriveSize ShowAll MinWarnFree=10% MinCritFree=5% Drive=c:\
d NSClient++.cpp(1106) Injecting: CheckDriveSize: ShowAll, MinWarnFree=10%, MinC
ritFree=5%, Drive=c:\
d NSClient++.cpp(1142) Injected Result: OK 'OK: c:\: 19.3G'
d NSClient++.cpp(1143) Injected Performance Result: ''c:\ %'=49%;10;5; 'c:\'=19.
32G;3.74;1.87;0;37.47; '
OK:OK: c:\: 19.3G|'c:\ %'=49%;10;5; 'c:\'=19.32G;3.74;1.87;0;37.47;
CheckCPU warn=80 crit=90 time=20m time=10s time=4
d NSClient++.cpp(1106) Injecting: CheckCPU: warn=80, crit=90, time=20m, time=10s
, time=4
d NSClient++.cpp(1142) Injected Result: OK 'OK CPU Load ok.'
d NSClient++.cpp(1143) Injected Performance Result: ''20m'=4%;80;90; '10s'=23%;8
0;90; '4'=22%;80;90; '
OK:OK CPU Load ok.|'20m'=4%;80;90; '10s'=23%;80;90; '4'=22%;80;90;


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Thursday, October 21, 2010

OPSI nightmares on Windows 2008 R2 64-bit

Back in September I worked for five days trying to make OPSI work for our Windows 2008 64-bit reference machine and failed miserably.

I reached a point where I wanted to uninstall OPSI and install it again from the right mount share. The problem is that uninstall process got the machine wedged and impossible to login through RDP. We finally figured out that we had to remove two registry keys to allow the machine to reach the login page (We used IPMI and safe mode to do this).

I will just list some comments that might help somebody searching for answers related to this. I won't put too much effort on making it presentable or coherent.

NOTE: many many thanks for cshields for having helped me through those days!

The versions (I think) we are using are:

The version for staging-opsi is 3.4.0.0 and I don't see preloginloader installed on the win64-ix-ref.build.mozilla.org machine. Nevertheless when I click on the package I get 3.4 v27mozilla2.
From bug 596735:
Blue screen before reaching login dialog
What I could see through IPMI: "Opsi Login Blocker: service request failed"
I asked on the OPSI forums and I got this answer:

Re: Cannot uninstall on Windows 2008 x64

Beitragvon wolfbardo » 17 Sep 2010, 08:19
Hello Armen,
armenzg hat geschrieben:The machine is MS2008 R2 x64 with OPSI versions 3.4.0.0 and preloginloader package 3.4 v27mozilla2.
In this version there some issues in the 64-Bit handling of Files and Registry

You should update your opsi-version to 3.4.0.14 and the opsi-packages
http://download.uib.de/opsi3.4/produkte ... 8.4-1.opsi
http://download.uib.de/opsi3.4/produkte ... .4-69.opsi

Also have a look at opsi 4.0 Release Candidate (viewtopic.php?f=10&t=1747)

armenzg hat geschrieben:
What could I try?
Delete the follwoing registry-Keys
Code: Alles auswählen
[Registry_vista_del_loginblocker] deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Providers\{d2028e19-82fe-44c6-ad64-51497c97a02a}] deletekey [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\Credential Provider Filters\{d2028e19-82fe-44c6-ad64-51497c97a02a    }]
regards
bardo wolf
The key was to login with IPMI on safe mode and remove those two registry keys.
After that the machine was good almost good (RDP was not working - jabba fixed this for me in bug 597573).

I also got another machine wedged in the exact same way but not from trying to uninstall OPSI but from forgetting to resume the opsiclientd service.

In bug 596735 comment 12 I mentioned:

This morning I found win64-ix-ref in the same condition and it is *not* from uninstalling OPSI but from having forgotten to resume last night opsiclientd service (which is mentioned here https://forum.uib.de/viewtopic.php?f=8&t=939#p4694).
After we fixed the prelogin issue bhearsum tried to mount from the right OPSI mount but he was not succesful either:
I tried a few things without success:
- preloginloader 3.4-27mozilla2 and 3.4-30 -- These versions successfully
talked to the config server but failed to mount the SMB share. Errors were like
this:
[5] [Oct 20 09:43:11] [action_processor_starter.exe] (2250,
'WNetCancelConnection2', 'The network connection could not be found.')
(Windows.pyo|237)
[4] [Oct 20 09:43:11] [action_processor_starter.exe] Mounting
'\\production-opsi\opt_pcbin' to 'P:' (Windows.pyo|239)
[2] [Oct 20 09:43:12] [action_processor_starter.exe] Cannot mount: (1312,
'WNetAddConnection2', 'A specified logon session does not exist. It may already
have been terminated.') (Windows.pyo|253)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] Traceback:
(Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] line 71 in
'' in file 'action_processor_starter.py' (Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] line 254 in 'mount'
in file 'OPSI\System\Windows.pyo' (Logger.pyo|647)
[1] [Oct 20 09:43:12] [action_processor_starter.exe] ==>>> Cannot mount:
(1312, 'WNetAddConnection2', 'A specified logon session does not exist. It may
already have been terminated.') (action_processor_starter.py|83)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Executing jsonrpc method
'setStatusMessage' (JSONRPC.pyo|225)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Options: {'params':
"Failed to process action requests: Cannot mount: (1312, 'WNetAddConnection2',
'A specified logon session does not exist. It may already have been
terminated.')"} (JSONRPC.pyo|229)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] Appending param: Failed
to process action requests: Cannot mount: (1312, 'WNetAddConnection2', 'A
specified logon session does not exist. It may already have been terminated.'),
type: (JSONRPC.pyo|238)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] jsonrpc string:
{"params":["Failed to process action requests: Cannot mount: (1312,
'WNetAddConnection2', 'A specified logon session does not exist. It may already
have been terminated.')"],"id":1,"method":"setStatusMessage"}
(JSONRPC.pyo|248)
[7] [Oct 20 09:43:12] [action_processor_starter.exe] requesting:
'https://localhost:4441/opsiclientd', query '{"params":["Failed to process
action requests: Cannot mount: (1312, 'WNetAddConnection2', 'A specified logon
session does not exist. It may already have been
terminated.')"],"id":1,"method":"setStatusMessage"}' (JSONRPC.pyo|250)
[6] [Oct 20 09:43:12] [action_processor_starter.exe] Using method POST
(JSONRPC.pyo|283)

I also tried preloginloader 3.4-69, which failed to event connect the config server -- claimed that it timed out.
So yeah... you can understand if I ever say I don't like OPSI and suggest you never to get close to it.


Creative Commons License
This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

Monday, June 07, 2010

How to build Firefox on Windows 64-bit machine with Visual Studio 2010

I thought you might want to know what are the instructions how to generate a Windows 64-bit build of Firefox with VS2010.

Setup instructions
  1. Install Visual Studio 2010 Pro (4/12/2010, vs_proweb.exe)
  2. Install Windows 7 SDK (winsdk_web.exe - v7 - 7/24/2009 )  
  3. Install MozillaBuild 1.4
  4. Download into your Mozilla Build installation directory (e.g. C:\mozilla-build) the following files:
  5. Rename or delete this file:


    • C:\mozilla-build\vim\vim72\install.exe
    NOTE: At the time of posting MozillaBuild 1.5 has not yet been released. Steps 4 and 5 should not apply with the new version.

     Now let's build
    • Start Mozilla Build with start-msvc10-x64.bat
    • Use for instance this mozconfig 
    . $topsrcdir/browser/config/mozconfig
    
    ac_add_options --target=x86_64-pc-mingw32
    ac_add_options --host=x86_64-pc-mingw32 
    # Once we fix jemalloc for VC10 + x64 - bug 569375
    export WIN32_REDIST_DIR="C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\redist\x64\Microsoft.VC100.CRT"
    •  and then:
    make -f client.mk build
      NOTE: The WIN32_REDIST_DIR is because we don't have jemalloc support for VS2010; we do have support for it in VS2008 and VS2005 (therefore you won't need the WIN32_REDIST_DIR line). For VS2005 use start-msvc8-x64.bat and for VS2008 use start-msvc9-x64.bat (NOTE I have not tested it myself).

      I hope this helps you out,
      Armen

      EDIT: I had to switch VS2005 and VS2008 in the last note of this post. I had them switched. Thanks Makoto!


      Creative Commons License
      This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

      Monday, May 31, 2010

      Pre-release Firefox Windows 64-bit builds now available (testing purposes)

      We started producing on Friday in an automated manner the 64 bit version of Firefox for Windows.

      The purpose of automating these pre-release builds is to allow developers to do work for this architecture and for  testers to use the builds and be able to file bugs. Currently, we are producing these builds twice a day and upon request from developers.



      If you go to:
      NOTE if you want to run the builds and you don't have Visual Studio installed you should install the Microsoft Visual C++ 2010 Redistributable Package (x64) to be able to start Firefox, otherwise, you will get a message saying that MSVCR100.dll cannot be found. Let me know if this works for you.

      You also have to note that Flash is not yet available for Windows 64 bit builds (currently only available for Linux).

      If you want to know what the big picture currently is you can visit this page:
      https://wiki.mozilla.org/User:Armenzg/Win64

      This was pre-announced last week in our mailing list and by Oduinn in his blog.


      Creative Commons License
      This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.

      Wednesday, May 19, 2010

      Win64 builds - what am I doing wrong?

      As I blogged before I started setting up a Windows 2008 64 bit machine that will be able to generate the 64 bit builds of Firefox.

      I have managed to build it but I have not been able to actually run it. It plainly crashes before running.

      Find below the tool-chain, mozconfig, logs and builds in case you think you can help me figure out what I am doing wrong.

      Documentation followed:
      Tool-chain:
      • MS Win2008 x86_64
      • MS VS2010 Pro
      • Win7 SDK
      • MozillaBuild 1.4
      Mozconfig:
      . $topsrcdir/browser/config/mozconfig
      
      mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir
      ac_add_options --target=x86_64-pc-mingw32
      ac_add_options --host=x86_64-pc-mingw32
      ac_add_options --disable-ogg

      Logs:
      Binaries:
      Check out the screen-shot of the crash (edited image to show problem box):

















        Creative Commons License
        This work by Zambrano Gasparnian, Armen is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License.