Planet Mozilla Interns

Lukas Blakk: Seneca and Mozilla in the news

Lukas Blakk — 2008-11-20T17:47:36+00:00

So Armen and I were interviewed a while back for the school paper - The Senecan - about our work with Mozilla through Seneca's open source courses. The article just came out, Download PDF

Wei Zhou: Service design & three Mozilla interns get together

weizhou — 2008-11-13T04:30:45+00:00

This story can only happen in Carnegie Mellon University. I always love CMU over Yale, because CMU promotes team work, I can work with people from different backgrounds here, where designers, psychologists, MBA students, HCI researchers and computer scientists get together, and we work with real clients all the time. I’m a way happier than I was in Yale.

Three ex-Mozilla interns, me, Maria, and Steve were assigned into the same team in Shelley Evenson’s service design class. It’s absolutely a coincidence, so we subconsciously keep low-key and not ware Mozilla T-shirts all three people together at the same time. So far we work very hard on making our design not suck. However we maybe embedded Mozilla’s soul into our process - a pure clean service system, with personalized service “plug-ins”. I’m really looking forward how the final prototype would be.

Yeah, we strive to be the best team in this class. GO TEAM!

Territory Map

User research & Exploratory phase

Concept development and generative phase
- Yeah, the next step is concept validation, then interface design, user-testing & prototyping. This is my first time in my life to design medical device interfaces, Hmmm…
Here’s the first small project: A self checkout service system.

Wei Zhou: Digital Self

weizhou — 2008-11-05T02:32:18+00:00

I haven’t updated my blog for a while. The FLU is really killing me. I just got to know there’re different kinds of FLUs, and I am the lucky person who is got attacked by more than one.

But nothing can stop me doing fun stuff. These are screen shots for my recent project “Digital Self”. I programmed it in processing. Haha, I have to admit it, when comes to the UI stuff, processing sucks.

Well, the basic idea, as I said before, the web browser knows more than you know about yourself. As much as I analyze the data, I realized those “Google map search” means more than just one keyword, it encapsulates all my feelings and emotions at that time. I vividly remember the moment I searched “Mountain view taxi” when my car was broken and I was anxious about being late to airport. These stories, alone with the search keywords, suddenly jump out of my mind, like I pressed the Replay button of a movie player. If we see in the near future, a software can be expressed in an art form, and the fundamental question would be “How do we enhance meaningful human experience “. The browsing history, literally becomes a backup memory library.

I didn’t do a good job because of time constraints. This application reads your local moz_places.sql file, and sorts the browsing data either by category or time. You can enter an estimation of how much time you spend online for sth, and the application will tell you how to monitor yourself better.

This is the animated beginning. Each dot represents one single site entry.(by site, not by url).

Yeah, that’s my own data. Haha. You can see I actually spend a lot of time studying and working!

Well, if you are interested in this visualization application, please email me your moz_place.sqlite file.

Anant Narayanan: EU_MozCamp and Friends

Anant — 2008-11-02T22:06:17+00:00

I’ve been doing a lot of traveling in the past few weeks, starting with the really exciting “Northside Weekend” at the Dutch towns of Tolbert and Groningen. Copious amounts of embarrassing pictures were generated on Facebook: I wouldn’t expect any less from a group of 60 students hanging out over the weekend

Allowing for just enough time to recover, I had to write my final exam for the Parallel Programming course. Having courses compressed into such short time periods can be both and good and bad, I guess I will decide for myself when the results come in!

I left almost immediately after the exam to catch a flight to Barcelona, for EU MozCamp 2008. There was a Labs Session at the Facultat d’informàtica de Barcelona the next day, part of Dan and Aza’s Europe tour. I managed to reach just before the session ended, thanks to me getting lost and then someone pointing me to a different place in the opposite direction. thunder, aza and myself then traversed the excellent Metro system in the city, to catch up with cbeard for lunch. I had my first experience with Tapas, and it wasn’t bad at all - the fact that thunder knows spanish helped a lot. mconnor joined us shortly after. A fabulous dinner later, we were all set for the first day of MozCamp

Most Mozilla events are all about meeting new people and catching up with people you knew already. I made the most of it, it was nice to see faces back from Mountain View, and some new ones. Aza was scheduled for a Labs keynote on the first day, which was pretty good. Unfortunately, he seemed to have contracted food poisoning, or maybe just general exhaustion and insomnia from the whole Labs Europe tour; so they postponed the session on Ubiquity to the next day. David Ascher’s talk was fantastic with some really cool demos, Thunderbird 3 is going to be prodigious! We had a session on Weave that evening, which also went off reasonably well, though the audience seemed more focused on the legal and privacy aspects of the project. Not a bad thing, just not interesting to me - I’d rather think and talk about technical challenges

I had the wonderful opportunity to have dinner with Mark Surman, Executive Director of the Mozilla Foundation. I also met up with Suneel who had just recently joined Mozilla at Mountain View, and we all had a great time! mconnor had to stay awake a little later than the rest of us, to prepare for his talk the next day.

Highlights of Day 2 were mconnor’s talk, 2010 goals discussion and the session on Ubiquity. I also had a brief discussion with Dan on some of my plans for Weave 0.3, it’s going to be awesome! Labs team did a little sightseeing that night, and then caught up with John Lilly, accompanied by his wife, for dinner. A perfect end to an excellent weekend.

Thanks Mozilla Europe, and Barcelona!

David Anderson: TraceMonkey and Type Instability in JavaScript

dvander — 2008-10-28T03:08:41+00:00

Work on TraceMonkey continues! TraceMonkey is Mozilla’s JavaScript JIT based on Franz-Gal trace compilation. The JIT ships with Firefox 3.1 and can be enabled in the beta via about:config. This article is about how TraceMonkey was recently changed to handle type changes in JavaScript data flow.

Introduction

The greatest advantage to a trace compiling JIT rather than a method compiling JIT, for dynamic languages, is type specialization. Observe the following loop:

function f(c) {
  for (var i = 0; i < 500; i++)
    c += 2;
  return c;
}

In a traditional JIT this method would need to be compiled generically enough to use all possible incoming types. The function could be invoked using either a string, object, number, or anything — and the compiled code must account for that. Optimizing further requires static analysis.

Type Specialization

TraceMonkey’s approach is remarkably different. If a hot path (loop) is encountered, the runtime behavior is observed, recorded, and compiled as a “trace tree.” If that loop is run using integers, the trace tree becomes a specialized chunk of assembly using straight integer math. Because trace compilation only compiles instructions that have been hit (for example, it may only compile one side to an if path), it’s much easier and faster to type specialize than doing aggressive static analysis.

This approach has worked very well so far. Except for type instability.

If that loop above were run a second time with c being a string instead, the original tree could not be used as the types are not compatible. This resulted in a “tree mismatch,” and excessive tree mismatches meant destroying the tree to make way for another. What if you could have multiple type-specialized trees for a given loop? Enter multi-trees.

Over the past week or so I have been working on (and now landed) my second major TraceMonkey patch — multi-trees.

Multiple Type Specializations per Path

TraceMonkey stores a cache of all trees. The bytecode location of each hot loop is mapped to a linked list of trees; the first in each list is called the “root peer.” Each tree records typemaps, which are a simple arrays describing the data type of each slot in the stack. A tree contains one entry typemap, which is the set of types the stack must have in order to execute the compiled tree.

The simplest case of multiple type specializations is the example above, where a simple loop’s typemap can alternate on entry. This is the easiest case to handle. The linked list is grown for each new combination of types. f(3) will create the root peer with an integer specialization. f(’a') will link in a new tree with a string specialization.

To invoke a tree, the root tree is fetched and the list is traversed until a matching entry typemap is found. The tree is then executed. If no matching typemap exists, the list is appended and a new tree using the new typemap is recorded. This means there are multiple trees recorded for the same loop, and we pick one matching the types on the stack.

Type Instability Inside Loops

The hard cases involve type instability within the same loop. TraceMonkey relies on being able to close loops, meaning that the loop edge (tail of the loop) can jump directly back to the top. This can only happen if the entry type map is equivalent to the type map at the loop edge. For example, a number turning into a string can’t be closed because the compiled code at the top of the loop expects a double, and we never recorded a path to bridge the conversion back.

There are many reasons this can happen. Take the following dumb example:

function g() {
  var q;
  for (var i = 0; i < 100; i++)
    q = 2.0;
  return q;
}

The first time we enter this loop, the entry typemap contains Boolean for q since it is undefined. When the loop reaches its first edge, q has changed to a Number. Recording starts on the second iteration, so now both the entry and exit typemaps will contain Number.

Now if g() is run again we can’t run the tree we just recorded, because it expects q to be a Number when it really starts out as Boolean. To solve this we immediately start recording a new tree. At the end we encounter a problem: in our tree, q started out as Boolean and now it’s a Number. This means the loop cannot be closed because the exit and entry points are not type compatible. Joining the ends would be incorrect.

To solve this we search for a tree X (on the same loop) whose entry types are identical to our ending types. If X exists we compile a “loop” that runs once, then immediately jumps into tree X. This resolves the type conflict by bridging the type flow to another loop. Visualization:

Figure 1

Delayed Bridging

What happens if no matching tree exists? Take this example:

function f(var c) {
   var q;
   for (var i = 0; i < c; i++)
     q = 2.0;
   return q;
}
f(1);
f(1);
f(20);

The first call to f puts a counter that says “record a trace for me next time I’m reached.” The second call to f records a trace with a type instability: q flows from undefined to Number. Unlike the previous example though, there is no stable tree to complete the bridge.

To solve this, we have the top of each tree (where the entry typemap is stored) contain a list of unstable loop edge exit points for that specific tree. During the third call to f a stable loop will be compiled. We then use the following algorithm:

1. For all other peers that have compiled code,
2. For all unstable exit points in that peer,
3. If any unstable exit point has a typemap matching our entry typemap, bridge the points together and remove the unstable exit from its peer’s list.

This algorithm is invoked every time a main trace is compiled (that is, a trace that’s not tying to extend a branch off a tree). Once it runs in the above example we a diagram very similar to Figure 1.

The important result of this is that we can bridge many types of instability together. For example, imagine a scenario where two traces are mutually unstable. This is easily bridged:

Figure 2

Other crazy situations are handled as well. For example, stable loops with unstable branch exits, or chaining multiple unstable traces together (multiple levels of mutual instability). One case I ran into was something like this:

Figure 3

Nested Type Instability

The situation gets hairy with nested trees. Consider a loop like:

for (var i = 0; i < 100; i++) {
  var q;
  for (var j = 0; j < 100; j++) {
    q = 2.0;
  }
}

In this example there’s an inner tree that will quickly type stabilize. When the outer loop starts recording, the inner loop’s incoming types will not match the existing root tree. In this case the outer loop’s recording is temporarily aborted and the inner loop immediately starts recording again under the assumption that it will record a type stabilizing trace. In this loop it does, and the two trees are connected together. The outer loop can then record and directly call into the tree that starts out with unstable types.

This solves all sorts of crazy data flow stability problems. Previously SunSpider was traced as very type unstable, and loop hoisted variables (such as the q in that example) served as tracing poison. They’re now handled fine. Our least favorite test, access-fannkuch, used to have 100,000 and even 200,000 side exits (exits from the JIT to the interpreter). It now only has 200 and it runs 2.3X faster than the interpreter (over an old 1.3X). Other cases like crypto-md5 are almost completely covered by tracing.

Thin Loops

Multi-trees go hand in hand with “thin loops,” a concept Andreas landed recently. The idea is that if a loop doesn’t close because it’s too short, we close it anyway and assume the best. That works really nicely for very thin loops, especially thin nested loops which would prevent the outer from recording. Unfortunately it doesn’t give thin loops time to type stabilize, so many thin loops are thrown out. For example:

function h(c, k) {
  var q = new String('');
  for (var i = 0; i < c; i++)
    q = k + q;
  return q;
}
h(1, 'a');
h(1, 'a');
h(5, 'a');

Multi-trees solves this. The first call to h tells the JIT to start recording the next time h is called. When h gets called the second time a thin loop is recorded. But the types are not stable, and the loop cannot be closed - we get another dangling trace that runs once and exits. When h gets called a third time it type stabilizes. A new loop is compiled and the two traces are bridged together.

Note: h is not immediately stable because new String returns a JSVAL_OBJECT in SpiderMonkey, whereas the addition returns a JSVAL_STRING.

Type Speculation

TraceMonkey has a very cool optimization called type speculation. Floating point values like 30.0 or -12.0 fit into an integer. When TraceMonkey starts recording a loop it speculates that such “promotable” numbers are integers for the entire loop. When the loop closes, if the number in that slot changed to a double, the loop can’t be closed and the recording is thrown out. How do you tell it to not perform integer promotion next time that loop is recorded?

The original solution was an oracle, a simple lossy hash table. If an integer->double conflict arose the oracle would be told “please blacklist stack slot N for bytecode location X, so we don’t try to promote it to an integer.” Since the hash table is lossy this led to false positives as traces filled the JIT cache.

Multitrees rids most uses of the oracle. If a loop can’t be closed because of integer->double conflicts, a new trace is immediately recorded with the same conflicts demoted to doubles on entry. This is better than compiling bridged traces (int->double and double->double), because the extra time spent compiling has very little gain.

There is one case where the oracle is still important (global variables aside), in that the oracle can help memoize type instability for nested loops. If the JIT cache is flushed the oracle will still remember which stack slots can’t be promoted, and outer loops can be compiled faster. Thus the oracle only memoizes stack slots when recording inner tree calls.

There is also a case where we build similar traces that are impossible to connect. For example, two variables going from int->int in one tree, and double->double in a peer tree. If a branch from the first goes from int->double for one the variables but not the other, the traces cannot be connected without some sort of intermediate conversion logic. Perhaps with very bushy trees it would make more sense to build such intermediate bridges. For now we simply throw out the tree that is causing the problems (the original integer tree). This is safe because integers can always be boxed into doubles, so the second trace is all we need.

Problems

It’s not perfect. There’s potential for code/trace explosion if types are too unstable. The heuristics for deeply nested trees can get messy too. If there are many inner trees that are type unstable it can take a long time to build a stable outer tree.

Compared to the original “single-tree specialization” TraceMonkey, there are cases that get a little slower (although still faster than with no JIT). This is because the original algorithm aggressively pegged incompatible values into mismatching trees. For example, undefined (Boolean) was passed as a 0 for integer slots and NaN for double slots. This was wrong. For example, ((undefined == undefined) == true) but ((NaN == NaN) == false). And ((undefined == undefined) == true) but ((undefined == 0) == false). Other operators are hurt too. NaN is a poison value and causes any expression to return false.

So there were cases where the compiled code was simply wrong and produced incorrect results (but boy was it fast!). By the time this fault was discovered it was too late. Removing this aggressive coercion corrected a small percentage of code at the price of greatly reducing performance. Trees would continually get trashed because they mismatched. Multitrees solved this, but in some cases it uncovers new paths that don’t yet trace well.

Conclusion

As TraceMonkey improves, type specialization will bring JIT’d code closer and closer to C equivalents. Multiple type specialized trees furthers us toward this goal. It solves a huge class of correctness bugs and improves performance in many scenarios where nested trees could not be compiled.

Lukas Blakk: The TTC on the move...

Lukas Blakk — 2008-10-24T02:03:48+00:00

This is an awesome video made with OpenGL by the guys who just did a PK on myttc.ca - an amazing visual of the transit system in Toronto throughout the day:

TTC Weekday Service (HD) from Kieran Huggins on Vimeo.

Lukas Blakk: Ready, Set, FSOSS

Lukas Blakk — 2008-10-23T16:01:05+00:00

Sitting in the workshop "ohai! art!" at FSOSS early on a Thursday morning. I'm downloading pure data (extended) to participate. Very curious to see what we'll do in this time.

Last night was the kick-off party at Mozilla for FSOSS speakers and also a time to do a dry run of the PK presentation that I'll be doing on Friday. I got a sneak-peek at Gale Moore's slides and then did a run-through myself. By setting the timing on the slides to 20 seconds I forced myself to stick to the format. It's not hard to do this when you're really just talking about your life. It's a whirlwind tour of what I call my "Zero to Hero Journey". Come by tomorrow and see it.

The best part of last night though was informal discussions about two topics I love; activism and enjoyable work environments. The former came up in discussion with Gale and Mike Hoye about the systems being used in schools. Seneca uses BlackBoard which is akin to being drawn and quartered on a regular basis. It's painful to use, ugly, has absolutely nothing to offer students, and even our teachers hardly touch it. Gale wondered why students weren't rebelling or otherwise agitating for a better system...Moodle for example. My take, granted that I am a bit different than the "average" student at Seneca, is that because of the decentralization of students - many live up to 3 hours away from campus - there is never going to be a decent level of student activism about this or any other issue because we don't spend time with each other outside of class.

On rare occasions, students may have time to compare notes with each other about their experience in their program - but without a common space for downtime, combined with an interest in making changes to the way things are, we will never join together to combat the system as it stands.

I'm 6.5 months away from being done school and it's not in my list of priorities to take on this kind of activity either. So no complaining about BlackBoard for me. I've got bigger fish to fry.

The second topic, the work environment, was fun because it wasn't just a Mozilla love-in (and there is so much to love) but instead I heard from someone in a completely different field how much their workplace supported and encouraged them, gave them a clear path to advancement and had a strong team with no weak links. I'm often griping about the team work that we are obliged to do in school, mostly because the teams are often nothing like anything found in the "real world" and then also due to the fact that the assignments are often fictitious and therefore don't give us as much of a challenge.

The open source class at Mozilla really raised the bar by giving us real work to do and that experience is invaluable. Instead of wasting time with teammates dividing up work and trying to manage other students' work ethics, we were project leaders on our Mozilla work and we pulled people into our teams as needed to get the work done. This kind of turns the model that other classes use on its ear. Next term I'll be doing a research and methodologies course where our deliverable is a significant research paper on some facet of technology. I intend to pitch an idea to research the current BSD program's project stream and how it currently leads to very little implementation. Hopefully with a little research, we can look at ways to turn that around and give the next generation of BSD students a level of satisfaction that will enable them to recognize this in their future workplaces.

Anant Narayanan: Exciting Happenings at Mozilla

Anant — 2008-10-14T00:55:36+00:00

Mozilla is making great strides towards making impacting innovations built on the open web. If you haven’t already heard, the always awesome Chris Beard was appointed as Mozilla’s Chief Innovation Officer. He’ll be heading the newly formed Developer Tools Lab which was formed after the co-founders of Ajaxian, Ben and Dion, were hired by Mozilla. That’s in addition to his existing responsibilities with Mozilla Labs and Evangelism.

I feel extremely privileged to be working with a group of such brilliant people

With projects like Geode, Ubiquity, Weave, and Snowl already on the table; I simply can’t hold my excitement in anticipation of the great things that are to come out from Mozilla in the near future!

Paul O'Shannessy: Yahoo! HackU 2008

Paul O’Shannessy — 2008-10-12T18:50:09+00:00

This past week Yahoo! was on campus for their annual HackU event. This is the third year they have been doing it and each year it has gotten progressively larger.

The Past

The first year (2006), I think there was a single entry. I made it in time for the presentation and was pretty sad I hadn’t had a chance to make anything.

Last year, I made a determined effort and hacked together my own entry – yWeather – and contributed to flolcatr. These were among the six or so entries from Carnegie Mellon. I didn’t win anything, though I’m still pretty proud of yWeather. There was a 24-hour hack session on the last night, but I’m pretty sure there was nobody there after midnight.

This year

This year there were 28 hacks! I was simply astounded at the number and variation of the entries this year. There were technically impressive hacks, humorous hacks, hacks that could change the web, and even some creepy hacks. We had some big hacker names from Yahoo! show up – Rasmus Lerdof (creator of PHP), Kent Brewster (hacker extraordinaire), and my new pal Paul Tarjan (creator of Search Monkey). I think there were people who actually pulled off the 24-hour hackathon, staying up the whole night. I know Paul was there when I left around 6am.

Last year I took advantage of the time and spent the week working on my hack. This year I didn’t start until after midnight. I contributed a little bit to a hack (“lolmonkey”) with Mattt Thompson which uses Search Monkey and creates an infobar in Yahoo! search results lolifying an image on the linked page. While not the craziest hack, we had fun with it. I think at this point we are both officially done creating tools/hacks that have anything to do with lolcats.

Busses & the Blueprint Framework

The other hack that I worked on uses the Yahoo! Blueprint framework for creating mobile applications. While not the most technically advanced hack, I think it was one with the most immediate value. It was called “CMU Busses”, and simply tells you the next 5 busses leaving Carnegie Mellon in either direction. You can see a screenshot on the project page. This is something I’ve been wanting to make for a long time, and something I would actually use (no offense to the other hacks). This was a true “hack” – I’m scraping the PAT website with Ruby. My Ruby code is outputting PHP code (the departure times are hardcoded instead of doing a DB lookup). Then the rest of the script is using the PHP Blueprint framework that Yahoo! provides (after spending a while working it out).

I had originally planned to make this an iPhone application, but having never done Objective-C, I figured now wasn’t the time to start. I had also wanted this application to be able to detect the user’s GPS coordinates and find the closes 5 bus stops before showing departure times, but that had some barriers as well. In a future iteration hopefully we’ll see that. The Blueprint framework is supposedly also able to convert a Blueprint application to an iPhone application, but that has not been released to the public yet. When it is released, I think I may publish what I have now as a first iteration.

Some Other Highlights

I got to hang out with some great people throughout the week. I also got treated to a dinner for all of the former Yahoo! interns at Fuel & Fuddle. It was a good time, getting to talk some technology and design with smart people.

As a result of working on the lolmonkey hack, Mattt and I were invited to dinner by Alicia at Bug Labs. They have a really cool product called BUG, which as they say on their website, “is a collection of easy-to-use electronic modules that snap together to build any gadget you can imagine”. I got to play with one and it’s really neat. I wish I had some spare cash to get one. They also have some of the coolest package design I’ve seen in a while. So if you’re interested in hacking hardware, take a look at them – they’re cool people with a cool product, hacking on open source hardware and software.

Brian Krausz: PHP is a /great/ first language…for a hacker

Brian — 2008-10-06T09:39:53+00:00

So I figured I’d jump on the blogger bandwagon in the debate on whether or not PHP is a good first language.

First, a little context. PHP was the first language I did anything useful in. I learned QBASIC in middle school, which I used to make some neat games. Around 7th grade I taught myself PHP by hacking on phpBB installations. I didn’t learn any other language until college, where I quickly picked up Java, C, and SML (in that order). I agree that PHP is a horrible language to teach proper programming practices, but that’s not the point of it.

I’ve had the privledge of talking to Rasmus a number of times over the last few years, and while I didn’t get permission to reproduce his comments and so I won’t (I will say he has some funny stories about David Filo…), I can quote an article he wrote approx. 4 years ago on PHP:

PHP was never meant to win any beauty contests. … It was designed to solve a single problem: the Web problem.

It does this very well, as shown by it’s outstanding adoption rate. The truth is that PHP is hacky, but it’s hacky in a logical way. It was written to help a programmer have an easier time writing for the web by one in that exact situation, and it does it’s job. If you need to throw a page together in 5 minutes, you can do that! Which is exactly what budding programmers should be exposed to.

Think of it this way: why do people learn programming in the first place? Nowadays, it’s either as a career path, in which case they probably aren’t destined to be hackers in any sense, or because they need to solve a problem and writing a program is the best way to do it. What is going to turn a newbie on to programming faster: learning proper structure, or being able to pump out a solution to his/her problem in 5 minutes? I would bet a lot it’s the later. That’s what hackers do, they pump out a solution to a problem quick and, quite possibly, dirty.

“But Brian, have you ever seen dirty code?” you ask. Of course I have, and I despise it. But I contest that proper style is something that can be learned after the fact. The ability to pump out a prototype in 5 minutes can’t be learned as easily. The ability to ignore all convention and generate an import script to move all of your data from A to B to get your site back up, or hack together a script to bulk rename files in a really wonky way, while trivial to do for most programmers, can be done much faster with a hacker mentality.

I am a hacker, but I have no problem following convention. I don’t like to, since it slows down coding, but when I’m in a group of programmers, or working on code that needs to be maintained, I do recognize the point of doing so and follow them (to the letter even). Of course, I’m by no means the norm on anything, so take what I say with a grain of salt, but it seems to have worked well for me: I can follow convention, but at the same time I can adjust, I can change my style as the situation demands, and I can code up a storm when need be.

Lukas Blakk: Third time's a charm - Unittest Production moves tomorrow morning

Lukas Blakk — 2008-09-23T20:22:45+00:00

Wednesday September 24th at 6:00 PDT I'll be connecting the new slaves to Tinderbox:Firefox and taking down the old ones.

Fingers crossed that it really happens this time.

Lukas Blakk: Postponed - Production MozCentral Unittest Moves - to the Build Network

Lukas Blakk — 2008-09-23T14:03:40+00:00

As you may already know, there is some major outage (power out at the San Jose colo) this morning - now dubbed "Black Tuesday" by me. So the switch is postponed again.

New time and date forthcoming when the network is up and stable again.

Lukas Blakk: Take 2 - mozilla-central unittest production waterfalls moves to build network

Lukas Blakk — 2008-09-22T18:09:59+00:00

Hopefully things will still look good tomorrow morning because the new time of the move is now:

This will take place on Tuesday September 23th, at 7am PDT

The new production buildbot will already be up and running smoothly
(currently reports to the UnitTest tinderbox tree) so there should
hopefully be very little impact when this switch over happens.

Basically, the new unittest buildbot slaves will start reporting to
the Firefox tinderbox tree, and the current slaves will stop reporting
there.

This means you will be looking at new slave names. There are 2
buildslaves for each platform and their names are as follows:

Linux mozilla-central moz2-linux-slave07 dep unit test
Linux mozilla-central moz2-linux-slave08 dep unit test
MacOSX Darwin 9.2.2 moz2-darwin8-slave01 dep unit test
MacOSX Darwin 9.2.2 moz2-darwin8-slave02 dep unit test
WINNT 5.2 mozilla-central moz2-win32-slave07 dep unit test
WINNT 5.2 mozilla-central moz2-win32-slave08 dep unit test

Wei Zhou: Internet Self

weizhou — 2008-09-21T05:12:02+00:00

I’m recently obsessed with human identity, perception and self-expression. In Stacie’s course ‘ Information and interaction’, I proposed to do a 6 weeks project naming “Digital Self”. The goal is quite simple: Explore the difference between your real self in the society and your digital self on the Internet. The method is even simpler: by analyzing people’s daily browsing activities. I plan to build this piece by Processing(as usual), the final delivery will be an “artistic” interactive piece - I’m suffering with CMU’s rigorous design research, usibility tesing and seminar reading education, it’s time to have some fun.

So what’s a Digital Self and why start from a web browsing history analysis?

People may scared if they realized that their browser knows themselves more than they do. The browser traced down every single activity you do on the Internet, that somehow reflects you as a whole slightly different than you are in the real world.

For example, here are some variables that may reflects your “Internet” personality.

Your most frequent visiting sites.

- If I ask you to list some favorite sites, the answer you give me might be different than the answer given by your browser. You may not realize why you actually spend 30 hours on Facebook monthly, that conflicts with your goal - study harder.

How long do you stay at a site

- Spending 10 hours shopping on Amazon and 10 seconds to check things on Amazon reflects different personalities. If we did a long-term research, you may find you are easily obsessed with one thing, or you may like to change your focus frequently. That’s a very interesting Internet personality.

Your browsing time

- This directly reflects your schedule, how busy you are.

Sites category you visited over time

- When you are relax and happy, you will visit a lot of sites you like, otherwise, if you are hitting a deadline, you will only visit sites related to your work. However the sites people visit reflects how they release their pressure, such as watching a cartoon after a long time work.

Your search result

- Reflects your curiosity

What do you tag or bookmark a page

- This may reflect your Internet self in a social context.

What much do you comment or upload things on Internet(In versus Out)?

- If you only get, never give, does that mean you have a selfish personality on the Internet?

…

It will be interesitng to build a framework for Internet Self Study, but I think that’s phychologists’ job.

Anant Narayanan: Meme(me)

Anant — 2008-09-20T23:58:12+00:00

Another meme has begun:

1. Take a picture of yourself right now.
2. Don’t change your clothes, don’t fix your hair… just take a picture.
3. Post that picture with NO editing.
4. Post these instructions with your picture.

Lukas Blakk: Postponed - Production MozCentral Unittest Moves - to the Build Network

Lukas Blakk — 2008-09-19T14:09:44+00:00

Due to some glitches in the buildslaves, the move of the mozilla-central unittest build master has been postponed for now.

I will post again soon with a new time and date for the switch.

http://groups.google.com/group/mozilla.dev.planning/post

Lukas Blakk: Production MozCentral Unittest Moves - to the Build Network

Lukas Blakk — 2008-09-17T21:53:50+00:00

So the last task in the move from QA network to Build network is moving the production mozilla-central unittest boxes.

This will take place on Friday September 19th, at 7am PDT

The new production buildbot is already up and running smoothly (currently reports to the UnitTest tinderbox tree) so there should hopefully be very little impact when this switch over happens.

Basically, the new unittest buildbot slaves will start reporting to the Firefox tinderbox tree, and the current slaves will stop reporting there.

This means you will be looking at new slave names. There are 2 buildslaves for each platform and their names are as follows:

Linux mozilla-central moz2-linux-slave07 dep unit test
Linux mozilla-central moz2-linux-slave08 dep unit test
MacOSX Darwin 9.2.2 moz2-darwin8-slave01 dep unit test
MacOSX Darwin 9.2.2 moz2-darwin8-slave02 dep unit test
WINNT 5.2 mozilla-central moz2-win32-slave07 dep unit test
WINNT 5.2 mozilla-central moz2-win32-slave08 dep unit test

Thanks to Jesse Ruderman and others for working so hard on bug #450637, as it means we are now running unit tests on win32 VMs with very consistent results.

As soon as this is done, and working well I can tie up the loose ends, close out some bugs, update the config files and start looking at some new projects...like the test results - and try server unittests.

Bobby Holley: So Many Colors

bholley — 2008-09-12T04:24:14+00:00

This summer, I’ve been working on getting Mozilla’s color management backend ready for the prime time. We’re finally turning it on in tonight’s nightly builds, so I thought I’d give a bit of background on the history of color management in Mozilla and on color management in general. You should all go download the latest nightly when it comes out and give it a try.

In a nutshell, color management involves using information about a computer’s display, and in some cases information about certain images, to render pages more accurately. Contrary to popular belief, an RGB value (such as #ffffff) alone does not represent a color. Rather, the values for the red, green, and blue channels can be thought of as coordinates in a color space that contains the actual colors. The color that an RGB tuple represents thus depends on how the color space is defined (see Wikipedia for more information).

This type information is generally stored in a file called an ICC color profile. Computer displays often ship with ICC profiles that specify exactly how various RGB values will look on the screen. Additionally, JPG and PNG images are often tagged with ICC profiles to describe the color space of the pixels. In theory, every image should come with a color profile specifying its color space. However, many images do not contain embedded profiles, nor would it make sense for a CSS stylesheet to embed a color profile for CSS colors. As a result, the W3C determined that untagged web content should be assumed to be in a standard color space called sRGB. So color management in Mozilla is really a two-part process: we map incoming RGB values out of their respective color spaces, and into the color space of your display (which we obtain by querying the operating system).

Colors are more complicated than you might think.

About a year ago, Mike Beltzner unveiled Mozilla’s color management support in a blog post. Unfortunately, there were some major issues that kept the color management mode set squarely to ‘off’ for the release of Firefox 3.0.

First of all, there were quite a few rendering bugs filed for our color management implementation. I was able to confirm a good number of them, and discover a few more myself, and I’ve landed the necessary fixes. There were also quite a few issues with our Reftest rendering test suite that needed fixing so that the tests would pass with color management on. These issues have also been fixed. However, several people reported bizarre rendering behavior (whites showing up yellowish, blues showing up purple) that turned out to be the result of bogus display profiles on the reporter’s system (many thanks to Chris Murphy for lending his expertise on this issue). This is scary, because it raises the possibility that any number of people might see inexplicably weird colors in their browser once we turn on color management. As such, we’re hoping to get as many testers as possible in the nightlies and betas to see how widespread the problem is.

Another concern we had over color management had to do with plugins. Since plugins like Flash and Java do their own drawing, we don’t have any control over whether they use color management. While an extension to the plugin API to allow this is in the works, there’s currently no way for the browser and plugins to coordinate on color management. If the browser color corrects a color while the plugin doesn’t, they no longer look exactly the same side by side. At this point, it’s unclear exactly how much of a problem this is, but we’d appreciate feedback.

However, the biggest reason color management was disabled for Firefox 3.0 was that it regressed performance by about 20-30% (according to our Talos tests) when switched on. This was deemed unacceptable for a default setting. As such, I spent a great deal of time profiling code and then devising, implementing, and testing dozens of strategies to squeeze the performance overhead as close to zero as possible.

This turned out to be really hard. Color management is a per-pixel transform, meaning that we need to call a function for every pixel of an image (CSS colors are easier since repeated pixels are known to the rendering code). Such an embarrassingly parallel problem would be a perfect candidate for GPU acceleration, but we’re just beginning to draw up plans for OpenGL and Direct3D backends to Cairo, so it will be quite a while before we can just write a custom shader and make the entire problem disappear. As a result, I dug in my heels and worked on making the transformation of one pixel really, really fast. After some rearchitecting, various precaching strategies, and a whole lot of hand-tuned SSE2 assembly, the performance overhead of full-page color management is finally at about 4-5% on all platforms.

For the past few months, this seemed to be enough. Beltzner was receptive to the idea of turning it on, and nobody in either of the development meetings raised any gripes on the issue. Robert O’Callahan and David Baron had voiced some concerns about standards compliance and interoperability, but both were willing to turn it on for Firefox 3.1 Beta 1 in order to get feedback from the community. However, as Vlad and I were landing the final patch to turn color management on, we were beset by a flurry of naysayers who didn’t feel like the performance hit was worth it.

As a contingency, I had previously implemented a third mode for color management, one that color-corrects tagged images only, that has a performance hit of about 1%. Safari has been color-correcting tagged images for a while now, and this is certainly the dominant use case for color management. Given the pushback on full color management, it looks like we’ll have to settle for a default of “tagged only” for now. The capability for full correction is available in Firefox, and hopefully we’ll have an easy way to switch modes by the time 3.1 rolls out the door. For now, you can go to about:config and set gfx.color_management.mode to ‘0′ for no color management, ‘1′ for full color management, and ‘2′ (the default) for tagged-only correction (for technical reasons, you’ll need to restart your browser for this setting to take effect). Deb Richardson put together a great blog post where she highlighted some of the differences between color-managing and not color-managing tagged images. I’d encourage you to check it out.

Color Management Off, Color Management On (Courtesy of Deb)

Personal disappointment aside, I think it would be a good idea to open up discussion on the subject of full color management on the web. In a world where every browser supported color management and nobody had bogus display profiles (idealistic, I know), a web designer could pick a color scheme for his or her web page and rest assured that it would actually look the same for each viewer. Whites would really be white, and that soothing hue of green would look just as good on your mother’s laptop as it does on your desktop. I’d encourage people to try browsing the web with gfx.color_management.mode=1. Is the performance impact noticeable? Do you run across any web pages that don’t look right? How many color-matched plugins do you see on a daily basis? Do you care about color correction? As always, Firefox is your browser, and we want to know what you want. So tell us.

Brian Krausz: Antiantisocial Networks

Brian — 2008-09-06T02:09:45+00:00

Today TechCrunch reported on a paper describing a way to use Facebook for malicious means. The paper describes a DDoS attack that can be done, leveraging the large number of users of an application to attack a victim site.

While this attack vector is legitimate, I see a number of things that make it inherently infeasibly, and don’t think it really warrants being called a “FaceBot” (implying similar power to a botnet).

In order to create an application, one obviously needs to create a Facebook account, though that can be done anonymously. The real issue is that in order to execute such an attack, one would need to make an application that is incredibly popular. The attacker would need to devote a large number of resources to keeping such a popular app up, which would all need to be done anonymously (though would need to be paid for in one way or another).

Let’s say an attacker has gone through all of this to make a popular application: why doesn’t he/she just use those resources for a direct attack? One possibly answer is that the Facebook DDoS would be hard to shut down, or better in some other way in executing the attack. This is false because as soon as someone realizes that their traffic is coming from Facebook (whether by referrers, or FB trying to pull images for its cache, or some other mechanism), it can in most instances be stopped immediately, especially considering how most Facebook calls to other sites include the application’s API keys. Even barring that, IP addresses and Facebook’s logging can be used to determine what application a user was in when they requested the victim’s site.

Additionally, DDoSs using this attack vector are relatively easy to mitigate. If a hacker already has all of these resources dedicated to keeping an application up, why wouldn’t they just launch a TCP SYN flood or similar lower-level attack, much more potent DoSs, even if launched from a more limited IP range.

Let’s take a different route: suppose a hacker attacks one of Slide’s applications and somehow manages to break in and add an attack iframe. This is a completely legitimate and anonymous way of attacking a site (though it begs the question of why the hacker didn’t just break into the target site in the first place, assuming both have similar levels of security). While this is a legitimate issue, the same holds true for all websites. Should someone hack into Yahoo! and figure out how to deploy a new home page (somewhere between almost-impossible and no-freaking-way on the difficulty scale), almost any site on the internet could easily be taken down. I certainly hope top app developers take security as seriously as top website owners, but this is nothing special for Facebook.

On the topic of information theft, this is why Facebook requires you to explicitly permit an application to access your information. The concept of an API implies this potential for theft…users are trusting applications to access their information and not keep it. There is no way to prevent this for the same reason DRM doesn’t work: if people can view things they can store things. While this is a legitimate concern, again it is nothing new, and not much can be done about it short of user education.

Rob Arnold: Science of the Web

Rob Arnold — 2008-09-03T15:32:51+00:00

My friend Dan Schultz is currently taking a course called “Science of The Web”. For one of his assignments (see problem 2), he needs as much help as he can get. Here’s where you come in. It’s quick and easy:

1) Go to http://boom.aladdin.cs.cmu.edu/cgi-bin/ipaddy (the server might encounter an error, just refresh and it should work)
2) Enter ‘dschultz’
3) Get as many other people as possible to do the same

Wei Zhou: Bauhous and Google products

weizhou — 2008-09-02T21:53:14+00:00

Actually I like Google products over a lot of others. But there’s something out there makes me not a big fan of it. Eventually I cannot endure it anymore and decide to write something. I think Google, being considered as one of the leading software companies in the world, has the responsibility to make their product user experience not suck, not only in terms of usability, but of aesthetics.

If everybody is using Gmail, Google Calendar and Google Doc, we begin to lose our control over our choices of web applications, just as we have to use Windows OS. So welcome to a software modernism age. Welcome Google’s fast-food web apps. Keep Google Bauhaus design doctrine in mind: “Our Web Apps are so simple, secure and fast, that’s all you need for your Internet.”

Is simple, secure and fast all we want? If design is about the value,not the truth, where can we target our values in this fast-food software culture? When can a piece of software become a form of art, bringing people altimate enjoyment?When can a web Apps functions beyond a tool, behave like a digital being or digital asistant that help us in an interesting and considerate way? Can softare have personalities, catering to their master(user)’s ever-changing needs and interests? Google basically ignores all of those possibilities and proves itself proudly: A group of engineers, make softwares, work hard to make every engineer happy, and change every Internet user to a engineer.

Wei Zhou: Master of Tangible Interaction Design & Gold Rush

weizhou — 2008-08-29T18:15:11+00:00

Recently my school(CMU)announces a new program: Master of Tangible Interaction Design. It’s a one-year master program, no GRE requirements, no need to have any design backgrounds, no Thesis project or thesis paper, no CHI submissions, and under the school of architecture. I regrect that I chose my current program -Interaction Design(well-known for its user-centered design theory and practice), for which I had to do everything the other program doesn’t need, for which I dropped Yale, and declined RCA and CMU HCI’s offer. It really hurts, we spent so much money and energy to define the word of “interaction design”, we worked so hard to distinguish interaction design/user experience design with other subjects, and a new name-misleading program simply ruin it all.

Paul O'Shannessy: Making the New Password Manager Storage Faster

Paul O’Shannessy — 2008-08-29T01:01:47+00:00

When I first wrote the new storage module for the Password Manager, I took a few “shortcuts,” trying to keep my code DRY. Partially this was because of the first patch by Mrinal Kant, but mostly it was because I like to reuse code. This bit us just a bit.

I mentioned in my first post about this change that we were initially considerably slower in the critical countLogins method. While it got improved before being checked in, it was still marginally slower (milliseconds on an abnormally large dataset).

As I said before, this was most likely since we were doing a SELECT * on the moz_logins table, and looping over the results and counting. This allowed me to reuse more code. Loops are a kind of slow, and since this was so important, I decided to speed it up.

I filed a bug just over a week ago entitled “storage-mozStorage should use COUNT in countLogins” – which pretty much explains what the solution to the above problem. I created a patch which essentially just switched the mosStorage module to use SELECT COUNT(1). I reran the performance tests I created and we’re doing much better now. There’s still a miniscule loss in speed from the legacy storage module, but at this point, we’ve done all that we can, and where the difference was milliseconds, its closer to millisecond.

This was checked in today (thanks Justin!).

And that’s it. I have another patch in the pipeline and hopefully I’ll have time to get it finished, approved, and reviewed for the freeze (whenever that is now).

Wei Zhou: Scratching the surface

weizhou — 2008-08-27T18:36:46+00:00

I have been using iPhone for three weeks now. Here’s a normal iPhone user daily schedule.

7:00 AM: iPhone Alarm wakes me up
7:01 AM: Check facebook, twitter, mySpace, Blog, School email, Gmail, company email, open MC chats, log into msn and AIM, Gmail, skype(wifi), IRC(wifi).Quickly check TODO.
7:05 AM: Fully wake up. Hold iPhone and run to the bathroom. Listen to Stitcher’s new radios. Cook breakfast, check iPhone for nutrition info.
7:30 AM: Bring my iPhone(and my iPhone charger!), walk to the bus stop, read Newsstand RSS feeds, check Stanza, Jamed, NetNewsWire, and Shovel, digg news and comment on other people’s blogs.
8:00 AM: On the bus stop, search iWant for local bus schedule.
8:30 AM: Class begin. Use Note&Sketch to take notes, use FileMagnet to view course outline.
8:35 AM: Bored. Check iGotchi and feed my pet.
9:00 AM: Teacher forgot my name. Use Shout it, display “Wei Zhou” and wave to him.
9:30 AM: Use Camera to take a picture of my teacher, publish through ShoZu, send it to facebook, flickr, MySpace, WordPress and Picasa.
10:30 AM: Class ends, update my Diary. Play Tris, Tap Tap.
11:30 AM: Run for another class, focus on teaching, Camera bad students sleep in class.
12:00 AM. Break. Use VoiceNotes to record student’s requirements.
1:30 PM: Starving. Check Urbanspoon for food.Drive up there using maps.Turn on Pathtracker because I’m directionally challenged. Using where to find Starbucks.
2:30 PM: Graffitio notify me another CMU friend in Starbucks is using iPhone as well.Exchange cellphone number with him.Textmessage another friend to join us. Send him our location using Over Here. Share Loopt with them.
3:00 PM:Another class. Bored. Use Lifecast to record class discussion.
5:50 PM: End class. Repeat 1:30 PM.
6:30 PM: Another class. Explore new Apps on App Store.
8:30 PM: School ends. Walk back home. Study english using Blanks. Update TODO. Repeat 7:01 AM.
9:30 PM: Homework time, use SnatchTest as a mouse for my Macpro.Use controller for my iTV and iPod player.
10:30 PM: Play Ambient to get to sleep.

P.S.: Repeat 7:01 AM many times during the day.

What else do I need?

Support remote printing.
Every time I jump from one App to another App, the previous one stops working, I have to reopen it afterwards. That really stops my work-flow. I wish it works like Tabs in a browser(Or even better). In summery, the navigation is bad.
Constantly pressing the main home button is annoying. Especially when I lay on the bed, holding the iPhone with one hand, I don’t want to move another hand. Can I just do everything use one hand?
I wish I could track my iPhone using history.
Searching is painful. iPhone should have an universal Google search button on the device.
Typing is painful. I need a auto-type App.
How can I copy and paste info from one App to another???!!!
I need a Firefox browser App

How to design a Multi-touch interaction framework from ground up? IPhone just scratched the surface of a smart phone design - for me iPhone is a device that combines a bunch of unrelated separate gadgets together. It’s getting there, but hardly to be considered as “Smart”. Multi-touch gesture standards will be set up within next 5 years. What we need is no longer a “smart phone“, but a smart browser that embedded in a mobile device, In this way we would be able to use a series of small Apps in a meaningful combination, adjusting to people’s ever-changing task flow and context, without interrupting people’s thinking model over time(also see Ubiquity). That makes a digital device really become part of our body, like our hands and feet.

Yes. Building a variety of cool Apps is important, but not as important as arranging them in a useful way(Outsourcing part of human thinking to machines?). That applies to Small screen browser design, as well as browser for OS. Human beings are task-oriented, not tool-oriented. Guess that’s the difference between a human and a computer.

As a designer we usually need to answer three questions:”What”, “Why”, and “How”. Here I delve into what and why, not much how. I found Aza’s blog particularly interesting - Ambient information. Because it also talked about how. We need more “Hows”.

David Anderson: SpiderMonkey + Tracing = TraceMonkey

dvander — 2008-08-22T20:00:32+00:00

Firefox has a JIT for JavaScript now. Whoa. Before I continue on, here are links to the blogs of other Mozilla people close to the TraceMonkey team:

So, I started at Mozilla by working on Tamarin-Tracing. Tracing is Andreas Gal’s fancy new idea for run-time guided JIT optimization, a powerful new concept that poses huge benefits over whole method compilation. I talked about this before, perhaps erroneously, but the concepts are there.

The old style of compilation is to perform static analysis on entire methods at a time, compiling them to assembly when necessary. Without running the program, you decide how to compile loops and nested loops efficiently, perhaps even trying to decide if they’re expensive or not. Methods may or may not be inlined, but they are still the fundamental building block of most compilers.

The concept of methods quickly disappears in a tracing compiler. Everything is inlined as the tracer only compiles exactly what low level operations it sees being performed (and any sort of control flow is essentially a no-op). A tracing JIT essentially turns an expensive loop into its own isolated method call, optimized for its run-time properties, regardless of where it is or what the loop has to call into.

Andreas’s original paper on tracing was targeted toward mobile performance, where whole-method compilation and static analysis are too expensive. For dynamic languages one instruction can have many decision paths at run-time. Whole method JITing is a real problem because the code required for each opcode becomes very large, and static analysis is either unfruitful (because of dynamic types) or just too expensive. This is especially problematic for JavaScript where browser performance is critical, and time spent analyzing code is time wasted.

Thus it’s no surprise that Adobe decided to try tracing in the next generation of their Tamarin project. Adobe’s approach to tracing ActionScript is to create very primitive building blocks and trace those at the lowest level. It does this by converting ActionScript bytecode to a Forth dialect, and tracing the primitive Forth operations.

Mozilla’s JavaScript engine (”SpiderMonkey”) is very different. It has a decade worth of optimization hacks and very “fat” opcodes (instructions that have a lot of internal decisions, rather than performing one single operation). Although there were originally plans for Mozilla to switch to Tamarin, throwing out SpiderMonkey had a lot of hurdles, and the TraceMonkey project was started instead.

Luckily Adobe had very nicely separated the tracing backend from their interpreter. Tamarin-Tracing has a “nanojit” component with a simple IR. Interpreters are responsible for emitting the IR, and nanojit can compile straight-line IR blocks into native code. It can also link compiled code fragments together for attaching branches and building trees of traces.

Using Adobe’s nanojit, Andreas decided to take a top-down approach to tracing SpiderMonkey. The edge of every loop is monitored. If a loop is executed enough times, the tracer is activated. Every opcode is hooked and critical decision points are emitted as nanojit IR where possible. When the control flow reaches the loop edge again, the IR is compiled and the loop will run as native code thereafter.

There are some fascinating aspects to Andreas’s work. Type speculation and specialization, the native stack versus the script stack, tree specialization, his handling of global variables — are all intricate and critical to the rapid progress and success he’s made on TraceMonkey. And he (and Mike Shaver and Brendan Eich) did it all in 60 days, which is amazing.

What’s my role in all this? My summer intern project was porting the code generator and tracer to AMD64, which has landed and seems to work in the shell. I’ve also been debugging anything that goes wrong on the 32-bit port. Working with nanojit was a lot of fun - Adobe did a great job making it usable by other projects, and it’s definitely something that could become a generic library for dynamic languages to use for tracing.

The big news today is that TraceMonkey has landed in mozilla-central and will probably be turned on by default for Firefox 3.1 beta 1. Although it was open source and downloadable during development, it is now being officially announced and publicized, and can be used in the official nightly builds. The speed difference is noticeable in sites doing intense JavaScript processing. And though the SunSpider benchmarks can be considered superficial, it’s great to see the improvements we’re getting on them versus the old SpiderMonkey.

This is just the beginning. A lot more is planned for TraceMonkey and for tracing in general. Code that used to be considered too crazy for JavaScript, like graphics and crypto loops, is becoming plausible. We’re already noticing smoother play quality in some 3D JavaScript games on the web (using Canvas) and in other heavy applications. In Mike Shaver’s words, this could change the way people use JavaScript.

In terms of portability, nanojit still needs a bit of work, but it’s been hammered into shape for AMD64 for the time being. It can use the extra eight registers available with REX prefixes, and it will perform 32-bit integer math versus 64-bit pointer math correctly (given using the correct LIR instructions for safety). I also took the liberty of prettying up the macros used for code generation. Some of the work remaining that I’d like to do in terms of the overall x32/x64 assembly process:

Taking more advantage of addressing modes — we can reduce register pressure by combining redundant store/load ALU operations.
Improving SSE2 logic which currently uses LAHF/PUSHF.
Improving calling conventions for SSE2 and reducing register spilling.
Inheriting type information from child instructions, to remove the need for separately typed IR instructions (i.e. no need for add versus fadd versus qadd).
Enabling 64-bit jitting in the browser (too unstable right now so it’s only on in the shell).

I should thank Edwin Smith at Adobe for putting up with my intense nanojit nagging; Mozilla for giving me the opportunity to work on this project as an intern; and Andreas Gal for coaching me through the tracing concepts every time I got them wrong.

For people who follow this blog from the SourceMod project, will SourcePawn get tracing? It’s something I’m experimenting with and will talk more about later. There are some hurdles to JITing Pawn in that very careful escape analysis is needed to make any of the nice optimizations.

Paul O'Shannessy: Password Manager now uses mozStorage

Paul O’Shannessy — 2008-08-19T03:55:43+00:00

The other day my patch landed switching the Password Manager to use mozStorage (our wrapper around SQLite). The bug had been up on Bugzilla for a long time, over 3 years, when I came across it at the end of June. I had been doing some Password Manager related work already by that point, so I decided I would do it. This is the story of that bug: the process, the hardships, the code (at least a bit). Keep in mind I was also doing work on my DTrace Treemaps at the time, went to Summit, and encountered more edge cases than I wanted, so this took longer than expected.

Quick Features & Change Summary

Some of this is discussed further down, so bear with me. One of the primary differences in the switch to using mozStorage is that we now store data in a database. Previously we were storing data in a text file, using lines and periods to separate data fields. Open signons3.txt in your profile directory and take a look at it (assuming you’ve saved a password before). All information was kept in memory, and when a new password was saved, the whole file would need to be rewritten. The same thing happened if you ever removed a saved password. Using a database means that we don’t have to keep any (potentially) sensitive information in memory. It also means faster reading and writing since we don’t have to read the whole file every time. These speed boosts are apparent especially in the speed tests, attached to the bug and summarized below.

Since this is really just a drop in change that must implement an API, to the outside world nothing has changed. Although the inner workings are different, it’s the same to anybody who happens to use it (extensions or other parts of Firefox).

v0.1 – The beginning

I began work by really taking a look at the legacy storage module to make sure I knew what was supposed to be happening. Then I looked at the initial attempts by other developers. The first attempts were made before Password Manager got rewritten, so those weren’t relevant. Mrinal Kant came in (over 2 years after the first patches) and wrote what I used as the basis for my code. I don’t think I ran it as it was, but it looked functional, at least at the core.

I actually started by just copying the legacy storage file, removing all of the code from methods that would need to be changed, and started fresh. I copied in some code from Mrinal’s work and used some of the conventions, but the bulk of it was rewritten. I opted to use the wrapper we have for Storage, which makes it easy to do parameter replacement. This also automatically binds the parameters to a type – so when you give it a string, it will ensure it’s treated as a string. It’s very handy.

This first version “worked” (at least as far as I remember), though it definitely had problems. I brought it up at our weekly status meeting and it became one of the “nice to have” features for 3.1. That gave it some attention it needed and I got some quick feedback from Shawn and Justin.

v0.2 – Database details

One of the primary changes here was some of the database stuff. Shawn had pointed out that I needed a way to version the database. All I had was a method to create the tables. In order to future-proof this, I needed to make sure the schema was stored somewhere and there was a procedure for migrating the database.

I took a look at nsContentPrefService (which stores your preferences for specific sites, like remembering zoom settings) since that was another component using storage and written in JavaScript. I “hijacked” the code related to the database stuff, and modified it a bit to fit my needs.

The other major change here was to replace var with let – “let is the new var” as my shirt says. A number of other changes were also made – cleaning up queries, hard-coding the table names, and making sure I was using statements correctly.

v0.3, v0.4 – Cleanup & Optimization

These versions were pretty light – mostly involving cleanup. modifyLogin and the process involved got improved. I also reduced the number of queries we were making by hand and so all SELECTs were done from just 2 places (one for each table).

v0.5 – Importing

v0.5 focused on importing from the legacy module. The basics were in the original code Mrinal wrote, but all of the edge cases were difficult to handle. The one case that caused a lot of problems were the “user has a master password, but presses cancel when we import”. We needed to handle that gracefully, and Justin and I decided the best way was to introduce a _deferredInit method, which did the bulk of the initialization work. At the beginning of each public method, we then check the initialization state and try to import again. It can get annoying, but everything about the master password is annoying.

This also resulted in a couple bugs being spun off to help us with importing: one bug was just a few lines added to the legacy module and the password manager UI to handle an additional error, and the other was just to ensure the legacy module didn’t create an empty file if it was never used.

v0.6 – Tests

I finally got to writing the tests, which involved a lot of copy & paste from the legacy tests, then making small modifications. It’s not the most efficient way, but it works ok. Since the mozStorage module works slightly differently, the code duplication is a necessity for now. In theory they can be cleaned up, but that’ll be a task for the next intern :)

At this point I thought I was pretty much done, and I was. There were still a few problems though, and also not quite enough test coverage.

v0.7, v0.8 – Cleanup, Tests, Corrupt Databases

v0.7 involved a lot of cleanup and adding tests. v0.8 was an important milestone in that I finally added the handling of a corrupt database. Before this point, if we encountered a corrupt database, we would fail and then as with a failed import, just try again and again. This was bad, really bad. So a “thank you” to Shawn for catching that. Now we backup the corrupt database and just create a new one. It should be difficult to get a corrupt database, but just in case (and to cover the case when people think they know what they’re doing, but don’t).

v0.9, v1.0 – Performance

In one of the recent status meetings, Mike Shaver asked me about performance. At that point I hadn’t really done much except throw it up on the try server. The try server gives decent ball park figures, but it’s not perfect. So Justin worked on getting Standalone Talos working while I wrote some XPC shell “tests”. I discovered that while we were generally faster – faster init, faster add, faster remove – we were considerably slower for countLogin, which is a critical path, since it gets called on every page. Over 90% of the that time was actually spent initializing nsLoginInfo objects (since I tried to reuse code). This got improved, though we are still a couple milliseconds slower. This could likely be improved a little bit since we are still doing a SELECT and looping over the results. A little bit more work needs to be done that way, so making the query use COUNT would cut that out. Maybe I’ll write another patch to do that before Beta 1. For now though, we’ll keep doing it how we’re doing it.

And that’s all… almost

After this got checked in, the Windows boxes turned orange on Tinderbox. This was because of the tests (trying to delete files). Justin and I thought we had a quick fix, committed that. As I was packing to come home, I got pinged on IRC since the boxes were still orange and it was my fault. So the tests for my changeset got backed out and we switched back to the legacy module, but the code was still in there.

On Sunday, my module was re-enabled (thanks Justin!). There was another hiccup related to packages-static, but that was fixed as well. If you have any problems please report them on Bugzilla.