Saturday, June 28, 2008

Inspired by a book: Planning Extreme Programming

I have just finished reading the book: Planning Extreme Programming (The XP Series) and I have to say I found it brilliant. I read it in a day and could not put it down until I finally finished it around 3 am.
First of all the book is short, some 160 pages long, and that is very good! Simply everything is up to the point, you would not want to take anything out. It is concise, understandable, written in easy to follow language. Every paragraph brings some new value. And at the end it does not have ton of useless code examples, it does not have much of the examples though, but they were enough for me and most possibly any more would just be a distraction.
I strongly believe that any book should not be more then 200 or so pages long. Everything longer just puts me off. It is likely I won't have time to read it in continuation and at the end I won't know what I read in the beginning or at least how the things I'm reading now relate to the things I read two weeks ago. I like the Duplex Book idea, maybe that is why I enjoy reading Martin Fowler's books, it is like they are just the right size (or is it the content ;) )

Let me get back to the book. Even though the book is mostly project manager oriented developers should definitely read it. Sometimes I have difficulties explaining what is wrong with the process to our managers. Sometimes it is just a kind of a gut feeling, however you need far more solid arguments if you want to challenge some decisions regarding your project/process/whatever. The book nicely explains what the agile is all about and now I can safely say that I can easily explain to anyone why something is not agile or something is, and suggest improvements. And that alone is why it is worth reading.
One thing developer will definitely like is chapter about estimates. It just hits the spot. I recognized all the problems I had with my estimates (well more with project managers to be honest) and now I have a working plan how to improve my estimates even when not working on agile projects. And the advice to stop using the phrase: "I don't have enough time" with the phrase "I have too much to do" is simply brilliant, and what is more important it is usually more accurate.

So what have I learned from this book? The key to agile development is communication with the customer. If there is no proper and timely communication with the customer, if the customer is most of the time unavailable for clarifications regarding the functionality, you can't do agile. Without this key ingredient agile is not possible. The second key to agile is response to customer changes - the customer has the right and will change his mind and we should enable him to do so.
What the book also points is that all projects are different and what worked for one might not work for another one. So I guess being flexible when it comes to the process is a must. If something works for your project then use it, don't just blindly follow the process.

What you might notice I did not mention technology, testing, frequent releases, stand-up meetings or pair programming in the key elements? Your technology and practices should be determined by your project and your process, not vice verse. While it is very questionable if you can build software of decent quality without proper unit tests you might very well do without stand-up meetings and pair programming. At the end of the day use what works for you, don't force something that is not going to fit. Being agile should be company's strategy it is not a matter of simply coming and saying: great lets go agile.

Read the book, you'll feel better. I found what mistakes I made and what problems I might have prevented if I had educated arguments. It adds great value in couple of days reading.

Sunday, June 22, 2008

Make JSUnit work in Firefox 3.0

I lost hours figuring out this one. I was getting an error: Permission denied to get property Window.xbDEBUG, and eventually getting timeout on running my test page. The first error was happening on different lines of code while I was debugging in Firebug. At the end it was probably some concurrency issue, timeout was happening due to JSUnit test runner not being able to load my test file.
Now, of course I checked if the file exists, and it did exist. I figured out that it cannot load file since it is stuck in function: isPageLoaded in testContainerController.html. Why? At the end I found an answer in an unlikely place: Why doesn't my test suite run under JSCoverage in Firefox 3?
The problem is in the tightened Firefox security regarding local files, and security.fileuri.origin_policy parameter is not the correct one as suggested by the FAQ, it is new configuration parameter that superseded it:
Security.fileuri.strict_origin_policy, it should be set to false so JSUnit will run correctly. The article is mumbling something about suggesting to keep it true for the improved security sake, but I'm too tired of banging my head of the wall to really pay attention.

*** UPDATE ***
I just realised that the version of JSUnit I'm using ( 2.2alpha11 ) is not working correctly when you brows for the test page. I am getting an error:
Reading Test Page file:///someTest.html timed out.
Make sure that the file exists and is a Test Page.
The page does not exist since I browsed to a different file - file:///c:\temp\someTest.html. So there is a bug with loading page.
However it works if I supply the test page through testPage url parameter like this:
file:///.../testRunner.html?testpage=c:/temp/someTest.html
If the tests were not working after fixing the configuration then this might be the next thing to try out.

Thursday, June 19, 2008

Martin Fowler’s Syntactic Noise in JavaScript

I am closely following Martin Fowler's work on his DSL book. It is very interesting stuff and I wish I had more time to follow it more closely ... His latest post is related to the book and is about Syntactic Noise. This is my humble contribution with attempt to translate his example to JavaScript (btw, you have to read the article to make anything out of this):


var stateMachine = 
new StateMachine(
{
events: {
doorClosed : "D1CL",
drawOpened : "D2OP",
lightOn : "L1ON"
},

commands: {
unlockDoor : "D1UL",
lockPanel : "PNLK"
},

states: {

idle : {
actions : ["unlockDoor", "lockPanel"],
transitions: {
doorClosed : "active"
}
},

active: {
transitions: {
drawOpened : "waitingForLight",
lightOn: "waitingForDraw"
}
}
}
}
);



I know there is lots of red and I don't use => as is used in most of the examples, but I think I got it pretty good, and actually JavaScript is not that bad at all when it comes to internal DSLs. What do you think?
Oh, and I was just thinking, I bet I can do DSL in MPS that will have 0 (zero) syntactic noise! Keep tuned, I really might. That brings up another question: the example and the whole article is based on text based DSLs (blast, he was talking about internal DSLs, and I thought to be a smart a**) and what about graphical DSLs or Language Workbenches? As I said in some more advanced language oriented tools (MPS) we might use no syntactic nose without sacrificing any clarity. Text presented for clarification in this tools is part of the editor and is not there because of the syntax, you type in only significant data.

Tuesday, June 17, 2008

Improving security of JSONP in Chrome

Injecting scripts in chrome are raising security concerns, and JSONP is just that - injecting JavaScript. As I pointed out in Make JSONP Work in Chrome, number of security concerns are arising from the fact that, in chrome, scripts have as much privileges as the user running the browser. That means that if malicious code is injected by JSONP provider it could access private and sensitive data or do some real damage to the system as I will show later on with couple of examples.
Unfortunately Firefox is not offering much control over the privileges of the script execution: in chrome script has all the privileges. netscape.security.PrivilegeManager.disablePrivilege is not working and, correct me if I'm wrong, all you have as alternative is to run the code in the sandbox, where you have limited privileges and again no way to control them. I would much more prefere to have sandbox privileges by default and ask for more privileges I require.

Enough ranting and lets show the code. Here is the old version of the code:


var scriptCollection = document.getElementsByTagName("script");
var reader = new httpReader(scriptCollection[scriptCollection.length - 1].src);

function evaluateJs(aReader){
if(aReader && aReader.mData) {
eval(aReader.mData);
}
}

reader.AsyncLoad(bind(evaluateJs, this, reader));

Basically the first part is getting the URL of the JSONP script to execute - jQuery AJAX function hack. The second one is the function that will evaluate received script. As I pointed out earlier, by doing this the received script is executed in chrome with all the privileges and that is not very good idea if you have internet server serving you the script.

Here is improved script to execute this code in the sandbox:


var scriptCollection = document.getElementsByTagName("script");
var reader = new httpReader(scriptCollection[scriptCollection.length - 1].src);

function runInSandbox(scriptToEval){
//this line is for testing purposes this privilege will be enabled in chrome anyway
//netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
var safewin = new XPCNativeWrapper(window);

var sandbox = Components.utils.Sandbox(safewin);
sandbox.window = safewin;
sandbox.document = sandbox.window.document;
sandbox.__proto__ = safewin;

//add jsonp method to the safe window
for(var prop in window){
if(typeof window[prop] == "function"
&& prop.substring(0, 5) == "jsonp"){
sandbox.window[prop] = window[prop];
break;
}
}

Components.utils.evalInSandbox(scriptToEval, sandbox);
}

function evaluateJs(aReader){
if(aReader && aReader.mData) {
//eval(aReader.mData);
runInSandbox(aReader.mData);
}
}

reader.AsyncLoad(bind(evaluateJs, this, reader));

This is significantly longer but the only change is usage of runInSandbox function. This code is inspired by Greasemonkey implementation. We are creating safe window using XPCNativeWrapper. Using the safe window we are creating sandbox and setting sandbox properties - window, document and __proto__. The XPCNativeWrapper is wrapping the object and limiting access to its properties and functions. Mainly, it is not allowing access to added or changed properties and functions of the wrapped object. Look at: How the History of Greasemonkey Security Affects You Now for more informati on about possible attacks and pitfalls.
Since after wrapping the window the JSONP function is missing we are adding it by cycling through all the properties of the original window and upon finding the right function we are adding it to the safe window. We know that jQuery created the JSONP function so it is pretty safe to do this. After that evalInSandbox can be called and we have our JSONP function executed in sandbox without possiblity to do any damage.
To illustrate what's been shown so far I will create a couple of simple tests.

Simplistic injection testing

First of all to note that all of this I am showing is done by opening XUL window in Firefox, feel free to wrap it up into extension and test it properly.
Lets prepare our testing ground for exploring vulnerabilities arising. The full source code is here. Please put injection*.js files somewhere on your local web server if you have it so you can access it easily. They are simulating JSONP scripts injection and basically they are simulating what some malicious server might serve to our extension.
The injection1.js is simply showing alert window so we know that we have access to injection files.
Injection2.js is reading C:\temp\security.txt file from the disk (you create some test file) and showing its contents. It is there just to prove the point, it might easily delete your system files if you are running Firefox as admin, or read your Gmail username and password and send it to some 3rd party via web service, etc; the point is it can do anything if called from chrome.
Injection3.js is trying to add click event handler to existing window and attempt to make malicious script pass boundaries of the sandbox.
Injection4.js is using some function we deliberately made unsafe to make the point that even when running in sandbox we have to keep in mind what we are doing and that we might cause voulnerabilities ourselves.
The securityText.xul has 5 buttons and they are doing the following stuff:
Simple test - loads injection1.js - shows alert 'injection', if it shows alert we are injecting scripts as expected
Voulnerability! - executes Injection2.js in chrome and is reading content of C:\temp\security.txt
Safe 1! - executes Injection2.js in sandbox - showing that it will not read the file
Safe ! - on click changes txt to still safe - we are attaching malicious code to click event of the button, but since it is executed on the wrapper it will not read the file
Safe (not really) ! - on click will change the text to Unsafe! since it will use voulnerable function we added to safe window and will actually add malicious code to click event of the button. Click on it again and it will read the contents of the file.

To change the location of the injection files change line 2: var scripturl = 'http://localhost/' + scriptFile; to wherever you put the script files. To change what file is read from the local disk find c:\\\\temp\\\\security.txt in injection scripts and change them to whatever file you want to read.
Hopefuly this helps.

Saturday, June 14, 2008

Intranet Feed Reader 0.2.0 - Support for Atom, and improved security

New version of Intranet Feed Reader add-on - 0.2.0 is available now. Source code is available here.
The add-on now has improved security by executing JSONP in sandbox instead of running it in chrome. This improvement will stop malicious code served by XML2JSON proxy, there is still possibility that unsafe code might leave sandbox but threat surface is much smaller now. Blog entry about that will follow and will continue to work on ways to improve security so stay tuned.
New version brings support for Atom feeds so now you can use: http://feeds.feedburner.com/blogspot/digitalmihailo as XML Url :) I added some HTML cleansing code as well - in our intranet we are using confluence and some feeds are featuring unclosed img tags. What the code does is using very simple algorithm (and un-optimized) to close unclosed tags and remove closing tags without openning tag. These improvements are implemented by following functions in content/sidebar.js of the add-on source code:

desinfictHtml
runInSandbox
adaptData - for Atom to RSS (we are using RSS to render content)
so give it a look.

Sandbox execution is inspired by Greasemonkey implementation.
Converting Atom dates is inspired by Convert Atom documents to JSON document by IBM.


Testing remarks

For setting up testing environment locally one possible solution would be to download feed's XML and put it to your local web server - url something like: http://localhost/testfeed/somefeed.xml. Using XML2JSON proxy is described in the first post on the Intranet Feed Reader Add-on.

Sunday, June 8, 2008

jQuery and dynamic HTML in XUL

There is a problem with doing dynamic HTML in Firefox Extensions using jQuery, actually only if you use XUL windows. This is probably applicable to all JavaScript frameworks that are allowing dynamic manipulation of DOM.
Consider the following example:




<?xml version="1.0"?>
<window id="rootWnd" title="Test"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns:html="http://www.w3.org/1999/xhtml" >
<script type="application/x-javascript" src="jquery-1.2.3.js"/>
<script type="application/x-javascript">
function test(){
$("#content").append('&lt;p&gt;test&lt;/p&gt;');
alert('done');
}
</script>
<vbox hidden="false" height="0">
<button label="click me" oncommand="test();"/>
<html:div id="content"></html:div>
</vbox>
</window>




There is nothing special about this code, highlighted text JavaScript should add paragraph to content div and show text "test". That is probably very common statement when using jQuery to change some HTML content (and probably doing something more useful). Nothing special about the code as I said but it does not work. If you try this into web browser it will work though however in XUL it won't.
This example follows a tutorial on adding HTML elements to XUL. It is obvious now that $("#content").append('test') won't work. According to the article it has to be: $("#content").append('test'). Unfortunately it does not work again. The problem is in jQuery.1.2.3.js, more precisely in the line 968:
var tags = jQuery.trim( elem ).toLowerCase(), div = context.createElement("div");
Again if we try to put "html:div" it will not work and now the problem is in the Mozilla's document.createElement implementation which will create the element in the default namespace of http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul since it is XUL document. I think this is not how it should be but non the less it is the way it works so solution to the problem would be:
div = context.createElementNS("http://www.w3.org/1999/xhtml", "html:div")
which will create the element in the correct namespace. Unfortunately it will not work again since exception will be thrown in the line 998: div.innerHTML = wrap[1] + elem + wrap[2]; it does not like html:... part I guess.
So is there a solution? Here it comes:

Solution

If we rewrite the XUL document to look like this:




<?xml version="1.0"?>
<xul:window id="rootWnd" title="Test"
xmlns:xul="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
xmlns="http://www.w3.org/1999/xhtml" >
<xul:script type="application/x-javascript" src="jquery-1.2.3.js"/>
<xul:script type="application/x-javascript">
function test(){
$("#content").append('&lt;p&gt;test&lt;/p&gt;');
alert('done');
}
</xul:script>
<xul:vbox hidden="false" height="0">
<xul:button label="click me" oncommand="test();"/>
<div id="content"></div>
</xul:vbox>
</xul:window>




What we are doing here is setting the default namespace to be the XHTML namespace instead of XUL. It requires a bit of rewriting the XUL elements but it will presumably enable us to use any JavaScript framework. however there is another bit that has to be changed in the jQuery script:
line 968 should stand as:
div = context.createElementNS("http://www.w3.org/1999/xhtml", "div")
And of course any createElement calls should be amended.

Another problem

At the end to conclude with another unsolved problem. This will all work if you have control over the html contents you want to change, but if you don't, e.g. showing rss feed, then you might end up with a problem. If you check the namespaces in the XUL document you'll see that namespace is of XHTML and actually the document is XML document. Because of that, sometimes you might end up with error similar to:
[Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIDOMNSHTMLElement.innerHTML]" nsresult: "0x80004003 (NS_ERROR_INVALID_POINTER)"
Which is Firefox's friendly way to say that your XHTML is invalid. Thanks to this article in Russian (here is the English version, courtasy of the Google language tools, mind that translation is a bit awkward but it can be understood if you badly need it) I managed to decypher the error message.
This message can occure because of many things, unclosed tags, text in elements that are not supposed to have text etc. It can pass in regular browser since browser will do its best to parse whatever you give it but if it ain't proper XHTML it won't work in Firefox. If you have any suggestions on how to clean the HTML to get proper XHTML please let me know.

Tuesday, June 3, 2008

Firebug Net Panel History Overlay version 0.2.1

Just uploaded new version of the Firebug Net Panel History Overlay. The new version is 0.2.1. It adds support for Firefox 3.0.* and adds new menu item Clear History.
I don't know why the addon's main page is not showing the latest version but the file can be downloaded from here, and source code from here. As with the previous version login is required since the addon is in the sandbox. Talking about sandbox, some review/feedback would be highly appreciated so I can nominate it for becoming public. Thanks.

**Update**
The public page of the add on has been updated, seems there is a bit of lag between uploading new version and showing on the site, none the less all links should work.