Pragmatix Blog

The art of developing quality code and the science of security and reliability

Run vulnerability scans on your websites

clock May 6, 2010 10:58 by author Bill Abram

This entry is part of the series Ten things you can do to improve IT security and reliability without spending money

Both the SANS Institute report I mentioned in a previous blog entry and IBM’s X-Force 2009  Trend and Risk Report documented significant growth in website vulnerabilities, particularly across three attack categories:  Cross-Site Scripting, SQL Injection, and File Include vulnerabilities. The IBM X-Force report provides these succinct descriptions.

Attack Technique

Description
Cross-Site Scripting  Cross-Site Scripting vulnerabilities occur when Web applications do not properly validate user input from form fields, the syntax of URLs, etc. These vulnerabilities allow attackers to embed their own script into a page the user is visiting, manipulating the behavior or appearance of the page. These page changes can be used to steal sensitive information, manipulate the Web application in a malicious way, or embed more content on the page that exploits other vulnerabilities. 

The attacker first has to create a specially-crafted Web link and then entice the victim into clicking it (through spam, user forums, etc.) The user is more likely to be tricked clicking the link because the domain name of the URL is a trusted or familiar company. The attack attempt may appear to the user to come from the trusted organization itself and not the attacker that compromised the organization’s vulnerability.
SQL Injection  SQL Injection vulnerabilities are also related to improper validation of user input, and they occur when this input (from a form field, for example) is allowed to dynamically include SQL statements that are then executed by a database. Access to a back-end database may allow attackers to read, delete, and modify sensitive information and, in some cases, execute arbitrary code.

In addition to exposing confidential customer information (like credit card data), SQL Injection vulnerabilities can also allow attackers to embed other attacks inside the database that can then be used against visitors to the Web site.
File Include  File Include vulnerabilities (typically found in PHP applications) occur when the application retrieves code from a remote source to be executed in the local application. Oftentimes, the remote source is not validated for authenticity, which allows an attacker to use the Web application to remotely execute malicious code.
Other  This category includes some denial-of-service attacks and miscellaneous techniques that allow attackers to view or obtain unauthorized information and/or change files, directories, user information or other components of Web applications.

 

What You Should Do
First, you should make sure that any people you have developing or maintaining your websites are familiar with these types of vulnerabilities and that they are using current methods to protect against them.

Second, you should check with whoever is responsible for hosting your websites to see if they offer any kind of web vulnerability scanning as part of their hosting service.  (In some cases, hosting companies may charge extra for this; in other cases it’s included as part of the fee you already pay.)

And third, you should use web vulnerability scanning software.  Some vendors may offer to perform the web scans for free, as a means of getting in the door to talk to you about purchasing other services.  (Full disclosure: Pragmatix has been known to do this also.)

There are several free tools available on the internet.  My network engineering team has identified two that are definitely worth looking into.  One product is named Rapid7 (see http://www.rapid7.com/vulnerability-scanner.jsp ).  While there is a cost to purchase a full copy of their software, they do offer a very useful free version.  Another product is named Gamja (see http://sourceforge.net/projects/gamja/).

 

 



Desktop Security Tip: Always Log In As Local User, Not Administrator

clock May 6, 2010 06:14 by author jnandy

This entry is part of the series Ten things you can do to improve IT security and reliability without spending money

When you install Windows, or when it comes pre-installed on a new computer, the default settings provide user accounts that have full Administrator privileges.  It is very dangerous to use your computer when you are logged in as Administrator.  Here’s why: When you are logged in as Administrator, you have totally unrestricted access to every corner of your system, every setting, every software application, every network connection, every file, etc.  If you happen to come in contact with a virus, spyware, or other malware program—even just by clicking an infected link on a website—that virus or malware will then be able to attack anything and everything on your computer.

Instead, you should login in as a local user, with limited privileges.  Encountering that same virus, spyware, or malware as a local user will have much less impact—and in many cases no impact—on your computer.

Sometimes, you will need to login as Administrator to install software or perform other tasks.  As soon as those tasks are complete, you should log off.  Then you can log in again, as a local user with limited privileges. With limited privileges, your computer will be much less susceptible to virus, spyware, and other types of attacks.

How to determine if you are logging in as the Administrator
1. Right click on your “my computer” icon and select “manage”. If you do not see “my computer” icon on your desktop, then hit start and right click on “computer.

2. Expand the “Local users and group” from the left panel and right click on “groups” and double click/open “Administrator” on the right panel

 

3. You should see your user name in this box if you are the Administrator or equivalent user.


Setting up a local user account with limited privileges
1. Right click on your “my computer” icon and select “manage”. If you do not see “my computer” icon on your desktop, then hit start and right click on “computer.

2. Expand the “Local users and group” from the left panel and right click on “users” and  select “New User”

 

3. You should see this box. Proceed to fill in the information and hit “create”. This will create a regular windows user.

You can logout and log back in with this user which will have reduced privileges and hence would be safe to use.

Follow the instruction in the section “How to Determine if you are the admin” to make sure that this user does not show up in the Administrator group. If it does , highlight and remove.



Spotlight on Mobile Banking Security

clock April 21, 2010 03:44 by author barbaraa

More and more we use our mobile phones for everything, including applications where we expect the highest level of security, such as online banking. Many banks provide apps for the Google Android phones and Apple iPhones that let you check your account balances, transfer funds, and pay bills. These are incredibly convenient, allowing you to check your balances before making a purchase, move funds to avoid overdrafts, and pay bills whenever you remember, not just when you are in front of a computer. But what are the risks of having these apps on a device that could easily be stolen or lost?

The first line of defense is the security of the mobile device itself. Your phone should be password locked requiring you to enter a password in order to access any of the phone features. Apple also provides a powerful option, allowing you to remotely transmit a command to erase the contents of your phone in the event that it is lost or stolen. But what is at risk if someone gets access to your mobile banking apps? If you store your username and password in the app, the person with your phone may be able to do everything you can do with your accounts. Most banking apps require that a password be re-entered before proceeding with operations that move money, greatly reducing your risk as long as you have maintained good password practices (see The Care and Feeding of Passwords). So a great deal of security is obtained by password protecting your phone, never saving your banking username and password on the phone, and choosing passwords that are easy for you to remember without being easy for others to crack.

Banks design their software to reduce the risk of fraud, but thieves and hackers are working just as hard to break the applications. Make sure that you keep your antivirus service up to date on your phone and keep your bank's software up to date also. Your bank will make updates available in order to keep you more secure (as well as upgrade service). Finally, it is a good idea to keep your bank emergency contact information with you separate from your phone so that if you your phone is lost or stolen, you can notify the bank and/or log in via computer to change your password.

By taking a just a few simple precautions, you can safely take advantage of the power and convenience of mobile banking.



Automating ASP.NET Application Deployment

clock April 1, 2010 12:43 by author pkrishnan

Being a relatively small development shop, we developers do deployments ourselves, without a dedicated person to handle versioning, compiling, and pushing of code. We need to ensure that deployment goes as smoothly and as automated as possible. We support our clients with separate environments for development, testing, production, and DR, requiring multiple code and database deployments for the lifecycle of a set of changes.

Database Synchronization using SQL Examiner Suite
In particular, deploying database updates can get complicated and cumbersome simply to create a record of the changes that need to be pushed. To automate database deployment we use SQL Examiner Suite (http://www.sqlaccessories.com/Products.aspx). The SQL Examiner Suite is relatively simple to use and it allows you to visually see what is changed between the development and the production database.

The SQL Examiner tool allows you to compare the database schema. All you need to do is to enter the location and authentication information of your databases and run compare. If you don’t need to compare all database objects, you can select a subset to compare. The comparison results are displayed in a tree view with differences color coded and the changes highlighted in right panel. You can select the changes you want to synchronize and push the changes from one database to the other. 

Once the schemas are synched, the SQL Data Examiner tool allows you to synchronize the data in the tables. Just like with the schema comparison, the data comparison results are displayed in a tree view and tables with differences color coded. Clicking on a table in the tree view will display the data differences in the right panel. You can select either the entire data or just the records to be added or deleted and synchronize them.

Depending on how automated you would like, it also allows you to save the project. Use Console Mode Setting Wizard to create a config  file and run it in future by simply creating a batch file to run the configuration file by using command SQLDCmd /config:Config_Schema.xml for schema synchronizing and SQLDECmd.exe /config:config.xml for data synchronization. For straightforward, controlled environments this can be an amazing timesaver.

Deploying Changes to a Website Using Beyond Compare
Website changes are updated using the Beyond Compare software (http://www.scootersoftware.com). We push changes over FTP and Beyond Compare lets you select a local folder and the remote FTP folder for comparison. The differences are color coded, with the newer files displayed in red and the orphaned files in blue. After identifying a file that is different on the two sites, Beyond Compare allows you to select the file and view the differences (there are a variety of specialized built-in viewers, including images, text, HTML, CSV, etc.). Changed files can be copied over or, where appropriate, portions of code files or text files can be copied without moving the whole file. You can choose files or folders to exclude from the comparison (e.g., configuration files) by adding them in the Filters.  You can Save the session and rerun the comparison each time by simply opening the session and also share your settings with other users by exporting the settings to a configuration file.

Conclusion
With these two reasonably priced pieces of software, we have been able to reduce our deployment times to less than 1/3 of our previous times. The highly functional, clear, and easy to use interfaces of these apps not only speed the work, but greatly increase our assurance that everything was deployed and deployed properly. In fact, the bulk of the time savings comes from not having to check, recheck, and debug deployment problems. Although we spent a couple of days learning the software and setting up our automation configuration files, the time was paid back after just a few deployments.



Don't Leave Your Passwords On A Sticky Note: The Care and Feeding of Passwords

clock February 22, 2010 07:14 by author Brian Kennedy

This entry is part of the series Ten things you can do to improve IT security and reliability without spending money

The average person juggles dozens of passwords for accessing everything from social networking sites to their bank accounts. Given our increasing reliance on the internet via password-protected sites, it is worth reviewing some basics about passwords and how they should be managed.A password is a secret bit of information that allows you to prove to a web site that you are indeed yourself. There are many ways other than passwords to prove your identity online (biometrics, tokens, etc.), but passwords have the advantage of being simple to implement and easy to understand so almost every site uses them.

The Basics

The key aspect of a password is that it is secret. Only you, and the site you are using, should know it. Anyone else who knows your passwords can do anything else online that you can, from reading your mail to draining your bank account. Thus, all good rules about managing passwords revolve around keeping your passwords secret.

How The Beans Get Spilled

To understand the best way to choose and maintain your passwords, let's quickly review the ways in which a password could stop being secret.

1) You give it away
2) Someone else gives it away for you
3) Someone guesses it

Giving Away Your Password: This method covers a lot of ground, from the post-it note on the monitor, to spyware on your computer capturing passwords as you type, to phishing attacks. The best defense against these is simply to be careful. Never reply to email from your bank online or click a link in a message that claims to be from your bank. Never give technical support your password (they never need it). Don't keep your passwords written down or in a text file labeled "secret_passwords.txt" on your desktop. Don't share your password with others.

Someone Else Gives It Away: Basically, this boils down to hackers stealing your login information from one of the web sites you frequent or a careless employee of that web site losing a laptop with usernames and passwords stored on it. While there is little you can do to prevent this from happening, you can take steps to reduce the damage. The key thing is to use *different* passwords for each site you access. That way, if Site A is hacked, the hackers can't start trying the logins on popular sites B through Z. If Twitter loses your password, at least they won’t be losing your banking password.

Someone Guesses It: This method should influence your actual choice of passwords. Hackers have a very sophisticated set of tools available to them to attempt to guess your password, and given the speed of computers, they can guess thousands of passwords in a second. Since hackers are very, very good guessers, you should design your passwords to be resistant to guessing.Just like the infamous luggage lock code of 1-2-3-4, there are common passwords that every password guessing tool will try right away. A good password cracking tool can run through every combination of birthday, month, year, first name faster than you can type your password, so avoid those. In fact, most hacking tools will run through the entire dictionary, so using real words is to be avoided as well.The best possible password is an extremely long string of random numbers and letters, with some punctuation thrown in. From a technical standpoint, this is a GREAT password:

          f25$$haPAsWacU!sta$8esWa9ruBr3%wRuSPUyu9uquswAbr6cEfrA9eyeR^6r*ufreyuw8

Reality Check

While that is a perfect password in theory, the reality is, no one can remember that password. Which means it will get written down, and if memorized, it would certainly be reused for every possible purpose. Which kind of defeats the purpose.

Compromise

A good password strategy needs to find a happy medium. You want to create passwords that are simple enough to remember even if you use a different one for each site. The passwords should be reasonably resistant against guessing. So what should you use to generate your passwords? One good password generation strategy involves taking some simple, easily remembered text and tacking on enough extra bits that it is hard to guess. Another good algorithm is to embed actual words, possibly linked to the site you are accessing into a template.

For example, I might choose a template of

          $6<first word>*<SECONDWORD>!

and use this template for all of my passwords.

Now, my Facebook password might be

          $6social*FRIENDS!

while my bank password might be

          $6money*RICH!

For each site I only have to remember the two words for that site and my simple, but hard to guess template. But I have accomplished my goals of making my passwords hard to guess and different across all my sites.

There are a bunch of other similar algorithms out there to accomplish this goal. Farhad Manjoo of Slate has another good one here:

http://www.slate.com/id/2223478/

Another concession to reality

To further simplify your password management, we can admit that not all sites are equally important. It is possible to divide sites into tiers based on what a hacker could do if she had your password. Acess to sites like banking and email could allow a hacker to do a lot of damage. So could any site that stores your credit card (e.g. Amazon One-Click). But for things like commenting on someone's blog you could probably skip this whole process since the risk is low. It takes some pressure off to choose one or two ‘easy’ passwords that you use for temporary or non-critical purposes, and reduces the temptation to reuse a secure password when faced with this need.

In a Nutshell

1) Don't give your passwords to anyone or write them down.

2) Use a different password for every site you access (except the really, really unimportant ones).

3) Use a password generation strategy to turn simple to remember phrases or key words into hard to guess passwords.

 



JavaScript Inheritance

clock January 18, 2010 07:33 by author Brian Kennedy
When we first started embarking on larger scale JavaScript projects (not just the standard event handler snippets) it took us a while to adjust to the differing inheritance style since we were used to class based programming. Initially, we flirted with the Microsoft AJAX.NET approach, which attempts to bolt on a class based, strongly typed inheritance model onto JavaScript (even to the point of mimicking enumerations and interfaces), but eventually we realized that it was best to use the language the way it was intended to be used.

After a bit of searching among the variety of approaches out there, we ended up using the parasitic inheritance model proposed by Douglas Crockford. The essence of this approach is having the constructor for a subclass instantiate an instance of its superclass, modify it by adding methods, and then returning the modified object. A simple inheritance would look like this:




   1:  car= function(manufacturer) {

   2:      // class we inherit from

   3:      var me = new vehicle(manufacturer);

   4:      me.honkHorn = new function() {

   5:          // do stuff

   6:      };

   7:      return me;

   8:  }

   9:   

  10:  var myBlueVolvo = new car= ("Volvo");



This seems pretty natural and gives most of what we were looking for (code reuse and some variable hiding) without torturing the language too badly.

We do occasionally need to call methods in the superclass from the subclass though. Douglas implements a relatively complex "sugar" that automatically provides reference to a "uber" object that mimics the "base" object in C#, but we were worried that this was a lot of extra code for something we really only do with a handful of functions. Complexity is bad.

Instead, when we need to reference an overridden method in the superclass, we took advantage of JavaScript closures to provide a simple way to call a superclass method. Here is an example:



   1:  var super = me.honkHorn; 

   2:  me.honkHorn = new function() {

   3:      // call the method on the superclass

   4:      super.honkHorn();

   5:   

   6:      // now do stuff in the subclass

   7:  }
;
By saving the superclass function in the super var, it gets captured and is still around to be called, even though it is no longer references by the new object as a public function.


XML Serialization With Subclasses

clock September 18, 2009 06:13 by author Brian Kennedy
The .NET XML serialization libraries are a huge time saver for quickly implementing configuration files and small local data stores. All it takes is a few attribute markup tags to allow for quick saving and loading of an object graph to disk.

One of the shortcomings of this approach is how .NET handles the serialization of subclasses. To illustrate this, lets take a quick look at a configuration file I developed for an data processing service I was working on recently. The service allowed the user to configure jobs that ran on a scheduled basis, performed configurable data searches against a remote database, transformed the data, and than ran the data through one or more output plug-ins. These plug-ins allowed perform a variety of different actions on the retrieved data. Each output plug-in needed different configuration object, since it did different things. The service architecture allowed us to write new plug-ins as needed to perform custom integration tasks for specific clients without needed to release a new version of the app.

The architecture of the output system is pretty simple: configuration classes all implement the same IOutputPluginConfig interface and we used a simple factory class to instantiate the corresponding IOutputPlugin classes from their saved config. This means that each persistent job configuration class had a property like this:



   1:  public SerializableList&lt;IOutputConfig&gt; Outputs

   2:  {

   3:       get { return m_Outputs; }

   4:       set { m_Outputs = value; }

   5:  }

   6:  private SerializableList&lt;IOutputConfig&gt; m_Outputs;



This list might contain several different output configuration types.


Now, .NET will serialize this list of IOutputConfig objects to disk just fine, but on loading the configuration it will fail, since it doesn't know what classes to create for each saved XML output configuration element.

There is a built-in mechanism to resolve this problem; using the XmlArrayItemAttribute you can specified all the subclasses that should be deserialized.



   1:  [XmlArrayItem(typeof(EmailOutputConfig)),

   2:  XmlArrayItem(typeof(DiskOutputConfig)),

   3:  XmlArrayItem(typeof(SharepointOutputConfig)),]

   4:  public List&lt;IOutputConfig&gt; Outputs;



This didn't really suit my need for two reasons:
  1. Its messy and breaks encapsulation. We are leaking knowledge of the subclasses upwards, which is bad design. New implementations of outputs would have to modify the attribute in the library classes.
  2. Since we are implementing new functionality via plugin-ins (which are loaded dynamically), we don't know the full list of possible subclasses at compile time and thus can't list them in the attribute, even if we were willing to hold our noses and do so.
We could also work around this by having our subclasses implement the IXmlSerializable interface, but this would require mucking about with XML readers and writers for every configuration type we implemented, which takes time and thus negates a lot of the benefits of this approach.

The approach I used was to implement two custom classes, ISerializableList and ISerializableDictionary. These classes implement IXmlSerializable to wrap each of the subclasses in an tab. This tag records the type of the saved object so we know what type to feed an XmlSerialzer when restoring the data.



   1:  &lt;Outputs&gt;

   2:  &lt;Item type="APP.Output.DiskOutputConfig, APPDAC"&gt;

   3:    &lt;Disk&gt;

   4:      &lt;BasePath&gt;c:\test\exrs\two\&lt;/BasePath&gt;

   5:      &lt;PathPattern&gt;{DATESTAMP}_{TIMESTAMP}&lt;/PathPattern&gt;

   6:      &lt;OverwriteFile&gt;true&lt;/OverwriteFile&gt;

   7:    &lt;/Disk&gt;

   8:  &lt;/Item&gt;

   9:  &lt;Item type="APP.Output.DiskOutputConfig, APPDAC"&gt;

  10:    &lt;Email&gt;

  11:      &lt;SMTPServer&gt;127.0.0.1&lt;/SMTPServer&gt;

  12:      &lt;Recipient&gt;a@test.com&lt;/Recipient&gt;

  13:      &lt;Recipient&gt;b@test.com&lt;/Recipient&gt;

  14:      &lt;FilePattern&gt;{DATESTAMP}_{TIMESTAMP}&lt;/FilePattern&gt;

  15:    &lt;/Email&gt;

  16:  &lt;/Item&gt;

  17:  &lt;/Outputs&gt;



One of the points to note is that I am stripping the assembly version info from the saved type name. This was to avoid version changes in the asembly causing load errors. For major schema changes I would need to implement another set of classes anyway, so as to be able to have both loaded at once for translation.

Here is the full code for both classes and well as a link to a project you can use directly.



   1:  [Serializable]

   2:  public class SerializableList&lt;TValue&gt;

   3:  : List&lt;TValue&gt;, IXmlSerializable

   4:  {

   5:      #region IXmlSerializable Members

   6:   

   7:      public XmlSchema GetSchema()

   8:      {

   9:          return null;

  10:      }

  11:   

  12:      public void ReadXml(XmlReader reader)

  13:      {

  14:          bool wasEmpty = reader.IsEmptyElement;

  15:          reader.Read();

  16:          if (wasEmpty)

  17:          return;

  18:   

  19:          while (reader.NodeType != XmlNodeType.EndElement)

  20:          {

  21:          string StateTypeDescriptor = reader.GetAttribute("type");

  22:          Type StateType = Type.GetType(StateTypeDescriptor);

  23:   

  24:          reader.ReadStartElement();

  25:          XmlSerializer valueSerializer = new XmlSerializer(StateType);

  26:          this.Add((TValue)valueSerializer.Deserialize(reader));

  27:   

  28:          reader.ReadEndElement();

  29:          reader.MoveToContent();

  30:          }

  31:          reader.ReadEndElement();

  32:      }

  33:   

  34:      public void WriteXml(XmlWriter writer)

  35:      {

  36:          foreach (TValue item in this)

  37:          {

  38:          Type ValueType = item.GetType();

  39:          XmlSerializer valueSerializer = new XmlSerializer(ValueType);

  40:          string SubElementName = "Item";

  41:   

  42:          writer.WriteStartElement(SubElementName);

  43:   

  44:          writer.WriteStartAttribute("type");

  45:          writer.WriteString(Serialization.GetTypeName(ValueType));

  46:          writer.WriteEndAttribute();

  47:   

  48:          valueSerializer.Serialize(writer, item);

  49:   

  50:          writer.WriteEndElement();

  51:          }

  52:      }

  53:   

  54:      #endregion

  55:   

  56:  }





   1:  [Serializable]

   2:  public class SerializableDictionary&lt;TKey, TValue&gt;

   3:  : Dictionary&lt;TKey, TValue&gt;, IXmlSerializable

   4:  {

   5:      #region IXmlSerializable Members

   6:      public XmlSchema GetSchema()

   7:      {

   8:          return null;

   9:      }

  10:   

  11:      public void ReadXml(XmlReader reader)

  12:      {

  13:          XmlSerializer keySerializer = new XmlSerializer(typeof(TKey));

  14:   

  15:          bool wasEmpty = reader.IsEmptyElement;

  16:          reader.Read();

  17:          if (wasEmpty)

  18:          return;

  19:   

  20:          while (reader.NodeType != System.Xml.XmlNodeType.EndElement)

  21:          {

  22:          string StateTypeDescriptor = reader.GetAttribute("type");

  23:          Type StateType = Type.GetType(StateTypeDescriptor);

  24:          XmlSerializer valueSerializer = new XmlSerializer(StateType);

  25:   

  26:          reader.ReadToFollowing("key");

  27:          reader.ReadStartElement("key");

  28:          TKey key = (TKey)keySerializer.Deserialize(reader);

  29:          reader.ReadEndElement();

  30:   

  31:          reader.ReadStartElement("value");

  32:          TValue value = (TValue)valueSerializer.Deserialize(reader);

  33:          reader.ReadEndElement();

  34:   

  35:          this.Add(key, value);

  36:          reader.ReadEndElement();

  37:          }

  38:   

  39:          reader.ReadEndElement();

  40:      }

  41:   

  42:      public void WriteXml(System.Xml.XmlWriter writer)

  43:      {

  44:          // the keys can't be subclassed,only the values can

  45:          XmlSerializer keySerializer = new XmlSerializer(typeof(TKey));

  46:   

  47:          foreach (TKey key in this.Keys)

  48:          {

  49:          TValue value = this[key];

  50:          Type ValueType = this[key].GetType();

  51:          XmlSerializer valueSerializer = new XmlSerializer(ValueType);

  52:   

  53:          writer.WriteStartElement("item");

  54:          writer.WriteStartAttribute("type");

  55:          writer.WriteString(Serialization.GetTypeName(ValueType));

  56:          writer.WriteEndAttribute();

  57:   

  58:          // serialize the key

  59:          writer.WriteStartElement("key");

  60:          keySerializer.Serialize(writer, key);

  61:          writer.WriteEndElement();

  62:          writer.WriteStartElement("value");

  63:          valueSerializer.Serialize(writer, value);

  64:          writer.WriteEndElement();

  65:   

  66:          writer.WriteEndElement();

  67:          }

  68:      }

  69:      #endregion

  70:  }

  71:   





   1:  namespace Pragmatix.Serialization

   2:  {

   3:      public class Serialization

   4:      {

   5:   

   6:          public static string GetTypeName(Type t)

   7:          {

   8:              string ClassName = t.FullName;

   9:              string SimpleAssembly = t.Assembly.FullName.Split(new string[] {","}, StringSplitOptions.RemoveEmptyEntries)[0];

  10:              return ClassName + ", " + SimpleAssembly;

  11:          }

  12:      }

  13:  }



Source code: Serialization.zip (14.19 kb)


Improving Performance and Bandwidth Usage in Rich Web Apps

clock April 28, 2009 02:43 by author Brian Kennedy

The Problem

Rich client web applications tend use a lot of JavaScript. Not only do they rely on extensive libraries to support client side functionality, but good modular code design tends to lead to lots of JavaScript files (especially since the Visual Studio JavaScript editor lacks code folding or regions and thus makes working with large files tiresome to say the least). Typically I find it easiest to place each client side class in a single file, much like we do with our server side code. Thus, for a typical page, consisting of a list, a filter, and one or two editor controls I might have the following script includes:

·         JSON library

·         jQuery / jQuery UI

·         Five or six standard jQuery plug-ins (blockui, timer, dimensions, hoverintent,  flot, bgiframe, etc) to provide extended functionality beyond that of the base libraries)

·         Our own library of base controls that the domain specific controls inherit from to obtain our form, list, and filter functionality

·         A control for each editor and list on the page,  with these controls  often having sub controls

Looking over a sample page from a recent application, I found 18 scripts included! This can cause significant performance issue for client, especially those with more limited bandwidth, for two reasons:

·        Total script size: We are downloading source code, which can be quite large. Source code contains white space and comments, and is formatted to aid maintainability, not to produce svelte downloads.

·        Number of Requests: Since there may be dependencies between the scripts (almost everything we use has some dependency on jQuery for instance), browsers will only download and evaluate one script at a time. Given the overhead in fetching and evaluating each script, this time can add up.

There are several established techniques to address these issues:

·         Compression: Most moderns browsers support gzip compression. Using gzip can reduce the size of the scripts significantly.

·         Concatenation: By combining all of your scripts into a single file, the client only needs to make one request. Order is still important, so you need to concatenate the scripts in the correct order of their dependencies.

·         Remove whitespace: Tools like JSMIN(http://www.crockford.com/javascript/jsmin.html) and packer(http://dean.edwards.name/packer/) can remove whitespace and comments, greatly reducing the download size.

·         Caching: By enabling caching for libraries and other scripts that are unlikely to change frequently, the browser will only download the script file once.

 

While using these techniques can make an enormous difference to the end user, they can make life very hard on the developer. Compression and caching require you to configure and maintain settings in IIS which add additional deployment and maintenance concerns. And combining / minifying your scripts makes them very difficult to maintain. Imagine having all your developers work on a single, uncommented, non whitespace JavaScript file!

 The Solution

The solution is to apply these techniques at run time, rather than at development or build time. There are a variety of implementations  for various frameworks, but none quite suited my needs. Specifically, I wanted something that would:

·         Work with minimal IIS configuration and work in both IIS6 and IIS7

·         Allow for a "debug" mode that delivered readable scripts to aid in debugging

·         Require minimal configuration and ideally use a centralized configuration file

·         Be flexible enough to allow for varying cache intervals and to compensate for the fact that some library scripts I've used seem to dislike being minified

·         Work well with .NET, but not require the client side AJAX.NET libraries (no ScriptManager base implementations)

·         Be simple

Overview

My solution is based on a custom ASP.NET HttpHandler that allows  web pages to include requests for scripts groups instead of just single scripts. The handler dynamically concatenates and minifies the script as well as emitting caching and gzip compression headers as appropriate.

A request for a script group would look like:

    <script src="/ScriptOptimizer.pragmatix?groups=jQuery" type="text/javascript"></script>

Multiple scripts can be combined into a single request by combining in a comma separated list:

    <script src="/ScriptOptimizer.pragmatix?groups=jQuery,PragmatixBase" type="text/javascript"></script>

I define my script group mappings in the web.config inside a custom <ScriptOptimizer> configuration section. I've seen a few folks online try to auto discover the scripts referenced by a page to avoid a configuration file, but I felt the overhead in terms of code wasn't worth the work.

  <ScriptOptimizer>

    <ScriptGroup Name="jQuery" Compress="true" AllowCache="true" CacheLengthInDays="7">

      <Script Path="/scripts/jquery-1.3.2.min.js" Enabled="true" Minify="false"/>

      <Script Path="/scripts/jquery-ui-1.7.1.custom.min.js" Enabled="true" Minify="false"/>

       …

     </ScriptGroup>

    <ScriptGroup Name="PragmatixBase" Compress="true" AllowCache="false" CacheLengthInDays="1">

      <Script Path="/scripts/V2/PragmatixControl.js" Enabled="true" Minify="true"/>

      <Script Path="/scripts/V2/PragmatixModalEditor.js" Enabled="true" Minify="true"/>

    </ScriptGroup>

  </ScriptOptimizer>

As you can see the caching options apply to an entire script group while enabling and minifying can be controlled on a per script level. If you combine groups, the lowest caching interval applies to all included scripts.

The Configuration

In order to keep the configuration info inside the web.config and avoid yet another config file I implemented my own configuration section. I just found out that my previous way of doing this, using the IConfigurationSectionHandlerinterface is deprecated, but  I chose to ignore this for now instead of using the newer ConfigurationSection  (http://msdn.microsoft.com/en-us/library/2tw134k3.aspx) class.  This allowed me to stick with the same XmlSerializer loading and saving of procedures  that I use for other stuff. Plus I didn’t feel like implementing a custom class for all my collections (although using generics is a pretty solid work around for that, see: http://utahdnug.org/blogs/josh/archive/2007/08/21/generic-configurationelementcollection.aspx)

The configuration section is pretty simple.  The configuration data is loaded up as an object by deserializing  the contents of the config section.

        public object Create(object parent, object configContext, XmlNode section)
        {
            XmlElement root = (XmlElement)section;
            XmlSerializer s = new XmlSerializer(typeof(ScriptOptimizerConfig));
            return (ScriptOptimizerConfig)s.Deserialize(new XmlNodeReader(section));
        }

The ScriptOptimizerConfig class contains a collection of script groups, which in turn contain a collection of script definitions. Using XML markup attributes like this:

    [XmlRoot(ElementName = "ScriptOptimizer")]
    public class ScriptOptimizerConfig
    {
         #region Properties
        [XmlElement("ScriptGroup")]
        public List<ScriptOptimizerScriptGroupConfig> ScriptGroups
        {
            get { return m_ScriptGroups; }
            set { m_ScriptGroups = value; }
        }
        private List<ScriptOptimizerScriptGroupConfig> m_ScriptGroups;
         #endregion

its possible to whip up a quick and nicely structured configuration file in just a few minutes.

The Code

The actual work of script optimization is done inside the ScriptOptimizerHttpHandler class. The entry point is the ProcessRequest method.

        public void ProcessRequest(HttpContext context)
        {            // load our configuration section
            ScriptOptimizerConfig Config = (ScriptOptimizerConfig)ConfigurationManager.GetSection("ScriptOptimizer");
            string[] groups = context.Request.QueryString["groups"].Split(new char[] { ',', ';', ':' });

             // determine the combined settings when multiple groups are requested
            ResultantScriptGroupSetting CombinedGroupSettings = Config.GetCachingSettings(groups);
               // set up GZIP compression if configured for such and the client allows it
            if (CombinedGroupSettings.Compress.Value && IsGZipSupported(context))
            {
              context.Response.AppendHeader("Content-Encoding", "gzip");
              ICSharpCode.SharpZipLib.GZip.GZipOutputStream OutputGZIPStream;
              OutputGZIPStream = new ICSharpCode.SharpZipLib.GZip.GZipOutputStream(context.Response.Filter);
              OutputGZIPStream.SetLevel(ICSharpCode.SharpZipLib.Zip.Compression.Deflater.BEST_COMPRESSION);
              context.Response.Filter = OutputGZIPStream;
            }
             // ready the response for writing out the scripts
            context.Response.Clear();
            context.Response.ContentType = "application/x-javascript";
             // write caching headers
            // if we are combining groups, we need to generate a combined set of caching headers
            // that makes sense. In this implementation, the lowest caching interval wins
             if (CombinedGroupSettings.AllowCache.Value)
            {
                context.Response.Cache.SetCacheability(HttpCacheability.Public);
                context.Response.Cache.SetExpires(DateTime.Now.AddDays(CombinedGroupSettings.CacheLengthInDays.Value));
             }
            else
            {
                context.Response.Cache.SetCacheability(HttpCacheability.NoCache);
            }
             // append the scripts
            AppendScripts(context, Config, groups, CombinedGroupSettings);
         }

First I load the configuration section, parse out my requested groups from the query string, and calculate some of the group-level settings if more than one group is requested. Next we emit our compression and caching headers. Finally we call the AppendScripts function to return each of the requested scripts.

        public void AppendScripts(System.Web.HttpContext context, ScriptOptimizerConfig Config, string[] groups, ResultantScriptGroupSetting CombinedGroupSettings)
        {
             foreach (string groupName in groups)
            {
                ScriptOptimizerScriptGroupConfig groupconfig = Config.GetGroup(groupName);
                // loop over each script. minify is necessary and append to the output stream
                foreach (ScriptOptimizerScriptConfig scriptconfig in groupconfig.Scripts)
                {
                    if (scriptconfig.Enabled)
                    {
                        string FullScriptPath = context.Server.MapPath(scriptconfig.Path);
                        // we can choose to exclude scripts from the minifications process
                        // some libraries we don't need to debug and some scritps react poorly
                        if (scriptconfig.Minify)
                        {
                            MemoryStream FileJavaScript = new MemoryStream(Encoding.ASCII.GetBytes(File.ReadAllText(FullScriptPath)));
                            JavaScriptMinifier min = new JavaScriptMinifier();
                            min.Minify(FileJavaScript, context.Response.OutputStream);
                        }
                        else
                        {
                            context.Response.WriteFile(FullScriptPath);
                        }
                         // this helps correct for scripts that aren't properly terminated
                        // this is fine when they are individual files, but causes issues when concantenated
                        context.Response.Write(";;");
                    }
                }
            }
          }

I am using a modified version of the JSMIN C# class provided by Douglas Crockford (http://www.crockford.com/javascript/jsmin.html). My only modification was to rework some of the input and output functions to make it easier to interface with the library.

 

Installation

 

First, you need to add the configuration section to your web.config :

    <section name="ScriptOptimizer" type="Pragmatix.ScriptOptimizer.ScriptOptimizerConfigurationHandler, CDXLibrary"/>

Next register the HTTPHandler under the <system.web> <httpHandlers> section like so:

      <add verb="*" path="ScriptOptimizer.pragmatix" type="Pragmatix.ScriptOptimizer.ScriptOptimizerHttpHandler,CDXLibrary "/>

You can choose any path you like, my choice was pretty arbitrary. One thing to remember is that, while this will work fine in Visual Studio (using the built in Cassini debugger), it will fail in IIS 6 unless you register whatever path you chose to be handled by ASP.NET isapi dll. IIS 7 doesn't have this problem if you use integrated mode, since all extensions are routed through .NET. 

 

 

Once this is configured, define your script groups, add the references in your pages, and you should be good to go.

The Results

The firebug windows below show the results of adding the script optimizer to a page. The number of blocking script requests is greatly reduced, shortening the "JavaScript load ladder" and reducing page load time by almost two thirds. (NB This is running locally in a test server with Firebug and Fiddler running. 9.91 seconds is not a good production page time!)

After:

 

Plans For The Future

This same technique could be applied to combine and compress css and possible html fragments if you load html dynamically in the client side. Eventually I plan to add a server control to replace the script reference tag, so that in debug mode I can return multiple script references for easier searching through code in Firebug. I would also like to do some server side caching of the resultant output, thus reducing the overhead of running JSMIN and reading all the file from disk. 

Sample code discussed in this posting is available here: script_optimizer.zip (8.37 kb)



Adding Interactive Graphs to Web Pages Using FLOT

clock April 10, 2009 10:50 by author Brian Kennedy

Overview

Adding graphs to web pages used to require building up images server side or using a flash applet. Server side images had bandwidth concerns and limited interactivity while flash applets required a another skill set to develop.

With the advent of client side JavaScript libraries like jQuery, several graphing toolkits have been written to allow client side graph generation and interaction. The one we use at Pragmatix is called flot (http://code.google.com/p/flot/). It is open source, easy to develop with, provides excellent cross browser functionality, is well integrated with our primary JavaScript library (jQuery) and generates very attractive graphs.

Figure 1 - Sample Flot Chart From a Pragmatix Product

Flot provides a host of useful features, including:

·         Intelligent scaling of axis based on series data

·         Zooming  and drill down with clickable data points

·         Automatically legend creation

·         Complete control over formatting, with sensible defaults

·         Simple binding to JSON data for interacting with web services

·         Interpolation of data. Data sets do not need to have a uniform collection of x-axis values.

Variety of graph types, including time series, scatter plot, bar, and stacked series. A variety of graph type can be combines into a single graph

[picture]

Walkthrough of Simple Graph

 

Our typical usage for flot is to calculate the data set server side and return a set of data series to load into flot. Flot expects the data for a series to be a array of point values, each point value being a simple two element array. This is then contained inside another object, with additional properties describing the series. We have a collection of classes we use to load the data that serialize well into JSON and that can be fed directly into our flot graph using the flot series format. A simple data series object is show below:

    public class SimpleDataSeries

    {
        #region Properties 

        public string label
        {
            get { return m_label; }
            set { m_label = value; }
        }

        private string m_label;

        public List<double[]> data
        {
            get { return m_data; }
            set { m_data = value; }
        }

        private List<double[]> m_data;

        #endregion

        public SimpleDataSeries(string label)
        {
            m_label = label;
            m_data = new List<double[]>();
        }

        public void AddDataPoint(double x, double y)
        {
            data.Add(new double[] { x, y });
        }
       } 

Flot has automatic capabilities for displaying and formatting time series data. For time series charts it expects the x values to be in JavaScript timestamps, which are the number of seconds since 1/1/1970. To handle time series we use another server side class:

    public class TimeDataSeries : SimpleDataSeries
    {
        #region Properties

        private DateTime UnixStart
        {
            get { return m_UnixStart; }
            set { m_UnixStart = value; }
        }
        private DateTime m_UnixStart;

        #endregion
        public TimeDataSeries(string Label)
            :base(Label)
        {
            UnixStart = new DateTime(1970, 1, 1);
        }

        public void AddDataPoint(DateTime x, double y)
        {
            double JavascriptTimestamp = x.Subtract(UnixStart).TotalMilliseconds;
            AddDataPoint(JavascriptTimestamp, y);
        }
    }

 

In order to get this data to the client, we use a web service that returns a listing of these data series in the response object.  Here is a web service call that returns a set of data series describing the storage utilization of several servers on our network, both for each machine individually and as a set of summary series:

        [WebMethod(true)]
        public IUpdateResponse Update(IUpdateRequest request)
        {
            // get the current user
            User CurrentUser = User.GetUser(HttpContext.Current.User.Identity.Name);
            // create a response and initialize it with base reponse data
            UpdateResponse response = new UpdateResponse();
            // add our main response properties
            VaultStorageCalculator storagecalc = new VaultStorageCalculator(request.VaultId, request.MachineIds, request.StartDate, request.EndDate, 45);

            GraphData gd = storagecalc.GetGraphData();
            response.DataSeries = gd.Machines;
            response.SummarySeries  = gd.Total;
            // return the response
            return response;
         }

    interface IUpdateResponse
    {
        List<SimpleDataSeries> MachineDataSeries { get; set; }
        List<SimpleDataSeries> SummarySeries { get; set; }
    }

We take advantage of the excellent serialization provided by the Microsoft AJX.NET server side libraries to serialize this to the client. Using this method, we keep most of the heavy lifting of calculating the series data on the server side, where we can use C# code that has direct access to our data model.

 

Once we have fetched this data on the client, building a simple chart on the client side can be done with only a few lines of code, like so:

        // plot the individual machines
        var options = {
            lines: { show: true },
            points: { show: true }
        };
        $.plot(this._element.find("div[type=chartArea]"), response.DataSeries, options);

By adding items to the options object, we can get custom control over display elements like the formatting of the axis labels.  By adding the code below to the options object, I can format a y-axis to display the server storage utilization in logical units of megabytes and gigabytes.

            options.yaxis:
            {
                mode: "time",
                tickFormatter:   function suffixFormatter(val, axis) {
                    if (val > 1000000000)
                      return (val / 1000000000).toFixed(axis.tickDecimals) + " GB";
                    else if (val > 1000000)
                      return (val / 1000000).toFixed(axis.tickDecimals) + " MB";
                    else if (val > 1000)
                      return (val / 1000).toFixed(axis.tickDecimals) + " kB";
                    else
                      return val.toFixed(axis.tickDecimals) + " B";
                  }
             };

We can tie event handlers to plot events, capturing a variety of events including:

·         clicking on data series elements

·         clicking on legend items

·         hovering over plot elements

·         selecting regions of the plot

For our server storage plot, we can bind an event to the chart to display a tooltip style popup when a user hovers over a data point. This allows us to show the actual storage values without cluttering up the main graph. Here is the code to bind to the hover event.

        $(this._element.find("div[type=chartArea_Total]")).bind("plothover", $.context(this).callback('_showTooltip'));

Another neat feature is the in depth control over the legend. By adding a legend object to our options definition, we can add arbitrary HTML markup to the legend. Here I use the legend markup to allow users to click on a machine in the legend to view another page with detailed machine data.

             options.legend =
             {
                show: true,
                container: this._element.find("div[type=legendArea]"),
                noColumns: 2,
                labelFormatter: function(label, series) {
                    // series is the series object for the label
                    return '<a href="#' + label + ' machine=' + series  + ' >' + label + '</a>';
                }
            }

 

 

Using these relatively simple technique, you can build highly interactive graphs using only JavaScript and .NET web services.



Calendar

<<  September 2010  >>
MoTuWeThFrSaSu
303112345
6789101112
13141516171819
20212223242526
27282930123
45678910

View posts in large calendar

Sign in