Archive

Archive for September, 2011

Improve Azure Table performance with query projections

September 27, 2011 Leave a comment

                At build 2011 Microsoft announced improvements in REST APIs used for Azure Storage with version (“2011-08-18”). Out of these improved features, a specific feature we are going to discuss here is about query projection while selecting entities from Azure Table storage.  In one of our project we had requirement for this feature but due to it’s unavailability we couldn’t implement it. Now, with this feature we will have to change our implementation but this change is good as it helps improve performance.

Earlier, selecting a specific or few properties from the Azure Table Storage was not supported. For example, in one of our implementations we were using a flag IsDeleted in table storage (with somewhere around 50 more properties) to maintain soft delete of an entity. Just to retrieve IDs of the entities we had to retrieve entire entity/ies with 50 properties and that would unnecessarily increase the latency and bandwidth load for the application.

e.g.

var query = from entity in playersServiceContext.CreateQuery<Players>(PlayersTableName)  where entity.IsDeleted == true;

 Now, with Version (“2011-08-18”) we can select specific properties using Select predicates which reduces the latency, bandwidth and improves the performance of application.

var query = from entity in playersServiceContext.CreateQuery<Players>(PlayersTableName) where entity.IsDeleted == true
select new 
{
  PlayerName =  entity.PlayerName,
  Rank = entity.Rank
};

Here, select new does the trick using anonymous entity.

We could explicitly project the properties into a specific entity type like 

var query = from entity in playersServiceContext.CreateQuery<Players>(PlayersTableName) where entity.IsDeleted == true
select new DeletedPlayers
           {
             PlayerName =  entity.PlayerName,
            Rank = entity.Rank
           };

Here, DeletedPlayers is a DataServiceEntity can be called as partial view entity.

In one of the other scenarios, we wanted to display the name, rank and age of players in a HTML table and then user might go into the details (view/edit) of the player using a link from the table. We had used paging size of 10 to decrease latency as there were around 50 properties attached with every player entity. Earlier, we had to retrieve all the 50 properties but now with this select predicate we can get only specific properties (Name, Rank, Age and Core properties) for the entity. This gives us better performance and an option to think about displaying more than 10 entities on a page with a good performance.

A few things to remember when using select predicate with Azure Table Storage.

  • If you are not defining core properties [PartitionKey, RowKey and ETag] in your data service entity definition then it would by default get selected in select predicate
  • In case you are explicitly defining core properties then you will have to explicitly select it in select predicate which would be useful for updating the properties
  • Prefer explicit entities (Partial) over anonymous entity as it makes the looping and updates explicit and improves readability
  • Remember even if you want to select only one property in the select predicate still you will have to at least use the anonymous entity. Just select entity.entityproperty would not work

So, go ahead and revisit and change your code for betterment where you had taken decisions just because query projections were not supported for Azure tables.

Portable Configurations for Windows Azure and Local IIS

September 22, 2011 Leave a comment

We were involved in migration an ASP.Net web application with SQL Server as backend to Windows Azure and SQL Azure. In this application, we wanted to perform functional and basic integration testing on local IIS server (not Azure Emulator but the IIS Server) and after the basic validations wanted to deploy the web application to Windows Azure.  In the process the basic requirement was to do this with minimal or no change in the code or configuration. So the problem statement:

Provide a portable mechanism for ASP.Net web application to run on local IIS connecting to on premise SQL Server and with minimal or no changes to code/configuration files, the same application should be able to deploy and run on Windows Azure connecting to SQL Azure database.

So, before jumping to the solution let us first drill down to the problem statement a bit. Basically, to achieve this portability between IIS and Windows Azure the configuration settings are the items of concern.  We could develop a custom configuration manager responsible for returning the respective configuration value to the caller depending on the environment (IIS/Azure) it is running on. 

We could replicate the appsettings from web.config to serviceconfiguration.cscfg file in cloud project. As the connection strings being used in the project are required to be in web.config we wanted a mechanism to select local or SQL Azure connection string depending on the whether it’s running on local IIS or on Windows Azure.

Below is the static method we implemented in a utility class (PortableConfigurationManager) to achieve the retrieval of configurations from the configuration files either web.config or serviceconfiguration.cscfg.  Basically it checks whether the RoleEnvironment.IsAvailable or not and accordingly it is going to return configuration values from either .cscfg or web.config.

        public static string GetConfigurationValue(string configurationName)

        {

            if (RoleEnvironment.IsAvailable)

            {

                return (TryGetConfigurationSetting(configurationName) == null) ?     TryGetAppSetting(configurationName) : TryGetConfigurationSetting(configurationName);

            }

            else

            {

                return TryGetAppSetting(configurationName);

            }        

        }

        private static string TryGetConfigurationSetting(string configName)

        {

            string ret = null;

            try

            {

                ret = RoleEnvironment.GetConfigurationSettingValue(configName);

            }

            catch (RoleEnvironmentException)

            {

                return null;

            }

            return ret;

        }

       private static string TryGetAppSetting(string configName)
        {
            string ret = null;
            try
            {
             ret = System.Configuration.ConfigurationManager.AppSettings[configName];
            }           
            catch (Exception)
            {
                return null;
            }
            return ret;
        }

Method also contains logic to fall back on web.config configuration settings in case it doesn’t exist in serviceconfiguration.cscfg while running on Azure.

The caller is going to retrieve the configuration values using

returnValue = CommonConfigurationManager.PortableConfigurationManager.GetConfigurationValue(txtConfigurationSettingName.Text);

         Configuration values existed at both the places in web.config (under appsetings) and serviceconfiguration.cscfg (under configurationsettings). Above method resolved the appsettings configuration for web.config.

 Now, the interesting part is to manage connection strings. As we know unlike serviceconfiguration.cscfg a change in web.config we have to redeploy the application package while deploying to Windows azure. To address this we added multiple connection strings to web.config file and for selecting the required one we pointed the name of the connection string from serviceconfiguration.cscfg.

Let us first look at the method which is responsible for returning the connection string in PortableConfigurationManager class.

 public static string GetConnectionString(string connectionName)
        {
            string connectionCofigurationName = TryGetConfigurationSetting(connectionName);
            if (string.IsNullOrEmpty(connectionCofigurationName))
            {
                return null;
            }
            else
            {
                return (TryGetConnectionString(connectionCofigurationName));
            }
        }

private static string TryGetConnectionString(string connectionName)
        {
            string ret = null;
            try
            {
                ret = System.Configuration.ConfigurationManager.ConnectionStrings[connectionName].ConnectionString;
            }            
            catch (Exception)
            {
                return null;
            }
            return ret;
        }
 

The public method takes parameter as string which indicates the connection name and the connection name is just a redirection from configuration to point to the connection string name to be used. The redirection is done through web.config for local IIS and serviceconfiguration.cscfg for Windows Azure.

Web.config:

   <connectionStrings>
    <add name="Local"
         connectionString="@Local Connection String"
         providerName="System.Data.SqlClient" />
    <add name="Azure"
        connectionString="@Azure Connection String"
        providerName="System.Data.SqlClient" />
  </connectionStrings>
  <appSettings>    
    <add key ="ConnectionName" value ="Local"/>
  </appSettings>

          ServiceConfiguration.cscfg

  <ConfigurationSettings>
      <Setting name ="ConnectionName" value ="Azure"/>
  </ConfigurationSettings>

You might have understood by now that we are using the configuration ConnectionName to select the appropriate connection string. When the application is running on Local IIS the configuration ConnectionName is going to return value “Local” and the method would return connection string which has name “Local”. Similarly when it is running on Azure it would return connection string which has name “Azure”.

Using this PortableConfigurationManager class and redirection in configuration we could achieve the configuration portability between local IIS and Windows Azure.

Sample code can be downloaded from here

Hope it helps.

Windows Azure AppFabric Caching Intro to Best Practices

September 21, 2011 Leave a comment

Get introduced to Windows Azure Appfabric Caching and get to know a few guidelines and best practices. It does talk about gotchas of using Azure Appfabric ASP.Net Session State Provider.

Improve surveillance with Cloud

September 12, 2011 Leave a comment

Cloud is everywhere and what else would be a better option (literally) to surveillance than using Cloud.

I don’t want to go in details of why surveillance is required and what is happening without it. I would like to share my thoughts on how patrolling systems in developing countries can use cloud computing to improve on surveillance. Diclaimer, I am not an expert of surveillance but I can surely tell you it is not being used efficiently or in some cases not used at all.

If you simply look at it, there are only three functions involved in the Surveillance System (SS now onwards) Capture, Transmit and Store. Capture function just generates the data and real information part happens at the store function. Search, View, Analysis etc. would happen on the Storage part of it. If an entire city has to be monitored using SS, imagine how much of data it would be and how much of infrastructure it would require.

I think decent tools for first two functions Capture and Transmit are readily available and the challenge might be to have a good tool for Storage. With all the benefits of cloud computing, its availability, affinity and nearly unlimited storage at inexpensive cost should be an option to consider.