Introduction

Previous Parts of This Series.

Note, the contents of the post covers non-ASP.NET applications. ASP.NET uses its own approach to configuration inheritance, and this article already contains more than enough information.

Also note, I call this configuration inheritance because it expresses the idea of one thing overriding another. Given the limited conceptual documentation about .NET’s configuration system I don’t think there is an official term. Until there is I will stick with “inheritance”.

When one uses the .NET configuration APIs to load settings into an application there is a rich structure supporting not just the kind of data that can be held, but also the scope of that data. Different applications can share data at the machine level, different AppDomains in the same process can have different configuration, and users can have their own settings either in the roaming part of their profile or purely local, or both.

Where this data is stored, and what influences where is the subject of this posting, but on the way there is a little coverage of how to get it and manage it but mostly that will be for the future.

The Four Levels

For non-ASP.NET there are four files, with each able—if the section allows it—to override the previous level. Some times of collection can also be modified by subsequence levels: clearing the collection, adding or removing members. The four levels are

1. Machine
2. Application
3. User Roaming
4. User Local

The Machine Configuration Level

This is the global file, normally stored in %SystemRoot%\Microsoft.NET\Framework64\‹Version›\Config\machine.config for 64bit .NET or %SystemRoot%\Microsoft.NET\Framework\‹Version›\Config\machine.config for 32bit. Where ‹Version› is the full version, e.g. v4.0.30319 for .NET 4.

It is instructive to read through this file—if you have enabled local active content in Internet Explorer the ability to collapse elements makes this easier—because you will see all the standard sections and section groups defined with the classes that implement them. For instance <appSettings> is implemented by System.Configuration.AppSettingsSection in assembly System.Configuration. It is also in the <connectionStrings> section at the machine level that connection “LocalSqlServer” is defined to be SQL Express using data file …\App_Data\aspnetdb.mdf which is then used as the store for ASP.NET profiles in their default configuration. (ASP.NET also has a global configuration “web.config” in the same folder with yet more global settings.)

As this file is read, by the configuration system, first any of the other levels can override it, unless the section is defined with allowDefinition or allowExeDefinition attribute is set to “MachineOnly” to ensure it can only be used at the machine level.

The Application Level

This is the most familiar level, creating a file called “app.config” in a Visual Studio project will create a file “‹assembly-name›.config” in the project output directory.

User Roaming and Local Levels

If you set things up correctly, per user files can also be used. The roaming file will be part of the user’s roaming profile, shared across machines if roaming is enabled. The local file will stay local. But you can override the filenames (see below) and as the content of a user’s roaming profile is based on a set of configurable folders, these names are strictly speaking indicative rather than absolute.

User levels are most useful when the ability to save configuration content is enabled. Rather than storing user settings and options in the registry use these configuration files.

The only problem with these files is the default naming, it is designed to ensure different versions of the same thing running side by side have separate configuration. If the location, assembly version or signing key changes then the filenames will be changed and any previous content (e.g. from before an upgrade) will be lost. This is bad news. The good news is that it is easy to specify your own filenames—and I would suggest doing that in most cases.

Using The User Levels

If you just use the static properties of ConfigurationManager (or one of the other configuration entry points) then machine and application configuration files are included, but user files are not.

To access either user-roaming, or user-roaming and user-local levels (you cannot use the local level without the roaming level—the roaming level could be left empty) one just uses one of the static ConfigurationManager methods that take a ConfigurationUserLevel parameter. The ConfigurationUserLevel enumeration allows selection of no user levels: ConfigurationUserLevel.None, just roaming: ConfigurationUserLevel.PerUserRaoming, or all levels: ConfigurationUserLevel.PerUserRaomingAndLocal.

If the configuration is opened with the ConfigurationManager.OpenExeConfiguration method then the default filenames, created by the configuration system, are used. Alternately using ConfigurationManager.OpenMappedExeConfiguration allows the filenames to be specified (more on this below, after covering the default filenames).

What Are The Default Filenames?

Disclaimer: none of this, as far as I can determine, is formally documented. Treat this information as subject to change in a future .NET version.

Default Filenames For The Default AppDomain of a Unsigned Executable

This is the simplest, and likely starting, case. The application configuration is the starting assembly’s path with “.config” appended. The user files will be:

  %USERPROFILE%\‹folder›\‹companyName›\‹AssemblyBase›_Url_‹hash›\‹version›\user.config

where

• folder: Is either “Roaming” or “Local” depending on the level.
• companyName: Is the value of the AssemblyCompanyAttribute if set. If not then the value of the AssemblyProductAttribute. If neither is set then this path fragment is not included (one less directory in the path).
• AssemblyBase: is the first part of the assembly’s filename (including the first period and two further characters).
• hash: is an alphanumeric string created by hashing the full path of the assembly in some implementation determined way.
• version: is the value of the AssemblyVersionAttrinbute.

With Code Signing

If the assembly is signed then there is a small change. Rather than ‹AssemblyBase›_Url_‹hash› we have ‹AssemblyBase›_StrongName_‹hash› with the hash being based on the signing key (the rest of the path is the same). This means by signing the assembly the location of the user configuration files becomes independent of the location of the assembly. If you want xcopy deployment then signing would seem to be a good idea to allow the user to relocate the application while preserving their configuration.

With Other AppDomains

If the configuration is loaded from a different AppDomain the machine and application configuration files remain the same. However the user levels file names change. In the above path AssemblyBase is replaced by the AppDomain name. Another case of trying to ensure uniqueness.

One of the options on creating an AppDomain is to specify a different configuration file:

var adConfigPath = Path.GetDirectoryName(AppDomain.CurrentDomain.SetupInformation.ConfigurationFile);
adConfigPath = Path.Combine(adConfigPath, "AppDomain.config");
var adSetup = new AppDomainSetup {
  ConfigurationFile = adConfigPath
};
otherDomain = AppDomain.CreateDomain("WithDifferentConfig", null, adSetup);

In this case replacing the AssemblyName.config with AppDomain.config. This results in the assembly level configuration file being the filename specified, but no other levels are changed. Specifically the user level filenames remain the same whatever application level configuration is specified.

Note, if you don’t give an AppDomain a name on creation, .NET will: there are no unnamed AppDomains.

Defining You Own Filenames

As noted above the default naming is not ideal (if you want to be able to revise the application) except in being confident of its uniqueness. You can however define all the filenames, except at machine level, yourself.

To specify your own filenames you need to specify all the levels you are going to use, except the machine level. This includes the application level (to use the default just use AppDomain.CurrentDomain.SetupInformation.ConfigurationFile (this seems to be always populated whether not specified on AppDomain creation or in the default AppDomain) or Assembly.GetEntryAssembly().Location + ".config".

The real benefit is being able to specify you own path, to be independent of assembly/AppDomain name, assembly version, signing key and assembly location. Just construct the path you need, and remember with a new version you need to ensure that structural changes are either backwards compatible, or you include conversion code. Something like:

private static string GetProfileConfigPath(Assembly targetAssembly, Environment.SpecialFolder pathId) {
  var profile = Environment.GetFolderPath(pathId,
                                          Environment.SpecialFolderOption.None);
  var company = targetAssembly.GetAttribute<AssemblyCompanyAttribute>().Company;
  if (String.IsNullOrWhiteSpace(company)) {
    throw new InvalidOperationException("AssemblyCompanyAttribute must be set");
  }
  var configPath = Path.Combine(profile, company);
  configPath = Path.Combine(configPath, targetAssembly.GetName().Name);
  configPath = Path.Combine(configPath, "user.config");
  return configPath;
}

This can be called with each of Environment.SpecialFolder.ApplicationData and Environment.SpecialFolder.LocalApplicationData to get the current users roaming and local profile root folders (respectively). (The GetAttribute<T>() method is a helper extension method over Assembly I have which gets the single instance of attribute T from the assembly.) Normally this would be passed the entry assembly, but for an extension could be passed the extension‘s assembly to keep configuration separate (and thus avoid conflicts) from the containing application.

Once you have the filenames you want to use, populate an instance of ExeConfigurationFileMap and pass to ConfigurationManager.OpenMappedExeConfiguration to get a Configuration instance, which is used in exactly the same way as an instance obtained any other way.

Discovering the Names Being Used At Runtime

There are two approaches: defined and laborious, or using reflection to peer inside the Configuration instance. The safe method is based on Configuration.FilePath being the path of the lowest level of configuration in use. So open with different ConfigurationUserLevel values and get each path in turn.

The reflection method relies on Configuration having private member _configRecord of private type System.Configuration.MgmtConfigurationRecord. This is a linked list of MgmtConfigurationRecord instances, one per level. Each has a _configName field which gives the level as a string (“MACHINE”, “EXE”, …) giving the level, and a ConfigurationFilePath property with the path. Given an instance of Configuration and a few helpers a populated ExeConfigurationFileMap can be returned.

Reference

In the posting: “Client Settings FAQ” Raghavendra Prabhu covers some details of the user-roaming and user-local configuration levels. Including that the hash following the _Url_ or _StrongName_ is based on the evidence—i.e. of the URL or signing key.

Previous Part of This Series on .NET Custom Configuration

1. Demonstrating Custom Configuration Sections for .NET
2. Demonstrating Custom Configuration for .NET (Part 2): Custom Elements
3. Demonstrating Custom Configuration for .NET (Part 3): Validation
4. Custom Configuration for .NET (Part 4): Backgrounder on Configuration Sections

Updated 2010-06-22: Added link to part 5.

Introduction

Previous Parts of This Series.

There are two remaining approaches to providing structure that I have not yet covered: collections of elements and section groups. Element collections will be a topic for several future posts as there are a number of different approaches to creating element collections, and two of them make little sense without first covering configuration inheritance (see part 5).

Section groups are much simpler, and mostly a matter of declaring them.

Creating Section Groups

When custom sections are declared using the <configSections> element of a configuration file, one can declare both sections (with the <sections> element, using attributes to define the associated custom type and its name) and section groups.

To define a section group use the—unsurprisingly—<sectionsGroup> element. This needs a name and a type. Often the type will be System.Configuration.ConfigurationSectionGroup which adds no behaviour (but one can customise this, more below) and just allows programmatic access to its child sections and section groups (there is no specified limit to the depth of structure you can define—but don’t go overboard with deep structures which will become increasingly hard to understand and maintain).

For instance:

   <configuration>
    <configSections>
    <sectionGroup name="userSettings"
          type="System.Configuration.ConfigurationSectionGroup, System.Configuration, Version=4.0.0.0,
                                                                Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
      <section name="recentFiles"
           type="CustomConfiguration.RecentFilesConfigurationSection, MyAssembly"/>
    </sectionGroup>
    </configSections>

    <userSettings>
    <recentFiles showFiles="5" removeNoneExistant="true"/>
    </userSettings>

  </configuration>

Which defines a userSettings element to act as a container for (presumably) multiple sections describing many aspects of user behaviour.

Note, replace the strong name for System.Configuration with “System.Configuration.ConfigurationSectionGroup, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a” for .NET versions 2, 3 and 3.5.

Reading Section Groups and Contained Sections

In previous parts of this series the application code used Configuration.GetSection(‹name›) to get to its custom section (and the instance of Configuration was obtained with ConfigurationManager.OpenExeConfiguration). This works for configuration sections that are immediate children of the root <configuration> node.

var config = ConfigurationManager.OpenExeConfiguration(…);
var userSettingsGroup = config.GetSectionGroup("userSettings");
var recentFiles = (RecentFilesConfigurationSection) userSettingsGroup.GetSection("recentFiles");

GetSectionGroup returns a ConfigurationSectionGroup instance which has GetSection and GetSectionGroup methods as Configuration has.

Configuration Section and Group Collection Properties

You might notice that the Configuration and ConfigurationSectionGroup also have properties Sections and SectionGroups which can be indexed by name. These give another approach to getting a section or section group:

  var myConfig = (MyConfigurationSection)config.GetSection("mine");
  var myConfig2 = (MyConfigurationSection)config.Sections["mine"];

are completely equivalent.

Custom Section Group Types

It is also possible to create your own custom configuration section group by deriving from ConfigurationSectionGroup, and a number of .NET Framework types do this. E.g. System.Xml.Serialization.Configuration.SerializationSectionGroup defined in the System.Xml assembly, which adds properties to type safely access contained sections (i.e. the properties contain the access into the Sections collection and cast the results to the correct type).

Previous Part of This Series on .NET Custom Configuration

1. Demonstrating Custom Configuration Sections for .NET
2. Demonstrating Custom Configuration for .NET (Part 2): Custom Elements
3. Demonstrating Custom Configuration for .NET (Part 3): Validation

Introduction

Previous Parts of This Series.

Parts 1 and 2 added structure and data, but this is often not enough, one needs valid data—the domain of the data for your configuration being narrower than the domain of the underlying type of the parsed values. Hence the need to extend the custom types representing the structure with more types and attributes to limit the ranges of those values.

Value Validation

Checking single values is the easiest part, albeit needing two custom types to be defined for each distinct kind of validation, but at least each case can be parameterised (i.e. if you need to check a date range, you only need two types, not four).

First: The Validator

The first type is derived from System.Configuration.ConfigurationValidatorBase. This will actually do the validation in two steps:

1. Overriding CanValidate to confirm that the type can be validated. Return true to indicate you can, otherwise validation will fail. This override is usually used to confirm the type of the value matches the expectations of the next method.
2. Override Validate to perform the validation. To indicate failure throw an exception (which will be wrapped by the configuration runtime, into a ConfigurationErrorsException.

NB. the validation needs to handle the default (before the configuration content is parsed) value. E.g. a DateTime value will be initially validated with a value of DateTime.MinValue before being called again with the value read from the configuration file. (For types which can be used with code attributes—see §17.1.3 of the C# specification, the default value is an optional parameter for the ConfigurastionProperty attribute. For other types, the code attribute based declaration of configuration values can be replaced with a more programmatic one, which I should cover later in this series.)

For instance to check that a DateTime value has a minimum year the following validation code will work (note the check to allow MinValue):

public class DateTimeValidator : ConfigurationValidatorBase {
    private DateTime lowerLimit;

    internal DateTimeValidator(DateTime lowerLimit) {
      this.lowerLimit = lowerLimit;
    }

    public override bool CanValidate(Type type) {
      return type.Equals(typeof(DateTime));
    }

    public override void Validate(object value) {
      Trace.Assert(value.GetType().Equals(typeof(DateTime)));
      DateTime val = (DateTime)value;

      // MinValue is used as a placeholder before the actual value is set
      if (val < lowerLimit && val != DateTime.MinValue) {
        throw new ArgumentException("Invalid DateTime value in configuration");
      }
    }
  }

Second: The Attribute

The second type is derived from ConfigurationValidatorAttribute (which itself derives from System.Attribute). This is used to (1) annotate the configuration property in the ConfigurationElement (or ConfigurationSection) type, and (2) be an object factory for the first, validation, type. Additional parameters can be passed from this attribute to the validator.

The key override is ValidatorInstance which needs to return an initialised instance of the validator class.

For example, to support the DateTimeValidator type, the following:

  [AttributeUsage(AttributeTargets.Property)]
  public class DateTimeValidatorAttribute : ConfigurationValidatorAttribute {
    public DateTimeValidatorAttribute() {
    }

    public override ConfigurationValidatorBase ValidatorInstance {
      get {
        return new DateTimeValidator(new DateTime(minYear, 1, 1));
      }
    }

    private int minYear = 1900;
    public int MinimumYear {
      get { return minYear; }
      set { minYear = value; }
    }
  }

Using Validation

The only change is to annotate the configuration property with the just defined attribute, passing any necessary additional parameters, e.g.:

  [ConfigurationProperty("startDate", IsRequired=true),
   DateTimeValidator(MinimumYear=2000)]
  public DateTime StartDate {
    get { return (DateTime)this["startDate"]; }
    set { this["startDate"] = value; }
  }

  [ConfigurationProperty("endDate", IsRequired=true),
   DateTimeValidator()]
  public DateTime EndDate {
    get { return (DateTime)this["endDate"]; }
    set { this["endDate"] = value; }
  }

Note the use of an explicit lower limit for StartDate but not for EndDate. (DateTime is not a type with literals available for attribute parameters, hence just using the year here.)

What Happens on Validation Failure?

Validation is performed when the configuration section is read, i.e. when Configuration.GetSection(‹name›) is called. Importantly this means that if configuration sections are not used, validation will not be performed, and no errors will be reported, however many invalid, missing or extra values there are. Also note that extra attributes and elements will be reported as errors, this behaviour can be modified by overriding ConfigurationElement’s OnDeserializeUnrecognizedAttribute or OnDeserializeUnrecognizedElement.

When an error, validation or otherwise (including malformed XML) occurs a ConfigurationErrorsException is thrown. If this is based on another exception being thrown internally (e.g. on malformed XML, a XmlException) then that original exception may be the ConfigurationErrorsException’s InnerException (sometimes this seems to be the base, other times not—I don’t see much consistency).

Any reporting of this to the user or logging is up to the application, this is not trivial as while the text of the message does include key information (like where the error was in the configuration file) it is not exactly in a user friendly format. But then configuration files are not targeted at (typical) end user direct editing.

Checking Values Together

There is no direct support for further validation after each individual value has been loaded and (given suitable attributes) validated. But ConfigurationElement does have the PostDeserialize method, which is called at the right time.

But the error message is not ideal, starting with the text: “An error occurred creating the configuration section handler for ‹section-name›”. But it does work.

See the code for AppConfigSection.cs for an example

The Complete Solution

DateTimeValidator.cs

namespace ConfigurationDemoPart3 {
  public class DateTimeValidator : ConfigurationValidatorBase {
    private DateTime lowerLimit;

    internal DateTimeValidator(DateTime lowerLimit) {
      this.lowerLimit = lowerLimit;
    }

    public override bool CanValidate(Type type) {
      return type.Equals(typeof(DateTime));
    }

    public override void Validate(object value) {
      Trace.Assert(value.GetType().Equals(typeof(DateTime)));
      DateTime val = (DateTime)value;

      // MinValue is used as a placeholder before the actual value is set
      if (val < lowerLimit && val != DateTime.MinValue) {
        throw new ArgumentException("Invalid DateTime value in configuration");
      }
    }
  }

  [AttributeUsage(AttributeTargets.Property)]
  public class DateTimeValidatorAttribute : ConfigurationValidatorAttribute {
    public DateTimeValidatorAttribute() {
    }

    public override ConfigurationValidatorBase ValidatorInstance {
      get {
        return new DateTimeValidator(new DateTime(minYear, 1, 1));
      }
    }

    private int minYear = 1900;
    public int MinimumYear {
      get { return minYear; }
      set { minYear = value; }
    }
  }
}

AppConfigSection.cs

namespace ConfigurationDemoPart3 {
  public class AppConfigSection : ConfigurationSection {

    [ConfigurationProperty("startDate", IsRequired=true),
     DateTimeValidator(MinimumYear=2000)]
    public DateTime StartDate {
      get { return (DateTime)this["startDate"]; }
      set { this["startDate"] = value; }
    }

    [ConfigurationProperty("endDate", IsRequired=true),
     DateTimeValidator()]
    public DateTime EndDate {
      get { return (DateTime)this["endDate"]; }
      set { this["endDate"] = value; }
    }

    protected override void PostDeserialize() {
      if (StartDate > EndDate) {
        throw new ArgumentException("EndDate must be after StartDate");
      }
      base.PostDeserialize();
    }
  }
}

Program.cs

namespace ConfigurationDemoPart3 {
  class Program {
    static void Main(string[] args) {
      var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
      ShowSection(config, "custom");
      ShowSection(config, "custom2");
    }

    private static void ShowSection(Configuration config, string secName) {
      try {
        var customSection = config.GetSection(secName) as AppConfigSection;

        Console.WriteLine("Config section \"{2}\": start = {0:d}, end = {1:d}", customSection.StartDate, customSection.EndDate, secName);

      } catch (ConfigurationErrorsException e) {
        var inner = e.InnerException;
        Console.WriteLine("Loading config section \"{0}\" failed: {1} (inner type {2})",
          secName, e.Message, inner == null ? "<None>" : inner.GetType().Name);
      }
    }
  }
}

app.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
  <section name="custom"
       type="ConfigurationDemoPart3.AppConfigSection, ConfigurationDemoPart3"/>
  <section name="custom2"
       type="ConfigurationDemoPart3.AppConfigSection, ConfigurationDemoPart3"/>
  </configSections>

  <custom startDate="2000-05-20" endDate="2010-05-22"/>
  <custom2 startDate="2010-05-20" endDate="2009-05-22"/>
</configuration>

Previous Part of This Series on .NET Custom Configuration

1. Demonstrating Custom Configuration Sections for .NET
2. Demonstrating Custom Configuration for .NET (Part 2): Custom Elements

Introduction

Previous Parts of This Series.

Part 1 kept things really simple. But for non-trivial custom configuration just having XML attributes to work with starts to get very hard to edit in the file. Rather one wants to employ the richer hierarchical structures that XML allows: child elements. This turns out to be very easy.

Creating A Custom Element

To keep things simple, this example will just change part 1’s structure:

<custom date="2010-05-13" name="element name" />

into this:

<custom>
  <data date="2010-05-21" name="custom data name"/>
</custom>

All this takes is deriving a new class from System.Configuration.ConfigurationElement with code attributes for the two configuration attributes, and then using this new type as the type of the code property in the ConfigurationSection derived class. See the source code below: DataConfigurationElement is the new custom element, and AppConfigSection is the updated custom section, now with just a single code property.

One limitation of this use of attributes is that text content is not supported (<element>text content</element>), custom elements still just use attributes, or child custom elements (there does not seem to be any depth limit). Unless one completely overrides the de-serialisation code that makes custom elements easy (by overriding the ConfigurationElement.DeserializeElement method), but then all the work—including parsing the attributes—would have to be manually coded using the passed XmlReader instance.

The Complete Solution

DataConfigurationElement.cs

namespace ConfigurationDemoPart2 {
  public class DataConfigurationElement : ConfigurationElement {
    [ConfigurationProperty("date", IsRequired=true)]
    public DateTime Date {
      get { return (DateTime) this["date"]; }
      set { this["date"] = value; }
    }
    [ConfigurationProperty("name", IsRequired=true)]
    public string Name {
      get { return (string)this["name"]; }
      set { this["name"] = value; }
    }
  }
}

AppConfigSection.cs

namespace ConfigurationDemoPart2 {
  public class AppConfigSection : ConfigurationSection {

    [ConfigurationProperty("data", IsRequired=true)]
    public DataConfigurationElement Data {
      get { return (DataConfigurationElement)this["data"]; }
      set { this["data"] = value; }
    }
  }
}

Program.cs

namespace ConfigurationDemoPart2 {
  class Program {
    static void Main(string[] args) {
      var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
      var customSection = config.GetSection("custom") as AppConfigSection;
      var dataElement = customSection.Data;

      Console.WriteLine("Config: date = {0}, name = {1}", dataElement.Date, dataElement.Name);
    }
  }
}

app.config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="custom"
             type="ConfigurationDemoPart2.AppConfigSection, ConfigurationDemoPart2"/>
  </configSections>

  <custom>
    <data date="2010-05-21" name="custom data name"/>
  </custom>
</configuration>

Previous Part of This Series on .NET Custom Configuration

1. Demonstrating Custom Configuration Sections for .NET

Introduction

The <appSettings> element of a .NET .config with the ConfigurationSettings.AppSettings static collection to read them make adding a few simple string configuration settings to an application easy. But, limited to simple name-value pairs, this is (to put it nicely) limited. And then one compares it to the richness of structure (for example) the CLR and ASP.NET support. However all that capability to create rich hierarchical structures, including multiple types of collections with all values validated is also open to the developer.

It is just not obvious where to start (at least the documentation for .NET 4, e.g. here, is a big improvement over .NET 2's).

Hence this series of posts which aim to be an aide-mémoire of various scenarios for different types of .config structures.

Keeping It Simple: A Single Element

How to add this

  <custom date="2010-05-13" name="some text" />

to the application (or ASP.NET) configuration file?

Note: both XML and C# code have attributes, so I’ll write configuration attribute for XML attributes on the config file, and code attribute for those in the source code.

Setting Up The Project

Any kind of .NET project can be used, I’m using C# in a console project here (so I can just focus on configuration). This will also work against a .NET 2, 3, 3.5 or 4 target. The sample here is a console application called ConfigurationDemoPart1, so that will also be the namespace.

Start by adding a reference to the System.Configuration assembly.

Then add a new public class, for this sample called AppConfigSection. Derive from System.Configuration.ConfigurationSection. This base is needed for any custom type that is referenced by the <configSections> to define the configuration file section being created.

Defining Attributes on the Configuration Element

To create each configuration attribute to appear on the custom element add a property to the section class.

Each property needs a System.Configuration.ConfigurationPropertyAttribute code attribute with the name of the configuration attribute, and must use the default dictionary inherited from ConfigurationSection as a backing store: do not use automatic properties or a local field. The key name used in the dictionary must match the name in the ConfigurationProperty.

This is easier to show by example than explain:

  [ConfigurationProperty("date")]
  public DateTime Date {
      get { return (DateTime)this["date"]; }
      set { this["date"] = value; }
  }

Accessing the Custom Configuration

Without this the whole effort would be pointless, why write configuration file content if you are not going to use it?

Create an instance of System.Configuration.Configuration by using the OpenExeConfiguration method of System.Configuration.ConfigurationManager. Pass System.Configuration.ConfigurationUserLevel.None to just access the application configuration file (or web.config). Per-user configuration files, and directly loading a specific file (i.e. not the default name) is also possible, but that’s something for later.

Get to the custom configuration with GetSection(name) and casting to the custom configuration type. The properties will have the values from the configuration file.

You can also set the value of the properties. While this will change the value subsequent reads of the properties returns it will not change the configuration file. It is possible to save changes, but that’s a topic for a later post.

Adding To The Configuration File

As well as adding the new custom element, the configuration system needs to be told about the new element. Do this with the <configSections> element, giving the new section a name (which you also need to pass, in code, to GetSection) and a type:

  <configSections>
    <section name="custom"
             type="ConfigurationDemoPart1.AppConfigSection, ConfigurationDemoPart1"/>
  </configSections>

Possible Errors

A null reference on accessing a property in code to get a configuration attribute’s value probably means one of the code attribute on the custom configuration property, the index into the values dictionary or the configuration attribute is spelt differently. (This took ages to work out the first time it happened to me, now it is always the first thing to check.)

The Complete Solution

AppConfigSection.cs

This defines the custom type

  namespace ConfigurationDemoPart1 {
    public class AppConfigSection : ConfigurationSection {

      [ConfigurationProperty("date")]
      public DateTime Date {
        get { return (DateTime)this["date"]; }
        set { this["date"] = value; }
      }

      [ConfigurationProperty("name")]
      public string Name {
        get { return (String)this["name"]; }
        set { this["name"] = value; }
      }
    }
  }

Program.cs

Minimal demonstration code using the custom configuration

  namespace ConfigurationDemoPart1 {
    class Program {
      static void Main(string[] args) {
        var config = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
        var customSection = config.GetSection("custom") as AppConfigSection;

        Console.WriteLine("Config: date = {0}, name = {1}", customSection.Date, customSection.Name);
      }
    }
  }

app.config

Visual Studio will convert this to ConfigurationDemoPart1.exe.config when the project is built.

  <configuration>
    <configSections>
      <section name="custom"
               type="ConfigurationDemoPart1.AppConfigSection, ConfigurationDemoPart1"/>
    </configSections>

    <custom date="2010-05-13" name="element name" />
  </configuration>

Introduction

The next step in this series on creating my own IObservable<T> implementation is concurrency. (See below for links to previous parts, Part 5 has a full code listing.)

This is the hard part. Too easy to think it is OK when it isn’t, and too hard to validate that is really thread safe.

The only option is careful analysis, and then double (triple, …) check everything.

After all even the best set of unit tests will fail to find problems if it takes a particular combination of OS version, hardware and other processes running to show the problem. With race conditions, deadlocks and all the other concurrency fun such is not just a possibility but likely.

But first, the new 1.0.2350.0 (2010-03-15) build of Rx makes no change to my code, all tests pass without any changes. And this is now running on my main system (Rx has go-live licence, so why not) with its multiple codes (previously was working on a VM with a single virtual CPU).

What Needs Protection?

At the heart of my implementation there is some shared data. This needs protecting where multiple threads could operate concurrently. E.g. while events are pushed to subscribers using a copy of the list of subscribers (this allows re-entrant subscription and un-subscription) taking a copy of the values of a Dictionary<K,V> certainly is not safe.

Some operations are inherently safe (defined by the platforms object model, see Joe Duffy’s blog for more details). In this case reading or writing an aligned, volatile int or reference is safe (such reads and writes are inherently atomic). And only a single field needs to be checked, written to, or both compared & exchanged (via Interlocked.CompareExchange):

• Checking if the endOfEvents field (reference to a Notification<T>) is null.

• The boolean field pushingEvents used only in PushEvents (the method that loops through a snapshot of the current subscribers) to avoid re-entrant or multi-threaded pushing of events.

  When it is read, and true in a simple check there is no problem (just return from PushEvents. But if it is false it then needs to be set. So two operations are needed: a read and a write. Interlocked.CompareExchange will take care of this without a lock. (Before returning from pushing events it is set back to false, but a single write is also OK.)

  As there is no overload of Interlocked.CompareExchange for bool the field will need to be an int with values 0 and 1.

• To avoid races in Next, Completed and Error between checking for endOfEvents and setting it one can use Interlocked.CompareExchange.

There are also two pieces of functionality that require multiple operations on fields (or objects that are not thread safe). In each case a lock is needed.

• The collection of subscribers and subscriber id counter. With operations to add or remove a subscriber, and to snapshot a list of all subscribers. This will require a lock, there is no way (I can see) around this.

• To operate on the queue of pending events can be handled by changing to an instance of ConcurrentQueue<T> with modifications in PushEvents to use TryDequeue rather than a combination of checking the count and then Dequeue.

What Doesn’t Need Protection

This implementation is not about being fair. So if two threads both try and use an instance together, with one calling Next and the other Completed only the Windows thread scheduler gets any say in which wins (if Completed is called first, no subscriber will see the value passed to Next).

Ready to Implement?

Even as I wrote the analysis above (which was already the second pass through what would be needed) I noted that I could avoid a lock for Next, Completed, and Error and use CompareExchange. So I’ve probably still missed something: time for another pass over the code.

And another question is how to test…? Throwing a load of cocurrent tasks from the Parallel Extensions (see here) will be a start, but cannot really cover mixing in Completed and Error because I cannot determine which thread will win and which will not, and thus no way to verify the “right one” won in a unit test assert.

More to work on.

Previously

• Part 1: The basic implementation of Subscribe, unsubscribe and Next.

• Part 2: Making Subscribe and unsubscribe re-entrant.

• Part 3: Why re-entrancy is important.

• Part 4: Making Next re-entrant.

• Part 5: Implementing OnCompleted and OnError to complete support for full IObserver<T> interface.

• Part 5a: Updates for build 1.0.2317.0 of the Reactive Extensions (2010-03-05).

The new code drop came through, 2½ months after the last drop, on 5^th March. The release notes are suitably long. With a fair amount of change, much of it under the covers to edge cases.

But a fair number of things have moved assemblies and changed name.

Impact on My Implementation

Given all the change, there was relatively little impact. The only thing that broke was my use of Notification<T>: the Current property is now called Value. And it now has an Exception property, which saves a cast when handling an OnError notification.

My (unpublished) test harness used GroupDisposable to clean up subscriptions, this was renamed CompositeDisposable and moved to the CoreEx assembly.

Both of these renames give slightly better and more consistent names, and the addition of the Exception property is definitely better. So overall the new drop is, for me, an improvement so far: once the code compiled all tests just passed.

The only function that changed was SendNotification so following DRY helped. The new implementation:

private static void SendNotification(IObserver<T> obs, Notification<T> n) {
    switch (n.Kind) {
    case NotificationKind.OnNext:
        Debug.Assert(n.HasValue);
        obs.OnNext(n.Value);
        break;
    case NotificationKind.OnCompleted:
        obs.OnCompleted();
        break;
    case NotificationKind.OnError:
        obs.OnError(n.Exception);
        break;
    }
}

So Where’s The Concurrency Support…?

Coming! :-)
I've been rather head down in learning WPF for something else, and trying to avoid too many distractions while getting my head around the basics of WPF. Concurrency support is perhaps the biggest challenge here and I therefore want to be able to focus on it and get it right. This will mean taking a block of focused time.

Introduction

So far in this series (links below) the focus has been on implementing Next and Subscribe in a way that is re-entrant. But that leaves two methods uncovered. Both Completed and Error end the sequence, but in different ways. Error passes an exception indicating the source has failed, whereas Completed just indicates it has ended.

Based on Subject<T> when a sequence of events is ended:

• Existing subscribers received a call to their OnError or OnCompleted (as applicable).

• Further calls to any of Subject<T>'s IObserver<T> methods is a no-op.

• Any new subscribers immediately get a call to their OnError or OnCompleted (as applicable) and nothing more.

Previously

• Part 1: The basic implementation of Subscribe, unsubscribe and Next.

• Part 2: Making Subscribe and unsubscribe re-entrant.

• Part 3: Why re-entrancy is important.

• Part 4: Making Next re-entrant.

Notification<T> As A Way to Store An Observable Sequence

The reactive Framework (Rx) distributable comes with three assemblies:

• System.CoreEx

• System.Reactive

• System.Interactive

System.CoreEx contains types that would have been in System or System.Core had .NET been designed with Rx in it from the start. System.Interactive extends LINQ to Objects by back-porting many of IObservable<T>'s extension methods (or ’operators”) to IEnumerable<T>. System.Reactive implements LINQ like operators for IObservable<T>.

My observable implementation almost doesn’t need any of these three assemblies (IObservable<T> and IObserver<T> being defined in the mscorlib assembly).

However one rather useful group of types defined in the System.CoreEx assembly in namespace System.Collections.Generic are Notification<T> and its subtypes. These types allow the “value” of an observable event to be stored. Using one of the three subtypes (one each for OnNext, OnCompleted and OnError) defined as members of Notification<T> (hence created as, for example, new Notification<int>.OnNext(1)). Therefore a collection of events covering all three IObserver<T> methods can be easily created without some custom helper types.

These types are provided for the Materialize and Dematerialise enumerable and observable operators (the former being one the back-ported operators in System.Interactive).

Overview of The Implementation of Completed and OnError

Both follow the same pattern as the existing Next implementation:

1. If ended, return

2. Add the new event to a queue of pending events.

3. If not already pushing an event, process the queue.

The first of these steps is new, and handles the requirement that either Completed or OnError ends the sequence.

To make handling new subscriptions after the end of the sequence the endOfSequnec field is a Notification<T> reference. If null the sequence has not ended, otherwise this is the event to immediately push. Also a helper from System.Reactive is used as the return value: System.Disposables.Disposable.Empty which is an instance of a type which implements IDisposable as a no-op and these is no need to track this after the end subscriber (it will never be called again).

The New Implementation

The single biggest change to support storing notifications (rather than values of type T>) is in PushEvents where different kinds of notifications need to call different IObserver<T> methods, so a switch is needed. This is in its own method (SendNotification) because it can then be reused to send the ending event to post-end subscribers.

The implementations of Next, Completed and Error end up being very similar and simple.

Overall, compared to the code from part 4, the implementation is a somewhat longer, but certainly better factored with each method doing one thing. The most complex (PushEvents) does have to loop over both pending events and subscribers, but that is all it does.

The Code

using System;
using System.Collections.Generic;
using System.Linq;

public class Observable4<T> : IObservable<T> {
    private Dictionary<int, IObserver<T>> subscribers = new Dictionary<int, IObserver<T>>();
    private int nextSubscriber = 0;
    private Queue<Notification<T>> pendingEvents = new Queue<Notification<T>>();
    private bool pushingEvent = false;  // If true, only queue the event.
    private Notification<T> endOfEvents;   // Not null => have had ended, and this is the Completed or Error that ended it.

    public IDisposable Subscribe(IObserver<T> observer) {
        if (endOfEvents != null) {
            // Already have end of sequence, so just tell this new subscriber
            // and return no-op.
            SendNotification(observer, endOfEvents);
            return System.Disposables.Disposable.Empty;
        } else {
            subscribers.Add(nextSubscriber, observer);
            return new Observable4Disposer(this, nextSubscriber++);
        }
    }

    private void Unsubscribe(int index) {
        subscribers.Remove(index);
    }

    public void Next(T value) {
        if (endOfEvents != null) { return; }
        pendingEvents.Enqueue(new Notification<T>.OnNext(value));
        PushEvents();
    }

    public void Completed() {
        if (endOfEvents != null) { return; }
        endOfEvents = new Notification<T>.OnCompleted();
        pendingEvents.Enqueue(endOfEvents);
        PushEvents();
    }

    public void Error(Exception exn) {
        if (endOfEvents != null) { return; }
        endOfEvents = new Notification<T>.OnError(exn);
        pendingEvents.Enqueue(endOfEvents);
        PushEvents();
    }

    private void PushEvents() {
        if (!pushingEvent) {
            try {
                pushingEvent = true;
                while (pendingEvents.Count > 0) {
                    // To allow subscribers to unsubscribe while calling them, don't
                    // directly iterate over the collection, but use a copy.
                    // Take this each time, in case some event triggers a change
                    var subs = subscribers.Values.ToArray();
                    var n = pendingEvents.Dequeue();
                    foreach (var s in subs) {
                        SendNotification(s, n);
                    }
                }
            } finally {
                pushingEvent = false;
            }
        }
    }

    private static void SendNotification(IObserver<T> obs, Notification<T> n) {
        switch (n.Kind) {
        case NotificationKind.OnNext:
            obs.OnNext(n.Current);
            break;
        case NotificationKind.OnCompleted:
            obs.OnCompleted();
            break;
        case NotificationKind.OnError:
            obs.OnError(((Notification<T>.OnError)(n)).Exception);
            break;
        }
    }


    private class Observable4Disposer : IDisposable {
        private Observable4<T> target;
        private int index;
        internal Observable4Disposer(Observable4<T> target, int index) {
            this.target = target;
            this.index = index;
        }

        public void Dispose() {
            target.Unsubscribe(index);
        }
    }
}

Further Steps

The toughest part to get right is still left: concurrency. This will be difficult as much to test as to implement, tests using workers are never simple and just being confident that they are doing things concurrently is hard (after all a simple Interlocked.Increment might be slow enough to effectively eliminate any concurrency due to forcing CPU cache synchronisation even with multiple idle cores).

Introduction

In building my own implementation of IObservable<T> across a series of posts (Part 1, Part 2, Part 3 and Part 4) I’ve been making heavy use of Observer.Create<T> to create helper observer instances to ensure that the right methods of the subscribed observers are called, the right number of times (and even—although somewhat harder to track in all but the simplest cases—the right order).

The Discovery

In working towards Part 5 which will cover implementing the two, so far missing, methods of IObserver<T>: OnCompleted and OnError I was rather surprise when this test passed:

[TestMethod]
public void Observable4_CallingCompletedAgainIsNoOp() {
    bool hasCompleted = false;
    var sub = Obs.Create<int>(
        i => { Assert.Fail("OnNext should not have been called"); },
        exn => { Assert.Fail("OnError should not have been called"); },
        () => {
            Assert.IsFalse(hasCompleted, "OnCompleted for single subscriber called multiple times");
            hasCompleted = true;
        });

    var source = new Observable4<int>();
    using (var unsub = source.Subscribe(sub)) {
        source.Completed();
        source.Completed();
    }
    Assert.IsTrue(hasCompleted);
}

When I knew that Observable4<int>.Completed had no logic to prevent IObserver<T>.OnComplete being called multiple times. Under the debugger it was quite clear that the third lambda was not being called (briefly I considered a debugger bug, but decided to check more fully first).

I looked at the implementation of Observer.Create<T> in Reflector. That factory method creates an instance of an internal type, which derives from another internal type AbstractObserver<T>. So far largely as expected.

But in looking at the implementation of the IObserver<T> methods I saw:

public void OnCompleted() {
    if (!this.IsStopped) {
        this.IsStopped = true;
        this.Completed();
    }
}

I.e. it keeps track of a call to OnCompleted and blocks further events. All three IObserver<T> methods check for IsStopped, and it is also set in OnError.

The Implication

If I want an observer that I can use to test an observable for compliance to the IObserver<T> semantic contract, I cannot use Observer.Create<T>. It will hide breaking the “OnCompleted or OnError end the observable’s event sequence”.

The Solution

In the end the fix is rather easy: make my own Observer.Create<T> which doesn’t contain any such logic. This is rather simple:

public static class Obs {
    public static AnonObserver<T> Create<T>(Action<T> onNext) {
        return new AnonObserver<T>(onNext);
    }
    public static AnonObserver<T> Create<T>(Action<T> onNext, Action<Exception> onError, Action onComplete) {
        return new AnonObserver<T>(onNext, onError, onComplete);
    }
}

public class AnonObserver<T> : IObserver<T> {
    private Action<T> onNext;
    private Action onComplete;
    private Action<Exception> onError;

    public AnonObserver(Action<T> onNext) {
        if (onNext == null) { throw new ArgumentNullException("onNext"); }
        this.onNext = onNext;
    }
    public AnonObserver(Action<T> onNext, Action<Exception> onError, Action onComplete) {
        if (onNext == null) { throw new ArgumentNullException("onNext"); }
        if (onError == null) { throw new ArgumentNullException("onError"); }
        if (onComplete == null) { throw new ArgumentNullException("onComplete"); }
        this.onNext = onNext;
        this.onError = onError;
        this.onComplete = onComplete;
    }

    public void OnCompleted() {
        if (onComplete != null) {
            onComplete();
        }
    }

    public void OnError(Exception error) {
        if (onError != null) {
            onError(error);
        }
    }

    public void OnNext(T value) {
        onNext(value);
    }
}

I now need to just replace all the use of Observer.Create<T> with Obs.Create<T> (and try and think of a better name).

Introduction

In Part 3 I looked at the effect of calling Subject<T>.OnNext from within a subscriber to that Subject<T>. I really do not think that the out of order events that creates are going to create anything other than confusion.

In my implementation, continuing on from Part 1 and Part 2, I want to avoid the implicit re-ordering that happens when Next(value) is called from within a subscriber.

The Re-Entrant Implementation

In the basic implementation a snapshot of the subscribers is taken (to avoid problems with the collection being modified by re-entrant subscribe and unsubscribe operations), then iterated over:

var subs = subscribers.Values.ToArray();
foreach (var sub in subs) {
   sub.OnNext(value);
}

In the case of a re-entrant Next call this loop will simply exist on the stack twice:

Observable2.Next()
ATestObserver.OnNext()
Observable2.Next()

With the second call iterating through the list of subscribers before the first moves on to the next.

The solution here is to recognise that an iteration is taking place already when Next is called and, rather than just looping to call all subscribers, buffer up the new value. The original call keeps looping until all buffered values are sent out. (Once, later in this series, concurrency is introduced locking is going to be needed here.)

First two new fields are needed. A Queue to act as buffer, and a flag to indicate events are currently being pushed to the subscribers:

private Queue<T> pendingEvents = new Queue<T>();
private bool pushingEvent = false;

Then reworking the Next implementation:

pendingEvents.Enqueue(value);
if (!pushingEvent) {
    try {
        pushingEvent = true;
        while (pendingEvents.Count > 0) {
            var subs = subscribers.Values.ToArray();
            var v = pendingEvents.Dequeue();
            foreach (var s in subs) {
                s.OnNext(v);
            }
        }
    } finally {
        pushingEvent = false;
    }
}

A few points worth noting:

• By always putting the new event value in the queue there is no special cases needed for the first event

• The try…finally ensures the pushingEvent gets reset, but this could leave un-pushed events. This is a consequence of the RX exception model (propogate back to the called), but with the added twist that a re-entrant call to this will not get the exception, the original caller of Next will.

• I have chosen to re-snapshot the subscribers for each event. This will avoid a subscriber that unsubscribes in response to one event getting more, even if already queued up.

Next Steps

This is still missing Exception and Complete methods (to call IObserver<T>.Error() and .OnCompleted.

…and concurrency support of course.