Managing Application Configuration

Posted by on July 07, 2009

Finally off school and with a lot of free time in the summer to devote to side projects. To warm myself back into writing blog posts I decided to start with a fairly simple but very useful topic: application configuration.

I’ve seen a few small Erlang projects resort to custom configuration syntax, strange XML parsing, etc. for configuration but Erlang has support for easy application configuration built right in! The syntax is simple enough for a non-programmer to grok and is extremely flexible for application developers. As an added bonus, no external libraries are required.

Design Goals for Configuration Files

Before diving into the “how” let’s step back and take a look at the overall design goals of configuration files:

  1. Key/Value Association- Configuration is generally a key associated with a single value. More complicated configurations also have values which consist of more keys and values (such as Apache configuration).
  2. Easy to edit and read for a human - Configuration files are written by humans, its important for human beings to be able to easily read and edit the files. Since configuration files are typically only parsed once and usually only while booting up, the speed for parsing for the computer is not as important.
  3. Simple parsing implementation - When working on a project, we don’t want to spend 8 hours of a developer’s time simply writing the configuration parsing implementation.
  4. Simple access API - Similar to above, we don’t want an overly complex structure which makes it hard to access configuration values from the application code.

Writing Configuration the Erlang Way

The built-in types of Erlang are perfect for configuration: lists (strings included), atoms, numbers, tuples. A key can be an atom and a value can be any other data type, right? So using just Erlang syntax, a configuration file could look something like this:

{host, "foo.com"}.
{port, 1234}.
{database, [{type, mysql},
            {user, root},
            {pass, root},
            {host, localhost}]}.

I’m sure you noticed that I’ve intermixed some atoms with some strings. I did this on purpose to demonstrate some things later!

Looking at the above configuration, we can put ourselves in multiple shoes and ask:

  1. As a user, is this easy to read? Yes, I think so, since its just {key, value}.
  2. As a user, is this easy to edit? Sure, just change the value. If anything, there can be examples up top on how to edit.
  3. As a developer, does this provide me with enough flexibility? Yes. You can have simple KV (Key/Value) configurations or sub configurations (as with the database key). Values can be integers or strings.

Let’s assume I saved this configuration file as “application.cfg”

Parsing Configuration the Erlang way

This will be a short section. To parse it:

file:consult("application.cfg").

Really! That’s it. That will return:

{ok,[{host,"foo.com"},
     {port,1234},
     {database,[{type,mysql},
                {user,root},
                {pass,root},
                {host,localhost}]}]}

Basically, file:consult/1 parses any file and returns a list of all the Erlang terms within it. The consult method does not run the code; function calls and function declarations will cause syntax errors. Terms are simply read. This is great for configuration!

Note: If there is a syntax error within the configuration file, then the consult method will return a helpful error message with the line the error occurred along with a simple programmer-friendly description. For example, if I left out a period on a line, I get {error,{5,erl_parse,["syntax error before: ","'{'"]}}. This tells me the error was on line 5 along with the error.

Of course the error message it returns is made for programmers and not general users so if you plan on writing a general purpose configuration parser, it may be a good idea to write a method that converts that to something more human-readable. :)

Extracting Configuration Values

As I’m sure most of you reading this have at least some Erlang experience, I’m sure you can already see that extracting configuration values is just a matter of pattern matching the term list returned. The easiest way to do this is for your app to have a simple configuration module which handles parsing it out. The following is a simple get method to read out values:

get(_Key, []) ->
  {error, not_found};
get(Key, [{Key, Value} | _Config]) ->
  {ok, Value};
get(Key, [{_Other, _Value} | Config]) ->
  get(Key, Config).

The above is a simple functional and tail recursive approach to getting a value from a key. Some examples below with their return values:

1> {ok, Cfg} = config:read("application.cfg").
2> config:get(host, Cfg).
{ok, "foo.com"}
3> {ok, SubCfg} = config:get(database, Cfg).
{ok,[{type,mysql},{user,root},{pass,root},{host,localhost}]}
4> config:get(type, SubCfg).
{ok, mysql}

From the examples, you can see that its easy to read values and just as easy to read sub-values from more complicated configurations such as for the database.

The above example is only an extremely simple case where you want to get a value for a key. Some configurations require more complicated traversals and so on, but this is just as easy with pattern matching that Erlang gives us.

Advanced Cases

Some configuration files require more advanced features such as including other config files. For example, a web server may include a single configuration which includes separate configuration files for each virtual host on the server (the way Apache on Ubuntu works out of the box).

Including other configuration files isn’t too much more difficult. Take the following configuration file:

{host, "foo.com"}.
{port, 1234}.
{include_config, "database.cfg"}.

I’ve taken the first configuration we used and pulled out the database configuration into a separate file and threw a include statement into the original configuration. If you’ve read up to this point and you think that this is it and it works, I want to tell you now: This doesn’t work on its own! The include logic has to be implemented manually.

Here is the updated configuration logic:

read(FileName) ->
  {ok, Terms} = file:consult(FileName),
  read_includes(Terms).

read_includes(Terms) ->
  read_includes(Terms, []).

read_includes([{include_config, File} | Terms], Acc) ->
  case file:consult(File) of
    {ok, NewTerms} ->
      read_includes(Terms ++ NewTerms, Acc);
    {error,enoent} ->
      {error, {bad_include, File}};
    Other ->
      {error, Other}
  end;
read_includes([Other | Terms], Acc) ->
  read_includes(Terms, [Other | Acc]);
read_includes([], Result) ->
  {ok, Result}.

The above looks scary but really, take the time to read through it because its not. There are about 10 more lines of code there because I did some defensive programming, handling some special error cases. Typically in Erlang this is a bad idea since errors are so readable but since we want errors to be more readable for humans, I made a special case.

The config:read/1 was modified to call config:read_includes/1 which begins reading the includes. config:read_includes/2 works by keeping an accumulator of configuration values, traversing the current values, and replacing include_config configurations with their respective files.

For those who don’t want to run this on their own, here is the result of reading the new application.cfg:

1> config:read("application.cfg").
{ok,[{database,[{type,mysql},
                {user,root},
                {pass,root},
                {host,localhost}]},
     {port,1234},
     {host,"foo.com"}]}

The ordering of the configuration changed a little (its in reverse, can you figure out why? ;) ), but this shouldn’t matter.

Conclusion

It feels great to be writing a spawn_link article again! I apologize if this topic was too “beginner” for the readers but I feel like configuration is a very important part of an application and wanted to share how its generally approached.

And don’t worry, I already have finished writing a few more articles. They’re drafted and in the pipeline for publishing in the next couple weeks. :)

Trackbacks

Use this link to trackback from your own site.

Comments

Leave a response

  1. James Lee Jul 07, 2009 20:26

    Thanks for this. I’m working on my first little project in Erlang and was looking for some tips on how to store configuration. I’ve been spoiled by http://java.sun.com/j2se/1.4.2/docs/guide/lang/preferences.html, but I think I can whip up a few more functions to add to your examples to get something comparable.

  2. Harish Mallipeddi Jul 07, 2009 22:26

    Good to see you writing again! I learnt a lot from your series of blog posts on OTP.

  3. [...] Managing Application Configuration (tags: erlang programming) Leave a Comment [...]

  4. Anselmo Silva Jul 08, 2009 02:20

    Tks, great stuff as usual.

  5. [...] Managing Application Configuration (Mitchell Hashimoto) [...]

  6. Brendon Murphy Jul 08, 2009 10:14

    Thanks for sharing this. Having spent most my time recently in Rails (or even just plain Ruby) I’m addicted to being able to use YAML for reading a config. As I’m new to Erlang, this’ll help fill the gap I’m sure.

  7. Ferd T-H Jul 08, 2009 16:18

    To read form a proplist ([{key,Value}]), see the proplist module with the functions get_value/2 and get_value/3. They’re going to do what you want without you needing to write anything.

    1> Conf = [{appname, "Hello"}, {user,"Ferd"}].
    [{appname,"Hello"},{user,"Ferd"}]
    2> proplists:get_value(user,Conf).
    “Ferd”
    3> proplists:get_value(password, Conf, “Default if not found”).
    “Default if not found”

    Check around the module for some other helpful functions to deal with proplists rather than writing them yourself.

  8. Mitchell Jul 08, 2009 18:39

    Ferd T-H,

    Awesome! After I read about it more and find some time I’ll update the post to reflect these findings. Thank you very much.

  9. grantmichaels Jul 09, 2009 14:13

    It’s awesome to see that you are planning to start blogging again!

  10. Witold Baryluk Aug 12, 2009 06:03

    It is very cool thing.

    But sometimes file:consult/1 isn’t enaugh, for example if you want to use, expression (like multiply something, or append list to another, or use list comprehension).

    This snipet will do this:

    eval(Filename) ->
    {ok, B} = file:read_file(Filename),
    T = binary_to_list(B),
    {ok, Tokens, _EndLocation} = erl_scan:string(T),
    {ok, [Expression]} = erl_parse:parse_exprs(Tokens),
    Bindings = erl_eval:new_bindings(),
    {value, Value, _NewBindings} = erl_eval:expr(Expression, Bindings),
    {ok, Value}.

    Use like eval(”file.cfg”);

    example file.cfg:
    {hosts,
    [ {active, Host, 100} || X <- [a1,a2,a3,a4,a5] ] ++
    [ {passive, Host, 200} || X <- [p1, p2, p3, p4] ]
    }.
    {acl, [ {joe, md5("password') } ]}.

    this is just example. You need to consider that users can write any expressions there (like erlang:halt() ).

Comments

Comments: