Performing Real-time Upgrades to an OTP System

Posted by on September 24, 2008

This is the seventh and final article of the Erlang/OTP introduction series. If you haven’t already, I recommend you read the first article which lays the foundation for the application which we’ll be upgrading in addition to teaching you the basics of Erlang/OTP before jumping into the topic of this article. If you’re a quick learner or you wish to jump straight into this article, you may click here to download a ZIP of all the files up to this point.

The Scenario: ErlyBank has been running strong for a few months now and based on customer feedback, the bank wants to implement some additional features. First, they want us to implement a credit-based account. This is similar to a normal account except that withdrawals may be made to go into the negative, meaning that money is owed on the credit account. They also want us to change the ATM so that people can only use the ATM to pay bills with a credit account. And to top this all off, they want us to do these upgrades without significant downtime.

The Result: We’ll create a credit server to easily add a credit account system and following that we’ll change the ATM. Luckily for us, once we make these changes, there is a straightforward way of upgrading the system in real-time so that ErlyBank won’t experience much, if any, downtime.

Credit Based Account

First things first, we need to implement the credit based account. ErlyBank wants a completely different server to handle this feature so I created eb_credit.erl as another gen_server to handle the creation and logic behind credit accounts. If you wish to challenge yourself I recommend trying to write this yourself first. The only features necessary are creating an account, withdrawing, and depositing (paying bills) since that is all we will use in this article. For fun you can also implement other features such as deleting accounts, sending events to the event manager, etc.

Since this article is about real-time upgrades and not more about gen_server, I’ve created the eb_credit.erl file already, which you can view here.

Changes to ATM

ErlyBank also wanted changes to the ATM so that if a person logs in with a credit account, they are only allowed to deposit money to pay off a negative balance or gain a positive balance. For this, I am going to change the authorize method of eb_server to actually check the credit server too. And eb_server:authorize will return ok and tell the caller what kind of account it is. These are a lot of changes so I’ll walk you through the basics of what I did. Note: There are some very obvious problems with this approach, since it is possible that a credit account and debit account have the same name and PIN, in which case the authorize may return the wrong account. The reason I’m making these “mistakes” is to demonstrate different methods of release upgrades. After this article, if you wish, you can try and fix up the server to be a bit more realistic.

authorize(Name, PIN) ->
  case gen_server:call(?SERVER, {authorize, Name, PIN}) of
    ok ->
      {ok, debit};
    {error, _Reason} ->
      case eb_credit:authorize(Name, PIN) of
        ok ->
          {ok, credit};
        {error, Reason} ->
          {error, Reason}
      end
  end.

 

This method should seem pretty straightforward by now. It first checks to see if there is a debit account with the name and pin, and if so, returns {ok, debit}, otherwise it checks for a credit account.

After changing the authorize method for the server, we need to change the authorization method for the ATM to note the credit account. Remember that ErlyBank wants credit accounts to be able to deposit, but not withdraw money. Using Erlang’s pattern matching, implementing this change is trivial.

unauthorized({authorize, Name, Pin}, _From, State) ->
  case eb_server:authorize(Name, Pin) of
    {ok, debit} ->
      {reply, ok, authorized, Name};
    {ok, credit} ->
      {reply, ok, authorized, {credit, Name}};
    {error, Reason} ->
      {reply, {error, Reason}, unauthorized, State}
  end;

 

First is to change the authorization request for the ATM. Its a simple change to test whether the response is either credit or debit. If the response is debit, we make no changes and the same code as the previous eb_atm is used. If it is a credit account, the internal state data of the ATM is set to {credit, Name}.

Now to restrict withdrawing to only debit accounts, we can just check to make sure the state is a list (a “string” in Erlang) in the deposit method, as you can see here:

authorized({withdraw, Amount}, _From, State) when is_list(State) ->

 

Now when a credit account attempts to withdraw, it will just give an invalid error message since it’ll skip down to the catch-all function. For the authorization methods, this is now done:

authorized({deposit, Amount}, {credit, Account}=State) ->
  eb_credit:deposit(Account, Amount),
  {next_state, thank_you, State, 5000};
authorized({deposit, Amount}, State) ->
  eb_server:deposit(State, Amount),
  {next_state, thank_you, State, 5000};
authorized(_Event, State) ->
  {next_state, authorized, State}.

 

The first function catches credit accounts and deposits its to them, the second will catch debit accounts, and the final catches invalid messages.

That should do it for the changes to the ATM!

Release Handling Instructions

Finally, to the interesting part. In Erlang/OTP an upgrade is given a set of instructions on how to transition from one version to the next. Each instruction is a tuple that is part of a list, where each instruction is executed in order of the list. There are a few instructions described as high-level upgrade instructions, and then there are low-level upgrade instructions. A brief list is given below:

  • {load_module, Module} - Simply reloads a new version of the module Module. If no internal state of a module is changed and simply new code is added or changed, this command is sufficient. An example is if you have a math library and simply fix a bug and add a new method.
  • {update, Module, {advanced, Extra}} - If internal state is changed of a module, simply reloading a new version will not work as it will corrupt the current state for running processes. The update command calls the callback code_change passing the current state and Extra. Following this call, the module is updated to the latest version. The code_change callback function should be used to update the state to the newest format.
  • {add_module/delete_module, Module} - If a new module is introduced, add_module simply loads it into the address space. If a module is removed, delete_module deletes it. Any running processes from delete_module are killed.
  • {apply, {M, F, A}} - A low level instruction which applies the function onto the running system.

There are many more but the above are the most common that I’ve used. You can view the rest here.

ErlyBank Upgrade Instructions

Now, thinking back on the changes we made, what upgrade instructions do we need, and in what order? The following are the changes:

  • New module, eb_credit.erl
  • Changed internal state of eb_server
  • Changed code of eb_atm
  • Added eb_credit to the supervisor

For a fun challenge, you can try to think of the instructions needed to upgrade erlybank now. The following is the order and instructions I will use, accompanied with reasons:

  1. {add_module, eb_credit} - Adding the module won’t do much except load it into the memory space. But the other instructions are dependent on this so I do this first.
  2. {update, eb_sup, supervisor} - Next, updating the supervisor so that eb_credit will be started.
  3. {load_module, eb_server} - eb_server depends on eb_credit running, and now that it is we can load that up!
  4. {update, eb_atm, {advanced, []}} - Finally we update the ATM with an advanced update since the internal state changed. We do this last since it depends on eb_server.

The above instructions must go into an application upgrade file (an “appup” file), which has the following general format:

{NewVsn,
  [{OldVsn1, [Instructions]}],
  [{OldVsn1, [Instructions]}]
}.

 

The first value is the new version, followed by a list of tuples. Each tuple represents an upgrade path for a specific version, with a list of instructions to upgrade from that version. The next list is the same format as the first list but contains instructions for downgrading to that version. Following this format, the application upgrade file for ErlyBank looks like this:

{"2.0",
 [{"1.0", [{add_module, eb_credit},
           {update, eb_sup, supervisor},
           {load_module, eb_server},
           {update, eb_atm, {advanced, []}},
            ]}],
  [{"1.0", [{update, eb_atm, {advanced, []}},
           {load_module, eb_server},
           {update, eb_sup, supervisor},
           {delete_module, eb_credit}]}
 ]
}.

 

As you can see, the instructions are the same except in the opposite order for the upgrade/downgrade paths. And we only need to be able to upgrade/downgrade from version 1 since that is the only other version! This file should be saved as erlybank.appup and should be placed in the ebin/ directory with the app file.

Also, at this point, you should update the rel file to have the new version “2″ and rename it to eb_rel-2.rel.

Release Upgrade File

There also needs to be a release upgrade file, or relup file, to describe how the entire release should be upgraded. Luckily for us, this doesn’t need to be manually created. Start an erlang shell in the root directory of ErlyBank and append ebin/ to the code path, along with the prior version’s ebin path and path to the location of the prior version’s rel file, which you should know how to do by now. If not, the complete command for me is: erl -pz ebin/ -pz ../old/ -pz ../old/ebin. Then invoke the following command:

systools:make_relup("eb_rel-2", ["eb_rel-1"], ["eb_rel-1"]).

 

If it was able to find all the files, it should return with an ok. If not, there should be a descriptive error of what happened. This command should create a “relup” file in the working directory.

Packaging and Installing ErlyBank 2.0

You should also know how to package the release now, it is the same as in the last article. If you’re unsure, go back and reference the previous article. The first steps to installing the release are also the same as the last article, so unpack the release in the releases directory.

After unpacking the release, since we already have a prior release running in memory, we need to install this new release. Installing a release will run the relup file commands. To do this, invoke:

release_handler:install_release("2").

 

Now, although the new release is installed, the “current” code is still version 1. To make version 2 the new default, you must mark it as permanent by using the following command:

release_handler:make_permanent("2").

&nsbp;

And that’s it! ErlyBank has been seamlessly upgraded.

Final Notes

In this article, I introduced real-time upgrades to an OTP system. I covered the most used release instructions and guided you through upgrading the old system. There were some upgrade instructions I forced into this article, however. For example, although the internal state of eb_atm changed, it wasn’t necessary to do the advanced update of the code since we didn’t change the structure of the old state information, just added new state information. If you’d like to learn more about application and release upgrades, I recommend reading the appup cookbook.

With the conclusion of this article, the series is also concluded, almost as soon as it started. I have more posts coming up next week, but hopefully this series has helped more people become more familiar with OTP and its true power.

Ah, one final thing, if you want to download all the final files for this project, you can download the ZIP here.

Trackbacks

Use this link to trackback from your own site.

Comments

Leave a response

  1. [...] Performing Real-time Upgrades to an OTP System (tags: erlang programming spawnlink) [...]

  2. Ricardo Sep 30, 2008 11:24

    I managed to unpack and release the version 2, but get the following error when trying to install the new release:
    4> release_handler:which_releases().
    [{"eb_rel","2",
    ["kernel-2.12","stdlib-1.15","sasl-2.1.5.2","erlybank-2.0"],
    unpacked},
    {”eb_rel”,”1″,
    ["kernel-2.12","stdlib-1.15","sasl-2.1.5.2","erlybank-1.0"],
    unpacked},
    {”OTP APN 181 01″,”R12B”,
    ["kernel-2.12","stdlib-1.15","sasl-2.1.5.2"],
    permanent}]
    5> release_handler:install_release(”2″).
    {error,{enoent,”/opt/local/lib/erlang/releases/R12B/relup”}}

    Why is the system trying to find the relup file in the R12B directory and not in the 2 directory where it really is?

    Thanks.

  3. Tim Oct 01, 2008 09:52

    I had this same problem. I even tried moving the relup file where it would be found, but then it just complained that the relup file was bad.

  4. David Oct 30, 2008 08:00

    I got the same problem too. any1 knows why?

  5. [...] bug fixes or features. Before jumping at the “with a little forethought”, take a read here and see how easy it really is. And it is easy, but even if it was hard when was the last time you [...]

  6. [...] of the neatest aspects of erlang is the OTP release system, which allows you to do real-time upgrades of your application code. But before you can take advantage of it, you need to create a embedded [...]

Comments

Comments: