In the past I have unit tested my Erlang source files by manually creating a test method and running it via a bash script. This works, but it is a bit hard to manage and you don’t get a lot of things for free; I just used pattern matching for the unit tests. Now that I’m starting a new project, I decided to start right with a proper unit testing framework. For those of you not familiar with the practice of “unit testing,” this article is not for you and you should google around, there are thousands of resources available.

Getting EUnit

For some reason I assumed Erlang would come shipped with EUnit, and spent a good 10 minutes “debugging” why my unit tests weren’t available! Of course I was wrong, I had to go grab EUnit myself. You can grab it by checking it out of the EUnit trunk via SubVersion. I know there is a ZIP file also available on process-one’s website but the code is pretty outdated so I recommend checking out the trunk.

Go into the download eunit/ directory and type make to compile all the Erlang source files. After doing this, make sure to copy your eunit/ directory to the Erlang lib directory, which can be found by running code:lib_dir().

Basics of EUnit

There are quite a few resources for EUnit which I’ll link at the end of this article. So as not to totally reinvent the wheel I’m only going to cover a few topics here and will trust that you have the drive to read the other articles which are fantastic for getting off the ground.

EUnit is minimally invasive to your code. By this I mean the only real change you need to make to your source files to get it up and running is to include a header file. Here is an example of the preprocessor command I use:

-include_lib("eunit/include/eunit.hrl").

This header file does do a small amount of magic. That is, it automatically exports a test/0 method from the module if it doesn’t exist already. This test method automatically calls any methods ending in _test or _test_ (a subtle but important difference).

The *_test methods are expected to return anything on success, or throw an exception on failure. Erlang actually has a great testing construct baked into the language: pattern matching. This is what 90% of your tests will use. The following is an example test:

reverse_test() ->
  [3, 2, 1] = lists:reverse([1, 2, 3]).

This test will pass. If instead I put [3, 1, 2] it would have raised a badmatch exception and the test would have failed. Simple, isn’t it?

Getting More Advanced

This is just the surface of EUnit, but luckily, the rest of it isn’t much harder to understand. Unfortunately, EUnit doesn’t have mocks/stubs, which many Ruby programmers may miss (as we are all spoiled with our RSpec). I’m not going to cover any more of EUnit in this article since there are a handful of articles who do this already, and I don’t want to steal their thunder.

But one last thing I really liked…

Hiding Your Tests In Production Systems

Its great that EUnit automagically exports your test methods and runs them and all that but after the tests pass and the green icon shows, nobody really wants these in a running system. Unsurprisingly, EUnit developers already thought of this and offer an easy way out: Simply define NOTEST before you include the EUnit header file. If this is defined, nothing will happen.

I put NOTEST in all my files right away. Don’t worry, to run the tests you don’t have to manually remove this line. When you compile your Erlang source files for testing, add the -DTEST flag to the compiler, which will define the TEST macro. This overrides NOTEST. An example erlc command:

erlc -DTEST module.erl

The Resources

The following is a list of valuable resources I found picking up Erlang. All gain my stamp of approval (if that means anything to you).

• The Official EUnit Summar Page
• Paul Nelson’s EUnit-Based Erlang Tutorials
• Kevin Smith’s Screencast on EUnit - Highly recommend if you’re willing to spend a small amount of money. Its worth it.

As a final note or disclaimer: I haven’t used EUnit a great amount yet in a professional environment so this is a very high-level overview of EUnit. I’m sure as time goes on and this new project goes on, I will have more to say.

The Scenario: ErlyBank has been running strong for a few months now and based on customer feedback, the bank wants to implement some additional features. First, they want us to implement a credit-based account. This is similar to a normal account except that withdrawals may be made to go into the negative, meaning that money is owed on the credit account. They also want us to change the ATM so that people can only use the ATM to pay bills with a credit account. And to top this all off, they want us to do these upgrades without significant downtime.

The Result: We’ll create a credit server to easily add a credit account system and following that we’ll change the ATM. Luckily for us, once we make these changes, there is a straightforward way of upgrading the system in real-time so that ErlyBank won’t experience much, if any, downtime.

Credit Based Account

First things first, we need to implement the credit based account. ErlyBank wants a completely different server to handle this feature so I created eb_credit.erl as another gen_server to handle the creation and logic behind credit accounts. If you wish to challenge yourself I recommend trying to write this yourself first. The only features necessary are creating an account, withdrawing, and depositing (paying bills) since that is all we will use in this article. For fun you can also implement other features such as deleting accounts, sending events to the event manager, etc.

Since this article is about real-time upgrades and not more about gen_server, I’ve created the eb_credit.erl file already, which you can view here.

Changes to ATM

ErlyBank also wanted changes to the ATM so that if a person logs in with a credit account, they are only allowed to deposit money to pay off a negative balance or gain a positive balance. For this, I am going to change the authorize method of eb_server to actually check the credit server too. And eb_server:authorize will return ok and tell the caller what kind of account it is. These are a lot of changes so I’ll walk you through the basics of what I did. Note: There are some very obvious problems with this approach, since it is possible that a credit account and debit account have the same name and PIN, in which case the authorize may return the wrong account. The reason I’m making these “mistakes” is to demonstrate different methods of release upgrades. After this article, if you wish, you can try and fix up the server to be a bit more realistic.

authorize(Name, PIN) ->
  case gen_server:call(?SERVER, {authorize, Name, PIN}) of
    ok ->
      {ok, debit};
    {error, _Reason} ->
      case eb_credit:authorize(Name, PIN) of
        ok ->
          {ok, credit};
        {error, Reason} ->
          {error, Reason}
      end
  end.

This method should seem pretty straightforward by now. It first checks to see if there is a debit account with the name and pin, and if so, returns {ok, debit}, otherwise it checks for a credit account.

After changing the authorize method for the server, we need to change the authorization method for the ATM to note the credit account. Remember that ErlyBank wants credit accounts to be able to deposit, but not withdraw money. Using Erlang’s pattern matching, implementing this change is trivial.

unauthorized({authorize, Name, Pin}, _From, State) ->
  case eb_server:authorize(Name, Pin) of
    {ok, debit} ->
      {reply, ok, authorized, Name};
    {ok, credit} ->
      {reply, ok, authorized, {credit, Name}};
    {error, Reason} ->
      {reply, {error, Reason}, unauthorized, State}
  end;

First is to change the authorization request for the ATM. Its a simple change to test whether the response is either credit or debit. If the response is debit, we make no changes and the same code as the previous eb_atm is used. If it is a credit account, the internal state data of the ATM is set to {credit, Name}.

Now to restrict withdrawing to only debit accounts, we can just check to make sure the state is a list (a “string” in Erlang) in the deposit method, as you can see here:

authorized({withdraw, Amount}, _From, State) when is_list(State) ->

Now when a credit account attempts to withdraw, it will just give an invalid error message since it’ll skip down to the catch-all function. For the authorization methods, this is now done:

authorized({deposit, Amount}, {credit, Account}=State) ->
  eb_credit:deposit(Account, Amount),
  {next_state, thank_you, State, 5000};
authorized({deposit, Amount}, State) ->
  eb_server:deposit(State, Amount),
  {next_state, thank_you, State, 5000};
authorized(_Event, State) ->
  {next_state, authorized, State}.

The first function catches credit accounts and deposits its to them, the second will catch debit accounts, and the final catches invalid messages.

That should do it for the changes to the ATM!

Release Handling Instructions

Finally, to the interesting part. In Erlang/OTP an upgrade is given a set of instructions on how to transition from one version to the next. Each instruction is a tuple that is part of a list, where each instruction is executed in order of the list. There are a few instructions described as high-level upgrade instructions, and then there are low-level upgrade instructions. A brief list is given below:

• {load_module, Module} - Simply reloads a new version of the module Module. If no internal state of a module is changed and simply new code is added or changed, this command is sufficient. An example is if you have a math library and simply fix a bug and add a new method.
• {update, Module, {advanced, Extra}} - If internal state is changed of a module, simply reloading a new version will not work as it will corrupt the current state for running processes. The update command calls the callback code_change passing the current state and Extra. Following this call, the module is updated to the latest version. The code_change callback function should be used to update the state to the newest format.
• {add_module/delete_module, Module} - If a new module is introduced, add_module simply loads it into the address space. If a module is removed, delete_module deletes it. Any running processes from delete_module are killed.
• {apply, {M, F, A}} - A low level instruction which applies the function onto the running system.

There are many more but the above are the most common that I’ve used. You can view the rest here.

ErlyBank Upgrade Instructions

Now, thinking back on the changes we made, what upgrade instructions do we need, and in what order? The following are the changes:

• New module, eb_credit.erl
• Changed internal state of eb_server
• Changed code of eb_atm
• Added eb_credit to the supervisor

For a fun challenge, you can try to think of the instructions needed to upgrade erlybank now. The following is the order and instructions I will use, accompanied with reasons:

1. {add_module, eb_credit} - Adding the module won’t do much except load it into the memory space. But the other instructions are dependent on this so I do this first.
2. {update, eb_sup, supervisor} - Next, updating the supervisor so that eb_credit will be started.
3. {load_module, eb_server} - eb_server depends on eb_credit running, and now that it is we can load that up!
4. {update, eb_atm, {advanced, []}} - Finally we update the ATM with an advanced update since the internal state changed. We do this last since it depends on eb_server.

The above instructions must go into an application upgrade file (an “appup” file), which has the following general format:

{NewVsn,
  [{OldVsn1, [Instructions]}],
  [{OldVsn1, [Instructions]}]
}.

The first value is the new version, followed by a list of tuples. Each tuple represents an upgrade path for a specific version, with a list of instructions to upgrade from that version. The next list is the same format as the first list but contains instructions for downgrading to that version. Following this format, the application upgrade file for ErlyBank looks like this:

{"2.0",
 [{"1.0", [{add_module, eb_credit},
           {update, eb_sup, supervisor},
           {load_module, eb_server},
           {update, eb_atm, {advanced, []}},
            ]}],
  [{"1.0", [{update, eb_atm, {advanced, []}},
           {load_module, eb_server},
           {update, eb_sup, supervisor},
           {delete_module, eb_credit}]}
 ]
}.

As you can see, the instructions are the same except in the opposite order for the upgrade/downgrade paths. And we only need to be able to upgrade/downgrade from version 1 since that is the only other version! This file should be saved as erlybank.appup and should be placed in the ebin/ directory with the app file.

Also, at this point, you should update the rel file to have the new version “2″ and rename it to eb_rel-2.rel.

Release Upgrade File

There also needs to be a release upgrade file, or relup file, to describe how the entire release should be upgraded. Luckily for us, this doesn’t need to be manually created. Start an erlang shell in the root directory of ErlyBank and append ebin/ to the code path, along with the prior version’s ebin path and path to the location of the prior version’s rel file, which you should know how to do by now. If not, the complete command for me is: erl -pz ebin/ -pz ../old/ -pz ../old/ebin. Then invoke the following command:

systools:make_relup("eb_rel-2", ["eb_rel-1"], ["eb_rel-1"]).

If it was able to find all the files, it should return with an ok. If not, there should be a descriptive error of what happened. This command should create a “relup” file in the working directory.

Packaging and Installing ErlyBank 2.0

You should also know how to package the release now, it is the same as in the last article. If you’re unsure, go back and reference the previous article. The first steps to installing the release are also the same as the last article, so unpack the release in the releases directory.

After unpacking the release, since we already have a prior release running in memory, we need to install this new release. Installing a release will run the relup file commands. To do this, invoke:

release_handler:install_release("2").

Now, although the new release is installed, the “current” code is still version 1. To make version 2 the new default, you must mark it as permanent by using the following command:

release_handler:make_permanent("2").

&nsbp;

And that’s it! ErlyBank has been seamlessly upgraded.

Final Notes

In this article, I introduced real-time upgrades to an OTP system. I covered the most used release instructions and guided you through upgrading the old system. There were some upgrade instructions I forced into this article, however. For example, although the internal state of eb_atm changed, it wasn’t necessary to do the advanced update of the code since we didn’t change the structure of the old state information, just added new state information. If you’d like to learn more about application and release upgrades, I recommend reading the appup cookbook.

With the conclusion of this article, the series is also concluded, almost as soon as it started. I have more posts coming up next week, but hopefully this series has helped more people become more familiar with OTP and its true power.

Ah, one final thing, if you want to download all the final files for this project, you can download the ZIP here.

Sorry about postponing the final post of my Erlang/OTP series. Its in about a half-written, unedited state right now and I mean to finish it but this week I’m actually moving in to my apartment back at school at the University of Washington! I finally got internet hooked up today and am just making final touches.

I have great things planned however and the feedback I’ve received on this blog has been great.

Thanks!

This article will introduce release management of an Erlang application. This is not necessarily restricted to Erlang/OTP applications, and is a general tool that may be used for all your Erlang applications.

Release Management refers to consolidating an application and its dependencies into a single package which can be easily started and stopped as a whole. Releases can be “installed” into an Erlang runtime system. Releases can also be upgraded in real-time, so when a new version of the application is created, it can be seamlessly installed into the system without bringing it down. (Note that real-time system upgrades are covered in the next article, not this one!)

After bundling the ErlyBank system as an application, restructuring it to be an Erlang release is very easy. But first, we’ll need to do some housecleaning to the directory structure.

Release Directory Structure

In general, all application releases follow a certain directory structure. This structure is not required but it is a common practice which increases in readability of application files:

Application-Vsn/ebin
               /include
               /priv
               /src

The src directory contains all the *.erl files, the sources for an application. The ebin directory contains all the compiled *.beam files for the sources and the application specification file. The include directory contains all the *.hrl files which may be included by the sources. And the priv directory contains any executables that are used by the application, such as port applications and Erlang drivers.

All these directories are under a parent directory named Application-Vsn where Application is the name of the application and Vsn is the version.

Reorganizing ErlyBank to follow this structure is quite easy and I will do this before moving on. Simply move all the *.erl files to the src/ directory, and move erlybank.app to the ebin/ directory.

Another important thing to remember is that when compiling Erlang sources, the resulting beam file is usually placed in the same directory as the erl file. To make it so that our sources are compiled into the ebin/ directory, compilation should be done with the following command, which you may easily turn into a make command or whatever you want:

erlc -o ../ebin eb_app.erl eb_sup.erl eb_event_manager.erl eb_atm.erl eb_withdrawal_handler.erl eb_server.erl

The above command should be run from the src/ directory and will place all beam files in the corresponding ebin/ directory.

Release Specification File

The one thing you need to create a release is a release specification file. This file is used by the systools module to figure out how to package the Erlang application. This file format is documented very well at the release resource documentation, which I recommend you glance over now. The release specification I’ve created for the ErlyBank release is:

{release, {"eb_rel", "1"},
          {erts, "5.6.3"},
          [{kernel, "2.12.3"},
           {stdlib, "1.15.3"},
           {sasl, "2.1.5.3"},
           {erlybank, "1.0"}]
}.

The contents of the file are saved as eb_rel-1.rel. eb_rel is the name of the release with “1″ being the version number. The tuple {erts, “5.6.3″} specifies the version of the Erlang runtime system that the release is meant for. This can be obtained by opening the Erlang shell and seeing what version you have. Following these tuples, there is a list of more tuples which specify the applications for this release. By convention they are listed in order of dependency but systools is smart enough to figure this out on its own when you create the boot script later.

The versions of dependencies can be retrieved by using the application:loaded_applications/0 method.

One thing you may notice that was added was the dependency on sasl, which stands for “System Architecture Support Libraries.” The sasl application is required for the release handler module which is used to install releases into a system.

Creating a Boot Script

The purpose of the release specification file is create a boot script that we will use to boot up the entire release quickly and easily. To create the boot script, run the following commands:

Chip ~/.../6_release_management: erl -pa ./ebin
Erlang (BEAM) emulator version 5.6.3 [source] [async-threads:0] [kernel-poll:false]                                     

Eshell V5.6.3  (abort with ^G)
1> systools:make_script("eb_rel-1", [local]).
ok

The important thing is that the Erlang shell is started by appending ebin to the code path. This is what the pa flag does. This is so that the systools module can find the erlybank.app file.

Once the shell is open, the systools:make_script/2 method is used to create the boot script. The make_script method accepts as its first argument the name of the release spec file without the extension, and it is expected to be in the current working directory. The second argument is a list of options. The option specified here, local, uses the complete path of where the dependency applications are found instead of using variables such as $ROOT. This is useful for testing in a development environment. When you’re ready to package it for true release, omit this option.

If the result of calling the function is ok, then you should find a eb_rel-1.boot file in the current working directory. This file is used to start the Erlang system.

Starting the System with the Boot File

Remember how easy it was to start the ErlyBank system using application:start/1? Well now its even easier by typing the following into the shell:

erl -boot eb_rel-1

The Erlang system will locate the boot file specified and will follow it to startup the system. You can test that it worked by using the ErlyBank server to create accounts, make deposits, etc.

To shut down the system, use the normal q() command in the shell. All processes will be gracefully stopped.

Packaging the Release

The final step is to make the system a nice single package that can be carried around different systems to ease in setup and installation. This can all be done in a single step now that we already have the correct directory structure and a compiled boot script.

4> systools:make_tar("eb_rel-1").
ok

In the working directory, there should now be an eb_rel-1.tar.gz. That’s all there is to it!

Installing a Packaged Release

To install the packaged release, we first need to copy the tar.gz package we created earlier into the $ROOT/releases directory. $ROOT can be determined by running code:root_dir().. Next, start up and Erlang shell and make sure the sasl application is started so we can use the release_handler module. Also, make sure the shell is running with valid permissions to make changes to the $ROOT directory. This may require running erl with sudo.

3> release_handler:unpack_release("eb_rel-1").
{ok,"1"} 

The unpack_release method does just what it says. It unpacks the package file, copying the libs to the code:lib_dir() and created a proper release directory. Since this is the initial release, there is no need to “install” it, as its already done.

Now restart the Erlang shell, start sasl, and you should now be able to start erlybank with application:start(erlybank) anywhere! The system has been installed!

Alternatively you can start the ErlyBank application with a single command:

erl -boot $ROOT/releases/1/start

$ROOT must be replaced with the root directory of the Erlang install!

Final Notes

In this article I introduced creating a basic release, covering topics such as creating boot scripts, packaging a release, and installing a release. The real power of releases, however, comes at the ability to perform system upgrades to a running system, and this will be covered in the next, and final, article in the series.

If you wish to download all the files as of the end of this post, I have zipped them up here.

The Scenario: The ErlyBank system is in a nice state right now with everything wrapped up under the supervisor, but ErlyBank is pressuring us to come up with a package which specifies an overall version so the system can be upgraded as a whole and can be controlled easily. They’ve commissioned us to do this task.

The Result: Erlang/OTP has a built-in module for making a set of modules which achieve some set of functionality into an application. An application can be started and stopped as a unit and can easily be reused in other systems.

What is an application?

An application consists of a typical OTP-style callback module and an application resource file which tells the Erlang system how to handle the application. In every Erlang runtime there is an application controller process, which stores the information for every loaded application resource and manages the processes if they have started. Note that the application controller manager is not responsible and will not restart your applications if they terminate. It merely stores process information to know whether or not the application is running.

There are only two callbacks for the application callback module:

• start/2 - Called when an application is starting to initialize the processes that are part of that application. In a proper Erlang/OTP application, this will usually start a supervisor process which manages the other processes.
• stop/1 - Called when an application is stopping so that the processes associated with an application can stop. This usually involves stopping the overall supervisor process for the application.

Application Callback Module

A skeleton for an application module can be viewed here. As you can see there are only two methods: the behavior methods.

So save that skeleton as eb_app.erl and let’s get it working with our ErlyBank system!

The Application Start Callback

Based on the skeleton, you’ll probably be able to figure out how to do this on your own, but if you need some guidance, then continue reading this paragraph. Here is the start method for eb_app:

start(_Type, _StartArgs) ->
  case eb_sup:start_link() of
    {ok, Pid} -> 
      {ok, Pid};
    Error ->
      Error
  end.

To start the application, all we have to do is start the supervisor. The application behavior requires us to return {ok, Pid} on success so it can verify that this application is running. Type is used to tell us why we’re starting, and is usually just “normal.” If we write a distributed application, then it can also signify that this application is starting due to failover and takeover, but that won’t be covered in this introductory article.

Also, start arguments may be specified in the application resource file, but we won’t be using them for ErlyBank so we mark the variable as unused.

The Application Stop Callback

The stop callback is even easier than the start!

stop(_State) ->
  exit(whereis(eb_sup), shutdown).

In the article on supervisors , I made a subtle hint to exit with kill to test that the supervisors worked. The kill message is actually a brutal kill and doesn’t call any graceful termination methods. In the stop method of the applicaton, we use the shutdown message which stops the supervisor gracefully.

There is one last piece to the application before we can actually run it, and that is to create the application resource file. But eb_app.erl is completed and can be viewed here.

Application Resource File Syntax

The application resource file is suffixed with .app but is actually just a file containing some erlang terms. Copied directly from the application documentation:

{application, Application,
  [{description,  Description},
   {id,           Id},
   {vsn,          Vsn},
   {modules,      Modules},
   {maxP,         MaxP},
   {maxT,         MaxT},
   {registered,   Names},
   {included_applications, Apps},
   {applications, Apps},
   {env,          Env},
   {mod,          Start},
   {start_phases, Phases}]}.

             Value                Default
             -----                -------
Application  atom()               -
Description  string()             ""
Id           string()             ""
Vsn          string()             ""
Modules      [Module]             []
MaxP         int()                infinity
MaxT         int()                infinity
Names        [Name]               []
Apps         [App]                []
Env          [{Par,Val}]          []
Start        {Module,StartArgs}   undefined
Phases       [{Phase,PhaseArgs}]  undefined
  Module = Name = App = Par = Phase = atom()
  Val = StartArgs = PhaseArgs = term()

As you can see, the application file contains an Erlang tuple which has various information about the application which the Erlang runtime will use to know how to handle the application. All the options, the tuples contained within the list, are optional, and if any are omitted, their respective default value will be used, which are listed above, also.

First, Application is an atom representing the name of the application. This atom must be the same as the filename of the application resource file. So if they resource file is named “erlybank.app,” then the application name is “erlybank.”

Those are a lot of options, but the ones that are important are the options that systools requires, because we will be using that module in the future to handle upgrading and packaging our system! systools requires description, vsn, modules, registered, and applications.

Description is a one line description of the application. That’s it!

vsn is the version number represented as a string. It can be anything you want! There is no standard format.

modules are all modules introduced by this application. These are the modules you wrote specifically for this application, ignoring any modules in any other applications.

registered is a list of names of all registered processes. systools will use this in the future to automatically detect any clashing process names in a system.

applications is a list of applications which must be started before this application. Include the application name of any external modules you use here so systools can create proper boot scripts in the future for the application.

ErlyBank’s Application Resource File

The following is the contents of the application resource file I made for ErlyBank:

{application, erlybank,
  [{description, "ErlyBank system."},
  {vsn, "1.0"},
  {modules, [eb_app, eb_sup, eb_server, eb_atm, eb_event_manager, eb_withdrawal_handler]},
  {registered, [eb_sup, eb_server, eb_atm, eb_event_manager]},
  {applications, [kernel, stdlib]},
  {mod, {eb_app, []}}
]}.

All the systools required options are included in addition to mod, which specifies the module that is used to start the application.

As application dependencies I explicitly list kernel and stdlib, but even if I left that list blank, they would be started anyways, as they’re required by the Erlang system. I just like to make everything as explicit as possible so when I come back to look at code later, I know exactly what is happening.

Save the above code as erlybank.app and we’re ready to test run it!

Loading and Running the Application From the Shell

First, you can use the loaded_applications method to check what applications are already loaded into the Erlang system. This is the output from my shell:

14> application:loaded_applications().
[{kernel,"ERTS  CXC 138 10","2.12.3"},
 {stdlib,"ERTS  CXC 138 10","1.15.3"}]

As you can see, it’s not loaded yet. But actually, if you run the start method and application sees that that application spec is not loaded yet, it will attempt to load it itself. But to show you how this works, I will run it explicitly:

1> application:load(erlybank).
ok
2> application:loaded_applications().
[{kernel,"ERTS  CXC 138 10","2.12.3"},
 {erlybank,"ErlyBank system.","1.0"},
 {stdlib,"ERTS  CXC 138 10","1.15.3"}]

The load method finds the app file locally and loads it. It does not load any of the actual Erlang code for ErlyBank, yet. After loading the spec, I ran loaded_applications again and you can easily see that the ErlyBank application is listed.

To start the ErlyBank system, just run application:start(erlybank) and to stop it, run the stop method with erlybank as the parameter. Play around with the bank system just to make sure it’s working.

Final Notes

In this article I talked about application. I explained what an application is, the files that are required for an application, how to load an application, and how to start and stop an application. I breezed over the options required by systools, which we will use in a future article.

And that is the end of article five in the Erlang/OTP introduction series. The sixth article will be published in an another few days and will cover release management, including topics such as creating boot scripts, packaging applications, and more.

The Scenario: The thing that makes us feel good about banks and ATMs is that they are always there. We can deposit and get money whenever want, 24 hours a day, using an ATM. Or we can go into any bank branch when they’re open, and know we have complete access to our funds. To achieve this security, we need to make sure that our system to run ErlyBank always stays working: the processes must always be running. ErlyBank has commissioned us to achieve this goal. 100% uptime! (Or as close to that as we can get)

The Result: Using an OTP supervisor, we will create a process whose responsibility it is to watch the running processes and make sure they stay up.

What is a Supervisor?

A supervisor is a process which monitors what are called child processes. If a child process goes down, it uses that child’s restart strategy to restart the process. This system can keep Erlang systems running forever.

The supervisor is part of what is called a supervision tree. A well written Erlang/OTP application starts with a root supervisor, which watches over child supervisors, which in turn watch over more supervisors or processes. The idea is that if a supervisor goes down, the parent supervisor will restart it, all the way up to the root supervisor. The Erlang runtime has a heart option which will watch the entire system and restart it if the root supervisor were to die. This way, the supervision tree will always be intact.

There is only one callback for a supervisor: init/1. Its role is to return a list of child processes and restart strategies for each process, so the supervisor knows what to watch and what actions to take if something goes wrong.

Decoupling eb_server and the Event Manager

One of the things I did in the last article on gen_event was explicitly start the event manager process in the init method of eb_server. I did this at the time because it was the only option I really had if I wanted to easily start the server with that dependency. But now that we’re going to be implementing startup and stop using a supervisor, we can start the event manager within the supervisor tree. So let’s take out the eb_event_manager startup from the server.

To do this, simply remove line 84 from eb_server.erl, which is the startup for the event manager. Additionally, I added at this location the add_handler call to add eb_withdrawal_handler to the event manager. So the init method of eb_server now looks like this:

init([]) ->
  eb_event_manager:add_handler(eb_withdrawal_handler),
  {ok, dict:new()}.

Click here to view eb_server.erl after this change.

The Supervisor Skeleton

A basic skeleton for writing a supervisor can be viewed here. As you can see, it has a start method and has a basic init method, which is returning a restart strategy and a fake child spec for now. Restart strategies and child specifications are covered in the next section of this article.

Save the skeleton as eb_sup.erl. The naming of this file is another convention. The supervisor for a certain group is always suffixed with “_sup.” Its not mandatory but its standard practice.

Restart Strategies

A supervisor has one restart strategy, which it uses in conjunction with the child specification, to determine what it should in case one of the supervisor’s child processes dies. The following are the possible restart strategies:

• one_for_one - When one of the child processes dies, the supervisor restarts it. Other child processes aren’t affected.
• one_for_all - When one of the child processes dies, all the other child processes are terminated, and then all restarted.
• rest_for_one - When one of the child processes dies, the “rest” of the child processes defined after it in the child specification list are terminated, then all restarted.

When specifying the restart strategy, it takes the following format:

{RestartStrategy, MaxRetries, MaxTime}

This is very simple to understand after you wrap your mind around the english: If a child process is restarted more than MaxRetries times in MaxTime seconds, then the supervisor terminates all child processes and then itself. This is to avoid an infinite loop of restarting a child process.

Child Specification Syntax and Concepts

The init callback for the supervisor is responsible for returning a list of child specifications. These specs tell the supervisor which processes to start and how to start them. The supervisor starts the processes in order from left to right (beginning of the list to the end of the list). A restart strategy is a tuple with the following format:

{Id, StartFunc, Restart, Shutdown, Type, Modules}

Definitions:
Id = term()
 StartFunc = {M,F,A}
  M = F = atom()
  A = [term()]
 Restart = permanent | transient | temporary
 Shutdown = brutal_kill | int()>=0 | infinity
 Type = worker | supervisor
 Modules = [Module] | dynamic
  Module = atom()

Id is only used internally by the supervisor to store the child specification, but its a general convention to have the ID be the same as the module name of the child process unless you’re starting multiple instances of your module, in that case suffix the ID with the number.

StartFunc is a tuple in the format of {Module, Function, Args} which specifies the function to call to start the process. REALLY IMPORTANT: The start function must create and link to the process, and should return {ok, Pid}, {ok, Pid, Other}, or {error, Reason}. The normal OTP start_link methods follow this rule. But if you implement a module which starts its own custom processes, make sure you use spawn_link to start them (hence the blog title, if you didn’t know).

Restart is one of three atoms, defined above in the code block. If restart is “permanent” then the process is always restarted. If the value is “temporary” then the process is never restarted. And if the value is “transient” the process is only restarted if it terminated abnormally.

Shutdown tells the supervisor how to terminate child processes. The atom “brutal_kill” shuts the child process down without calling the terminate method. Any integer greater than zero represents a timeout for a graceful shutdown. And the atom “infinity” will gracefully shutdown the process and wait forever for it to stop.

Type tells the supervisor whether the child is another supervisor or any other process. If it is a supervisor, use the atom “supervisor” otherwise use the atom “worker.”

Modules is either a list of modules this process affects or the atom “dynamic.” 95% of the time, you will just use the single OTP callback module in a list for this value. You use “dynamic” if the process is a gen_event process, since the modules it affects are dynamic (multiple handlers that can’t be determined right away). This list is only used for release handling and is not important in the context of this article, but will be used in a future article about release handling.

Whew! That was a lot of information to soak up in so little time. It took me quite a long time to remember the format of the child specs and the different restart strategies, so don’t sweat it if you can’t remember. You can always reference this information on the supervisor manual page.

Event Manager Child Spec

The first thing we want to start is the event manager, since the server depends on it. The child specification looks something like this:

  EventManager = {eb_event_manager,{eb_event_manager, start_link,[]},
            permanent,2000,worker,dynamic}.

After reading the child specification syntax section this piece of code should be fairly straightforward. You will probably need to go back and reference the spec to see what each parameter does, and that is completely normal! Its better to go back and understand the code than nod your head and forget it in a few minutes. The one “weird” thing in the spec, I suppose, is the module list is set to “dynamic.” This is because it is a gen_event and the number of modules it uses is dynamic because of the handlers plugging into it. In other cases, you would list out all modules the process uses.

Here is the init method after adding this child spec:

init([]) ->
  EventManager = {eb_event_manager,{eb_event_manager, start_link,[]},
            permanent,2000,worker,dynamic},
  {ok,{{one_for_one,5,10}, [EventManager]}}.

I like to assign each child spec to a variable, and then use these variables for the return value, rather than putting the specs directly into the return value. One of my biggest peeves in Erlang is when a programmer nests lists and tuples so deeply that you can’t see where one ends and another begins, so I recommend you assign each to a variable too.

If you compile and run the supervisor now (I think you should!), after running the start_link method of the supervisor, type whereis(eb_event_manager) and it should return the pid of the event manager process. Then, if you kill the supervisor, by doing exit(whereis(eb_sup), kill), and then try to get the eb_event_manager pid again, you should get the result that it is undefined, since the process has been killed.

Also, for fun, kill the eb_event_manager while it is running under the supervisor. Wait a couple seconds and check the process again. It should be back up!

Server and ATM

With the child spec reference and the example given above, you should have enough know-how to get the server and ATM up and running. So if you feel like challenging yourself, do that now. If not, I’ve posted the specs to get both up below:

Server = {eb_server, {eb_server, start_link, []},
            permanent,2000,worker,[eb_server]},
  ATM = {eb_atm, {eb_atm, start_link, []},
         permanent,2000,worker,[eb_atm]},

After you create these specs, add them to the list returned by the init method. Make sure that you add them after the event manager.

You can view the completed eb_sup.erl by clicking here.

Adding and Removing Children at Runtime

Unfortunately I couldn’t think of a witty scenario to fit this into ErlyBank, but I felt that it was important to mention that you can dynamically add and remove child specs to an already running supervisor process by using the start_child and delete_child methods.

They are pretty straightforward so I won’t repeat what the manual says here and I’ve linked the methods so you can go directly to them to check them out.

Final Notes

In this article about supervisors I introduced concepts such as the supervisor tree, restart strategies, child specifications, and dynamically adding and removing children.

This concludes article four of the Erlang/OTP introduction series. Article five is already written and queued up for publishing in another few days and will introduce applications.

The Scenario: With central server and ATM software in place, ErlyBank is started to feel good about their technological foundation. But as a security measure, one of their competitors, they’d like a system implemented which dispatches notifications when a withdrawal over a certain amount is executed. They want the ability to change the withdrawal notification amount threshold during runtime. They’ve chosen to hire us to streamline this into the current system software.

The Result: We’ll create an event-based notification system using gen_event. This will give us a base foundation for creating more notifications in the future, while allowing us to easily plug-in to the current server software.

What is gen_event?

gen_event is an Erlang/OTP behavior module for implementing event handling functionality. This works by running an event manager process, with many handler processes running along-side it. The event manager receives events from other processes, and each handler in turn is notified about the events, and can do what it pleases with them.

The callbacks expected for a gen_event handler module are:

• init/1 - Initializes the handler.
• handle_event/2 - Handles any events sent to the notification manager it is listening to.
• handle_call/2 - Handles an event which is sent as a call to the notification manager. A call in this context is the same as a gen_server call: it blocks until a response is sent.
• handle_info/2 - Handles any non-event and non-call messages sent to the handler.
• terminate/2 - Called when the handler is quitting so the process can clean up any open resources.
• code_change/3 - Called when the module is experiencing a real-time system upgrade. This will not be covered in this article, but it will play a central role of a future article in this series.

Compared to the previous two behavior modules covered in this series, there are relatively few callback methods. But gen_event has just as much use and just as much power as the other OTP modules.

Creating the Event Manager

First thing we need to do is create the event manager. This is a relatively simple task, since its really just starting a gen_event server and adding a few API methods on top easily add new handlers, send notifications, etc.

View my event manager skeleton here. I will be pasting in snippets from this point on.

As you can see, there is nothing out of the ordinary in the file. The start-up method is the same as every other OTP behavior module. And I’ve included a basic API to add a handler to the event manager:

%%--------------------------------------------------------------------
%% Function: add_handler(Module) -> ok | {'EXIT',Reason} | term()
%% Description: Adds an event handler
%%--------------------------------------------------------------------
add_handler(Module) ->
  gen_event:add_handler(?SERVER, Module, []).

This just adds an event handler located at Module to the event manager. And I’ve also added an easy to use notify method to send along a notification through the event manager:

%%--------------------------------------------------------------------
%% Function: notify(Event) -> ok | {error, Reason}
%% Description: Sends the Event through the event manager.
%%--------------------------------------------------------------------
notify(Event) ->
  gen_event:notify(?SERVER, Event).

This also should be pretty easy to understand. It just sends the event along to the event manager. gen_event:notify/2 is an asynchronous request, it will return immediately.

Hooking the Event Manager Into the Server

We want to make sure that the event manager is always up before the server starts, so for now we will be explicitly putting the start code into the server module. Later, in another article, we will use supervisor trees to do this for us. Here is my new init code for the server:

init([]) ->
  eb_event_manager:start_link(),
  {ok, dict:new()}.

Now that we can assume the event manager will be up during the life of the server process, we can dispatch events at certain times. Our client, ErlyBank, wants to know when a deposit over a certain amount occurs. Here is how I hooked this into the server:

handle_call({withdraw, Name, Amount}, _From, State) ->
  case dict:find(Name, State) of
    {ok, {_PIN, Value}} when Value < Amount ->
      {reply, {error, not_enough_funds}, State};
    {ok, {PIN, Value}} ->
      NewBalance = Value - Amount,
      NewState = dict:store(Name, {PIN, NewBalance}, State),
      % Send notification
      eb_event_manager:notify({withdraw, Name, Amount, NewBalance}),
      {reply, {ok, NewBalance}, NewState};
    error ->
      {reply, {error, account_does_not_exist}, State}
  end;

Now whenever a withdrawal occurs, the event is raised through the event manager too. Remember that the notify method is asynchronous and the event manager and handlers all run on separate processes. This makes all this notification happen concurrently, therefore it won’t slow down the withdrawal transaction. Of course, if the cpu is being slammed, it may take more time to execute each process, but in theory, they should happen at the same time.

Also notice that I don’t check here whether the withdrawal is over a certain amount. ErlyBank didn’t clarify with me what amount they wanted as the threshold, and it would be rather silly to hardcode it into the server process. Its generally a good idea to keep all the logic in the handlers, and just raise the notification. And this is what we will do next!

The entire contents of eb_server.erl can be viewed here.

The Handler Skeleton

As with all OTP modules, I have a basic skeleton I always start with. The one for event handlers can be viewed here.

One thing that is different about this module is that there is no start_link or start method. This is because to add an event handler we will be using the eb_event_manager:add_handler(Module) method, which actually starts and spawns the process for us!

init([]) ->
  {ok, 500}.

The init method for a gen_event handler is similar to all other Erlang/OTP behavior modules in that it returns {ok, State} where State represents the state data for the process. In this case we’ve returned 500, which we will use to signify what the warning threshold for withdrawal notifications is.

Handling the Withdrawal Notification

The sole purpose of this event handler is to process the withdrawal notification and do something if the amount withdrawn is over a certain threshold. The event is sent with gen_event:notify/2 which is an asynchronous message. Asynchronous notifications to handlers are handled in the handle_event method.

handle_event({withdraw, Name, Amount, NewBalance}, State) when Amount >= State ->
  io:format("WITHDRAWAL NOTIFICATION: ~p withdrew ~p leaving ~p left.~n", [Name, Amount, NewBalance]),
  {ok, State};
handle_event(_Event, State) ->
  {ok, State}.

Handling the message is simple. We add a matcher to match the withdrawal message, and we add a guard when Amount >= State to only get events when the amount withdrawn is above the threshold.

When the amount is above the threshold, we output it to the terminal.

The complete eb_withdrawal_handler.erl can be viewed here.

Changing the Threshold During Runtime

ErlyBank also mentioned that they want the ability to change the withdrawal notification amount threshold during runtime. To do this, we will add an API method to the actual handler. Here is the API method:

%%--------------------------------------------------------------------
%% Function: change_threshold(Amount) -> {ok, Old, NewThreshold}
%% | {error, Reason}
%% Description: Changes the withdrawal amount threshold during runtime
%%--------------------------------------------------------------------
change_threshold(Amount) ->
  gen_event:call(eb_event_manager, ?MODULE, {change_threshold, Amount}).

This introduces a new gen_event method, the call method. This method sends a request to a specific handler and expects a response, and therefore is synchronous. The arguments are: call(EventManager, Handler, Message). So for our arguments we put the event manager module, which that process is registered as, as the first parameter. We put the handler module as the second parameter, and we send a message to change the threshold.

We handle this request in a callback handle_call/2:

handle_call({change_threshold, Amount}, State) ->
  io:format("NOTICE: Changing withdrawal threshold from ~p to ~p~n", [State, Amount]),
  {ok, {ok, State, Amount}, Amount};
handle_call(_Request, State) ->
  Reply = ok,
  {ok, Reply, State}.

We first output to the terminal that the threshold is changing, and then we reply with {ok, OldThreshold, NewThreshold} and set the new state data to the new threshold. Upon receiving the next withdrawal notification, the handler will begin using the new threshold!

The complete eb_withdrawal_handler can be viewed here.

Final Notes

In this article about gen_event I introduced writing an event manager, dispatch events, writing an event handler, processing those events, and calling an event handler. The only thing I didn’t really cover which is part of gen_event is the ability for an event handler to remove itself or swap itself with another handler. The reason for this is because I don’t have much experience with these myself in a production environment. I haven’t found a position yet where I’ve needed to use them. But if you wish to learn about them, check out the gen_event manual page.

And that concludes part three of the Erlang/OTP introduction series. Article four is queued up for publishing in a few days and will cover supervisors.

The Scenario: We delivered ErlyBank’s server to them and they were very pleased. But this is the 21st century so they also need a secure and easy to use ATM system, so they’ve asked that we extend the server and also create their ATM backend. User accounts are to be protected with a four-digit PIN. From the ATM you should be able to log in to a previously existing account, and deposit or withdraw funds. There is no need to create a pretty user interface on top of the ATM backend, they have another team doing that.

The Result: First, we’ll extend the server software to hold a PIN for accounts, and add an authorization call to it. Then, we’ll use gen_fsm to create the ATM backend. We’ll rely on the server to do all the data validation.

As always, click read more to get started, if you haven’t already…

What is gen_fsm?

gen_fsm is another Erlang/OTP behavior module, and is used for implementing finite state machines.

I apologize in advance, since in this article “state” will be used to reference two different things:

• gen_fsm state - The state of a finite state machine, this is the current “mode” its in. It has nothing to do with the state you learned about with gen_server.
• state data - The state data of the server, this is the same as the state you learned about with gen_server.

A bit confusing, I know, but I will do my best to only refer to them with the above terms.

A gen_fsm starts in a state, and any calls/casts made to it are received by a special callback method which is the named the same as the state the gen_fsm module is in. Based on an action, the module can change states. A textbook example of a finite state machine is a locked door. The door starts in the state of “locked.” It requires four digits to be unlocked. After entering a single digit, the door stores it, but it needs more digits, and waits, still “locked.” After entering four digits, if they are correct, the door changes state to “unlocked” for a period of time. If the digits are incorrect, it remains “locked” and clears its memory. You’re probably getting an idea now of how we’re going to implement an ATM using gen_fsm now

As I did with gen_server, here is the list of callbacks we need to implement for gen_fsm. You’ll notice many of them are similar to gen_server:

• init/1 - Initializers the server. Almost identical to gen_server.
• StateName/2 - In this case, StateName actually will be replaced with a state name. This is called when a message is sent to the server; an action occurs on the finite state machine. This is an asynchronous callback.
• handle_event/3 - Similar to StateName/2, except that this is sent no matter what state you’re in, when the client calls gen_fsm:send_all_state_event. Again, this is asynchronous.
• StateName/3 - Equivalent to StateName/2 except this is the synchronous version. The client waits for a response from the server before continuing.
• handle_sync_event/4 - Equivalent to handle_event/3 except this is the synchronous version.
• handle_info/3 - Equivalent to gen_server’s handle_info. This receives all messages which weren’t sending using a standard gen_fsm command. This can include timeout messages, process exit messages, or any messages sent manually with the “!” symbol to the server process.
• terminate/3 - Called when the server is terminating so you can clean up any resources.
• code_change/4 - Called when a real-time system upgrade of the server is occuring. We won’t deal with this in this article, but it will be used extensively in a future article.

The gen_fsm Skeleton

Just like with gen_server, I start off every finite state machine with a generic skeleton. The gen_fsm skeleton can be viewed here.

There should be nothing out of the ordinary there. The start_link method looks just like the gen_server one did. Save that skeleton as eb_atm.erl Now we’re ready to start going!

Extending eb_server to do Account Authorization

This is again one of those tasks which I feel comfortable allowing you to do on your own. The following are the changes we need to make:

1. Account creation should require a PIN now, which will be stored with the account, unencrypted.
2. Add an authorize/2 method, where the arguments are Name and PIN. The return values should be ok or {error, Reason}.

Also, it would be a good idea to require the PIN, or some sort of authorization token with every request to deposit or withdraw money, but in order to save time, and since ErlyBank is fake (breaks my heart hah!), we will not do this.

To be honest, this isn’t really a quick-fix, but if you’re learning Erlang now on your own, you must be pretty smart So I think you can do it! Be sure to test your changes before moving on, or at least compare them to the answer linked below.

After implementing the above changes, your eb_server.erl should look like this. Note that the messages you send to the server may be different, and that is okay. Programmers think differently. The important thing is that the API outputs the same data, correctly.

ATM Design Strategy

I’m going to quick a take coding break to explain the plan for this ATM system. We’re going to implement it following the flow chart below:

The three blue boxes represent the different states the server will be in. The arrows represent what actions are necessary to switch to those states.

Initializing gen_fsm

To start the ATM, we use the same start_link method as a gen_server. But initialization is a little bit different.

init([]) ->
  {ok, unauthorized, nobody}.

The init/1 method of a gen_fsm is expected to return {ok, StateName, StateData}. StateName is the initial state of the server, and state data is the initial data of the server. In the case of ErlyBank’s ATM, we start out in the unauthorized state, and the data is set to nobody. The state data will be the account name we’ll be dealing with, so initially its nothing. Erlang has no null/nil/nothing data type, and a descriptive atom is usually used instead, like this one.

Account Authorization

We now need to implement the authorization API for the ATM. First, the API definition:

authorize(Name, PIN) ->
  gen_fsm:sync_send_event(?SERVER, {authorize, Name, PIN}).

The sync_send_event method is equivalent to the call method of gen_server. It sends the message, the second argument, to the current state of the server represented with the first argument. So now we have to write a handler for the message:

unauthorized({authorize, Name, Pin}, _From, State) ->
  case eb_server:authorize(Name, Pin) of
    ok ->
      {reply, ok, authorized, Name};
    {error, Reason} ->
      {reply, {error, Reason}, unauthorized, State}
  end;
unauthorized(_Event, _From, State) ->
  Reply = {error, invalid_message},
  {reply, Reply, unauthorized, State}.

This should read pretty logically as well. The function is named unauthorized because that is the state we expect the message to be received at. We then write a matcher to match the tuple {authorize, Name, Pin}. We then use the API methods exported by the server we wrote to authorize the user.

If the account and PIN are valid, we send the response ok back to the client. The format of the response is {reply, Response, NewStateName, NewStateData}. Following that format, we’ve changed states to authorized and for the data we store the account name.

If the account information was invalid, we respond with the error and reason and stay in the same state with the same state data.

Finally, we implement another catch-all function at the end. You should do this throughout functional programming environments but it is especially important here, since different states might be receiving actions meant for other states. For example, what if, for some reason, someone attempts to deposit while we’re still unauthenticated? We need a catch all to send back an invalid message error. This is what we do here.

Deposits

Once we’re authorized, a user is going to either deposit or withdraw from his or her bank account. We’re going to implement deposits using an asynchronous call to the server. Again, this is a bit insecure, since we’re not verifying the deposit was actually successful, but since this is a fake bank, and an ATM prototype at best, I am letting this slip.

First, the API!

%%--------------------------------------------------------------------
%% Function: deposit(Amount) -> ok
%% Description: Deposits a certain amount in the currently authorized
%% account.
%%--------------------------------------------------------------------
deposit(Amount) ->
  gen_fsm:send_event(?SERVER, {deposit, Amount}).

Simple, and this time we use the send_event/2 method instead of sync_send_event. This sends an asynchronous call to the server. Now the handling part…

authorized({deposit, Amount}, State) ->
  eb_server:deposit(State, Amount),
  {next_state, thank_you, State, 5000};
authorized(_Event, State) ->
  {next_state, authorized, State}.

Again, very simple. The method actual just forwards the information to the eb_server deposit API we wrote, which will do all the validation. But there is something strange in the return value of the deposit method! Not only does the state change to thank_you, but there is this odd “5000″ at the end there. That is actually a timeout. If no message is received in 5000 milliseconds (or 5 seconds), then a timeout message is sent to the current state. Which leads us to our next topic…

The Short “Thank You!” State

Anyone (everyone?) who has gone to an ATM knows that after you’re done banking, there is a short “Thank You” screen that shows up for a little while. And although this really shouldn’t be implemented in the back end, I wanted to show off the timeout feature of gen_fsm. After 5000 milliseconds, or if any other message is received, I swap the state back over to “unauthorized” so the ATM can start over again for the next customer. Here is the code:

thank_you(timeout, _State) ->
  {next_state, unauthorized, nobody};
thank_you(_Event, _State) ->
  {next_state, unauthorized, nobody}.

Note: The trained eye will note that both methods are equal, and its unnecessary to have the first matcher. This is true, and I just included the first matcher to be explicit about catching the timeout.

Here is the complete eb_atm.erl up to this point.

Withdrawals

Again, I will be leaving withdrawals as an exercise to the reader. You can implement it any way you’d like! Just make sure it actually does withdraw from the account

Here is my eb_atm.erl after implementing withdrawals. Note that successful withdrawals transition to the thank_you state with the timeout in place.

A Cancel-No-Matter-What Button

One of my biggest gripes of many machines is that there isn’t a big “Cancel” button that cancels everything you have been doing. And though I’ve found that killing the power to a machine does this cancel functionality quite nicely, this isn’t an option for people at an ATM. So let’s implement a giant cancel method which cancels the transaction, no matter what state it is in.

How would you do this? Well, using only the knowledge of this article, I would guess that you’d put a cancel API method which sends a cancel message to the event. Then in each event you can handle this and go back to the unauthorized state.

Witty, but not correct, by no fault of your own! I didn’t mention (or I did extremely briefly, you probably missed it) that there is a method gen_fsm:send_all_state_event/2 which sends a message, regardless of what state the server is in. We will use this, to keep our code DRY.

Our API:

%%--------------------------------------------------------------------
%% Function: cancel/0
%% Description: Cancels the ATM transaction no matter what state.
%%--------------------------------------------------------------------
cancel() ->
  gen_fsm:send_all_state_event(?SERVER, cancel).

This event is sent to the handle_event/3 method, which we extend below:

handle_event(cancel, _StateName, _State) ->
  {next_state, unauthorized, nobody};
handle_event(_Event, StateName, State) ->
  {next_state, StateName, State}.

If we receive the cancel message, the server resets the state to unauthorized and the state data to nobody: a fresh ATM!

As always, the current complete state of eb_atm.erl can be viewed here.

Final Notes

In this article, I showed you how to create a basic ATM system using gen_fsm. I showed you how to handle messages in a given state, change states, change states with timeouts, and send-to-all messages, among other things.

There are some warts in the system though, and I leave it to you to fix them. Here are two exercises for you to do, if you wish. Trust me, you have the ability:

1. Add error checking to deposits. Make it return {error, Reason} and {ok, Balance} instead of just “ok” all the time.
2. Add a “check my balance” function to the ATM. This should only be available while authorized, and should not end the transaction. This means that it should not send the user to the thank_you state. This is so that after seeing their balance, the user can withdraw/deposit if necessary.

These two features of the exercise won’t be used in future articles, and as such I won’t post the answers here. You can test them yourself to make sure they work!

Part two of this Erlang/OTP series is over. The third article is already in the pipeline and will be published in another few days. It will cover gen_event. For fun, you can think what I’ll be adding to ErlyBank using this!

I hope you have been enjoying these introductions to Erlang/OTP as much as I’ve enjoyed writing them. Thanks for all the support, and good luck!

The Scenario: ErlyBank is just starting and they want to start off on the right foot by creating a scalable system for managing the bank accounts of its important customer-base. Having heard about the powers of Erlang, we have been hired to do that! But to test us out, they first want a simple server to handle the creation and destruction of accounts, depositing money, and withdrawing money. They just want a prototype, not something they’ll actually be putting into production.

The Result: We will create a simple server and client to do everything they asked using gen_server. Since its just a prototype, we’ll store the account information in memory for now. The accounts will only be referenced by name, no other information is necessary to create an account. And of course we’ll have validation on withdrawals and deposits.

Note: Before beginning this series, I am assuming you have a basic knowledge of the Erlang syntax. If not, I recommend reading my compendium of beginner erlang resources to find a place you can learn Erlang.

If you’re ready, click read more below to get started! (Unless you’re viewing the whole article already )

What is gen_server?

gen_server is a behavior module for implementing client-server relationships. By using this OTP module, you’re given a lot of things “for free,” as I’ll explain later. Also, later in the series, when I talk about supervisors and error reporting, this module will fit right in without many changes.

Since gen_server is a behavior, you’re expected to adhere to a certain number of callbacks. These callbacks, in no particular order, are:

• init/1 - Initializes the server.
• handle_call/3 - Handles a call made to the server. The client who called the server is blocked until you return from this method.
• handle_cast/2 - Handles a cast made to the server. A cast is similar to a call except that it is asynchronous; the client continues while this cast is running.
• handle_info/2 - A sort of “catch all” method. If a message is sent to the server process that isn’t a call or a cast, it will come here. An example of such a message is an EXIT process message if your server is linked to any other process.
• terminate/2 - Called while the server is stopping, so any cleanup can be done.
• code_change/3 - Called when the server is being upgraded in real-time. This method will be covered in much greater detail in future articles, but you must at least put the stub of the method into your module to adhere to the behavior.

The Server Skeleton

To start off every gen_server file, I always use a common skeleton. If you can, create a snippet or something with your text editor to paste the skeleton. The skeleton can be viewed here. Please take a quick look at it.

Note: To save space, I won’t ever paste the entire file here in this post, I will try to link to the whole thing when such a change occurs. But for the most part, I will just be pasting in snippets.

As you can see, the module is named eb_server, for “ErlyBank Server.” It implements all the callback methods I mentioned above and also adds one more: start_link/0. This will be the method used to start the server. I’ve pasted this portion of the code below:

start_link() ->
  gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).

 
This calls the start_link/4 method of gen_server, which starts the server and registers the process to the atom defined by the macro SERVER, which by default is just the module name. The remaining arguments are the module of the gen_server, in this case itself, any arguments, and finally any options. We leave the arguments blank because we don’t need any and we don’t specify any options. For more information, an in-depth description of the method is available on the gen_server manual page.

Initializing The ErlyBank Server

Once the gen_server:start_link method is called, it calls the init method of the actual server. It is in this method you should initialize any state for the server. State can be anything you want: An atom, a list of values, a fun, anything! This state is passed around in every callback to the server. In this case, we’d like to maintain a list of all accounts and their account balance. Additionally, we’d like to look up these accounts by name. To do this, I’m going to use the dict module, which stores key-value pairs in memory.

Note to Object-Oriented Minds: If you come from an OOP language, you can think of the server state as its instance variables. In every gen_server callback, you’ll have these instance variables available, and you can change them, too.

So here is my final init method:

init(_Args) ->
  {ok, dict:new()}.

 
Really, its that simple! One of the return values expected for init is {ok, State} so for the ErlyBank server, I’m just returning ok, and an empty dictionary for the state. And we don’t use the arguments to init (which is an empty list anyway, remember, from start_link), so I just prepend the argument with an underscore to note that.

To Call or To Cast, That is the Question

Before we implement the bulk of the server, I want to quickly reiterate the difference between a call and a cast.

A call to the gen_server is a blocking method for the client. This means that when the client sends a message to the server, it waits for a response before moving on. You use a call when you need a response, such as if you’re asking what the balance of an account is.

A cast to the gen_server is a non-blocking or asynchronous method. This means that when the client sends a message to the server, it continues on, without receiving a response. Now, Erlang guarantees that all messages sent to a process are received, so unless you explicitly need a response value from the server, you should cast, as it will keep your client executing. That is to say, you don’t need to make a “call” to the server if you simply want a response to make sure the server received the message, since Erlang guarantees this.

Bank Account Creation

First things first, ErlyBank needs a way to create new accounts. Quickly, test yourself: If you were creating a bank account, would you cast or call to the server? Think hard… what value would you need returned? If you said call you’re correct, normally anyways. You usually want to make sure the account was created successfully without just assuming it was so. But in this case, I’m going to implement it as a cast, since we won’t be error checking for now.

First, I’m going to create the API method, the method outside modules will call to create an account:

%%--------------------------------------------------------------------
%% Function: create_account(Name) -> ok
%% Description: Creates a bank account for the person with name Name
%%--------------------------------------------------------------------
create_account(Name) ->
  gen_server:cast(?SERVER, {create, Name}).

This sends a cast to the server, which we registered as ?SERVER in start_link. The request is a tuple {create, Name}. Since this is a cast, it returns immediately with “ok,” which is also returned from the function.

Now, we need to write the part to handle this cast, which is a callback for the gen_server:

handle_cast({create, Name}, State) ->
  {noreply, dict:store(Name, 0, State)};
handle_cast(_Msg, State) ->
  {noreply, State}.

As you can see, we just add another function definition for handle_cast to capture our create request. We then store it in the dictionary, with a value of 0, representing the current account balance. The return value expected for a handle_cast is {noreply, State} where State is the new state of the server. So this time, we return a new dictionary with the added account.

Also, notice I added a catch-all function at the end. This is not just good practice here, but for all functional programming in general. You can use this method to just absorb the message silently, or you could always raise an exception if you have to. In this case, I absorb it silently.

The entire eb_server.erl at this point can be viewed here.

Money Deposit

We promised our client, ErlyBank, that we’d add an API to deposit money into a person’s account, and that we’d include basic validation. So we need to write a deposit API method, and the server also needs to verify that the account exists before depositing money, since ErlyBank doesn’t want would-be customer’s money going into a black hole. Again, test yourself: A cast or call? This one is easy: A call. We need to make sure the deposit was successful, and notify the user.

Just like before, I’m going to first write the API method:

%%--------------------------------------------------------------------
%% Function: deposit(Name, Amount) -> {ok, Balance} | {error, Reason}
%% Description: Deposits Amount into Name's account. Returns the
%% balance if successful, otherwise returns an error and reason.
%%--------------------------------------------------------------------
deposit(Name, Amount) ->
  gen_server:call(?SERVER, {deposit, Name, Amount}).

Nothing exciting here. We just send along a message to the server. The above should look familiar to you, as its the exact same as a cast so far. But the difference is in the server code:

handle_call({deposit, Name, Amount}, _From, State) ->
  case dict:find(Name, State) of
    {ok, Value} ->
      NewBalance = Value + Amount,
      Response = {ok, NewBalance},
      NewState = dict:store(Name, NewBalance, State),
      {reply, Response, NewState};
    error ->
      {reply, {error, account_does_not_exist}, State}
  end;
handle_call(_Request, _From, State) ->
  Reply = ok,
  {reply, Reply, State}.

Whoa! Lots of new stuff! First, the method definition looks similar to handle_cast, except for the added From argument, which we don’t use. This argument is the pid of the calling process, so we can send additional messages if needed.

We promised ErlyBank we’d validate the existence of accounts, and the first line is where we do it. We attempt to find a value from the state dictionary for the person trying to deposit the account. The find method of dict returns one of two values: Either {ok, Value} or error.

In the case that the account exists, Value equals the current balance in the account, so we add the deposited amount to that. We then store the new balance for the account into the dictionary and set that to a variable. I also store the response in a variable, which should look just like the comment for the deposit API said it would: {ok, Balance}. Then, by returning the tuple {reply, Reply, State}, the server sends the Reply back to the calling process, and stores the new state.

On the other hand, if the account doesn’t exist, we don’t change the state at all, but we reply with the tuple {error, account_does_not_exist}, which again follows the spec in the comments of the deposit API.

Again, here is the updated eb_server.erl source after this addition.

Account Destruction and Money Withdrawals

I am actually going to leave it as an exercise to the reader to complete the account destruction and withdrawal APIs for the module. You have all the necessary knowledge to do these things. If you need any help with the dict module, refer to the dict module API reference. For withdrawals, please validate both that the account exists and that it has enough funds for the withdrawal. You do not have to handle the case of withdrawing negative funds.

Once you’ve finished, or if you’ve given up (I hope not!), you can view the answer here.

Final Notes

In this article, I showed you the basics of gen_server and how to create a client-server relationship. I did not cover the full capabilities of gen_server which include such things as timeouts for message receiving and stopping the server, but I feel I’ve explained a solid portion of the behavior module.

If you wish to learn more about the callback methods, possible return values, and more advanced usages of gen_server, read the manual on gen_server. Read all of it, really.

Also, I know I did not touch and I barely mentioned the code_change/3 method, which for some reason causes the most excitement in people. Don’t worry, I have an entire article (near the end of the series) dedicated to deploying real-time upgrades to running systems, and this is where this method will come into play.

The next article, in a few days, will be on gen_fsm. So if this article tickled your brain, feel free to jump in on that manual and get a head start! Maybe you can guess what wacky continuation of ErlyBank I’ll be adding with gen_fsm.

OTP is much too complex of a thing to explain in a single article, and I will not attempt to do so here. Instead, I will be making this a series of articles spanning a couple weeks.

Why should I learn about Erlang/OTP?

The platform which comes with Erlang gives you a rich and very generalized set of libraries and behaviors to easily create highly available, fault tolerant, hot code swapping ready systems. The following is a basic list of what it gives you “for free:”

• Generic server, finite state machine, and event manager behaviors
• Standardized application structure
• Hot swappable code
• Supervisor Trees so your processes always stay up

In addition to these free features, OTP is the standardized way to create systems in Erlang. The big name open source Erlang applications such as ejabberd,