QCon 2008

This has been a great week. We were in London to show XSTM at QCon. Have a look at our luxurious booth! Photos are not of great quality but the phone did its job.

Nicolas and I waiting for other geeks!

Great trip, we met lots of interesting people and lots of them were interested by the project. This is encouraging. Next version is a little late on planning, probably not before one or two weeks... In the meantime I assembled a benchmarking demo for the conference based on our latest build. I will publish it soon on the web site so you can benchmark XSTM on your machines and network.

“You should write more documentation”

The code works quite well so it's time to document. The web site needed a refresh and more content. There is now a .NET version of this project called NSTM in addition to the Java one. The new site also has a tutorial to get started, and you can download a book that I have been working on for some time. It goes through some recurring problems developers have to face today in concurrent and distributed programming and the observations that started this project. It tries to give you a good sense of what XSTM does, the way it works and what can and cannot be done with it. I give it for free but of course if you want to buy the printed version it will be much appreciated!

I also made a second demo which shows how you can use XSTM to manage data entered through a form.

For those of you who where at Javapolis, did you see the Adobe Flex Data Services demo? It was about having two flex clients talking to a web server. Each client shows a grid with identical data. If some data is edited in the grid on one of the clients, the modifications can be sent to the server and to the other clients using a Commit button. Conflicts are detected, and modifications can be cancelled.

This new demo shows how to do this with XSTM. It is implemented as an Ajax application using GWT but it could be done in Java or .NET with almost the same code. Adobe also provides support for offline use of the application, which might be possible using Google Gears, but there is no support for this yet in XSTM.

JSTM 0.3

For this release I made a demo to show the compatibility between the Java, GWT and .NET versions of JSTM. It is made of three identical applications written with each platform, which communicate with each other (The first one launched starts a socket server, the next ones connect to it, so it is normal that they show warnings that the socket port is already used). They allow you to drag images on the screen, keeping the positions synchronized. You can download it here.

This release brings a few improvements. First it is possible to modify shared objects without starting a transaction. A transaction is automatically started for each field update. For performance reasons it is still better to start one yourself if you have multiple fields to change on an object.

It is also now possible to generate your shared objects using XML. The schema allows you to specify fields and methods for your objects. A "transient" keyword on a field means that if you share an object with another machine, the objects it references will not be shared. The default false value makes JSTM walk the graph of references and add all objects to the share. I will try to make transforms to create XML from existing Hibernate HBM or Microsoft EDM data models.

JSTM API changed a little. Now after generating your shared objects, you have to register the resulting ObjectModel class on your local site, and not on your transports as before. Look at the samples for more info. This allows you to have multiple object models in an application.

For the releases to come, I will work on persisting shared objects and enabling Terracotta to have a better cluster support. If you have ideas or suggestions about this please contact me or post on the forum.

Javapolis

I am at Javapolis today and I am meeting lots of very interesting people. It seems the data synchronization problem is getting more and more attention. Adobe presented their Flex Data Services. It keeps track of local modifications to data, and offers a way to commit them to the server or undo them so the local data returns to the previous values, which ressembles a lot what JSTM does.

JSTM 0.3 is almost ready, with .NET support. Also with this version is the ability to define the object model of an application using XML instead of today's hand-written description. And lastly, a transaction will be automatically created if you modify an object without starting a transaction first, which reduces the code you need to write.

JSTM 0.2

Good news everyone! Sorry for the much too long wait. Finally the 0.2 version is ready, and this time the GWT version comes along too. After we released 0.1 with Luciano, a few features have been asked for, and they took much longer than expected because of design flaws in the previous version.

Now things are much simpler and homogenous. The new JSTM library exposes only a handful of classes, but they cover the main aspects of a distributed application. There is now a Site class which represents a process and a Group class for groups of sites like “a server and all its clients”. The Share class is functionally the same as in version 0.1 but it is now decoupled from communication. This is now done by the various implementations of the Transport class. The transaction manager is removed, now transactions are managed by each site. E.g. to start a transaction: Site.getLocal().startTransaction().

A site can start transports to connect to other sites, and create shares to replicate objects with them. Each share can involve a different set of sites and groups. As there is no security model yet, any site can modify any object, including who is involved in a share. Lastly, now every transacted object has a getOrigin() method that returns the site where it has been created. Transactions also have this method so it is possible to know which site initiated a modification on a given object when you are notified of it. This also makes possible to log centrally who is doing what to which object.

It will take time to make samples and documentation about this so don’t hesitate to post questions on the forum.

Chris Brumme in the mail

First, a warning: This post gets into issues that I think are complicated, but they are mostly implementation details not related to using jstm, which is easy :)

A few days ago I sent links to jstm to people I thought could be interested. I had the surprise to have an answer from Chris Brumme. In person. What a great day :) He made an interesting comment about distributed transactions. I quote also the beginning of his message because I like it:

It’s an interesting idea and may lead to an appealing programming model.

Of course, distributed transactions work best over short distances. I reject the notion of 2-phase commit over the Internet, and I dislike it in a data center or cluster.

The “indeterminate” state of 2PC is difficult to build on top of, particularly if partitions can endure. In some ways, it’s more appealing to build on top of asynchronous messages that can get delayed or lost. At least there you know that indeterminacy is unavoidable and the application will naturally build it into its assumptions.

Certainly transactional memory is a fascinating topic. It has the potential to evolutionize concurrency the way garbage collection revolutionized memory management. But for me it’s still too soon to know whether it will fulfil that potential.

I agree that it is difficult to build upon distributed transactions, but here is why I think that it is not a big issue with jstm:

Jstm transactions are not really distributed in the sense that there is only one authority who decides if they are committed or discarded. So even if transactions modify objects spread on multiple machines, the actual commit is always done on the same machine. In the client-server topology, the server is the authority. In the cluster one, a coordinator is elected (It will be possible to change it at runtime in the future). Let’s take an example with the client-server topology:

A server and clients applications have a share with identical objects. A client starts a transaction, modifies a few objects. For now those changes are private to the client, they are kept in the transaction. When the client wants to commit, the share will "intercept" this commit, the whole transaction with private changes is sent to the server, and it is applied on the server’s copies of the objects. The server decides if the transaction can commit, and then broadcasts the new version of the share: It sends an acknowledgement to the initiating client and forwards the changes to the other clients.

The server does not wait for all clients to have acknowledged the changes before considering a transaction committed, because we would then return to the joys and issues of 2 phases commit. It simplifies a lot, but then what happens if a client does not receive a change? To go in Chris’s direction, a library should not hide this indeterminacy but expose it to the developer, if possible in a way that naturally makes him deal with it. Jstm could probably do better with this. For now the solution is simple:

First transaction commits are asynchronous. The method is beginCommit(callback), so the developer has to wait explicitly for the callback to know that his transaction has succeeded. Second, jstm has a simple heartbeat mechanism which disconnect clients is they do not acknowledge changes after a certain time. When a client gets disconnected, all his local state is considered invalid so his share is cleared and pending transactions are discarded. If he manages to reconnect, he can download the content of the share again and continue to work.

If you are in Paris on July 4, go see Luciano show our GWT version here

Hello world!

In this first post, let’s have an overview of the current state of jstm. This project started early 2006, and the first solid release is almost ready. The GWT version is running great. Luciano, the creator of gwm, is building some impressive demos. For the java version, last benchmark between two applications showed a quite impressive sustained rate of almost 15 000 transactions/s on standard desktop PCs. We are starting to get positive feedback from developers and future developments suggestions. The next feature I will work on is a way to call a method on a remote share. In a client-server topology, this will facilitate the use of jstm to query a database on the server and return a set of objects through replication. More on this in a future post. I hope you will find this project useful, some documentation and great samples are on the way to help you get started!