Reflections on EclipseCon

Thanks to everyone at EclipseCon this past week for making it a great experience. JaxEnter asked me to write a short reflection about my experiences. I don't want to steal traffic from them, so you can see the article by following the link above :-).

The article is technology-centric. It was difficult for me to choose from among the many great things that are going on with Eclipse since it is such a vibrant and innovative community. The technologies in the article have heavy support from the top-level projects, so they will be interesting to follow. Also, in reality, they only reflect my own interests and experience and I could not have possibly seen everything going on at EclipseCon. That isn't to diminish any of the other great technologies that are built on Eclipse. There are a tonne of them that have become so integrated into my daily work, that I barely even notice them anymore. That's a real compliment, I think.

The best thing, really, though is the community. I have to admit that it was a little difficult to get to talk to people before I gave my talk on Wednesday. After that, though, I had a few great conversations. I'd like to make a shout-out to the guys over at Ericsson. For the past while, we have been working on the same set of problems, with support by the same funders. We had to go all the way to California for EclipseCon before we had heard about each other, though. It's a funny world.

Oh well, time to head back home. Good-bye cold, rainy California. I'll be going back to the rain and palm trees of Victoria :-).

Tracing Web Apps Using Diver

Diver is a tool designed to help Java Developers understand how unfamiliar, or long-forgotten, software works. It lets developers easily trace the actions of Java Software and analyze it using an advanced Sequence Diagram View and some workbench filters that help focus on the software artifacts associated with particular features of the software.

With more and more applications moving to the Web, I've been asked a number of times, "Can Diver trace web apps?" I've always told people that it is possible to do, but I haven't given detailed instructions on how to do it. This post will hopefully remedy that.

First, a few notes. The version of Diver used to illustrate the points in this post is 0.3.1. It isn't an official release yet because there is still some testing to be done. I've had to do some tweaking to Diver, though, to make tracing web apps a little easier. So, you can get your copy of Diver 0.3.1 by pointing p2 to http://diver.svn.sourceforge.net/svnroot/diver/Development. Another thing: Diver only traces Java, not JavaScript. So, Diver can't be used to help get at any JavaScript errors that are running in the browser. It can, however, trace Java apps that have been deployed on a Java-enabled web server such as Jetty, Tomcat, or JBoss. That includes code invoked from JSPs. In this post, I'll be using JBoss 5.0.1GA. There is no particular reason for that. I've used Diver to trace applications deployed in Jetty as well.

Install And Deploy Your Web App

The focus of this post is on tracing web apps, not on setting up web servers. So, I assume that you have your server ready to go. It's probably best, though, that you run on a small test server on your localhost rather than on a deployment machine. That just keeps things compact and easier to manage.

Diver works best if you have the source code for that app. It isn't strictly necessary, but it makes the tools that Diver offers a lot more powerful. At the very least, you have to be able to deploy your app into your own web server. You can just build and deploy your web app the way you normally would. For example, in JBoss, this would typically involve copying a WAR file into the jboss_root/server/server_name/deploy directory. When you are tracing your app, Diver will actually be tracing the copy that has been deployed on your server, not the version that is in your workspace. So, after you make any changes to your code, be sure to rebuild and redeploy before running your trace. Then, Diver will be able to match the calls to various methods/classes in the deployed WAR to those classes in your workspace, making it a lot easier to find things.

Preparing for Launch

When tracing a Java Web App, what we are really doing is capturing specific events that happen inside a Java web server instance. So, the application we will be tracing is the Java Web Server. Ideally, Diver would be able to interact with the Eclipse Web Tools Platform so that you could just fire up a trace of a web server launch. Unfortunately, I haven't gotten that far yet. Fortunately, though, programs like JBoss are really just Java applications, so we can use the Java Application Trace functionality of Diver to grab a trace, but it will take a little set up first.

The Diver Java Application Trace is just an extension of the JDT Java Application Launch. The JDT requires that the applications it launches are visible in the workspace, so Diver requires the same. That doesn't mean, though, that you have to have all of the JBoss source to get started. You can launch directly from your installed JBoss instance. To get JBoss to run, you just need to make the right jar files visible. This is how you do that.

I suggest that you make the appropriate jars visible in their own project. You could, possibly, create dependencies to JBoss or Jetty, or what-have-you, in your web app's project, but that could cause build problems or runtime errors if your web app needs its jars to be isolated from the server. I just create a small Java Project called JBoss:

Once the project is created, you need to make the JBoss launch jar visible. To do that, just select the JBoss project in the Package Explorer or the Navigator and choose File>Properties <Alt+Enter> from the menu. Jump over to the Libraries tab of the Java Build Path settings page, and select Add External Jars:

The jar that you are looking for is run.jar. It's just the JBoss launcher. On my computer, it is found in C:\jboss-5.0.1.GA\bin:

That's all that you need for preparation. Now, we can get onto the launch.

Setting Up The Trace

Now that the JBoss launcher is visible to Eclipse, we can get onto running it to gather a trace of our web app.  Open up your Run or Debug Configurations dialog, and create a new Java Application Trace. In the Main  tab, set the Project to the one we created earlier. The Main class that we will use is just org.jboss.Main. Simple enough:

Your web server will likely require a number of arguments to be passed both into the program and into the VM. Go to the Arguments tab. JBoss requires the following arguments:

Program Arguments:

-c <server> where <server> is the name of the configured server that you deployed your web app into. I typically just use default one supplied by JBoss, so my program arguments are -c default.

VM Arguments:

-Xshare:off -Dprogram.name=run.bat -server 

-Dorg.jboss.resolver.warning=true 

-Dsun.rmi.dgc.client.gcInterval=3600000 

-Dsun.rmi.dgc.server.gcInterval=3600000 -Xms128m -Xmx512m  

-XX:MaxPermSize=256m

-Xshare:off is a Sun/Oracle specific VM argument that is there for Diver's sake. It tells the VM that this launch will not share its classes with anyone (the Java VM allows different processes to share the same loaded classes unless this flag is set). If you aren't using the Sun/Oracle VM, get rid of this argument. The other ones are there for JBoss. The really important ones (I think) are -Dprogram.name=run.bat (or run.sh if you are running on Linux) and -server. You can set the heap and Perm Gen sizes according to your liking.

Working Directory:

/jboss_home/bin. On my computer, this is just C:\jboss-5.0.1.GA\bin. This has to be set so that JBoss can know where it is running from and resolve all of its own resources.

If you are wondering how I discovered what all of these arguments should be, I did it by inspecting the run.bat script that is used to launch JBoss. The script just sets a bunch of default variables, and I used those. Another thing that you should be aware of is that you need to have your JAVA_HOME environment variable set to a valid JDK. A normal JRE might work, but (depending on the version of your server) it may not be able to compile JSPs unless you have the JDK. I'm sure you already knew that, but I'm just throwing it out there for good measure.

Tweaking The Trace

That is all you technically need to do to get started. Now, you can just select the Run button and start gathering information like you normally would with Diver (see the Diver documentation). But there are a few things that you can do to tweak the trace and make sure it is efficient. Web apps can be slow as it is. Tracing them introduces i/o overhead that we would like to minimize. Go over to the Java Trace tab of your configuration, and set it up to look something like this:

First of all, you are likely only interested in the things that your web-app is doing, so I suggest selecting the Set Filter Manually option, and adding only the packages that are a part of your web app to the Inclusion Filters area. By Default, Diver will analyze the trace and only watch for methods defined in the project that is named in the Main tab of the launch configuration. In this launch, there is nothing but the JBoss launching jar, so the analysis would end up being mostly useless unless you set the filters manually.

I also suggest making sure Pause On Start and Apply Filters at Runtime are checked. Pause On Start is the default, and it just means that none of the start-up process of the program will be traced. You can define what actually gets recorded in real-time using the Eclipse Debug View as described in the Diver documentation.

Older versions of Diver (0.2 and earlier) recorded absolutely every method call, and "rejected" data based on the defined filters during a separate analysis step. Since version 3.0, Diver allows you to reject that data before it gets stored, which can speed up your traces and your analysis by many, many times. Make sure Apply Filters at Runtime is checked to enable this speed up.

Get Going

There you go, that is all you need to get a trace of a web app using Diver. Now, you can record and compare traces to focus in on specific bugs or features in your app and use Diver's sequence diagram to analyze what is going on. The Diver documenation, and the video tutorials tell you how to do all that fun stuff:

There is just one more small bit of housekeeping that you might want to be aware of. Launching a web server in this way doesn't give you a clean, interactive way to shut it down. You can always just use the "Stop" button in the Debug View to force it to quit, but that could leave your server in an undefined state. I suggest opening up a console, and using your web server's shutdown command. For JBoss, all you have to do is type shutdown -S.

That's all for now. If you are out at EclipseCon, make sure to come to my talk where I'll be presenting Diver and how it helps users of Eclipse understand their software. Happy coding!

Diver is a finalist for an Eclipse Community Award

I've been swamped with work lately, so I'm afraid that I haven't been able to devote much time to blogging. But I got news today that my Diver tool has been selected as a finalist for the Best Developer Tool category along side the excellent Pydev environment for the Python language. Diver has really been my baby for the past couple of years. I've been working hard on trying to solve the problems associated with understanding software. I hope that Diver can help developers in that respect (I know it helps me :-) ). Even so, it is a great honour to be selected as a finalist from among more than 40 other excellent solutions. Eclipse has such a vibrant developer community. There is a solution out there for almost every problem.

If you are going to EclipseCon, be sure to check out my talk to find out more about Diver and how to incorporate reverse engineering into your everyday work to improve your understanding of software.

See you at EclipseCon!

I felt like I won an award this morning when I found this in my email:

Dear Del Myers,
        
We are pleased to accept your proposal "Put It In Reverse: Using Eclipse to
Understand Code that has Already Been Written" for EclipseCon 2011. 

The Program Committee received many excellent submissions, and we're 
very happy to include "Put It In Reverse: Using Eclipse to Understand Code
that has Already Been Written" in that select set.

Thank you for helping to make EclipseCon 2011 the best one yet!

I'm certain that EclipseCon will be excellent, and I hope to do my part in making it that way. So, everyone come and check out my talk if you have the time: https://www.eclipsecon.org/submissions/2011/view_talk.php?id=2099

Interview about Diver

The online magazine JaxEnter.COM recently interviewed me about my tool Dynamic Interactive Views for Reverse Engineering (Diver), which is an open source tool for bringing reverse engineering processes into the hands of us ordinary developers. Wondering what this tool has to offer? Check out the interview: http://jaxenter.com/diver-interview-32838.html

Diver version 0.3.0

Last night, I released version 0.3.0 of Diver. It has been a while since I've written here. I've been busy with other work that had to be done. So, for those who may no longer be familiar with Diver, I will take a moment to step back and let you know what it is all about.

Diver stands for Dynamic Interactive Views for Reverse Engineering. The basic of Diver is to make some reverse engineering tools and techniques easily accessible to us Java developers who use Eclipse. Some really common problems that I have had in maintaining software are trying to figure out answers to questions like, "Why did that happen?",  "Where is that implemented?" and, "How does that work?" Diver offers tools to help you answer those kinds of questions. The techniques realized in Diver use execution traces of your running Java software so that you can get a picture of what is really going on at runtime rather than having to guess at what might be happening based on class hierarchies or keyword and reference searches. It isn't meant to replace your current debugging and investigation techniques. It is meant to offer you a few extra tools to augment your current work practices. Diver has its own website: http://diver.sf.net. You can go there to find out more.

Now, onto the cool new features of Diver 0.3.0. Some highlights are on Diver's New and Noteworthy page, but these are the ones that I'm most excited about:

More Efficient Tracing


I've put a lot of effort into cutting back trace sizes and reducing the amount of time that it takes to analyze the traces for use in Diver. Previous versions recorded everything (every single method call) that was executed during the run of your program and applied custom package and class filters during a separate analysis step. In Diver 0.3.0, it is possible to apply those filters up-front so that you can control which methods get called during execution. This can make your target program more responsive and greatly reduce the amount of time that it takes to do the second analysis step. The Diver documentation gives more information about how to set this up.

Compare Traces Using the Sequence Diagram


Diver has always offered a cool feature which allows you to compare the difference between traces. It offers a filter on the Eclipse package explorer which allows you to locate method calls unique to a particular trace (or to a particular thread of execution within that trace). That filter has never been applied to the sequence diagram viewer offered by Diver, however. Any solution just seemed to inefficient to be able to apply to the massive amounts of data that the sequence diagram has to be able to display. Well, I've figured out a way to do it which seems to be quick enough for my own usage, so hopefully it will be for you as well. It is now possible to see which method calls are unique to a particular trace directly within the sequence diagram:

Methods that are not unique to the trace can either be shown in grey, or hidden completely:

This feature (and several others) are highlighted in the following video. It is a pretty good video for just giving you an overview of what Diver has to offer as well:

If you want to install Diver, the p2 site is here: http://diver.svn.sourceforge.net/svnroot/diver/Release. Or, if you are using Eclipse 3.6, you can use the totally awesome Eclipse Marketplace to install it directly. Just do a search for the keyword: Diver.

Diver in linux

Some people may be having difficulty with Diver in various distributions of linux. You may be getting the following message when trying to launch a trace.

And you may see this in the console:


Error occurred during initialization of VM
Could not find agent library in absolute path: /[path-to-plugin]/libsketch_linux32.so


Now, you may be surprised to find out that these errors may have nothing to do with open ports or whether or not the file listed exists. This is not Diver's fault. The client can't find an open port because the Java virtual machine couldn't start and open one. The Virtual machine couldn't start not because the listed library is missing but because you might not have some of the dependencies installed on your system. Unfortunately, the Java VM sometimes gives some pretty uninformative error messages.

Diver depends on the C++ boost libraries to do socket communication and multi-threading. It also uses boost to do multi-platform file system manipulation. Many distributions of linux come with boost installed. Some do not. If you have a debian-based version of linux, installing the required boost libraries is easy:

$ sudo apt-get install libboost-iostreams1.40.0 libboost-date-time1.40.0 libboost-filesystem1.40.0 libboost-system1.40.0 libboost-thread1.40.0

If you are running a redhat distro, you should be able to use a similar yum command. I hope that this all works for everyone.