Tuesday, December 21, 2010

See you at EclipseCon!

I felt like I won an award this morning when I found this in my email:

Dear Del Myers,
We are pleased to accept your proposal "Put It In Reverse: Using Eclipse to
Understand Code that has Already Been Written" for EclipseCon 2011. 

The Program Committee received many excellent submissions, and we're 
very happy to include "Put It In Reverse: Using Eclipse to Understand Code
that has Already Been Written" in that select set.

Thank you for helping to make EclipseCon 2011 the best one yet!

I'm certain that EclipseCon will be excellent, and I hope to do my part in making it that way. So, everyone come and check out my talk if you have the time: https://www.eclipsecon.org/submissions/2011/view_talk.php?id=2099

Tuesday, December 7, 2010

Interview about Diver

The online magazine JaxEnter.COM recently interviewed me about my tool Dynamic Interactive Views for Reverse Engineering (Diver), which is an open source tool for bringing reverse engineering processes into the hands of us ordinary developers. Wondering what this tool has to offer? Check out the interview: http://jaxenter.com/diver-interview-32838.html

Thursday, November 25, 2010

Diver version 0.3.0

Last night, I released version 0.3.0 of Diver. It has been a while since I've written here. I've been busy with other work that had to be done. So, for those who may no longer be familiar with Diver, I will take a moment to step back and let you know what it is all about.

Diver stands for Dynamic Interactive Views for Reverse Engineering. The basic of Diver is to make some reverse engineering tools and techniques easily accessible to us Java developers who use Eclipse. Some really common problems that I have had in maintaining software are trying to figure out answers to questions like, "Why did that happen?",  "Where is that implemented?" and, "How does that work?" Diver offers tools to help you answer those kinds of questions. The techniques realized in Diver use execution traces of your running Java software so that you can get a picture of what is really going on at runtime rather than having to guess at what might be happening based on class hierarchies or keyword and reference searches. It isn't meant to replace your current debugging and investigation techniques. It is meant to offer you a few extra tools to augment your current work practices. Diver has its own website: http://diver.sf.net. You can go there to find out more.

Now, onto the cool new features of Diver 0.3.0. Some highlights are on Diver's New and Noteworthy page, but these are the ones that I'm most excited about:

More Efficient Tracing

I've put a lot of effort into cutting back trace sizes and reducing the amount of time that it takes to analyze the traces for use in Diver. Previous versions recorded everything (every single method call) that was executed during the run of your program and applied custom package and class filters during a separate analysis step. In Diver 0.3.0, it is possible to apply those filters up-front so that you can control which methods get called during execution. This can make your target program more responsive and greatly reduce the amount of time that it takes to do the second analysis step. The Diver documentation gives more information about how to set this up.

Compare Traces Using the Sequence Diagram

Diver has always offered a cool feature which allows you to compare the difference between traces. It offers a filter on the Eclipse package explorer which allows you to locate method calls unique to a particular trace (or to a particular thread of execution within that trace). That filter has never been applied to the sequence diagram viewer offered by Diver, however. Any solution just seemed to inefficient to be able to apply to the massive amounts of data that the sequence diagram has to be able to display. Well, I've figured out a way to do it which seems to be quick enough for my own usage, so hopefully it will be for you as well. It is now possible to see which method calls are unique to a particular trace directly within the sequence diagram:

Methods that are not unique to the trace can either be shown in grey, or hidden completely:

This feature (and several others) are highlighted in the following video. It is a pretty good video for just giving you an overview of what Diver has to offer as well:

If you want to install Diver, the p2 site is here: http://diver.svn.sourceforge.net/svnroot/diver/Release. Or, if you are using Eclipse 3.6, you can use the totally awesome Eclipse Marketplace to install it directly. Just do a search for the keyword: Diver.

Friday, June 25, 2010

Diver in linux

Some people may be having difficulty with Diver in various distributions of linux. You may be getting the following message when trying to launch a trace.

And you may see this in the console:

Error occurred during initialization of VM
Could not find agent library in absolute path: /[path-to-plugin]/libsketch_linux32.so

Now, you may be surprised to find out that these errors may have nothing to do with open ports or whether or not the file listed exists. This is not Diver's fault. The client can't find an open port because the Java virtual machine couldn't start and open one. The Virtual machine couldn't start not because the listed library is missing but because you might not have some of the dependencies installed on your system. Unfortunately, the Java VM sometimes gives some pretty uninformative error messages.

Diver depends on the C++ boost libraries to do socket communication and multi-threading. It also uses boost to do multi-platform file system manipulation. Many distributions of linux come with boost installed. Some do not. If you have a debian-based version of linux, installing the required boost libraries is easy:

$ sudo apt-get install libboost-iostreams1.40.0 libboost-date-time1.40.0 libboost-filesystem1.40.0 libboost-system1.40.0 libboost-thread1.40.0

If you are running a redhat distro, you should be able to use a similar yum command. I hope that this all works for everyone.

Friday, June 18, 2010

Awesomeness in Helios

There was a recent post by Wayne about how Eclipse is an IDE Platform. For all practical purposes, that is the way that I use it. But what is really awesome about Eclipse as an IDE platform is that it is an IDE platform that just keeps improving.

I have to admit that I honestly don't keep up with the latest improvements to the Eclipse IDE. The reason is simply practical. I work on making tools for doing research on IDEs. I would like my current project, Diver to be as compatible as is reasonable so that I can decrease the barrier to adoption. If I work with the "latest and greatest" version of Eclipse, I am just too tempted to use the "latest and greatest" features of Eclipse in my own tools, which means that people running the current release may not be able to work with what I am building. It's really just a matter of discipline.

But now that Helios is about to be released I've started to use it and I find it totally awesome. Ian Bull has been faithful to his top ten list of the newest and greatest features of Eclipse. I don't know if I will do the same, but I think I might report on the great things about Eclipse as I discover them.

So far, the number 1 through 10 greatest feature of Helios for me has been its new integration with the Eclipse Marketplace (and the Yoxos Marketplace catalog). I just recently had to change computers which meant that I also had to reinstall Eclipse. Normally, I would use Yoxos to create my own custom distribution of Eclipse, but I wanted to try Helios and all its cool new features. I use a number of 3rd party tools that aren't available in the standard Eclipse p2 repositories. For example, I work in academia which means that I write papers. I use LaTeX for writing papers, which is a surprisingly process to writing software. So, I use Eclipse to write text as well as software. It used to be total pain to search for my LaTeX plug-in, add its p2 repository to my list of "Available Software Sites", and install. That is no longer the case. Now, I can just load up my Eclipse Marketplace client and do a quick search for "LaTeX":

And there you go. No hassle, no fuss, I just install and that's it. It's beautiful.

As an aside (and a shameless plug), you can do the same thing with my project "Diver". Just load your Eclipse Marketplace client. A quick search for "reverse engineering" will point you straight to my favorite Eclipse tool ;-) :

Thursday, June 3, 2010

Eclipse Labs: To Migrate or Not to Migrate?

I'm really happy about the recent creation of the new Eclipse Labs. While I would absolutely love to have my Diver project as an official Eclipse project in incubation, I'm afraid that the process is too heavy-weight. I work in research which often requires a very malleable programming schedule... I don't think that I would be able to hit the release targets of Eclipse. Plus the IP process can be a little long. My friend and colleague Ian Bull was able to make his Zest project an official Eclipse project (it is now part of GMF/draw2D) while he was doing his Ph.D... he did have a little programing help from some of the other members of our lab ;-). I'm a Masters student, though: the program is too short to wait for my project to be approved by the committers.

So, Eclipse Labs is exactly perfect for what I need. It has the benefit of the Eclipse brand, but it doesn't carry the baggage of official Eclipse projects. It's too bad that it wasn't available six months ago when I was making Diver a tool open to the public. The best option I had at the time was SourceForge, which has been quite good, really.

So, now I'm left with the question of whether or not to migrate Diver to the Eclipse Labs hosting. I would really like Diver to have a closer tie to the Eclipse brand. But, I would lose a lot of work that was put into creating the project content for Diver over at SourceForge. Most specifically, I would lose are the hard work that I put into creating the web site. Google code has its wiki, which is OK, but it isn't as rich as what I've been able to make with my web page.

So: to migrate, or not to migrate? I don't know. Is there any real benefit?

Tuesday, May 25, 2010

New version of Diver available today!

Actually, it's been available for about a week, but I just haven't had time to blog about it. I've been busy with other work.

For those of you who don't know, Diver is a research project of the CHISEL group at the University of Victoria. We are trying to find better ways to help programmers develop, debug, and understand their software. Diver does this by bringing some techniques from reverse engineering into your Eclipse IDE. It can generate execution traces of your software which it uses to adjust the UI of your IDE to draw your attention to the features of your software that you are looking for. Diver is the product of several years of research and development, so it is feature-rich and robust. You can find out more about Diver at our website: http://diver.sf.net

OK, that's enough of the sell... I should let you know what you get with this new version of Diver.

Faster Tracing The tracer has been optimized. Tests indicate that it is between
2 and 4 times faster than version 0.1.0
Capture Screenshots You can now save screenshots of your sequence diagram. Select
the "camera" icon,
and you will be able to save an image of the main pane of the sequence
diagram as a PNG, JPG, or BMP.
Exporting Traces Diver is now able to export a complete trace to be shared with your
Importing Traces If you can export, you have to be able to import. Use the import
trace wizard to open a Diver Trace (dvt) file. The file will be imported
and analyzed just as if you ran the trace yourself.
Filtering Empty Threads Diver keeps a record for all threads in your traces, even if no events
occur in them. In previous versions, this could cause a lot of clutter
in the Program Traces view. Now, you can filter empty threads and
get rid of the clutter.
Marking Code Executed in Jar Files In previous versions, it wasn't possible to view executed
lines of code in .class files contained in Jars because of limitations
in Eclipse's Resource framework. So, I used Diver to figure out how the
JDT sets breakpoints in Jar files, and I was able to use what I found
to add markers to Jar files for use within Diver.
Toggle Combied Fragments and Loops If, for some reason (and I don't know what that would be :-) ),
you don't like seeing your sequence diagram decorated with source
code information, you can shut off combined fragments and loop compaction.

Exceptions If a method returns due to an exception, it is now indicated in
the sequence diagram.

Diver Research Diver is a research project at the University of Victoria. Diver
now makes it easier for you to help us with our research by logging
information about how you use it. You can automatically upload these
logs to our server, or you can shut off logging using the Workbench
Logger preference page

If you already have Diver installed, you should be able to get the new version just by updating your Eclipse. Otherwise, you can use our p2 repository site: https://diver.svn.sourceforge.net/svnroot/diver/Release.

I hope that this new version will be useful to everyone. I've put a lot of hard work into it. It's 100% open source, so its free to you. But, please remember that Diver is a research project, so help me out with my research. Use it for a while, and then fill out our survey. Also, this new version of Diver has a logging feature that will keep track of some of your interactions with Diver. It's all anonymous information, and you can turn it off at any time (using the Workbench Logger) preference page. We don't get the data unless you explicitly agree to send it to us (Diver will ask you permission before uploading). But, please participate. I have to graduate sometime ;-).

Anyway, happy coding!

Tuesday, February 2, 2010

The Perils of Multi-Platform Eclipse Development

Writing code for multiple platforms is a challenge. It not only increases the complexity of your code and tests, but it also increases the complexity of your build process. The build process is what I have had trouble with as of late. So, I'm logging my struggles so that I, and you, might have some place to go the next time multi-platform deployment causes headaches.

So, here is my problem. I have a plug-in that has a little bit of platform-specific code in it. The code works on both windows and linux, and I have precompiled binaries for them. In other words, the build of the (C++) native code doesn't need to be part of my build process. I do that separately just because it is easier. Setting up cross-compilers is a nightmare, and takes more time than I have.

So, here is what I do: I separate my native code into plug-in fragments for windows and for linux. After that, I have to go through all of my different manifests to make sure that P2 knows what to do with the the fragments. First, a p2.inf file has to be made for each fragment and the host plug-in, so that P2 knows the fragments are installable units that should be downloaded. OK, that's fine. Then, I have to the feature.xml for the feature that contains my plugins and fragments to make sure that I specify the supporting environments:

OK, then I have to set up my fragments to make sure that the platform filters match their target environment:

Platform filters are basically read as a functional notation, where sets of attributes are bound to operators. For example, (|(osgi.arch=x86)(osgi.arch=x86_64)) binds the selection of the attributes (osgi.arch=x86 and osgi.arch=x86_64) to the logical OR operator (|), so that it reads, "the osgi architecture can be either x86 or x86_64." So, I've got that down. I also had to set up the Bundle-NativeCode attribute in the manifest for the fragment so that my application knows where to find the binaries, but I'll get back to that later.

OK, now one might expect that everything is ready to export. After all, the binaries don't depend on any native Eclipse fragments (they don't reference SWT or anything). So, it seems that my fragments should be able to be packaged up just like anything else.

That's not the case however. When you run a build in PDE, and you put a platform filter on a fragment, Eclipse checks your current target platform and builds only for what you currently have. I, for example, have a machine with Windows XP on it, and that is the version of Eclipse that I downloaded. So, Eclipse will build only the windows fragment for me. There is no option to build for other platforms. If I use the Export Deployable Features wizard, this is what I get:

Even though my fragments don't have any dependencies on any other platform-specific code and all I really need is a jar with my binaries in it, the PDE says, "No! I can't build that for you." OK, that's fine. I know that it is possible to build for multiple platforms (the SWT guys do it), so I did some searching.

It turns out that what I need is something called the Eclipse Delta Pack. The Delta Pack is basically a bunch of fragments for different platforms which include native code for user interfaces and other i/o such as file manipulation. Getting the Delta Pack can transform your Eclipse workbench into a catch-all platform which targets everything that Eclipse can manage.

The unfortunate thing is that the Delta Pack isn't that easy to find. If you go to the Eclipse Download page, you won't find any reference to it. You have to go down to the bottom of the page, and find the Classic Eclipse distribution. And select Other Downloads:

Once there, you will want to get the release that you are targeting (probably somewhere in the 3.5.1 stream or later), and look to the left-hand menu. You will find the Delta Pack there:

Or, you can follow this link for the 3.5.1 drop.

When you download the Delta Pack, you have to unpack it. Make sure that you don't unpack it over your current Eclipse install. That could make things go bad. Instead, unpack it somewhere else and you can tell Eclipse to add it to your target platform.

To do that, go to your Eclipse Preferences, and find the page for Target Platform, select the running platform, and Add the Delta Pack to it as an Eclipse Install:

After that's done, you will have a new magic option in your Export Deployable Features wizard called Export for multiple platforms. That's exactly what I was looking for!

So, one would think at this point that one could simply export all of the features, upload them to a P2 repository, and release the new version. Not so fast. If you are me and you do that, you get this bug report.

It was silly of me. I shouldn't have assumed that the build would work flawlessly without testing. So, I checked my P2 repository, and low-and-behold, the Linux fragment was missing. That seemed strange. After all I did export for Linux. There's no rocket science here. It's just a pre-compiled binary after all.

So, after hours of fighting with trying different combinations of platform filters, and different ways of building, there were no warnings and no errors in the build. Unfortunately, there was also no Linux fragment. It just would not get created on my windows box. I was about to go and file a bug on P2, when I decided that maybe I should just try to get the build on Linux. So, I went through all the above steps again on a fresh install of Eclipse on a Linux machine. What I expected to happen was that it would build fine on the Linux machine, but instead of not having a Linux fragment, I would be left without a windows fragment.

That isn't what happened. I did get a windows fragment. What was missing? You guessed it: the Linux fragment! My Linux machine failed to build the Linux fragment. But, at least I got an error this time:

It looks like there was something about my Bundle-NativeCode property in the Linux fragment which the PDE build system didn't like. So, I went to investigate it.

I had used native code previously in an RCP toy application in which I was experimenting with OpenGL. When I looked up how to do it at that time, I learnt that I was supposed to use various keys such as processor= and os= to tell the plug-in when to use different compiled libraries. In fact, that is what I used in previous versions of Diver. When both the Windows and Linux binaries were in one fragment, it worked fine. But apparently, now that I had them in two fragments, that was very, very wrong:

I didn't really know what to do, though since this is what I had read was correct, and it was previously working. In fact it still works for the different Windows versions. Not for Linux, though. I don't know why. So, after another hour or so on Google, I learned about the key selection-filter which works in the same way that the Eclipse-PlatformFilter does. So, I copied my platform filter into the Bundle-NativeCode like this:

And it works! So, I learned a few things from this whole process:

  1. You Need The Delta Pack. Why you need it for all multi-platform builds, I don't know. In my example, all I needed was to package up pre-compiled binaries into jar files. There doesn't need to be any dependency resolution for that, but the PDE needs the Delta Pack to do the build. That is relatively painless, so I don't mind.

  2. Pay Attention to the Bundle-NativeCode Format. Eclipse won't necessarily tell you what is wrong. It might just fail to build a fragment or two without you even noticing. The safest way to get it to work just seems to be to copy your Eclipse-PlatformFilter directly into the Bundle-NativeCode property. At least you don't have to manage multiple different syntaxes that way.

Anyway, that is all for now. I hope this can be helpful to anyone who is doing builds for multiple platforms. It isn't an easy problem to solve, so there are bound to be pain points. This post will hopefully get you (and me) through them quickly next time.

Diver 0.1.0 Released (For Real This Time)

If anyone tried to download Diver 0.1.0 last week on their Linux boxes, you may have had a little trouble. It turns out that I didn't have everything set up correctly for a multi-platform build, and the Linux plugins weren't getting created. I'm going to blog about my experience in a minute, to keep it for posterity so that I can remember how to do it. Hopefully it will be useful for others as well. For now, here is a pretty picture to prove to you that things are good to go for you Linux users.

If you downloaded for Windows last week, I think that you should be fine. But, there won't be any harm in updating.

Wednesday, January 27, 2010

Diver 0.1.0 Released

There is a new version of Diver out, with a few new features which should make reverse engineering easier in Eclipse. A couple of highlights are the new "Notes" feature which can be accessed using Eclipse's Properties View, and the new Trace Search Page which helps you to find things in your traces. Diver is also a lot smaller in terms of its memory footprint now. For more information on what's new, check out the New To Diver 0.1.0 Page. To get Diver, just point your P2 "Available Software Sites" to https://diver.svn.sourceforge.net/svnroot/diver/Release. Diver works on Windows and Linux.

Some fancy pictures of the new features:

The new Diver Trace Search

Finding all the activations of methods that start with get*

Finding all personalized notes stored in a trace

Friday, January 22, 2010

Diver tutorial videos available

Reverse engineering tools have traditionally suffered from a tremendous lack of adoption. There are likely several reasons for this. One is that they tend to come out of academia, and they never really get past the prototype stage. That is a shame because a lot of the ideas that have been proposed are really good ones. Another reason that they aren't often adopted is because it can be difficult to understand how to use them. The tools can be very cumbersome or they aren't very well documented.

I've tried to overcome both of these issues with Diver so that people will be able to see its use and it can be widely adopted. I have to say, my motivations aren't entirely altruistic: I do have to get my thesis done :-).

I've written a lot of in-depth documentation for Diver. But, let's be honest, I'm not even interested in reading all of those pages of documentation. So, for those of us with a shorter attention span, I've published several tutorial videos to get you started.

By the way, I made these tutorials using Wink. I highly recommend it for anyone who has to make a tutorial of their software. It's very easy to use and it gives you pretty well all the features that you could ask for. I searched for free screen capture software for a long while. This is the best that I could come by. If you've got any other suggestions, let me know.

Thursday, January 21, 2010

Survey on Reverse Engineering

Though I'm really a coder at heart, I am a researcher by day. I've been working hard on the Diver project to try and figure out ways that reverse engineering can be used to aid coders like you and me. If you've downloaded it and used it, please fill out my survey.

If you haven't tried Diver yet, please try it. I'm really hopeful that it can help you Java developers out.