Sprint EVDO Card and Linux

Recently I upgraded to a new phone, and while I was at it I bought one of those mobile broadband cards for my laptop. I had seen enough blogs and e-mail posts on various mailing lists to know that it would probably work. My carrier is Sprint, so I naturally received an EVDO card from them.

Upon my purchase, I noticed from Sprint's own instructions for Linux that you couldn't activate the card on Linux. That could only be done from a Mac or Windows based PC. Well, I happen to have Macs in my house along with my Linux systems, so I went to a Mac, and activated the card. It was very simple, as it automatically ran a Mac version of some software built into the card, and it activated, and connected without issue. So, then I wanted to get it configured to work on my Linux laptop. That's where the fun began.

I'm running Fedora 9, but its also a 64-bit AMD system, and I am running the x86_64 version of Fedora on it. When I plugged in the card, I could see that it was recognized by the OS, and the USB product id and vendor id would display. So, that was promising. I wanted to do this the easiest way possible, so I thought I would just go into the Edit Connections... menu item in the Network Manager applet, and configure from there. So, I did that, and I selected the broadband tab, and clicked on Add. Well, nothing happened. Nothing at all. After a while of digging around, I decided to run the same connection application from the command-line. Sometimes applications will spew out errors to standard error that never are displayed in the GUI, so I thought that I might learn something.

So, I ran nm-connection-editor from the command-line, instead of from the Network Manager applet, and what do you know, an error message spews out when I try to add a broadband connection. Here it is:

** (nm-connection-editor:4208): WARNING **: create_new_connection_for_type:
unhandled connection type 'NMSettingCdma'

** (nm-connection-editor:4208): WARNING **: Can't add new connection of type
'cdma'

After some searching, I found out that the x86_64 version of Network Manager actually didn't have the broadband code implemented in the version that was in Fedora 9. So, I opened a bugzilla on it, and I ended up getting a response saying it was fixed in a version that was in Fedora 9 updates testing.

So, after getting the appropriate link to where these packages where, I decided to install them and try it out. Well, I must say I couldn't be happier. It worked flawlessly, and it detected the correct type of card, and filled in all the information for it, and I didn't change anything other than the default connection name that the wizard had filled in. At first, I thought no one has ever documented adding a mobile broadband card through the Network Manager interface, so I would do that. After thinking about it though, it was so easy, that there really isn't any value in documenting it.

This is truly the way things should be. Plug in your device, right-click and select Edit Connections..., then select the Mobile Broadband tab, click Add, and click OK! It couldn't get much easier than that! Whoops, I guess I just documented it!

So, if you haven't used the Network Manager for doing this, its certainly a lot easier than setting up PPP scripts that I have seen documented by many people. Give it a try, you'll like it.

JVM Performance Tuning

Last week was JBoss World, and it was exciting to be a part of it. I gave a presentation on performance tuning our Enterprise Application Platform or EAP, and it was packed. In fact, people were sitting on the floor in pretty much all available space. What struck me about the presentation, and much of the discussion I had with individuals afterwards, is that JVM tuning is a big topic. So, I thought I would share some of what I learned over the past couple of months as I was preparing for my presentation.

In preparing for my presentation, I wrote an EJB 3 application, wrote a load test for it, and applied optimizations to various configuration parameters within the EAP, the JVM and the operating system. In particular, one JVM and OS setting really made a huge difference in throughput, and its something that I wanted to share here.

When using a 64-bit OS, in my case Fedora 8 and RHEL 5.1, I wanted to investigate the usage of large page memory support, or HugeTLB as its referred to within the Linux kernel. What I found was very scarce documentation around using this, and that that documentation was incomplete to actually make it work. What I also found, is it makes a huge difference in overall throughput and response times of an application, when using heap sizes above 2GB.

So, without further ado, let's dive into how to set this up. These instructions are for Linux, specifically for Fedora 8 and RHEL 5.1, but the results should be generally applicable to any 64-bit OS and 64-bit JVM that supports large page memory (which all the proprietary UNIX's do, and I found an MSDN article describing how to use this on 64-bit Windows).

You must have root access for these settings. First, you need to set the kernel parameter for shared memory to be at least as big as you need for the amount of memory you want to set aside for the JVM to use as large page memory. Personally, I like to just set it to the maximum amount of memory in the server, so I can play with different heap sizes without having to adjust this every time. You set this by putting the following entry into /etc/sysctl.conf:

kernel.shmmax = n

where n is the number of bytes. So, if you have a server with 8GB of RAM, then you would set it to 8589934592, or 1024*1024*1024*8, which is 8GB.

Second, you need to set a virtual memory kernel parameter to tell the OS how many large memory pages you want to set aside. You set this by putting the following entry into /etc/sysctl.conf:

vm.nr_hugepages = n

where n is the number of pages, based on the page size listed in /proc/meminfo. If you cat /proc/meminfo you will see the large page size of your particular system. This varies depending on the architecture of the system. Mine, is an old Opteron system, and it has a page size of 2048 KB, as shown by the following line in /proc/meminfo:

Hugepagesize: 2048 kB

So, I wanted to set this to 6GB. I set the parameter to 3072, which is (1024*1024*1024*6)/(1024*1024*2). Which is 6GB divided by 2MB, since 2048 KB is 2MB.

After this, you need to set another virtual memory parameter, to give permission for your process to access the shared memory segment. In /etc/group, I created a new group, called hugetlb, you can call it whatever you like, as long as it doesn't collide with any other group name, and it had a value of 501 on my system (it will vary, depending on whether you use the GUI tool, like I did, or whether you do it at the command line, and vary depending on what groups you already have defined). You put that group id in /etc/sysctl.conf as follows:

vm.hugetlb_shm_group = gid

where gid, in my case was 501. You also add that group to whatever your user id is that the JVM will be running as. In my case this was a user called jboss.

Now, that concludes the kernel parameter setup, but there is still one more OS setting, which changes the users security permissions to allow the user to use the memlock system call, to access the shared memory. Large page shared memory is locked into memory, and cannot be swapped to disk. Another major advantage to using large page memory. Having your heap space swapped to disk can be catastrophic for an application. So, you set this parameter in /etc/security/limits.conf as follows:

jboss soft memlock n
jboss hard memlock n

where n is equal to the number of huge pages, set in vm.nr_hugepages, times the page size from /proc/meminfo, which in my example would be, 3072*2048 = 6291456. This concludes the OS setup, and now we can actually configure the JVM.

The JVM parameter for the Sun JVM is -XX:+UseLargePages (for BEA JRocket its -XXlargePages, and for IBM's JVM its -lp). If you have everything setup correctly, then you should be able to look at /proc/meminfo and see that the large pages are being used after starting up the JVM.

A couple of additional caveats and warnings. First, you can dynamically have the kernel settings take affect by using sysctl -p. In most cases, if the server has been running for almost any length of time, you may not get all the pages you requested, because large pages requires contiguous memory. You may have to reboot to have the settings take affect. Second, when you allocate this memory, it is removed from the general memory pool and is not accessible to applications that don't have explicit support for large page memory, and are configured to access it. So, what kind of results can you expect?

Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-thirds, and GC time was also cut down by a similar percentage (each individual GC event was much shorter in duration). Of course, your mileage will vary, but this one optimization is worth looking at for any high throughput application.

Good luck!

The State of ATI Graphics for Linux

Over a year ago, my laptop that I use for work, had a meltdown, and would no longer work. I had to get back to work as quickly as possible, so I went to Best Buy to purchase a new one. In that situation, I made a decision to purchase a laptop with the ATI XPress 200M PCI-E integrated graphics chip. Now, at the time I was very hesitant, because I had read many reports of problems with all ATI discreet and integrated graphics under Linux. Well, it turns out that the reports were well founded, and I had lots of problems trying to use the proprietary graphics drivers from ATI. They were buggy, slow, and my laptops suspend and hibernate functions simply didn't work at all. I had none of these problems with my old laptop which used an Nvidia chip set. Needless to say, this was frustrating, and the open source driver, which I would prefer, but ATI doesn't release their specifications, so I couldn't actually use the majority of the features of the graphics chip. For example, no 3D support, no support for proper widescreen resolutions, and generally poor performance. Again, these issues with the open source drivers aren't the fault of the developers, but still falls squarely on ATI as well. So, what to do?

At this point, I used whatever seemed to work best at the time, and many times found myself switching between the proprietary driver and the open source driver, depending on which one at the time seemed to work the best. Needless to say, this was a real pain to deal with, but I really didn't have much of a choice in the matter. I still needed to make a living, and didn't want to take the laptop back and try another one, just to find other issues. So, I stayed patient, and kept testing each successive release of the available drivers. Then a breakthrough occurred.

First, a couple of months ago, ATI released a new proprietary driver that was based on a new code base, and I have to say, the performance is impressive. While it didn't address all of my issues, at least I had a stable driver that actually had good graphics performance. Many applications, that were really frustrating to use, suddenly became responsive and a joy to use. Whew! Now, this wasn't the end of the problems, but it certainly was a new beginning that had a lot of promise.

Now, the latest release of the driver, finally has my laptop usable in all situations. After reading the release notes for the latest release, I noticed that suspend and hibernate fixes were included, which had me intrigued. Maybe, I would finally be able to use my laptop without power for longer periods of time? Well, I installed the new release, and after seeing that everything was still stable for all my daily activities, I decided to test suspend/resume and hibernate functions. Well, I have very good news to report. Both functions work as expected! I couldn't be more happy at this point. I have a fully functional laptop where all the graphics features work, and I can use it with or without power in confidence. That's not the entire story either.

It turns out, that along with this transition to the new code base, that ATI has also started to release the specifications for their newer graphics architecture. While, that will not impact me, it certainly is a great step in the right direction. I actually wish that they would simply stop the proprietary driver, and just work in conjunction with the community to produce and support great open source drivers, but at least its a step in the right direction.

While I still have some reservations about using ATI products under Linux, the progress lately has me thinking that ATI products should be something I evaluate when making my next purchase.

Greater than 4GB files on an External USB Hard Drive

Several months ago, I purchased a Western Digital USB/Firewire external hard drive to backup my laptop's home directory. Considering that I was using it with Linux, specifically Fedora Core 6, I wasn't sure how things were going to work.

After plugging it in, and attaching it via the USB 2.0 cable, it mounted and was presented on the GNOME desktop, and I could browse the contents of the disk without issue. Trying to keep things simple, I merely used tar and created a gzipped tar of my home directory, making sure to preserve all the permissions of the files with the following command:

tar -czpf /tmp/[file name with date].tar.gz /home/[my home directory]

This works quite well, but it presented me with my very first issue. My home directory is quite large, and the very first tar file I created was larger than 4 GB, so I couldn't write it to the external drive. It couldn't be written for the simple reason that the drive was using the FAT file system, and it doesn't support file sizes larger than the 32-bit maximum of 4 GB.

So, I looked through my home directory, and I found some obvious culprits to my size problem, and deleted those files, because I no longer needed them. Mostly it was old ISO images, that I had burned to CD long ago, and didn't need anymore. Okay, problem solved right?

Well, not quite. This worked for several months, but I was still dangerously close to the 4 GB limit. Eventually I spilled over the limit, and really couldn't delete files to get back under it.

With this in mind, I decided to see if I could change the file system to one that supported files larger than 4 GB. Considering that I am only using this with Linux, cross platform compatibility was not an issue for me, so the obvious choice was to use the ext3 file system from Linux. This would give me the large file support I needed, and also be more reliable, as ext3 is more robust than FAT, and it supports journaling, so there is significantly less risk to losing data.

During my investigation of making this change, I found nothing but individuals having problems with trying to do this. Many individuals had even rendered their drives unusable. Considering this, I took a step back and wondered whether I should try this, or see if I could think of another resolution.

I really couldn't think of a better way to deal with this problem, and I wanted to keep things simple, so I went ahead and tried to make the file system change, and here is the procedure I used.

• First, I moved all my backups of my home directory that were currently on the drive, and copied them to /tmp on my laptop.
• Second, I fired up GParted, considering that it is a graphical partitioning tool, that also will format partitions. This proved to be an excellent choice, because it helped me to avoid one pitfall.
• Considering that the drive was plugged into the USB port, and mounted under /media/My Book, GParted would not let me format the drive until I unmounted it.
• I used GParted to unmount the drive, and then I selected from the menu "Format to->ext3".
• I watched as it automatically changed the partition type to the correct one, and then formatted the partition with the ext3 file system.
• It completed with no issues, but here is where one of the problems reared its ugly head.
• After formatting, the drive would no longer auto mount, and show itself on the desktop. I could manually mount it with the mount command, and it was working. I even wrote some files to it just to make sure everything was fine, and it was.
• The guys on the Fedora Core mailing list were most helpful with this problem.
• As it turns out, I needed to label the new file system with the e2label utility, which I did with the following command:
• e2label /dev/sdc1 "My Book"
• Finally, I moved the backups I put in /tmp back to the drive with the new file system.
  

After, these simple steps, I had a newly formatted external USB hard drive that I could write files larger than 4 GB to, without issues. It would auto mount, just the way it did when it was a FAT file system, and I now have some very large backups on it, and didn't have to change my very simple backup procedure.