HTTPClient FAQ

Contents

• General
  • How do I get rid of the Cookie-Popup
  • Will the HTTPClient work with JDK 1.3? (1.2, 1.1)
  • Dialog box doesn't appear or hangs
  • Displaying images and playing sound
  • File upload using multipart/form-data
  • UnknownHostException: http://www.foo.bar/x
  • https support
  • When will V0.4 be out?
  • What are the licensing conditions?
  • I'm trying to emulate Netscape/Exploder/whatever but I'm getting different results
  • I'm trying to POST to a form but I'm not getting the expected results
  • The site is saying "Your browser doesn't support cookies"
  • How do I simulate multiple independent clients or users?
• Applet specific
  • SecurityException: properties
  • SecurityException: thread
• Application specific
  • Application doesn't exit

General

Every time the server tries to set a cookie I get a pop-window asking whether I want to accept the cookie. How do I get rid of it?

If you just want to accept all cookies blindly then set the cookie-policy-handler to null:

	CookieModule.setCookiePolicyHandler(null); 

See the advanced_info doc for more information.

Will the HTTPClient work with JDK 1.3? (or 1.1, or 1.2)

The HTTPClient will work fine with JDK 1.1 or later, including JDK 1.2 and JDK 1.3. Only JDK 1.0.2 is not supported anymore.

All goes fine, until I try to do a request to a site that requires authorization or tries to set cookies. The dialog box either doesn't appear, or it appears but then everything hangs.

What you are probably doing is issuing the request from inside an AWT event handler. Don't! The problem is that while you're in an event handler no other events can get processed, including any event for the popup. This leads to a deadlock: no events can be processed till the popup has been closed again (thereby allowing the request to finish and to return to the caller), and the popup won't do anything unless it gets the corresponding event (e.g. a keyboard or mouse event).

There are various solutions to this:

1. Don't do any request from inside an event handler. This can be achieved by either starting a new thread which does the request, or by setting a flag and having the main thread do it (this is the way it's done in the Example Applet).
2. Install your own authorization and cookie policy handlers which don't use the AWT (see AuthorizationInfo.setAuthHandler and CookieModule.setCookiePolicyHandler). However, I recommend the previous solution, as in general it is bad practice to do any sort of extended work inside an event handler (i.e. the code in the handler should execute quickly and return so that other events can be handled).
3. If the site requires authorization then make the necessary authorization info available to the HTTPClient beforehand; see Authorization Handling for more info.
4. Use HTTPConnection.setAllowUserInteraction() or HTTPConnection.setDefaultAllowUserInteraction() to prevent the popups from being used. Note that this means that cookies will be silently accepted and that authorization info must be provided via the addXXXAuthorization() methods (see previous point).

How can I display an image or play an audio file retrieved via the HTTPClient? Do I have to write my own ContentHandler's?

You certainly could write your own ContentHandler's, but there is an easier way (however, one solution relies on undocumented sun classes):

Sound (.au)

This tip is from a JavaWorld article by Chong Ser Wah. Assuming resp is the HTTPResponse, use the following construct:

	    import sun.audio.AudioPlayer;
	    import sun.audio.AudioStream;

	    AudioStream as = new AudioStream(resp.getStream());
	    AudioPlayer.player.start(as); 

Images (gif and jpeg)

Assuming resp is the HTTPResponse, use the following:

	    import java.awt.Toolkit;

	    Image img = ToolKit.getDefaultToolkit().createImage(resp.getData()); 

I would like to upload files using POST and the multipart/form-data encoding just like certain browsers do - how do I do this with the HTTPClient?

See Codecs.mpFormDataEncode().

I'm having trouble creating a connection. All I get is

        java.net.UnknownHostException: http://www.foo.bar/x 

The code I use looks like:

        con = new HTTPConnection("http://www.foo.bar"); 

You are passing the wrong data to that particular constructor - it expects a host name, not a full url. Either use

        con = new HTTPConnection("www.foo.bar"); 

or use something like

        con = new HTTPConnection(new URL("http://www.foo.bar")); 

.

Does HTTPClient support HTTPS (i.e. HTTP on top of SSL)?

No, not yet. This is on the top of the list of things to do for version 0.4. The only real problem is finding a suitable SSL implementation. However, see the https support page for a list of patches available for various commercial SSL packages.

When will V0.4 be available?

I don't know. I'm fairly busy with a job and helping to renovate a house, so it'll definitely be a few months more.

May I use the HTTPClient in a commercial product? What are the conditions? How much does it cost?

The HTTPClient is covered by the GNU Lesser General Public License (LGPL). Note that this is not the same as the GPL, which is quite a bit more restrictive. One of the main differences is that the copyleft only applies to modifications of the library code itself, not to any product using the HTTPClient or to modules and handlers you write for the HTTPClient.

Here is a summary of the most important points:

• You may use the HTTPClient in any product you wish, whether commercial or otherwise. This product may be put under any license you wish (i.e. the HTTPClient's license is not infectious), and no source for the product need be distributed.
• The above liberties also apply to any modules and handlers you may have written for the HTTPClient.
• The HTTPClient itself is free for all uses, and will stay so. Others (including you) may however redistribute the HTTPClient too and charge for the cost of distribution.
• You may package the HTTPClient with your product, or you may even redistribute the HTTPClient by itself, provided you either distribute the complete HTTPClient package (including the source to it) or, if distributed over the net, provide pointers to where the source and the license to the HTTPClient can be obtained.
• You may make any modifications to the HTTPClient that you wish. However, if you distribute the modified HTTPClient then you must put those modifications under LGPL too and distribute them at no cost (or at the cost of distribution). This is the only infectious part of the license. Again, to restate: this applies only to modifications of the HTTPClient source itself.

I'm trying to emulate Netscape/Exploder/whatever but I'm getting different results

Here are a few things that may be causing problems:

• The site may be expecting certain browser types, i.e it may be looking at the User-Agent field. You can try setting the User-Agent header with something like the following:

              con.setDefaultHeaders(
                  new NVPair[] { new NVPair("User-Agent", "Mozilla/4.5") });

  (do this before you send the request, of course).
• The site may be expecting a Referer header containing a valid url. Try setting that header appropriately.
• The site may be requiring cookies. Try accepting any cookies it sends.

I'm trying to POST a form but I'm not getting the expected results

Any number of things may be happening here too. But first, check that you are not forgetting any data. Many forms have hidden fields (<INPUT TYPE=HIDDEN NAME=... VALUE=...>). If you are taking an html form and trying to do the same request as a browser would then make sure your sending along the info from those fields too (the fields may be anywhere on the page, so look closely). Also, take a look at the Emulating Forms doc. If that isn't the problem, see above.

The site is saying "Your browser doesn't support cookies"

yahoo.com and similarly broken sites just look at the User-Agent header to determine if the client supports cookies or not. Therefore, just set the header to something used by Netscape or Exploder. Example:

    con.setDefaultHeaders(
            new NVPair[] { new NVPair("User-Agent", "Mozilla/4.5") });

(do this before you send the request, of course).

How do I simulate multiple independent clients or users?

See the section on Contexts in the advanced-info doc.

Applet specific

I keep getting the message "#Security Exception: properties" (Netscape 3.x), "#Security Exception: checkpropsaccess.key" (Netscape 4.x), or "com.ms.security.SecurityExceptionEx[HTTPClient/HTTPConnection.<clinit>]: Unable to access system property:" (Internet Exploder) in the Java Console every time my Applet starts.

These can be ignored. What is happening is that the static initializer of the HTTPConnection class tries to read a number of properties and this results in a SecurityException (in an Applet). These exceptions are caught inside the initializer, but Netscape's AppletSecurityManager and Exploder's StandardSecurityManager print the above messages before throwing each exception.

I keep getting the message "#Security Exception: thread" in the Java Console.

This one is a little trickier. I assume the HTTPClient is being called from an AWT event handler? Then what is happening is the following. To keep a connection from staying open indefinitely when using persistent connections, the HTTPClient uses a timeout thread to close the connection if it's been idle for more than 10 seconds. To prevent applications from hanging at exit this thread is made a daemon thread when created (using setDaemon(true)). Furthermore, the thread is created when sending the first request, and is therefore created in the context of whatever thread is calling HTTPClient. The problem now is that the AWT event handler runs in thread of its own which belongs to the main thread group java.lang.ThreadGroup, but applets are only allowed to manipulate threads in the AppletThreadGroup (this includes stop()'ing a thread or doing a setDaemon()). Now if the HTTPClient is called from an event handler the timeout thread is created belonging to the main thread group, and any attempt at modifying it will therefore result in a security exception.

This message can actually be ignored too, as it doesn't matter whether the thread is a daemon or not in applets and the security exception is caught internally. However, it's not good practice to call HTTPClient from an event handler, as potentially long running stuff should be done in the main thread or a thread of its own (otherwise you lock up the event handling during that time). For an example of how to let the main thread do things see the simple Example Applet.

Application specific

My application reaches the end, and then instead of exiting just hangs.

What is probably happening is that there are some (non daemon) threads still around that haven't exited. An application won't exit until all (non daemon) threads are dead. Here is one reason why a thread might still be around (apart from any you might have started and not stopped yourself):

1. The AWT uses a number of (non daemon) threads which are never killed. If the authorization popup or the cookie handler popup appeared then these AWT threads were started and will therefore still be around at program exit time. In this case you need an explicit "System.exit(0)" at the end of your program.