Java: Be cautious while using URL equals method

In this post, I am going to explain why you should be cautious while using the equals method of URL class.

 

How URL equals method works?

As per the Java documentation, equals method documented like below.

 

That means, while checking two urls equality, Java performs a blocking DNS lookup operation to resolve hostname to IP address and then compare the ip addresses.

URL url1 = new URL("https://abc.test.com");
URL url2 = new URL("https://xyz.test.com");

Let's say both of these domains map to same IP address.

https://abc.test.com => 246.103.91.15
https://xyz.test.com => 246.103.91.15

Now when you compare url1 with url2 using equals method, it returns true. But both are different domains, I am expecting it to be false.

 

Apart from this, Since two url comparison using equals method perform a blocking DNS lookup operation to resolve the hostname, it takes lot of time for the comparison. In this case better to use URI (instead of URL) to compare two urls.

 

Let’s see it with an example.

 

URLEquals.java

package com.sample.app.net;

import java.net.MalformedURLException;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class URLEquals {

	public static List<URL> urls(List<String> urls) throws MalformedURLException {
		List<URL> urlList = new ArrayList<>();

		for (String url : urls) {
			urlList.add(new URL(url));
		}

		return urlList;
	}

	public static List<URI> uris(List<String> urls) throws MalformedURLException, URISyntaxException {
		List<URI> uriList = new ArrayList<>();

		for (String url : urls) {
			uriList.add(new URI(url));
		}

		return uriList;
	}

	public static void main(String args[]) throws URISyntaxException, MalformedURLException {
		boolean state = false;

		List<String> urlStrs = Arrays.asList("https://google.com", "https://youtube.com", "https://facebook.com",
				"https://www.wikipedia.org/", "https://twitter.com/", "https://www.quora.com/",
				"https://www.similarweb.com/", "https://hypestat.com/", "https://in.search.yahoo.com/",
				"https://www.amazon.com/", "https://www.walmart.com/", "https://javarevisited.blogspot.com/",
				"https://javarevisited.blogspot.com/", "https://javahungry.blogspot.com/",
				"https://javabypatel.blogspot.com/", "https://dnb-java.blogspot.com/p/home.html",
				"https://self-learning-java-tutorial.blogspot.com/", "https://javadevelopersguide.blogspot.com/",
				"https://javaknowhow.blogspot.com/", "https://javabasictoadvanced.blogspot.com/",
				"http://biemond.blogspot.com/", "https://javaworld-abhinav.blogspot.com/",
				"http://randomthoughtsonjavaprogramming.blogspot.com/", "https://javadiscover.blogspot.com/",
				"http://marxsoftware.blogspot.com/", "http://craftingjava.blogspot.com/",
				"https://www.java-success.com/", "https://javasolutionsguide.blogspot.com/",
				"https://ebclj.blogspot.com/", "https://www.tabnine.com/", "https://www.linkedin.com/",
				"https://github.com/", "https://www.overops.com/", "https://blog.javapapo.com/",
				"https://www.facebook.com/", "https://www.buggybread.com", "https://www.baeldung.com/",
				"https://www.onlyfullstack.com/", "https://apexapps.oracle.com/", "https://guava.dev/",
				"https://www.java67.com/", "https://blog.feedspot.com/", "https://javapapers.com/",
				"https://www.ideamotive.co/", "https://blogs.sap.com/", "https://www.eclipse.org/",
				"https://www.sololearn.com/", "https://brainly.in/", "https://www.javacodegeeks.com/",
				"https://freecomputerbooks.com/", "https://www.freecodecamp.org/", "https://en.wikipedia.org/");

		List<URL> urls = urls(urlStrs);

		long time1 = System.currentTimeMillis();
		for (URL url1 : urls) {
			for (URL url2 : urls) {

				if (url1.equals(url2)) {
					// System.out.println(url1 + " and " + url2 + " are equal ");
					state = true;// some dummy action
				}

			}
		}

		long time2 = System.currentTimeMillis();

		List<URI> uris = uris(urlStrs);

		long time3 = System.currentTimeMillis();

		for (URI uri1 : uris) {
			for (URI uri2 : uris) {

				if (uri1.equals(uri2)) {
					// System.out.println(uri1 + " and " + uri2 + " are equal ");
					state = true;
				}

			}
		}
		long time4 = System.currentTimeMillis();

		System.out.println("url comparison take " + (time2 - time1) + " milliseconds");
		System.out.println("uri comparison take " + (time4 - time3) + " milliseconds");
	}

}

Output

url comparison take 13478 milliseconds
uri comparison take 2 milliseconds

How URI equals method works?

Two uris are considered to be equals, if

a.   Their schemes must either both be undefined or else be equal without regard to case.

b.   Their fragments must either both be undefined or else be equal.

c.     For two opaque URIs to be considered equal, their scheme-specific parts must be equal.

d.   For two hierarchical URIs to be considered equal, their paths must be equal and their queries must either both be undefined or else be equal.  Their authorities must either both be undefined, or both be registry-based, or both be server-based.  If their authorities are defined and are registry-based, then they must be equal.  If their authorities are defined and are server-based, then their hosts must be equal without regard to case, their port numbers must be equal, and their user-information components must be equal.

 

 

Previous                                                    Next                                                    Home

Java: How to get the domain name from url?

URL stands for Uniform resource locator, it is a special type of uri which tells how to access the resource.

 

Example

https://self-learning-java-tutorial.blogspot.com/search?q=test

 

In the above example.

https -> specific scheme

self-learning-java-tutorial.blogspot.com -> represents the path

/search -> specifies the path

q=test -> specifies query parameter.

 

Example 2

https://www.self-learning-java-tutorial.blogspot.com/search?q=test

 

In the above example, url contains the string www after //.

 

We need to address both the examples while finding the domain name. Below snippet takes an url and return the domain name.

public static String getHostName(String url) throws URISyntaxException {
	URI uri = new URI(url);
	String hostComponent = uri.getHost();
	return hostComponent.startsWith("www.") ? hostComponent.substring(4) : hostComponent;
}

 

Find the below working application.

 

DomainNameDemo.java

 

package com.sample.app.net;

import java.net.URI;
import java.net.URISyntaxException;

public class DomainNameDemo {

	public static String getHostName(String url) throws URISyntaxException {
		URI uri = new URI(url);
		String hostComponent = uri.getHost();
		return hostComponent.startsWith("www.") ? hostComponent.substring(4) : hostComponent;
	}

	public static void main(String args[]) throws URISyntaxException {
		String url1 = "https://self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html";
		String url2 = "http://self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html";
		String url3 = "https://www.self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html";
		String url4 = "http://www.self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html";

		System.out.println("Host name in " + url1 + " is \t" + getHostName(url1));
		System.out.println("Host name in " + url2 + " is \t\t" + getHostName(url2));
		System.out.println("Host name in " + url3 + " is \t" + getHostName(url3));
		System.out.println("Host name in " + url4 + " is \t" + getHostName(url4));
	}

}

 

Output

Host name in https://self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html is 	self-learning-java-tutorial.blogspot.com
Host name in http://self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html is 		self-learning-java-tutorial.blogspot.com
Host name in https://www.self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html is 	self-learning-java-tutorial.blogspot.com
Host name in http://www.self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html is 	self-learning-java-tutorial.blogspot.com

 

 

Previous                                                    Next                                                    Home

URI vs URL

While working with web applications, you may come across the terms URI and URL. Most of the people think that both URI and URL are same, but actually not.

 

In this post, lets’ dig into it and try to understand the difference between URI and URL.

 

What is URI?

A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource.

 

Syntax of URI

scheme:[//authority]path[?query][#fragment]  

Examples

ftp://xyz.ab.com/text/text1808.txt
http://www.abc.xy.org/text/text2396.txt
mailto:ram.krishna@example.com
news:comp.abc.www.servers.unix
tel:+1-987-123-1212
telnet://192.168.2.255:80/

Components of URI syntax

a.   Schema: Every URI begins with a scheme name consist of a sequence of characters beginning with a letter and followed by any combination of letters, digits, plus ("+"), period ("."), or hyphen ("-"). For example http, https, ftp, telnet represent schemes.

b.   Authority: The authority component is preceded by a double slash ("//") and is terminated by the next slash ("/"), question mark ("?"), or number  sign ("#") character, or by the end of the URI. Authority component can contain port number (it is optional).

c.    Path: If a URI contains an authority component, then the path component must either be empty or begin with a slash ("/") character.

d.   Query: Query component is used to represent query parameters and it is preceded by a question mark(?). Query component is optional.

e.   Fragement: It is an optional component, preceded by a hash (#) symbol. The semantics of a fragment identifier are defined by the set of representations that might result from a retrieval action on the primary resource.

 

You can refer below link to get list of registered uri schemes.

http://www.iana.org/assignments/uri-schemes

 

 

What is URL?

Definition 1: URL stands for Uniform resource locator, it is a special type of uri which tells how to access the resource.

 

Definition 2: URL is used to find the location of a resource in the web. URL is a subset of URIs, in addition to identifying the resource, url provide a means of locating the resource by describing its primary access mechanism (e.g., its network "location").

 

In simple terms, URI is a superset of URLs, a URL is a URI, but a URI not necessarily be a URI.

What is URN?

URN stands for "Uniform Resource Name" is a Uniform Resource Identifier (URI) that uses the urn scheme.

 

Examples

 

urn:isbn:0451450523
urn:ietf:rfc:2648
urn:ISSN:0167-6423

 

URI vs URL vs URN

 

 

Reference

https://datatracker.ietf.org/doc/html/rfc3986

https://datatracker.ietf.org/doc/html/rfc1738

https://datatracker.ietf.org/doc/html/rfc2141

https://datatracker.ietf.org/doc/html/rfc3305

https://en.wikipedia.org/wiki/Uniform_Resource_Name

Previous                                                    Next                                                    Home

Java: How to read the content of a webpage or url?

In this post, I am going to show multiple programs/ways to read the content of a webpage or url in Java.

 

Approach 1: Using built-in HttpURLConnection class.

 

ReadDataFromUrlDemo1.java

package com.sample.app;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;

public class ReadDataFromUrlDemo1 {

	public static String getContentFromUrl(String urlToRead) throws Exception {

		URL url = new URL(urlToRead);
		HttpURLConnection httpUrlConnection = (HttpURLConnection) url.openConnection();
		httpUrlConnection.setRequestMethod("GET");

		StringBuilder result = new StringBuilder();

		try (InputStream inputStream = httpUrlConnection.getInputStream();
				InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
				BufferedReader reader = new BufferedReader(inputStreamReader)) {
			
			for (String line; (line = reader.readLine()) != null;) {
				result.append(line);
			}
			
		}
		return result.toString();
	}

	public static void main(String args[]) throws Exception {
		String urlToRead = "https://self-learning-java-tutorial.blogspot.com/2014/02/blog-post.html";
		String result = getContentFromUrl(urlToRead);
		System.out.println(result);
	}

}

 

Approach 2: Using Spring RestTemplate. One of the key advantages of Http client over HttpURLConnection is, it can handle URL redirects and proxy negotiations etc.,

 

ReadDataFromUrlDemo2.java

package com.sample.app;

import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;

public class ReadDataFromUrlDemo2 {

	private static final RestTemplate REST_TEMPLATE = new RestTemplate(getClientHttpRequestFactory());

	private static ClientHttpRequestFactory getClientHttpRequestFactory() {
		int timeout = 5000;
		RequestConfig config = RequestConfig.custom().setConnectTimeout(timeout).setConnectionRequestTimeout(timeout)
				.setSocketTimeout(timeout).build();
		CloseableHttpClient client = HttpClientBuilder.create().setDefaultRequestConfig(config).build();
		return new HttpComponentsClientHttpRequestFactory(client);
	}

	public static String getContentFromUrl(String urlToRead) throws Exception {
		ResponseEntity<String> responseEntity = REST_TEMPLATE.exchange(urlToRead, HttpMethod.GET, null, String.class);
		return responseEntity.getBody();

	}

	public static void main(String args[]) throws Exception {
		String urlToRead = "https://self-learning-java-tutorial.blogspot.com/2018/08/spring-framework.html";
		String result = getContentFromUrl(urlToRead);
		System.out.println(result);
	}
}

Dependencies used

<dependencies>
	<dependency>
		<groupId>org.apache.httpcomponents</groupId>
		<artifactId>httpclient</artifactId>
		<version>4.5.10</version>
	</dependency>

	<dependency>
		<groupId>org.springframework</groupId>
		<artifactId>spring-web</artifactId>
		<version>5.3.11</version>
	</dependency>

</dependencies>

Approach 3: Using apache http client.

 

ReadDataFromUrlDemo3.java

package com.sample.app;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;

import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public class ReadDataFromUrlDemo3 {

	public static String getContentFromUrl(String urlToRead) throws Exception {
		CloseableHttpClient httpclient = HttpClients.createDefault();
		HttpGet httpget = new HttpGet(urlToRead);
		CloseableHttpResponse response = httpclient.execute(httpget);
		HttpEntity entity = response.getEntity();

		StringBuilder result = new StringBuilder();

		try (InputStream inputStream = entity.getContent();
				InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
				BufferedReader reader = new BufferedReader(inputStreamReader)) {

			for (String line; (line = reader.readLine()) != null;) {
				result.append(line);
			}

		}
		return result.toString();

	}

	public static void main(String args[]) throws Exception {
		String urlToRead = "https://self-learning-java-tutorial.blogspot.com/2018/08/spring-framework.html";
		String result = getContentFromUrl(urlToRead);
		System.out.println(result);
	}
}

  

Previous                                                    Next                                                    Home