Making Tokens of a String

Tokens can be used where we want to break an application into tokens.

Tokens can be used where we want to break an application into tokens. We have to break a String into tokens as well as we will know how many tokens has been generated. That's what we are trying to do in this program. In the program a string is passed into a constructor of StringTokenizer class. StringTokenizer is a class in java.util.package. We are using while loop to generate tokens. The following methods have been used in this program.

countTokens(): It gives the number of tokens remaining in the string.

hasMoreTokens(): It gives true if more tokens are available, else false.

nextToken(): It gives the next token available in the string.

To break a string into tokens what we need first is to create a class named StringTokenizing. Inside this class we will declare our main method. Make an object of class StringTokenizer and pass one string inside the constructor which you want to break into tokens. By using the instance of StringTokenizer call the method countTokens() which gives the number of tokens remaining in the string. It is a method of StringTokenizer class. If the object have more tokens available then it will call method hasMoreTokens() and print the tokens by using nextToken().

The code of the program is given below:


import java.util.*;



public class StringTokenizing{

  public static void main(String[] args) {

  StringTokenizer stringTokenizer = new 

   StringTokenizer("You are tokenizing a string");

  System.out.println("The total no. of tokens 

   generated :  " + stringTokenizer.countTokens());

  while(stringTokenizer.hasMoreTokens()){

  System.out.println(stringTokenizer.nextToken());

  }

  }

}

Output of this program is given below:

C:\java>java StringTokenizing
The total no. of tokens generated : 5
You
are
tokenizing
a
string

Download this program:

Making Tokens of a String

Making Tokens of a String

Making Tokens of a String

Tutorials