When tokenizing a string, the delimiters are ignored by default.

Java: Tokenizing Delimiters

Purpose

When tokenizing a string, the delimiters are ignored by default. However, it may be useful to get them returned as tokens too. Each delimiter will be returned as a one-character string.

Constructor

Specify that you want the delimiters returned by specifying true in the third parameter of the constructor.

StringTokenizer st = new StringTokenizer(s, d, true);

Creates a StringTokenizer for the String s using delimiters from the String d

Example: Tokenizing an expression

Here is an example method that tokenizes a string, making all delimiters except whitespace into tokens.

 private String tokenizeEx(String expression, String delims) {
    StringBuffer sb = new StringBuffer(100);
    StringTokenizer scanner;    
    scanner = new StringTokenizer(expression, delims, true);
    
    while (scanner.hasMoreTokens()) {
       String aWord = scanner.nextToken();
       if (!Character.isWhitespace(aWord.charAt(0))) {
          sb.append(aWord + "\n");
       }
    }
    return sb.toString();
 }//end tokenizeEx

Example Applet

The following applet uses the above method. TokenizeExpr.java. Unfortunately, most browsers don't run Java 2 applets correctly yet, so it's doubtful that you can see it working here. You may have to compile and run it yourself.

Related Tags for Tokenizing Delimiters:
c,  string,  ant,  constructor,  struct,  get,  const,  char,  default,  new,  limit,  parameter,  return,  using,  oo,  character,  create,  if,  for,  cte,  ignore,  tokenizing,  to,  ram,  token,  each,  tokenizer,  pos,  ci,  delimiter,  e,  il,  it,  li,  mit,  use,  ul,  pe,  im,  from,  fault,  in,  no,  tokenize,  as,  m,  nt,  par,  tr,  tokens,  wan,  os,  os,  es,  spec,  true,  em,  me,  how,  param,  stringtokenizer,  tor,  when,  s,  urn,  sp,  del,  ctor,  at,  rac,  k,  ir,  ha,  ll,  meter,  ea,  ar,  act,  cons,  str,  want,  z,  s,  s,  ri,  ring,  rd,  th,  st,  hat,  cto,  efault,  miter,  mi,  on,  om,  o,  nor