All Packages Class Hierarchy This Package Previous Next Index
Class sdsu.util.SimpleTokenizer
java.lang.Object
|
+----sdsu.util.SimpleTokenizer
- public class SimpleTokenizer
- extends Object
This class performs some simple parsing of strings or streams.
The input is a sequence of ascii characters. The sequence is
divided into tokens, whitespace, and comments.
Comments start with the comment character and continue to the next
newline (\n ) character. Comments are removed from the input
characters and not returned as part of a token.
A token is string from the current location to the next separator
or whitespace character. Characters defined as whitespace
(tab, newline, and space default values ) help delineate tokens
but are not part of tokens. That is whitespace characters are
removed after finding a token. If a token must contain whitespace
character, a possible separator, or comment character,
the token must be placed between two quote characters. A quoted
token can contain a quote character.
- Version:
- 1.2 2 June 1998
- Author:
- Roger Whitney
(whitney@cs.sdsu.edu)
- See Also:
- Stringizer, TokenCharacters
-
SimpleTokenizer(InputStream)
- Create a SimpleTokenizer on tokenSource with default settings
-
SimpleTokenizer(InputStream, TokenCharacters)
- Create a SimpleTokenizer on tokenSource
-
SimpleTokenizer(Reader)
- Create a SimpleTokenizer on tokenSource with default settings
-
SimpleTokenizer(Reader, TokenCharacters)
- Create a SimpleTokenizer on tokenSource
-
SimpleTokenizer(String)
- Create a SimpleTokenizer on string with default settings
-
SimpleTokenizer(String, TokenCharacters)
- Create a SimpleTokenizer on string
-
hasMoreElements()
- Returns true if not at end of source stream or source string
-
hasMoreTokens()
- Returns true if not at end of source stream or source string
-
nextToken()
- Returns string containing all characters up to the given
separator, unquoted whitespace, or EOF
if the separator is not found.
-
nextToken(String)
- Returns string containing all characters up to the given
separator, unquoted whitespace, or EOF
if the separator is not found.
-
separator()
- Returns the separator found by the last call to nextToken
SimpleTokenizer
public SimpleTokenizer(String parsable)
- Create a SimpleTokenizer on string with default settings
SimpleTokenizer
public SimpleTokenizer(String parsable,
TokenCharacters charTable)
- Create a SimpleTokenizer on string
- Parameters:
- commentChar - character used to indicate start of a comment
- quoteChar - character used to quote a string containing
special characters
- whitespace - characters used for whitespace. Use null or
empty string for no whitespace characters
SimpleTokenizer
public SimpleTokenizer(InputStream tokenSource)
- Create a SimpleTokenizer on tokenSource with default settings
SimpleTokenizer
public SimpleTokenizer(Reader tokenSource)
- Create a SimpleTokenizer on tokenSource with default settings
SimpleTokenizer
public SimpleTokenizer(InputStream tokenSource,
TokenCharacters charTable)
- Create a SimpleTokenizer on tokenSource
- Parameters:
- commentChar - character used to indicate start of a comment
- beginQuoteChar - character used to start a quote of a
string containing special characters
- endQuoteChar - character used to end a quote of a
string containing special characters
- whitespace - characters used for whitespace. Use null
or empty string
for no whitespace characters
SimpleTokenizer
public SimpleTokenizer(Reader tokenSource,
TokenCharacters charTable)
- Create a SimpleTokenizer on tokenSource
- Parameters:
- commentChar - character used to indicate start
of a comment
- beginQuoteChar - character used to start a
quote of a string containing special characters
- endQuoteChar - character used to end a quote
of a string containing special characters
- whitespace - characters used for whitespace.
Use null or empty string for no whitespace characters
hasMoreTokens
public boolean hasMoreTokens()
- Returns true if not at end of source stream or source string
hasMoreElements
public boolean hasMoreElements()
- Returns true if not at end of source stream or source string
separator
public char separator()
- Returns the separator found by the last call to nextToken
nextToken
public String nextToken(String newSeparators) throws IOException
- Returns string containing all characters up to the given
separator, unquoted whitespace, or EOF
if the separator is not found. The separator is removed
from the stream, but not returned as part of token.
- Parameters:
- separator - can be any character except the current
comment or quote character
- Throws: IOException
- If separator or EOF does not
follow this token
nextToken
public String nextToken() throws IOException
- Returns string containing all characters up to the given
separator, unquoted whitespace, or EOF
if the separator is not found. The separator is removed
from the stream, but not returned as part of token.
- Parameters:
- separator - set of characters to be used as separator
after token. Can be any non-null or nonempty string of
characters except the current comment or quote character
- Throws: IOException
- If separator or EOF does not follow
this token
All Packages Class Hierarchy This Package Previous Next Index