Module dev.civl.abc
Package dev.civl.abc.token.IF

Interface CivlcTokenSequence

public interface CivlcTokenSequence

Represents a finite sequence of CivlcToken. Methods are provided to access the sequence in a variety of convenient ways. Most importantly, method newSource() returns a CivlcTokenSource, which can be used to construct an ANTLR TokenStream, which can in turn be used as the input to an ANTLR parser.

The tokens are never duplicated by any of these methods, so a modification to a token will be reflected in any subsequence method calls to access that token.

This sequence representation is best use for relatively short sequences, such as the tokens comprising a single pragma. It is not recommended for representing an entire translation unit.

This object will also include an addition "EOF" token. That token is not part of the sequence, but it is used by method newSource(). That method will in effect append an infinite sequence of EOF tokens to the end of the sequence.

  • Method Details

    • getEOFToken

      CivlcToken getEOFToken()
      Returns the extra EOF token which is used by method newSource().
      Returns:
      the extra EOF token

    • newSource

      CivlcTokenSource newSource()

      Returns a new CivlcTokenSource which produces the tokens in this sequence, in order, with an "EOF" token appended. Since a token source is basically infinite, the final EOF token is stuttered, i.e., repeated calls to TokenSource.nextToken() will continue to produce the EOF token after the EOF token has been reached.

      This source may be used to construct a TokenStream and then used as the input to a parser.

      Returns:
      a new CivlcTokenSource yielding the elements of this sequence.

    • getTokens

      CivlcToken[] getTokens()
      Returns the tokens as an array. Whether or not this array can be modified depends on the implementation. The array does not include the extra "EOF" token. The length of this array is the same as the number returned by size().
      Returns:
      the tokens as an array

    • size

      int size()
      Returns the number of tokens in the sequence. This number does not include the extra "EOF" token.
      Returns:
      the number of tokens in the sequence