Compiler course. Chapter 3 Lexical Analysis


 Eustacia Jean Barton
 3 years ago
 Views:
Transcription
1 Compiler course Chapter 3 Lexical Analysis 1 A. A. Pourhaji Kazem, Spring 2009
2 Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator 2 A. A. Pourhaji Kazem, Spring 2009
3 The role of lexical analyzer Source program Lexical Analyzer token getnexttoken Parser To semantic analysis Symbol table 3 A. A. Pourhaji Kazem, Spring 2009
4 Why to separate Lexical analysis and parsing 1. Simplicity of design 2. Improving compiler efficiency 3. Enhancing compiler portability 4 A. A. Pourhaji Kazem, Spring 2009
5 Tokens, Patterns and Lexemes A token is a pair, a token name and an optional token value A pattern is a description of the form that the lexemes of a token may take A lexeme is a sequence of characters in the source program that matches the pattern for a token 5 A. A. Pourhaji Kazem, Spring 2009
6 Example Token Informal description Sample lexemes if else comparison id number literal Characters i, f Characters e, l, s, e < or > or <= or >= or == or!= Letter followed by letter and digits Any numeric constant Anything but sorrounded by if else <=,!= pi, score, D , 0, 6.02e23 core dumped printf( total = %d\n, score); 6 A. A. Pourhaji Kazem, Spring 2009
7 Attributes for tokens E = M * C ** 2 <id, pointer to symbol table entry for E> <assignop> <id, pointer to symbol table entry for M> <multop> <id, pointer to symbol table entry for C> <expop> <number, integer value 2> 7 A. A. Pourhaji Kazem, Spring 2009
8 Lexical errors Some errors are out of power of lexical analyzer to recognize: fi (a == f(x)) However it may be able to recognize errors like: d = 2r Such errors are recognized when no pattern for tokens matches a character sequence 8 A. A. Pourhaji Kazem, Spring 2009
9 Error recovery Panic mode: successive characters are ignored until we reach to a well formed token Delete one character from the remaining input Insert a missing character into the remaining input Replace a character by another character Transpose two adjacent characters 9 A. A. Pourhaji Kazem, Spring 2009
10 Input buffering Sometimes lexical analyzer needs to look ahead some symbols to decide about the token to return In C language: we need to look after , = or < to decide what token to return In Fortran: DO 5 I = 1.25 We need to introduce a two buffer scheme to handle large lookaheads safely E = M * C * * 2 eof 10 A. A. Pourhaji Kazem, Spring 2009
11 Sentinels E = M eof * C * * 2 eof eof Switch (*forward++) { case eof: if (forward is at end of first buffer) { reload second buffer; forward = beginning of second buffer; } else if {forward is at end of second buffer) { reload first buffer;\ forward = beginning of first buffer; } else /* eof within a buffer marks the end of input */ terminate lexical analysis; break; cases for the other characters; } 11 A. A. Pourhaji Kazem, Spring 2009
12 Specification of tokens In theory of compilation regular expressions are used to formalize the specification of tokens Regular expressions are means for specifying regular languages Example: Letter_(letter_ digit)* Each regular expression is a pattern specifying the form of strings 12 A. A. Pourhaji Kazem, Spring 2009
13 Regular expressions Ű is a regular expression, L(Ű) = {Ű} If a is a symbol in then a is a regular expression, L(a) = {a} (r) (s) is a regular expression denoting the language L(r) L(s) (r)(s) is a regular expression denoting the language L(r)L(s) (r)* is a regular expression denoting (L9r))* (r) is a regular expression denting L(r) 13 A. A. Pourhaji Kazem, Spring 2009
14 Regular definitions d1 > r1 d2 > r2 dn > rn Example: letter_ > A B Z a b Z _ digit > id > letter_ (letter_ digit)* 14 A. A. Pourhaji Kazem, Spring 2009
15 Extensions One or more instances: (r)+ Zero of one instances: r? Character classes: [abc] Example: letter_ > [AZaz_] digit > [09] id > letter_(letter digit)* 15 A. A. Pourhaji Kazem, Spring 2009
16 Recognition of tokens Starting point is the language grammar to understand the tokens: stmt > if expr then stmt if expr then stmt else stmt Ű expr > term relop term term term > id number 16 A. A. Pourhaji Kazem, Spring 2009
17 Recognition of tokens (cont.) The next step is to formalize the patterns: digit > [09] Digits > digit+ number > digit(.digits)? (E[+]? Digit)? letter > [AZaz_] id > letter (letter digit)* If > if Then > then Else > else Relop > < > <= >= = <> We also need to handle whitespaces: ws > (blank tab newline)+ 17 A. A. Pourhaji Kazem, Spring 2009
18 Transition diagrams Transition diagram for relop 18 A. A. Pourhaji Kazem, Spring 2009
19 Transition diagrams (cont.) Transition diagram for reserved words and identifiers 19 A. A. Pourhaji Kazem, Spring 2009
20 Transition diagrams (cont.) Transition diagram for unsigned numbers 20 A. A. Pourhaji Kazem, Spring 2009
21 Transition diagrams (cont.) Transition diagram for whitespace 21 A. A. Pourhaji Kazem, Spring 2009
22 Architecture of a transitiondiagrambased lexical analyzer TOKEN getrelop() { TOKEN rettoken = new (RELOP) while (1) { switch(state) { } case 0: c= nextchar(); case 1: case 8: retract(); /* repeat character processing until a if (c == < ) state = 1; return or failure occurs */ else if (c == = ) state = 5; else if (c == > ) state = 6; else fail(); /* lexeme is not a relop */ break; rettoken.attribute = GT; return(rettoken); 22 A. A. Pourhaji Kazem, Spring 2009
23 Lexical Analyzer Generator  Lex Lex Source program lex.l Lexical Compiler lex.yy.c lex.yy.c C compiler a.out Input stream a.out Sequence of tokens 23 A. A. Pourhaji Kazem, Spring 2009
24 Structure of Lex programs declarations %% translation rules %% auxiliary functions Pattern {Action} 24 A. A. Pourhaji Kazem, Spring 2009
25 Example %{ /* definitions of manifest constants LT, LE, EQ, NE, GT, GE, IF, THEN, ELSE, ID, NUMBER, RELOP */ %} /* regular definitions delim [ \t\n] ws {delim}+ letter [AZaz] digit[09] id {letter}({letter} {digit})* number {digit}+(\.{digit}+)?(e[+]?{digit}+)? IntinstallID() {/* funtionto install the lexeme, whose first character is pointed to by yytext, and whose length is yyleng, into the symbol table and return a pointer thereto */ } IntinstallNum() { /* similar to installid, but puts numerical constants into a separate table */ } %% {ws}{/* no action and no return */} if {return(if);} then {return(then);} else {return(else);} {id} {yylval = (int) installid(); return(id); } {number} {yylval = (int) installnum(); return(number);} 25 A. A. Pourhaji Kazem, Spring 2009
26 Finite Automata Regular expressions = specification Finite automata = implementation A finite automaton consists of An input alphabet Σ A set of states S A start state n A set of accepting states F S A set of transitions state input state 26 A. A. Pourhaji Kazem, Spring 2009
27 Finite Automata Transition Is read s 1 a s 2 In state s 1 on input a go to state s 2 If end of input If in accepting state => accept, othewise => reject If no transition possible => reject 27 A. A. Pourhaji Kazem, Spring 2009
28 Finite Automata State Graphs A state The start state An accepting state A transition a 28 A. A. Pourhaji Kazem, Spring 2009
29 A Simple Example A finite automaton that accepts only 1 A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state 1 29 A. A. Pourhaji Kazem, Spring 2009
30 Another Simple Example A finite automaton accepting any number of 1 s followed by a single 0 Alphabet: {0,1} Check that 1110 is accepted but 110 is not A. A. Pourhaji Kazem, Spring 2009
31 And Another Example Alphabet {0,1} What language does this recognize? A. A. Pourhaji Kazem, Spring 2009
32 And Another Example Alphabet still { 0, 1 } 1 1 The operation of the automaton is not completely defined by the input On input 11 the automaton could be in either state 32 A. A. Pourhaji Kazem, Spring 2009
33 Epsilon Moves Another kind of transition: moves A B Machine can move from state A to state B without reading input 33 A. A. Pourhaji Kazem, Spring 2009
34 Deterministic and Nondeterministic Automata Deterministic Finite Automata (DFA) One transition per input per state No moves Nondeterministic Finite Automata (NFA) Can have multiple transitions for one input in a given state Can have moves Finite automata have finite memory Need only to encode the current state 34 A. A. Pourhaji Kazem, Spring 2009
35 Execution of Finite Automata A DFA can take only one path through the state graph Completely determined by input NFAs can choose Whether to make moves Which of multiple transitions for a single input to take 35 A. A. Pourhaji Kazem, Spring 2009
36 Acceptance of NFAs An NFA can get into multiple states Input: Rule: NFA accepts if it can get in a final state 36 A. A. Pourhaji Kazem, Spring 2009
37 NFA vs. DFA (1) NFAs and DFAs recognize the same set of languages (regular languages) DFAs are easier to implement There are no choices to consider 37 A. A. Pourhaji Kazem, Spring 2009
38 NFA vs. DFA (2) For a given language the NFA can be simpler than the DFA NFA DFA DFA can be exponentially larger than NFA 38 A. A. Pourhaji Kazem, Spring 2009
39 Regular Expressions to Finite Automata Highlevel sketch NFA Regular expressions DFA Lexical Specification Tabledriven Implementation of DFA 39 A. A. Pourhaji Kazem, Spring 2009
40 Regular Expressions to NFA (1) For each kind of rexp, define an NFA Notation: NFA for rexp A A For For input a a 40 A. A. Pourhaji Kazem, Spring 2009
41 Regular Expressions to NFA (2) For AB A B For A B B A 41 A. A. Pourhaji Kazem, Spring 2009
42 Regular Expressions to NFA (3) For A* A 42 A. A. Pourhaji Kazem, Spring 2009
43 Example of RegExp > NFA conversion Consider the regular expression (1 0)*1 The NFA is A B C D 1 0 E F G H I 1 J 43 A. A. Pourhaji Kazem, Spring 2009
44 Next NFA Regular expressions DFA Lexical Specification Tabledriven Implementation of DFA 44 A. A. Pourhaji Kazem, Spring 2009
45 NFA to DFA. The Trick Simulate the NFA Each state of resulting DFA = a nonempty subset of states of the NFA Start state = the set of NFA states reachable through moves from NFA start state Add a transition S a S to DFA iff S is the set of NFA states reachable from the states in S after seeing the input a considering moves as well 45 A. A. Pourhaji Kazem, Spring 2009
46 NFA > DFA Example A B ABCDHI 0 1 C D 1 0 E F FGABCDHI 0 1 EJGABCDHI 1 G H I J A. A. Pourhaji Kazem, Spring 2009
47 NFA to DFA. Remark An NFA may be in many states at any time How many different states? If there are N states, the NFA must be in some subset of those N states How many nonempty subsets are there? 2 N  1 = finitely many, but exponentially many 47 A. A. Pourhaji Kazem, Spring 2009
48 Implementation A DFA can be implemented by a 2D table T One dimension is states Other dimension is input symbols For every transition S i a S k define T[i,a] = k DFA execution If in state S i and input a, read T[i,a] = k and skip to state S k Very efficient 48 A. A. Pourhaji Kazem, Spring 2009
49 Table Implementation of a DFA S 0 1 T 0 1 U S T U T T U U T U 49 A. A. Pourhaji Kazem, Spring 2009
50 Implementation (Cont.) NFA > DFA conversion is at the heart of tools such as flex or jflex But, DFAs can be huge In practice, flexlike tools trade off speed for space in the choice of NFA and DFA representations 50 A. A. Pourhaji Kazem, Spring 2009
51 Readings Chapter 3 of the book 51 A. A. Pourhaji Kazem, Spring 2009
Chapter 3 Lexical Analysis
Chapter 3 Lexical Analysis Outline Role of lexical analyzer Specification of tokens Recognition of tokens Lexical analyzer generator Finite automata Design of lexical analyzer generator The role of lexical
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationProf. Mohamed Hamada Software Engineering Lab. The University of Aizu Japan
Compilers Prof. Mohamed Hamada Software Engineering Lab. The University of Aizu Japan Lexical Analyzer (Scanner) 1. Uses Regular Expressions to define tokens 2. Uses Finite Automata to recognize tokens
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs) Implementation
More informationLexical Analysis. Chapter 2
Lexical Analysis Chapter 2 1 Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexical analyzers (lexers) Regular
More informationLexical Analysis. Lecture 24
Lexical Analysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 2 1 Administrivia Moving to 60 Evans on Wednesday HW1 available Pyth manual available on line.
More informationCSE450. Translation of Programming Languages. Lecture 20: Automata and Regular Expressions
CSE45 Translation of Programming Languages Lecture 2: Automata and Regular Expressions Finite Automata Regular Expression = Specification Finite Automata = Implementation A finite automaton consists of:
More informationImplementation of Lexical Analysis
Implementation of Lexical Analysis Lecture 4 (Modified by Professor Vijay Ganesh) Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested
More informationLexical Analysis. Lecture 34
Lexical Analysis Lecture 34 Notes by G. Necula, with additions by P. Hilfinger Prof. Hilfinger CS 164 Lecture 34 1 Administrivia I suggest you start looking at Python (see link on class home page). Please
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 13, 2007 Outline Recap
More informationKinder, Gentler Nation
Lexical Analysis Finite Automata (Part 2 of 2) # Kinder, Gentler Nation In our post dropdeadline world things get easier. While we re here: reading quiz. #2 Summary Regular expressions provide a concise
More informationLexical Analysis. Finite Automata. (Part 2 of 2)
# Lexical Analysis Finite Automata (Part 2 of 2) PA0, PA Although we have included the tricky file ends without a newline testcases in previous years, students made good cases against them (e.g., they
More informationPRACTICAL CLASS: Flex & Bison
Master s Degree Course in Computer Engineering Formal Languages FORMAL LANGUAGES AND COMPILERS PRACTICAL CLASS: Flex & Bison Eliana Bove eliana.bove@poliba.it Install On Linux: install with the package
More informationImplementation of Lexical Analysis
Outline Implementation of Lexical nalysis Specifying lexical structure using regular expressions Finite automata Deterministic Finite utomata (DFs) Nondeterministic Finite utomata (NFs) Implementation
More informationCSE450. Translation of Programming Languages. Automata, Simple Language Design Principles
CSE45 Translation of Programming Languages Automata, Simple Language Design Principles Finite Automata State Graphs A state: The start state: An accepting state: A transition: a A Simple Example A finite
More informationUNIT 2 LEXICAL ANALYSIS
OVER VIEW OF LEXICAL ANALYSIS UNIT 2 LEXICAL ANALYSIS o To identify the tokens we need some method of describing the possible tokens that can appear in the input stream. For this purpose we introduce
More informationZhizheng Zhang. Southeast University
Zhizheng Zhang Southeast University 2016/10/5 Lexical Analysis 1 1. The Role of Lexical Analyzer 2016/10/5 Lexical Analysis 2 2016/10/5 Lexical Analysis 3 Example. position = initial + rate * 60 2016/10/5
More informationFormal Languages and Compilers Lecture VI: Lexical Analysis
Formal Languages and Compilers Lecture VI: Lexical Analysis Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/ artale/ Formal
More informationLexical Analysis. Implementation: Finite Automata
Lexical Analysis Implementation: Finite Automata Outline Specifying lexical structure using regular expressions Finite automata Deterministic Finite Automata (DFAs) Nondeterministic Finite Automata (NFAs)
More informationAdministrivia. Lexical Analysis. Lecture 24. Outline. The Structure of a Compiler. Informal sketch of lexical analysis. Issues in lexical analysis
dministrivia Lexical nalysis Lecture 24 Notes by G. Necula, with additions by P. Hilfinger Moving to 6 Evans on Wednesday HW available Pyth manual available on line. Please log into your account and electronically
More informationLexical Analysis (ASU Ch 3, Fig 3.1)
Lexical Analysis (ASU Ch 3, Fig 3.1) Implementation by hand automatically ((F)Lex) Lex generates a finite automaton recogniser uses regular expressions Tasks remove white space (ws) display source program
More informationLexical Analysis. Implementing Scanners & LEX: A Lexical Analyzer Tool
Lexical Analysis Implementing Scanners & LEX: A Lexical Analyzer Tool Copyright 2016, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California
More informationFigure 2.1: Role of Lexical Analyzer
Chapter 2 Lexical Analysis Lexical analysis or scanning is the process which reads the stream of characters making up the source program from lefttoright and groups them into tokens. The lexical analyzer
More informationLex Spec Example. Int installid() {/* code to put id lexeme into string table*/}
Class 5 Lex Spec Example delim [ \t\n] ws {delim}+ letter [AAaz] digit [09] id {letter}({letter} {digit})* number {digit}+(\.{digit}+)?(e[+]?{digit}+)? %% {ws} {/*no action and no return*?} if {return(if);}
More informationImplementation of Lexical Analysis. Lecture 4
Implementation of Lexical Analysis Lecture 4 1 Tips on Building Large Systems KISS (Keep It Simple, Stupid!) Don t optimize prematurely Design systems that can be tested It is easier to modify a working
More informationCOMPILER DESIGN UNIT I LEXICAL ANALYSIS. Translator: It is a program that translates one language to another Language.
UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another Language. Source Code Translator Target Code 1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System
More informationImplementation of Lexical Analysis
Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week :59pm Electronic handin Prof. iken CS 43 Lecture 4 Prof. iken CS 43 Lecture 4 2 Tips on uilding Large Systems
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsansheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationConcepts Introduced in Chapter 3. Lexical Analysis. Lexical Analysis Terms. Attributes for Tokens
Concepts Introduced in Chapter 3 Lexical Analysis Regular Expressions (REs) Nondeterministic Finite Automata (NFA) Converting an RE to an NFA Deterministic Finite Automatic (DFA) Lexical Analysis Why separate
More informationLexical Analysis. Chapter 1, Section Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual
Lexical Analysis Chapter 1, Section 1.2.1 Chapter 3, Section 3.1, 3.3, 3.4, 3.5 JFlex Manual Inside the Compiler: Front End Lexical analyzer (aka scanner) Converts ASCII or Unicode to a stream of tokens
More informationLexical Analyzer Scanner
Lexical Analyzer Scanner ASU Textbook Chapter 3.1, 3.3, 3.4, 3.6, 3.7, 3.5 Tsansheng Hsu tshsu@iis.sinica.edu.tw http://www.iis.sinica.edu.tw/~tshsu 1 Main tasks Read the input characters and produce
More informationImplementation of Lexical Analysis
Written ssignments W assigned today Implementation of Lexical nalysis Lecture 4 Due in one week y 5pm Turn in In class In box outside 4 Gates Electronically Prof. iken CS 43 Lecture 4 Prof. iken CS 43
More informationPRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis  The Role of the Lexical Analyzer
PRINCIPLES OF COMPILER DESIGN UNIT II LEXICAL ANALYSIS 2.1 Lexical Analysis  The Role of the Lexical Analyzer As the first phase of a compiler, the main task of the lexical analyzer is to read the input
More informationRecognition of Tokens
Recognition of Tokens Lecture 3 Section 3.4 Robb T. Koether HampdenSydney College Mon, Jan 19, 2015 Robb T. Koether (HampdenSydney College) Recognition of Tokens Mon, Jan 19, 2015 1 / 21 1 A Class of
More information1. INTRODUCTION TO LANGUAGE PROCESSING The Language Processing System can be represented as shown figure below.
UNIT I Translator: It is a program that translates one language to another Language. Examples of translator are compiler, assembler, interpreter, linker, loader and preprocessor. Source Code Translator
More informationChapter 3: Lexical Analysis
Chapter 3: Lexical Analysis A simple way to build a lexical analyzer is to construct a diagram that illustrates the structure of tokens of the source language, and then to hand translate the diagram into
More informationPrinciples of Compiler Design Presented by, R.Venkadeshan,M.TechIT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology
Principles of Compiler Design Presented by, R.Venkadeshan,M.TechIT, Lecturer /CSE Dept, Chettinad College of Engineering &Technology 6/30/2010 Principles of Compiler Design R.Venkadeshan 1 Preliminaries
More information2. λ is a regular expression and denotes the set {λ} 4. If r and s are regular expressions denoting the languages R and S, respectively
Regular expressions: a regular expression is built up out of simpler regular expressions using a set of defining rules. Regular expressions allows us to define tokens of programming languages such as identifiers.
More informationCS Lecture 2. The Front End. Lecture 2 Lexical Analysis
CS 1622 Lecture 2 Lexical Analysis CS 1622 Lecture 2 1 Lecture 2 Review of last lecture and finish up overview The first compiler phase: lexical analysis Reading: Chapter 2 in text (by 1/18) CS 1622 Lecture
More informationLexical Analysis. Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata
Lexical Analysis Dragon Book Chapter 3 Formal Languages Regular Expressions Finite Automata Theory Lexical Analysis using Automata Phase Ordering of FrontEnds Lexical analysis (lexer) Break input string
More informationFront End: Lexical Analysis. The Structure of a Compiler
Front End: Lexical Analysis The Structure of a Compiler Constructing a Lexical Analyser By hand: Identify lexemes in input and return tokens Automatically: LexicalAnalyser generator We will learn about
More informationCOMP421 Compiler Design. Presented by Dr Ioanna Dionysiou
COMP421 Compiler Design Presented by Dr Ioanna Dionysiou Administrative! [ALSU03] Chapter 3  Lexical Analysis Sections 3.13.4, 3.63.7! Reading for next time [ALSU03] Chapter 3 Copyright (c) 2010 Ioanna
More informationCD Assignment I. 1. Explain the various phases of the compiler with a simple example.
CD Assignment I 1. Explain the various phases of the compiler with a simple example. The compilation process is a sequence of various phases. Each phase takes input from the previous, and passes the output
More informationCS 403: Scanning and Parsing
CS 403: Scanning and Parsing Stefan D. Bruda Fall 2017 THE COMPILATION PROCESS Character stream Scanner (lexical analysis) Token stream Parser (syntax analysis) Parse tree Semantic analysis Abstract syntax
More informationUNIT II LEXICAL ANALYSIS
UNIT II LEXICAL ANALYSIS 2 Marks 1. What are the issues in lexical analysis? Simpler design Compiler efficiency is improved Compiler portability is enhanced. 2. Define patterns/lexeme/tokens? This set
More informationCSc 453 Lexical Analysis (Scanning)
CSc 453 Lexical Analysis (Scanning) Saumya Debray The University of Arizona Tucson Overview source program lexical analyzer (scanner) tokens syntax analyzer (parser) symbol table manager Main task: to
More informationTHE COMPILATION PROCESS EXAMPLE OF TOKENS AND ATTRIBUTES
THE COMPILATION PROCESS Character stream CS 403: Scanning and Parsing Stefan D. Bruda Fall 207 Token stream Parse tree Abstract syntax tree Modified intermediate form Target language Modified target language
More informationModule 8  Lexical Analyzer Generator. 8.1 Need for a Tool. 8.2 Lexical Analyzer Generator Tool
Module 8  Lexical Analyzer Generator This module discusses the core issues in designing a lexical analyzer generator from basis or using a tool. The basics of LEX tool are also discussed. 8.1 Need for
More informationMonday, August 26, 13. Scanners
Scanners Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved words, literals, etc. What do we need to know? How do we define tokens? How can
More informationWednesday, September 3, 14. Scanners
Scanners Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved words, literals, etc. What do we need to know? How do we define tokens? How can
More informationCompiler phases. Nontokens
Compiler phases Compiler Construction Scanning Lexical Analysis source code scanner tokens regular expressions lexical analysis Lennart Andersson parser context free grammar Revision 2011 01 21 parse tree
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwthaachen.de/teaching/ss16/cc/ Recap: FirstLongestMatch Analysis Outline of Lecture
More informationLexical Analysis  1. A. Overview A.a) Role of Lexical Analyzer
CMPSC 470 Lecture 02 Topics: Regular Expression Transition Diagram Lexical Analyzer Implementation A. Overview A.a) Role of Lexical Analyzer Lexical Analysis  1 Lexical analyzer does: read input character
More informationLexical analysis. Syntactical analysis. Semantical analysis. Intermediate code generation. Optimization. Code generation. Target specific optimization
Second round: the scanner Lexical analysis Syntactical analysis Semantical analysis Intermediate code generation Optimization Code generation Target specific optimization Lexical analysis (Chapter 3) Why
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwthaachen.de/teaching/ss17/cc/ Recap: FirstLongestMatch Analysis The Extended Matching
More informationScanners. Xiaokang Qiu Purdue University. August 24, ECE 468 Adapted from Kulkarni 2012
Scanners Xiaokang Qiu Purdue University ECE 468 Adapted from Kulkarni 2012 August 24, 2016 Scanners Sometimes called lexers Recall: scanners break input stream up into a set of tokens Identifiers, reserved
More informationChapter 4. Lexical analysis. Concepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Chapter 4 Lexical analysis Lexical scanning Regular expressions DFAs and FSAs Lex Concepts CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley
More informationRegular Expressions. Agenda for Today. Grammar for a Tiny Language. Programming Language Specifications
Agenda for Today Regular Expressions CSE 413, Autumn 2005 Programming Languages Basic concepts of formal grammars Regular expressions Lexical specification of programming languages Using finite automata
More informationCSEP 501 Compilers. Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter /8/ Hal Perkins & UW CSE B1
CSEP 501 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2008 1/8/2008 200208 Hal Perkins & UW CSE B1 Agenda Basic concepts of formal grammars (review) Regular expressions
More informationCSE302: Compiler Design
CSE302: Compiler Design Instructor: Dr. Liang Cheng Department of Computer Science and Engineering P.C. Rossin College of Engineering & Applied Science Lehigh University February 01, 2007 Outline Recap
More information10/4/18. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntactic Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationConcepts. Lexical scanning Regular expressions DFAs and FSAs Lex. Lexical analysis in perspective
Concepts Lexical scanning Regular expressions DFAs and FSAs Lex CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 1 CMSC 331, Some material 1998 by Addison Wesley Longman, Inc. 2 Lexical analysis
More informationCompiler Construction
Compiler Construction Thomas Noll Software Modeling and Verification Group RWTH Aachen University https://moves.rwthaachen.de/teaching/ss16/cc/ Conceptual Structure of a Compiler Source code x1 := y2
More informationLexical Analysis. Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast!
Lexical Analysis Lexical analysis is the first phase of compilation: The file is converted from ASCII to tokens. It must be fast! Compiler Passes Analysis of input program (frontend) character stream
More informationLexical and Syntax Analysis
Lexical and Syntax Analysis In Text: Chapter 4 N. Meng, F. Poursardar Lexical and Syntactic Analysis Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input
More informationUNIT I LEXICAL ANALYSIS. 1.Interpreter: It is one of the translators that translate high level language to low level language.
INRODUCTION TO COMPILING UNIT I LEXICAL ANALYSIS Translator: It is a program that translates one language to another. Types of Translator: 1.Interpreter 2.Compiler 3.Assembler source code Translator target
More informationG Compiler Construction Lecture 4: Lexical Analysis. Mohamed Zahran (aka Z)
G22.2130001 Compiler Construction Lecture 4: Lexical Analysis Mohamed Zahran (aka Z) mzahran@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Autumn 2012 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationCSCIGA Compiler Construction Lecture 4: Lexical Analysis I. Hubertus Franke
CSCIGA.2130001 Compiler Construction Lecture 4: Lexical Analysis I Hubertus Franke frankeh@cs.nyu.edu Role of the Lexical Analyzer Remove comments and white spaces (aka scanning) Macros expansion Read
More informationCSE 413 Programming Languages & Implementation. Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions
CSE 413 Programming Languages & Implementation Hal Perkins Winter 2019 Grammars, Scanners & Regular Expressions 1 Agenda Overview of language recognizers Basic concepts of formal grammars Scanner Theory
More informationIntroduction to Lexical Analysis
Introduction to Lexical Analysis Outline Informal sketch of lexical analysis Identifies tokens in input string Issues in lexical analysis Lookahead Ambiguities Specifying lexers Regular expressions Examples
More informationCS308 Compiler Principles Lexical Analyzer Li Jiang
CS308 Lexical Analyzer Li Jiang Department of Computer Science and Engineering Shanghai Jiao Tong University Content: Outline Basic concepts: pattern, lexeme, and token. Operations on languages, and regular
More informationAdministrativia. Extra credit for bugs in project assignments. Building a Scanner. CS164, Fall Recall: The Structure of a Compiler
dministrativia Building a canner Extra credit for bugs in project assignments in starter kits and handouts Ts are final arbiters of what s a bug only the first student to report the bug gets credit C64,
More informationLexical Analysis. Sukree Sinthupinyo July Chulalongkorn University
Sukree Sinthupinyo 1 1 Department of Computer Engineering Chulalongkorn University 14 July 2012 Outline Introduction 1 Introduction 2 3 4 Transition Diagrams Learning Objectives Understand definition of
More informationLexical Analysis. Introduction
Lexical Analysis Introduction Copyright 2015, Pedro C. Diniz, all rights reserved. Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies
More informationPart 5 Program Analysis Principles and Techniques
1 Part 5 Program Analysis Principles and Techniques Front end 2 source code scanner tokens parser il errors Responsibilities: Recognize legal programs Report errors Produce il Preliminary storage map Shape
More informationFormal Languages and Compilers Lecture IV: Regular Languages and Finite. Finite Automata
Formal Languages and Compilers Lecture IV: Regular Languages and Finite Automata Free University of BozenBolzano Faculty of Computer Science POS Building, Room: 2.03 artale@inf.unibz.it http://www.inf.unibz.it/
More informationCS415 Compilers. Lexical Analysis
CS415 Compilers Lexical Analysis These slides are based on slides copyrighted by Keith Cooper, Ken Kennedy & Linda Torczon at Rice University Lecture 7 1 Announcements First project and second homework
More informationProjects for Compilers
Projects for Compilers 1. Project One: Lexical Analysis (Required) (1) Directions Implement a transitiondiagrambased lexical analysis for the programming language TINY. (2) Outputs Source code (implemented
More informationBuffering Techniques: Buffer Pairs and Sentinels
Week 3 Lexical Analysis Tasks of Lexical Analysis Why separating lexical analysis and parsing? Tokens, Patterns and Lexemes Complex tokens like identifier and numeral are described using regularexpression
More informationCompiler Construction
Compiler Construction Lecture 2: Lexical Analysis I (Introduction) Thomas Noll Lehrstuhl für Informatik 2 (Software Modeling and Verification) noll@cs.rwthaachen.de http://moves.rwthaachen.de/teaching/ss14/cc14/
More informationCS 314 Principles of Programming Languages. Lecture 3
CS 314 Principles of Programming Languages Lecture 3 Zheng Zhang Department of Computer Science Rutgers University Wednesday 14 th September, 2016 Zheng Zhang 1 CS@Rutgers University Class Information
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview Tokens and regular expressions Syntax and contextfree grammars Grammar derivations More about parse trees Topdown and bottomup
More informationCompilers: table driven scanning Spring 2008
Compilers: table driven scanning Spring 2008 1 Contents Transition diagrams (3.4.1) Sketch implementation (3.4.4) Tabledriven scanner (3.8) Objectoriented implementation 2 Transition diagrams (ALSU,
More informationProgramming Languages (CS 550) Lecture 4 Summary Scanner and Parser Generators. Jeremy R. Johnson
Programming Languages (CS 550) Lecture 4 Summary Scanner and Parser Generators Jeremy R. Johnson 1 Theme We have now seen how to describe syntax using regular expressions and grammars and how to create
More informationLanguages, Automata, Regular Expressions & Scanners. Winter /8/ Hal Perkins & UW CSE B1
CSE 401 Compilers Languages, Automata, Regular Expressions & Scanners Hal Perkins Winter 2010 1/8/2010 200210 Hal Perkins & UW CSE B1 Agenda Quick review of basic concepts of formal grammars Regular
More informationDavid Griol Barres Computer Science Department Carlos III University of Madrid Leganés (Spain)
David Griol Barres dgriol@inf.uc3m.es Computer Science Department Carlos III University of Madrid Leganés (Spain) OUTLINE Introduction: Definitions The role of the Lexical Analyzer Scanner Implementation
More informationComputer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres
Computer Science Department Carlos III University of Madrid Leganés (Spain) David Griol Barres dgriol@inf.uc3m.es Introduction: Definitions Lexical analysis or scanning: To read from lefttoright a source
More informationLexical Analysis. Lecture 3. January 10, 2018
Lexical Analysis Lecture 3 January 10, 2018 Announcements PA1c due tonight at 11:50pm! Don t forget about PA1, the Cool implementation! Use Monday s lecture, the video guides and Cool examples if you re
More informationTypical tradeoffs in compiler design are: speed of compilation size of the generated code speed of the generated code Speed of Execution Foundations
Compilers overview There are many aspects to be considered in the study of compilers. Usually the study encompasses more than just the strict definition of a compiler. In General a compiler is: A program
More information[Lexical Analysis] Bikash Balami
1 [Lexical Analysis] Compiler Design and Construction (CSc 352) Compiled By Central Department of Computer Science and Information Technology (CDCSIT) Tribhuvan University, Kirtipur Kathmandu, Nepal 2
More informationCSc 453 Compilers and Systems Software
CSc 453 Compilers and Systems Software 3 : Lexical Analysis I Christian Collberg Department of Computer Science University of Arizona collberg@gmail.com Copyright c 2009 Christian Collberg August 23, 2009
More informationAnnouncements! P1 part 1 due next Tuesday P1 part 2 due next Friday
Announcements! P1 part 1 due next Tuesday P1 part 2 due next Friday 1 Finitestate machines CS 536 Last time! A compiler is a recognizer of language S (Source) a translator from S to T (Target) a program
More informationCS 314 Principles of Programming Languages
CS 314 Principles of Programming Languages Lecture 2: Syntax Analysis Zheng (Eddy) Zhang Rutgers University January 22, 2018 Announcement First recitation starts this Wednesday Homework 1 will be release
More information10/5/17. Lexical and Syntactic Analysis. Lexical and Syntax Analysis. Tokenizing Source. Scanner. Reasons to Separate Lexical and Syntax Analysis
Lexical and Syntactic Analysis Lexical and Syntax Analysis In Text: Chapter 4 Two steps to discover the syntactic structure of a program Lexical analysis (Scanner): to read the input characters and output
More informationEdited by Himanshu Mittal. Lexical Analysis Phase
Edited by Himanshu Mittal Lexical Analysis Phase Lexical Analyzer The main task of Lexical analysis is to read input characters of source program and group them into lexemes and produce as output a sequence
More informationCOP4020 Programming Languages. Syntax Prof. Robert van Engelen
COP4020 Programming Languages Syntax Prof. Robert van Engelen Overview n Tokens and regular expressions n Syntax and contextfree grammars n Grammar derivations n More about parse trees n Topdown and
More informationOutline. 1 Scanning Tokens. 2 Regular Expresssions. 3 Finite State Automata
Outline 1 2 Regular Expresssions Lexical Analysis 3 Finite State Automata 4 Nondeterministic (NFA) Versus Deterministic Finite State Automata (DFA) 5 Regular Expresssions to NFA 6 NFA to DFA 7 8 JavaCC:
More information2010: Compilers REVIEW: REGULAR EXPRESSIONS HOW TO USE REGULAR EXPRESSIONS
2010: Compilers Lexical Analysis: Finite State Automata Dr. Licia Capra UCL/CS REVIEW: REGULAR EXPRESSIONS a Character in A Empty string R S Alternation (either R or S) RS Concatenation (R followed by
More information