Instructor

Dr. Rudolf Riedi
Duncan Hall 2025, 713 / 348 3020,
Office Hours: Tu 4-6pm (DH 2025), or by appointment

Nasir Ahmed
Duncan Hall DH2123, 713 / 348 2827,
Office Hours: Thursday 4pm-6pm, or by appointment

Alireza Keshavarz-Haddad
Duncan Hall, 713 / 348 3579,
Office Hours: W 3-5pm, or by appointment

Monday Wednesday Friday 10:00 - 10:50 am, AL (Abercrombie Lab) 126

For current updates check the official Rice page

Requirement
 

Outline
 

Textbook

• T. Cover and J. Thomas, `Elements of Information Theory'.

The course will closely follow this book; it is available at the campus bookstore.

• R. Gallager, `Information Theory and Reliable Communication', (1968)
• N. Abramson, `Information Theory and Coding' (1963)
• C. Shannon, `A mathematical theory of communication', Bell Sys. Tech. Journal, vol 27, p 379-423, 623-656, (1949)

• H. Stark and J. Woods, `Probability, Random Processes, and Estimation Theory for Engineers'.
• A. Papoulis, `Probability, Random Variables, and Stochastic Processes'

Grading

15%  QUIZ
30%  Midterm EXAM
30%  Last EXAM
20%  Homework
5%  Notes and participation in class

Classes and Reading Assignments
  This doubles as a calendar for the course.

	Covered material	Reading: Cover&Thomas
January 13	Orientation, history
January 15	Discrete and differential entropy	2-8, 12-14, 224-225
January 17	Quantization, joint+conditional entropy	228-230, 15-16
January 22	Chain rule, KL distance, Mutual Information	16-21, 231-232
January 24	Basic properties of information, Markov Chains	22-23, 231; 32
January 27	Data processing inequality	32-33, (33-38)
January 29	Jensen's inequality	23-26
January 31	Convexity; thermodynamics	26-31; (33-38)
February 3	Convexity and maximal entropy	233-234
up to here:	Material for Quiz
February 5	Codes: basics	78-82
February 7	Kraft Inequality	82-86
February 10	HW3, D-ary expansions	90-92
February 12	QUIZ in class
February 14	extended Kraft, McMillan	90-92
February 17	Optimal Codes, Lagrange multipliers	87
February 19	Shannon Codes	88-89
February 21	Entropy rate, Cesaro means	60-66
February 24	Penalty for mismatched code	89-90
February 26	Huffman Codes: optimality	92-97, 97-101
February 28	Shannon-Fano-Elias Codes	101-103
March 3	Markov Chains, entropy rate	60-66
March 5	Arithmetic Codes	104-107
March 7	Homework solutions
Up to here	Material for Test 1
March 10	Midterm Recess
March 12	Midterm Recess
March 14	Midterm Recess
March 17	LLN, Asymptotic Equipartition Property	50-51
March 19	AEP and compression; Channel capacity	52-54; 183-185
March 21	Examples of channels	185-188
March 24	Symmetric channels	189-191
March 26	Joint AEP	191-197
March 28	Shannon's Channel Coding Theorem	198-203
March 31	Fano' s inequality (2nd part of Shannon)	204-207
April 2	Discussion Test 1
April 4	no class
April 7	Feedback, Joint Source-Channel Coding	213-214, 215-218
April 9	Hamming codes	209-212
April 11	Gaussian Channel: Capacity	239-243
April 14	AEP for continuous r.v.; Achieving capacity	226-227;243-247
April 16	Parallel channels and water filling	250-256
Up to here	Material for Test 2	download Test 2 from the following URL (access only within Rice, request password here) [ps] [pdf] [tex]
April 18	Review Homework; Rate distortion intro	336-339
April 21	Achievable distortion rates	340-349,357

Homework
(tex-source and solutions restricted to Rice University)

 

Homework sheet	Due date (in class)	Solutions
Problem Set 1 [ps] [pdf] [tex]	January 24, 2003	Jan 31 [ps] [pdf] [tex]
Problem Set 2 [ps] [pdf] [tex]	January 31, 2001	handed out Feb 7 [ps] [pdf] [tex]
Problem Set 3 [ps] [pdf] [tex]	February 7, 2003	handed out Feb 10 [ps] [pdf] [tex]
Problem Set 4 [ps] [pdf] [tex]	February 21, 2003	handed out February 28 [ps] [pdf] [tex]
Problem Set 5 [ps] [pdf] [tex]	February 28, 2003	handed out March 7 [ps] [pdf] [tex]
Problem Set 6 [ps] [pdf] [tex]	March 7, 2003	handed out March 7 [ps] [pdf] [tex]
Problem Set 7 [ps] [pdf] [tex]	March 28, 2003	handed out April 7 [ps] [pdf] [tex]
Problem Set 8 [ps] [pdf] [tex]	April 7, 2003	handed out April 11 [ps] [pdf] [tex]
Problem Set 9 [ps] [pdf] [tex]	April 16, 2003	handed out April 16

 

Late Homework Policy
 

Homework is due at the beginning of class on the due date. After the due date, but before solutions are handed out, homework can be turned in for 50% credit. In this case, please slip your homework under the door of DH 2025 or bring it to class. After solutions are handed out, 0% credit will be issued. You are encouraged to work in groups for homeworks but you will hand in your own solution which you are expected to understand.

Tests
 

Quiz (15% towards the grade)	February 12, 2003, 10 am AL126	In class, 30 min, (open: only two hand-written pages )
Test 1 (30%)	Handed out: March 7. Due March 21, midnight. download Test 1 from the following URL (access only within Rice, request password here) [ps] [pdf] [tex]	Take home, 3 hours, (open: personal notes and lecture notes)
Test 2 (30%)	Handed out: April 18. Due April 25, midnight. download Test 2 from the following URL (access only within Rice, request password here) [ps] [pdf] [tex]	Take home, 3 hours (open books)