General

• On the syllabus the class is instructed to Re-Read TWF Ch 1-8. Is TWF "Models for Investors in Real World Markets" or "The Intelligent Investor". Also, I was not aware that we were suppose to have already read these chapters, and it will be quite a task to have them complete by Thursday. Are there specific chapters that would be more useful for class Thursday and other chapters that could be read this weekend?
• Should a group consist of only undergraduates or graduate students, since there are different requirements for 686 students? If a group does contain students from both 486 and 686, then how does it deal with the differing requirements?
• Is assignment 0 to be presented in a "professional manner" or can it be hand written?
• Should we list all of the constituent company names for the indexes or can we just provide a link? The instructions didn't specify and listing all of the names would go well above the 3 page limit.
• How do I obtain returns with dividends from CRSP? Do I have adjust prices for dividends and splits, etc?
• How do I obtain daily index data? WRDS says we do not have Compustat Global Index Prices subscription.

Market Indexes and Components

• Should we be working on homework 1 in a group?
• Our group is having trouble finding FTDOY, LTDOY, and number of trading days for each year. Does anyone have any idea?

Mandelbrot Review and A Bad Backtest (k-Best Stock selection)

• Where did the S486 universe come from?
• Past data is not meaningfrul for this exercise.

Benchmarks

• With regard to the dates, are you requiring that we calculate returns over 1 year defined as the time between FTDOY and LTDOY, and NOT the full 365 (or 366 for leap year) calendar year? 
• When you use the phrase "if possible" when discussing benchmarking annual returns...what do you mean?
• Where did you get the benchmark data for the Dow and the Russell 2000/3000? We could not locate the data on WRDS or CRSP. I have data downloaded from Bloomberg that is similar to the benchmark, however the data runs from (12/31-12/31) rather than (1/01-1/01), so I was wondering where you got Dow and Russell so I can calibrate exactly to your data.
• Our team is having great difficulty finding data for the S&P 500 values for each of the four categories: market weighted dividends, market weighted no dividends, equal weighted dividends, and equal weighted no dividends. While CRSP provides information, it only gives the percentages and not the actual annual dollar values. Furthermore, the S&P website seems to provide only market weighted no dividends annual values and only back to 1988.
• I wonder whether the Dow Jones annual return data come from WRDS. I wasn't able to find it under either CRSP or Compustat. If it's not, could you please point out a direction of where we should be looking for the data? You have suggested us to go to the Dow Jones official site for total return data. The Dow Jones Total Return on the site is only available in daily, not yearly, and it only goes back to 1987. See http://www.djaverages.com/?go=industrial-index-data
• We are having difficulty finding data for Russell and NASDAQ as well... It seems the only place that provides market weighted vs value weighted data is WRDS. Yet we could not find any data of Russell or NASDAQ.
• Additionally, how do we normalize data for the year, if we are able to get it.
• I've been exploring WRDS but with the exception of NYSE/AMEX, I can't find any information summarized by index with which to construct benchmarks. Do you intend for us to identify the membership of each index by year (86 years), look up the permno's for each stock, and pull this information from CRSP for each of the DJIA, SP500, NYSE/AMEX, RUT3000, RUT2000? If that is your intention, can you please tell me if you will be assigning this amount of work for each of the homeworks? We did something similar for just the 10 dogs of the Dow over 30 years in Stat 686. It took a very long time and was considered a project level undertaking. I've added 482 on top of an already heavy load because I'm interested in the topic. So I'm eager to know if I should be auditing the class instead of taking it.
• How do we get the master list of "permno" of each company we need the data for. I was thinking of taking the list of "permno" and putting it in the CRSP/CRSP-Merged to get the annual variables for each of the companies(public and private).
• The instructions state "If a strategy outperforms a benchmark return by 1 percentage point over 30 years, we need to know if that is a significant amount." What does strategy refer to in this context? What does it mean by "outperforms a benchmark return by 1% over 30 years?" And should we make decision about the significance of this strategy as part of our solution?

CAPM Validation

• I am wondering what ARCA index means in CAPM project.
• What is "w'r"?
• How do we calculate the "actual annual return" for a portfolio?

Data Collection and Cleaning

• Where we could find the data for market cap from 1926-1949? We couldn't seem to find it on the CCM database.
• The instructions use the abbreviations TA and CA in various definitions; however, based on Chapter 4 in QFA and the CRSP/Compustat terms, it seems as though ACT (Current Assets - Total) works in place of both TA and CA. Am I understanding this correctly?
• Do you want three separate portfolios based on each variable or can we do one portfolio based on a combination of our three variables? We have already constructed a portfolio that ranks our companies by each variable at different levels. For instance, we used market cap as our primary variable, then sorted by dividend yield as our secondary variable, and then sorted by price/sales as our third variable. Furthermore, we discussed why we ranked each variables in this way.

Dogs of the Dow

• In some of the sample assignments people distinguished between annual return and total return. Do you know the difference between the two?

Buy and Hold with the Best (Graham/Dodd Analysis)

• I cannot fiund the Buy and Hold assignment?
• Where can we find Graham's Table 14-3? Is it in the textbook? Or is it in the Security Analysis?
• For the Earnings per Share (and cash flows for that matter), do you recommend (Diluted or Basic) and including or excluding extraordinary items?
• What should we use for debt?
• If the company de-lists or goes private (ie American Can) that the portfolio is rebalanced and we stay in 100% equity for the positional analysis as of 12/31/2010, or do we just place that portion in some lame money market?
• How are the stability index and Growth values calculated?
• We aren't sure exactly what types of tables we are expected to make: should we make a different table for each year from 1970 on, or something else?
• We noticed in the PERMO's we were given in the assignment, one of the companies listed is "Beatrice," yet we cannot find any record of this company in the DJIA. Maybe you can clarify this for us?
• We couldn't seem to match our result for PE ratio. We used the variable NIADJ -- Net Income Adjusted for Common/Ordinary Stock (Capital) Equivalents, divided by CSHPRI -- Common Shares Used to Calculate Earnings Per Share - Basic. But we got a much smaller number than the listed 11. Do you know where the problem could be?

Max-Median Rule

• Could we use log return for ranking the portfolio rather than simple gross return? Why or why not?
• For reproducing the results, are the charts and tables in the provided slides the only ones we need to reproduce? Or do we need to produce any other result mentioned in the paper?
• Hi I was wondering how long we should back-test our program, my computer crashes if I go from 1925-2010, R freezes because the file is too large (3GB). Would back-testing from 1980-2010 be acceptable? Otherwise is there a more powerful computer that I can use at Rice?
• Since the S&P 500 is always changing, is there anyway that we can get the prices or returns of the changing companies directly online without manually type in the permno in compustat?
• We are wondering what role the CFACPR variable plays in the calculations. Our understanding of the assignment is that we compute the daily ratios for each stock in the S&P 500, then find the median ratio value for each stock for the year and select the 20 highest to invest in the next year. We then calculate our returns from investing in those 20 stocks for the year. Where exactly would the CFACPR variable come into play?
• We having some issues with Excel and Bloomberg. We are pulling daily data from bloomberg using the compustat database's records of S&P constituents, but it is taking an extremely long time to pull the data for the S&P 100 and 500. The s&p 100 constituents will take around 2 hours to pull, and we're estimating that for the 500, it will take about 10 hours. We are also having problems with excel crashing due to the size of these spreadsheets (around 11,000,000 data points in our S&P 500 spreadsheet). Any suggestions that you could give us would be greatly appreciated.
• In the table on slide 7, there is a column that lists "Median Weight Return." Could you describe the methodology behind calculating those returns? Our best guess at this point, hesitantly confirmed by Sarah Tooth, is that the column describes the weighted median of the 20-stock Max Median portfolio.
• For investing with one present value dollar starting from 1970, does it mean we initially invest $1 dollar in 1970 currency value or initially invest $1 in 2016 value (aka 1/(1.02^36) in 1970 value)?

WRDS/CRSP Variables

• In the description of HW #3 you say that market cap should be readily available however I am not seeing it on Crsp merged database. I have found a variable called "Market Value" but I do not think this is what I want. If I cannot find the variable should I just multiply "common shares outstanding" by "closing annual price"? I am reluctant to do this though because you said to avoid using price data in our calculation.
• Regarding CFO, "CFOPS" does not appear in the CCM variable list. What should we use for this?
• Regarding CFO, we used XOPR, would that be best?
• The data is improperly formatted. The data entered for fiscal year end and actual filing data is not recognized as a date. Because of this the functions that calculate the amount of days in between 2 dates do not work. There are about 2000 observations for each of the 16 years, thus I think it's a little unreasonable for me to go through and manually calculate this for each observation. I'm not really sure what I should do.
• What kind of returns are we using? Are we pulling these directly from CRSP database or are we calculating these? CRSP has Equal-Weighted and Value-Weighted Returns. Which should we use?

Backtesting

• Which one exactly is the result we are supposed to beat? The "value decile intersection, 50 stocks" returns, or what?
• Do we have to beat him on total return since 1963? Or by 10-yr returns, (average or compound)? Sharpe ratio?
• Do we have any restriction on the number of stocks in the portfolio?
• Are we rebalancing on an annual basis?

Final project

• Where is the assignment posted?
• How much flexibility do we have in designing our project?
• Which one exactly is the result we are supposed to beat? The "value decile intersection, 50 stocks" returns, or what?
• Do we have to beat him on total return since 1963? Or by 10-yr returns, (average or compound)? Sharp ratio?
• Do we have any restriction on the number of stocks in the portfolio?
• Are we rebalancing on an annual basis?

Sorry, TWF refers to Thompson, Williams and Findlay, as opposed to Graham and Zweig; I would recommend that you skim them as you would any book before you really get into it, just go thru the Ch 1-8 and see what they're about.

Generally we would like each group to represent the STAT 486/686 enrollment. In unique cases with a very strong UG, they may participate in a graduate class group; the grading will be adjusted for their being enrolled in STAT 486.

It should be professionally done, i.e., type up a paragraph, check your spelling, etc. However, we will not cont off for handwritten on the 0th assignment, but we will for poor grammar, etc.

It would be impractical to ask you to provide the constituent companies for this assignment, just the counts are all you need, and these are available from all timeframes of the CRSP stock indexes.

Holding period return (RET) includes dividends. Using this variable will automatically reinvest your dividend payouts when they're received. RETX excludes dividends, and is what one calculates when one calculates Pt/Pt-1

It is true that we do not have subscription for that Compustat database, which would be used for indexes other than the CRSP indexes. Daily data for CRSP indexes is available from CRSP; however, only return data is provided (no levels). You would have to create your own level data from a reference level and string together the returns. Too much work. For this reason and for indexes other than the CRSP indexes (such as the DJIA, S&P 100, Shanghai, etc.) you will need to get the price data from any number of financial data websites (Yahoo Finance, Google Finance, ADVFN, etc.), or from the vendor. All have download tools.

Ideally, yes. However, it is possible that you might not have formed a group yet by the due dat e, in which case you should complete the assignment individually. The other assignments must be completed in groups.

In this exercise let the data speak to you. Pull the the NYSE/AMEX/NASDAQ daily index data and find out which is the FTDOY and LTDOY. And then just count the number of trading days in that year.

These 27 stocks started with a list of the 30 current DOW stocks (and possibly historical DOW stocks) which had continuously traded since 1970; these weren't enough so I went to the SP100 and added a couple more; that wasn't enough so I went to the SP500 and did the same. Eventually I amassed a collection of 27 stocks. These are the S486.

Unfortunately, past data is all one ever has in this discipline, so that's what we have to use. The Expected Returns recommended text discusses this. The EMH posits that there is no past data that has any bearing on future performance; that's what we are questioning in this course.

First, there are about 252 trading days in a calendar year for equities. We generally obtain this return and report it as an annual return. These dates are the ones that WRDS/CRSP provides without modification.

Sorry, this was unclear. It should read: "You should find benchmark annual returns for DJIA, S&P500, NYSE/AMEX, NYSE/AMEX/NASDAQ, RUT3000, RUT2000, etc. You should obtain returns for level data (no dividends) as well as dividends included (this is called Total Return). You should also try and get benchmark return data for Market-Weighted as well as Equal-weighted. This is most easily accomplished using WRDS/CRSP."

Our data goes from FTDOY to LTDOY, usually 1/2/xx thru 12/31/xx, so there should be very little difference between our results, although this sort of precision from you is what I was hoping for. If you check I'll bet Bloomberg is really based on LTDOY, BTW. Indeed, RUT3000/RUT2000 are not on CRSP, although the DOW is, but it does not go back very far. Our data comes from another service like Bloomberg which accounts for all dividends, including mergers/spinoffs and liquidating dividends. I would be very interested in seeing how these compare wit the Bloomberg data. I don't think Bloomberg goes back much past 1985, but perhaps you found that it did. You can also get the data directly from Russell.

We are interested in benchmark RETURNS for these periods, so we do not care about an index level. The only thing you would use an index level for (besides stochastic modeling) would be to calculate the daily/monthly/annual returns! So when CRSP provides the returns, all the work is done for you! Just use the CRSP indexes and use their return calculations. The S&P website might be limited or require registration, but since CRSP has this data there is really no need.

Unfortunately, CRSP is making it harder to get Dow data; their Dow index only goes back to 1987, and only goes forward to 2007 for some reason; I suppose they stopped providing. What's worse, YAHOO no longer allows you to download all historical data in a spreadsheet. However, the Dow website does have data for the regular DJIA and the total return, so take that back as far as you can. You'll have to use daily returns to build up the annual return; you could do a sensitivity analysis to see how the daily granularity affects the end of year return, but that is not required.

WRDS provides NASDAQ data back through 1987 or so; YAHOO provides R2K thru 1987, and NASDAQ composite data back to 1971. YOu might have to get total return data from the NASDAQ or Russell websites. You don't have to include every index to so well on this project. You do not a good benchmark for the total stock market, hence we use NYSE/AMEX, maybe the Russell 3000, and also popluar indexes such as Dow and S&P500. Index data from the data providers such as DOW, S&P and RUssell which includes dividends is normally called "Total Return" and the total return CAGR's can be computed directly from these.

I'm not sure we do "normalizing" on our time series data, the observations being just either an varibale level, or relative change, or category, etc. This is seen over time.

Thanks for the question, fear not! You just need to get annual stock market index returns, available from numerous sources. In CRSP there is S&P, Dow, as well as those you mentioned.

Actually in this all this work would be unnecessary. You just select "Search Entire Database" option for the screen input. This will pull all PERMNO (or GVKEY, depending if you are in CRSP or Compustat.) This the fiscal year exercise, you should be in Compustat. You will then be provided whatever data variables you requested for ALL companies.

This does not refer to a particular strategy, but is just cluing you in that the standard error of the CAGR needs to be calculated. So if I have a strategy and it outperform by 1%, and the standard error is 2%, then I have not shown any significant outperformance.

On 1/20/1997 the Archipelago ECN began trading. In 2005, Archipelago Holdings, the owner of ArcaEx, bought the Pacific Stock Exchange; Archipelago Holdings merged with NYSE Group in April 2006 in a reverse takeover and the business was renamed NYSE Arca. In WRDS, the NYSE Arca daily and monthly data were added in July 2007 for securities with primary listings on that exchange. NYSE Arca coverage begins on March 8, 2006 and appeared in CRSP as a separate index with Equal and value-weighted returns. Sometime after 2013, WRDS stopped providing a separate index for ARCA and those trades are now reflected in the NYSE index. Current exchanges realtime volume is available at https://iextrading.com/apps/market/

This is the inner (dot) product of the weights and the returns, I.e., sum(wi*ri)

This would be the realized portfolio return at the end of the year versus the beginning, I.e. the dot product of the weights with the annual returns for stocks in the portfolio.

Remember, in the problem statement you are reminded you'll have to calculate the market cap from CRSP's PRCC and SHROUT, since the CS or CCM coverage for MKTVALT is not good (and ZERO for years before 1950; be sure to note this in your coverage analysis). Note that split adjustments are not needed since both PRCC and SHROUT will adjust automatically.

Compustat/CCM variables have their own naming styles, that's why you have to work with the data and current financial statements to make sure you are using the correct variable name. ACT I believe is total current assets; total assets would be another variable. (TA includes non-current assets.)

For the Mini-Project we do not need to make portfolios at all, we are only trying to get the distribution of the data you might use in the final project.

The total return index includes dividends. You have to get total return data from DJIA website (requires signup). Sometimes people report 'total return" over the entire period (r1+r2+...rn) which is not very meaningful. It equals n times the average return, and the average return overstates the CAGR. This may be what you saw in some of the sample projects. We however are interested in CAGR, and maybe drawdowns, etc.

One could debate which benchmark to use. Since this is a dividend play, one could argue to use a total return index. However, this index is almost never quoted, only the DJIA (without dividends). As of 9/30/2016, the DJIA closed at 18,308 and the Total Return Index closed at 38,302. So either choice is fine, but know that claims of outperformance would usualy be versus the simple DJIA or S&P 500.

This assignment was discontinued circa 2015.

It is table 14-3 in the graham book you purchased for this class.

Great question. See Graham's excellent chapter on Earnings. I believe he would use diluted and excluding extraordinary items.

For debt you should use LTD total.

If a company delists, then if you can determine on which exchange it subsequently was listed on, then you can continue to track the stock. If it goes private, then you should determine the terms of the private stock exchange and assume that you are still invested in the private stock, which then has a problem due to lack of publicly traded prices. If the public shareholders are given cash in lieu of private stock, then you should reinvest equally amongst the remaining 4 equities on the date of the distribution (i.e., do not put into money market fund). How you handle the LTCG/LTCL is your choice as regards net funds to reinvest.

You should read the commentary in the next chapter. However, it is not really clear from the text in pages 338 and 371 exactly how they obtain their numbers. You could calculate the percent change of diluted EPS last FY vs. average of last 3 years; for 1970 for example, you would calculate (EPS.70-average(EPX; 1968, 1969, 1970)/EPS.70

All we are looking for is a new table for 12/31/18 for all 5 stocks. Only 4 have survived to today.

As you will find from the 1970 table, Swift WAS in the DJIA; it was later acquired by Beatrice, which then went private in 1986. You would then use your cash proceeds from Beatrice and reinvest in the remaining 4 stocks. Or not, leave it in money market (not a good choice).

These are tricky, even if you are familiar with CRSP/Compustat. As always, when variable-hunting, you should experiment on data you know to be correct, like 10-K data, or perhaps published data from Google or Yahoo, and try and zero in on the correct variables. And, when possible, do not make your own formulas, you should use Compustat, e.g., for EPS I would use EPXPX, or possible EPSPI, but not try and calculate from Net Income ($M)/Shares (M). Note that PE is EPS/Common Shares, and you were calculating EPS. So you would need to use Price/EPSPX, or better, find a PE already calculated.

You could also use percent returns as well as log returns. Since the log is a monotone function, then the inference should be the same. We suggest you try it both ways and report on the difference for extra credit.

You are trying to reproduce the overall results of the Max Median (MM) process as outlined in the provided papers. The main point is to get the overall CAGR (along with risk statistics) using the MM methodology. You will note that the MM strategy provides CAGR almost equal to the EW index but with a portfolio of only 20 stocks. That is huge.

Backtesting from 1980 is an appropriate lookback period for this assignment.

DO NOT type in PERMNO's in CCM. Use the PERMNO file provided in Owl-Space STAT 486/686 001 Sp14 Resources / Projects / MaxMedian.

You do not COMPUTE the daily returns, you DOWNLOAD them useing RET or RETX. CFACPR is a factor which takes into account splits so you could calculate a split-adjusted price, but it does not include dividends. For that you need to use CRSP's return variable RET. It includes both splits, spinoffs, and dividends into the returns. RETX does the same but without dividends.

Sorry to hear you're having problems. These are big files, others are around 1GB or so. Here you really kind of have to leave the Excel world; it generally will not able to deal with it. Office 2007 can only handle 1 million rows.... R has problems enough, but will work. Much bigger files than these are found in the business world, but most everyone uses SQL databases. We typically query the database to get toy datasets for Excel analysis. If you cannot use R or SAS, then you might have to split the exercise into 30 downloads for only one year?

BTW, How is Bloomberg getting the data? I thought it only went back to around 1985? Please write-up your Bloomberg procedures since this is unique and we would appreciate seeing it.

The "EqualWeight" return is based on the MaxMedian portfolio weight of 0.05 for each stock; the "MedianWeightReturn" is indeed a weight based on the median return, but in this assignment and in various papers we use the "Equal Weight" return. The "Equal Weight" return does NOT refer to the equal-weighted S&P500.

You would be starting with $1 nominal investment in 1970. Many people use $100,000 but that makes calculating CAGR's from terminal values unnecessarily complicated. For example, if a strategy grows $1 to $100 in 48 years (1970-2017), its CAGR is 10.07%. Of course, if you were doing a trading-cost analysis, which we wish more people would do, you would have to start with some more reasonable investment size.

When you say that you found a variable called "Market Value", what was the variable? Was it MKVALT? Was it MKVALTQ? Did you download some data and see if it matched Yahoo Finance? This is sort of how you figure these out. I looked at MKVALT and saw that it appears to quote in terms of $M, so I think this is what you want.

You could use OANCF, Operating Activities - Net Cash Flow, in the cash flow items input section of the WDRS CCM GUI. This item represents the net change in cash from all items classified in the Operating Activities section on a Statement of Cash Flows. Of course you might notice that coverage might be poor for this variable depending on how far back you are checking, in which case you could use O'Shaughnessy's construct, IB + DP. This calculated CF will have much better coverage. A spot check for IBM and GM and other companies reveals that OANCF is missing before FY 1988. A full universe pull (23MB) from 1970-2016 shows 88% coverage for DP and 93% coverage for IB, with a calculated CF therefore exhibiting 93% coverage, with only 60% coverage for OANCF.

No, XOPR is total operating expenses; it represents the sum of: Cost of Goods Sold (COGS) and Selling, General and Administrative Expenses (XSGA), it is an operating expense, not a cashflow item. When in doubt, always check again published 10-K's. When you do you can see that XOPR is clearly not CFOPS, or that IB+DP is very clearly only an approximation. (Note there are difference in DP and DPC, where the latter is a cash flow item - to resolve the discrepancy you'd have to check to 10-K's to see exactly which is correct to use.) The OANCF matches the 10-K data but not necessarly online sources such as YHOO and ADVFN because they would be reporting TTM rather than published numbers since these are more relevant for investors.

Welcome to the real world, this is how it is most of the time. If you are talking about dates such 20011231, you have to parse it and turn it into a date! There would be lots of ways to do this, in Excel you could use:
CRSP dates: 19800522 (Say this is E62)
text formula: 05/22/80 (F62)
datenum: 05/22/80 (G62)
E62: 19800522
F62: =MID(TEXT(E62,0), 5, 2)&"/"&RIGHT(TEXT(E62,0),2)&"/"&RIGHT(LEFT(TEXT(E62,0),4),2)
G62: =VALUE(F62)
You could just make a single formula:
=value(MID(TEXT(E62,0), 5, 2)&"/"&RIGHT(TEXT(E62,0),2)&"/"&RIGHT(LEFT(TEXT(E62,0),4),2))

You might start maintaining a spreadsheet of these sort of utilities since they will continue to arise. This is true whether you go into business or academia.

We assume you mean parts (1 and 2) of the question pertaining to the CRSP index volatility. We're interested in index returns so you may use the CRSP-provided returns (i.e., DO NOT calculate the returns yourself). Since the EW indexes are hard to find in the real world, you should use VW (i.e., VWRETD or VWRETX; the dividends should not affect the volatility calculations very much).

You're not trying really to beat O'Shaughnessy, just the market benchmarks of your choice, such as SP500, AMEX/NYSE, etc.  We recommend your trying to understand his results and statistics, which you should incorporate into your report.

Of course the further back you go the more stable the results will be, but things also change!  You should have a backtest period going back at least to 1975, but much past that might not be all that helpful if you are really trying to do something going forward.

You should employ a manageable number, such as 20-30-50, etc., unless you are proposing some high-frequency trading system, which is not really in the spirit of this course.

Yes, assume tax-deferred account such as IRA, etc.

The final project is assigned in the Assignments section of Canvas, with reference to the powerpoint slides in Files/Projects section. These are the slides we discussed in class.

Lots. But the more time you spend searching for and cleaning your data, the less time you'll have to actually work with it. Time management will be key, whatever form your final project takes.

You're not trying really to beat O’Shaughnessy, just the market benchmarks of your choice, such as SP500, AMEX/NYSE, etc. We recommend your trying to understand his results and statistics, which you should incorporate into your report.

Of course the further back you go the more stable the results will be, but things also chance! You should have a backtest period gogin back at least to 1975, but much past that might not be all that helpful if you are really trying to do something gogin forward.

You should employ a manageable number, such as 20-30-50, etc., unless you are proposing some high-frequency trading system, which is not really in the spirit of this course.

Yes, assume tax-deferred account such as IRA, etc.