Is more efficiency a good thing anyway?

This really begs at the question are Capital Markets good? I would say a resounding yes. That's not to say that our capital markets are perfect or couldn't be improved upon, but its better than not having an equity market.

Now what does the stock market do for the company and the investor and how is the stock market and in particularly the accuracy of the price at any one moment have to relate to that. Equity is one way for a company to fund itself. You sell a portion of company ownership for cash in an IPO or another offering. Then going forward you can also use the equity you retain and the current price in the market as one of your assets. For an investor equity is part ownership in the future cash flow of the company. If the company has more money they might issue a higher dividend or do a stock buyback thereby increasing the value of what you own.

The Efficiency of Price Discovery

I can try to answer the question in the context of something material happening to JCP. Let's say their holiday sales are released and it's bad. If you are an owner of JCP stock you probably want to compare your expectation of future cash flow with the current price of the stock. Would you rather sell now for cash or hold it for future appreciation. If you think you are much better off selling now then you go to the market and sell. This is going to affect the price in the market, because the level at which someone was willing to buy at may no longer be there. Only so many people are going to be willing to buy at price X until they move the price down to X-1. So the stock price moves down eventually it stops moving down and market participants are willing to push it back up again. This process is called price discovery and it's something that market makers help to achieve. Now here's the kicker, if you as an investor want the option to be able to buy or sell in a reasonable time after material information is released then price discovery has to happen! The price can't sit at it's last price and everyone is allowed to execute at that price. Clearly that doesn't work because who is actually going to receive the sold securities? There has to be real time buyers and sellers to facilitate this movement.

The Efficiency of Alternate Markets

Now above I described the efficiency of price discovery and how market makers help with the process. Now there is also the efficiency of price being equal across different exchanges. This is easier to understand, if there is a resting offer to sell BAC for 12.99 at NYSE and a resting bid to buy BAC at 13.00 at NASDAQ then it only makes sense that someone is going to come along and try to buy BAC at 12.99 at NYSE and sell it for 13.00 at NASDAQ. If you want more than one market center then this has to happen! And you do want more than one market center, because competing market centers help to keep exchange fees low via competition among the exchanges. Not to mention that competition among the exchanges helps them to improve their market technology being able to handle more bandwidth and coming up with new trading technologies that help market participants. The CME futures market is an interesting counterexample to the US equity markets, because although there are about 7 lit equity markets there are only 2 big futures markets. So CME has a lot more power. I cannot say definitively whether it's cheaper to trade in equities or futures because its like comparing apples to oranges. With CME they are also a central clearing house so they help facilitate the risk if a counter-party defaults and cannot fulfill an order (imagine Smithfield goes broke and cannot deliver on its pork futures). The problems with trading equities are not equivalent.

The Efficiency of Related Securities

Many of you are likely familiar with ETFs. Exchange Traded Funds are a security backed by a corporation that is designed to back this security with specific other securities. The simplest one is SPY, which is made up of the securities of the S&P 500. So the person who issues SPY buys the basket of stocks that relate to the S&P 500 then they issues an equivalent amount of SPY shares. When they get dividends from the basket that they own they issue dividends to the owners of SPY. Now why is this a good thing, because owning SPY instead of owning 500 stocks is a cheaper way to diversify your portfolio. Why do you want to diversify your portfolio, because mathematically you have higher expected returns with lower risk (check out modern portfolio theory). Now just with the alternate exchange example above, if I could calculate the fair price of SPY and I see that the price SPY is being traded lower than the cost of its constituents then I'm going to buy SPY and sell the constituents. This pushes the price SPY to the true price of SPY.

Aside, now one other thing that could happen that is different than it works today is there could be mini auctions every few milliseconds instead of continual trading. I wouldn't necessarily object to this idea. The idea is similar to what happens in the morning or closing auction. Everyone who wants to buy or sell a stock issues the price they would buy or sell at. If the market crosses then the exchange finds the fair value and matches as many buyer and sellers as possible at the price. You would have to match everyone pro rata though, because if the match still occurred based on time priority you wouldn't have solved any of the negative sides of the industry such as high technology cost.

I don't consider the high cost of technology to be a huge negative of the trading industry. What automated trading has accomplished is take the job of 1000's of human market makers and automate them. The old human market makers used to make a lot of money for what they did. They also couldn't handle more than a few stocks at a time so there had to be a lot of them. A modern moderately sized team can handle the entire universe of stocks though. This should make it clear why it's cheaper to trade now than it ever has been. In the process they have brought the spread between buyers and sellers down to penny and subpenny levels as well as enable you as an investor to click on your Scotttrade account and actually own the shares you want to own before the page even finishes loading. That's amazing if you think about it.

In response to the this

HFT often works in a way that is agnostic to the market itself.

In my experience strategies do not work in a way that is agnostic to the market. Usually strategies are formulated off of some premise related to the market. One example would be if there are more orders resting on the buy side then the market might move up. Or if the price has been moving up keep moving it up until reach some reversion point that you have calibrated by backtesting. These two strategies here may seem like they are agnostic to the fundamental price information of the stock, but they are not agnostic to the market itself. As a market maker you want to facilitate trading between buyers and sellers and using information that is coming into the market (like resting orders or recent trades) allows you to get a better idea of what the market participants are going to do next.

Now I imagine that there are other things that traders could do that are way out there. Attempting to correlate twitter traffic to price fluctuations seems particularly hard to me. Similarly doing anything that is deep machine learning like Neural networks or Genetic Algorithms run a bigger risk in my opinion because if the premise that you trained your learning algorithm on changes then your strategy might work in the complete wrong way. As an analogy imagine that you trained a neural network facial recognition algorithm in a room with only yellow incandescent light and then you bring in a test corpus to see if your training has worked. It will probably work pretty well but what if the yellow light gets changed to fluorescent light. If it's a neural network it will be hard to know if it will continue working. Similarly in the market there might be a certain market sentiment all of 2013 but if something big happens in 2014 like a sovereign default or another externality you have no idea how the market is going to behave moving forward. If you at least tie your strategy to something simple like order flow you are likely to run less risk.

EDIT: Another thought came to me: Is more efficiency a good thing anyway? I can imagine that some inefficiency between market changes and stock prices could actually be beneficial for the health of the economy. Inertia is a stabilizing factor of many systems.

The slight inefficiency of the market creates a small time buffer that means a lot of the meaningless fluctuations of the market can be ignored (have to be), which would seem to lend stability. HFT lets computers exploit these fluctuations instead, perhaps blowing them out of proportion.

And the other thing you said is true as well -- if you have the computing power it seems much more foolproof to work the market via HFT than it would be through doing actual investment research and accumulating knowledge.