Note for next iteration: Instead of literally modeling schema columns and only hiding columns on CQL generation, we need to model the logical table. See https://issues.apache.org/jira/browse/CASSANDRA-9813
The next step in CASSANDRA-6717 is to move secondary index metadata out of column definitions and into a separate table. I've pushed a preliminary branch with a new system_schema.indexes table here.
Things to note, further changes are imminent (i.e. before 3.0 beta 1), which will include support for multiple indexes on a column, multiple target columns for a given index and real per-row indexes, which don't target any specific columns. The new indexes table is designed for those new features, but they're not actually implemented in that branch.
This ended up spread over multiple commits. Trying to break into phases for review:
Add server version detection, normalize schema event processing, refactor schema parsing:
Metadata file updates from that point:
Mike has reviewed some of this over the course of C* betas and rc1.