Remove SCC shutdown hook
SCC uses a shutdown hook to close C* session cache. It caused a lot of problems in the past, e.g. SPARKC-620. The hook should be replaced with a more reliable solution. Adding a close method to SCC api seams like a reasonable thing to do.
Unfortunately Spark DSv2 API doesn't come with a full catalog lifecycle management - it only has initialize method. Up to a discussion.