We've all been there, I'm guessing. Swift is a particularly interesting case. It's a very modern language with a very modern face. It looks welcoming to people new to programming, but also familiar yet refreshing to veteran programmers. It sometimes feels as if Swift hands you the world on a silver platter only for it to yank it back, firlmy asking you whether you thoroughly cleaned your hands first.
This isn't a problem with the language nor with the users, in my opinion. Every language has its own ideology and requires a different mindset. Every language has its own little rulebook. Swift is no different. It may seem Swift's rulebook is bigger and more strict than of other languages. But I think we have to accept this as a trade-off for the language's modernism and safety it provides.
Those problems you described with protocols et al? I've been there as well, for real. And after 7+ years with the language everything makes sense now, where I can catch these gotchas the moment I'm imagining the code I want to write. I don't know if this is due to experience, being delusional or stockholm syndrome... but I swear I can read Swift code like Neo can read the Matrix. It just takes some time :)
My real frustrations with Swift lie beyond the language. The fact that Swift is still super inaccessible on platforms besides macOS. The fact that IDE support - other than Xcode - is still pure bogus. (SourceKit you say? Pff, even a Russian dashcam compilation has less crashes than SourceKit experiences in an hour). The fact the Windows version has to be taken care off by the community. The fact that Swift has only evolved this much despite being developed by a $2 trillion dollar company.