If you felt a connection with the drums, go for it. You can always go back and learn guitar after you have learned your "soul instrument."
But know that no instrument will be EASY to learn. Some may not be as hard as others, but it won't be EASY. I play drums, piano, guitar, french horn, and bass. None of them were easy to learn, but it was well worth it in the long run.