I am not really sure why Danes would hold Swedes responsible for not understanding Danish. I worked with a girl who was Danish and had just moved to Stockholm and she had a hard time understanding Swedes. She spoke English to them. So it might be that Danes in general also have a hard time understanding Swedes. I think there is a passive aggressiveness going on between the two cultures due their complex history.
I am not a native Swedish speaker, but I agree that spoken Danish is more difficult to understand. But I still have an easier time than my boyfriend who is a native speaker. I think its because I watched a lot of Danish and Norwegian movies while I was learning Swedish (since they were available on Netflix).
It depends on many things. Where came the Danes from? If they came from Jylland they were not used to Swedish and could have problems to understand. If they were native from Shelland, they should understand.
Then there are people everywhere who doesn't understand more than they want to.
It depends of age too. Shelland and Skåne have so much in common, so we talk our own languages with each other without any problems.