The downsides are resource usage. For the sending server, it's got to deliver public activities to every server it's interacted with. That could be thousands or tens of thousands of servers! If that's happening for every public activity, it's a real drag on resources.
The other downside is for the receiving servers. There are millions of actors on the Fediverse, publishing all kinds of activities -- new content, likes, shares, and so on. The receiving servers would be getting that full firehose to their `sharedInbox` endpoint -- the vast majority of which was not interesting to any users on that server.
If you have a fediverse account, you can quote this note from your own instance. Search https://cosocial.ca/users/evan/statuses/115894953736477393 on your instance and quote it. (Note that quoting is not supported in Mastodon.)