you're right, it's not very polite for someone to scroll when you're having a conversation with them.
a good way to respond might be to pause and let them finish.
plus, if they have a good reason, like taking a note or looking up something relevant, it gives them time to do so.