Doing some more digging. it looks like both the 'updated' and 'lastActivityDate' are not consistent at all.
Voting on an idea changes "lastActivityDate" however, "updated" does not update
Changing categories changes "updated" but does not change "lastActivityDate"
and filter=modificationDate doesn't use either "updated" or "lastActivityDate", it uses some other date that's not available through the REST API.
What gives? How is someone supposed to sort or filter when there are 2 different activity variables that are arbitrarily updated and the real modified date is hidden?