From a data standpoint, I think you are doing the right thing. This will enable you to see a history of anyone who was ever assigned to an activity, regardless of whether or not they actually served/posted attendance. This is useful to see who registered for events, but didn't make it or who could be scheduled to work for an activity, but never actually has. If you make the record inactive, but the person changes their mind later, you can re-activate that record without too much trouble.
The only big reasons I've heard for removing assignments are incorrect assignments (testing, data entry error, etc) or just having too many old assignments that aren't relevant anymore and are affecting portal usage or reports. I still tend to argue against removing anything just because of the possible data value at a later point. I've been in too many situations where the decision was made to not keep data and then it was realized that we actually wanted/needed that data for analysis later on. At that point, the backups were too old to retrieve anything past a couple of months and we ended up paying a pretty large sum to have someone process the data based on what we had retained.
In Him,
-Peter Schott