Bdtyktl

I am GMing a homebrew story and was trying to think of an enemy that my party could encounter that would be more story driven than just one that gets more dice to hurt them with. I have been fascinated with the idea of a revenant for a while, but as nobody in my party has murdered anyone important, and I don't wanna be a jerk and just throw a random vengeful bandit at them.

Then I got to thinking... What if my bbeg got a revenant after him, but instead of killing it, used modify memory to make the revenant "think" it's new target was one of my party members.

In this case I'm asking two things: can modify memory bypass a revenant's ability to always know their target, and, if so, do they get the perks against the new target, like bonus damage, tracking, etc?

No, because Revenants cannot be charmed.

When a creature is targeted by modify memory, it must succeed a Wisdom saving throw or become charmed.

On a failed save, the target becomes charmed by you for the duration.

...

While this charm lasts...

Unfortunately for your BBEG, according to the Monster Manual p239, the Revenant is immune to the charmed condition. So even on a failed saving throw, the Revenant would not be charmed by the modify memory spell.

You can't use modify memory, but if you found another way...

From MikeQ's answer, a revenant is immune to the charmed condition, which makes it immune to the spell modify memory. But if you were to find some other way to alter a revenant's memory, one which bypasses its immunity to the charmed condition (wish is the obvious answer, but there are better wishes the BBEG can make so you should probably devise a more original solution. Maybe some ability which lets the BBEG ignore the charmed immunity on undead, if he's a necromancer), then there still remains the question of whether an altered memory will change the revenant's sworn targets.

The description of the revenant in the Monster Manual (p.259) says that

No magic can hide a creature pursued by a revenant.

However, it says nothing about magic's ability to make the revenant pursue a different creature. On one hand, one could argue that a revenant is fueled by a soul's desire for revenge, regardless of whether it is correct or not, and that it only matters who the revenant believes has wronged it, regardless of whether the target is actually guilty. This could apply if revenants emerged from some process independent of external agents. This view is supported by the first paragraph of the revenant's description, which describes the soul acting by its own agency. If this is the case, then altering memories could alter a revenant's target.

On the other hand, it is possible that a revenant's target is fixed from the moment it is raised into undeath. The following two paragraphs of the revenant's description, Hunger for Revenge and Divine Justice, hint at this powerful supernatural link between the revenant and its target. If revenants are raised by a god so it can enact divine justice, then no mortal effort could alter the target. If this is the case, then altering memories will not alter a revenant's target, but it might leave the revenant rather confused.

So either case can be argued to be correct. Whichever way you rule, you should be consistent. If the PCs learn that revenants can have their target changed via whatever means the BBEG used, they might use this tactic later if it uses abilities and items which PCs acquire, although since NPCs are not PCs the BBEG could very likely be using some ability which is beyond what is available to PCs.

But since there is no rules-as-written answer to the revenant's behaviour (at least, not that I can glean from the Monster Manual), the real question becomes 'which option makes for a better story?' In my opinion, having the BBEG trick a revenant into hunting the party by using memory modification is a cool story and you should do it. Note also that there are multiple possible outcomes to this plot. Beyond combat, the player characters might be able to detect and reverse the curse which was placed upon the revenant, at which point they would gain a useful ally and strong lead towards the BBEG. This sounds like a brilliant plot arc and you should do it. Even if the official revenant description opposed this plan, you could overrule it to make for a better story.