Game Within a Game
A series of traits for a digital program converge. The program isn't meant for productivity per se, but to entertain. This is a game. A number of games sharing these traits are made. This is a genre. The traits might be called mechanics. If the mechanics of two genres meet, we might be looking at a mash-up, or one genre using aspects of another.
A grappling hook in a sidescroller is a mechanic, but not from another genre. A whole levelling system of stats and acquired skills, seems to be from the RPG genre, but can be used in the sidescroller genre, like in Castlevania Symphony of the Night. If enemies drop physical experience points to pick up, and your equipped weapon levels up, but also loses levels when you are damaged, that's a system from outside the sidescroller genre.
I'll just list a few systems. They might be battle systems, or meta progression, or something in between. Maybe you can pick an element and insert somewhere else, or think about how it's not often reused. They should be supplemental systems, and not core.
TRANSFORMATION
Touch (eat) something to transform for a while. Pac-Man did it, and it's good. It's barely a system yet, and it's a bit hard to define a genre for Pac-Man, so it's hard to say if it's outside the genre. Mario built upon it.
Old mario games are generally 1 health point, instant death. But pick up an upgrade, and you transform. A mushroom turns you into the titular Big-Tall Mario. This also grants you an extra health point. The many other upgrades convey abilities and two extra health points. If you take damage you either revert to Larger-than-life Mario, or to Mario without human rights.
Kirby's Adventure moves the complexity elsewhere. Now you eat the enemies to upgrade. It feels different, but is essentially the same. Honestly why haven't any shooters done this? I know it seems obvious that weapons are picked up and changed on the fly... But why not try something different?
ZELDA MAGIC
This is a funny one. In some of the games, Link gets a green bar of magic, which limits the usage for some of his tools. In Wind Waker, it signifies that he's becoming something of a wizard, Harry, and that only those with magical potential can use these items. It's a nice narrative twist on what are ostensibly just magic points.
DYNAMIC EXPERIENCE POINTS
This classic is so clever. Quote has a few varied guns available. Each is mechanically different. Enemies drop physical experience points that quickly disappear, you have to touch them to increase the level of your currently equipped weapon. Once you reach a threshold, it'll upgrade, and behave differently (most often better). But careful: taking damage makes the weapon lose experience too! Wonderful active risk/reward in this one.
GUNSTAR COCKTAIL HEROES
This game has 4 pretty different guns available. But pick up another, and you can either use the other, or combine them for a new kind of gun. Works with the same type as well, for a total of 14 guns! And it's all done in the heat of some pretty intense fighting.
GEARS OF WAR
Just a little twist: if you time a button press correctly while reloading, it finishes instantly. Time it correctly, it takes longer. Don't do anything, it takes the normal amount of time. Something to gamble, something to master.
TIME ACTION MAGIC RESOURCE
4 Heroes of Light, Bravely Default, and Triangle Strategy: just a little action point currency that you manipulate directly and indirectly. Simplifies the unwieldy abstraction of MP, makes the time management of actions more concrete. ... Why aren't there shooters with action points?
COMBAT CRAFTING
I wanna give my own The Girl Who Kicked a Rabbit a little shout out here, but clearly The Last of Us had already done it, and very well. That said, it shows how a seemingly similar system works differently in a cover-based shooter, compared to a turn-based puzzle game. The idea is that you can craft helpful upgrades during combat, from knickknacks you find before and during combat, from the environment or fallen enemies themselves. It's stressful and thrilling in The Last of Us, and strategic and surprising in The Girl Who Kicked a Rabbit.
KILL TO MOVE, MOVE TO KILL
Movement is everywhere, but most often it's either completely free or all about limitations. Maybe you can jump, roll, dash, double jump, triple jump, backflip, crouch-slide jump, wall kick, re-jump, ground pound, pound bounce, swing, rapel, ski dive, float, glide, fly, swim, or boost. But they're just actions you do. You can get good at them, and they might be hard to pull off.
On the other side, we got grids, action points, movement points, stamina, charges, energy. Systems might occur here, but in most cases, they're not sub systems to a main gameplay, but the entire focus.
Doom's (2016) system where you weaken enemies, get close, finish them off, they drop healing items, and you continue. Melee is risky, distance is death. Now that's a beautiful little sub system.
I tried adding a system of limited movement actions that could be regained by killing, in Like a Pig. It's still a good idea, since it forces deliberate movement. Yet in practice, you don't connect your sudden stops with running out of leg fuel, and there's no connection between killing, and getting to move again. I haven't given up on solving it though.
ANTHEM'S DELIBERATE COMBOS
In Anthem, some guns have a primer effect: shoot enemies until they are affected. Once they are, shoot them with a gun with the detonator effect. Lots of effects and be combined, and it encourages experimentation.
MOTHER 3 MUSIC BATTLES
I don't like these, cause I suck at them. That said, I madly respect them: tap the confirm button during a normal attack. Time it with the beat of the music to make a 16 hit combo. Just because I suck, doesn't make it not brilliant.
PIKMIN. JUST ALL OF IT
Oh look at me, I'm just John Fucking Mario, coming to show all you idiots how to make an entire RTS on a console: You're the cursor, your units are both bullets and workers. They have easily distinguishable traits, and need constant, intuitive micro management. It's incredibly difficult to make something so complex yet so simple and intuitive. Doesn't get half the praise it deserves.
PIKMIN AGAIN
You know what? Let's do another: Pikmin 3 has bingo battles. You race to collect junk on your respective bingo cards. You win by getting an entire line. It transposes the simple concept from other place into this multiplayer mode, and it works wonderfully.
GEEZ HOMER I THOUGHT SOMEBODY WITH TWO HEALTH BARS WOULD BE PRETTY HAPPY
God of War uses the idea of a stagger-metre delightfully, though it's most often not necessary. Either you deal damage with weapons, or you stagger with your fists. Lots of Final Fantasy games use staggering as well, from a whole ecosystem of "use Assess to discover a pressure point, then pound it until they stagger " in FF7 remakes, to the chain bonus builder in FF13. There's also the staggering locks in Octopath Traveller, where you have to hit each lock type to induce a stagger state. It's very clever, but in practice makes even the simplest enemy a tedious task. Dissidia Final Fantasy does something like it, but it's too complex to even begin describing.
ALTERNATIVE HEALTH
What if you don't even have health? In Yoshi's Island, health is sorta the baby you drag around. Get hit, and your health-baby will float away. Catch it quick!
RECHARGABLE HEALTH
I don't know who invented it, but Halo popularised it. Before, if you lost health in a westernly developed game, you'd have to scrounge for health kits. Suddenly, you just find cover and wait a bit. Honestly, why hasn't anyone used this in a side-scrolling game, or a vertical shooter?
REVERSE SHOT
In Sin & Punishment 2, you can reverse all missiles by striking them with a melee attack. Good luck pulling it off consistently though! Very simple, satisfying, and dangerous.
CHARGING
Metroid, Zelda, MegaMan. Hold a button. If you hold it for a while, you get a stronger or special attack. Not quite a system yet. Cave Story has more than one stage, so there's a risk of charging and not dealing damage while charging, and possibly wasting your time. Fighting games do the same, though the mechanic feels inherent to the genre, compared to somehow charging energy in Zelda. Another World built upon it, with wildly different mechanics upon charging. Videoball based its shooting system upon Another World, but finally turned it into a system: tap to shoot, charge for continuous push, long charge for smash, which is reversible, and over charge for shield, which is a consolation for wasting your charge. It ought to exist in an actual shooter (and I take it that it was supposed to in Shadows of the Damned).
TOOLS WITH SPELLS
This one is so widely used that no one considers it anymore, but let's just try and separate the system from the games. The original real-time strategy games like Dune 2, had units you moved around, and they could attack. I might be misremembering, but I don't recall them having extra skills that could be used occasionally. Warcraft changed this, by adding a set of so-called spells to various units. No longer were your units just pens and marquee tools that could kill. Each unit suddenly had a repertoire of sub-tools. This system within a system continues today in many similar games like Dota 2, but also shooters, where you select a character to play as, and it is defined by a few select abilities and equipment from the outset.
SHOPPING
Speaking of shooters and how your character is defined, let's round this off, by looking at Counter-Strike, a game that I respect, but thoroughly loathe to play. Each round of this kill-the-other-guys-basketball-without-basketball, has you use the money you received from how well you performed in the previous round. It's a decidedly keyboard-based interface, but it doesn't have to be. It changes how each round plays, encourages friends to invent impromptu rules ("guns only, guys"), and doesn't really belong in the setting (why wouldn't these guys arrive at the situation already equipped?). Very clever, rarely copied. Kinda odd.