What are house rules that could make the game a lot more quicker, without spoiling the "skeleton" of the game?
To be honest, Yu-Gi-Oh is actually quite fast-paced, even more now with all the changes that have been to the format (and I am not the only one that thinks this is a thing). In some cases, if you or your opponent pull out a OTK or FTK the game could well last less than 5 minutes.
In official tournaments, the rulings indicate that rounds have a 40 minutes duration. Each round is played in a best-of-three Duels, meaning that you have about 13 minutes per game. We can see that Duels are already fast enough, and that the official rules consider time limits according to this average duration.
With this being said, we can see that one Duel should at most last 13 minutes, so it is reasonable that a single duel can well last 10 minutes or less. Perhaps a duel can take longer if there is debate or argument while playing it, as to the correct ruling of a card, or on legal aspect of some move.
In this case, to effectively speed your games it would help to reduce the time spent debating or arguing so you have more time to actually play. This can only be achieved if both parties understand and know the rules on a decent or high degree, so there is no room for misinterpretation. Also, the more you play with someone the better you know each other's decks and possible moves, thus you will be able to play your games more swiftly.
Now, if you and your friend(s) are still taking longer than that to play a single game there could be other options (besides being swifter on the debating of rules) you may consider:
As you already suggested, reducing Life Points to 4000 can effectively speed some duels. You even say that "Kind of removes the length of the game significantly", but isn't this what you are asking for?
A slight variant of this idea is to reduce LP to 2000 points, as it was done in the anime/manga on Duel Monsters. This way you games will be really swift on the current format, while not deviating much from the real game (actually, it would be even more like the anime).
Other option I once tried with some friends was to penalize a player that takes much time deciding their moves. The penalty was represented by a fixed deduction of LP (say 500?) each time that player stalls or takes much time to decide. Of course, this is better when done with a third person that can oversee the game as judge and call the penalties. This encouraged fast-play while actually capping each game if a player stalls too much.
Yet another option you got is to perhaps try playing those swift duels on an app or similar, like the well-known YGO Pro. This way you spare yourself from interpreting the rules and timings, so you "just" play and let the game automatically carry out the legal actions for you. Of course that YGO Pro and similar may have some bugs, bit if they do they are minimal.
Then, when you got more free time, you can play with your physical decks and take all the time you need to debate if a move or strategy is valid or not. As a final suggestion, sometimes it is better to postpone a game and play it when you have time to play it properly, without rushing and being able to enjoy it more.
